starting February 13, 2024 at 09:39:23 PM UTC
Creating downstairs regions
Creating region directory at: /var/tmp/dsc/region
Creating dsc directory at: /tmp/dsc
Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.37101352
Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.059144393
Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.069718294
Region set with 3 regions was created
Starting downstairs
Using existing output directory "/tmp/dsc"
start ds: 8810
start ds: 8820
start ds: 8830
start access at:127.0.0.1:9998
Starting downstairs at port 8820
Starting downstairs at port 8810
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8820.txt"
Make output file at "/tmp/dsc/downstairs-8810.txt"
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8820][1] reports Starting
[8810][0] reports Starting
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1154)
[8810][0] reports Running
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1155)
[8830][2] reports Running
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1156)
[8820][1] reports Running
Feb 13 21:39:24.098 INFO listening, local_addr: 127.0.0.1:9998
Control access at:127.0.0.1:9998
Running initial fill
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.113241199Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"29185f5538632a13039785e911d1aaa0146bfa14\",\n git_commit_timestamp: \"2024-02-13T21:13:12.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.75.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"82e1608dfa6e0b5569232559e3d385fea5a93112\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.113933792Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160}
{"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.113953072Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160}
{"msg":"Crucible f77c8ab7-7da0-4ccd-98ba-a2665eb8dff7 has session id: 35cc3fe0-4062-4755-abb9-91ba6b18b47b","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.113967021Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"Upstairs opts: Upstairs UUID: f77c8ab7-7da0-4ccd-98ba-a2665eb8dff7, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.11398487Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"Crucible stats registered with UUID: f77c8ab7-7da0-4ccd-98ba-a2665eb8dff7","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.11400126Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
Crucible runtime is spawned
{"msg":"The guest has requested activation with gen:1","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.11452701Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160}
{"msg":"f77c8ab7-7da0-4ccd-98ba-a2665eb8dff7 active request set","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.114663274Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.114681124Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.114696393Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.114715172Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.114729762Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.114743711Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.114757701Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.114841628Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"":"io task","client":"0","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.114860627Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"":"io task","client":"1","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.114877616Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"":"io task","client":"2","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.115255122Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"":"io task","client":"1","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.115291061Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"":"io task","client":"0","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.11531383Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"":"io task","client":"2","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.117842793Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.117875392Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.117903661Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.1179323Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.117960768Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.117990597Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.119204101Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.119351025Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.119398764Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.120404515Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.120443194Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.120521851Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.12053843Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.120553649Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.120567919Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.120580738Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.120594878Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"[0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.120729463Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"[0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.120748482Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.120886997Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"[1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.120910376Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"[1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.120924805Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.120939575Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"[2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.120953874Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"[2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.120967904Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.120982663Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"Max found gen is 1","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.120996212Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"Generation requested: 1 >= found:1","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.121010142Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"Next flush: 1","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.121023651Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.121037141Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"":"downstairs","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.121051Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.12106376Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"0","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.121077789Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.121094219Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"client":"2","":"downstairs","session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.121108338Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.121121208Z","Activate successful
hostname"Wait for a query_work_queue command to finish before sending IO
:"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"f77c8ab7-7da0-4ccd-98ba-a2665eb8dff7 is now active with session: 35cc3fe0-4062-4755-abb9-91ba6b18b47b","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.121140537Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.121153037Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160,"session_id":"35cc3fe0-4062-4755-abb9-91ba6b18b47b"}
{"msg":"The guest has finished waiting for activation with:1","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:29.121165666Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160}
Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
Fill test
Read and Verify all blocks (0..5000 range:false)
Wrote out file "alan"
CLIENT: Tests done. All submitted work has been ACK'd
----------------------------------------------------------------
Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:23
States: Active Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
80 Acked 1079 Read 100 Done Done Done false
81 Acked 1080 Read 100 Done Done Done false
82 Acked 1081 Read 100 Done Done Done false
83 Acked 1082 Read 100 Done Done Done false
84 Acked 1083 Read 100 Done Done Done false
85 Acked 1084 Read 100 Done Done Done false
86 Acked 1085 Read 100 Done Done Done false
87 Acked 1086 Read 100 Done Done Done false
88 Acked 1087 Read 100 Done Done Done false
89 Acked 1088 Read 100 Done Done Done false
90 Acked 1089 Read 100 Done Done Done false
91 Acked 1090 Read 100 Done Done Done false
92 Acked 1091 Read 100 Done Done Done false
93 Acked 1092 Read 100 Done Done Done false
94 Acked 1093 Read 100 Done Done Done false
95 Acked 1094 Read 100 Done Done Done false
96 Acked 1095 Read 100 Done Done Done false
97 Acked 1096 Read 100 Done Done Done false
98 Acked 1097 Read 100 Done Done Done false
99 Acked 1098 Read 100 Done Done Done false
100 Acked 1099 Read 100 Done Done Done false
101 Acked 1100 Read 100 Done Done Done false
102 Acked 1101 Read 100 Done Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
New 0 0 0 0
Sent 0 0 0 0
Done 23 23 23 69
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1078 1078 1078
Downstairs last five completed: 1078 1077 1076 1075 1074
Upstairs last five completed: 102 101 100 99 98
CLIENT: Up:0 ds:23 act:3
----------------------------------------------------------------
Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 1102 1101 1100 1099 1098
Upstairs last five completed: 103 102 101 100 99
CLIENT: Up:0 ds:0 act:3
CLIENT: All crucible jobs finished, exiting program
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:34.383627559Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1160}
Replay loop starts now February 13, 2024 at 09:39:34 PM UTC
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.408124943Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"29185f5538632a13039785e911d1aaa0146bfa14\",\n git_commit_timestamp: \"2024-02-13T21:13:12.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.75.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"82e1608dfa6e0b5569232559e3d385fea5a93112\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.408352814Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164}
{"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.408368984Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164}
{"msg":"Crucible 47490775-8a57-4654-a177-f4e45fd74727 has session id: 7daa0b95-273f-4ce8-935e-89c0961e151d","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.408382553Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Upstairs opts: Upstairs UUID: 47490775-8a57-4654-a177-f4e45fd74727, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.408402372Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Crucible stats registered with UUID: 47490775-8a57-4654-a177-f4e45fd74727","v":0,"name":"crucible","level":30,"time"Crucible runtime is spawned
:"2024-02-13T21:39:34.408416022Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"The guest has requested activation with gen:2","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.408508698Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164}
{"msg":"47490775-8a57-4654-a177-f4e45fd74727 active request set","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.408586745Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.408603975Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.408626714Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.408652982Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.408678961Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.408704301Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.40873503Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.408762368Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.408793907Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.408821256Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.408848705Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.408874794Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.409012429Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.409530389Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.409559838Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.409585757Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.409614126Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.409642205Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.409665634Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.410159655Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.41029205Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.410316479Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.41081832Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.410856418Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.410884977Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.410967464Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.410991933Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.411007583Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.411020902Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.411035031Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.411049341Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.411063621Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.41107847Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.411092789Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.411107249Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.411125678Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.411140358Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.411154647Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.411169537Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Max found gen is 2","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.411183036Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Generation requested: 2 >= found:2","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.411196625Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Next flush: 2"Activate successful
,Wait for a query_work_queue command to finish before sending IO
"v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.411214925Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.411228574Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.411242324Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.411254823Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.411269353Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.411283552Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.411297472Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.411310281Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"47490775-8a57-4654-a177-f4e45fd74727 is now active with session: 7daa0b95-273f-4ce8-935e-89c0961e151d","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.4113254Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.41133822Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"The guest has finished waiting for activation with:2","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.411351199Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164}
Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
Loading write count information from file "alan"
Feb 13 21:39:34.461 INFO accepted connection, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Main task has work to do, go find it
got dsc Stop(0)
stop 0
[8810] Got stop action so:false kr:false
Feb 13 21:39:34.462 INFO request completed, latency_us: 392, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 5650e2d3-0a39-4211-805f-6593673ec118, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:34.463365182Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:34.463485517Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:34.463517966Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.463532036Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.463630642Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:34.46366777Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:39:34.464 INFO request completed, latency_us: 172, response_code: 200, uri: /state/cid/0, method: GET, req_id: 6634af6d-3e5f-4cf7-a76e-34e80b7ae11b, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:35.465158222Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:35.465302236Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:35.465330175Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:35.465356254Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:35.465372863Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:35.465390513Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:35.465423361Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:36.46698968Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:36.467046058Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:36.467065977Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:36.467082686Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:36.467095536Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:36.467117695Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:36.467144904Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:37.467663033Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:37.467723981Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:37.467753579Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:37.467774488Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:37.467790518Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:37.467808237Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:37.467831426Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:39:38.466 INFO request completed, latency_us: 203, response_code: 200, uri: /state/cid/0, method: GET, req_id: 780e9b72-8c53-45a7-a6b5-e1a60accd79a, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Read block 283 len 28672
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:38.469245991Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:38.469312508Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:38.469342327Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:38.469367346Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:38.469386635Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 1 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:38.469401354Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:38.469587787Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
002/300 Read block 2140 len 12288
003/300 Write block 3237 len 4096 data: 2
004/300 Read block 3055 len 36864
005/300 Read block 2929 len 8192
006/300 Write block 4322 len 24576 data: 2 2 2 2 2 2
007/300 Read block 4545 len 12288
008/300 Read block 3482 len 20480
009/300 Write block 4939 len 32768 data: 2 2 2 2 2 2 2 2
010/300 Read block 4272 len 8192
011/300 Write block 920 len 36864 data: 2 2 2 2 2 2 2 2 2
012/300 Read block 900 len 8192
013/300 Read block 1531 len 24576
014/300 Write block 117 len 12288 data: 2 2 2
015/300 Write block 4536 len 28672 data: 2 2 2 2 2 2 2
016/300 Read block 2942 len 16384
017/300 Read block 2544 len 24576
018/300 Write block 531 len 40960 data: 2 2 2 2 2 2 2 2 2 2
019/300 Write block 2732 len 8192 data: 2 2
020/300 Read block 110 len 20480
021/300 Read block 2618 len 40960
022/300 Flush
023/300 Read block 2147 len 12288
024/300 Read block 1731 len 28672
025/300 Read block 2819 len 12288
026/300 Write block 2908 len 24576 data: 2 2 2 2 2 2
027/300 Read block 4375 len 8192
028/300 Write block 2508 len 36864 data: 2 2 2 2 2 2 2 2 2
029/300 Read block 4337 len 32768
030/300 Write block 3915 len 24576 data: 2 2 2 2 2 2
031/300 Read block 2276 len 8192
032/300 Write block 1618 len 16384 data: 2 2 2 2
033/300 Read block 2313 len 20480
034/300 Flush
035/300 Read block 797 len 8192
036/300 Flush
037/300 Write block 4379 len 24576 data: 2 2 2 2 2 2
038/300 Read block 4496 len 8192
039/300 Read block 3692 len 32768
040/300 Read block 488 len 40960
041/300 Read block 2403 len 24576
042/300 Read block 4091 len 24576
043/300 Write block 1605 len 36864 data: 2 2 2 2 2 2 2 2 2
044/300 Read block 847 len 20480
045/300 Write block 3683 len 28672 data: 2 2 2 2 2 2 2
046/300 Write block 4218 len 16384 data: 2 2 2 2
047/300 Read block 363 len 28672
048/300 Read block 4886 len 8192
049/300 Write block 428 len 8192 data: 2 2
050/300 Read block 3708 len 36864
051/300 Write block 1104 len 16384 data: 2 2 2 2
052/300 Read block 4272 len 36864
053/300 Read block 1331 len 12288
054/300 Read block 3056 len 8192
055/300 Write block 3408 len 32768 data: 2 2 2 2 2 2 2 2
056/300 Read block 4108 len 32768
057/300 Write block 4714 len 36864 data: 2 2 2 2 2 2 2 2 2
058/300 Write block 161 len 16384 data: 2 2 2 2
059/300 Write block 4665 len 28672 data: 2 2 2 2 2 2 2
060/300 Flush
061/300 Write block 2164 len 36864 data: 2 2 2 2 2 2 2 2 2
062/300 Read block 3156 len 36864
063/300 Read block 1046 len 36864
064/300 Flush
065/300 Read block 4608 len 24576
066/300 Read block 3181 len 36864
067/300 Write block 2239 len 36864 data: 2 2 2 2 2 2 2 2 2
068/300 Write block 991 len 8192 data: 2 2
069/300 Read block 473 len 16384
070/300 Read block 4201 len 36864
071/300 Write block 2003 len 12288 data: 2 2 2
072/300 Read block 1842 len 16384
073/300 Write block 3379 len 28672 data: 2 2 2 2 2 2 2
074/300 Write block 3655 len 16384 data: 2 2 2 2
075/300 Write block 129 len 28672 data: 2 2 2 2 2 2 2
076/300 Write block 2649 len 24576 data: 2 2 2 2 2 2
077/300 Read block 845 len 20480
078/300 Read block 4787 len 28672
079/300 Write block 2392 len 20480 data: 2 2 2 2 2
080/300 Write block 4383 len 24576 data: 3 3 2 2 2 2
081/300 Write block 54 len 36864 data: 2 2 2 2 2 2 2 2 2
082/300 Read block 645 len 12288
083/300 Read block 3705 len 16384
084/300 Read block 1199 len 12288
085/300 Flush
086/300 Write block 3546 len 28672 data: 2 2 2 2 2 2 2
087/300 Write block 4870 len 16384 data: 2 2 2 2
088/300 Write block 123 len 40960 data: 2 2 2 2 2 2 3 3 3 3
089/300 Write block 3643 len 4096 data: 2
090/300 Write block 633 len 36864 data: 2 2 2 2 2 2 2 2 2
091/300 Read block 806 len 32768
092/300 Read block 2446 len 8192
093/300 Write block 4057 len 36864 data: 2 2 2 2 2 2 2 2 2
094/300 Read block 1038 len 16384
095/300 Read block 1299 len 28672
096/300 Read block 1567 len 28672
097/300 Write block 1440 len 20480 data: 2 2 2 2 2
098/300 Write block 2442 len 28672 data: 2 2 2 2 2 2 2
099/300 Read block 545 len 36864
100/300 Flush
101/300 Write block 4859 len 40960 data: 2 2 2 2 2 2 2 2 2 2
102/300 Flush
103/300 Write block 309 len 16384 data: 2 2 2 2
104/300 Write block 1449 len 40960 data: 2 2 2 2 2 2 2 2 2 2
105/300 Write block 3306 len 4096 data: 2
106/300 Read block 3168 len 24576
107/300 Write block 1484 len 4096 data: 2
108/300 Read block 2483 len 8192
109/300 Read block 3707 len 28672
110/300 Read block 1594 len 8192
111/300 Read block 3915 len 16384
112/300 Read block 2299 len 32768
113/300 Read block 3404 len 32768
114/300 Flush
115/300 Read block 3996 len 12288
116/300 Write block 1352 len 4096 data: 2
117/300 Read block 1856 len 32768
118/300 Read block 1476 len 24576
119/300 Write block 1761 len 4096 data: 2
120/300 Read block 2235 len 24576
121/300 Write block 4591 len 40960 data: 2 2 2 2 2 2 2 2 2 2
122/300 Read block 4408 len 12288
123/300 Write block 4729 len 16384 data: 2 2 2 2
124/300 Write block 2644 len 36864 data: 2 2 2 2 2 3 3 3 3
125/300 Flush
126/300 Write block 300 len 24576 data: 2 2 2 2 2 2
127/300 Write block 2073 len 8192 data: 2 2
128/300 Write block 3740 len 28672 data: 2 2 2 2 2 2 2
129/300 Write block 447 len 28672 data: 2 2 2 2 2 2 2
130/300 Read block 1993 len 16384
131/300 Write block 522 len 28672 data: 2 2 2 2 2 2 2
132/300 Read block 2751 len 12288
133/300 Read block 757 len 20480
134/300 Write block 509 len 32768 data: 2 2 2 2 2 2 2 2
135/300 Write block 621 len 12288 data: 2 2 2
136/300 Write block 1135 len 8192 data: 2 2
137/300 Write block 465 len 16384 data: 2 2 2 2
138/300 Read block 3380 len 16384
139/300 Read block 4533 len 28672
140/300 Read block 552 len 36864
141/300 Read block 3364 len 12288
142/300 Read block 2158 len 20480
143/300 Write block 1568 len 40960 data: 2 2 2 2 2 2 2 2 2 2
144/300 Flush
145/300 Write block 4740 len 36864 data: 2 2 2 2 2 2 2 2 2
146/300 Write block 690 len 12288 data: 2 2 2
147/300 Write block 3623 len 4096 data: 2
148/300 Write block 4287 len 28672 data: 2 2 2 2 2 2 2
149/300 Read block 257 len 16384
150/300 Read block 4900 len 36864
151/300 Write block 3367 len 36864 data: 2 2 2 2 2 2 2 2 2
152/300 Write block 4448 len 24576 data: 2 2 2 2 2 2
153/300 Write block 3866 len 8192 data: 2 2
154/300 Flush
155/300 Read block 3728 len 32768
156/300 Read block 1275 len 12288
157/300 Read block 3559 len 12288
158/300 Read block 3665 len 32768
159/300 Read block 4270 len 16384
160/300 Flush
161/300 Write block 4441 len 28672 data: 2 2 2 2 2 2 2
162/300 Write block 3401 len 8192 data: 2 2
163/300 Write block 4 len 4096 data: 2
164/300 Write block 4260 len 24576 data: 2 2 2 2 2 2
165/300 Read block 1483 len 36864
166/300 Write block 4284 len 32768 data: 2 2 2 3 3 3 3 3
167/300 Read block 3927 len 28672
168/300 Write block 3620 len 4096 data: 2
169/300 Read block 4138 len 8192
170/300 Read block 4282 len 28672
171/300 Read block 3439 len 4096
172/300 Read block 1923 len 8192
173/300 Read block 3603 len 28672
174/300 Write block 2675 len 36864 data: 2 2 2 2 2 2 2 2 2
175/300 Flush
176/300 Read block 4478 len 28672
177/300 Read block 4887 len 12288
178/300 Read block 2158 len 40960
179/300 Read block 2633 len 40960
180/300 Read block 137 len 36864
181/300 Read block 3145 len 12288
182/300 Flush
183/300 Write block 3783 len 40960 data: 2 2 2 2 2 2 2 2 2 2
184/300 Write block 3219 len 40960 data: 2 2 2 2 2 2 2 2 2 2
185/300 Read block 609 len 16384
186/300 Read block 4219 len 36864
187/300 Flush
188/300 Write block 2566 len 40960 data: 2 2 2 2 2 2 2 2 2 2
189/300 Write block 3381 len 36864 data: 3 3 3 3 3 2 2 2 2
190/300 Read block 4097 len 12288
191/300 Write block 869 len 8192 data: 2 2
192/300 Read block 1850 len 12288
193/300 Read block 2095 len 20480
194/300 Write block 3675 len 8192 data: 2 2
195/300 Write block 1294 len 8192 data: 2 2
196/300 Write block 6 len 8192 data: 2 2
197/300 Write block 3148 len 16384 data: 2 2 2 2
198/300 Read block 2925 len 36864
199/300 Read block 284 len 20480
200/300 Write block 3387 len 16384 data: 3 3 3 2
201/300 Write block 3928 len 4096 data: 2
202/300 Write block 3785 len 12288 data: 3 3 3
203/300 Read block 3606 len 12288
204/300 Write block 4195 len 32768 data: 2 2 2 2 2 2 2 2
205/300 Write block 1452 len 20480 data: 3 3 3 3 3
206/300 Read block 3925 len 32768
207/300 Read block 4127 len 36864
208/300 Flush
209/300 Read block 3045 len 28672
210/300 Write block 3342 len 20480 data: 2 2 2 2 2
211/300 Write block 3343 len 12288 data: 3 3 3
212/300 Read block 4634 len 24576
213/300 Write block 4833 len 40960 data: 2 2 2 2 2 2 2 2 2 2
214/300 Read block 2933 len 4096
215/300 Write block 1766 len 16384 data: 2 2 2 2
216/300 Write block 373 len 4096 data: 2
217/300 Read block 3274 len 28672
218/300 Read block 4447 len 32768
219/300 Write block 3553 len 36864 data: 2 2 2 2 2 2 2 2 2
220/300 Flush
221/300 Read block 207 len 16384
222/300 Read block 4309 len 24576
223/300 Read block 3899 len 20480
224/300 Read block 4536 len 20480
225/300 Write block 2484 len 20480 data: 2 2 2 2 2
226/300 Write block 1785 len 4096 data: 2
227/300 Read block 3298 len 20480
228/300 Read block 1034 len 4096
229/300 Read block 672 len 36864
230/300 Write block 48 len 24576 data: 2 2 2 2 2 2
231/300 Read block 2385 len 32768
232/300 Write block 3002 len 8192 data: 2 2
233/300 Read block 954 len 8192
234/300 Read block 3554 len 8192
235/300 Read block 2864 len 36864
236/300 Write block 3281 len 28672 data: 2 2 2 2 2 2 2
237/300 Flush
238/300 Flush
239/300 Write block 2993 len 24576 data: 2 2 2 2 2 2
240/300 Write block 503 len 28672 data: 2 2 2 2 2 2 3
241/300 Write block 3895 len 36864 data: 2 2 2 2 2 2 2 2 2
242/300 Read block 1632 len 20480
243/300 Read block 3939 len 32768
244/300 Read block 2051 len 4096
245/300 Read block 3418 len 4096
246/300 Write block 2875 len 16384 data: 2 2 2 2
247/300 Write block 3105 len 28672 data: 2 2 2 2 2 2 2
248/300 Write block 1433 len 28672 data: 2 2 2 2 2 2 2
249/300 Write block 3616 len 40960 data: 2 2 2 2 3 2 2 3 2 2
250/300 Flush
251/300 Read block 4273 len 40960
252/300 Read block 1232 len 16384
253/300 Write block 4366 len 24576 data: 2 2 2 2 2 2
254/300 Flush
255/300 Write block 559 len 36864 data: 2 2 2 2 2 2 2 2 2
256/300 Flush
257/300 Write block 4580 len 4096 data: 2
258/300 Flush
259/300 Read block 3280 len 12288
260/300 Write block 3939 len 12288 data: 2 2 2
261/300 Write block 2879 len 4096 data: 2
262/300 Read block 3604 len 4096
263/300 Read block 611 len 4096
264/300 Write block 57 len 40960 data: 3 3 3 3 3 3 2 2 2 2
265/300 Write block 4406 len 12288 data: 2 2 2
266/300 Write block 238 len 16384 data: 2 2 2 2
267/300 Read block 608 len 32768
268/300 Read block 3062 len 20480
269/300 Write block 590 len 40960 data: 2 2 2 2 2 2 2 2 2 2
270/300 Write block 4476 len 24576 data: 2 2 2 2 2 2
271/300 Write block 1744 len 12288 data: 2 2 2
272/300 Read block 591 len 16384
273/300 Write block 19 len 8192 data: 2 2
274/300 Write block 1263 len 4096 data: 2
275/300 Write block 3768 len 32768 data: 2 2 2 2 2 2 2 2
276/300 Write block 2497 len 16384 data: 2 2 2 2
277/300 Write block 2221 len 20480 data: 2 2 2 2 2
278/300 Write block 710 len 32768 data: 2 2 2 2 2 2 2 2
279/300 Write block 1298 len 20480 data: 2 2 2 2 2
280/300 Write block 4286 len 16384 data: 3 4 4 4
281/300 Flush
282/300 Write block 1251 len 12288 data: 2 2 2
283/300 Write block 3717 len 4096 data: 2
284/300 Read block 2244 len 16384
285/300 Read block 1016 len 28672
286/300 Read block 4957 len 20480
287/300 Read block 4306 len 36864
288/300 Read block 1068 len 8192
289/300 Write block 123 len 40960 data: 3 3 3 3 3 3 4 4 4 4
290/300 Read block 4669 len 24576
291/300 Write block 3008 len 40960 data: 2 2 2 2 2 2 2 2 2 2
292/300 Read block 63 len 32768
293/300 Flush
294/300 Flush
295/300 Write block 4842 len 16384 data: 3 2 2 2
296/300 Read block 4472 len 8192
297/300 Write block 2158 len 32768 data: 2 2 2 2 2 2 3 3
298/300 Read block 2200 len 16384
299/300 Write block 2827 len 16384 data: 2 2 2 2
300/300 Read block 2319 len 8192
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 13 21:39:39.288 INFO request completed, latency_us: 204, response_code: 204, uri: /start/cid/0, method: POST, req_id: 553df2a6-40f3-438d-8a9b-2ccf035f5fbb, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
1 Acked 1000 Read 7 New Done Done true
2 Acked 1001 Read 3 New Done Done false
3 Acked 1002 Write 1 New Done Done false
4 Acked 1003 Read 9 New Done Done false
5 Acked 1004 Read 2 New Done Done false
6 Acked 1005 Write 6 New Done Done false
7 Acked 1006 Read 3 New Done Done false
8 Acked 1007 Read 5 New Done Done false
9 Acked 1008 Write 8 New Done Done false
10 Acked 1009 Read 2 New Done Done false
11 Acked 1010 Write 9 New Done Done false
12 Acked 1011 Read 2 New Done Done false
13 Acked 1012 Read 6 New Done Done false
14 Acked 1013 Write 3 New Done Done false
15 Acked 1014 Write 7 New Done Done false
16 Acked 1015 Read 4 New Done Done false
17 Acked 1016 Read 6 New Done Done false
18 Acked 1017 Write 10 New Done Done false
19 Acked 1018 Write 2 New Done Done false
20 Acked 1019 Read 5 New Done Done false
21 Acked 1020 Read 10 New Done Done false
22 Acked 1021 Flush 0 New Done Done false
23 Acked 1022 Read 3 New Done Done false
24 Acked 1023 Read 7 New Done Done false
25 Acked 1024 Read 3 New Done Done false
26 Acked 1025 Write 6 New Done Done false
27 Acked 1026 Read 2 New Done Done false
28 Acked 1027 Write 9 New Done Done false
29 Acked 1028 Read 8 New Done Done false
30 Acked 1029 Write 6 New Done Done false
31 Acked 1030 Read 2 New Done Done false
32 Acked 1031 Write 4 New Done Done false
33 Acked 1032 Read 5 New Done Done false
34 Acked 1033 Flush 0 New Done Done false
35 Acked 1034 Read 2 New Done Done false
36 Acked 1035 Flush 0 New Done Done false
37 Acked 1036 Write 6 New Done Done false
38 Acked 1037 Read 2 New Done Done false
39 Acked 1038 Read 8 New Done Done false
40 Acked 1039 Read 10 New Done Done false
41 Acked 1040 Read 6 New Done Done false
42 Acked 1041 Read 6 New Done Done false
43 Acked 1042 Write 9 New Done Done false
44 Acked 1043 Read 5 New Done Done false
45 Acked 1044 Write 7 New Done Done false
46 Acked 1045 Write 4 New Done Done false
47 Acked 1046 Read 7 New Done Done false
48 Acked 1047 Read 2 New Done Done false
49 Acked 1048 Write 2 New Done Done false
50 Acked 1049 Read 9 New Done Done false
51 Acked 1050 Write 4 New Done Done false
52 Acked 1051 Read 9 New Done Done false
53 Acked 1052 Read 3 New Done Done false
54 Acked 1053 Read 2 New Done Done false
55 Acked 1054 Write 8 New Done Done false
56 Acked 1055 Read 8 New Done Done false
57 Acked 1056 Write 9 New Done Done false
58 Acked 1057 Write 4 New Done Done false
59 Acked 1058 Write 7 New Done Done false
60 Acked 1059 Flush 0 New Done Done false
61 Acked 1060 Write 9 New Done Done false
62 Acked 1061 Read 9 New Done Done false
63 Acked 1062 Read 9 New Done Done false
64 Acked 1063 Flush 0 New Done Done false
65 Acked 1064 Read 6 New Done Done false
66 Acked 1065 Read 9 New Done Done false
67 Acked 1066 Write 9 New Done Done false
68 Acked 1067 Write 2 New Done Done false
69 Acked 1068 Read 4 New Done Done false
70 Acked 1069 Read 9 New Done Done false
71 Acked 1070 Write 3 New Done Done false
72 Acked 1071 Read 4 New Done Done false
73 Acked 1072 Write 7 New Done Done false
74 Acked 1073 Write 4 New Done Done false
75 Acked 1074 Write 7 New Done Done false
76 Acked 1075 Write 6 New Done Done false
77 Acked 1076 Read 5 New Done Done false
78 Acked 1077 Read 7 New Done Done false
79 Acked 1078 Write 5 New Done Done false
80 Acked 1079 Write 6 New Done Done false
81 Acked 1080 Write 9 New Done Done false
82 Acked 1081 Read 3 New Done Done false
83 Acked 1082 Read 4 New Done Done false
84 Acked 1083 Read 3 New Done Done false
85 Acked 1084 Flush 0 New Done Done false
86 Acked 1085 Write 7 New Done Done false
87 Acked 1086 Write 4 New Done Done false
88 Acked 1087 Write 10 New Done Done false
89 Acked 1088 Write 1 New Done Done false
90 Acked 1089 Write 9 New Done Done false
91 Acked 1090 Read 8 New Done Done false
92 Acked 1091 Read 2 New Done Done false
93 Acked 1092 Write 9 New Done Done false
94 Acked 1093 Read 4 New Done Done false
95 Acked 1094 Read 7 New Done Done false
96 Acked 1095 Read 7 New Done Done false
97 Acked 1096 Write 5 New Done Done false
98 Acked 1097 Write 7 New Done Done false
99 Acked 1098 Read 9 New Done Done false
100 Acked 1099 Flush 0 New Done Done false
101 Acked 1100 Write 10 New Done Done false
102 Acked 1101 Flush 0 New Done Done false
103 Acked 1102 Write 4 New Done Done false
104 Acked 1103 Write 10 New Done Done false
105 Acked 1104 Write 1 New Done Done false
106 Acked 1105 Read 6 New Done Done false
107 Acked 1106 Write 1 New Done Done false
108 Acked 1107 Read 2 New Done Done false
109 Acked 1108 Read 7 New Done Done false
110 Acked 1109 Read 2 New Done Done false
111 Acked 1110 Read 4 New Done Done false
112 Acked 1111 Read 8 New Done Done false
113 Acked 1112 Read 8 New Done Done false
114 Acked 1113 Flush 0 New Done Done false
115 Acked 1114 Read 3 New Done Done false
116 Acked 1115 Write 1 New Done Done false
117 Acked 1116 Read 8 New Done Done false
118 Acked 1117 Read 6 New Done Done false
119 Acked 1118 Write 1 New Done Done false
120 Acked 1119 Read 6 New Done Done false
121 Acked 1120 Write 10 New Done Done false
122 Acked 1121 Read 3 New Done Done false
123 Acked 1122 Write 4 New Done Done false
124 Acked 1123 Write 9 New Done Done false
125 Acked 1124 Flush 0 New Done Done false
126 Acked 1125 Write 6 New Done Done false
127 Acked 1126 Write 2 New Done Done false
128 Acked 1127 Write 7 New Done Done false
129 Acked 1128 Write 7 New Done Done false
130 Acked 1129 Read 4 New Done Done false
131 Acked 1130 Write 7 New Done Done false
132 Acked 1131 Read 3 New Done Done false
133 Acked 1132 Read 5 New Done Done false
134 Acked 1133 Write 8 New Done Done false
135 Acked 1134 Write 3 New Done Done false
136 Acked 1135 Write 2 New Done Done false
137 Acked 1136 Write 4 New Done Done false
138 Acked 1137 Read 4 New Done Done false
139 Acked 1138 Read 7 New Done Done false
140 Acked 1139 Read 9 New Done Done false
141 Acked 1140 Read 3 New Done Done false
142 Acked 1141 Read 5 New Done Done false
143 Acked 1142 Write 10 New Done Done false
144 Acked 1143 Flush 0 New Done Done false
145 Acked 1144 Write 9 New Done Done false
146 Acked 1145 Write 3 New Done Done false
147 Acked 1146 Write 1 New Done Done false
148 Acked 1147 Write 7 New Done Done false
149 Acked 1148 Read 4 New Done Done false
150 Acked 1149 Read 9 New Done Done false
151 Acked 1150 Write 9 New Done Done false
152 Acked 1151 Write 6 New Done Done false
153 Acked 1152 Write 2 New Done Done false
154 Acked 1153 Flush 0 New Done Done false
155 Acked 1154 Read 8 New Done Done false
156 Acked 1155 Read 3 New Done Done false
157 Acked 1156 Read 3 New Done Done false
158 Acked 1157 Read 8 New Done Done false
159 Acked 1158 Read 4 New Done Done false
160 Acked 1159 Flush 0 New Done Done false
161 Acked 1160 Write 7 New Done Done false
162 Acked 1161 Write 2 New Done Done false
163 Acked 1162 Write 1 New Done Done false
164 Acked 1163 Write 6 New Done Done false
165 Acked 1164 Read 9 New Done Done false
166 Acked 1165 Write 8 New Done Done false
167 Acked 1166 Read 7 New Done Done false
168 Acked 1167 Write 1 New Done Done false
169 Acked 1168 Read 2 New Done Done false
170 Acked 1169 Read 7 New Done Done false
171 Acked 1170 Read 1 New Done Done false
172 Acked 1171 Read 2 New Done Done false
173 Acked 1172 Read 7 New Done Done false
174 Acked 1173 Write 9 New Done Done false
175 Acked 1174 Flush 0 New Done Done false
176 Acked 1175 Read 7 New Done Done false
177 Acked 1176 Read 3 New Done Done false
178 Acked 1177 Read 10 New Done Done false
179 Acked 1178 Read 10 New Done Done false
180 Acked 1179 Read 9 New Done Done false
181 Acked 1180 Read 3 New Done Done false
182 Acked 1181 Flush 0 New Done Done false
183 Acked 1182 Write 10 New Done Done false
184 Acked 1183 Write 10 New Done Done false
185 Acked 1184 Read 4 New Done Done false
186 Acked 1185 Read 9 New Done Done false
187 Acked 1186 Flush 0 New Done Done false
188 Acked 1187 Write 10 New Done Done false
189 Acked 1188 Write 9 New Done Done false
190 Acked 1189 Read 3 New Done Done false
191 Acked 1190 Write 2 New Done Done false
192 Acked 1191 Read 3 New Done Done false
193 Acked 1192 Read 5 New Done Done false
194 Acked 1193 Write 2 New Done Done false
195 Acked 1194 Write 2 New Done Done false
196 Acked 1195 Write 2 New Done Done false
197 Acked 1196 Write 4 New Done Done false
198 Acked 1197 Read 9 New Done Done false
199 Acked 1198 Read 5 New Done Done false
200 Acked 1199 Write 4 New Done Done false
201 Acked 1200 Write 1 New Done Done false
202 Acked 1201 Write 3 New Done Done false
203 Acked 1202 Read 3 New Done Done false
204 Acked 1203 Write 8 New Done Done false
205 Acked 1204 Write 5 New Done Done false
206 Acked 1205 Read 8 New Done Done false
207 Acked 1206 Read 9 New Done Done false
208 Acked 1207 Flush 0 New Done Done false
209 Acked 1208 Read 7 New Done Done false
210 Acked 1209 Write 5 New Done Done false
211 Acked 1210 Write 3 New Done Done false
212 Acked 1211 Read 6 New Done Done false
213 Acked 1212 Write 10 New Done Done false
214 Acked 1213 Read 1 New Done Done false
215 Acked 1214 Write 4 New Done Done false
216 Acked 1215 Write 1 New Done Done false
217 Acked 1216 Read 7 New Done Done false
218 Acked 1217 Read 8 New Done Done false
219 Acked 1218 Write 9 New Done Done false
220 Acked 1219 Flush 0 New Done Done false
221 Acked 1220 Read 4 New Done Done false
222 Acked 1221 Read 6 New Done Done false
223 Acked 1222 Read 5 New Done Done false
224 Acked 1223 Read 5 New Done Done false
225 Acked 1224 Write 5 New Done Done false
226 Acked 1225 Write 1 New Done Done false
227 Acked 1226 Read 5 New Done Done false
228 Acked 1227 Read 1 New Done Done false
229 Acked 1228 Read 9 New Done Done false
230 Acked 1229 Write 6 New Done Done false
231 Acked 1230 Read 8 New Done Done false
232 Acked 1231 Write 2 New Done Done false
233 Acked 1232 Read 2 New Done Done false
234 Acked 1233 Read 2 New Done Done false
235 Acked 1234 Read 9 New Done Done false
236 Acked 1235 Write 7 New Done Done false
237 Acked 1236 Flush 0 New Done Done false
238 Acked 1237 Flush 0 New Done Done false
239 Acked 1238 Write 6 New Done Done false
240 Acked 1239 Write 7 New Done Done false
241 Acked 1240 Write 9 New Done Done false
242 Acked 1241 Read 5 New Done Done false
243 Acked 1242 Read 8 New Done Done false
244 Acked 1243 Read 1 New Done Done false
245 Acked 1244 Read 1 New Done Done false
246 Acked 1245 Write 4 New Done Done false
247 Acked 1246 Write 7 New Done Done false
248 Acked 1247 Write 7 New Done Done false
249 Acked 1248 Write 10 New Done Done false
250 Acked 1249 Flush 0 New Done Done false
251 Acked 1250 Read 10 New Done Done false
252 Acked 1251 Read 4 New Done Done false
253 Acked 1252 Write 6 New Done Done false
254 Acked 1253 Flush 0 New Done Done false
255 Acked 1254 Write 9 New Done Done false
256 Acked 1255 Flush 0 New Done Done false
257 Acked 1256 Write 1 New Done Done false
258 Acked 1257 Flush 0 New Done Done false
259 Acked 1258 Read 3 New Done Done false
260 Acked 1259 Write 3 New Done Done false
261 Acked 1260 Write 1 New Done Done false
262 Acked 1261 Read 1 New Done Done false
263 Acked 1262 Read 1 New Done Done false
264 Acked 1263 Write 10 New Done Done false
265 Acked 1264 Write 3 New Done Done false
266 Acked 1265 Write 4 New Done Done false
267 Acked 1266 Read 8 New Done Done false
268 Acked 1267 Read 5 New Done Done false
269 Acked 1268 Write 10 New Done Done false
270 Acked 1269 Write 6 New Done Done false
271 Acked 1270 Write 3 New Done Done false
272 Acked 1271 Read 4 New Done Done false
273 Acked 1272 Write 2 New Done Done false
274 Acked 1273 Write 1 New Done Done false
275 Acked 1274 Write 8 New Done Done false
276 Acked 1275 Write 4 New Done Done false
277 Acked 1276 Write 5 New Done Done false
278 Acked 1277 Write 8 New Done Done false
279 Acked 1278 Write 5 New Done Done false
280 Acked 1279 Write 4 New Done Done false
281 Acked 1280 Flush 0 New Done Done false
282 Acked 1281 Write 3 New Done Done false
283 Acked 1282 Write 1 New Done Done false
284 Acked 1283 Read 4 New Done Done false
285 Acked 1284 Read 7 New Done Done false
286 Acked 1285 Read 5 New Done Done false
287 Acked 1286 Read 9 New Done Done false
288 Acked 1287 Read 2 New Done Done false
289 Acked 1288 Write 10 New Done Done false
290 Acked 1289 Read 6 New Done Done false
291 Acked 1290 Write 10 New Done Done false
292 Acked 1291 Read 8 New Done Done false
293 Acked 1292 Flush 0 New Done Done false
294 Acked 1293 Flush 0 New Done Done false
295 Acked 1294 Write 4 New Done Done false
296 Acked 1295 Read 2 New Done Done false
297 Acked 1296 Write 8 New Done Done false
298 Acked 1297 Read 4 New Done Done false
299 Acked 1298 Write 4 New Done Done false
300 Acked 1299 Read 2 New Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
New 300 0 0 300
Sent 0 0 0 0
Done 0 300 300 600
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 0 1293 1293
Downstairs last five completed:
Upstairs last five completed: 300 299 298 297 296
CLIENT: Up:0 ds:300 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:39.470718093Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:39.470811509Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:39.470854838Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:39.470894356Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:39.470921475Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:39.470951064Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:39.471017472Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:40.471902406Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:40.471979993Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:40.472001582Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:40.472022791Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:40.472047841Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:40.472078419Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:40.472206564Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:41.472738113Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:41.472793071Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:41.47281736Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:41.472840049Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:41.472855999Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:41.472873358Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:41.473018532Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:42.473586719Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:42.473644937Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:42.473665936Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:42.473686205Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:42.473708674Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:42.473726643Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:42.473840839Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
1 Acked 1000 Read 7 New Done Done true
2 Acked 1001 Read 3 New Done Done true
3 Acked 1002 Write 1 New Done Done true
4 Acked 1003 Read 9 New Done Done true
5 Acked 1004 Read 2 New Done Done true
6 Acked 1005 Write 6 New Done Done true
7 Acked 1006 Read 3 New Done Done true
8 Acked 1007 Read 5 New Done Done true
9 Acked 1008 Write 8 New Done Done true
10 Acked 1009 Read 2 New Done Done true
11 Acked 1010 Write 9 New Done Done true
12 Acked 1011 Read 2 New Done Done true
13 Acked 1012 Read 6 New Done Done true
14 Acked 1013 Write 3 New Done Done true
15 Acked 1014 Write 7 New Done Done true
16 Acked 1015 Read 4 New Done Done true
17 Acked 1016 Read 6 New Done Done true
18 Acked 1017 Write 10 New Done Done true
19 Acked 1018 Write 2 New Done Done true
20 Acked 1019 Read 5 New Done Done true
21 Acked 1020 Read 10 New Done Done true
22 Acked 1021 Flush 0 New Done Done true
23 Acked 1022 Read 3 New Done Done true
24 Acked 1023 Read 7 New Done Done true
25 Acked 1024 Read 3 New Done Done true
26 Acked 1025 Write 6 New Done Done true
27 Acked 1026 Read 2 New Done Done true
28 Acked 1027 Write 9 New Done Done true
29 Acked 1028 Read 8 New Done Done true
30 Acked 1029 Write 6 New Done Done true
31 Acked 1030 Read 2 New Done Done true
32 Acked 1031 Write 4 New Done Done true
33 Acked 1032 Read 5 New Done Done true
34 Acked 1033 Flush 0 New Done Done true
35 Acked 1034 Read 2 New Done Done true
36 Acked 1035 Flush 0 New Done Done true
37 Acked 1036 Write 6 New Done Done true
38 Acked 1037 Read 2 New Done Done true
39 Acked 1038 Read 8 New Done Done true
40 Acked 1039 Read 10 New Done Done true
41 Acked 1040 Read 6 New Done Done true
42 Acked 1041 Read 6 New Done Done true
43 Acked 1042 Write 9 New Done Done true
44 Acked 1043 Read 5 New Done Done true
45 Acked 1044 Write 7 New Done Done true
46 Acked 1045 Write 4 New Done Done true
47 Acked 1046 Read 7 New Done Done true
48 Acked 1047 Read 2 New Done Done true
49 Acked 1048 Write 2 New Done Done true
50 Acked 1049 Read 9 New Done Done true
51 Acked 1050 Write 4 New Done Done true
52 Acked 1051 Read 9 New Done Done true
53 Acked 1052 Read 3 New Done Done true
54 Acked 1053 Read 2 New Done Done true
55 Acked 1054 Write 8 New Done Done true
56 Acked 1055 Read 8 New Done Done true
57 Acked 1056 Write 9 New Done Done true
58 Acked 1057 Write 4 New Done Done true
59 Acked 1058 Write 7 New Done Done true
60 Acked 1059 Flush 0 New Done Done true
61 Acked 1060 Write 9 New Done Done true
62 Acked 1061 Read 9 New Done Done true
63 Acked 1062 Read 9 New Done Done true
64 Acked 1063 Flush 0 New Done Done true
65 Acked 1064 Read 6 New Done Done true
66 Acked 1065 Read 9 New Done Done true
67 Acked 1066 Write 9 New Done Done true
68 Acked 1067 Write 2 New Done Done true
69 Acked 1068 Read 4 New Done Done true
70 Acked 1069 Read 9 New Done Done true
71 Acked 1070 Write 3 New Done Done true
72 Acked 1071 Read 4 New Done Done true
73 Acked 1072 Write 7 New Done Done true
74 Acked 1073 Write 4 New Done Done true
75 Acked 1074 Write 7 New Done Done true
76 Acked 1075 Write 6 New Done Done true
77 Acked 1076 Read 5 New Done Done true
78 Acked 1077 Read 7 New Done Done true
79 Acked 1078 Write 5 New Done Done true
80 Acked 1079 Write 6 New Done Done true
81 Acked 1080 Write 9 New Done Done true
82 Acked 1081 Read 3 New Done Done true
83 Acked 1082 Read 4 New Done Done true
84 Acked 1083 Read 3 New Done Done true
85 Acked 1084 Flush 0 New Done Done true
86 Acked 1085 Write 7 New Done Done true
87 Acked 1086 Write 4 New Done Done true
88 Acked 1087 Write 10 New Done Done true
89 Acked 1088 Write 1 New Done Done true
90 Acked 1089 Write 9 New Done Done true
91 Acked 1090 Read 8 New Done Done true
92 Acked 1091 Read 2 New Done Done true
93 Acked 1092 Write 9 New Done Done true
94 Acked 1093 Read 4 New Done Done true
95 Acked 1094 Read 7 New Done Done true
96 Acked 1095 Read 7 New Done Done true
97 Acked 1096 Write 5 New Done Done true
98 Acked 1097 Write 7 New Done Done true
99 Acked 1098 Read 9 New Done Done true
100 Acked 1099 Flush 0 New Done Done true
101 Acked 1100 Write 10 New Done Done true
102 Acked 1101 Flush 0 New Done Done true
103 Acked 1102 Write 4 New Done Done true
104 Acked 1103 Write 10 New Done Done true
105 Acked 1104 Write 1 New Done Done true
106 Acked 1105 Read 6 New Done Done true
107 Acked 1106 Write 1 New Done Done true
108 Acked 1107 Read 2 New Done Done true
109 Acked 1108 Read 7 New Done Done true
110 Acked 1109 Read 2 New Done Done true
111 Acked 1110 Read 4 New Done Done true
112 Acked 1111 Read 8 New Done Done true
113 Acked 1112 Read 8 New Done Done true
114 Acked 1113 Flush 0 New Done Done true
115 Acked 1114 Read 3 New Done Done true
116 Acked 1115 Write 1 New Done Done true
117 Acked 1116 Read 8 New Done Done true
118 Acked 1117 Read 6 New Done Done true
119 Acked 1118 Write 1 New Done Done true
120 Acked 1119 Read 6 New Done Done true
121 Acked 1120 Write 10 New Done Done true
122 Acked 1121 Read 3 New Done Done true
123 Acked 1122 Write 4 New Done Done true
124 Acked 1123 Write 9 New Done Done true
125 Acked 1124 Flush 0 New Done Done true
126 Acked 1125 Write 6 New Done Done true
127 Acked 1126 Write 2 New Done Done true
128 Acked 1127 Write 7 New Done Done true
129 Acked 1128 Write 7 New Done Done true
130 Acked 1129 Read 4 New Done Done true
131 Acked 1130 Write 7 New Done Done true
132 Acked 1131 Read 3 New Done Done true
133 Acked 1132 Read 5 New Done Done true
134 Acked 1133 Write 8 New Done Done true
135 Acked 1134 Write 3 New Done Done true
136 Acked 1135 Write 2 New Done Done true
137 Acked 1136 Write 4 New Done Done true
138 Acked 1137 Read 4 New Done Done true
139 Acked 1138 Read 7 New Done Done true
140 Acked 1139 Read 9 New Done Done true
141 Acked 1140 Read 3 New Done Done true
142 Acked 1141 Read 5 New Done Done true
143 Acked 1142 Write 10 New Done Done true
144 Acked 1143 Flush 0 New Done Done true
145 Acked 1144 Write 9 New Done Done true
146 Acked 1145 Write 3 New Done Done true
147 Acked 1146 Write 1 New Done Done true
148 Acked 1147 Write 7 New Done Done true
149 Acked 1148 Read 4 New Done Done true
150 Acked 1149 Read 9 New Done Done true
151 Acked 1150 Write 9 New Done Done true
152 Acked 1151 Write 6 New Done Done true
153 Acked 1152 Write 2 New Done Done true
154 Acked 1153 Flush 0 New Done Done true
155 Acked 1154 Read 8 New Done Done true
156 Acked 1155 Read 3 New Done Done true
157 Acked 1156 Read 3 New Done Done true
158 Acked 1157 Read 8 New Done Done true
159 Acked 1158 Read 4 New Done Done true
160 Acked 1159 Flush 0 New Done Done true
161 Acked 1160 Write 7 New Done Done true
162 Acked 1161 Write 2 New Done Done true
163 Acked 1162 Write 1 New Done Done true
164 Acked 1163 Write 6 New Done Done true
165 Acked 1164 Read 9 New Done Done true
166 Acked 1165 Write 8 New Done Done true
167 Acked 1166 Read 7 New Done Done true
168 Acked 1167 Write 1 New Done Done true
169 Acked 1168 Read 2 New Done Done true
170 Acked 1169 Read 7 New Done Done true
171 Acked 1170 Read 1 New Done Done true
172 Acked 1171 Read 2 New Done Done true
173 Acked 1172 Read 7 New Done Done true
174 Acked 1173 Write 9 New Done Done true
175 Acked 1174 Flush 0 New Done Done true
176 Acked 1175 Read 7 New Done Done true
177 Acked 1176 Read 3 New Done Done true
178 Acked 1177 Read 10 New Done Done true
179 Acked 1178 Read 10 New Done Done true
180 Acked 1179 Read 9 New Done Done true
181 Acked 1180 Read 3 New Done Done true
182 Acked 1181 Flush 0 New Done Done true
183 Acked 1182 Write 10 New Done Done true
184 Acked 1183 Write 10 New Done Done true
185 Acked 1184 Read 4 New Done Done true
186 Acked 1185 Read 9 New Done Done true
187 Acked 1186 Flush 0 New Done Done true
188 Acked 1187 Write 10 New Done Done true
189 Acked 1188 Write 9 New Done Done true
190 Acked 1189 Read 3 New Done Done true
191 Acked 1190 Write 2 New Done Done true
192 Acked 1191 Read 3 New Done Done true
193 Acked 1192 Read 5 New Done Done true
194 Acked 1193 Write 2 New Done Done true
195 Acked 1194 Write 2 New Done Done true
196 Acked 1195 Write 2 New Done Done true
197 Acked 1196 Write 4 New Done Done true
198 Acked 1197 Read 9 New Done Done true
199 Acked 1198 Read 5 New Done Done true
200 Acked 1199 Write 4 New Done Done true
201 Acked 1200 Write 1 New Done Done true
202 Acked 1201 Write 3 New Done Done true
203 Acked 1202 Read 3 New Done Done true
204 Acked 1203 Write 8 New Done Done true
205 Acked 1204 Write 5 New Done Done true
206 Acked 1205 Read 8 New Done Done true
207 Acked 1206 Read 9 New Done Done true
208 Acked 1207 Flush 0 New Done Done true
209 Acked 1208 Read 7 New Done Done true
210 Acked 1209 Write 5 New Done Done true
211 Acked 1210 Write 3 New Done Done true
212 Acked 1211 Read 6 New Done Done true
213 Acked 1212 Write 10 New Done Done true
214 Acked 1213 Read 1 New Done Done true
215 Acked 1214 Write 4 New Done Done true
216 Acked 1215 Write 1 New Done Done true
217 Acked 1216 Read 7 New Done Done true
218 Acked 1217 Read 8 New Done Done true
219 Acked 1218 Write 9 New Done Done true
220 Acked 1219 Flush 0 New Done Done true
221 Acked 1220 Read 4 New Done Done true
222 Acked 1221 Read 6 New Done Done true
223 Acked 1222 Read 5 New Done Done true
224 Acked 1223 Read 5 New Done Done true
225 Acked 1224 Write 5 New Done Done true
226 Acked 1225 Write 1 New Done Done true
227 Acked 1226 Read 5 New Done Done true
228 Acked 1227 Read 1 New Done Done true
229 Acked 1228 Read 9 New Done Done true
230 Acked 1229 Write 6 New Done Done true
231 Acked 1230 Read 8 New Done Done true
232 Acked 1231 Write 2 New Done Done true
233 Acked 1232 Read 2 New Done Done true
234 Acked 1233 Read 2 New Done Done true
235 Acked 1234 Read 9 New Done Done true
236 Acked 1235 Write 7 New Done Done true
237 Acked 1236 Flush 0 New Done Done true
238 Acked 1237 Flush 0 New Done Done true
239 Acked 1238 Write 6 New Done Done true
240 Acked 1239 Write 7 New Done Done true
241 Acked 1240 Write 9 New Done Done true
242 Acked 1241 Read 5 New Done Done true
243 Acked 1242 Read 8 New Done Done true
244 Acked 1243 Read 1 New Done Done true
245 Acked 1244 Read 1 New Done Done true
246 Acked 1245 Write 4 New Done Done true
247 Acked 1246 Write 7 New Done Done true
248 Acked 1247 Write 7 New Done Done true
249 Acked 1248 Write 10 New Done Done true
250 Acked 1249 Flush 0 New Done Done true
251 Acked 1250 Read 10 New Done Done true
252 Acked 1251 Read 4 New Done Done true
253 Acked 1252 Write 6 New Done Done true
254 Acked 1253 Flush 0 New Done Done true
255 Acked 1254 Write 9 New Done Done true
256 Acked 1255 Flush 0 New Done Done true
257 Acked 1256 Write 1 New Done Done true
258 Acked 1257 Flush 0 New Done Done true
259 Acked 1258 Read 3 New Done Done true
260 Acked 1259 Write 3 New Done Done true
261 Acked 1260 Write 1 New Done Done true
262 Acked 1261 Read 1 New Done Done true
263 Acked 1262 Read 1 New Done Done true
264 Acked 1263 Write 10 New Done Done true
265 Acked 1264 Write 3 New Done Done true
266 Acked 1265 Write 4 New Done Done true
267 Acked 1266 Read 8 New Done Done true
268 Acked 1267 Read 5 New Done Done true
269 Acked 1268 Write 10 New Done Done true
270 Acked 1269 Write 6 New Done Done true
271 Acked 1270 Write 3 New Done Done true
272 Acked 1271 Read 4 New Done Done true
273 Acked 1272 Write 2 New Done Done true
274 Acked 1273 Write 1 New Done Done true
275 Acked 1274 Write 8 New Done Done true
276 Acked 1275 Write 4 New Done Done true
277 Acked 1276 Write 5 New Done Done true
278 Acked 1277 Write 8 New Done Done true
279 Acked 1278 Write 5 New Done Done true
280 Acked 1279 Write 4 New Done Done true
281 Acked 1280 Flush 0 New Done Done true
282 Acked 1281 Write 3 New Done Done true
283 Acked 1282 Write 1 New Done Done true
284 Acked 1283 Read 4 New Done Done true
285 Acked 1284 Read 7 New Done Done true
286 Acked 1285 Read 5 New Done Done true
287 Acked 1286 Read 9 New Done Done true
288 Acked 1287 Read 2 New Done Done true
289 Acked 1288 Write 10 New Done Done true
290 Acked 1289 Read 6 New Done Done true
291 Acked 1290 Write 10 New Done Done true
292 Acked 1291 Read 8 New Done Done true
293 Acked 1292 Flush 0 New Done Done true
294 Acked 1293 Flush 0 New Done Done true
295 Acked 1294 Write 4 New Done Done true
296 Acked 1295 Read 2 New Done Done true
297 Acked 1296 Write 8 New Done Done true
298 Acked 1297 Read 4 New Done Done true
299 Acked 1298 Write 4 New Done Done true
300 Acked 1299 Read 2 New Done Done true
301 Acked 1300 Flush 0 New Done Done true
STATES DS:0 DS:1 DS:2 TOTAL
New 301 0 0 301
Sent 0 0 0 0
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 0 1300 1300
Downstairs last five completed:
Upstairs last five completed: 301 300 299 298 297
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:43.474397137Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:43.474455704Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:43.474477003Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:43.474497493Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:43.474513172Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:43.474530621Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:43.474650647Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:44.47557284Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:44.475628958Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:44.475652007Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:44.475672596Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:44.475688475Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:44.475705805Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:44.47582125Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:45.477276664Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:45.477329191Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:45.477350691Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:45.47737118Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:45.477387129Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:45.477404528Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:45.477518954Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1165)
[8810][0] reports Running
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:46.479124171Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:46.479174009Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:46.479192779Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:46.479209608Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:46.479224147Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 0","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:46.479248966Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:46.479346302Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
1 Acked 1000 Read 7 New Done Done true
2 Acked 1001 Read 3 New Done Done true
3 Acked 1002 Write 1 New Done Done true
4 Acked 1003 Read 9 New Done Done true
5 Acked 1004 Read 2 New Done Done true
6 Acked 1005 Write 6 New Done Done true
7 Acked 1006 Read 3 New Done Done true
8 Acked 1007 Read 5 New Done Done true
9 Acked 1008 Write 8 New Done Done true
10 Acked 1009 Read 2 New Done Done true
11 Acked 1010 Write 9 New Done Done true
12 Acked 1011 Read 2 New Done Done true
13 Acked 1012 Read 6 New Done Done true
14 Acked 1013 Write 3 New Done Done true
15 Acked 1014 Write 7 New Done Done true
16 Acked 1015 Read 4 New Done Done true
17 Acked 1016 Read 6 New Done Done true
18 Acked 1017 Write 10 New Done Done true
19 Acked 1018 Write 2 New Done Done true
20 Acked 1019 Read 5 New Done Done true
21 Acked 1020 Read 10 New Done Done true
22 Acked 1021 Flush 0 New Done Done true
23 Acked 1022 Read 3 New Done Done true
24 Acked 1023 Read 7 New Done Done true
25 Acked 1024 Read 3 New Done Done true
26 Acked 1025 Write 6 New Done Done true
27 Acked 1026 Read 2 New Done Done true
28 Acked 1027 Write 9 New Done Done true
29 Acked 1028 Read 8 New Done Done true
30 Acked 1029 Write 6 New Done Done true
31 Acked 1030 Read 2 New Done Done true
32 Acked 1031 Write 4 New Done Done true
33 Acked 1032 Read 5 New Done Done true
34 Acked 1033 Flush 0 New Done Done true
35 Acked 1034 Read 2 New Done Done true
36 Acked 1035 Flush 0 New Done Done true
37 Acked 1036 Write 6 New Done Done true
38 Acked 1037 Read 2 New Done Done true
39 Acked 1038 Read 8 New Done Done true
40 Acked 1039 Read 10 New Done Done true
41 Acked 1040 Read 6 New Done Done true
42 Acked 1041 Read 6 New Done Done true
43 Acked 1042 Write 9 New Done Done true
44 Acked 1043 Read 5 New Done Done true
45 Acked 1044 Write 7 New Done Done true
46 Acked 1045 Write 4 New Done Done true
47 Acked 1046 Read 7 New Done Done true
48 Acked 1047 Read 2 New Done Done true
49 Acked 1048 Write 2 New Done Done true
50 Acked 1049 Read 9 New Done Done true
51 Acked 1050 Write 4 New Done Done true
52 Acked 1051 Read 9 New Done Done true
53 Acked 1052 Read 3 New Done Done true
54 Acked 1053 Read 2 New Done Done true
55 Acked 1054 Write 8 New Done Done true
56 Acked 1055 Read 8 New Done Done true
57 Acked 1056 Write 9 New Done Done true
58 Acked 1057 Write 4 New Done Done true
59 Acked 1058 Write 7 New Done Done true
60 Acked 1059 Flush 0 New Done Done true
61 Acked 1060 Write 9 New Done Done true
62 Acked 1061 Read 9 New Done Done true
63 Acked 1062 Read 9 New Done Done true
64 Acked 1063 Flush 0 New Done Done true
65 Acked 1064 Read 6 New Done Done true
66 Acked 1065 Read 9 New Done Done true
67 Acked 1066 Write 9 New Done Done true
68 Acked 1067 Write 2 New Done Done true
69 Acked 1068 Read 4 New Done Done true
70 Acked 1069 Read 9 New Done Done true
71 Acked 1070 Write 3 New Done Done true
72 Acked 1071 Read 4 New Done Done true
73 Acked 1072 Write 7 New Done Done true
74 Acked 1073 Write 4 New Done Done true
75 Acked 1074 Write 7 New Done Done true
76 Acked 1075 Write 6 New Done Done true
77 Acked 1076 Read 5 New Done Done true
78 Acked 1077 Read 7 New Done Done true
79 Acked 1078 Write 5 New Done Done true
80 Acked 1079 Write 6 New Done Done true
81 Acked 1080 Write 9 New Done Done true
82 Acked 1081 Read 3 New Done Done true
83 Acked 1082 Read 4 New Done Done true
84 Acked 1083 Read 3 New Done Done true
85 Acked 1084 Flush 0 New Done Done true
86 Acked 1085 Write 7 New Done Done true
87 Acked 1086 Write 4 New Done Done true
88 Acked 1087 Write 10 New Done Done true
89 Acked 1088 Write 1 New Done Done true
90 Acked 1089 Write 9 New Done Done true
91 Acked 1090 Read 8 New Done Done true
92 Acked 1091 Read 2 New Done Done true
93 Acked 1092 Write 9 New Done Done true
94 Acked 1093 Read 4 New Done Done true
95 Acked 1094 Read 7 New Done Done true
96 Acked 1095 Read 7 New Done Done true
97 Acked 1096 Write 5 New Done Done true
98 Acked 1097 Write 7 New Done Done true
99 Acked 1098 Read 9 New Done Done true
100 Acked 1099 Flush 0 New Done Done true
101 Acked 1100 Write 10 New Done Done true
102 Acked 1101 Flush 0 New Done Done true
103 Acked 1102 Write 4 New Done Done true
104 Acked 1103 Write 10 New Done Done true
105 Acked 1104 Write 1 New Done Done true
106 Acked 1105 Read 6 New Done Done true
107 Acked 1106 Write 1 New Done Done true
108 Acked 1107 Read 2 New Done Done true
109 Acked 1108 Read 7 New Done Done true
110 Acked 1109 Read 2 New Done Done true
111 Acked 1110 Read 4 New Done Done true
112 Acked 1111 Read 8 New Done Done true
113 Acked 1112 Read 8 New Done Done true
114 Acked 1113 Flush 0 New Done Done true
115 Acked 1114 Read 3 New Done Done true
116 Acked 1115 Write 1 New Done Done true
117 Acked 1116 Read 8 New Done Done true
118 Acked 1117 Read 6 New Done Done true
119 Acked 1118 Write 1 New Done Done true
120 Acked 1119 Read 6 New Done Done true
121 Acked 1120 Write 10 New Done Done true
122 Acked 1121 Read 3 New Done Done true
123 Acked 1122 Write 4 New Done Done true
124 Acked 1123 Write 9 New Done Done true
125 Acked 1124 Flush 0 New Done Done true
126 Acked 1125 Write 6 New Done Done true
127 Acked 1126 Write 2 New Done Done true
128 Acked 1127 Write 7 New Done Done true
129 Acked 1128 Write 7 New Done Done true
130 Acked 1129 Read 4 New Done Done true
131 Acked 1130 Write 7 New Done Done true
132 Acked 1131 Read 3 New Done Done true
133 Acked 1132 Read 5 New Done Done true
134 Acked 1133 Write 8 New Done Done true
135 Acked 1134 Write 3 New Done Done true
136 Acked 1135 Write 2 New Done Done true
137 Acked 1136 Write 4 New Done Done true
138 Acked 1137 Read 4 New Done Done true
139 Acked 1138 Read 7 New Done Done true
140 Acked 1139 Read 9 New Done Done true
141 Acked 1140 Read 3 New Done Done true
142 Acked 1141 Read 5 New Done Done true
143 Acked 1142 Write 10 New Done Done true
144 Acked 1143 Flush 0 New Done Done true
145 Acked 1144 Write 9 New Done Done true
146 Acked 1145 Write 3 New Done Done true
147 Acked 1146 Write 1 New Done Done true
148 Acked 1147 Write 7 New Done Done true
149 Acked 1148 Read 4 New Done Done true
150 Acked 1149 Read 9 New Done Done true
151 Acked 1150 Write 9 New Done Done true
152 Acked 1151 Write 6 New Done Done true
153 Acked 1152 Write 2 New Done Done true
154 Acked 1153 Flush 0 New Done Done true
155 Acked 1154 Read 8 New Done Done true
156 Acked 1155 Read 3 New Done Done true
157 Acked 1156 Read 3 New Done Done true
158 Acked 1157 Read 8 New Done Done true
159 Acked 1158 Read 4 New Done Done true
160 Acked 1159 Flush 0 New Done Done true
161 Acked 1160 Write 7 New Done Done true
162 Acked 1161 Write 2 New Done Done true
163 Acked 1162 Write 1 New Done Done true
164 Acked 1163 Write 6 New Done Done true
165 Acked 1164 Read 9 New Done Done true
166 Acked 1165 Write 8 New Done Done true
167 Acked 1166 Read 7 New Done Done true
168 Acked 1167 Write 1 New Done Done true
169 Acked 1168 Read 2 New Done Done true
170 Acked 1169 Read 7 New Done Done true
171 Acked 1170 Read 1 New Done Done true
172 Acked 1171 Read 2 New Done Done true
173 Acked 1172 Read 7 New Done Done true
174 Acked 1173 Write 9 New Done Done true
175 Acked 1174 Flush 0 New Done Done true
176 Acked 1175 Read 7 New Done Done true
177 Acked 1176 Read 3 New Done Done true
178 Acked 1177 Read 10 New Done Done true
179 Acked 1178 Read 10 New Done Done true
180 Acked 1179 Read 9 New Done Done true
181 Acked 1180 Read 3 New Done Done true
182 Acked 1181 Flush 0 New Done Done true
183 Acked 1182 Write 10 New Done Done true
184 Acked 1183 Write 10 New Done Done true
185 Acked 1184 Read 4 New Done Done true
186 Acked 1185 Read 9 New Done Done true
187 Acked 1186 Flush 0 New Done Done true
188 Acked 1187 Write 10 New Done Done true
189 Acked 1188 Write 9 New Done Done true
190 Acked 1189 Read 3 New Done Done true
191 Acked 1190 Write 2 New Done Done true
192 Acked 1191 Read 3 New Done Done true
193 Acked 1192 Read 5 New Done Done true
194 Acked 1193 Write 2 New Done Done true
195 Acked 1194 Write 2 New Done Done true
196 Acked 1195 Write 2 New Done Done true
197 Acked 1196 Write 4 New Done Done true
198 Acked 1197 Read 9 New Done Done true
199 Acked 1198 Read 5 New Done Done true
200 Acked 1199 Write 4 New Done Done true
201 Acked 1200 Write 1 New Done Done true
202 Acked 1201 Write 3 New Done Done true
203 Acked 1202 Read 3 New Done Done true
204 Acked 1203 Write 8 New Done Done true
205 Acked 1204 Write 5 New Done Done true
206 Acked 1205 Read 8 New Done Done true
207 Acked 1206 Read 9 New Done Done true
208 Acked 1207 Flush 0 New Done Done true
209 Acked 1208 Read 7 New Done Done true
210 Acked 1209 Write 5 New Done Done true
211 Acked 1210 Write 3 New Done Done true
212 Acked 1211 Read 6 New Done Done true
213 Acked 1212 Write 10 New Done Done true
214 Acked 1213 Read 1 New Done Done true
215 Acked 1214 Write 4 New Done Done true
216 Acked 1215 Write 1 New Done Done true
217 Acked 1216 Read 7 New Done Done true
218 Acked 1217 Read 8 New Done Done true
219 Acked 1218 Write 9 New Done Done true
220 Acked 1219 Flush 0 New Done Done true
221 Acked 1220 Read 4 New Done Done true
222 Acked 1221 Read 6 New Done Done true
223 Acked 1222 Read 5 New Done Done true
224 Acked 1223 Read 5 New Done Done true
225 Acked 1224 Write 5 New Done Done true
226 Acked 1225 Write 1 New Done Done true
227 Acked 1226 Read 5 New Done Done true
228 Acked 1227 Read 1 New Done Done true
229 Acked 1228 Read 9 New Done Done true
230 Acked 1229 Write 6 New Done Done true
231 Acked 1230 Read 8 New Done Done true
232 Acked 1231 Write 2 New Done Done true
233 Acked 1232 Read 2 New Done Done true
234 Acked 1233 Read 2 New Done Done true
235 Acked 1234 Read 9 New Done Done true
236 Acked 1235 Write 7 New Done Done true
237 Acked 1236 Flush 0 New Done Done true
238 Acked 1237 Flush 0 New Done Done true
239 Acked 1238 Write 6 New Done Done true
240 Acked 1239 Write 7 New Done Done true
241 Acked 1240 Write 9 New Done Done true
242 Acked 1241 Read 5 New Done Done true
243 Acked 1242 Read 8 New Done Done true
244 Acked 1243 Read 1 New Done Done true
245 Acked 1244 Read 1 New Done Done true
246 Acked 1245 Write 4 New Done Done true
247 Acked 1246 Write 7 New Done Done true
248 Acked 1247 Write 7 New Done Done true
249 Acked 1248 Write 10 New Done Done true
250 Acked 1249 Flush 0 New Done Done true
251 Acked 1250 Read 10 New Done Done true
252 Acked 1251 Read 4 New Done Done true
253 Acked 1252 Write 6 New Done Done true
254 Acked 1253 Flush 0 New Done Done true
255 Acked 1254 Write 9 New Done Done true
256 Acked 1255 Flush 0 New Done Done true
257 Acked 1256 Write 1 New Done Done true
258 Acked 1257 Flush 0 New Done Done true
259 Acked 1258 Read 3 New Done Done true
260 Acked 1259 Write 3 New Done Done true
261 Acked 1260 Write 1 New Done Done true
262 Acked 1261 Read 1 New Done Done true
263 Acked 1262 Read 1 New Done Done true
264 Acked 1263 Write 10 New Done Done true
265 Acked 1264 Write 3 New Done Done true
266 Acked 1265 Write 4 New Done Done true
267 Acked 1266 Read 8 New Done Done true
268 Acked 1267 Read 5 New Done Done true
269 Acked 1268 Write 10 New Done Done true
270 Acked 1269 Write 6 New Done Done true
271 Acked 1270 Write 3 New Done Done true
272 Acked 1271 Read 4 New Done Done true
273 Acked 1272 Write 2 New Done Done true
274 Acked 1273 Write 1 New Done Done true
275 Acked 1274 Write 8 New Done Done true
276 Acked 1275 Write 4 New Done Done true
277 Acked 1276 Write 5 New Done Done true
278 Acked 1277 Write 8 New Done Done true
279 Acked 1278 Write 5 New Done Done true
280 Acked 1279 Write 4 New Done Done true
281 Acked 1280 Flush 0 New Done Done true
282 Acked 1281 Write 3 New Done Done true
283 Acked 1282 Write 1 New Done Done true
284 Acked 1283 Read 4 New Done Done true
285 Acked 1284 Read 7 New Done Done true
286 Acked 1285 Read 5 New Done Done true
287 Acked 1286 Read 9 New Done Done true
288 Acked 1287 Read 2 New Done Done true
289 Acked 1288 Write 10 New Done Done true
290 Acked 1289 Read 6 New Done Done true
291 Acked 1290 Write 10 New Done Done true
292 Acked 1291 Read 8 New Done Done true
293 Acked 1292 Flush 0 New Done Done true
294 Acked 1293 Flush 0 New Done Done true
295 Acked 1294 Write 4 New Done Done true
296 Acked 1295 Read 2 New Done Done true
297 Acked 1296 Write 8 New Done Done true
298 Acked 1297 Read 4 New Done Done true
299 Acked 1298 Write 4 New Done Done true
300 Acked 1299 Read 2 New Done Done true
301 Acked 1300 Flush 0 New Done Done true
STATES DS:0 DS:1 DS:2 TOTAL
New 301 0 0 301
Sent 0 0 0 0
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 0 1300 1300
Downstairs last five completed:
Upstairs last five completed: 301 300 299 298 297
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:47.48093248Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:47.481008057Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:47.482353856Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:47.482391184Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 0","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:47.482410233Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 0","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:47.482605086Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:47.482647754Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:47.482671434Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:47.482689463Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:47.482707872Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 1300 1299 1298 1297 1296
Upstairs last five completed: 301 300 299 298 297
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Feb 13 21:39:51.295 INFO request completed, latency_us: 337, response_code: 204, uri: /stop/cid/0, method[8810] Got stop action so:false kr:false
: POST, req_id: bd6b2f75-f0dc-4973-858b-6c3f30e7276c, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:51.296438583Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:39:51.296 INFO request completed, latency_us: 174, response_code: 200, uri: /state/cid/0, method: GET, req_id:{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:51.296493601Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
632e184e-254b-414c-aee3-1b0e8edfe688, remote_addr: 127.0.0.1:52124{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:51.29651713Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
, local_addr: 127.0.0.1:9998
{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:51.296534969Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 1300","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:51.296558848Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:51.296583987Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:52.297453012Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:52.29751422Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:52.297536759Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:52.297557178Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:52.297572957Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 1300","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:52.297590127Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:52.297607116Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:53.299187084Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:53.299250022Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:53.299271281Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:53.29929134Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:53.29930723Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 1300","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:53.299333749Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:53.299351268Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:54.300812121Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:54.300871318Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:54.300892167Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:54.300912817Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:54.300928656Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 1300","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:54.300969844Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:54.301045192Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:39:55.298 INFO request completed, latency_us: 190, response_code: 200, uri: /state/cid/0, method: GET, req_id: f08bfcfa-1147-4dbb-b3dc-98071843358d, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Read block 870 len 4096
002/300 Read block 2434 len 8192
003/300 Flush
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:55.301370348Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:55.301440675Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:55.301475704Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:55.301518102Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:55.301546821Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 3 jobs since flush 1300","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:55.30157513Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:55.301602089Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
004/300 Write block 1978 len 16384 data: 2 2 2 2
005/300 Read block 3527 len 36864
006/300 Write block 3869 len 12288 data: 2 2 2
007/300 Write block 3894 len 12288 data: 2 3 3
008/300 Flush
009/300 Read block 1749 len 20480
010/300 Read block 1051 len 28672
011/300 Read block 1440 len 36864
012/300 Write block 163 len 36864 data: 3 3 2 2 2 2 2 2 2
013/300 Read block 1406 len 16384
014/300 Write block 1209 len 28672 data: 2 2 2 2 2 2 2
015/300 Write block 2832 len 36864 data: 2 2 2 2 2 2 2 2 2
016/300 Read block 4972 len 16384
017/300 Write block 2655 len 8192 data: 2 2
018/300 Read block 2032 len 36864
019/300 Write block 4164 len 28672 data: 2 2 2 2 2 2 2
020/300 Flush
021/300 Write block 4034 len 32768 data: 2 2 2 2 2 2 2 2
022/300 Read block 1719 len 16384
023/300 Read block 1812 len 8192
024/300 Read block 1279 len 32768
025/300 Read block 4841 len 28672
026/300 Flush
027/300 Read block 1037 len 24576
028/300 Read block 313 len 12288
029/300 Read block 4342 len 36864
030/300 Read block 2773 len 4096
031/300 Write block 517 len 20480 data: 2 2 2 2 2
032/300 Read block 2786 len 36864
033/300 Flush
034/300 Read block 3575 len 24576
035/300 Write block 3818 len 32768 data: 2 2 2 2 2 2 2 2
036/300 Write block 1611 len 4096 data: 3
037/300 Write block 3283 len 36864 data: 3 3 3 3 3 2 2 2 2
038/300 Write block 732 len 40960 data: 2 2 2 2 2 2 2 2 2 2
039/300 Read block 3890 len 12288
040/300 Write block 3902 len 12288 data: 3 3 2
041/300 Write block 36 len 24576 data: 2 2 2 2 2 2
042/300 Read block 1725 len 4096
043/300 Flush
044/300 Write block 986 len 8192 data: 2 2
045/300 Write block 2586 len 20480 data: 2 2 2 2 2
046/300 Write block 3128 len 4096 data: 2
047/300 Read block 1290 len 20480
048/300 Read block 626 len 24576
049/300 Write block 730 len 20480 data: 2 2 3 3 3
050/300 Write block 4428 len 28672 data: 2 2 2 2 2 2 2
051/300 Read block 4297 len 12288
052/300 Flush
053/300 Read block 1329 len 16384
054/300 Write block 4987 len 16384 data: 2 2 2 2
055/300 Write block 4308 len 24576 data: 2 2 2 2 2 2
056/300 Read block 2147 len 8192
057/300 Write block 1350 len 32768 data: 2 2 3 2 2 2 2 2
058/300 Read block 1514 len 28672
059/300 Read block 297 len 16384
060/300 Read block 2217 len 16384
061/300 Flush
062/300 Read block 3327 len 36864
063/300 Flush
064/300 Read block 1070 len 28672
065/300 Write block 1719 len 36864 data: 2 2 2 2 2 2 2 2 2
066/300 Read block 4474 len 12288
067/300 Read block 3029 len 4096
068/300 Write block 3037 len 12288 data: 2 2 2
069/300 Read block 4179 len 32768
070/300 Read block 731 len 40960
071/300 Read block 643 len 28672
072/300 Write block 1385 len 28672 data: 2 2 2 2 2 2 2
073/300 Write block 1095 len 24576 data: 2 2 2 2 2 2
074/300 Write block 1656 len 16384 data: 2 2 2 2
075/300 Write block 4948 len 32768 data: 2 2 2 2 2 2 2 2
076/300 Write block 1958 len 4096 data: 2
077/300 Write block 3557 len 32768 data: 3 3 3 3 3 2 2 2
078/300 Flush
079/300 Read block 4754 len 8192
080/300 Read block 319 len 20480
081/300 Write block 1863 len 20480 data: 2 2 2 2 2
082/300 Write block 854 len 12288 data: 2 2 2
083/300 Write block 565 len 24576 data: 3 3 3 2 2 2
084/300 Flush
085/300 Read block 1066 len 20480
086/300 Read block 2162 len 40960
087/300 Write block 4176 len 4096 data: 2
088/300 Write block 1742 len 20480 data: 2 2 3 3 3
089/300 Write block 162 len 16384 data: 3 4 4 3
090/300 Read block 3443 len 4096
091/300 Read block 1445 len 16384
092/300 Flush
093/300 Flush
094/300 Read block 2041 len 36864
095/300 Read block 4355 len 16384
096/300 Flush
097/300 Read block 1969 len 16384
098/300 Read block 4417 len 4096
099/300 Read block 85 len 40960
100/300 Flush
101/300 Write block 1659 len 12288 data: 3 2 2
102/300 Read block 1177 len 4096
103/300 Write block 4066 len 20480 data: 2 2 2 2 2
104/300 Write block 4038 len 40960 data: 3 3 3 3 2 2 2 2 2 2
105/300 Read block 2017 len 8192
106/300 Write block 709 len 24576 data: 2 3 3 3 3 3
107/300 Write block 2284 len 24576 data: 2 2 2 2 2 2
108/300 Read block 67 len 24576
109/300 Read block 4293 len 12288
110/300 Write block 3221 len 24576 data: 3 3 3 3 3 3
111/300 Write block 3171 len 32768 data: 2 2 2 2 2 2 2 2
112/300 Flush
113/300 Flush
114/300 Read block 1690 len 28672
115/300 Read block 2388 len 4096
116/300 Read block 4554 len 20480
117/300 Flush
118/300 Write block 3851 len 40960 data: 2 2 2 2 2 2 2 2 2 2
119/300 Read block 3441 len 28672
120/300 Read block 3403 len 24576
121/300 Read block 238 len 36864
122/300 Read block 4717 len 40960
123/300 Flush
124/300 Write block 2713 len 12288 data: 2 2 2
125/300 Read block 4080 len 4096
126/300 Read block 3034 len 24576
127/300 Flush
128/300 Write block 271 len 8192 data: 2 2
129/300 Write block 3721 len 4096 data: 2
130/300 Write block 3246 len 32768 data: 2 2 2 2 2 2 2 2
131/300 Flush
132/300 Write block 1053 len 4096 data: 2
133/300 Read block 152 len 24576
134/300 Flush
135/300 Flush
136/300 Read block 644 len 32768
137/300 Read block 4022 len 16384
138/300 Read block 43 len 20480
139/300 Write block 1203 len 36864 data: 2 2 2 2 2 2 3 3 3
140/300 Write block 4215 len 40960 data: 2 2 2 3 3 3 3 2 2 2
141/300 Read block 2647 len 28672
142/300 Read block 4213 len 28672
143/300 Read block 3790 len 8192
144/300 Write block 1072 len 4096 data: 2
145/300 Read block 2030 len 40960
146/300 Read block 2177 len 36864
147/300 Read block 1278 len 20480
148/300 Read block 953 len 12288
149/300 Read block 1037 len 20480
150/300 Write block 842 len 28672 data: 2 2 2 2 2 2 2
151/300 Write block 2101 len 36864 data: 2 2 2 2 2 2 2 2 2
152/300 Read block 3827 len 40960
153/300 Write block 1206 len 16384 data: 3 3 3 4
154/300 Read block 2194 len 20480
155/300 Read block 3477 len 40960
156/300 Write block 1158 len 12288 data: 2 2 2
157/300 Read block 1479 len 12288
158/300 Read block 4816 len 40960
159/300 Read block 1021 len 36864
160/300 Read block 302 len 40960
161/300 Read block 3507 len 28672
162/300 Read block 2444 len 24576
163/300 Read block 2651 len 24576
164/300 Write block 61 len 20480 data: 4 4 3 3 3
165/300 Write block 3336 len 36864 data: 2 2 2 2 2 2 3 4 4
166/300 Read block 2713 len 16384
167/300 Read block 307 len 4096
168/300 Read block 2647 len 20480
169/300 Read block 3855 len 8192
170/300 Write block 411 len 36864 data: 2 2 2 2 2 2 2 2 2
171/300 Read block 4806 len 40960
172/300 Read block 4643 len 32768
173/300 Write block 665 len 32768 data: 2 2 2 2 2 2 2 2
174/300 Write block 4722 len 20480 data: 3 2 2 2 2
175/300 Read block 3293 len 32768
176/300 Read block 1641 len 40960
177/300 Read block 1532 len 28672
178/300 Write block 2501 len 16384 data: 2 2 2 2
179/300 Write block 2887 len 20480 data: 2 2 2 2 2
180/300 Read block 2116 len 40960
181/300 Write block 3075 len 16384 data: 2 2 2 2
182/300 Flush
183/300 Flush
184/300 Write block 3304 len 32768 data: 2 2 3 2 2 2 2 2
185/300 Read block 2090 len 20480
186/300 Read block 1119 len 40960
187/300 Read block 4772 len 4096
188/300 Write block 164 len 24576 data: 5 4 3 3 3 3
189/300 Write block 141 len 4096 data: 2
190/300 Read block 4216 len 40960
191/300 Read block 1775 len 16384
192/300 Read block 1253 len 8192
193/300 Write block 3042 len 12288 data: 2 2 2
194/300 Read block 1364 len 40960
195/300 Read block 2481 len 28672
196/300 Write block 555 len 8192 data: 2 2
197/300 Read block 882 len 24576
198/300 Write block 1587 len 12288 data: 2 2 2
199/300 Flush
200/300 Flush
201/300 Write block 1722 len 8192 data: 3 3
202/300 Read block 1344 len 40960
203/300 Write block 2937 len 4096 data: 2
204/300 Write block 1994 len 32768 data: 2 2 2 2 2 2 2 2
205/300 Write block 3238 len 24576 data: 2 2 2 2 2 2
206/300 Read block 1669 len 4096
207/300 Read block 4034 len 32768
208/300 Read block 1402 len 24576
209/300 Write block 411 len 28672 data: 3 3 3 3 3 3 3
210/300 Read block 371 len 16384
211/300 Write block 992 len 36864 data: 3 2 2 2 2 2 2 2 2
212/300 Flush
213/300 Read block 917 len 8192
214/300 Read block 1341 len 24576
215/300 Write block 3607 len 36864 data: 2 2 2 2 2 2 2 2 2
216/300 Write block 4942 len 4096 data: 3
217/300 Read block 3270 len 12288
218/300 Read block 2872 len 28672
219/300 Read block 1490 len 28672
220/300 Read block 222 len 36864
221/300 Read block 4317 len 40960
222/300 Read block 4583 len 8192
223/300 Read block 4882 len 32768
224/300 Write block 3485 len 28672 data: 2 2 2 2 2 2 2
225/300 Write block 1281 len 32768 data: 2 2 2 2 2 2 2 2
226/300 Write block 758 len 16384 data: 2 2 2 2
227/300 Write block 1555 len 4096 data: 2
228/300 Read block 2308 len 32768
229/300 Read block 3435 len 28672
230/300 Read block 2250 len 20480
231/300 Read block 2127 len 20480
232/300 Write block 3117 len 32768 data: 2 2 2 2 2 2 2 2
233/300 Read block 3743 len 36864
234/300 Read block 4133 len 12288
235/300 Read block 3111 len 4096
236/300 Write block 3775 len 32768 data: 3 2 2 2 2 2 2 2
237/300 Write block 3219 len 32768 data: 3 3 4 4 4 4 4 4
238/300 Read block 4392 len 12288
239/300 Read block 1067 len 4096
240/300 Read block 324 len 24576
241/300 Write block 2170 len 4096 data: 3
242/300 Write block 4114 len 12288 data: 2 2 2
243/300 Write block 4508 len 8192 data: 2 2
244/300 Read block 3724 len 4096
245/300 Flush
246/300 Read block 4841 len 12288
247/300 Read block 1345 len 32768
248/300 Flush
249/300 Read block 2490 len 12288
250/300 Read block 1171 len 40960
251/300 Read block 202 len 4096
252/300 Read block 4092 len 36864
253/300 Write block 178 len 40960 data: 2 2 2 2 2 2 2 2 2 2
254/300 Write block 4700 len 12288 data: 2 2 2
255/300 Read block 2940 len 40960
256/300 Flush
257/300 Read block 4263 len 28672
258/300 Read block 3203 len 24576
259/300 Write block 1061 len 20480 data: 2 2 2 2 2
260/300 Read block 200 len 32768
261/300 Write block 4794 len 32768 data: 2 2 2 2 2 2 2 2
262/300 Read block 2849 len 24576
263/300 Read block 1190 len 40960
264/300 Write block 2365 len 28672 data: 2 2 2 2 2 2 2
265/300 Read block 451 len 24576
266/300 Flush
267/300 Read block 4271 len 28672
268/300 Write block 83 len 20480 data: 2 2 2 2 2
269/300 Read block 1303 len 12288
270/300 Read block 2687 len 8192
271/300 Read block 3 len 36864
272/300 Write block 4379 len 36864 data: 3 3 3 3 4 4 3 3 3
273/300 Read block 570 len 20480
274/300 Write block 4484 len 28672 data: 2 2 2 2 2 2 2
275/300 Read block 2328 len 8192
276/300 Write block 1771 len 32768 data: 2 2 2 2 2 2 2 2
277/300 Read block 2487 len 8192
278/300 Read block 3595 len 40960
279/300 Read block 1946 len 4096
280/300 Read block 2912 len 24576
281/300 Write block 2587 len 40960 data: 3 3 3 3 2 2 2 2 2 2
282/300 Read block 3594 len 16384
283/300 Write block 4946 len 16384 data: 3 2 3 3
284/300 Read block 3899 len 20480
285/300 Read block 1085 len 16384
286/300 Write block 1227 len 12288 data: 2 2 2
287/300 Read block 2650 len 28672
288/300 Write block 4088 len 20480 data: 2 2 2 2 2
289/300 Write block 2020 len 16384 data: 2 2 2 2
290/300 Read block 273 len 28672
291/300 Read block 4059 len 16384
292/300 Write block 121 len 28672 data: 2 2 4 4 4 4 4
293/300 Read block 403 len 36864
294/300 Flush
295/300 Read block 2172 len 36864
296/300 Flush
297/300 Read block 1828 len 8192
298/300 Read block 3575 len 24576
299/300 Write block 2327 len 8192 data: 2 2
300/300 Read block 1066 len 36864
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 13 21:39:56.166 INFO request completed, latency_us: 204, response_code: 204, uri: /start/cid/0, method: POST, req_id: 265d48a9-49a3-4568-9f75-6e3a5f099665, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
302 Acked 1301 Read 1 New Done Done true
303 Acked 1302 Read 2 New Done Done true
304 Acked 1303 Flush 0 New Done Done true
305 Acked 1304 Write 4 New Done Done false
306 Acked 1305 Read 9 New Done Done false
307 Acked 1306 Write 3 New Done Done false
308 Acked 1307 Write 3 New Done Done false
309 Acked 1308 Flush 0 New Done Done false
310 Acked 1309 Read 5 New Done Done false
311 Acked 1310 Read 7 New Done Done false
312 Acked 1311 Read 9 New Done Done false
313 Acked 1312 Write 9 New Done Done false
314 Acked 1313 Read 4 New Done Done false
315 Acked 1314 Write 7 New Done Done false
316 Acked 1315 Write 9 New Done Done false
317 Acked 1316 Read 4 New Done Done false
318 Acked 1317 Write 2 New Done Done false
319 Acked 1318 Read 9 New Done Done false
320 Acked 1319 Write 7 New Done Done false
321 Acked 1320 Flush 0 New Done Done false
322 Acked 1321 Write 8 New Done Done false
323 Acked 1322 Read 4 New Done Done false
324 Acked 1323 Read 2 New Done Done false
325 Acked 1324 Read 8 New Done Done false
326 Acked 1325 Read 7 New Done Done false
327 Acked 1326 Flush 0 New Done Done false
328 Acked 1327 Read 6 New Done Done false
329 Acked 1328 Read 3 New Done Done false
330 Acked 1329 Read 9 New Done Done false
331 Acked 1330 Read 1 New Done Done false
332 Acked 1331 Write 5 New Done Done false
333 Acked 1332 Read 9 New Done Done false
334 Acked 1333 Flush 0 New Done Done false
335 Acked 1334 Read 6 New Done Done false
336 Acked 1335 Write 8 New Done Done false
337 Acked 1336 Write 1 New Done Done false
338 Acked 1337 Write 9 New Done Done false
339 Acked 1338 Write 10 New Done Done false
340 Acked 1339 Read 3 New Done Done false
341 Acked 1340 Write 3 New Done Done false
342 Acked 1341 Write 6 New Done Done false
343 Acked 1342 Read 1 New Done Done false
344 Acked 1343 Flush 0 New Done Done false
345 Acked 1344 Write 2 New Done Done false
346 Acked 1345 Write 5 New Done Done false
347 Acked 1346 Write 1 New Done Done false
348 Acked 1347 Read 5 New Done Done false
349 Acked 1348 Read 6 New Done Done false
350 Acked 1349 Write 5 New Done Done false
351 Acked 1350 Write 7 New Done Done false
352 Acked 1351 Read 3 New Done Done false
353 Acked 1352 Flush 0 New Done Done false
354 Acked 1353 Read 4 New Done Done false
355 Acked 1354 Write 4 New Done Done false
356 Acked 1355 Write 6 New Done Done false
357 Acked 1356 Read 2 New Done Done false
358 Acked 1357 Write 8 New Done Done false
359 Acked 1358 Read 7 New Done Done false
360 Acked 1359 Read 4 New Done Done false
361 Acked 1360 Read 4 New Done Done false
362 Acked 1361 Flush 0 New Done Done false
363 Acked 1362 Read 9 New Done Done false
364 Acked 1363 Flush 0 New Done Done false
365 Acked 1364 Read 7 New Done Done false
366 Acked 1365 Write 9 New Done Done false
367 Acked 1366 Read 3 New Done Done false
368 Acked 1367 Read 1 New Done Done false
369 Acked 1368 Write 3 New Done Done false
370 Acked 1369 Read 8 New Done Done false
371 Acked 1370 Read 10 New Done Done false
372 Acked 1371 Read 7 New Done Done false
373 Acked 1372 Write 7 New Done Done false
374 Acked 1373 Write 6 New Done Done false
375 Acked 1374 Write 4 New Done Done false
376 Acked 1375 Write 8 New Done Done false
377 Acked 1376 Write 1 New Done Done false
378 Acked 1377 Write 8 New Done Done false
379 Acked 1378 Flush 0 New Done Done false
380 Acked 1379 Read 2 New Done Done false
381 Acked 1380 Read 5 New Done Done false
382 Acked 1381 Write 5 New Done Done false
383 Acked 1382 Write 3 New Done Done false
384 Acked 1383 Write 6 New Done Done false
385 Acked 1384 Flush 0 New Done Done false
386 Acked 1385 Read 5 New Done Done false
387 Acked 1386 Read 10 New Done Done false
388 Acked 1387 Write 1 New Done Done false
389 Acked 1388 Write 5 New Done Done false
390 Acked 1389 Write 4 New Done Done false
391 Acked 1390 Read 1 New Done Done false
392 Acked 1391 Read 4 New Done Done false
393 Acked 1392 Flush 0 New Done Done false
394 Acked 1393 Flush 0 New Done Done false
395 Acked 1394 Read 9 New Done Done false
396 Acked 1395 Read 4 New Done Done false
397 Acked 1396 Flush 0 New Done Done false
398 Acked 1397 Read 4 New Done Done false
399 Acked 1398 Read 1 New Done Done false
400 Acked 1399 Read 10 New Done Done false
401 Acked 1400 Flush 0 New Done Done false
402 Acked 1401 Write 3 New Done Done false
403 Acked 1402 Read 1 New Done Done false
404 Acked 1403 Write 5 New Done Done false
405 Acked 1404 Write 10 New Done Done false
406 Acked 1405 Read 2 New Done Done false
407 Acked 1406 Write 6 New Done Done false
408 Acked 1407 Write 6 New Done Done false
409 Acked 1408 Read 6 New Done Done false
410 Acked 1409 Read 3 New Done Done false
411 Acked 1410 Write 6 New Done Done false
412 Acked 1411 Write 8 New Done Done false
413 Acked 1412 Flush 0 New Done Done false
414 Acked 1413 Flush 0 New Done Done false
415 Acked 1414 Read 7 New Done Done false
416 Acked 1415 Read 1 New Done Done false
417 Acked 1416 Read 5 New Done Done false
418 Acked 1417 Flush 0 New Done Done false
419 Acked 1418 Write 10 New Done Done false
420 Acked 1419 Read 7 New Done Done false
421 Acked 1420 Read 6 New Done Done false
422 Acked 1421 Read 9 New Done Done false
423 Acked 1422 Read 10 New Done Done false
424 Acked 1423 Flush 0 New Done Done false
425 Acked 1424 Write 3 New Done Done false
426 Acked 1425 Read 1 New Done Done false
427 Acked 1426 Read 6 New Done Done false
428 Acked 1427 Flush 0 New Done Done false
429 Acked 1428 Write 2 New Done Done false
430 Acked 1429 Write 1 New Done Done false
431 Acked 1430 Write 8 New Done Done false
432 Acked 1431 Flush 0 New Done Done false
433 Acked 1432 Write 1 New Done Done false
434 Acked 1433 Read 6 New Done Done false
435 Acked 1434 Flush 0 New Done Done false
436 Acked 1435 Flush 0 New Done Done false
437 Acked 1436 Read 8 New Done Done false
438 Acked 1437 Read 4 New Done Done false
439 Acked 1438 Read 5 New Done Done false
440 Acked 1439 Write 9 New Done Done false
441 Acked 1440 Write 10 New Done Done false
442 Acked 1441 Read 7 New Done Done false
443 Acked 1442 Read 7 New Done Done false
444 Acked 1443 Read 2 New Done Done false
445 Acked 1444 Write 1 New Done Done false
446 Acked 1445 Read 10 New Done Done false
447 Acked 1446 Read 9 New Done Done false
448 Acked 1447 Read 5 New Done Done false
449 Acked 1448 Read 3 New Done Done false
450 Acked 1449 Read 5 New Done Done false
451 Acked 1450 Write 7 New Done Done false
452 Acked 1451 Write 9 New Done Done false
453 Acked 1452 Read 10 New Done Done false
454 Acked 1453 Write 4 New Done Done false
455 Acked 1454 Read 5 New Done Done false
456 Acked 1455 Read 10 New Done Done false
457 Acked 1456 Write 3 New Done Done false
458 Acked 1457 Read 3 New Done Done false
459 Acked 1458 Read 10 New Done Done false
460 Acked 1459 Read 9 New Done Done false
461 Acked 1460 Read 10 New Done Done false
462 Acked 1461 Read 7 New Done Done false
463 Acked 1462 Read 6 New Done Done false
464 Acked 1463 Read 6 New Done Done false
465 Acked 1464 Write 5 New Done Done false
466 Acked 1465 Write 9 New Done Done false
467 Acked 1466 Read 4 New Done Done false
468 Acked 1467 Read 1 New Done Done false
469 Acked 1468 Read 5 New Done Done false
470 Acked 1469 Read 2 New Done Done false
471 Acked 1470 Write 9 New Done Done false
472 Acked 1471 Read 10 New Done Done false
473 Acked 1472 Read 8 New Done Done false
474 Acked 1473 Write 8 New Done Done false
475 Acked 1474 Write 5 New Done Done false
476 Acked 1475 Read 8 New Done Done false
477 Acked 1476 Read 10 New Done Done false
478 Acked 1477 Read 7 New Done Done false
479 Acked 1478 Write 4 New Done Done false
480 Acked 1479 Write 5 New Done Done false
481 Acked 1480 Read 10 New Done Done false
482 Acked 1481 Write 4 New Done Done false
483 Acked 1482 Flush 0 New Done Done false
484 Acked 1483 Flush 0 New Done Done false
485 Acked 1484 Write 8 New Done Done false
486 Acked 1485 Read 5 New Done Done false
487 Acked 1486 Read 10 New Done Done false
488 Acked 1487 Read 1 New Done Done false
489 Acked 1488 Write 6 New Done Done false
490 Acked 1489 Write 1 New Done Done false
491 Acked 1490 Read 10 New Done Done false
492 Acked 1491 Read 4 New Done Done false
493 Acked 1492 Read 2 New Done Done false
494 Acked 1493 Write 3 New Done Done false
495 Acked 1494 Read 10 New Done Done false
496 Acked 1495 Read 7 New Done Done false
497 Acked 1496 Write 2 New Done Done false
498 Acked 1497 Read 6 New Done Done false
499 Acked 1498 Write 3 New Done Done false
500 Acked 1499 Flush 0 New Done Done false
501 Acked 1500 Flush 0 New Done Done false
502 Acked 1501 Write 2 New Done Done false
503 Acked 1502 Read 10 New Done Done false
504 Acked 1503 Write 1 New Done Done false
505 Acked 1504 Write 8 New Done Done false
506 Acked 1505 Write 6 New Done Done false
507 Acked 1506 Read 1 New Done Done false
508 Acked 1507 Read 8 New Done Done false
509 Acked 1508 Read 6 New Done Done false
510 Acked 1509 Write 7 New Done Done false
511 Acked 1510 Read 4 New Done Done false
512 Acked 1511 Write 9 New Done Done false
513 Acked 1512 Flush 0 New Done Done false
514 Acked 1513 Read 2 New Done Done false
515 Acked 1514 Read 6 New Done Done false
516 Acked 1515 Write 9 New Done Done false
517 Acked 1516 Write 1 New Done Done false
518 Acked 1517 Read 3 New Done Done false
519 Acked 1518 Read 7 New Done Done false
520 Acked 1519 Read 7 New Done Done false
521 Acked 1520 Read 9 New Done Done false
522 Acked 1521 Read 10 New Done Done false
523 Acked 1522 Read 2 New Done Done false
524 Acked 1523 Read 8 New Done Done false
525 Acked 1524 Write 7 New Done Done false
526 Acked 1525 Flush 0 New Done Done false
527 Acked 1526 Write 8 New Done Done false
528 Acked 1527 Write 4 New Done Done false
529 Acked 1528 Write 1 New Done Done false
530 Acked 1529 Read 8 New Done Done false
531 Acked 1530 Read 7 New Done Done false
532 Acked 1531 Read 5 New Done Done false
533 Acked 1532 Read 5 New Done Done false
534 Acked 1533 Write 8 New Done Done false
535 Acked 1534 Read 9 New Done Done false
536 Acked 1535 Read 3 New Done Done false
537 Acked 1536 Read 1 New Done Done false
538 Acked 1537 Write 8 New Done Done false
539 Acked 1538 Write 8 New Done Done false
540 Acked 1539 Read 3 New Done Done false
541 Acked 1540 Read 1 New Done Done false
542 Acked 1541 Read 6 New Done Done false
543 Acked 1542 Write 1 New Done Done false
544 Acked 1543 Write 3 New Done Done false
545 Acked 1544 Write 2 New Done Done false
546 Acked 1545 Read 1 New Done Done false
547 Acked 1546 Flush 0 New Done Done false
548 Acked 1547 Read 3 New Done Done false
549 Acked 1548 Read 8 New Done Done false
550 Acked 1549 Flush 0 New Done Done false
551 Acked 1550 Read 3 New Done Done false
552 Acked 1551 Read 10 New Done Done false
553 Acked 1552 Read 1 New Done Done false
554 Acked 1553 Read 9 New Done Done false
555 Acked 1554 Write 10 New Done Done false
556 Acked 1555 Write 3 New Done Done false
557 Acked 1556 Read 10 New Done Done false
558 Acked 1557 Flush 0 New Done Done false
559 Acked 1558 Read 7 New Done Done false
560 Acked 1559 Read 6 New Done Done false
561 Acked 1560 Write 5 New Done Done false
562 Acked 1561 Read 8 New Done Done false
563 Acked 1562 Write 8 New Done Done false
564 Acked 1563 Read 6 New Done Done false
565 Acked 1564 Read 10 New Done Done false
566 Acked 1565 Write 7 New Done Done false
567 Acked 1566 Read 6 New Done Done false
568 Acked 1567 Flush 0 New Done Done false
569 Acked 1568 Read 7 New Done Done false
570 Acked 1569 Write 5 New Done Done false
571 Acked 1570 Read 3 New Done Done false
572 Acked 1571 Read 2 New Done Done false
573 Acked 1572 Read 9 New Done Done false
574 Acked 1573 Write 9 New Done Done false
575 Acked 1574 Read 5 New Done Done false
576 Acked 1575 Write 7 New Done Done false
577 Acked 1576 Read 2 New Done Done false
578 Acked 1577 Write 8 New Done Done false
579 Acked 1578 Read 2 New Done Done false
580 Acked 1579 Read 10 New Done Done false
581 Acked 1580 Read 1 New Done Done false
582 Acked 1581 Read 6 New Done Done false
583 Acked 1582 Write 10 New Done Done false
584 Acked 1583 Read 4 New Done Done false
585 Acked 1584 Write 4 New Done Done false
586 Acked 1585 Read 5 New Done Done false
587 Acked 1586 Read 4 New Done Done false
588 Acked 1587 Write 3 New Done Done false
589 Acked 1588 Read 7 New Done Done false
590 Acked 1589 Write 5 New Done Done false
591 Acked 1590 Write 4 New Done Done false
592 Acked 1591 Read 7 New Done Done false
593 Acked 1592 Read 4 New Done Done false
594 Acked 1593 Write 7 New Done Done false
595 Acked 1594 Read 9 New Done Done false
596 Acked 1595 Flush 0 New Done Done false
597 Acked 1596 Read 9 New Done Done false
598 Acked 1597 Flush 0 New Done Done false
599 Acked 1598 Read 2 New Done Done false
600 Acked 1599 Read 6 New Done Done false
601 Acked 1600 Write 2 New Done Done false
602 Acked 1601 Read 9 New Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
New 301 0 0 301
Sent 0 0 0 0
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1300 1597 1597
Downstairs last five completed: 1300 1299 1298 1297 1296
Upstairs last five completed: 602 601 600 599 598
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:56.30311551Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:56.303173577Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:56.303197996Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:56.303219046Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:56.303234855Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 1300","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:56.303252045Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:56.30337816Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:57.30387027Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:57.303937487Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:57.303961886Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:39:57.303990675Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:57.304007154Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 1300","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:57.304024364Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:57.304194987Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1166)
[8810][0] reports Running
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:58.304670257Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:58.304797932Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:58.306110512Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:58.306137791Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 1300","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:58.30615645Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 1300","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:58.306350463Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:58.306391521Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:58.30641782Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:58.30643569Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:39:58.306452859Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 1602 1601 1600 1599 1598
Upstairs last five completed: 603 602 601 600 599
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(2)
stop 2
Feb 13 21:40:00.169 INFO request completed, latency_us: [8830] Got stop action so:false kr:false
321, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 2532a81f-a155-4c82-a33e-242f367a95c2, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:00.170630916Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:00.170688074Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:00.170709463Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:40:00.170 INFO request completed, latency_us: 104, response_code: 200, uri:{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:00.170730222Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
/state/cid/2, method: GET, req_id: 20655a11-58e0-477d-a319-92b15d05d0f6, remote_addr: 127.0.0.1:52124, local_addr: 127{"msg":"[2] client re-new 0 jobs since flush 1602","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:00.170763161Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
.0.0.1:9998
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:00.1707985Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:01.171410694Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:01.171465362Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:01.171484601Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:01.171501531Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:01.17151476Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 1602","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:01.17153576Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:01.171550449Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:02.173113868Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:02.173160026Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:02.173178325Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:02.173194995Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:02.173207764Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 1602","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:02.173228633Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:02.173257132Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:03.175121229Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:03.175178857Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:03.175201596Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:03.175222945Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:03.175247004Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 1602","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:03.175266414Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:03.175283653Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:40:04.172 INFO request completed, latency_us: 224, response_code: 200, uri: /state/cid/2, method: GET, req_id: 209fc10a-38db-417b-90cd-77589d189d85, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Flush
002/300 Write block 4475 len 12288 data: 2 3 3
003/300 Write block 1997 len 16384 data: 3 3 3 3
004/300 Write block 2471 len 28672 data: 2 2 2 2 2 2 2
005/300 Read block 280 len 28672
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:04.176650819Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:04.176697897Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:04.176716306Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:04.176733146Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:04.176745815Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 5 jobs since flush 1602","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:04.176760055Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:04.176774344Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
006/300 Write block 1895 len 16384 data: 2 2 2 2
007/300 Read block 1418 len 8192
008/300 Read block 1270 len 32768
009/300 Read block 3389 len 28672
010/300 Read block 3615 len 8192
011/300 Read block 4658 len 8192
012/300 Write block 4158 len 24576 data: 2 2 2 2 2 2
013/300 Read block 4518 len 8192
014/300 Read block 824 len 20480
015/300 Read block 1576 len 32768
016/300 Read block 2688 len 32768
017/300 Write block 1644 len 4096 data: 2
018/300 Read block 462 len 32768
019/300 Write block 3403 len 40960 data: 2 2 2 2 2 3 3 3 3 3
020/300 Write block 3969 len 24576 data: 2 2 2 2 2 2
021/300 Write block 238 len 36864 data: 3 3 3 3 2 2 2 2 2
022/300 Write block 2830 len 4096 data: 3
023/300 Read block 2253 len 20480
024/300 Read block 254 len 28672
025/300 Write block 4500 len 32768 data: 2 2 2 2 2 2 2 2
026/300 Write block 1864 len 20480 data: 3 3 3 3 2
027/300 Write block 3434 len 12288 data: 2 2 2
028/300 Read block 69 len 40960
029/300 Read block 4649 len 24576
030/300 Write block 4806 len 20480 data: 2 2 2 2 2
031/300 Write block 4803 len 4096 data: 2
032/300 Read block 3460 len 36864
033/300 Flush
034/300 Write block 2320 len 16384 data: 2 2 2 2
035/300 Write block 351 len 32768 data: 2 2 2 2 2 2 2 2
036/300 Write block 3647 len 40960 data: 2 2 2 2 2 2 2 2 3 3
037/300 Flush
038/300 Write block 1306 len 36864 data: 2 2 2 2 2 2 2 2 2
039/300 Write block 4416 len 28672 data: 2 2 2 2 2 2 2
040/300 Write block 3883 len 16384 data: 2 2 2 2
041/300 Write block 3770 len 12288 data: 3 3 3
042/300 Read block 998 len 40960
043/300 Write block 3641 len 40960 data: 2 2 3 2 2 2 3 3 3 3
044/300 Write block 17 len 28672 data: 2 2 3 3 2 2 2
045/300 Write block 835 len 8192 data: 2 2
046/300 Flush
047/300 Read block 1031 len 28672
048/300 Write block 2406 len 24576 data: 2 2 2 2 2 2
049/300 Read block 3945 len 4096
050/300 Write block 1556 len 20480 data: 2 2 2 2 2
051/300 Read block 1706 len 40960
052/300 Write block 2032 len 8192 data: 2 2
053/300 Flush
054/300 Read block 4180 len 4096
055/300 Write block 4121 len 28672 data: 2 2 2 2 2 2 2
056/300 Write block 1565 len 8192 data: 2 2
057/300 Write block 4325 len 24576 data: 3 3 3 2 2 2
058/300 Write block 797 len 20480 data: 2 2 2 2 2
059/300 Write block 4209 len 4096 data: 2
060/300 Flush
061/300 Write block 1405 len 12288 data: 2 2 2
062/300 Write block 2009 len 4096 data: 2
063/300 Read block 1898 len 28672
064/300 Read block 415 len 40960
065/300 Read block 4351 len 4096
066/300 Flush
067/300 Write block 4155 len 28672 data: 2 2 2 3 3 3 3
068/300 Read block 483 len 40960
069/300 Write block 1594 len 40960 data: 2 2 2 2 2 2 2 2 2 2
070/300 Read block 2660 len 28672
071/300 Write block 3908 len 32768 data: 2 2 2 2 2 2 2 3
072/300 Read block 977 len 8192
073/300 Write block 4636 len 28672 data: 2 2 2 2 2 2 2
074/300 Flush
075/300 Write block 3258 len 4096 data: 2
076/300 Read block 4499 len 4096
077/300 Read block 2694 len 8192
078/300 Flush
079/300 Write block 2157 len 40960 data: 2 3 3 3 3 3 3 4 4 3
080/300 Read block 1694 len 8192
081/300 Write block 4576 len 8192 data: 2 2
082/300 Read block 3426 len 32768
083/300 Read block 3447 len 32768
084/300 Read block 1976 len 40960
085/300 Read block 4073 len 40960
086/300 Write block 1724 len 16384 data: 3 3 3 3
087/300 Read block 2196 len 4096
088/300 Write block 3197 len 12288 data: 2 2 2
089/300 Flush
090/300 Flush
091/300 Write block 3586 len 36864 data: 2 2 2 2 2 2 2 2 2
092/300 Read block 2213 len 8192
093/300 Write block 900 len 24576 data: 2 2 2 2 2 2
094/300 Write block 2075 len 8192 data: 2 2
095/300 Read block 4623 len 8192
096/300 Write block 4770 len 12288 data: 2 2 2
097/300 Read block 1417 len 24576
098/300 Write block 105 len 12288 data: 2 2 2
099/300 Write block 1359 len 20480 data: 2 2 2 2 2
100/300 Write block 3147 len 32768 data: 2 3 3 3 3 2 2 2
101/300 Read block 4980 len 28672
102/300 Flush
103/300 Flush
104/300 Read block 264 len 40960
105/300 Read block 2252 len 28672
106/300 Flush
107/300 Read block 4257 len 20480
108/300 Flush
109/300 Write block 856 len 8192 data: 3 2
110/300 Read block 150 len 36864
111/300 Flush
112/300 Write block 2232 len 4096 data: 2
113/300 Read block 1886 len 8192
114/300 Read block 3578 len 40960
115/300 Read block 4942 len 36864
116/300 Write block 443 len 24576 data: 2 2 2 2 3 3
117/300 Read block 982 len 28672
118/300 Write block 4383 len 36864 data: 5 5 4 4 4 3 2 2 2
119/300 Read block 2655 len 32768
120/300 Write block 858 len 4096 data: 2
121/300 Read block 438 len 16384
122/300 Write block 1854 len 12288 data: 2 2 2
123/300 Write block 1245 len 24576 data: 2 2 2 2 2 2
124/300 Write block 694 len 32768 data: 2 2 2 2 2 2 2 2
125/300 Read block 2802 len 4096
126/300 Flush
127/300 Read block 4969 len 32768
128/300 Write block 2627 len 4096 data: 2
129/300 Write block 2350 len 28672 data: 2 2 2 2 2 2 2
130/300 Read block 1401 len 4096
131/300 Write block 372 len 16384 data: 2 3 2 2
132/300 Write block 1353 len 36864 data: 3 3 3 3 3 2 3 3 3
133/300 Read block 2187 len 40960
134/300 Read block 3048 len 16384
135/300 Read block 2380 len 40960
136/300 Read block 4418 len 24576
137/300 Write block 1757 len 8192 data: 2 2
138/300 Write block 388 len 36864 data: 2 2 2 2 2 2 2 2 2
139/300 Write block 3046 len 4096 data: 2
140/300 Write block 3427 len 16384 data: 2 2 2 2
141/300 Write block 395 len 36864 data: 3 3 2 2 2 2 2 2 2
142/300 Write block 1402 len 4096 data: 2
143/300 Read block 1610 len 36864
144/300 Write block 3347 len 32768 data: 2 2 2 2 2 2 2 2
145/300 Read block 3157 len 12288
146/300 Write block 674 len 16384 data: 2 2 2 2
147/300 Read block 2770 len 28672
148/300 Write block 966 len 24576 data: 2 2 2 2 2 2
149/300 Write block 2978 len 20480 data: 2 2 2 2 2
150/300 Read block 3929 len 32768
151/300 Read block 24 len 40960
152/300 Write block 4405 len 12288 data: 2 3 3
153/300 Write block 3883 len 8192 data: 3 3
154/300 Read block 3975 len 40960
155/300 Write block 2310 len 40960 data: 2 2 2 2 2 2 2 2 2 2
156/300 Write block 73 len 16384 data: 2 2 2 2
157/300 Read block 2765 len 20480
158/300 Write block 4197 len 28672 data: 3 3 3 3 3 3 2
159/300 Read block 310 len 36864
160/300 Read block 1136 len 32768
161/300 Read block 2235 len 36864
162/300 Read block 3420 len 40960
163/300 Write block 2712 len 8192 data: 2 3
164/300 Read block 93 len 32768
165/300 Read block 3243 len 12288
166/300 Read block 3935 len 24576
167/300 Read block 2413 len 28672
168/300 Write block 683 len 28672 data: 2 2 2 2 2 2 2
169/300 Flush
170/300 Read block 3233 len 28672
171/300 Read block 1275 len 16384
172/300 Read block 1663 len 4096
173/300 Read block 644 len 8192
174/300 Flush
175/300 Write block 3313 len 24576 data: 2 2 2 2 2 2
176/300 Read block 4080 len 28672
177/300 Write block 3309 len 20480 data: 3 3 3 2 3
178/300 Write block 2352 len 36864 data: 3 3 3 3 3 2 2 2 2
179/300 Read block 1256 len 24576
180/300 Read block 102 len 40960
181/300 Write block 2203 len 4096 data: 2
182/300 Read block 4609 len 12288
183/300 Write block 3577 len 24576 data: 2 2 2 2 2 2
184/300 Write block 138 len 16384 data: 2 2 2 3
185/300 Write block 4434 len 20480 data: 3 2 2 2 2
186/300 Write block 865 len 32768 data: 2 2 2 2 3 3 2 2
187/300 Read block 2167 len 40960
188/300 Write block 3423 len 40960 data: 2 2 2 2 3 3 3 3 2 2
189/300 Flush
190/300 Flush
191/300 Write block 3949 len 28672 data: 2 2 2 2 2 2 2
192/300 Read block 313 len 16384
193/300 Read block 3933 len 24576
194/300 Read block 3964 len 20480
195/300 Read block 2030 len 36864
196/300 Write block 1909 len 28672 data: 2 2 2 2 2 2 2
197/300 Write block 3171 len 12288 data: 3 3 3
198/300 Flush
199/300 Read block 1020 len 16384
200/300 Write block 3339 len 20480 data: 3 3 3 4 5
201/300 Flush
202/300 Flush
203/300 Write block 4222 len 4096 data: 3
204/300 Read block 2284 len 20480
205/300 Read block 4961 len 20480
206/300 Read block 3993 len 36864
207/300 Read block 3656 len 8192
208/300 Read block 1552 len 32768
209/300 Read block 3813 len 20480
210/300 Write block 3050 len 12288 data: 2 2 2
211/300 Read block 3003 len 32768
212/300 Flush
213/300 Read block 3935 len 32768
214/300 Read block 1724 len 24576
215/300 Write block 181 len 32768 data: 3 3 3 3 3 3 3 2
216/300 Flush
217/300 Read block 214 len 20480
218/300 Read block 4130 len 4096
219/300 Write block 4775 len 24576 data: 2 2 2 2 2 2
220/300 Write block 2895 len 12288 data: 2 2 2
221/300 Write block 1536 len 8192 data: 2 2
222/300 Read block 114 len 16384
223/300 Read block 2820 len 36864
224/300 Write block 4578 len 12288 data: 2 2 3
225/300 Flush
226/300 Read block 752 len 16384
227/300 Write block 4123 len 36864 data: 3 3 3 3 3 2 2 2 2
228/300 Read block 3189 len 40960
229/300 Read block 94 len 16384
230/300 Read block 1918 len 20480
231/300 Read block 2095 len 8192
232/300 Read block 3607 len 12288
233/300 Write block 4134 len 4096 data: 2
234/300 Write block 1759 len 24576 data: 2 2 3 2 2 2
235/300 Write block 3101 len 12288 data: 2 2 2
236/300 Read block 1102 len 36864
237/300 Read block 2983 len 32768
238/300 Write block 3581 len 24576 data: 3 3 2 2 2 3
239/300 Read block 1354 len 32768
240/300 Write block 3144 len 8192 data: 2 2
241/300 Flush
242/300 Write block 4991 len 16384 data: 2 2 2 2
243/300 Read block 3103 len 12288
244/300 Write block 2485 len 24576 data: 3 3 3 3 2 2
245/300 Read block 3411 len 16384
246/300 Flush
247/300 Read block 461 len 24576
248/300 Write block 1057 len 36864 data: 2 2 2 2 3 3 3 3 3
249/300 Read block 1073 len 12288
250/300 Write block 3485 len 8192 data: 3 3
251/300 Write block 3600 len 8192 data: 2 2
252/300 Write block 4883 len 40960 data: 2 2 2 2 2 2 2 2 2 2
253/300 Read block 776 len 36864
254/300 Read block 1255 len 28672
255/300 Write block 3048 len 36864 data: 2 2 3 3 3 2 2 2 2
256/300 Write block 4487 len 24576 data: 3 3 3 3 2 2
257/300 Write block 4763 len 12288 data: 2 2 2
258/300 Write block 2331 len 36864 data: 2 2 2 2 2 2 2 2 2
259/300 Read block 1219 len 20480
260/300 Flush
261/300 Read block 706 len 16384
262/300 Write block 3062 len 4096 data: 2
263/300 Write block 810 len 36864 data: 2 2 2 2 2 2 2 2 2
264/300 Flush
265/300 Read block 258 len 36864
266/300 Read block 2107 len 24576
267/300 Read block 663 len 16384
268/300 Read block 2134 len 16384
269/300 Write block 4263 len 24576 data: 3 3 3 2 2 2
270/300 Write block 4488 len 8192 data: 4 4
271/300 Read block 960 len 12288
272/300 Write block 4530 len 40960 data: 2 2 2 2 2 2 3 3 3 3
273/300 Read block 1578 len 12288
274/300 Read block 4376 len 12288
275/300 Write block 774 len 12288 data: 2 2 2
276/300 Read block 735 len 32768
277/300 Read block 3457 len 16384
278/300 Write block 3211 len 12288 data: 2 2 2
279/300 Write block 1549 len 36864 data: 2 2 2 2 2 2 3 3 3
280/300 Write block 4713 len 8192 data: 2 3
281/300 Write block 2313 len 4096 data: 3
282/300 Read block 3224 len 8192
283/300 Read block 1212 len 36864
284/300 Read block 1588 len 8192
285/300 Write block 4476 len 12288 data: 4 4 3
286/300 Write block 4298 len 8192 data: 2 2
287/300 Read block 3758 len 36864
288/300 Write block 2990 len 12288 data: 2 2 2
289/300 Write block 1912 len 8192 data: 3 3
290/300 Read block 4082 len 40960
291/300 Flush
292/300 Read block 4198 len 4096
293/300 Flush
294/300 Read block 1238 len 36864
295/300 Read block 3785 len 24576
296/300 Read block 3481 len 16384
297/300 Write block 146 len 32768 data: 2 2 2 2 2 2 2 2
298/300 Read block 805 len 4096
299/300 Read block 4762 len 40960
300/300 Write block 1406 len 12288 data: 3 3 2
Main task has work to do, go find it
got dsc Start(2)
start 2
Feb 13 21:40:04.877 INFO request completed, latency_us: 220, response_code: 204, uri: /start/cid/2, method: POST, req_id: 9891f0b4-2a01-41ae-a505-6c90bf705780, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
States: Active Active Offline
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
604 Acked 1603 Flush 0 Done Done New true
605 Acked 1604 Write 3 Done Done New true
606 Acked 1605 Write 4 Done Done New true
607 Acked 1606 Write 7 Done Done New true
608 Acked 1607 Read 7 Done Done New true
609 Acked 1608 Write 4 Done Done New false
610 Acked 1609 Read 2 Done Done New false
611 Acked 1610 Read 8 Done Done New false
612 Acked 1611 Read 7 Done Done New false
613 Acked 1612 Read 2 Done Done New false
614 Acked 1613 Read 2 Done Done New false
615 Acked 1614 Write 6 Done Done New false
616 Acked 1615 Read 2 Done Done New false
617 Acked 1616 Read 5 Done Done New false
618 Acked 1617 Read 8 Done Done New false
619 Acked 1618 Read 8 Done Done New false
620 Acked 1619 Write 1 Done Done New false
621 Acked 1620 Read 8 Done Done New false
622 Acked 1621 Write 10 Done Done New false
623 Acked 1622 Write 6 Done Done New false
624 Acked 1623 Write 9 Done Done New false
625 Acked 1624 Write 1 Done Done New false
626 Acked 1625 Read 5 Done Done New false
627 Acked 1626 Read 7 Done Done New false
628 Acked 1627 Write 8 Done Done New false
629 Acked 1628 Write 5 Done Done New false
630 Acked 1629 Write 3 Done Done New false
631 Acked 1630 Read 10 Done Done New false
632 Acked 1631 Read 6 Done Done New false
633 Acked 1632 Write 5 Done Done New false
634 Acked 1633 Write 1 Done Done New false
635 Acked 1634 Read 9 Done Done New false
636 Acked 1635 Flush 0 Done Done New false
637 Acked 1636 Write 4 Done Done New false
638 Acked 1637 Write 8 Done Done New false
639 Acked 1638 Write 10 Done Done New false
640 Acked 1639 Flush 0 Done Done New false
641 Acked 1640 Write 9 Done Done New false
642 Acked 1641 Write 7 Done Done New false
643 Acked 1642 Write 4 Done Done New false
644 Acked 1643 Write 3 Done Done New false
645 Acked 1644 Read 10 Done Done New false
646 Acked 1645 Write 10 Done Done New false
647 Acked 1646 Write 7 Done Done New false
648 Acked 1647 Write 2 Done Done New false
649 Acked 1648 Flush 0 Done Done New false
650 Acked 1649 Read 7 Done Done New false
651 Acked 1650 Write 6 Done Done New false
652 Acked 1651 Read 1 Done Done New false
653 Acked 1652 Write 5 Done Done New false
654 Acked 1653 Read 10 Done Done New false
655 Acked 1654 Write 2 Done Done New false
656 Acked 1655 Flush 0 Done Done New false
657 Acked 1656 Read 1 Done Done New false
658 Acked 1657 Write 7 Done Done New false
659 Acked 1658 Write 2 Done Done New false
660 Acked 1659 Write 6 Done Done New false
661 Acked 1660 Write 5 Done Done New false
662 Acked 1661 Write 1 Done Done New false
663 Acked 1662 Flush 0 Done Done New false
664 Acked 1663 Write 3 Done Done New false
665 Acked 1664 Write 1 Done Done New false
666 Acked 1665 Read 7 Done Done New false
667 Acked 1666 Read 10 Done Done New false
668 Acked 1667 Read 1 Done Done New false
669 Acked 1668 Flush 0 Done Done New false
670 Acked 1669 Write 7 Done Done New false
671 Acked 1670 Read 10 Done Done New false
672 Acked 1671 Write 10 Done Done New false
673 Acked 1672 Read 7 Done Done New false
674 Acked 1673 Write 8 Done Done New false
675 Acked 1674 Read 2 Done Done New false
676 Acked 1675 Write 7 Done Done New false
677 Acked 1676 Flush 0 Done Done New false
678 Acked 1677 Write 1 Done Done New false
679 Acked 1678 Read 1 Done Done New false
680 Acked 1679 Read 2 Done Done New false
681 Acked 1680 Flush 0 Done Done New false
682 Acked 1681 Write 10 Done Done New false
683 Acked 1682 Read 2 Done Done New false
684 Acked 1683 Write 2 Done Done New false
685 Acked 1684 Read 8 Done Done New false
686 Acked 1685 Read 8 Done Done New false
687 Acked 1686 Read 10 Done Done New false
688 Acked 1687 Read 10 Done Done New false
689 Acked 1688 Write 4 Done Done New false
690 Acked 1689 Read 1 Done Done New false
691 Acked 1690 Write 3 Done Done New false
692 Acked 1691 Flush 0 Done Done New false
693 Acked 1692 Flush 0 Done Done New false
694 Acked 1693 Write 9 Done Done New false
695 Acked 1694 Read 2 Done Done New false
696 Acked 1695 Write 6 Done Done New false
697 Acked 1696 Write 2 Done Done New false
698 Acked 1697 Read 2 Done Done New false
699 Acked 1698 Write 3 Done Done New false
700 Acked 1699 Read 6 Done Done New false
701 Acked 1700 Write 3 Done Done New false
702 Acked 1701 Write 5 Done Done New false
703 Acked 1702 Write 8 Done Done New false
704 Acked 1703 Read 7 Done Done New false
705 Acked 1704 Flush 0 Done Done New false
706 Acked 1705 Flush 0 Done Done New false
707 Acked 1706 Read 10 Done Done New false
708 Acked 1707 Read 7 Done Done New false
709 Acked 1708 Flush 0 Done Done New false
710 Acked 1709 Read 5 Done Done New false
711 Acked 1710 Flush 0 Done Done New false
712 Acked 1711 Write 2 Done Done New false
713 Acked 1712 Read 9 Done Done New false
714 Acked 1713 Flush 0 Done Done New false
715 Acked 1714 Write 1 Done Done New false
716 Acked 1715 Read 2 Done Done New false
717 Acked 1716 Read 10 Done Done New false
718 Acked 1717 Read 9 Done Done New false
719 Acked 1718 Write 6 Done Done New false
720 Acked 1719 Read 7 Done Done New false
721 Acked 1720 Write 9 Done Done New false
722 Acked 1721 Read 8 Done Done New false
723 Acked 1722 Write 1 Done Done New false
724 Acked 1723 Read 4 Done Done New false
725 Acked 1724 Write 3 Done Done New false
726 Acked 1725 Write 6 Done Done New false
727 Acked 1726 Write 8 Done Done New false
728 Acked 1727 Read 1 Done Done New false
729 Acked 1728 Flush 0 Done Done New false
730 Acked 1729 Read 8 Done Done New false
731 Acked 1730 Write 1 Done Done New false
732 Acked 1731 Write 7 Done Done New false
733 Acked 1732 Read 1 Done Done New false
734 Acked 1733 Write 4 Done Done New false
735 Acked 1734 Write 9 Done Done New false
736 Acked 1735 Read 10 Done Done New false
737 Acked 1736 Read 4 Done Done New false
738 Acked 1737 Read 10 Done Done New false
739 Acked 1738 Read 6 Done Done New false
740 Acked 1739 Write 2 Done Done New false
741 Acked 1740 Write 9 Done Done New false
742 Acked 1741 Write 1 Done Done New false
743 Acked 1742 Write 4 Done Done New false
744 Acked 1743 Write 9 Done Done New false
745 Acked 1744 Write 1 Done Done New false
746 Acked 1745 Read 9 Done Done New false
747 Acked 1746 Write 8 Done Done New false
748 Acked 1747 Read 3 Done Done New false
749 Acked 1748 Write 4 Done Done New false
750 Acked 1749 Read 7 Done Done New false
751 Acked 1750 Write 6 Done Done New false
752 Acked 1751 Write 5 Done Done New false
753 Acked 1752 Read 8 Done Done New false
754 Acked 1753 Read 10 Done Done New false
755 Acked 1754 Write 3 Done Done New false
756 Acked 1755 Write 2 Done Done New false
757 Acked 1756 Read 10 Done Done New false
758 Acked 1757 Write 10 Done Done New false
759 Acked 1758 Write 4 Done Done New false
760 Acked 1759 Read 5 Done Done New false
761 Acked 1760 Write 7 Done Done New false
762 Acked 1761 Read 9 Done Done New false
763 Acked 1762 Read 8 Done Done New false
764 Acked 1763 Read 9 Done Done New false
765 Acked 1764 Read 10 Done Done New false
766 Acked 1765 Write 2 Done Done New false
767 Acked 1766 Read 8 Done Done New false
768 Acked 1767 Read 3 Done Done New false
769 Acked 1768 Read 6 Done Done New false
770 Acked 1769 Read 7 Done Done New false
771 Acked 1770 Write 7 Done Done New false
772 Acked 1771 Flush 0 Done Done New false
773 Acked 1772 Read 7 Done Done New false
774 Acked 1773 Read 4 Done Done New false
775 Acked 1774 Read 1 Done Done New false
776 Acked 1775 Read 2 Done Done New false
777 Acked 1776 Flush 0 Done Done New false
778 Acked 1777 Write 6 Done Done New false
779 Acked 1778 Read 7 Done Done New false
780 Acked 1779 Write 5 Done Done New false
781 Acked 1780 Write 9 Done Done New false
782 Acked 1781 Read 6 Done Done New false
783 Acked 1782 Read 10 Done Done New false
784 Acked 1783 Write 1 Done Done New false
785 Acked 1784 Read 3 Done Done New false
786 Acked 1785 Write 6 Done Done New false
787 Acked 1786 Write 4 Done Done New false
788 Acked 1787 Write 5 Done Done New false
789 Acked 1788 Write 8 Done Done New false
790 Acked 1789 Read 10 Done Done New false
791 Acked 1790 Write 10 Done Done New false
792 Acked 1791 Flush 0 Done Done New false
793 Acked 1792 Flush 0 Done Done New false
794 Acked 1793 Write 7 Done Done New false
795 Acked 1794 Read 4 Done Done New false
796 Acked 1795 Read 6 Done Done New false
797 Acked 1796 Read 5 Done Done New false
798 Acked 1797 Read 9 Done Done New false
799 Acked 1798 Write 7 Done Done New false
800 Acked 1799 Write 3 Done Done New false
801 Acked 1800 Flush 0 Done Done New false
802 Acked 1801 Read 4 Done Done New false
803 Acked 1802 Write 5 Done Done New false
804 Acked 1803 Flush 0 Done Done New false
805 Acked 1804 Flush 0 Done Done New false
806 Acked 1805 Write 1 Done Done New false
807 Acked 1806 Read 5 Done Done New false
808 Acked 1807 Read 5 Done Done New false
809 Acked 1808 Read 9 Done Done New false
810 Acked 1809 Read 2 Done Done New false
811 Acked 1810 Read 8 Done Done New false
812 Acked 1811 Read 5 Done Done New false
813 Acked 1812 Write 3 Done Done New false
814 Acked 1813 Read 8 Done Done New false
815 Acked 1814 Flush 0 Done Done New false
816 Acked 1815 Read 8 Done Done New false
817 Acked 1816 Read 6 Done Done New false
818 Acked 1817 Write 8 Done Done New false
819 Acked 1818 Flush 0 Done Done New false
820 Acked 1819 Read 5 Done Done New false
821 Acked 1820 Read 1 Done Done New false
822 Acked 1821 Write 6 Done Done New false
823 Acked 1822 Write 3 Done Done New false
824 Acked 1823 Write 2 Done Done New false
825 Acked 1824 Read 4 Done Done New false
826 Acked 1825 Read 9 Done Done New false
827 Acked 1826 Write 3 Done Done New false
828 Acked 1827 Flush 0 Done Done New false
829 Acked 1828 Read 4 Done Done New false
830 Acked 1829 Write 9 Done Done New false
831 Acked 1830 Read 10 Done Done New false
832 Acked 1831 Read 4 Done Done New false
833 Acked 1832 Read 5 Done Done New false
834 Acked 1833 Read 2 Done Done New false
835 Acked 1834 Read 3 Done Done New false
836 Acked 1835 Write 1 Done Done New false
837 Acked 1836 Write 6 Done Done New false
838 Acked 1837 Write 3 Done Done New false
839 Acked 1838 Read 9 Done Done New false
840 Acked 1839 Read 8 Done Done New false
841 Acked 1840 Write 6 Done Done New false
842 Acked 1841 Read 8 Done Done New false
843 Acked 1842 Write 2 Done Done New false
844 Acked 1843 Flush 0 Done Done New false
845 Acked 1844 Write 4 Done Done New false
846 Acked 1845 Read 3 Done Done New false
847 Acked 1846 Write 6 Done Done New false
848 Acked 1847 Read 4 Done Done New false
849 Acked 1848 Flush 0 Done Done New false
850 Acked 1849 Read 6 Done Done New false
851 Acked 1850 Write 9 Done Done New false
852 Acked 1851 Read 3 Done Done New false
853 Acked 1852 Write 2 Done Done New false
854 Acked 1853 Write 2 Done Done New false
855 Acked 1854 Write 10 Done Done New false
856 Acked 1855 Read 9 Done Done New false
857 Acked 1856 Read 7 Done Done New false
858 Acked 1857 Write 9 Done Done New false
859 Acked 1858 Write 6 Done Done New false
860 Acked 1859 Write 3 Done Done New false
861 Acked 1860 Write 9 Done Done New false
862 Acked 1861 Read 5 Done Done New false
863 Acked 1862 Flush 0 Done Done New false
864 Acked 1863 Read 4 Done Done New false
865 Acked 1864 Write 1 Done Done New false
866 Acked 1865 Write 9 Done Done New false
867 Acked 1866 Flush 0 Done Done New false
868 Acked 1867 Read 9 Done Done New false
869 Acked 1868 Read 6 Done Done New false
870 Acked 1869 Read 4 Done Done New false
871 Acked 1870 Read 4 Done Done New false
872 Acked 1871 Write 6 Done Done New false
873 Acked 1872 Write 2 Done Done New false
874 Acked 1873 Read 3 Done Done New false
875 Acked 1874 Write 10 Done Done New false
876 Acked 1875 Read 3 Done Done New false
877 Acked 1876 Read 3 Done Done New false
878 Acked 1877 Write 3 Done Done New false
879 Acked 1878 Read 8 Done Done New false
880 Acked 1879 Read 4 Done Done New false
881 Acked 1880 Write 3 Done Done New false
882 Acked 1881 Write 9 Done Done New false
883 Acked 1882 Write 2 Done Done New false
884 Acked 1883 Write 1 Done Done New false
885 Acked 1884 Read 2 Done Done New false
886 Acked 1885 Read 9 Done Done New false
887 Acked 1886 Read 2 Done Done New false
888 Acked 1887 Write 3 Done Done New false
889 Acked 1888 Write 2 Done Done New false
890 Acked 1889 Read 9 Done Done New false
891 Acked 1890 Write 3 Done Done New false
892 Acked 1891 Write 2 Done Done New false
893 Acked 1892 Read 10 Done Done New false
894 Acked 1893 Flush 0 Done Done New false
895 Acked 1894 Read 1 Done Done New false
896 Acked 1895 Flush 0 Done Done New false
897 Acked 1896 Read 9 Done Done New false
898 Acked 1897 Read 6 Done Done New false
899 Acked 1898 Read 4 Done Done New false
900 Acked 1899 Write 8 Done Done New false
901 Acked 1900 Read 1 Done Done New false
902 Acked 1901 Read 10 Done Done New false
903 Acked 1902 Write 3 Done Done New false
STATES DS:0 DS:1 DS:2 TOTAL
New 0 0 300 300
Sent 0 0 0 0
Done 300 300 0 600
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1895 1895 1602
Downstairs last five completed: 1602 1601 1600 1599 1598
Upstairs last five completed: 903 902 901 900 899
CLIENT: Up:0 ds:300 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:05.177763195Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:05.177816613Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:05.177839782Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:05.177857041Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:05.1778699Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 301 jobs since flush 1602","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:05.17788424Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:05.178029365Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:06.178489805Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:06.178545493Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:06.178574221Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:06.178603691Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:06.17862599Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 301 jobs since flush 1602","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:06.178647719Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:06.178768924Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1167)
[8830][2] reports Running
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:07.18040381Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:07.180534765Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:07.181832665Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:07.181863564Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 1602","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:07.181882733Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 1602","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:07.182042997Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:07.182081546Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:07.182107925Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:07.182126154Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:07.182140333Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 1903 1902 1901 1900 1899
Upstairs last five completed: 904 903 902 901 900
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Feb 13 21:40:08.881 INFO request completed[8810] Got stop action so:false kr:false
, latency_us: 369, response_code: 204, uri: /stop/cid/0, method: POST, req_id: bed60a61-f14f-4ba2-9cf1-5ecc98f4fe5c, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:08.881850735Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:08.881911522Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:08.881950771Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:08.88196879Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:40:08.881 INFO request completed, latency_us: 164, response_code: 200, uri: /state/cid/0, method: GET, req_id: 0cc63d93-1e52-479d-b9f2-8d501fba0d4c, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"[0] client re-new 0 jobs since flush 1903","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:08.882059027Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:08.882086115Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:09.883250389Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:09.883316876Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:09.883353805Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:09.883387004Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:09.883411053Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 1903","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:09.883437492Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:09.883468301Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:10.884964702Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:10.885047329Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:10.885080838Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:10.885192534Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:10.885254831Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 1903","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:10.885313579Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:10.885410176Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:11.885813668Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:11.885877126Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:11.885899045Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:11.885919724Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:11.885935783Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 1903","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:11.885965852Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:11.885999091Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:40:12.883 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/0, method: GET, req_id: 111389f5-5cac-4449-8fd2-92d8d6972917, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Read block 1568 len 24576
002/300 Read block 1082 len 32768
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:12.886525259Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:12.886582517Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:12.886601296Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:12.886618795Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:12.886631845Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 2 jobs since flush 1903","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:12.886645645Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:12.886659554Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
003/300 Write block 2880 len 20480 data: 2 2 2 2 2
004/300 Read block 2256 len 32768
005/300 Read block 514 len 4096
006/300 Read block 2520 len 28672
007/300 Write block 3225 len 20480 data: 5 5 3 3 2
008/300 Write block 2403 len 4096 data: 2
009/300 Read block 73 len 36864
010/300 Write block 244 len 8192 data: 3 3
011/300 Read block 90 len 12288
012/300 Write block 2981 len 28672 data: 3 3 2 2 2 2 2
013/300 Read block 2746 len 4096
014/300 Flush
015/300 Read block 3356 len 4096
016/300 Read block 2848 len 28672
017/300 Write block 565 len 36864 data: 4 4 4 3 3 3 2 2 2
018/300 Write block 268 len 20480 data: 2 2 2 3 3
019/300 Write block 2237 len 8192 data: 2 2
020/300 Read block 2504 len 36864
021/300 Flush
022/300 Read block 1464 len 16384
023/300 Flush
024/300 Read block 929 len 36864
025/300 Write block 4272 len 28672 data: 2 2 2 2 2 2 2
026/300 Read block 1931 len 28672
027/300 Write block 1100 len 4096 data: 3
028/300 Read block 2273 len 40960
029/300 Read block 3842 len 32768
030/300 Read block 61 len 20480
031/300 Read block 2199 len 24576
032/300 Read block 1429 len 36864
033/300 Write block 1717 len 32768 data: 2 2 3 3 3 4 4 4
034/300 Flush
035/300 Write block 4656 len 36864 data: 2 2 2 2 2 2 2 2 2
036/300 Write block 2132 len 40960 data: 2 2 2 2 2 2 2 2 2 2
037/300 Write block 4368 len 16384 data: 3 3 3 3
038/300 Read block 2481 len 32768
039/300 Read block 4122 len 36864
040/300 Read block 372 len 28672
041/300 Read block 2610 len 28672
042/300 Read block 2296 len 40960
043/300 Write block 242 len 28672 data: 3 3 4 4 3 2 2
044/300 Read block 2221 len 20480
045/300 Read block 3085 len 4096
046/300 Write block 2687 len 12288 data: 2 2 2
047/300 Read block 3376 len 20480
048/300 Write block 712 len 40960 data: 4 4 4 3 3 3 2 2 2 2
049/300 Read block 2354 len 4096
050/300 Write block 3498 len 16384 data: 2 2 2 2
051/300 Read block 2573 len 24576
052/300 Read block 3360 len 32768
053/300 Read block 410 len 32768
054/300 Write block 130 len 20480 data: 5 5 5 3 3
055/300 Write block 1989 len 36864 data: 2 2 2 2 2 3 3 3 4
056/300 Write block 3964 len 12288 data: 2 2 2
057/300 Read block 826 len 32768
058/300 Write block 4547 len 16384 data: 2 2 2 2
059/300 Write block 1644 len 36864 data: 3 2 2 2 2 2 2 2 2
060/300 Write block 3514 len 36864 data: 2 2 2 2 2 2 2 2 2
061/300 Write block 2662 len 28672 data: 2 2 2 2 2 2 2
062/300 Read block 1604 len 12288
063/300 Read block 2693 len 40960
064/300 Read block 166 len 24576
065/300 Write block 721 len 40960 data: 3 2 2 2 2 2 2 2 2 3
066/300 Write block 2503 len 4096 data: 3
067/300 Read block 1037 len 12288
068/300 Write block 3538 len 4096 data: 2
069/300 Write block 1930 len 16384 data: 2 2 2 2
070/300 Read block 806 len 36864
071/300 Read block 919 len 16384
072/300 Read block 4391 len 20480
073/300 Read block 2463 len 8192
074/300 Flush
075/300 Read block 4831 len 12288
076/300 Flush
077/300 Write block 2132 len 40960 data: 3 3 3 3 3 3 3 3 3 3
078/300 Read block 1887 len 32768
079/300 Write block 2142 len 32768 data: 2 2 2 2 2 2 2 2
080/300 Write block 103 len 4096 data: 2
081/300 Write block 718 len 16384 data: 3 3 3 4
082/300 Write block 136 len 36864 data: 2 2 3 3 3 4 2 2 2
083/300 Read block 511 len 32768
084/300 Read block 4791 len 24576
085/300 Read block 136 len 20480
086/300 Read block 4810 len 40960
087/300 Write block 2128 len 24576 data: 2 2 2 2 4 4
088/300 Flush
089/300 Read block 3381 len 36864
090/300 Read block 692 len 40960
091/300 Read block 299 len 40960
092/300 Read block 929 len 4096
093/300 Read block 2587 len 20480
094/300 Read block 1808 len 16384
095/300 Read block 2154 len 12288
096/300 Read block 4430 len 8192
097/300 Write block 2067 len 20480 data: 2 2 2 2 2
098/300 Write block 4447 len 20480 data: 3 3 3 3 3
099/300 Write block 2778 len 4096 data: 2
100/300 Flush
101/300 Write block 1121 len 28672 data: 2 2 2 2 2 2 2
102/300 Write block 3972 len 36864 data: 3 3 3 2 2 2 2 2 2
103/300 Read block 3614 len 4096
104/300 Read block 3528 len 32768
105/300 Write block 1460 len 24576 data: 2 2 2 2 2 2
106/300 Read block 1873 len 24576
107/300 Read block 611 len 24576
108/300 Read block 4434 len 28672
109/300 Read block 2537 len 28672
110/300 Write block 4940 len 24576 data: 3 3 4 3 3 3
111/300 Write block 4642 len 4096 data: 3
112/300 Read block 3975 len 12288
113/300 Write block 3429 len 40960 data: 4 4 3 3 2 3 3 3 2 2
114/300 Write block 1962 len 28672 data: 2 2 2 2 2 2 2
115/300 Read block 4185 len 24576
116/300 Write block 3443 len 32768 data: 2 2 2 2 2 2 2 2
117/300 Read block 564 len 28672
118/300 Write block 2290 len 16384 data: 2 2 2 2
119/300 Read block 4822 len 12288
120/300 Write block 2463 len 40960 data: 2 2 2 2 2 2 2 2 3 3
121/300 Read block 3024 len 16384
122/300 Read block 1883 len 32768
123/300 Read block 4583 len 24576
124/300 Read block 4001 len 32768
125/300 Write block 655 len 40960 data: 2 2 2 2 2 2 2 2 2 2
126/300 Read block 2473 len 12288
127/300 Write block 334 len 12288 data: 2 2 2
128/300 Write block 2563 len 24576 data: 2 2 2 3 3 3
129/300 Write block 1164 len 40960 data: 2 2 2 2 2 2 2 2 2 2
130/300 Write block 3828 len 40960 data: 2 2 2 2 2 2 2 2 2 2
131/300 Read block 584 len 4096
132/300 Write block 345 len 40960 data: 2 2 2 2 2 2 3 3 3 3
133/300 Flush
134/300 Read block 2425 len 16384
135/300 Flush
136/300 Read block 793 len 40960
137/300 Write block 1837 len 20480 data: 2 2 2 2 2
138/300 Write block 1389 len 36864 data: 3 3 3 2 2 2 2 2 2
139/300 Read block 3207 len 16384
140/300 Read block 1080 len 24576
141/300 Read block 3358 len 32768
142/300 Write block 4566 len 40960 data: 2 2 2 2 2 2 2 2 2 2
143/300 Read block 494 len 32768
144/300 Write block 2649 len 28672 data: 4 4 4 4 3 3 3
145/300 Read block 3771 len 40960
146/300 Read block 2091 len 20480
147/300 Write block 2146 len 4096 data: 3
148/300 Read block 3542 len 40960
149/300 Read block 1682 len 8192
150/300 Read block 2456 len 4096
151/300 Read block 3030 len 40960
152/300 Flush
153/300 Read block 2014 len 8192
154/300 Read block 4720 len 24576
155/300 Read block 872 len 36864
156/300 Write block 157 len 36864 data: 2 2 2 2 3 4 5 6 5
157/300 Write block 2782 len 12288 data: 2 2 2
158/300 Write block 4650 len 8192 data: 2 2
159/300 Read block 1076 len 4096
160/300 Read block 2598 len 28672
161/300 Read block 1694 len 40960
162/300 Read block 4566 len 16384
163/300 Read block 4751 len 36864
164/300 Flush
165/300 Read block 3598 len 8192
166/300 Flush
167/300 Read block 235 len 20480
168/300 Write block 1323 len 8192 data: 2 2
169/300 Read block 2868 len 40960
170/300 Read block 3229 len 40960
171/300 Read block 775 len 8192
172/300 Write block 4946 len 40960 data: 4 3 4 4 3 3 3 3 3 3
173/300 Write block 461 len 8192 data: 2 2
174/300 Flush
175/300 Write block 3146 len 8192 data: 2 3
176/300 Write block 4749 len 16384 data: 2 2 2 2
177/300 Write block 2756 len 28672 data: 2 2 2 2 2 2 2
178/300 Read block 3314 len 40960
179/300 Read block 2788 len 24576
180/300 Read block 4887 len 24576
181/300 Write block 1773 len 24576 data: 3 3 3 3 3 3
182/300 Flush
183/300 Write block 1655 len 36864 data: 2 3 3 3 4 3 3 2 2
184/300 Write block 2244 len 4096 data: 3
185/300 Read block 1352 len 24576
186/300 Read block 886 len 8192
187/300 Read block 1802 len 40960
188/300 Write block 3688 len 8192 data: 3 3
189/300 Write block 4004 len 24576 data: 2 2 2 2 2 2
190/300 Write block 120 len 20480 data: 2 3 3 5 5
191/300 Read block 143 len 20480
192/300 Read block 1246 len 24576
193/300 Flush
194/300 Flush
195/300 Read block 2054 len 40960
196/300 Read block 4007 len 36864
197/300 Read block 2034 len 28672
198/300 Read block 3633 len 16384
199/300 Flush
200/300 Write block 4552 len 24576 data: 2 2 2 2 2 2
201/300 Read block 63 len 4096
202/300 Write block 4402 len 4096 data: 2
203/300 Flush
204/300 Write block 1548 len 36864 data: 2 3 3 3 3 3 3 4 4
205/300 Write block 3994 len 40960 data: 2 2 2 2 2 2 2 2 2 2
206/300 Read block 3959 len 4096
207/300 Read block 3389 len 36864
208/300 Read block 3987 len 8192
209/300 Write block 3278 len 16384 data: 2 2 2 3
210/300 Write block 2085 len 36864 data: 2 2 2 2 2 2 2 2 2
211/300 Read block 238 len 36864
212/300 Write block 2134 len 36864 data: 4 4 4 4 4 4 4 4 3
213/300 Flush
214/300 Write block 3921 len 32768 data: 2 2 2 2 2 2 2 3
215/300 Read block 68 len 4096
216/300 Read block 869 len 8192
217/300 Read block 2437 len 40960
218/300 Read block 3367 len 24576
219/300 Read block 1318 len 8192
220/300 Flush
221/300 Read block 3984 len 28672
222/300 Write block 2655 len 36864 data: 4 3 2 2 2 2 2 3 3
223/300 Read block 985 len 28672
224/300 Flush
225/300 Write block 1596 len 32768 data: 3 3 3 3 3 3 3 3
226/300 Read block 3669 len 20480
227/300 Read block 1758 len 4096
228/300 Write block 1521 len 4096 data: 2
229/300 Write block 1103 len 40960 data: 2 3 3 3 3 2 2 2 2 2
230/300 Read block 3421 len 12288
231/300 Read block 83 len 36864
232/300 Read block 4238 len 8192
233/300 Write block 4733 len 20480 data: 2 2 2 2 2
234/300 Write block 3503 len 36864 data: 2 2 2 2 2 2 2 2 2
235/300 Read block 913 len 40960
236/300 Read block 1131 len 24576
237/300 Flush
238/300 Read block 3003 len 4096
239/300 Read block 458 len 16384
240/300 Write block 639 len 40960 data: 3 3 3 2 2 2 2 2 2 2
241/300 Read block 17 len 12288
242/300 Read block 545 len 32768
243/300 Write block 4985 len 36864 data: 2 2 3 3 3 3 3 3 3
244/300 Read block 1573 len 16384
245/300 Read block 4832 len 8192
246/300 Write block 1418 len 16384 data: 2 2 2 2
247/300 Write block 4564 len 16384 data: 2 2 3 3
248/300 Write block 4148 len 16384 data: 2 2 2 2
249/300 Flush
250/300 Write block 2985 len 12288 data: 3 3 3
251/300 Read block 286 len 8192
252/300 Write block 3669 len 28672 data: 2 2 2 2 2 2 3
253/300 Write block 4451 len 4096 data: 4
254/300 Read block 2091 len 36864
255/300 Write block 1410 len 16384 data: 2 2 2 2
256/300 Flush
257/300 Read block 3892 len 8192
258/300 Write block 2309 len 16384 data: 2 3 3 3
259/300 Write block 683 len 32768 data: 3 3 3 3 3 3 3 3
260/300 Flush
261/300 Write block 2096 len 20480 data: 2 2 2 2 2
262/300 Read block 4550 len 16384
263/300 Read block 839 len 36864
264/300 Read block 4114 len 16384
265/300 Read block 4226 len 24576
266/300 Write block 435 len 4096 data: 2
267/300 Read block 2791 len 40960
268/300 Write block 2 len 8192 data: 2 2
269/300 Write block 3586 len 32768 data: 4 3 3 3 3 3 3 3
270/300 Read block 2177 len 12288
271/300 Flush
272/300 Read block 1597 len 28672
273/300 Write block 3119 len 32768 data: 3 3 3 3 3 3 2 2
274/300 Read block 2861 len 4096
275/300 Read block 182 len 36864
276/300 Read block 490 len 28672
277/300 Read block 974 len 12288
278/300 Read block 2317 len 40960
279/300 Read block 1870 len 32768
280/300 Read block 2117 len 40960
281/300 Read block 1467 len 20480
282/300 Read block 4653 len 40960
283/300 Write block 151 len 20480 data: 3 3 3 2 2
284/300 Read block 854 len 4096
285/300 Write block 3821 len 36864 data: 3 3 3 3 3 2 2 3 3
286/300 Flush
287/300 Read block 1861 len 12288
288/300 Write block 3453 len 4096 data: 2
289/300 Read block 4807 len 20480
290/300 Read block 3256 len 16384
291/300 Read block 329 len 4096
292/300 Write block 380 len 28672 data: 2 2 2 2 2 2 2
293/300 Read block 3868 len 28672
294/300 Read block 2670 len 36864
295/300 Read block 4535 len 12288
296/300 Write block 1332 len 8192 data: 2 2
297/300 Write block 42 len 32768 data: 2 2 2 2 2 2 3 3
298/300 Read block 1125 len 16384
299/300 Write block 4323 len 8192 data: 3 3
300/300 Read block 1374 len 36864
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 13 21:40:13.722 INFO request completed, latency_us: 207, response_code: 204, uri: /start/cid/0, method: POST, req_id: b0e8edd5-3fbd-41bf-835f-cca4a0711293, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
905 Acked 1904 Read 6 New Done Done true
906 Acked 1905 Read 8 New Done Done true
907 Acked 1906 Write 5 New Done Done false
908 Acked 1907 Read 8 New Done Done false
909 Acked 1908 Read 1 New Done Done false
910 Acked 1909 Read 7 New Done Done false
911 Acked 1910 Write 5 New Done Done false
912 Acked 1911 Write 1 New Done Done false
913 Acked 1912 Read 9 New Done Done false
914 Acked 1913 Write 2 New Done Done false
915 Acked 1914 Read 3 New Done Done false
916 Acked 1915 Write 7 New Done Done false
917 Acked 1916 Read 1 New Done Done false
918 Acked 1917 Flush 0 New Done Done false
919 Acked 1918 Read 1 New Done Done false
920 Acked 1919 Read 7 New Done Done false
921 Acked 1920 Write 9 New Done Done false
922 Acked 1921 Write 5 New Done Done false
923 Acked 1922 Write 2 New Done Done false
924 Acked 1923 Read 9 New Done Done false
925 Acked 1924 Flush 0 New Done Done false
926 Acked 1925 Read 4 New Done Done false
927 Acked 1926 Flush 0 New Done Done false
928 Acked 1927 Read 9 New Done Done false
929 Acked 1928 Write 7 New Done Done false
930 Acked 1929 Read 7 New Done Done false
931 Acked 1930 Write 1 New Done Done false
932 Acked 1931 Read 10 New Done Done false
933 Acked 1932 Read 8 New Done Done false
934 Acked 1933 Read 5 New Done Done false
935 Acked 1934 Read 6 New Done Done false
936 Acked 1935 Read 9 New Done Done false
937 Acked 1936 Write 8 New Done Done false
938 Acked 1937 Flush 0 New Done Done false
939 Acked 1938 Write 9 New Done Done false
940 Acked 1939 Write 10 New Done Done false
941 Acked 1940 Write 4 New Done Done false
942 Acked 1941 Read 8 New Done Done false
943 Acked 1942 Read 9 New Done Done false
944 Acked 1943 Read 7 New Done Done false
945 Acked 1944 Read 7 New Done Done false
946 Acked 1945 Read 10 New Done Done false
947 Acked 1946 Write 7 New Done Done false
948 Acked 1947 Read 5 New Done Done false
949 Acked 1948 Read 1 New Done Done false
950 Acked 1949 Write 3 New Done Done false
951 Acked 1950 Read 5 New Done Done false
952 Acked 1951 Write 10 New Done Done false
953 Acked 1952 Read 1 New Done Done false
954 Acked 1953 Write 4 New Done Done false
955 Acked 1954 Read 6 New Done Done false
956 Acked 1955 Read 8 New Done Done false
957 Acked 1956 Read 8 New Done Done false
958 Acked 1957 Write 5 New Done Done false
959 Acked 1958 Write 9 New Done Done false
960 Acked 1959 Write 3 New Done Done false
961 Acked 1960 Read 8 New Done Done false
962 Acked 1961 Write 4 New Done Done false
963 Acked 1962 Write 9 New Done Done false
964 Acked 1963 Write 9 New Done Done false
965 Acked 1964 Write 7 New Done Done false
966 Acked 1965 Read 3 New Done Done false
967 Acked 1966 Read 10 New Done Done false
968 Acked 1967 Read 6 New Done Done false
969 Acked 1968 Write 10 New Done Done false
970 Acked 1969 Write 1 New Done Done false
971 Acked 1970 Read 3 New Done Done false
972 Acked 1971 Write 1 New Done Done false
973 Acked 1972 Write 4 New Done Done false
974 Acked 1973 Read 9 New Done Done false
975 Acked 1974 Read 4 New Done Done false
976 Acked 1975 Read 5 New Done Done false
977 Acked 1976 Read 2 New Done Done false
978 Acked 1977 Flush 0 New Done Done false
979 Acked 1978 Read 3 New Done Done false
980 Acked 1979 Flush 0 New Done Done false
981 Acked 1980 Write 10 New Done Done false
982 Acked 1981 Read 8 New Done Done false
983 Acked 1982 Write 8 New Done Done false
984 Acked 1983 Write 1 New Done Done false
985 Acked 1984 Write 4 New Done Done false
986 Acked 1985 Write 9 New Done Done false
987 Acked 1986 Read 8 New Done Done false
988 Acked 1987 Read 6 New Done Done false
989 Acked 1988 Read 5 New Done Done false
990 Acked 1989 Read 10 New Done Done false
991 Acked 1990 Write 6 New Done Done false
992 Acked 1991 Flush 0 New Done Done false
993 Acked 1992 Read 9 New Done Done false
994 Acked 1993 Read 10 New Done Done false
995 Acked 1994 Read 10 New Done Done false
996 Acked 1995 Read 1 New Done Done false
997 Acked 1996 Read 5 New Done Done false
998 Acked 1997 Read 4 New Done Done false
999 Acked 1998 Read 3 New Done Done false
1000 Acked 1999 Read 2 New Done Done false
1001 Acked 2000 Write 5 New Done Done false
1002 Acked 2001 Write 5 New Done Done false
1003 Acked 2002 Write 1 New Done Done false
1004 Acked 2003 Flush 0 New Done Done false
1005 Acked 2004 Write 7 New Done Done false
1006 Acked 2005 Write 9 New Done Done false
1007 Acked 2006 Read 1 New Done Done false
1008 Acked 2007 Read 8 New Done Done false
1009 Acked 2008 Write 6 New Done Done false
1010 Acked 2009 Read 6 New Done Done false
1011 Acked 2010 Read 6 New Done Done false
1012 Acked 2011 Read 7 New Done Done false
1013 Acked 2012 Read 7 New Done Done false
1014 Acked 2013 Write 6 New Done Done false
1015 Acked 2014 Write 1 New Done Done false
1016 Acked 2015 Read 3 New Done Done false
1017 Acked 2016 Write 10 New Done Done false
1018 Acked 2017 Write 7 New Done Done false
1019 Acked 2018 Read 6 New Done Done false
1020 Acked 2019 Write 8 New Done Done false
1021 Acked 2020 Read 7 New Done Done false
1022 Acked 2021 Write 4 New Done Done false
1023 Acked 2022 Read 3 New Done Done false
1024 Acked 2023 Write 10 New Done Done false
1025 Acked 2024 Read 4 New Done Done false
1026 Acked 2025 Read 8 New Done Done false
1027 Acked 2026 Read 6 New Done Done false
1028 Acked 2027 Read 8 New Done Done false
1029 Acked 2028 Write 10 New Done Done false
1030 Acked 2029 Read 3 New Done Done false
1031 Acked 2030 Write 3 New Done Done false
1032 Acked 2031 Write 6 New Done Done false
1033 Acked 2032 Write 10 New Done Done false
1034 Acked 2033 Write 10 New Done Done false
1035 Acked 2034 Read 1 New Done Done false
1036 Acked 2035 Write 10 New Done Done false
1037 Acked 2036 Flush 0 New Done Done false
1038 Acked 2037 Read 4 New Done Done false
1039 Acked 2038 Flush 0 New Done Done false
1040 Acked 2039 Read 10 New Done Done false
1041 Acked 2040 Write 5 New Done Done false
1042 Acked 2041 Write 9 New Done Done false
1043 Acked 2042 Read 4 New Done Done false
1044 Acked 2043 Read 6 New Done Done false
1045 Acked 2044 Read 8 New Done Done false
1046 Acked 2045 Write 10 New Done Done false
1047 Acked 2046 Read 8 New Done Done false
1048 Acked 2047 Write 7 New Done Done false
1049 Acked 2048 Read 10 New Done Done false
1050 Acked 2049 Read 5 New Done Done false
1051 Acked 2050 Write 1 New Done Done false
1052 Acked 2051 Read 10 New Done Done false
1053 Acked 2052 Read 2 New Done Done false
1054 Acked 2053 Read 1 New Done Done false
1055 Acked 2054 Read 10 New Done Done false
1056 Acked 2055 Flush 0 New Done Done false
1057 Acked 2056 Read 2 New Done Done false
1058 Acked 2057 Read 6 New Done Done false
1059 Acked 2058 Read 9 New Done Done false
1060 Acked 2059 Write 9 New Done Done false
1061 Acked 2060 Write 3 New Done Done false
1062 Acked 2061 Write 2 New Done Done false
1063 Acked 2062 Read 1 New Done Done false
1064 Acked 2063 Read 7 New Done Done false
1065 Acked 2064 Read 10 New Done Done false
1066 Acked 2065 Read 4 New Done Done false
1067 Acked 2066 Read 9 New Done Done false
1068 Acked 2067 Flush 0 New Done Done false
1069 Acked 2068 Read 2 New Done Done false
1070 Acked 2069 Flush 0 New Done Done false
1071 Acked 2070 Read 5 New Done Done false
1072 Acked 2071 Write 2 New Done Done false
1073 Acked 2072 Read 10 New Done Done false
1074 Acked 2073 Read 10 New Done Done false
1075 Acked 2074 Read 2 New Done Done false
1076 Acked 2075 Write 10 New Done Done false
1077 Acked 2076 Write 2 New Done Done false
1078 Acked 2077 Flush 0 New Done Done false
1079 Acked 2078 Write 2 New Done Done false
1080 Acked 2079 Write 4 New Done Done false
1081 Acked 2080 Write 7 New Done Done false
1082 Acked 2081 Read 10 New Done Done false
1083 Acked 2082 Read 6 New Done Done false
1084 Acked 2083 Read 6 New Done Done false
1085 Acked 2084 Write 6 New Done Done false
1086 Acked 2085 Flush 0 New Done Done false
1087 Acked 2086 Write 9 New Done Done false
1088 Acked 2087 Write 1 New Done Done false
1089 Acked 2088 Read 6 New Done Done false
1090 Acked 2089 Read 2 New Done Done false
1091 Acked 2090 Read 10 New Done Done false
1092 Acked 2091 Write 2 New Done Done false
1093 Acked 2092 Write 6 New Done Done false
1094 Acked 2093 Write 5 New Done Done false
1095 Acked 2094 Read 5 New Done Done false
1096 Acked 2095 Read 6 New Done Done false
1097 Acked 2096 Flush 0 New Done Done false
1098 Acked 2097 Flush 0 New Done Done false
1099 Acked 2098 Read 10 New Done Done false
1100 Acked 2099 Read 9 New Done Done false
1101 Acked 2100 Read 7 New Done Done false
1102 Acked 2101 Read 4 New Done Done false
1103 Acked 2102 Flush 0 New Done Done false
1104 Acked 2103 Write 6 New Done Done false
1105 Acked 2104 Read 1 New Done Done false
1106 Acked 2105 Write 1 New Done Done false
1107 Acked 2106 Flush 0 New Done Done false
1108 Acked 2107 Write 9 New Done Done false
1109 Acked 2108 Write 10 New Done Done false
1110 Acked 2109 Read 1 New Done Done false
1111 Acked 2110 Read 9 New Done Done false
1112 Acked 2111 Read 2 New Done Done false
1113 Acked 2112 Write 4 New Done Done false
1114 Acked 2113 Write 9 New Done Done false
1115 Acked 2114 Read 9 New Done Done false
1116 Acked 2115 Write 9 New Done Done false
1117 Acked 2116 Flush 0 New Done Done false
1118 Acked 2117 Write 8 New Done Done false
1119 Acked 2118 Read 1 New Done Done false
1120 Acked 2119 Read 2 New Done Done false
1121 Acked 2120 Read 10 New Done Done false
1122 Acked 2121 Read 6 New Done Done false
1123 Acked 2122 Read 2 New Done Done false
1124 Acked 2123 Flush 0 New Done Done false
1125 Acked 2124 Read 7 New Done Done false
1126 Acked 2125 Write 9 New Done Done false
1127 Acked 2126 Read 7 New Done Done false
1128 Acked 2127 Flush 0 New Done Done false
1129 Acked 2128 Write 8 New Done Done false
1130 Acked 2129 Read 5 New Done Done false
1131 Acked 2130 Read 1 New Done Done false
1132 Acked 2131 Write 1 New Done Done false
1133 Acked 2132 Write 10 New Done Done false
1134 Acked 2133 Read 3 New Done Done false
1135 Acked 2134 Read 9 New Done Done false
1136 Acked 2135 Read 2 New Done Done false
1137 Acked 2136 Write 5 New Done Done false
1138 Acked 2137 Write 9 New Done Done false
1139 Acked 2138 Read 10 New Done Done false
1140 Acked 2139 Read 6 New Done Done false
1141 Acked 2140 Flush 0 New Done Done false
1142 Acked 2141 Read 1 New Done Done false
1143 Acked 2142 Read 4 New Done Done false
1144 Acked 2143 Write 10 New Done Done false
1145 Acked 2144 Read 3 New Done Done false
1146 Acked 2145 Read 8 New Done Done false
1147 Acked 2146 Write 9 New Done Done false
1148 Acked 2147 Read 4 New Done Done false
1149 Acked 2148 Read 2 New Done Done false
1150 Acked 2149 Write 4 New Done Done false
1151 Acked 2150 Write 4 New Done Done false
1152 Acked 2151 Write 4 New Done Done false
1153 Acked 2152 Flush 0 New Done Done false
1154 Acked 2153 Write 3 New Done Done false
1155 Acked 2154 Read 2 New Done Done false
1156 Acked 2155 Write 7 New Done Done false
1157 Acked 2156 Write 1 New Done Done false
1158 Acked 2157 Read 9 New Done Done false
1159 Acked 2158 Write 4 New Done Done false
1160 Acked 2159 Flush 0 New Done Done false
1161 Acked 2160 Read 2 New Done Done false
1162 Acked 2161 Write 4 New Done Done false
1163 Acked 2162 Write 8 New Done Done false
1164 Acked 2163 Flush 0 New Done Done false
1165 Acked 2164 Write 5 New Done Done false
1166 Acked 2165 Read 4 New Done Done false
1167 Acked 2166 Read 9 New Done Done false
1168 Acked 2167 Read 4 New Done Done false
1169 Acked 2168 Read 6 New Done Done false
1170 Acked 2169 Write 1 New Done Done false
1171 Acked 2170 Read 10 New Done Done false
1172 Acked 2171 Write 2 New Done Done false
1173 Acked 2172 Write 8 New Done Done false
1174 Acked 2173 Read 3 New Done Done false
1175 Acked 2174 Flush 0 New Done Done false
1176 Acked 2175 Read 7 New Done Done false
1177 Acked 2176 Write 8 New Done Done false
1178 Acked 2177 Read 1 New Done Done false
1179 Acked 2178 Read 9 New Done Done false
1180 Acked 2179 Read 7 New Done Done false
1181 Acked 2180 Read 3 New Done Done false
1182 Acked 2181 Read 10 New Done Done false
1183 Acked 2182 Read 8 New Done Done false
1184 Acked 2183 Read 10 New Done Done false
1185 Acked 2184 Read 5 New Done Done false
1186 Acked 2185 Read 10 New Done Done false
1187 Acked 2186 Write 5 New Done Done false
1188 Acked 2187 Read 1 New Done Done false
1189 Acked 2188 Write 9 New Done Done false
1190 Acked 2189 Flush 0 New Done Done false
1191 Acked 2190 Read 3 New Done Done false
1192 Acked 2191 Write 1 New Done Done false
1193 Acked 2192 Read 5 New Done Done false
1194 Acked 2193 Read 4 New Done Done false
1195 Acked 2194 Read 1 New Done Done false
1196 Acked 2195 Write 7 New Done Done false
1197 Acked 2196 Read 7 New Done Done false
1198 Acked 2197 Read 9 New Done Done false
1199 Acked 2198 Read 3 New Done Done false
1200 Acked 2199 Write 2 New Done Done false
1201 Acked 2200 Write 8 New Done Done false
1202 Acked 2201 Read 4 New Done Done false
1203 Acked 2202 Write 2 New Done Done false
1204 Acked 2203 Read 9 New Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
New 300 0 0 300
Sent 0 0 0 0
Done 0 300 300 600
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1903 2189 2189
Downstairs last five completed: 1903 1902 1901 1900 1899
Upstairs last five completed: 1204 1203 1202 1201 1200
CLIENT: Up:0 ds:300 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:13.887600777Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:13.887659435Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:13.887682714Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:13.887703373Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:13.887719442Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 300 jobs since flush 1903","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:13.887736492Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:13.887846188Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:14.889354059Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:14.889416436Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:14.889441055Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:14.889461715Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:14.889477514Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 1903","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:14.889495023Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:14.889638828Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:15.891166287Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:15.891225265Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:15.891246875Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:15.891267504Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:15.891283003Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 1903","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:15.891308782Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:15.891429868Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:16.893088613Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:16.893185769Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:16.893231757Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:16.893298175Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:16.893344863Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 1903","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:16.893380102Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:16.893575104Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
905 Acked 1904 Read 6 New Done Done true
906 Acked 1905 Read 8 New Done Done true
907 Acked 1906 Write 5 New Done Done true
908 Acked 1907 Read 8 New Done Done true
909 Acked 1908 Read 1 New Done Done true
910 Acked 1909 Read 7 New Done Done true
911 Acked 1910 Write 5 New Done Done true
912 Acked 1911 Write 1 New Done Done true
913 Acked 1912 Read 9 New Done Done true
914 Acked 1913 Write 2 New Done Done true
915 Acked 1914 Read 3 New Done Done true
916 Acked 1915 Write 7 New Done Done true
917 Acked 1916 Read 1 New Done Done true
918 Acked 1917 Flush 0 New Done Done true
919 Acked 1918 Read 1 New Done Done true
920 Acked 1919 Read 7 New Done Done true
921 Acked 1920 Write 9 New Done Done true
922 Acked 1921 Write 5 New Done Done true
923 Acked 1922 Write 2 New Done Done true
924 Acked 1923 Read 9 New Done Done true
925 Acked 1924 Flush 0 New Done Done true
926 Acked 1925 Read 4 New Done Done true
927 Acked 1926 Flush 0 New Done Done true
928 Acked 1927 Read 9 New Done Done true
929 Acked 1928 Write 7 New Done Done true
930 Acked 1929 Read 7 New Done Done true
931 Acked 1930 Write 1 New Done Done true
932 Acked 1931 Read 10 New Done Done true
933 Acked 1932 Read 8 New Done Done true
934 Acked 1933 Read 5 New Done Done true
935 Acked 1934 Read 6 New Done Done true
936 Acked 1935 Read 9 New Done Done true
937 Acked 1936 Write 8 New Done Done true
938 Acked 1937 Flush 0 New Done Done true
939 Acked 1938 Write 9 New Done Done true
940 Acked 1939 Write 10 New Done Done true
941 Acked 1940 Write 4 New Done Done true
942 Acked 1941 Read 8 New Done Done true
943 Acked 1942 Read 9 New Done Done true
944 Acked 1943 Read 7 New Done Done true
945 Acked 1944 Read 7 New Done Done true
946 Acked 1945 Read 10 New Done Done true
947 Acked 1946 Write 7 New Done Done true
948 Acked 1947 Read 5 New Done Done true
949 Acked 1948 Read 1 New Done Done true
950 Acked 1949 Write 3 New Done Done true
951 Acked 1950 Read 5 New Done Done true
952 Acked 1951 Write 10 New Done Done true
953 Acked 1952 Read 1 New Done Done true
954 Acked 1953 Write 4 New Done Done true
955 Acked 1954 Read 6 New Done Done true
956 Acked 1955 Read 8 New Done Done true
957 Acked 1956 Read 8 New Done Done true
958 Acked 1957 Write 5 New Done Done true
959 Acked 1958 Write 9 New Done Done true
960 Acked 1959 Write 3 New Done Done true
961 Acked 1960 Read 8 New Done Done true
962 Acked 1961 Write 4 New Done Done true
963 Acked 1962 Write 9 New Done Done true
964 Acked 1963 Write 9 New Done Done true
965 Acked 1964 Write 7 New Done Done true
966 Acked 1965 Read 3 New Done Done true
967 Acked 1966 Read 10 New Done Done true
968 Acked 1967 Read 6 New Done Done true
969 Acked 1968 Write 10 New Done Done true
970 Acked 1969 Write 1 New Done Done true
971 Acked 1970 Read 3 New Done Done true
972 Acked 1971 Write 1 New Done Done true
973 Acked 1972 Write 4 New Done Done true
974 Acked 1973 Read 9 New Done Done true
975 Acked 1974 Read 4 New Done Done true
976 Acked 1975 Read 5 New Done Done true
977 Acked 1976 Read 2 New Done Done true
978 Acked 1977 Flush 0 New Done Done true
979 Acked 1978 Read 3 New Done Done true
980 Acked 1979 Flush 0 New Done Done true
981 Acked 1980 Write 10 New Done Done true
982 Acked 1981 Read 8 New Done Done true
983 Acked 1982 Write 8 New Done Done true
984 Acked 1983 Write 1 New Done Done true
985 Acked 1984 Write 4 New Done Done true
986 Acked 1985 Write 9 New Done Done true
987 Acked 1986 Read 8 New Done Done true
988 Acked 1987 Read 6 New Done Done true
989 Acked 1988 Read 5 New Done Done true
990 Acked 1989 Read 10 New Done Done true
991 Acked 1990 Write 6 New Done Done true
992 Acked 1991 Flush 0 New Done Done true
993 Acked 1992 Read 9 New Done Done true
994 Acked 1993 Read 10 New Done Done true
995 Acked 1994 Read 10 New Done Done true
996 Acked 1995 Read 1 New Done Done true
997 Acked 1996 Read 5 New Done Done true
998 Acked 1997 Read 4 New Done Done true
999 Acked 1998 Read 3 New Done Done true
1000 Acked 1999 Read 2 New Done Done true
1001 Acked 2000 Write 5 New Done Done true
1002 Acked 2001 Write 5 New Done Done true
1003 Acked 2002 Write 1 New Done Done true
1004 Acked 2003 Flush 0 New Done Done true
1005 Acked 2004 Write 7 New Done Done true
1006 Acked 2005 Write 9 New Done Done true
1007 Acked 2006 Read 1 New Done Done true
1008 Acked 2007 Read 8 New Done Done true
1009 Acked 2008 Write 6 New Done Done true
1010 Acked 2009 Read 6 New Done Done true
1011 Acked 2010 Read 6 New Done Done true
1012 Acked 2011 Read 7 New Done Done true
1013 Acked 2012 Read 7 New Done Done true
1014 Acked 2013 Write 6 New Done Done true
1015 Acked 2014 Write 1 New Done Done true
1016 Acked 2015 Read 3 New Done Done true
1017 Acked 2016 Write 10 New Done Done true
1018 Acked 2017 Write 7 New Done Done true
1019 Acked 2018 Read 6 New Done Done true
1020 Acked 2019 Write 8 New Done Done true
1021 Acked 2020 Read 7 New Done Done true
1022 Acked 2021 Write 4 New Done Done true
1023 Acked 2022 Read 3 New Done Done true
1024 Acked 2023 Write 10 New Done Done true
1025 Acked 2024 Read 4 New Done Done true
1026 Acked 2025 Read 8 New Done Done true
1027 Acked 2026 Read 6 New Done Done true
1028 Acked 2027 Read 8 New Done Done true
1029 Acked 2028 Write 10 New Done Done true
1030 Acked 2029 Read 3 New Done Done true
1031 Acked 2030 Write 3 New Done Done true
1032 Acked 2031 Write 6 New Done Done true
1033 Acked 2032 Write 10 New Done Done true
1034 Acked 2033 Write 10 New Done Done true
1035 Acked 2034 Read 1 New Done Done true
1036 Acked 2035 Write 10 New Done Done true
1037 Acked 2036 Flush 0 New Done Done true
1038 Acked 2037 Read 4 New Done Done true
1039 Acked 2038 Flush 0 New Done Done true
1040 Acked 2039 Read 10 New Done Done true
1041 Acked 2040 Write 5 New Done Done true
1042 Acked 2041 Write 9 New Done Done true
1043 Acked 2042 Read 4 New Done Done true
1044 Acked 2043 Read 6 New Done Done true
1045 Acked 2044 Read 8 New Done Done true
1046 Acked 2045 Write 10 New Done Done true
1047 Acked 2046 Read 8 New Done Done true
1048 Acked 2047 Write 7 New Done Done true
1049 Acked 2048 Read 10 New Done Done true
1050 Acked 2049 Read 5 New Done Done true
1051 Acked 2050 Write 1 New Done Done true
1052 Acked 2051 Read 10 New Done Done true
1053 Acked 2052 Read 2 New Done Done true
1054 Acked 2053 Read 1 New Done Done true
1055 Acked 2054 Read 10 New Done Done true
1056 Acked 2055 Flush 0 New Done Done true
1057 Acked 2056 Read 2 New Done Done true
1058 Acked 2057 Read 6 New Done Done true
1059 Acked 2058 Read 9 New Done Done true
1060 Acked 2059 Write 9 New Done Done true
1061 Acked 2060 Write 3 New Done Done true
1062 Acked 2061 Write 2 New Done Done true
1063 Acked 2062 Read 1 New Done Done true
1064 Acked 2063 Read 7 New Done Done true
1065 Acked 2064 Read 10 New Done Done true
1066 Acked 2065 Read 4 New Done Done true
1067 Acked 2066 Read 9 New Done Done true
1068 Acked 2067 Flush 0 New Done Done true
1069 Acked 2068 Read 2 New Done Done true
1070 Acked 2069 Flush 0 New Done Done true
1071 Acked 2070 Read 5 New Done Done true
1072 Acked 2071 Write 2 New Done Done true
1073 Acked 2072 Read 10 New Done Done true
1074 Acked 2073 Read 10 New Done Done true
1075 Acked 2074 Read 2 New Done Done true
1076 Acked 2075 Write 10 New Done Done true
1077 Acked 2076 Write 2 New Done Done true
1078 Acked 2077 Flush 0 New Done Done true
1079 Acked 2078 Write 2 New Done Done true
1080 Acked 2079 Write 4 New Done Done true
1081 Acked 2080 Write 7 New Done Done true
1082 Acked 2081 Read 10 New Done Done true
1083 Acked 2082 Read 6 New Done Done true
1084 Acked 2083 Read 6 New Done Done true
1085 Acked 2084 Write 6 New Done Done true
1086 Acked 2085 Flush 0 New Done Done true
1087 Acked 2086 Write 9 New Done Done true
1088 Acked 2087 Write 1 New Done Done true
1089 Acked 2088 Read 6 New Done Done true
1090 Acked 2089 Read 2 New Done Done true
1091 Acked 2090 Read 10 New Done Done true
1092 Acked 2091 Write 2 New Done Done true
1093 Acked 2092 Write 6 New Done Done true
1094 Acked 2093 Write 5 New Done Done true
1095 Acked 2094 Read 5 New Done Done true
1096 Acked 2095 Read 6 New Done Done true
1097 Acked 2096 Flush 0 New Done Done true
1098 Acked 2097 Flush 0 New Done Done true
1099 Acked 2098 Read 10 New Done Done true
1100 Acked 2099 Read 9 New Done Done true
1101 Acked 2100 Read 7 New Done Done true
1102 Acked 2101 Read 4 New Done Done true
1103 Acked 2102 Flush 0 New Done Done true
1104 Acked 2103 Write 6 New Done Done true
1105 Acked 2104 Read 1 New Done Done true
1106 Acked 2105 Write 1 New Done Done true
1107 Acked 2106 Flush 0 New Done Done true
1108 Acked 2107 Write 9 New Done Done true
1109 Acked 2108 Write 10 New Done Done true
1110 Acked 2109 Read 1 New Done Done true
1111 Acked 2110 Read 9 New Done Done true
1112 Acked 2111 Read 2 New Done Done true
1113 Acked 2112 Write 4 New Done Done true
1114 Acked 2113 Write 9 New Done Done true
1115 Acked 2114 Read 9 New Done Done true
1116 Acked 2115 Write 9 New Done Done true
1117 Acked 2116 Flush 0 New Done Done true
1118 Acked 2117 Write 8 New Done Done true
1119 Acked 2118 Read 1 New Done Done true
1120 Acked 2119 Read 2 New Done Done true
1121 Acked 2120 Read 10 New Done Done true
1122 Acked 2121 Read 6 New Done Done true
1123 Acked 2122 Read 2 New Done Done true
1124 Acked 2123 Flush 0 New Done Done true
1125 Acked 2124 Read 7 New Done Done true
1126 Acked 2125 Write 9 New Done Done true
1127 Acked 2126 Read 7 New Done Done true
1128 Acked 2127 Flush 0 New Done Done true
1129 Acked 2128 Write 8 New Done Done true
1130 Acked 2129 Read 5 New Done Done true
1131 Acked 2130 Read 1 New Done Done true
1132 Acked 2131 Write 1 New Done Done true
1133 Acked 2132 Write 10 New Done Done true
1134 Acked 2133 Read 3 New Done Done true
1135 Acked 2134 Read 9 New Done Done true
1136 Acked 2135 Read 2 New Done Done true
1137 Acked 2136 Write 5 New Done Done true
1138 Acked 2137 Write 9 New Done Done true
1139 Acked 2138 Read 10 New Done Done true
1140 Acked 2139 Read 6 New Done Done true
1141 Acked 2140 Flush 0 New Done Done true
1142 Acked 2141 Read 1 New Done Done true
1143 Acked 2142 Read 4 New Done Done true
1144 Acked 2143 Write 10 New Done Done true
1145 Acked 2144 Read 3 New Done Done true
1146 Acked 2145 Read 8 New Done Done true
1147 Acked 2146 Write 9 New Done Done true
1148 Acked 2147 Read 4 New Done Done true
1149 Acked 2148 Read 2 New Done Done true
1150 Acked 2149 Write 4 New Done Done true
1151 Acked 2150 Write 4 New Done Done true
1152 Acked 2151 Write 4 New Done Done true
1153 Acked 2152 Flush 0 New Done Done true
1154 Acked 2153 Write 3 New Done Done true
1155 Acked 2154 Read 2 New Done Done true
1156 Acked 2155 Write 7 New Done Done true
1157 Acked 2156 Write 1 New Done Done true
1158 Acked 2157 Read 9 New Done Done true
1159 Acked 2158 Write 4 New Done Done true
1160 Acked 2159 Flush 0 New Done Done true
1161 Acked 2160 Read 2 New Done Done true
1162 Acked 2161 Write 4 New Done Done true
1163 Acked 2162 Write 8 New Done Done true
1164 Acked 2163 Flush 0 New Done Done true
1165 Acked 2164 Write 5 New Done Done true
1166 Acked 2165 Read 4 New Done Done true
1167 Acked 2166 Read 9 New Done Done true
1168 Acked 2167 Read 4 New Done Done true
1169 Acked 2168 Read 6 New Done Done true
1170 Acked 2169 Write 1 New Done Done true
1171 Acked 2170 Read 10 New Done Done true
1172 Acked 2171 Write 2 New Done Done true
1173 Acked 2172 Write 8 New Done Done true
1174 Acked 2173 Read 3 New Done Done true
1175 Acked 2174 Flush 0 New Done Done true
1176 Acked 2175 Read 7 New Done Done true
1177 Acked 2176 Write 8 New Done Done true
1178 Acked 2177 Read 1 New Done Done true
1179 Acked 2178 Read 9 New Done Done true
1180 Acked 2179 Read 7 New Done Done true
1181 Acked 2180 Read 3 New Done Done true
1182 Acked 2181 Read 10 New Done Done true
1183 Acked 2182 Read 8 New Done Done true
1184 Acked 2183 Read 10 New Done Done true
1185 Acked 2184 Read 5 New Done Done true
1186 Acked 2185 Read 10 New Done Done true
1187 Acked 2186 Write 5 New Done Done true
1188 Acked 2187 Read 1 New Done Done true
1189 Acked 2188 Write 9 New Done Done true
1190 Acked 2189 Flush 0 New Done Done true
1191 Acked 2190 Read 3 New Done Done true
1192 Acked 2191 Write 1 New Done Done true
1193 Acked 2192 Read 5 New Done Done true
1194 Acked 2193 Read 4 New Done Done true
1195 Acked 2194 Read 1 New Done Done true
1196 Acked 2195 Write 7 New Done Done true
1197 Acked 2196 Read 7 New Done Done true
1198 Acked 2197 Read 9 New Done Done true
1199 Acked 2198 Read 3 New Done Done true
1200 Acked 2199 Write 2 New Done Done true
1201 Acked 2200 Write 8 New Done Done true
1202 Acked 2201 Read 4 New Done Done true
1203 Acked 2202 Write 2 New Done Done true
1204 Acked 2203 Read 9 New Done Done true
1205 Acked 2204 Flush 0 New Done Done true
STATES DS:0 DS:1 DS:2 TOTAL
New 301 0 0 301
Sent 0 0 0 0
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1903 2204 2204
Downstairs last five completed: 1903 1902 1901 1900 1899
Upstairs last five completed: 1205 1204 1203 1202 1201
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:17.894808375Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:17.894865893Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:17.894887902Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:17.894908161Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:17.894923891Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 1903","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:17.89494153Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:17.895063195Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:18.895783896Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:18.895842284Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:18.895873303Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:18.895895242Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:18.895911081Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 1903","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:18.895928401Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:18.896032617Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:19.896958149Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:19.897020307Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:19.897043156Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:19.897063785Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:19.897079625Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 1903","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:19.897096974Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:19.897238959Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:20.89768707Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:20.897745368Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:20.897767447Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:20.897788086Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:20.897803916Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 1903","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:20.897821385Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:20.89794552Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
905 Acked 1904 Read 6 New Done Done true
906 Acked 1905 Read 8 New Done Done true
907 Acked 1906 Write 5 New Done Done true
908 Acked 1907 Read 8 New Done Done true
909 Acked 1908 Read 1 New Done Done true
910 Acked 1909 Read 7 New Done Done true
911 Acked 1910 Write 5 New Done Done true
912 Acked 1911 Write 1 New Done Done true
913 Acked 1912 Read 9 New Done Done true
914 Acked 1913 Write 2 New Done Done true
915 Acked 1914 Read 3 New Done Done true
916 Acked 1915 Write 7 New Done Done true
917 Acked 1916 Read 1 New Done Done true
918 Acked 1917 Flush 0 New Done Done true
919 Acked 1918 Read 1 New Done Done true
920 Acked 1919 Read 7 New Done Done true
921 Acked 1920 Write 9 New Done Done true
922 Acked 1921 Write 5 New Done Done true
923 Acked 1922 Write 2 New Done Done true
924 Acked 1923 Read 9 New Done Done true
925 Acked 1924 Flush 0 New Done Done true
926 Acked 1925 Read 4 New Done Done true
927 Acked 1926 Flush 0 New Done Done true
928 Acked 1927 Read 9 New Done Done true
929 Acked 1928 Write 7 New Done Done true
930 Acked 1929 Read 7 New Done Done true
931 Acked 1930 Write 1 New Done Done true
932 Acked 1931 Read 10 New Done Done true
933 Acked 1932 Read 8 New Done Done true
934 Acked 1933 Read 5 New Done Done true
935 Acked 1934 Read 6 New Done Done true
936 Acked 1935 Read 9 New Done Done true
937 Acked 1936 Write 8 New Done Done true
938 Acked 1937 Flush 0 New Done Done true
939 Acked 1938 Write 9 New Done Done true
940 Acked 1939 Write 10 New Done Done true
941 Acked 1940 Write 4 New Done Done true
942 Acked 1941 Read 8 New Done Done true
943 Acked 1942 Read 9 New Done Done true
944 Acked 1943 Read 7 New Done Done true
945 Acked 1944 Read 7 New Done Done true
946 Acked 1945 Read 10 New Done Done true
947 Acked 1946 Write 7 New Done Done true
948 Acked 1947 Read 5 New Done Done true
949 Acked 1948 Read 1 New Done Done true
950 Acked 1949 Write 3 New Done Done true
951 Acked 1950 Read 5 New Done Done true
952 Acked 1951 Write 10 New Done Done true
953 Acked 1952 Read 1 New Done Done true
954 Acked 1953 Write 4 New Done Done true
955 Acked 1954 Read 6 New Done Done true
956 Acked 1955 Read 8 New Done Done true
957 Acked 1956 Read 8 New Done Done true
958 Acked 1957 Write 5 New Done Done true
959 Acked 1958 Write 9 New Done Done true
960 Acked 1959 Write 3 New Done Done true
961 Acked 1960 Read 8 New Done Done true
962 Acked 1961 Write 4 New Done Done true
963 Acked 1962 Write 9 New Done Done true
964 Acked 1963 Write 9 New Done Done true
965 Acked 1964 Write 7 New Done Done true
966 Acked 1965 Read 3 New Done Done true
967 Acked 1966 Read 10 New Done Done true
968 Acked 1967 Read 6 New Done Done true
969 Acked 1968 Write 10 New Done Done true
970 Acked 1969 Write 1 New Done Done true
971 Acked 1970 Read 3 New Done Done true
972 Acked 1971 Write 1 New Done Done true
973 Acked 1972 Write 4 New Done Done true
974 Acked 1973 Read 9 New Done Done true
975 Acked 1974 Read 4 New Done Done true
976 Acked 1975 Read 5 New Done Done true
977 Acked 1976 Read 2 New Done Done true
978 Acked 1977 Flush 0 New Done Done true
979 Acked 1978 Read 3 New Done Done true
980 Acked 1979 Flush 0 New Done Done true
981 Acked 1980 Write 10 New Done Done true
982 Acked 1981 Read 8 New Done Done true
983 Acked 1982 Write 8 New Done Done true
984 Acked 1983 Write 1 New Done Done true
985 Acked 1984 Write 4 New Done Done true
986 Acked 1985 Write 9 New Done Done true
987 Acked 1986 Read 8 New Done Done true
988 Acked 1987 Read 6 New Done Done true
989 Acked 1988 Read 5 New Done Done true
990 Acked 1989 Read 10 New Done Done true
991 Acked 1990 Write 6 New Done Done true
992 Acked 1991 Flush 0 New Done Done true
993 Acked 1992 Read 9 New Done Done true
994 Acked 1993 Read 10 New Done Done true
995 Acked 1994 Read 10 New Done Done true
996 Acked 1995 Read 1 New Done Done true
997 Acked 1996 Read 5 New Done Done true
998 Acked 1997 Read 4 New Done Done true
999 Acked 1998 Read 3 New Done Done true
1000 Acked 1999 Read 2 New Done Done true
1001 Acked 2000 Write 5 New Done Done true
1002 Acked 2001 Write 5 New Done Done true
1003 Acked 2002 Write 1 New Done Done true
1004 Acked 2003 Flush 0 New Done Done true
1005 Acked 2004 Write 7 New Done Done true
1006 Acked 2005 Write 9 New Done Done true
1007 Acked 2006 Read 1 New Done Done true
1008 Acked 2007 Read 8 New Done Done true
1009 Acked 2008 Write 6 New Done Done true
1010 Acked 2009 Read 6 New Done Done true
1011 Acked 2010 Read 6 New Done Done true
1012 Acked 2011 Read 7 New Done Done true
1013 Acked 2012 Read 7 New Done Done true
1014 Acked 2013 Write 6 New Done Done true
1015 Acked 2014 Write 1 New Done Done true
1016 Acked 2015 Read 3 New Done Done true
1017 Acked 2016 Write 10 New Done Done true
1018 Acked 2017 Write 7 New Done Done true
1019 Acked 2018 Read 6 New Done Done true
1020 Acked 2019 Write 8 New Done Done true
1021 Acked 2020 Read 7 New Done Done true
1022 Acked 2021 Write 4 New Done Done true
1023 Acked 2022 Read 3 New Done Done true
1024 Acked 2023 Write 10 New Done Done true
1025 Acked 2024 Read 4 New Done Done true
1026 Acked 2025 Read 8 New Done Done true
1027 Acked 2026 Read 6 New Done Done true
1028 Acked 2027 Read 8 New Done Done true
1029 Acked 2028 Write 10 New Done Done true
1030 Acked 2029 Read 3 New Done Done true
1031 Acked 2030 Write 3 New Done Done true
1032 Acked 2031 Write 6 New Done Done true
1033 Acked 2032 Write 10 New Done Done true
1034 Acked 2033 Write 10 New Done Done true
1035 Acked 2034 Read 1 New Done Done true
1036 Acked 2035 Write 10 New Done Done true
1037 Acked 2036 Flush 0 New Done Done true
1038 Acked 2037 Read 4 New Done Done true
1039 Acked 2038 Flush 0 New Done Done true
1040 Acked 2039 Read 10 New Done Done true
1041 Acked 2040 Write 5 New Done Done true
1042 Acked 2041 Write 9 New Done Done true
1043 Acked 2042 Read 4 New Done Done true
1044 Acked 2043 Read 6 New Done Done true
1045 Acked 2044 Read 8 New Done Done true
1046 Acked 2045 Write 10 New Done Done true
1047 Acked 2046 Read 8 New Done Done true
1048 Acked 2047 Write 7 New Done Done true
1049 Acked 2048 Read 10 New Done Done true
1050 Acked 2049 Read 5 New Done Done true
1051 Acked 2050 Write 1 New Done Done true
1052 Acked 2051 Read 10 New Done Done true
1053 Acked 2052 Read 2 New Done Done true
1054 Acked 2053 Read 1 New Done Done true
1055 Acked 2054 Read 10 New Done Done true
1056 Acked 2055 Flush 0 New Done Done true
1057 Acked 2056 Read 2 New Done Done true
1058 Acked 2057 Read 6 New Done Done true
1059 Acked 2058 Read 9 New Done Done true
1060 Acked 2059 Write 9 New Done Done true
1061 Acked 2060 Write 3 New Done Done true
1062 Acked 2061 Write 2 New Done Done true
1063 Acked 2062 Read 1 New Done Done true
1064 Acked 2063 Read 7 New Done Done true
1065 Acked 2064 Read 10 New Done Done true
1066 Acked 2065 Read 4 New Done Done true
1067 Acked 2066 Read 9 New Done Done true
1068 Acked 2067 Flush 0 New Done Done true
1069 Acked 2068 Read 2 New Done Done true
1070 Acked 2069 Flush 0 New Done Done true
1071 Acked 2070 Read 5 New Done Done true
1072 Acked 2071 Write 2 New Done Done true
1073 Acked 2072 Read 10 New Done Done true
1074 Acked 2073 Read 10 New Done Done true
1075 Acked 2074 Read 2 New Done Done true
1076 Acked 2075 Write 10 New Done Done true
1077 Acked 2076 Write 2 New Done Done true
1078 Acked 2077 Flush 0 New Done Done true
1079 Acked 2078 Write 2 New Done Done true
1080 Acked 2079 Write 4 New Done Done true
1081 Acked 2080 Write 7 New Done Done true
1082 Acked 2081 Read 10 New Done Done true
1083 Acked 2082 Read 6 New Done Done true
1084 Acked 2083 Read 6 New Done Done true
1085 Acked 2084 Write 6 New Done Done true
1086 Acked 2085 Flush 0 New Done Done true
1087 Acked 2086 Write 9 New Done Done true
1088 Acked 2087 Write 1 New Done Done true
1089 Acked 2088 Read 6 New Done Done true
1090 Acked 2089 Read 2 New Done Done true
1091 Acked 2090 Read 10 New Done Done true
1092 Acked 2091 Write 2 New Done Done true
1093 Acked 2092 Write 6 New Done Done true
1094 Acked 2093 Write 5 New Done Done true
1095 Acked 2094 Read 5 New Done Done true
1096 Acked 2095 Read 6 New Done Done true
1097 Acked 2096 Flush 0 New Done Done true
1098 Acked 2097 Flush 0 New Done Done true
1099 Acked 2098 Read 10 New Done Done true
1100 Acked 2099 Read 9 New Done Done true
1101 Acked 2100 Read 7 New Done Done true
1102 Acked 2101 Read 4 New Done Done true
1103 Acked 2102 Flush 0 New Done Done true
1104 Acked 2103 Write 6 New Done Done true
1105 Acked 2104 Read 1 New Done Done true
1106 Acked 2105 Write 1 New Done Done true
1107 Acked 2106 Flush 0 New Done Done true
1108 Acked 2107 Write 9 New Done Done true
1109 Acked 2108 Write 10 New Done Done true
1110 Acked 2109 Read 1 New Done Done true
1111 Acked 2110 Read 9 New Done Done true
1112 Acked 2111 Read 2 New Done Done true
1113 Acked 2112 Write 4 New Done Done true
1114 Acked 2113 Write 9 New Done Done true
1115 Acked 2114 Read 9 New Done Done true
1116 Acked 2115 Write 9 New Done Done true
1117 Acked 2116 Flush 0 New Done Done true
1118 Acked 2117 Write 8 New Done Done true
1119 Acked 2118 Read 1 New Done Done true
1120 Acked 2119 Read 2 New Done Done true
1121 Acked 2120 Read 10 New Done Done true
1122 Acked 2121 Read 6 New Done Done true
1123 Acked 2122 Read 2 New Done Done true
1124 Acked 2123 Flush 0 New Done Done true
1125 Acked 2124 Read 7 New Done Done true
1126 Acked 2125 Write 9 New Done Done true
1127 Acked 2126 Read 7 New Done Done true
1128 Acked 2127 Flush 0 New Done Done true
1129 Acked 2128 Write 8 New Done Done true
1130 Acked 2129 Read 5 New Done Done true
1131 Acked 2130 Read 1 New Done Done true
1132 Acked 2131 Write 1 New Done Done true
1133 Acked 2132 Write 10 New Done Done true
1134 Acked 2133 Read 3 New Done Done true
1135 Acked 2134 Read 9 New Done Done true
1136 Acked 2135 Read 2 New Done Done true
1137 Acked 2136 Write 5 New Done Done true
1138 Acked 2137 Write 9 New Done Done true
1139 Acked 2138 Read 10 New Done Done true
1140 Acked 2139 Read 6 New Done Done true
1141 Acked 2140 Flush 0 New Done Done true
1142 Acked 2141 Read 1 New Done Done true
1143 Acked 2142 Read 4 New Done Done true
1144 Acked 2143 Write 10 New Done Done true
1145 Acked 2144 Read 3 New Done Done true
1146 Acked 2145 Read 8 New Done Done true
1147 Acked 2146 Write 9 New Done Done true
1148 Acked 2147 Read 4 New Done Done true
1149 Acked 2148 Read 2 New Done Done true
1150 Acked 2149 Write 4 New Done Done true
1151 Acked 2150 Write 4 New Done Done true
1152 Acked 2151 Write 4 New Done Done true
1153 Acked 2152 Flush 0 New Done Done true
1154 Acked 2153 Write 3 New Done Done true
1155 Acked 2154 Read 2 New Done Done true
1156 Acked 2155 Write 7 New Done Done true
1157 Acked 2156 Write 1 New Done Done true
1158 Acked 2157 Read 9 New Done Done true
1159 Acked 2158 Write 4 New Done Done true
1160 Acked 2159 Flush 0 New Done Done true
1161 Acked 2160 Read 2 New Done Done true
1162 Acked 2161 Write 4 New Done Done true
1163 Acked 2162 Write 8 New Done Done true
1164 Acked 2163 Flush 0 New Done Done true
1165 Acked 2164 Write 5 New Done Done true
1166 Acked 2165 Read 4 New Done Done true
1167 Acked 2166 Read 9 New Done Done true
1168 Acked 2167 Read 4 New Done Done true
1169 Acked 2168 Read 6 New Done Done true
1170 Acked 2169 Write 1 New Done Done true
1171 Acked 2170 Read 10 New Done Done true
1172 Acked 2171 Write 2 New Done Done true
1173 Acked 2172 Write 8 New Done Done true
1174 Acked 2173 Read 3 New Done Done true
1175 Acked 2174 Flush 0 New Done Done true
1176 Acked 2175 Read 7 New Done Done true
1177 Acked 2176 Write 8 New Done Done true
1178 Acked 2177 Read 1 New Done Done true
1179 Acked 2178 Read 9 New Done Done true
1180 Acked 2179 Read 7 New Done Done true
1181 Acked 2180 Read 3 New Done Done true
1182 Acked 2181 Read 10 New Done Done true
1183 Acked 2182 Read 8 New Done Done true
1184 Acked 2183 Read 10 New Done Done true
1185 Acked 2184 Read 5 New Done Done true
1186 Acked 2185 Read 10 New Done Done true
1187 Acked 2186 Write 5 New Done Done true
1188 Acked 2187 Read 1 New Done Done true
1189 Acked 2188 Write 9 New Done Done true
1190 Acked 2189 Flush 0 New Done Done true
1191 Acked 2190 Read 3 New Done Done true
1192 Acked 2191 Write 1 New Done Done true
1193 Acked 2192 Read 5 New Done Done true
1194 Acked 2193 Read 4 New Done Done true
1195 Acked 2194 Read 1 New Done Done true
1196 Acked 2195 Write 7 New Done Done true
1197 Acked 2196 Read 7 New Done Done true
1198 Acked 2197 Read 9 New Done Done true
1199 Acked 2198 Read 3 New Done Done true
1200 Acked 2199 Write 2 New Done Done true
1201 Acked 2200 Write 8 New Done Done true
1202 Acked 2201 Read 4 New Done Done true
1203 Acked 2202 Write 2 New Done Done true
1204 Acked 2203 Read 9 New Done Done true
1205 Acked 2204 Flush 0 New Done Done true
STATES DS:0 DS:1 DS:2 TOTAL
New 301 0 0 301
Sent 0 0 0 0
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1903 2204 2204
Downstairs last five completed: 1903 1902 1901 1900 1899
Upstairs last five completed: 1205 1204 1203 1202 1201
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:21.898627253Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:21.898674311Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:21.898695001Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:21.89871615Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:21.898731649Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 1903","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:21.898756358Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:21.898894223Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1168)
[8810][0] reports Running
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:22.900060407Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:22.900118234Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:22.900139243Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:22.900164523Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:22.900180592Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 1903","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:22.900197861Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:22.900331436Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:23.901021858Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:23.901135233Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:23.902466443Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:23.902496341Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 1903","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:23.902514901Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 1903","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:23.902676255Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:23.902712163Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:23.902739252Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:23.902762261Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:23.90278351Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 2204 2203 2202 2201 2200
Upstairs last five completed: 1205 1204 1203 1202 1201
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(1)
stop 1
Feb 13 21:40:25.728 INFO request completed, latency_us:[8820] Got stop action so:false kr:false
357, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 796d2b66-9ef8-4883-a176-20f9f44c1866, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:25.729170132Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:25.72922109Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:25.72923848Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:25.729252449Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 0 jobs since flush 2204","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:25.729287628Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:25.729326116Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:40:25.729 INFO request completed, latency_us: 223, response_code: 200, uri: /state/cid/1, method: GET, req_id: 176b1e7a-7f25-4cd4-a185-0e6224bedb4b, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:26.731309839Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:26.731364246Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:26.731382566Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:26.731399205Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:26.731411815Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 0 jobs since flush 2204","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:26.731428004Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:26.731452043Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:27.732591898Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:27.732652516Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:27.732672085Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:27.732688964Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:27.732702163Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 0 jobs since flush 2204","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:27.732751622Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:27.732775741Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:28.73432711Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:28.734380478Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:28.734399068Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:28.734415367Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:28.734428197Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 0 jobs since flush 2204","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:28.734450106Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:28.734464665Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:40:29.731 INFO request completed, latency_us: 203, response_code: 200, uri: /state/cid/1, method: GET, req_id: 892b72f2-6413-4d59-83a1-e59671ea25ab, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Write block 331 len 8192 data: 2 2
002/300 Read block 508 len 32768
003/300 Write block 104 len 32768 data: 2 3 3 3 2 2 2 2
004/300 Read block 2756 len 4096
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:29.735585981Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:29.735649819Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:29.735668688Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:29.735685498Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:29.735698477Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 4 jobs since flush 2204","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:29.735712896Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:29.735727446Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
005/300 Read block 2009 len 40960
006/300 Write block 3670 len 36864 data: 3 3 3 3 3 4 3 2 2
007/300 Read block 3085 len 16384
008/300 Flush
009/300 Read block 2915 len 20480
010/300 Read block 563 len 36864
011/300 Read block 726 len 36864
012/300 Read block 1907 len 36864
013/300 Write block 45 len 24576 data: 3 3 3 4 4 3
014/300 Read block 1417 len 8192
015/300 Read block 1698 len 4096
016/300 Write block 3804 len 28672 data: 2 2 2 2 2 2 2
017/300 Write block 400 len 40960 data: 3 3 3 3 2 2 2 2 2 2
018/300 Read block 2905 len 16384
019/300 Write block 2017 len 16384 data: 2 2 2 3
020/300 Read block 2594 len 36864
021/300 Write block 3351 len 32768 data: 3 3 3 3 2 2 2 2
022/300 Write block 4331 len 40960 data: 2 2 2 2 2 2 2 2 2 2
023/300 Write block 2340 len 24576 data: 2 2 2 2 2 2
024/300 Read block 4613 len 32768
025/300 Read block 3543 len 4096
026/300 Read block 1528 len 4096
027/300 Read block 813 len 28672
028/300 Read block 2089 len 16384
029/300 Flush
030/300 Write block 1191 len 16384 data: 2 2 2 2
031/300 Read block 787 len 36864
032/300 Write block 330 len 28672 data: 2 3 3 2 3 3 3
033/300 Write block 4180 len 32768 data: 2 2 2 2 2 2 2 2
034/300 Read block 2828 len 28672
035/300 Write block 821 len 36864 data: 2 2 2 2 2 2 2 2 2
036/300 Flush
037/300 Read block 1571 len 16384
038/300 Read block 3927 len 8192
039/300 Write block 1495 len 40960 data: 2 2 2 2 2 2 2 2 2 2
040/300 Read block 3386 len 4096
041/300 Read block 3771 len 36864
042/300 Read block 478 len 20480
043/300 Write block 4111 len 36864 data: 2 2 2 3 3 3 2 2 2
044/300 Read block 2360 len 16384
045/300 Read block 4211 len 20480
046/300 Read block 16 len 28672
047/300 Read block 4763 len 20480
048/300 Flush
049/300 Read block 2860 len 32768
050/300 Read block 11 len 8192
051/300 Write block 2057 len 36864 data: 2 2 2 2 2 2 2 2 2
052/300 Write block 3692 len 4096 data: 2
053/300 Read block 29 len 24576
054/300 Flush
055/300 Read block 1778 len 32768
056/300 Read block 3580 len 4096
057/300 Write block 997 len 36864 data: 3 3 3 3 2 2 2 2 2
058/300 Read block 3578 len 16384
059/300 Read block 1166 len 4096
060/300 Read block 1188 len 40960
061/300 Read block 3111 len 4096
062/300 Read block 952 len 32768
063/300 Write block 3125 len 8192 data: 3 3
064/300 Read block 4461 len 8192
065/300 Read block 3505 len 36864
066/300 Read block 3912 len 28672
067/300 Write block 1334 len 16384 data: 2 2 2 2
068/300 Read block 1387 len 28672
069/300 Read block 1250 len 4096
070/300 Flush
071/300 Read block 345 len 40960
072/300 Read block 1845 len 28672
073/300 Write block 2717 len 40960 data: 2 2 2 2 2 2 2 2 2 2
074/300 Write block 4232 len 4096 data: 2
075/300 Read block 457 len 4096
076/300 Read block 3124 len 20480
077/300 Write block 1328 len 28672 data: 2 2 2 2 3 3 3
078/300 Read block 1990 len 8192
079/300 Write block 2938 len 32768 data: 2 2 2 2 2 2 2 2
080/300 Read block 3386 len 32768
081/300 Read block 3670 len 32768
082/300 Flush
083/300 Read block 156 len 32768
084/300 Read block 1790 len 40960
085/300 Read block 3272 len 24576
086/300 Read block 1369 len 32768
087/300 Read block 2293 len 24576
088/300 Read block 3935 len 4096
089/300 Flush
090/300 Read block 3 len 16384
091/300 Flush
092/300 Write block 1233 len 12288 data: 2 2 2
093/300 Read block 4274 len 20480
094/300 Write block 4051 len 4096 data: 2
095/300 Write block 986 len 36864 data: 3 3 2 2 2 3 4 3 3
096/300 Read block 1922 len 40960
097/300 Write block 4014 len 8192 data: 2 2
098/300 Write block 739 len 20480 data: 3 3 3 2 2
099/300 Flush
100/300 Read block 3406 len 40960
101/300 Write block 248 len 16384 data: 3 2 2 2
102/300 Write block 2186 len 8192 data: 2 2
103/300 Write block 3402 len 20480 data: 3 3 3 3 3
104/300 Write block 4321 len 24576 data: 2 3 4 4 4 4
105/300 Write block 2507 len 36864 data: 2 3 3 3 3 3 3 3 3
106/300 Read block 1310 len 32768
107/300 Write block 1475 len 8192 data: 2 2
108/300 Flush
109/300 Write block 4764 len 4096 data: 3
110/300 Write block 231 len 28672 data: 2 2 2 2 2 2 2
111/300 Write block 2532 len 16384 data: 2 2 2 2
112/300 Read block 43 len 36864
113/300 Read block 1070 len 16384
114/300 Read block 167 len 40960
115/300 Read block 2549 len 24576
116/300 Flush
117/300 Read block 3670 len 8192
118/300 Write block 4503 len 20480 data: 3 3 3 3 3
119/300 Read block 958 len 20480
120/300 Write block 519 len 12288 data: 3 3 3
121/300 Read block 3701 len 40960
122/300 Read block 1381 len 24576
123/300 Read block 2964 len 36864
124/300 Read block 1469 len 40960
125/300 Read block 1335 len 12288
126/300 Read block 1956 len 4096
127/300 Write block 4831 len 20480 data: 2 2 3 3 3
128/300 Read block 4431 len 40960
129/300 Read block 4026 len 36864
130/300 Write block 2090 len 40960 data: 3 3 3 3 2 2 3 3 3 3
131/300 Write block 697 len 24576 data: 3 3 3 3 3 2
132/300 Write block 1519 len 16384 data: 2 2 3 2
133/300 Read block 3812 len 20480
134/300 Read block 962 len 24576
135/300 Write block 73 len 8192 data: 3 3
136/300 Read block 370 len 36864
137/300 Read block 675 len 28672
138/300 Write block 90 len 36864 data: 2 2 2 2 2 2 2 2 2
139/300 Write block 3923 len 36864 data: 3 3 3 3 3 4 2 2 2
140/300 Flush
141/300 Read block 257 len 40960
142/300 Read block 862 len 24576
143/300 Read block 1642 len 28672
144/300 Write block 4022 len 36864 data: 2 2 2 2 2 2 2 2 2
145/300 Write block 4228 len 16384 data: 2 2 2 2
146/300 Read block 4697 len 36864
147/300 Flush
148/300 Flush
149/300 Read block 2060 len 36864
150/300 Read block 2201 len 4096
151/300 Read block 625 len 36864
152/300 Read block 883 len 28672
153/300 Write block 3482 len 12288 data: 2 2 2
154/300 Write block 2525 len 24576 data: 2 2 2 2 2 2
155/300 Write block 2959 len 16384 data: 2 2 2 2
156/300 Read block 4259 len 28672
157/300 Read block 2603 len 20480
158/300 Write block 3400 len 4096 data: 2
159/300 Write block 4705 len 24576 data: 2 2 2 2 2 2
160/300 Write block 2573 len 8192 data: 3 3
161/300 Read block 4395 len 32768
162/300 Write block 855 len 12288 data: 3 4 3
163/300 Read block 862 len 32768
164/300 Write block 531 len 4096 data: 3
165/300 Write block 4826 len 16384 data: 2 2 2 2
166/300 Read block 3014 len 40960
167/300 Flush
168/300 Read block 3043 len 4096
169/300 Flush
170/300 Write block 289 len 4096 data: 2
171/300 Write block 3334 len 24576 data: 2 2 3 3 3 4
172/300 Write block 617 len 20480 data: 2 2 2 2 3
173/300 Write block 776 len 8192 data: 3 2
174/300 Write block 4531 len 24576 data: 3 3 3 3 3 4
175/300 Read block 1221 len 32768
176/300 Write block 4487 len 20480 data: 4 5 5 4 3
177/300 Flush
178/300 Read block 855 len 40960
179/300 Write block 331 len 8192 data: 4 4
180/300 Flush
181/300 Read block 1181 len 32768
182/300 Read block 2291 len 40960
183/300 Flush
184/300 Read block 226 len 12288
185/300 Read block 427 len 16384
186/300 Flush
187/300 Write block 1279 len 16384 data: 2 2 3 3
188/300 Read block 1345 len 40960
189/300 Write block 3878 len 20480 data: 2 2 2 2 2
190/300 Write block 3788 len 12288 data: 3 3 3
191/300 Read block 2090 len 36864
192/300 Read block 1028 len 12288
193/300 Write block 1200 len 24576 data: 2 2 2 3 3 3
194/300 Read block 1205 len 24576
195/300 Read block 2797 len 28672
196/300 Read block 2292 len 24576
197/300 Flush
198/300 Read block 1645 len 20480
199/300 Read block 3730 len 40960
200/300 Write block 1819 len 32768 data: 2 2 2 2 2 2 2 2
201/300 Write block 3209 len 8192 data: 2 2
202/300 Read block 1931 len 4096
203/300 Write block 2830 len 36864 data: 4 2 3 3 3 3 3 3 3
204/300 Read block 4520 len 24576
205/300 Read block 811 len 36864
206/300 Read block 3940 len 12288
207/300 Read block 3125 len 36864
208/300 Read block 1850 len 32768
209/300 Read block 1301 len 32768
210/300 Read block 446 len 16384
211/300 Read block 3185 len 4096
212/300 Write block 3884 len 12288 data: 4 3 3
213/300 Write block 3948 len 40960 data: 2 3 3 3 3 3 3 3 2 2
214/300 Write block 334 len 36864 data: 4 4 4 2 2 2 2 2 2
215/300 Read block 3808 len 28672
216/300 Write block 590 len 40960 data: 3 3 3 3 3 3 3 3 3 3
217/300 Write block 1470 len 24576 data: 2 2 2 2 2 3
218/300 Read block 1437 len 28672
219/300 Write block 3411 len 4096 data: 4
220/300 Read block 4610 len 32768
221/300 Write block 1248 len 16384 data: 3 3 3 3
222/300 Read block 242 len 24576
223/300 Read block 536 len 4096
224/300 Read block 1679 len 36864
225/300 Write block 170 len 32768 data: 3 3 2 2 2 2 2 2
226/300 Flush
227/300 Read block 208 len 8192
228/300 Read block 1455 len 16384
229/300 Write block 351 len 20480 data: 4 4 4 4 3
230/300 Flush
231/300 Flush
232/300 Write block 4285 len 4096 data: 3
233/300 Read block 3553 len 24576
234/300 Read block 721 len 40960
235/300 Read block 511 len 4096
236/300 Flush
237/300 Read block 1558 len 12288
238/300 Write block 326 len 32768 data: 2 2 2 2 3 5 5 3
239/300 Flush
240/300 Read block 2877 len 28672
241/300 Read block 1542 len 8192
242/300 Read block 3180 len 12288
243/300 Write block 4276 len 16384 data: 3 3 3 2
244/300 Read block 585 len 36864
245/300 Read block 964 len 16384
246/300 Read block 1472 len 24576
247/300 Write block 4662 len 32768 data: 3 3 3 3 3 3 3 3
248/300 Read block 4270 len 4096
249/300 Read block 3465 len 16384
250/300 Write block 3201 len 24576 data: 2 2 2 2 2 2
251/300 Write block 1602 len 40960 data: 4 4 2 3 3 3 3 3 3 4
252/300 Read block 160 len 40960
253/300 Read block 1285 len 32768
254/300 Read block 3670 len 24576
255/300 Read block 39 len 20480
256/300 Read block 514 len 8192
257/300 Read block 3353 len 36864
258/300 Read block 4363 len 16384
259/300 Read block 3419 len 40960
260/300 Flush
261/300 Read block 2962 len 16384
262/300 Read block 998 len 16384
263/300 Read block 2220 len 32768
264/300 Read block 2385 len 28672
265/300 Read block 312 len 8192
266/300 Read block 2899 len 24576
267/300 Read block 3118 len 8192
268/300 Write block 3097 len 20480 data: 2 2 2 2 3
269/300 Write block 2367 len 36864 data: 3 3 3 3 3 2 2 2 2
270/300 Write block 2559 len 32768 data: 2 2 2 2 3 3 3 4
271/300 Write block 3509 len 36864 data: 3 3 3 2 2 3 3 3 3
272/300 Write block 1586 len 24576 data: 2 3 3 3 2 2
273/300 Write block 2414 len 24576 data: 2 2 2 2 2 2
274/300 Read block 1292 len 8192
275/300 Write block 4793 len 28672 data: 2 3 3 3 3 3 3
276/300 Read block 2962 len 40960
277/300 Read block 1128 len 40960
278/300 Read block 680 len 4096
279/300 Read block 3657 len 8192
280/300 Flush
281/300 Write block 3016 len 32768 data: 3 3 2 2 2 2 2 2
282/300 Read block 3601 len 16384
283/300 Write block 64 len 32768 data: 4 4 3 2 2 2 2 2
284/300 Flush
285/300 Read block 3897 len 16384
286/300 Write block 3101 len 32768 data: 4 3 3 2 3 3 3 3
287/300 Read block 2935 len 16384
288/300 Write block 3665 len 24576 data: 2 2 2 2 3 4
289/300 Write block 441 len 8192 data: 2 2
290/300 Read block 4491 len 28672
291/300 Read block 2548 len 20480
292/300 Write block 3097 len 12288 data: 3 3 3
293/300 Write block 2254 len 16384 data: 2 2 2 2
294/300 Read block 1029 len 24576
295/300 Read block 4074 len 40960
296/300 Write block 2327 len 16384 data: 3 3 2 2
297/300 Read block 709 len 20480
298/300 Read block 1472 len 24576
299/300 Write block 1946 len 40960 data: 2 2 2 2 2 2 2 2 2 2
300/300 Read block 4447 len 12288
Main task has work to do, go find it
got dsc Start(1)
start 1
Feb 13 21:40:30.361 INFO request completed, latency_us: 210, response_code: 204, uri: /start/cid/1, method: POST, req_id: 6884e425-83fe-480e-a88f-771710614302, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
1206 Acked 2205 Write 2 Done New Done true
1207 Acked 2206 Read 8 Done New Done true
1208 Acked 2207 Write 8 Done New Done true
1209 Acked 2208 Read 1 Done New Done true
1210 Acked 2209 Read 10 Done New Done false
1211 Acked 2210 Write 9 Done New Done false
1212 Acked 2211 Read 4 Done New Done false
1213 Acked 2212 Flush 0 Done New Done false
1214 Acked 2213 Read 5 Done New Done false
1215 Acked 2214 Read 9 Done New Done false
1216 Acked 2215 Read 9 Done New Done false
1217 Acked 2216 Read 9 Done New Done false
1218 Acked 2217 Write 6 Done New Done false
1219 Acked 2218 Read 2 Done New Done false
1220 Acked 2219 Read 1 Done New Done false
1221 Acked 2220 Write 7 Done New Done false
1222 Acked 2221 Write 10 Done New Done false
1223 Acked 2222 Read 4 Done New Done false
1224 Acked 2223 Write 4 Done New Done false
1225 Acked 2224 Read 9 Done New Done false
1226 Acked 2225 Write 8 Done New Done false
1227 Acked 2226 Write 10 Done New Done false
1228 Acked 2227 Write 6 Done New Done false
1229 Acked 2228 Read 8 Done New Done false
1230 Acked 2229 Read 1 Done New Done false
1231 Acked 2230 Read 1 Done New Done false
1232 Acked 2231 Read 7 Done New Done false
1233 Acked 2232 Read 4 Done New Done false
1234 Acked 2233 Flush 0 Done New Done false
1235 Acked 2234 Write 4 Done New Done false
1236 Acked 2235 Read 9 Done New Done false
1237 Acked 2236 Write 7 Done New Done false
1238 Acked 2237 Write 8 Done New Done false
1239 Acked 2238 Read 7 Done New Done false
1240 Acked 2239 Write 9 Done New Done false
1241 Acked 2240 Flush 0 Done New Done false
1242 Acked 2241 Read 4 Done New Done false
1243 Acked 2242 Read 2 Done New Done false
1244 Acked 2243 Write 10 Done New Done false
1245 Acked 2244 Read 1 Done New Done false
1246 Acked 2245 Read 9 Done New Done false
1247 Acked 2246 Read 5 Done New Done false
1248 Acked 2247 Write 9 Done New Done false
1249 Acked 2248 Read 4 Done New Done false
1250 Acked 2249 Read 5 Done New Done false
1251 Acked 2250 Read 7 Done New Done false
1252 Acked 2251 Read 5 Done New Done false
1253 Acked 2252 Flush 0 Done New Done false
1254 Acked 2253 Read 8 Done New Done false
1255 Acked 2254 Read 2 Done New Done false
1256 Acked 2255 Write 9 Done New Done false
1257 Acked 2256 Write 1 Done New Done false
1258 Acked 2257 Read 6 Done New Done false
1259 Acked 2258 Flush 0 Done New Done false
1260 Acked 2259 Read 8 Done New Done false
1261 Acked 2260 Read 1 Done New Done false
1262 Acked 2261 Write 9 Done New Done false
1263 Acked 2262 Read 4 Done New Done false
1264 Acked 2263 Read 1 Done New Done false
1265 Acked 2264 Read 10 Done New Done false
1266 Acked 2265 Read 1 Done New Done false
1267 Acked 2266 Read 8 Done New Done false
1268 Acked 2267 Write 2 Done New Done false
1269 Acked 2268 Read 2 Done New Done false
1270 Acked 2269 Read 9 Done New Done false
1271 Acked 2270 Read 7 Done New Done false
1272 Acked 2271 Write 4 Done New Done false
1273 Acked 2272 Read 7 Done New Done false
1274 Acked 2273 Read 1 Done New Done false
1275 Acked 2274 Flush 0 Done New Done false
1276 Acked 2275 Read 10 Done New Done false
1277 Acked 2276 Read 7 Done New Done false
1278 Acked 2277 Write 10 Done New Done false
1279 Acked 2278 Write 1 Done New Done false
1280 Acked 2279 Read 1 Done New Done false
1281 Acked 2280 Read 5 Done New Done false
1282 Acked 2281 Write 7 Done New Done false
1283 Acked 2282 Read 2 Done New Done false
1284 Acked 2283 Write 8 Done New Done false
1285 Acked 2284 Read 8 Done New Done false
1286 Acked 2285 Read 8 Done New Done false
1287 Acked 2286 Flush 0 Done New Done false
1288 Acked 2287 Read 8 Done New Done false
1289 Acked 2288 Read 10 Done New Done false
1290 Acked 2289 Read 6 Done New Done false
1291 Acked 2290 Read 8 Done New Done false
1292 Acked 2291 Read 6 Done New Done false
1293 Acked 2292 Read 1 Done New Done false
1294 Acked 2293 Flush 0 Done New Done false
1295 Acked 2294 Read 4 Done New Done false
1296 Acked 2295 Flush 0 Done New Done false
1297 Acked 2296 Write 3 Done New Done false
1298 Acked 2297 Read 5 Done New Done false
1299 Acked 2298 Write 1 Done New Done false
1300 Acked 2299 Write 9 Done New Done false
1301 Acked 2300 Read 10 Done New Done false
1302 Acked 2301 Write 2 Done New Done false
1303 Acked 2302 Write 5 Done New Done false
1304 Acked 2303 Flush 0 Done New Done false
1305 Acked 2304 Read 10 Done New Done false
1306 Acked 2305 Write 4 Done New Done false
1307 Acked 2306 Write 2 Done New Done false
1308 Acked 2307 Write 5 Done New Done false
1309 Acked 2308 Write 6 Done New Done false
1310 Acked 2309 Write 9 Done New Done false
1311 Acked 2310 Read 8 Done New Done false
1312 Acked 2311 Write 2 Done New Done false
1313 Acked 2312 Flush 0 Done New Done false
1314 Acked 2313 Write 1 Done New Done false
1315 Acked 2314 Write 7 Done New Done false
1316 Acked 2315 Write 4 Done New Done false
1317 Acked 2316 Read 9 Done New Done false
1318 Acked 2317 Read 4 Done New Done false
1319 Acked 2318 Read 10 Done New Done false
1320 Acked 2319 Read 6 Done New Done false
1321 Acked 2320 Flush 0 Done New Done false
1322 Acked 2321 Read 2 Done New Done false
1323 Acked 2322 Write 5 Done New Done false
1324 Acked 2323 Read 5 Done New Done false
1325 Acked 2324 Write 3 Done New Done false
1326 Acked 2325 Read 10 Done New Done false
1327 Acked 2326 Read 6 Done New Done false
1328 Acked 2327 Read 9 Done New Done false
1329 Acked 2328 Read 10 Done New Done false
1330 Acked 2329 Read 3 Done New Done false
1331 Acked 2330 Read 1 Done New Done false
1332 Acked 2331 Write 5 Done New Done false
1333 Acked 2332 Read 10 Done New Done false
1334 Acked 2333 Read 9 Done New Done false
1335 Acked 2334 Write 10 Done New Done false
1336 Acked 2335 Write 6 Done New Done false
1337 Acked 2336 Write 4 Done New Done false
1338 Acked 2337 Read 5 Done New Done false
1339 Acked 2338 Read 6 Done New Done false
1340 Acked 2339 Write 2 Done New Done false
1341 Acked 2340 Read 9 Done New Done false
1342 Acked 2341 Read 7 Done New Done false
1343 Acked 2342 Write 9 Done New Done false
1344 Acked 2343 Write 9 Done New Done false
1345 Acked 2344 Flush 0 Done New Done false
1346 Acked 2345 Read 10 Done New Done false
1347 Acked 2346 Read 6 Done New Done false
1348 Acked 2347 Read 7 Done New Done false
1349 Acked 2348 Write 9 Done New Done false
1350 Acked 2349 Write 4 Done New Done false
1351 Acked 2350 Read 9 Done New Done false
1352 Acked 2351 Flush 0 Done New Done false
1353 Acked 2352 Flush 0 Done New Done false
1354 Acked 2353 Read 9 Done New Done false
1355 Acked 2354 Read 1 Done New Done false
1356 Acked 2355 Read 9 Done New Done false
1357 Acked 2356 Read 7 Done New Done false
1358 Acked 2357 Write 3 Done New Done false
1359 Acked 2358 Write 6 Done New Done false
1360 Acked 2359 Write 4 Done New Done false
1361 Acked 2360 Read 7 Done New Done false
1362 Acked 2361 Read 5 Done New Done false
1363 Acked 2362 Write 1 Done New Done false
1364 Acked 2363 Write 6 Done New Done false
1365 Acked 2364 Write 2 Done New Done false
1366 Acked 2365 Read 8 Done New Done false
1367 Acked 2366 Write 3 Done New Done false
1368 Acked 2367 Read 8 Done New Done false
1369 Acked 2368 Flush 0 Done New Done false
1370 Acked 2369 Write 1 Done New Done false
1371 Acked 2370 Write 4 Done New Done false
1372 Acked 2371 Read 10 Done New Done false
1373 Acked 2372 Flush 0 Done New Done false
1374 Acked 2373 Read 1 Done New Done false
1375 Acked 2374 Flush 0 Done New Done false
1376 Acked 2375 Write 1 Done New Done false
1377 Acked 2376 Write 6 Done New Done false
1378 Acked 2377 Write 5 Done New Done false
1379 Acked 2378 Write 2 Done New Done false
1380 Acked 2379 Write 6 Done New Done false
1381 Acked 2380 Read 8 Done New Done false
1382 Acked 2381 Write 5 Done New Done false
1383 Acked 2382 Flush 0 Done New Done false
1384 Acked 2383 Read 10 Done New Done false
1385 Acked 2384 Write 2 Done New Done false
1386 Acked 2385 Flush 0 Done New Done false
1387 Acked 2386 Read 8 Done New Done false
1388 Acked 2387 Read 10 Done New Done false
1389 Acked 2388 Flush 0 Done New Done false
1390 Acked 2389 Read 3 Done New Done false
1391 Acked 2390 Read 4 Done New Done false
1392 Acked 2391 Flush 0 Done New Done false
1393 Acked 2392 Write 4 Done New Done false
1394 Acked 2393 Read 10 Done New Done false
1395 Acked 2394 Write 5 Done New Done false
1396 Acked 2395 Write 3 Done New Done false
1397 Acked 2396 Read 9 Done New Done false
1398 Acked 2397 Read 3 Done New Done false
1399 Acked 2398 Write 6 Done New Done false
1400 Acked 2399 Read 6 Done New Done false
1401 Acked 2400 Read 7 Done New Done false
1402 Acked 2401 Read 6 Done New Done false
1403 Acked 2402 Flush 0 Done New Done false
1404 Acked 2403 Read 5 Done New Done false
1405 Acked 2404 Read 10 Done New Done false
1406 Acked 2405 Write 8 Done New Done false
1407 Acked 2406 Write 2 Done New Done false
1408 Acked 2407 Read 1 Done New Done false
1409 Acked 2408 Write 9 Done New Done false
1410 Acked 2409 Read 6 Done New Done false
1411 Acked 2410 Read 9 Done New Done false
1412 Acked 2411 Read 3 Done New Done false
1413 Acked 2412 Read 9 Done New Done false
1414 Acked 2413 Read 8 Done New Done false
1415 Acked 2414 Read 8 Done New Done false
1416 Acked 2415 Read 4 Done New Done false
1417 Acked 2416 Read 1 Done New Done false
1418 Acked 2417 Write 3 Done New Done false
1419 Acked 2418 Write 10 Done New Done false
1420 Acked 2419 Write 9 Done New Done false
1421 Acked 2420 Read 7 Done New Done false
1422 Acked 2421 Write 10 Done New Done false
1423 Acked 2422 Write 6 Done New Done false
1424 Acked 2423 Read 7 Done New Done false
1425 Acked 2424 Write 1 Done New Done false
1426 Acked 2425 Read 8 Done New Done false
1427 Acked 2426 Write 4 Done New Done false
1428 Acked 2427 Read 6 Done New Done false
1429 Acked 2428 Read 1 Done New Done false
1430 Acked 2429 Read 9 Done New Done false
1431 Acked 2430 Write 8 Done New Done false
1432 Acked 2431 Flush 0 Done New Done false
1433 Acked 2432 Read 2 Done New Done false
1434 Acked 2433 Read 4 Done New Done false
1435 Acked 2434 Write 5 Done New Done false
1436 Acked 2435 Flush 0 Done New Done false
1437 Acked 2436 Flush 0 Done New Done false
1438 Acked 2437 Write 1 Done New Done false
1439 Acked 2438 Read 6 Done New Done false
1440 Acked 2439 Read 10 Done New Done false
1441 Acked 2440 Read 1 Done New Done false
1442 Acked 2441 Flush 0 Done New Done false
1443 Acked 2442 Read 3 Done New Done false
1444 Acked 2443 Write 8 Done New Done false
1445 Acked 2444 Flush 0 Done New Done false
1446 Acked 2445 Read 7 Done New Done false
1447 Acked 2446 Read 2 Done New Done false
1448 Acked 2447 Read 3 Done New Done false
1449 Acked 2448 Write 4 Done New Done false
1450 Acked 2449 Read 9 Done New Done false
1451 Acked 2450 Read 4 Done New Done false
1452 Acked 2451 Read 6 Done New Done false
1453 Acked 2452 Write 8 Done New Done false
1454 Acked 2453 Read 1 Done New Done false
1455 Acked 2454 Read 4 Done New Done false
1456 Acked 2455 Write 6 Done New Done false
1457 Acked 2456 Write 10 Done New Done false
1458 Acked 2457 Read 10 Done New Done false
1459 Acked 2458 Read 8 Done New Done false
1460 Acked 2459 Read 6 Done New Done false
1461 Acked 2460 Read 5 Done New Done false
1462 Acked 2461 Read 2 Done New Done false
1463 Acked 2462 Read 9 Done New Done false
1464 Acked 2463 Read 4 Done New Done false
1465 Acked 2464 Read 10 Done New Done false
1466 Acked 2465 Flush 0 Done New Done false
1467 Acked 2466 Read 4 Done New Done false
1468 Acked 2467 Read 4 Done New Done false
1469 Acked 2468 Read 8 Done New Done false
1470 Acked 2469 Read 7 Done New Done false
1471 Acked 2470 Read 2 Done New Done false
1472 Acked 2471 Read 6 Done New Done false
1473 Acked 2472 Read 2 Done New Done false
1474 Acked 2473 Write 5 Done New Done false
1475 Acked 2474 Write 9 Done New Done false
1476 Acked 2475 Write 8 Done New Done false
1477 Acked 2476 Write 9 Done New Done false
1478 Acked 2477 Write 6 Done New Done false
1479 Acked 2478 Write 6 Done New Done false
1480 Acked 2479 Read 2 Done New Done false
1481 Acked 2480 Write 7 Done New Done false
1482 Acked 2481 Read 10 Done New Done false
1483 Acked 2482 Read 10 Done New Done false
1484 Acked 2483 Read 1 Done New Done false
1485 Acked 2484 Read 2 Done New Done false
1486 Acked 2485 Flush 0 Done New Done false
1487 Acked 2486 Write 8 Done New Done false
1488 Acked 2487 Read 4 Done New Done false
1489 Acked 2488 Write 8 Done New Done false
1490 Acked 2489 Flush 0 Done New Done false
1491 Acked 2490 Read 4 Done New Done false
1492 Acked 2491 Write 8 Done New Done false
1493 Acked 2492 Read 4 Done New Done false
1494 Acked 2493 Write 6 Done New Done false
1495 Acked 2494 Write 2 Done New Done false
1496 Acked 2495 Read 7 Done New Done false
1497 Acked 2496 Read 5 Done New Done false
1498 Acked 2497 Write 3 Done New Done false
1499 Acked 2498 Write 4 Done New Done false
1500 Acked 2499 Read 6 Done New Done false
1501 Acked 2500 Read 10 Done New Done false
1502 Acked 2501 Write 4 Done New Done false
1503 Acked 2502 Read 5 Done New Done false
1504 Acked 2503 Read 6 Done New Done false
1505 Acked 2504 Write 10 Done New Done false
1506 Acked 2505 Read 3 Done New Done false
STATES DS:0 DS:1 DS:2 TOTAL
New 0 301 0 301
Sent 0 0 0 0
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 2489 2204 2489
Downstairs last five completed: 2204 2203 2202 2201 2200
Upstairs last five completed: 1506 1505 1504 1503 1502
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:30.737026784Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:30.737087562Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:30.737111301Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:30.73713899Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:30.737168039Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 302 jobs since flush 2204","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:30.737197538Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:30.737226657Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:31.737656439Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:31.737705247Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:31.737726326Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:31.737746256Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:31.737761995Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 302 jobs since flush 2204","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:31.737779434Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:31.737912799Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:32.738969557Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:32.739026355Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:32.739047274Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:32.739074953Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:32.739091153Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 302 jobs since flush 2204","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:32.739108392Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:32.739237637Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:33.739759775Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:33.739811213Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:33.739832422Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:33.739853081Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:33.739868791Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 302 jobs since flush 2204","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:33.73988799Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:33.740009935Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1169)
[8820][1] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Offline Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
1206 Acked 2205 Write 2 Done New Done true
1207 Acked 2206 Read 8 Done New Done true
1208 Acked 2207 Write 8 Done New Done true
1209 Acked 2208 Read 1 Done New Done true
1210 Acked 2209 Read 10 Done New Done true
1211 Acked 2210 Write 9 Done New Done true
1212 Acked 2211 Read 4 Done New Done true
1213 Acked 2212 Flush 0 Done New Done true
1214 Acked 2213 Read 5 Done New Done true
1215 Acked 2214 Read 9 Done New Done true
1216 Acked 2215 Read 9 Done New Done true
1217 Acked 2216 Read 9 Done New Done true
1218 Acked 2217 Write 6 Done New Done true
1219 Acked 2218 Read 2 Done New Done true
1220 Acked 2219 Read 1 Done New Done true
1221 Acked 2220 Write 7 Done New Done true
1222 Acked 2221 Write 10 Done New Done true
1223 Acked 2222 Read 4 Done New Done true
1224 Acked 2223 Write 4 Done New Done true
1225 Acked 2224 Read 9 Done New Done true
1226 Acked 2225 Write 8 Done New Done true
1227 Acked 2226 Write 10 Done New Done true
1228 Acked 2227 Write 6 Done New Done true
1229 Acked 2228 Read 8 Done New Done true
1230 Acked 2229 Read 1 Done New Done true
1231 Acked 2230 Read 1 Done New Done true
1232 Acked 2231 Read 7 Done New Done true
1233 Acked 2232 Read 4 Done New Done true
1234 Acked 2233 Flush 0 Done New Done true
1235 Acked 2234 Write 4 Done New Done true
1236 Acked 2235 Read 9 Done New Done true
1237 Acked 2236 Write 7 Done New Done true
1238 Acked 2237 Write 8 Done New Done true
1239 Acked 2238 Read 7 Done New Done true
1240 Acked 2239 Write 9 Done New Done true
1241 Acked 2240 Flush 0 Done New Done true
1242 Acked 2241 Read 4 Done New Done true
1243 Acked 2242 Read 2 Done New Done true
1244 Acked 2243 Write 10 Done New Done true
1245 Acked 2244 Read 1 Done New Done true
1246 Acked 2245 Read 9 Done New Done true
1247 Acked 2246 Read 5 Done New Done true
1248 Acked 2247 Write 9 Done New Done true
1249 Acked 2248 Read 4 Done New Done true
1250 Acked 2249 Read 5 Done New Done true
1251 Acked 2250 Read 7 Done New Done true
1252 Acked 2251 Read 5 Done New Done true
1253 Acked 2252 Flush 0 Done New Done true
1254 Acked 2253 Read 8 Done New Done true
1255 Acked 2254 Read 2 Done New Done true
1256 Acked 2255 Write 9 Done New Done true
1257 Acked 2256 Write 1 Done New Done true
1258 Acked 2257 Read 6 Done New Done true
1259 Acked 2258 Flush 0 Done New Done true
1260 Acked 2259 Read 8 Done New Done true
1261 Acked 2260 Read 1 Done New Done true
1262 Acked 2261 Write 9 Done New Done true
1263 Acked 2262 Read 4 Done New Done true
1264 Acked 2263 Read 1 Done New Done true
1265 Acked 2264 Read 10 Done New Done true
1266 Acked 2265 Read 1 Done New Done true
1267 Acked 2266 Read 8 Done New Done true
1268 Acked 2267 Write 2 Done New Done true
1269 Acked 2268 Read 2 Done New Done true
1270 Acked 2269 Read 9 Done New Done true
1271 Acked 2270 Read 7 Done New Done true
1272 Acked 2271 Write 4 Done New Done true
1273 Acked 2272 Read 7 Done New Done true
1274 Acked 2273 Read 1 Done New Done true
1275 Acked 2274 Flush 0 Done New Done true
1276 Acked 2275 Read 10 Done New Done true
1277 Acked 2276 Read 7 Done New Done true
1278 Acked 2277 Write 10 Done New Done true
1279 Acked 2278 Write 1 Done New Done true
1280 Acked 2279 Read 1 Done New Done true
1281 Acked 2280 Read 5 Done New Done true
1282 Acked 2281 Write 7 Done New Done true
1283 Acked 2282 Read 2 Done New Done true
1284 Acked 2283 Write 8 Done New Done true
1285 Acked 2284 Read 8 Done New Done true
1286 Acked 2285 Read 8 Done New Done true
1287 Acked 2286 Flush 0 Done New Done true
1288 Acked 2287 Read 8 Done New Done true
1289 Acked 2288 Read 10 Done New Done true
1290 Acked 2289 Read 6 Done New Done true
1291 Acked 2290 Read 8 Done New Done true
1292 Acked 2291 Read 6 Done New Done true
1293 Acked 2292 Read 1 Done New Done true
1294 Acked 2293 Flush 0 Done New Done true
1295 Acked 2294 Read 4 Done New Done true
1296 Acked 2295 Flush 0 Done New Done true
1297 Acked 2296 Write 3 Done New Done true
1298 Acked 2297 Read 5 Done New Done true
1299 Acked 2298 Write 1 Done New Done true
1300 Acked 2299 Write 9 Done New Done true
1301 Acked 2300 Read 10 Done New Done true
1302 Acked 2301 Write 2 Done New Done true
1303 Acked 2302 Write 5 Done New Done true
1304 Acked 2303 Flush 0 Done New Done true
1305 Acked 2304 Read 10 Done New Done true
1306 Acked 2305 Write 4 Done New Done true
1307 Acked 2306 Write 2 Done New Done true
1308 Acked 2307 Write 5 Done New Done true
1309 Acked 2308 Write 6 Done New Done true
1310 Acked 2309 Write 9 Done New Done true
1311 Acked 2310 Read 8 Done New Done true
1312 Acked 2311 Write 2 Done New Done true
1313 Acked 2312 Flush 0 Done New Done true
1314 Acked 2313 Write 1 Done New Done true
1315 Acked 2314 Write 7 Done New Done true
1316 Acked 2315 Write 4 Done New Done true
1317 Acked 2316 Read 9 Done New Done true
1318 Acked 2317 Read 4 Done New Done true
1319 Acked 2318 Read 10 Done New Done true
1320 Acked 2319 Read 6 Done New Done true
1321 Acked 2320 Flush 0 Done New Done true
1322 Acked 2321 Read 2 Done New Done true
1323 Acked 2322 Write 5 Done New Done true
1324 Acked 2323 Read 5 Done New Done true
1325 Acked 2324 Write 3 Done New Done true
1326 Acked 2325 Read 10 Done New Done true
1327 Acked 2326 Read 6 Done New Done true
1328 Acked 2327 Read 9 Done New Done true
1329 Acked 2328 Read 10 Done New Done true
1330 Acked 2329 Read 3 Done New Done true
1331 Acked 2330 Read 1 Done New Done true
1332 Acked 2331 Write 5 Done New Done true
1333 Acked 2332 Read 10 Done New Done true
1334 Acked 2333 Read 9 Done New Done true
1335 Acked 2334 Write 10 Done New Done true
1336 Acked 2335 Write 6 Done New Done true
1337 Acked 2336 Write 4 Done New Done true
1338 Acked 2337 Read 5 Done New Done true
1339 Acked 2338 Read 6 Done New Done true
1340 Acked 2339 Write 2 Done New Done true
1341 Acked 2340 Read 9 Done New Done true
1342 Acked 2341 Read 7 Done New Done true
1343 Acked 2342 Write 9 Done New Done true
1344 Acked 2343 Write 9 Done New Done true
1345 Acked 2344 Flush 0 Done New Done true
1346 Acked 2345 Read 10 Done New Done true
1347 Acked 2346 Read 6 Done New Done true
1348 Acked 2347 Read 7 Done New Done true
1349 Acked 2348 Write 9 Done New Done true
1350 Acked 2349 Write 4 Done New Done true
1351 Acked 2350 Read 9 Done New Done true
1352 Acked 2351 Flush 0 Done New Done true
1353 Acked 2352 Flush 0 Done New Done true
1354 Acked 2353 Read 9 Done New Done true
1355 Acked 2354 Read 1 Done New Done true
1356 Acked 2355 Read 9 Done New Done true
1357 Acked 2356 Read 7 Done New Done true
1358 Acked 2357 Write 3 Done New Done true
1359 Acked 2358 Write 6 Done New Done true
1360 Acked 2359 Write 4 Done New Done true
1361 Acked 2360 Read 7 Done New Done true
1362 Acked 2361 Read 5 Done New Done true
1363 Acked 2362 Write 1 Done New Done true
1364 Acked 2363 Write 6 Done New Done true
1365 Acked 2364 Write 2 Done New Done true
1366 Acked 2365 Read 8 Done New Done true
1367 Acked 2366 Write 3 Done New Done true
1368 Acked 2367 Read 8 Done New Done true
1369 Acked 2368 Flush 0 Done New Done true
1370 Acked 2369 Write 1 Done New Done true
1371 Acked 2370 Write 4 Done New Done true
1372 Acked 2371 Read 10 Done New Done true
1373 Acked 2372 Flush 0 Done New Done true
1374 Acked 2373 Read 1 Done New Done true
1375 Acked 2374 Flush 0 Done New Done true
1376 Acked 2375 Write 1 Done New Done true
1377 Acked 2376 Write 6 Done New Done true
1378 Acked 2377 Write 5 Done New Done true
1379 Acked 2378 Write 2 Done New Done true
1380 Acked 2379 Write 6 Done New Done true
1381 Acked 2380 Read 8 Done New Done true
1382 Acked 2381 Write 5 Done New Done true
1383 Acked 2382 Flush 0 Done New Done true
1384 Acked 2383 Read 10 Done New Done true
1385 Acked 2384 Write 2 Done New Done true
1386 Acked 2385 Flush 0 Done New Done true
1387 Acked 2386 Read 8 Done New Done true
1388 Acked 2387 Read 10 Done New Done true
1389 Acked 2388 Flush 0 Done New Done true
1390 Acked 2389 Read 3 Done New Done true
1391 Acked 2390 Read 4 Done New Done true
1392 Acked 2391 Flush 0 Done New Done true
1393 Acked 2392 Write 4 Done New Done true
1394 Acked 2393 Read 10 Done New Done true
1395 Acked 2394 Write 5 Done New Done true
1396 Acked 2395 Write 3 Done New Done true
1397 Acked 2396 Read 9 Done New Done true
1398 Acked 2397 Read 3 Done New Done true
1399 Acked 2398 Write 6 Done New Done true
1400 Acked 2399 Read 6 Done New Done true
1401 Acked 2400 Read 7 Done New Done true
1402 Acked 2401 Read 6 Done New Done true
1403 Acked 2402 Flush 0 Done New Done true
1404 Acked 2403 Read 5 Done New Done true
1405 Acked 2404 Read 10 Done New Done true
1406 Acked 2405 Write 8 Done New Done true
1407 Acked 2406 Write 2 Done New Done true
1408 Acked 2407 Read 1 Done New Done true
1409 Acked 2408 Write 9 Done New Done true
1410 Acked 2409 Read 6 Done New Done true
1411 Acked 2410 Read 9 Done New Done true
1412 Acked 2411 Read 3 Done New Done true
1413 Acked 2412 Read 9 Done New Done true
1414 Acked 2413 Read 8 Done New Done true
1415 Acked 2414 Read 8 Done New Done true
1416 Acked 2415 Read 4 Done New Done true
1417 Acked 2416 Read 1 Done New Done true
1418 Acked 2417 Write 3 Done New Done true
1419 Acked 2418 Write 10 Done New Done true
1420 Acked 2419 Write 9 Done New Done true
1421 Acked 2420 Read 7 Done New Done true
1422 Acked 2421 Write 10 Done New Done true
1423 Acked 2422 Write 6 Done New Done true
1424 Acked 2423 Read 7 Done New Done true
1425 Acked 2424 Write 1 Done New Done true
1426 Acked 2425 Read 8 Done New Done true
1427 Acked 2426 Write 4 Done New Done true
1428 Acked 2427 Read 6 Done New Done true
1429 Acked 2428 Read 1 Done New Done true
1430 Acked 2429 Read 9 Done New Done true
1431 Acked 2430 Write 8 Done New Done true
1432 Acked 2431 Flush 0 Done New Done true
1433 Acked 2432 Read 2 Done New Done true
1434 Acked 2433 Read 4 Done New Done true
1435 Acked 2434 Write 5 Done New Done true
1436 Acked 2435 Flush 0 Done New Done true
1437 Acked 2436 Flush 0 Done New Done true
1438 Acked 2437 Write 1 Done New Done true
1439 Acked 2438 Read 6 Done New Done true
1440 Acked 2439 Read 10 Done New Done true
1441 Acked 2440 Read 1 Done New Done true
1442 Acked 2441 Flush 0 Done New Done true
1443 Acked 2442 Read 3 Done New Done true
1444 Acked 2443 Write 8 Done New Done true
1445 Acked 2444 Flush 0 Done New Done true
1446 Acked 2445 Read 7 Done New Done true
1447 Acked 2446 Read 2 Done New Done true
1448 Acked 2447 Read 3 Done New Done true
1449 Acked 2448 Write 4 Done New Done true
1450 Acked 2449 Read 9 Done New Done true
1451 Acked 2450 Read 4 Done New Done true
1452 Acked 2451 Read 6 Done New Done true
1453 Acked 2452 Write 8 Done New Done true
1454 Acked 2453 Read 1 Done New Done true
1455 Acked 2454 Read 4 Done New Done true
1456 Acked 2455 Write 6 Done New Done true
1457 Acked 2456 Write 10 Done New Done true
1458 Acked 2457 Read 10 Done New Done true
1459 Acked 2458 Read 8 Done New Done true
1460 Acked 2459 Read 6 Done New Done true
1461 Acked 2460 Read 5 Done New Done true
1462 Acked 2461 Read 2 Done New Done true
1463 Acked 2462 Read 9 Done New Done true
1464 Acked 2463 Read 4 Done New Done true
1465 Acked 2464 Read 10 Done New Done true
1466 Acked 2465 Flush 0 Done New Done true
1467 Acked 2466 Read 4 Done New Done true
1468 Acked 2467 Read 4 Done New Done true
1469 Acked 2468 Read 8 Done New Done true
1470 Acked 2469 Read 7 Done New Done true
1471 Acked 2470 Read 2 Done New Done true
1472 Acked 2471 Read 6 Done New Done true
1473 Acked 2472 Read 2 Done New Done true
1474 Acked 2473 Write 5 Done New Done true
1475 Acked 2474 Write 9 Done New Done true
1476 Acked 2475 Write 8 Done New Done true
1477 Acked 2476 Write 9 Done New Done true
1478 Acked 2477 Write 6 Done New Done true
1479 Acked 2478 Write 6 Done New Done true
1480 Acked 2479 Read 2 Done New Done true
1481 Acked 2480 Write 7 Done New Done true
1482 Acked 2481 Read 10 Done New Done true
1483 Acked 2482 Read 10 Done New Done true
1484 Acked 2483 Read 1 Done New Done true
1485 Acked 2484 Read 2 Done New Done true
1486 Acked 2485 Flush 0 Done New Done true
1487 Acked 2486 Write 8 Done New Done true
1488 Acked 2487 Read 4 Done New Done true
1489 Acked 2488 Write 8 Done New Done true
1490 Acked 2489 Flush 0 Done New Done true
1491 Acked 2490 Read 4 Done New Done true
1492 Acked 2491 Write 8 Done New Done true
1493 Acked 2492 Read 4 Done New Done true
1494 Acked 2493 Write 6 Done New Done true
1495 Acked 2494 Write 2 Done New Done true
1496 Acked 2495 Read 7 Done New Done true
1497 Acked 2496 Read 5 Done New Done true
1498 Acked 2497 Write 3 Done New Done true
1499 Acked 2498 Write 4 Done New Done true
1500 Acked 2499 Read 6 Done New Done true
1501 Acked 2500 Read 10 Done New Done true
1502 Acked 2501 Write 4 Done New Done true
1503 Acked 2502 Read 5 Done New Done true
1504 Acked 2503 Read 6 Done New Done true
1505 Acked 2504 Write 10 Done New Done true
1506 Acked 2505 Read 3 Done New Done true
1507 Acked 2506 Flush 0 Done New Done true
STATES DS:0 DS:1 DS:2 TOTAL
New 0 302 0 302
Sent 0 0 0 0
Done 302 0 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 2506 2204 2506
Downstairs last five completed: 2204 2203 2202 2201 2200
Upstairs last five completed: 1507 1506 1505 1504 1503
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:34.741322284Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:34.741385232Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:34.742771209Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:34.742797248Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 2204","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:34.742815867Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 2204","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:34.743031809Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:34.743061647Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:34.743078277Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:34.743093146Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:34.743107426Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 2506 2505 2504 2503 2502
Upstairs last five completed: 1507 1506 1505 1504 1503
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Feb 13 21:40:38.366 INFO request completed, latency_us: 346, response_code: 204[8810] Got stop action so:false kr:false
, uri: /stop/cid/0, method: POST, req_id: 8e54517a-23c8-4382-a6be-20f81e8d9317, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:38.367611449Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:40:38.367 INFO request completed, latency_us: 133, response_code: 200, uri: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:38.367706845Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
/state/cid/0, method: GET, req_id: 9faa8de5-5d3d-43ba-ac7c-6629c5eaf3db{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:38.367729495Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
, remote_addr: 127.0.0.{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:38.367753223Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
1:52124, local_addr: 127.0.0.1:9998{"msg":"[0] client re-new 0 jobs since flush 2506","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:38.367769643Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:38.367803092Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:39.36886821Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:39.368939917Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:39.368965596Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:39.368986585Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:39.369002705Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 2506","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:39.369033154Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:39.369052923Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:40.370022464Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:40.370085522Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:40.370109281Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:40.3701306Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:40.370146949Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 2506","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:40.370164648Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:40.370182348Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:41.370726125Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:41.370785573Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:41.370808532Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:41.370829451Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:41.370845421Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 2506","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:41.37086352Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:41.370881149Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:40:42.369 INFO request completed, latency_us: 220, response_code: 200, uri: /state/cid/0, method: GET, req_id: 029bf418-3752-49bd-a0db-b18558f6085f, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Read block 3427 len 36864
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30002/300 Read block 897 len 4096
,"time":"2024-02-13T21:40:42.371657788Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:42.371707977Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:42.371726496Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:42.371743195Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:42.371756025Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 1 jobs since flush 2506","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:42.371769964Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:42.37187958Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
003/300 Write block 2571 len 32768 data: 3 3 4 4 3 2 2 2
004/300 Read block 3114 len 40960
005/300 Read block 3131 len 16384
006/300 Write block 1846 len 36864 data: 2 2 2 2 2 2 2 2 3
007/300 Read block 1395 len 28672
008/300 Write block 4721 len 24576 data: 3 4 3 3 3 3
009/300 Read block 961 len 4096
010/300 Read block 1246 len 32768
011/300 Write block 3014 len 20480 data: 3 3 4 4 3
012/300 Read block 4054 len 8192
013/300 Read block 2181 len 36864
014/300 Write block 3152 len 20480 data: 3 3 3 2 2
015/300 Read block 1836 len 32768
016/300 Write block 291 len 36864 data: 2 2 2 2 2 2 2 2 2
017/300 Write block 2794 len 28672 data: 2 2 2 2 2 2 2
018/300 Read block 2003 len 8192
019/300 Write block 2302 len 28672 data: 2 2 2 2 2 2 2
020/300 Write block 1729 len 20480 data: 2 2 2 2 2
021/300 Write block 4930 len 36864 data: 2 2 2 2 2 2 2 2 2
022/300 Flush
023/300 Read block 3784 len 36864
024/300 Write block 4623 len 4096 data: 2
025/300 Write block 4454 len 36864 data: 2 2 2 2 2 2 2 2 2
026/300 Read block 1018 len 20480
027/300 Write block 4907 len 16384 data: 2 2 2 2
028/300 Read block 3010 len 20480
029/300 Read block 4005 len 20480
030/300 Flush
031/300 Read block 56 len 4096
032/300 Flush
033/300 Read block 2827 len 28672
034/300 Write block 3680 len 40960 data: 2 2 2 3 3 3 3 3 4 4
035/300 Write block 1139 len 4096 data: 2
036/300 Write block 2915 len 36864 data: 2 2 2 2 2 2 2 2 2
037/300 Read block 534 len 4096
038/300 Read block 2641 len 32768
039/300 Read block 1942 len 12288
040/300 Flush
041/300 Write block 4397 len 20480 data: 2 2 2 2 2
042/300 Read block 3040 len 4096
043/300 Read block 4271 len 4096
044/300 Read block 2898 len 16384
045/300 Flush
046/300 Read block 4769 len 24576
047/300 Flush
048/300 Read block 2274 len 4096
049/300 Write block 3227 len 20480 data: 4 4 3 2 2
050/300 Read block 2831 len 28672
051/300 Write block 3399 len 32768 data: 2 3 3 4 4 4 4 4
052/300 Read block 2456 len 32768
053/300 Flush
054/300 Flush
055/300 Write block 893 len 12288 data: 2 2 2
056/300 Read block 237 len 24576
057/300 Write block 552 len 4096 data: 2
058/300 Read block 3518 len 28672
059/300 Read block 1323 len 4096
060/300 Write block 3309 len 20480 data: 4 4 4 3 4
061/300 Read block 2012 len 36864
062/300 Read block 3611 len 20480
063/300 Write block 2534 len 40960 data: 3 3 2 2 2 2 2 2 2 2
064/300 Write block 826 len 16384 data: 3 3 3 3
065/300 Write block 1260 len 32768 data: 2 2 2 3 2 2 2 2
066/300 Flush
067/300 Read block 585 len 32768
068/300 Read block 2655 len 36864
069/300 Read block 2463 len 36864
070/300 Read block 129 len 32768
071/300 Flush
072/300 Write block 1733 len 24576 data: 3 2 2 2 2 2
073/300 Read block 1581 len 28672
074/300 Read block 3197 len 8192
075/300 Read block 654 len 20480
076/300 Write block 249 len 28672 data: 3 3 3 2 2 2 2
077/300 Write block 66 len 24576 data: 4 3 3 3 3 3
078/300 Write block 3093 len 4096 data: 2
079/300 Write block 1612 len 32768 data: 3 3 2 2 2 2 3 3
080/300 Read block 1603 len 36864
081/300 Read block 2561 len 24576
082/300 Read block 4371 len 24576
083/300 Read block 1011 len 36864
084/300 Write block 3247 len 16384 data: 3 3 3 3
085/300 Write block 91 len 24576 data: 3 3 3 3 3 3
086/300 Write block 2593 len 32768 data: 3 3 3 3 2 2 2 2
087/300 Read block 4850 len 8192
088/300 Read block 3876 len 8192
089/300 Read block 3062 len 36864
090/300 Read block 2230 len 8192
091/300 Write block 4796 len 20480 data: 4 4 4 4 3
092/300 Read block 1667 len 24576
093/300 Read block 1896 len 24576
094/300 Read block 3076 len 32768
095/300 Write block 3192 len 36864 data: 2 2 2 2 2 3 3 3 2
096/300 Read block 3540 len 20480
097/300 Read block 4626 len 24576
098/300 Flush
099/300 Read block 4303 len 12288
100/300 Write block 3611 len 28672 data: 3 3 3 3 3 3 3
101/300 Write block 165 len 36864 data: 6 4 4 4 4 4 4 3 3
102/300 Read block 2621 len 40960
103/300 Read block 4688 len 32768
104/300 Read block 4313 len 40960
105/300 Flush
106/300 Write block 991 len 8192 data: 4 5
107/300 Write block 4379 len 8192 data: 4 4
108/300 Read block 1848 len 32768
109/300 Read block 3407 len 40960
110/300 Read block 1158 len 12288
111/300 Flush
112/300 Read block 1708 len 4096
113/300 Write block 4728 len 32768 data: 2 3 3 3 3 3 3 3
114/300 Write block 1698 len 40960 data: 2 2 2 2 2 2 2 2 2 2
115/300 Write block 3673 len 8192 data: 4 4
116/300 Write block 1752 len 8192 data: 2 2
117/300 Read block 2769 len 28672
118/300 Read block 1818 len 36864
119/300 Read block 1258 len 36864
120/300 Write block 2800 len 36864 data: 3 2 2 2 2 2 2 2 2
121/300 Read block 4424 len 24576
122/300 Read block 4223 len 16384
123/300 Read block 4199 len 16384
124/300 Read block 875 len 16384
125/300 Write block 1886 len 28672 data: 2 2 2 2 2 2 2
126/300 Write block 1306 len 16384 data: 3 3 3 3
127/300 Read block 1278 len 36864
128/300 Write block 3420 len 16384 data: 2 2 2 3
129/300 Read block 3312 len 12288
130/300 Read block 2307 len 12288
131/300 Write block 3587 len 12288 data: 4 4 4
132/300 Read block 1649 len 8192
133/300 Write block 2965 len 8192 data: 2 2
134/300 Write block 604 len 16384 data: 2 2 2 2
135/300 Write block 2374 len 4096 data: 3
136/300 Write block 3257 len 32768 data: 2 3 2 2 2 2 2 2
137/300 Read block 1613 len 4096
138/300 Write block 4607 len 28672 data: 2 2 2 2 2 2 2
139/300 Write block 382 len 20480 data: 3 3 3 3 3
140/300 Read block 3009 len 28672
141/300 Read block 3692 len 12288
142/300 Read block 3984 len 24576
143/300 Read block 3136 len 4096
144/300 Read block 816 len 16384
145/300 Write block 2952 len 36864 data: 2 2 2 2 2 2 2 3 3
146/300 Flush
147/300 Write block 721 len 24576 data: 5 3 3 3 3 3
148/300 Read block 3584 len 24576
149/300 Read block 155 len 16384
150/300 Write block 884 len 20480 data: 2 2 2 2 2
151/300 Write block 1952 len 8192 data: 3 3
152/300 Read block 1600 len 40960
153/300 Flush
154/300 Read block 179 len 32768
155/300 Read block 1324 len 40960
156/300 Read block 2851 len 20480
157/300 Write block 1038 len 32768 data: 2 2 2 2 2 2 2 2
158/300 Write block 3283 len 28672 data: 4 4 4 4 4 3 3
159/300 Write block 4691 len 24576 data: 2 2 2 2 2 2
160/300 Read block 3631 len 36864
161/300 Read block 3864 len 20480
162/300 Read block 553 len 40960
163/300 Read block 4562 len 4096
164/300 Read block 3567 len 36864
165/300 Write block 977 len 8192 data: 2 2
166/300 Flush
167/300 Write block 3073 len 8192 data: 2 2
168/300 Read block 133 len 40960
169/300 Write block 4357 len 4096 data: 2
170/300 Write block 3385 len 20480 data: 4 3 4 4 4
171/300 Write block 1306 len 8192 data: 4 4
172/300 Read block 1614 len 4096
173/300 Write block 1144 len 8192 data: 2 2
174/300 Read block 2999 len 40960
175/300 Write block 3455 len 4096 data: 2
176/300 Write block 1755 len 8192 data: 2 2
177/300 Read block 2796 len 4096
178/300 Read block 1740 len 24576
179/300 Read block 1467 len 20480
180/300 Write block 1886 len 8192 data: 3 3
181/300 Write block 4182 len 20480 data: 3 3 3 3 3
182/300 Read block 4066 len 8192
183/300 Write block 792 len 32768 data: 2 2 2 2 2 3 3 3
184/300 Flush
185/300 Write block 680 len 40960 data: 2 2 2 4 4 4 4 4 4 4
186/300 Read block 3761 len 40960
187/300 Write block 357 len 36864 data: 3 3 2 2 2 2 2 2 2
188/300 Read block 3488 len 8192
189/300 Flush
190/300 Flush
191/300 Read block 3864 len 16384
192/300 Read block 2722 len 12288
193/300 Read block 3459 len 36864
194/300 Write block 732 len 40960 data: 4 4 4 3 3 3 3 4 4 4
195/300 Write block 2353 len 24576 data: 4 4 4 4 3 3
196/300 Write block 3807 len 24576 data: 3 3 3 3 2 2
197/300 Read block 4990 len 24576
198/300 Read block 4418 len 12288
199/300 Write block 3296 len 28672 data: 2 2 2 2 2 2 2
200/300 Write block 39 len 32768 data: 3 3 3 3 3 3 4 4
201/300 Write block 3662 len 36864 data: 2 2 2 3 3 3 3 4 5
202/300 Read block 2224 len 40960
203/300 Read block 2758 len 32768
204/300 Flush
205/300 Write block 4287 len 24576 data: 5 5 5 4 4 3
206/300 Write block 2852 len 4096 data: 2
207/300 Read block 3301 len 36864
208/300 Write block 3192 len 40960 data: 3 3 3 3 3 4 4 4 3 3
209/300 Write block 3712 len 4096 data: 2
210/300 Write block 4572 len 32768 data: 3 3 3 3 3 3 3 3
211/300 Read block 4715 len 28672
212/300 Write block 2338 len 28672 data: 3 3 3 3 3 3 3
213/300 Read block 3901 len 36864
214/300 Write block 2583 len 28672 data: 2 2 2 3 4 4 4
215/300 Write block 3355 len 40960 data: 3 3 3 3 2 2 2 2 2 2
216/300 Write block 3904 len 16384 data: 3 2 2 2
217/300 Read block 4969 len 24576
218/300 Write block 4513 len 4096 data: 2
219/300 Read block 2907 len 20480
220/300 Read block 229 len 40960
221/300 Read block 3473 len 8192
222/300 Flush
223/300 Write block 1300 len 20480 data: 3 3 3 2 2
224/300 Read block 1012 len 28672
225/300 Read block 4369 len 20480
226/300 Write block 1590 len 32768 data: 3 3 2 2 3 3 4 4
227/300 Read block 3787 len 8192
228/300 Read block 4528 len 24576
229/300 Read block 620 len 24576
230/300 Read block 2224 len 12288
231/300 Read block 3153 len 8192
232/300 Read block 4059 len 16384
233/300 Read block 3357 len 20480
234/300 Read block 1073 len 24576
235/300 Read block 4183 len 32768
236/300 Read block 2696 len 4096
237/300 Flush
238/300 Read block 2062 len 36864
239/300 Write block 461 len 28672 data: 3 3 2 2 3 3 3
240/300 Write block 1391 len 4096 data: 4
241/300 Read block 2981 len 24576
242/300 Write block 3264 len 8192 data: 3 2
243/300 Write block 4486 len 4096 data: 3
244/300 Write block 1130 len 16384 data: 2 2 2 2
245/300 Read block 365 len 16384
246/300 Write block 1925 len 40960 data: 2 2 2 2 2 3 3 3 3 2
247/300 Write block 2836 len 28672 data: 4 4 4 3 3 2 2
248/300 Read block 3338 len 8192
249/300 Read block 4141 len 40960
250/300 Read block 1635 len 8192
251/300 Read block 1512 len 40960
252/300 Flush
253/300 Write block 2005 len 40960 data: 3 2 2 2 3 2 2 2 2 2
254/300 Read block 709 len 32768
255/300 Read block 78 len 12288
256/300 Read block 3333 len 36864
257/300 Read block 4077 len 12288
258/300 Write block 2298 len 28672 data: 2 2 2 2 3 3 3
259/300 Read block 1025 len 12288
260/300 Flush
261/300 Write block 1628 len 32768 data: 2 2 2 2 2 2 2 2
262/300 Read block 4631 len 16384
263/300 Read block 2996 len 28672
264/300 Write block 3076 len 32768 data: 3 3 3 2 2 2 2 2
265/300 Read block 1857 len 28672
266/300 Write block 243 len 16384 data: 4 5 5 4
267/300 Read block 1138 len 20480
268/300 Write block 3752 len 12288 data: 2 2 2
269/300 Read block 2282 len 40960
270/300 Write block 4624 len 20480 data: 2 2 2 2 2
271/300 Read block 1052 len 12288
272/300 Read block 3438 len 28672
273/300 Write block 1964 len 24576 data: 3 3 3 3 3 2
274/300 Read block 4641 len 12288
275/300 Read block 4268 len 24576
276/300 Read block 2038 len 40960
277/300 Read block 2517 len 12288
278/300 Read block 357 len 4096
279/300 Write block 2170 len 8192 data: 4 3
280/300 Read block 3739 len 16384
281/300 Write block 163 len 28672 data: 6 7 7 5 5 5 5
282/300 Write block 2011 len 40960 data: 3 3 3 3 2 2 3 3 3 4
283/300 Write block 1134 len 32768 data: 2 3 3 2 2 3 2 2
284/300 Read block 4588 len 8192
285/300 Read block 4465 len 4096
286/300 Read block 2038 len 36864
287/300 Read block 4883 len 20480
288/300 Write block 2734 len 24576 data: 2 2 2 2 2 2
289/300 Flush
290/300 Read block 3220 len 40960
291/300 Read block 903 len 8192
292/300 Write block 2517 len 32768 data: 2 2 2 2 2 2 2 2
293/300 Read block 964 len 4096
294/300 Read block 3189 len 40960
295/300 Read block 1064 len 24576
296/300 Read block 4702 len 28672
297/300 Write block 255 len 16384 data: 3 2 2 2
298/300 Read block 2716 len 32768
299/300 Write block 1761 len 16384 data: 4 3 3 3
300/300 Write block 3682 len 8192 data: 3 4
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 13 21:40:43.138 INFO request completed, latency_us: 250, response_code: 204, uri: /start/cid/0, method: POST, req_id: 4a856997-d638-4e6f-8810-0ce98eb3e1e0, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
1508 Acked 2507 Read 9 New Done Done true
1509 Acked 2508 Read 1 New Done Done false
1510 Acked 2509 Write 8 New Done Done false
1511 Acked 2510 Read 10 New Done Done false
1512 Acked 2511 Read 4 New Done Done false
1513 Acked 2512 Write 9 New Done Done false
1514 Acked 2513 Read 7 New Done Done false
1515 Acked 2514 Write 6 New Done Done false
1516 Acked 2515 Read 1 New Done Done false
1517 Acked 2516 Read 8 New Done Done false
1518 Acked 2517 Write 5 New Done Done false
1519 Acked 2518 Read 2 New Done Done false
1520 Acked 2519 Read 9 New Done Done false
1521 Acked 2520 Write 5 New Done Done false
1522 Acked 2521 Read 8 New Done Done false
1523 Acked 2522 Write 9 New Done Done false
1524 Acked 2523 Write 7 New Done Done false
1525 Acked 2524 Read 2 New Done Done false
1526 Acked 2525 Write 7 New Done Done false
1527 Acked 2526 Write 5 New Done Done false
1528 Acked 2527 Write 9 New Done Done false
1529 Acked 2528 Flush 0 New Done Done false
1530 Acked 2529 Read 9 New Done Done false
1531 Acked 2530 Write 1 New Done Done false
1532 Acked 2531 Write 9 New Done Done false
1533 Acked 2532 Read 5 New Done Done false
1534 Acked 2533 Write 4 New Done Done false
1535 Acked 2534 Read 5 New Done Done false
1536 Acked 2535 Read 5 New Done Done false
1537 Acked 2536 Flush 0 New Done Done false
1538 Acked 2537 Read 1 New Done Done false
1539 Acked 2538 Flush 0 New Done Done false
1540 Acked 2539 Read 7 New Done Done false
1541 Acked 2540 Write 10 New Done Done false
1542 Acked 2541 Write 1 New Done Done false
1543 Acked 2542 Write 9 New Done Done false
1544 Acked 2543 Read 1 New Done Done false
1545 Acked 2544 Read 8 New Done Done false
1546 Acked 2545 Read 3 New Done Done false
1547 Acked 2546 Flush 0 New Done Done false
1548 Acked 2547 Write 5 New Done Done false
1549 Acked 2548 Read 1 New Done Done false
1550 Acked 2549 Read 1 New Done Done false
1551 Acked 2550 Read 4 New Done Done false
1552 Acked 2551 Flush 0 New Done Done false
1553 Acked 2552 Read 6 New Done Done false
1554 Acked 2553 Flush 0 New Done Done false
1555 Acked 2554 Read 1 New Done Done false
1556 Acked 2555 Write 5 New Done Done false
1557 Acked 2556 Read 7 New Done Done false
1558 Acked 2557 Write 8 New Done Done false
1559 Acked 2558 Read 8 New Done Done false
1560 Acked 2559 Flush 0 New Done Done false
1561 Acked 2560 Flush 0 New Done Done false
1562 Acked 2561 Write 3 New Done Done false
1563 Acked 2562 Read 6 New Done Done false
1564 Acked 2563 Write 1 New Done Done false
1565 Acked 2564 Read 7 New Done Done false
1566 Acked 2565 Read 1 New Done Done false
1567 Acked 2566 Write 5 New Done Done false
1568 Acked 2567 Read 9 New Done Done false
1569 Acked 2568 Read 5 New Done Done false
1570 Acked 2569 Write 10 New Done Done false
1571 Acked 2570 Write 4 New Done Done false
1572 Acked 2571 Write 8 New Done Done false
1573 Acked 2572 Flush 0 New Done Done false
1574 Acked 2573 Read 8 New Done Done false
1575 Acked 2574 Read 9 New Done Done false
1576 Acked 2575 Read 9 New Done Done false
1577 Acked 2576 Read 8 New Done Done false
1578 Acked 2577 Flush 0 New Done Done false
1579 Acked 2578 Write 6 New Done Done false
1580 Acked 2579 Read 7 New Done Done false
1581 Acked 2580 Read 2 New Done Done false
1582 Acked 2581 Read 5 New Done Done false
1583 Acked 2582 Write 7 New Done Done false
1584 Acked 2583 Write 6 New Done Done false
1585 Acked 2584 Write 1 New Done Done false
1586 Acked 2585 Write 8 New Done Done false
1587 Acked 2586 Read 9 New Done Done false
1588 Acked 2587 Read 6 New Done Done false
1589 Acked 2588 Read 6 New Done Done false
1590 Acked 2589 Read 9 New Done Done false
1591 Acked 2590 Write 4 New Done Done false
1592 Acked 2591 Write 6 New Done Done false
1593 Acked 2592 Write 8 New Done Done false
1594 Acked 2593 Read 2 New Done Done false
1595 Acked 2594 Read 2 New Done Done false
1596 Acked 2595 Read 9 New Done Done false
1597 Acked 2596 Read 2 New Done Done false
1598 Acked 2597 Write 5 New Done Done false
1599 Acked 2598 Read 6 New Done Done false
1600 Acked 2599 Read 6 New Done Done false
1601 Acked 2600 Read 8 New Done Done false
1602 Acked 2601 Write 9 New Done Done false
1603 Acked 2602 Read 5 New Done Done false
1604 Acked 2603 Read 6 New Done Done false
1605 Acked 2604 Flush 0 New Done Done false
1606 Acked 2605 Read 3 New Done Done false
1607 Acked 2606 Write 7 New Done Done false
1608 Acked 2607 Write 9 New Done Done false
1609 Acked 2608 Read 10 New Done Done false
1610 Acked 2609 Read 8 New Done Done false
1611 Acked 2610 Read 10 New Done Done false
1612 Acked 2611 Flush 0 New Done Done false
1613 Acked 2612 Write 2 New Done Done false
1614 Acked 2613 Write 2 New Done Done false
1615 Acked 2614 Read 8 New Done Done false
1616 Acked 2615 Read 10 New Done Done false
1617 Acked 2616 Read 3 New Done Done false
1618 Acked 2617 Flush 0 New Done Done false
1619 Acked 2618 Read 1 New Done Done false
1620 Acked 2619 Write 8 New Done Done false
1621 Acked 2620 Write 10 New Done Done false
1622 Acked 2621 Write 2 New Done Done false
1623 Acked 2622 Write 2 New Done Done false
1624 Acked 2623 Read 7 New Done Done false
1625 Acked 2624 Read 9 New Done Done false
1626 Acked 2625 Read 9 New Done Done false
1627 Acked 2626 Write 9 New Done Done false
1628 Acked 2627 Read 6 New Done Done false
1629 Acked 2628 Read 4 New Done Done false
1630 Acked 2629 Read 4 New Done Done false
1631 Acked 2630 Read 4 New Done Done false
1632 Acked 2631 Write 7 New Done Done false
1633 Acked 2632 Write 4 New Done Done false
1634 Acked 2633 Read 9 New Done Done false
1635 Acked 2634 Write 4 New Done Done false
1636 Acked 2635 Read 3 New Done Done false
1637 Acked 2636 Read 3 New Done Done false
1638 Acked 2637 Write 3 New Done Done false
1639 Acked 2638 Read 2 New Done Done false
1640 Acked 2639 Write 2 New Done Done false
1641 Acked 2640 Write 4 New Done Done false
1642 Acked 2641 Write 1 New Done Done false
1643 Acked 2642 Write 8 New Done Done false
1644 Acked 2643 Read 1 New Done Done false
1645 Acked 2644 Write 7 New Done Done false
1646 Acked 2645 Write 5 New Done Done false
1647 Acked 2646 Read 7 New Done Done false
1648 Acked 2647 Read 3 New Done Done false
1649 Acked 2648 Read 6 New Done Done false
1650 Acked 2649 Read 1 New Done Done false
1651 Acked 2650 Read 4 New Done Done false
1652 Acked 2651 Write 9 New Done Done false
1653 Acked 2652 Flush 0 New Done Done false
1654 Acked 2653 Write 6 New Done Done false
1655 Acked 2654 Read 6 New Done Done false
1656 Acked 2655 Read 4 New Done Done false
1657 Acked 2656 Write 5 New Done Done false
1658 Acked 2657 Write 2 New Done Done false
1659 Acked 2658 Read 10 New Done Done false
1660 Acked 2659 Flush 0 New Done Done false
1661 Acked 2660 Read 8 New Done Done false
1662 Acked 2661 Read 10 New Done Done false
1663 Acked 2662 Read 5 New Done Done false
1664 Acked 2663 Write 8 New Done Done false
1665 Acked 2664 Write 7 New Done Done false
1666 Acked 2665 Write 6 New Done Done false
1667 Acked 2666 Read 9 New Done Done false
1668 Acked 2667 Read 5 New Done Done false
1669 Acked 2668 Read 10 New Done Done false
1670 Acked 2669 Read 1 New Done Done false
1671 Acked 2670 Read 9 New Done Done false
1672 Acked 2671 Write 2 New Done Done false
1673 Acked 2672 Flush 0 New Done Done false
1674 Acked 2673 Write 2 New Done Done false
1675 Acked 2674 Read 10 New Done Done false
1676 Acked 2675 Write 1 New Done Done false
1677 Acked 2676 Write 5 New Done Done false
1678 Acked 2677 Write 2 New Done Done false
1679 Acked 2678 Read 1 New Done Done false
1680 Acked 2679 Write 2 New Done Done false
1681 Acked 2680 Read 10 New Done Done false
1682 Acked 2681 Write 1 New Done Done false
1683 Acked 2682 Write 2 New Done Done false
1684 Acked 2683 Read 1 New Done Done false
1685 Acked 2684 Read 6 New Done Done false
1686 Acked 2685 Read 5 New Done Done false
1687 Acked 2686 Write 2 New Done Done false
1688 Acked 2687 Write 5 New Done Done false
1689 Acked 2688 Read 2 New Done Done false
1690 Acked 2689 Write 8 New Done Done false
1691 Acked 2690 Flush 0 New Done Done false
1692 Acked 2691 Write 10 New Done Done false
1693 Acked 2692 Read 10 New Done Done false
1694 Acked 2693 Write 9 New Done Done false
1695 Acked 2694 Read 2 New Done Done false
1696 Acked 2695 Flush 0 New Done Done false
1697 Acked 2696 Flush 0 New Done Done false
1698 Acked 2697 Read 4 New Done Done false
1699 Acked 2698 Read 3 New Done Done false
1700 Acked 2699 Read 9 New Done Done false
1701 Acked 2700 Write 10 New Done Done false
1702 Acked 2701 Write 6 New Done Done false
1703 Acked 2702 Write 6 New Done Done false
1704 Acked 2703 Read 6 New Done Done false
1705 Acked 2704 Read 3 New Done Done false
1706 Acked 2705 Write 7 New Done Done false
1707 Acked 2706 Write 8 New Done Done false
1708 Acked 2707 Write 9 New Done Done false
1709 Acked 2708 Read 10 New Done Done false
1710 Acked 2709 Read 8 New Done Done false
1711 Acked 2710 Flush 0 New Done Done false
1712 Acked 2711 Write 6 New Done Done false
1713 Acked 2712 Write 1 New Done Done false
1714 Acked 2713 Read 9 New Done Done false
1715 Acked 2714 Write 10 New Done Done false
1716 Acked 2715 Write 1 New Done Done false
1717 Acked 2716 Write 8 New Done Done false
1718 Acked 2717 Read 7 New Done Done false
1719 Acked 2718 Write 7 New Done Done false
1720 Acked 2719 Read 9 New Done Done false
1721 Acked 2720 Write 7 New Done Done false
1722 Acked 2721 Write 10 New Done Done false
1723 Acked 2722 Write 4 New Done Done false
1724 Acked 2723 Read 6 New Done Done false
1725 Acked 2724 Write 1 New Done Done false
1726 Acked 2725 Read 5 New Done Done false
1727 Acked 2726 Read 10 New Done Done false
1728 Acked 2727 Read 2 New Done Done false
1729 Acked 2728 Flush 0 New Done Done false
1730 Acked 2729 Write 5 New Done Done false
1731 Acked 2730 Read 7 New Done Done false
1732 Acked 2731 Read 5 New Done Done false
1733 Acked 2732 Write 8 New Done Done false
1734 Acked 2733 Read 2 New Done Done false
1735 Acked 2734 Read 6 New Done Done false
1736 Acked 2735 Read 6 New Done Done false
1737 Acked 2736 Read 3 New Done Done false
1738 Acked 2737 Read 2 New Done Done false
1739 Acked 2738 Read 4 New Done Done false
1740 Acked 2739 Read 5 New Done Done false
1741 Acked 2740 Read 6 New Done Done false
1742 Acked 2741 Read 8 New Done Done false
1743 Acked 2742 Read 1 New Done Done false
1744 Acked 2743 Flush 0 New Done Done false
1745 Acked 2744 Read 9 New Done Done false
1746 Acked 2745 Write 7 New Done Done false
1747 Acked 2746 Write 1 New Done Done false
1748 Acked 2747 Read 6 New Done Done false
1749 Acked 2748 Write 2 New Done Done false
1750 Acked 2749 Write 1 New Done Done false
1751 Acked 2750 Write 4 New Done Done false
1752 Acked 2751 Read 4 New Done Done false
1753 Acked 2752 Write 10 New Done Done false
1754 Acked 2753 Write 7 New Done Done false
1755 Acked 2754 Read 2 New Done Done false
1756 Acked 2755 Read 10 New Done Done false
1757 Acked 2756 Read 2 New Done Done false
1758 Acked 2757 Read 10 New Done Done false
1759 Acked 2758 Flush 0 New Done Done false
1760 Acked 2759 Write 10 New Done Done false
1761 Acked 2760 Read 8 New Done Done false
1762 Acked 2761 Read 3 New Done Done false
1763 Acked 2762 Read 9 New Done Done false
1764 Acked 2763 Read 3 New Done Done false
1765 Acked 2764 Write 7 New Done Done false
1766 Acked 2765 Read 3 New Done Done false
1767 Acked 2766 Flush 0 New Done Done false
1768 Acked 2767 Write 8 New Done Done false
1769 Acked 2768 Read 4 New Done Done false
1770 Acked 2769 Read 7 New Done Done false
1771 Acked 2770 Write 8 New Done Done false
1772 Acked 2771 Read 7 New Done Done false
1773 Acked 2772 Write 4 New Done Done false
1774 Acked 2773 Read 5 New Done Done false
1775 Acked 2774 Write 3 New Done Done false
1776 Acked 2775 Read 10 New Done Done false
1777 Acked 2776 Write 5 New Done Done false
1778 Acked 2777 Read 3 New Done Done false
1779 Acked 2778 Read 7 New Done Done false
1780 Acked 2779 Write 6 New Done Done false
1781 Acked 2780 Read 3 New Done Done false
1782 Acked 2781 Read 6 New Done Done false
1783 Acked 2782 Read 10 New Done Done false
1784 Acked 2783 Read 3 New Done Done false
1785 Acked 2784 Read 1 New Done Done false
1786 Acked 2785 Write 2 New Done Done false
1787 Acked 2786 Read 4 New Done Done false
1788 Acked 2787 Write 7 New Done Done false
1789 Acked 2788 Write 10 New Done Done false
1790 Acked 2789 Write 8 New Done Done false
1791 Acked 2790 Read 2 New Done Done false
1792 Acked 2791 Read 1 New Done Done false
1793 Acked 2792 Read 9 New Done Done false
1794 Acked 2793 Read 5 New Done Done false
1795 Acked 2794 Write 6 New Done Done false
1796 Acked 2795 Flush 0 New Done Done false
1797 Acked 2796 Read 10 New Done Done false
1798 Acked 2797 Read 2 New Done Done false
1799 Acked 2798 Write 8 New Done Done false
1800 Acked 2799 Read 1 New Done Done false
1801 Acked 2800 Read 10 New Done Done false
1802 Acked 2801 Read 6 New Done Done false
1803 Acked 2802 Read 7 New Done Done false
1804 Acked 2803 Write 4 New Done Done false
1805 Acked 2804 Read 8 New Done Done false
1806 Acked 2805 Write 4 New Done Done false
1807 Acked 2806 Write 2 New Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
New 300 0 0 300
Sent 0 0 0 0
Done 0 300 300 600
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 2506 2795 2795
Downstairs last five completed: 2506 2505 2504 2503 2502
Upstairs last five completed: 1807 1806 1805 1804 1803
CLIENT: Up:0 ds:300 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:43.373024895Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:43.373103772Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:43.373127501Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:43.37314861Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:43.37316478Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 300 jobs since flush 2506","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:43.373182609Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:43.373313744Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:44.373981386Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:44.374044384Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:44.374067823Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:44.374088692Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:44.374112691Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 2506","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:44.374130721Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:44.374276255Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:45.375088942Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:45.37514617Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:45.375168759Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:45.375189638Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:45.375206758Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 2506","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:45.375225037Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:45.375322423Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:46.375841333Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:46.375890081Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:46.37591165Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:46.375939719Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:46.375955858Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 2506","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:46.375973068Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:46.376114422Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1170)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
1508 Acked 2507 Read 9 New Done Done true
1509 Acked 2508 Read 1 New Done Done true
1510 Acked 2509 Write 8 New Done Done true
1511 Acked 2510 Read 10 New Done Done true
1512 Acked 2511 Read 4 New Done Done true
1513 Acked 2512 Write 9 New Done Done true
1514 Acked 2513 Read 7 New Done Done true
1515 Acked 2514 Write 6 New Done Done true
1516 Acked 2515 Read 1 New Done Done true
1517 Acked 2516 Read 8 New Done Done true
1518 Acked 2517 Write 5 New Done Done true
1519 Acked 2518 Read 2 New Done Done true
1520 Acked 2519 Read 9 New Done Done true
1521 Acked 2520 Write 5 New Done Done true
1522 Acked 2521 Read 8 New Done Done true
1523 Acked 2522 Write 9 New Done Done true
1524 Acked 2523 Write 7 New Done Done true
1525 Acked 2524 Read 2 New Done Done true
1526 Acked 2525 Write 7 New Done Done true
1527 Acked 2526 Write 5 New Done Done true
1528 Acked 2527 Write 9 New Done Done true
1529 Acked 2528 Flush 0 New Done Done true
1530 Acked 2529 Read 9 New Done Done true
1531 Acked 2530 Write 1 New Done Done true
1532 Acked 2531 Write 9 New Done Done true
1533 Acked 2532 Read 5 New Done Done true
1534 Acked 2533 Write 4 New Done Done true
1535 Acked 2534 Read 5 New Done Done true
1536 Acked 2535 Read 5 New Done Done true
1537 Acked 2536 Flush 0 New Done Done true
1538 Acked 2537 Read 1 New Done Done true
1539 Acked 2538 Flush 0 New Done Done true
1540 Acked 2539 Read 7 New Done Done true
1541 Acked 2540 Write 10 New Done Done true
1542 Acked 2541 Write 1 New Done Done true
1543 Acked 2542 Write 9 New Done Done true
1544 Acked 2543 Read 1 New Done Done true
1545 Acked 2544 Read 8 New Done Done true
1546 Acked 2545 Read 3 New Done Done true
1547 Acked 2546 Flush 0 New Done Done true
1548 Acked 2547 Write 5 New Done Done true
1549 Acked 2548 Read 1 New Done Done true
1550 Acked 2549 Read 1 New Done Done true
1551 Acked 2550 Read 4 New Done Done true
1552 Acked 2551 Flush 0 New Done Done true
1553 Acked 2552 Read 6 New Done Done true
1554 Acked 2553 Flush 0 New Done Done true
1555 Acked 2554 Read 1 New Done Done true
1556 Acked 2555 Write 5 New Done Done true
1557 Acked 2556 Read 7 New Done Done true
1558 Acked 2557 Write 8 New Done Done true
1559 Acked 2558 Read 8 New Done Done true
1560 Acked 2559 Flush 0 New Done Done true
1561 Acked 2560 Flush 0 New Done Done true
1562 Acked 2561 Write 3 New Done Done true
1563 Acked 2562 Read 6 New Done Done true
1564 Acked 2563 Write 1 New Done Done true
1565 Acked 2564 Read 7 New Done Done true
1566 Acked 2565 Read 1 New Done Done true
1567 Acked 2566 Write 5 New Done Done true
1568 Acked 2567 Read 9 New Done Done true
1569 Acked 2568 Read 5 New Done Done true
1570 Acked 2569 Write 10 New Done Done true
1571 Acked 2570 Write 4 New Done Done true
1572 Acked 2571 Write 8 New Done Done true
1573 Acked 2572 Flush 0 New Done Done true
1574 Acked 2573 Read 8 New Done Done true
1575 Acked 2574 Read 9 New Done Done true
1576 Acked 2575 Read 9 New Done Done true
1577 Acked 2576 Read 8 New Done Done true
1578 Acked 2577 Flush 0 New Done Done true
1579 Acked 2578 Write 6 New Done Done true
1580 Acked 2579 Read 7 New Done Done true
1581 Acked 2580 Read 2 New Done Done true
1582 Acked 2581 Read 5 New Done Done true
1583 Acked 2582 Write 7 New Done Done true
1584 Acked 2583 Write 6 New Done Done true
1585 Acked 2584 Write 1 New Done Done true
1586 Acked 2585 Write 8 New Done Done true
1587 Acked 2586 Read 9 New Done Done true
1588 Acked 2587 Read 6 New Done Done true
1589 Acked 2588 Read 6 New Done Done true
1590 Acked 2589 Read 9 New Done Done true
1591 Acked 2590 Write 4 New Done Done true
1592 Acked 2591 Write 6 New Done Done true
1593 Acked 2592 Write 8 New Done Done true
1594 Acked 2593 Read 2 New Done Done true
1595 Acked 2594 Read 2 New Done Done true
1596 Acked 2595 Read 9 New Done Done true
1597 Acked 2596 Read 2 New Done Done true
1598 Acked 2597 Write 5 New Done Done true
1599 Acked 2598 Read 6 New Done Done true
1600 Acked 2599 Read 6 New Done Done true
1601 Acked 2600 Read 8 New Done Done true
1602 Acked 2601 Write 9 New Done Done true
1603 Acked 2602 Read 5 New Done Done true
1604 Acked 2603 Read 6 New Done Done true
1605 Acked 2604 Flush 0 New Done Done true
1606 Acked 2605 Read 3 New Done Done true
1607 Acked 2606 Write 7 New Done Done true
1608 Acked 2607 Write 9 New Done Done true
1609 Acked 2608 Read 10 New Done Done true
1610 Acked 2609 Read 8 New Done Done true
1611 Acked 2610 Read 10 New Done Done true
1612 Acked 2611 Flush 0 New Done Done true
1613 Acked 2612 Write 2 New Done Done true
1614 Acked 2613 Write 2 New Done Done true
1615 Acked 2614 Read 8 New Done Done true
1616 Acked 2615 Read 10 New Done Done true
1617 Acked 2616 Read 3 New Done Done true
1618 Acked 2617 Flush 0 New Done Done true
1619 Acked 2618 Read 1 New Done Done true
1620 Acked 2619 Write 8 New Done Done true
1621 Acked 2620 Write 10 New Done Done true
1622 Acked 2621 Write 2 New Done Done true
1623 Acked 2622 Write 2 New Done Done true
1624 Acked 2623 Read 7 New Done Done true
1625 Acked 2624 Read 9 New Done Done true
1626 Acked 2625 Read 9 New Done Done true
1627 Acked 2626 Write 9 New Done Done true
1628 Acked 2627 Read 6 New Done Done true
1629 Acked 2628 Read 4 New Done Done true
1630 Acked 2629 Read 4 New Done Done true
1631 Acked 2630 Read 4 New Done Done true
1632 Acked 2631 Write 7 New Done Done true
1633 Acked 2632 Write 4 New Done Done true
1634 Acked 2633 Read 9 New Done Done true
1635 Acked 2634 Write 4 New Done Done true
1636 Acked 2635 Read 3 New Done Done true
1637 Acked 2636 Read 3 New Done Done true
1638 Acked 2637 Write 3 New Done Done true
1639 Acked 2638 Read 2 New Done Done true
1640 Acked 2639 Write 2 New Done Done true
1641 Acked 2640 Write 4 New Done Done true
1642 Acked 2641 Write 1 New Done Done true
1643 Acked 2642 Write 8 New Done Done true
1644 Acked 2643 Read 1 New Done Done true
1645 Acked 2644 Write 7 New Done Done true
1646 Acked 2645 Write 5 New Done Done true
1647 Acked 2646 Read 7 New Done Done true
1648 Acked 2647 Read 3 New Done Done true
1649 Acked 2648 Read 6 New Done Done true
1650 Acked 2649 Read 1 New Done Done true
1651 Acked 2650 Read 4 New Done Done true
1652 Acked 2651 Write 9 New Done Done true
1653 Acked 2652 Flush 0 New Done Done true
1654 Acked 2653 Write 6 New Done Done true
1655 Acked 2654 Read 6 New Done Done true
1656 Acked 2655 Read 4 New Done Done true
1657 Acked 2656 Write 5 New Done Done true
1658 Acked 2657 Write 2 New Done Done true
1659 Acked 2658 Read 10 New Done Done true
1660 Acked 2659 Flush 0 New Done Done true
1661 Acked 2660 Read 8 New Done Done true
1662 Acked 2661 Read 10 New Done Done true
1663 Acked 2662 Read 5 New Done Done true
1664 Acked 2663 Write 8 New Done Done true
1665 Acked 2664 Write 7 New Done Done true
1666 Acked 2665 Write 6 New Done Done true
1667 Acked 2666 Read 9 New Done Done true
1668 Acked 2667 Read 5 New Done Done true
1669 Acked 2668 Read 10 New Done Done true
1670 Acked 2669 Read 1 New Done Done true
1671 Acked 2670 Read 9 New Done Done true
1672 Acked 2671 Write 2 New Done Done true
1673 Acked 2672 Flush 0 New Done Done true
1674 Acked 2673 Write 2 New Done Done true
1675 Acked 2674 Read 10 New Done Done true
1676 Acked 2675 Write 1 New Done Done true
1677 Acked 2676 Write 5 New Done Done true
1678 Acked 2677 Write 2 New Done Done true
1679 Acked 2678 Read 1 New Done Done true
1680 Acked 2679 Write 2 New Done Done true
1681 Acked 2680 Read 10 New Done Done true
1682 Acked 2681 Write 1 New Done Done true
1683 Acked 2682 Write 2 New Done Done true
1684 Acked 2683 Read 1 New Done Done true
1685 Acked 2684 Read 6 New Done Done true
1686 Acked 2685 Read 5 New Done Done true
1687 Acked 2686 Write 2 New Done Done true
1688 Acked 2687 Write 5 New Done Done true
1689 Acked 2688 Read 2 New Done Done true
1690 Acked 2689 Write 8 New Done Done true
1691 Acked 2690 Flush 0 New Done Done true
1692 Acked 2691 Write 10 New Done Done true
1693 Acked 2692 Read 10 New Done Done true
1694 Acked 2693 Write 9 New Done Done true
1695 Acked 2694 Read 2 New Done Done true
1696 Acked 2695 Flush 0 New Done Done true
1697 Acked 2696 Flush 0 New Done Done true
1698 Acked 2697 Read 4 New Done Done true
1699 Acked 2698 Read 3 New Done Done true
1700 Acked 2699 Read 9 New Done Done true
1701 Acked 2700 Write 10 New Done Done true
1702 Acked 2701 Write 6 New Done Done true
1703 Acked 2702 Write 6 New Done Done true
1704 Acked 2703 Read 6 New Done Done true
1705 Acked 2704 Read 3 New Done Done true
1706 Acked 2705 Write 7 New Done Done true
1707 Acked 2706 Write 8 New Done Done true
1708 Acked 2707 Write 9 New Done Done true
1709 Acked 2708 Read 10 New Done Done true
1710 Acked 2709 Read 8 New Done Done true
1711 Acked 2710 Flush 0 New Done Done true
1712 Acked 2711 Write 6 New Done Done true
1713 Acked 2712 Write 1 New Done Done true
1714 Acked 2713 Read 9 New Done Done true
1715 Acked 2714 Write 10 New Done Done true
1716 Acked 2715 Write 1 New Done Done true
1717 Acked 2716 Write 8 New Done Done true
1718 Acked 2717 Read 7 New Done Done true
1719 Acked 2718 Write 7 New Done Done true
1720 Acked 2719 Read 9 New Done Done true
1721 Acked 2720 Write 7 New Done Done true
1722 Acked 2721 Write 10 New Done Done true
1723 Acked 2722 Write 4 New Done Done true
1724 Acked 2723 Read 6 New Done Done true
1725 Acked 2724 Write 1 New Done Done true
1726 Acked 2725 Read 5 New Done Done true
1727 Acked 2726 Read 10 New Done Done true
1728 Acked 2727 Read 2 New Done Done true
1729 Acked 2728 Flush 0 New Done Done true
1730 Acked 2729 Write 5 New Done Done true
1731 Acked 2730 Read 7 New Done Done true
1732 Acked 2731 Read 5 New Done Done true
1733 Acked 2732 Write 8 New Done Done true
1734 Acked 2733 Read 2 New Done Done true
1735 Acked 2734 Read 6 New Done Done true
1736 Acked 2735 Read 6 New Done Done true
1737 Acked 2736 Read 3 New Done Done true
1738 Acked 2737 Read 2 New Done Done true
1739 Acked 2738 Read 4 New Done Done true
1740 Acked 2739 Read 5 New Done Done true
1741 Acked 2740 Read 6 New Done Done true
1742 Acked 2741 Read 8 New Done Done true
1743 Acked 2742 Read 1 New Done Done true
1744 Acked 2743 Flush 0 New Done Done true
1745 Acked 2744 Read 9 New Done Done true
1746 Acked 2745 Write 7 New Done Done true
1747 Acked 2746 Write 1 New Done Done true
1748 Acked 2747 Read 6 New Done Done true
1749 Acked 2748 Write 2 New Done Done true
1750 Acked 2749 Write 1 New Done Done true
1751 Acked 2750 Write 4 New Done Done true
1752 Acked 2751 Read 4 New Done Done true
1753 Acked 2752 Write 10 New Done Done true
1754 Acked 2753 Write 7 New Done Done true
1755 Acked 2754 Read 2 New Done Done true
1756 Acked 2755 Read 10 New Done Done true
1757 Acked 2756 Read 2 New Done Done true
1758 Acked 2757 Read 10 New Done Done true
1759 Acked 2758 Flush 0 New Done Done true
1760 Acked 2759 Write 10 New Done Done true
1761 Acked 2760 Read 8 New Done Done true
1762 Acked 2761 Read 3 New Done Done true
1763 Acked 2762 Read 9 New Done Done true
1764 Acked 2763 Read 3 New Done Done true
1765 Acked 2764 Write 7 New Done Done true
1766 Acked 2765 Read 3 New Done Done true
1767 Acked 2766 Flush 0 New Done Done true
1768 Acked 2767 Write 8 New Done Done true
1769 Acked 2768 Read 4 New Done Done true
1770 Acked 2769 Read 7 New Done Done true
1771 Acked 2770 Write 8 New Done Done true
1772 Acked 2771 Read 7 New Done Done true
1773 Acked 2772 Write 4 New Done Done true
1774 Acked 2773 Read 5 New Done Done true
1775 Acked 2774 Write 3 New Done Done true
1776 Acked 2775 Read 10 New Done Done true
1777 Acked 2776 Write 5 New Done Done true
1778 Acked 2777 Read 3 New Done Done true
1779 Acked 2778 Read 7 New Done Done true
1780 Acked 2779 Write 6 New Done Done true
1781 Acked 2780 Read 3 New Done Done true
1782 Acked 2781 Read 6 New Done Done true
1783 Acked 2782 Read 10 New Done Done true
1784 Acked 2783 Read 3 New Done Done true
1785 Acked 2784 Read 1 New Done Done true
1786 Acked 2785 Write 2 New Done Done true
1787 Acked 2786 Read 4 New Done Done true
1788 Acked 2787 Write 7 New Done Done true
1789 Acked 2788 Write 10 New Done Done true
1790 Acked 2789 Write 8 New Done Done true
1791 Acked 2790 Read 2 New Done Done true
1792 Acked 2791 Read 1 New Done Done true
1793 Acked 2792 Read 9 New Done Done true
1794 Acked 2793 Read 5 New Done Done true
1795 Acked 2794 Write 6 New Done Done true
1796 Acked 2795 Flush 0 New Done Done true
1797 Acked 2796 Read 10 New Done Done true
1798 Acked 2797 Read 2 New Done Done true
1799 Acked 2798 Write 8 New Done Done true
1800 Acked 2799 Read 1 New Done Done true
1801 Acked 2800 Read 10 New Done Done true
1802 Acked 2801 Read 6 New Done Done true
1803 Acked 2802 Read 7 New Done Done true
1804 Acked 2803 Write 4 New Done Done true
1805 Acked 2804 Read 8 New Done Done true
1806 Acked 2805 Write 4 New Done Done true
1807 Acked 2806 Write 2 New Done Done true
1808 Acked 2807 Flush 0 New Done Done true
STATES DS:0 DS:1 DS:2 TOTAL
New 301 0 0 301
Sent 0 0 0 0
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 2506 2807 2807
Downstairs last five completed: 2506 2505 2504 2503 2502
Upstairs last five completed: 1808 1807 1806 1805 1804
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:47.376559273Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:47.37663711Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:47.377910531Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:47.377972599Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 2506","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:47.378000808Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 2506","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:47.378170461Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:47.37820197Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:47.37821802Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:47.378232289Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:47.378246349Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 2807 2806 2805 2804 2803
Upstairs last five completed: 1808 1807 1806 1805 1804
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Feb 13 21:40:51.144 INFO request completed, latency_us: 349, response_code: 204, uri: /stop/cid/0, method:[8810] Got stop action so:false kr:false
POST, req_id: 2f571e80-8c40-46d3-831c-c37b0c639b75, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Feb 13 21:40:51.145 INFO request completed{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:51.145128441Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
, latency_us: 139, response_code:{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:51.145211068Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
200, uri: /state/cid/0, method: GET, req_id: aa4e69be-f16a-41b9-af03-cdd8a60640b6{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:51.145244377Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
, remote_addr: 127.0.0.1:52124, local_addr: 127.{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:51.145286045Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
0.0.1:9998
{"msg":"[0] client re-new 0 jobs since flush 2807","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:51.145321384Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:51.145349353Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:52.147216919Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:52.147280467Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:52.147304396Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:52.147325345Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:52.147341335Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 2807","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:52.147358914Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:52.147387873Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:53.148934562Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:53.149159673Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:53.149208581Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:53.149299928Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:53.149318577Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 2807","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:53.149345886Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:53.149372205Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:54.151086137Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:54.151142805Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:54.151160045Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:54.151176364Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:54.151189004Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 2807","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:54.151220042Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:54.151245961Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:40:55.148 INFO request completed, latency_us: 189, response_code: 200, uri: /state/cid/0, method: GET, req_id: bc830b5e-7be3-4130-8ed0-f5afaa46bf01, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Read block 1893 len 32768
002/300 Read block 1922 len 32768
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:55.151409044Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:55.151489381Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:55.15151072Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:55.151528289Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:55.151549398Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 2 jobs since flush 2807","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:55.151576087Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:55.151604506Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
003/300 Write block 3005 len 20480 data: 2 2 2 3 3
004/300 Write block 319 len 12288 data: 2 2 2
005/300 Flush
006/300 Read block 2150 len 36864
007/300 Write block 2709 len 12288 data: 2 2 2
008/300 Write block 2517 len 28672 data: 3 3 3 3 3 3 3
009/300 Write block 1640 len 40960 data: 2 2 2 2 4 3 3 3 3 3
010/300 Read block 987 len 8192
011/300 Write block 3582 len 28672 data: 4 3 3 3 5 5 5
012/300 Read block 1103 len 4096
013/300 Read block 183 len 20480
014/300 Write block 4714 len 8192 data: 4 3
015/300 Read block 1742 len 28672
016/300 Read block 3551 len 28672
017/300 Write block 4958 len 8192 data: 2 2
018/300 Read block 1814 len 40960
019/300 Read block 683 len 28672
020/300 Read block 1997 len 36864
021/300 Read block 3665 len 28672
022/300 Flush
023/300 Write block 329 len 4096 data: 3
024/300 Write block 2101 len 32768 data: 3 3 3 3 3 3 3 3
025/300 Read block 4383 len 12288
026/300 Read block 2765 len 20480
027/300 Write block 3217 len 4096 data: 2
028/300 Read block 1675 len 40960
029/300 Read block 308 len 24576
030/300 Write block 859 len 36864 data: 2 2 2 2 2 2 3 3 3
031/300 Read block 1355 len 24576
032/300 Flush
033/300 Flush
034/300 Write block 4327 len 36864 data: 4 3 3 3 3 3 3 3 3
035/300 Read block 3628 len 8192
036/300 Write block 3324 len 8192 data: 2 2
037/300 Read block 330 len 8192
038/300 Read block 3168 len 36864
039/300 Write block 2580 len 4096 data: 2
040/300 Read block 845 len 4096
041/300 Read block 4796 len 16384
042/300 Read block 1214 len 12288
043/300 Flush
044/300 Write block 4293 len 32768 data: 3 2 2 2 2 3 3 2
045/300 Read block 1020 len 28672
046/300 Read block 272 len 28672
047/300 Write block 886 len 4096 data: 3
048/300 Flush
049/300 Read block 1422 len 36864
050/300 Read block 4908 len 8192
051/300 Write block 352 len 4096 data: 5
052/300 Flush
053/300 Read block 1732 len 32768
054/300 Write block 2249 len 36864 data: 2 2 2 2 2 3 3 3 3
055/300 Read block 295 len 32768
056/300 Read block 24 len 8192
057/300 Write block 2998 len 36864 data: 3 2 2 2 3 3 2 3 3
058/300 Read block 3312 len 4096
059/300 Write block 2437 len 32768 data: 2 2 2 2 2 3 3 3
060/300 Write block 4578 len 16384 data: 4 4 4 2
061/300 Read block 4213 len 40960
062/300 Read block 1093 len 32768
063/300 Write block 230 len 28672 data: 2 3 3 3 3 3 3
064/300 Read block 1556 len 16384
065/300 Read block 3597 len 32768
066/300 Read block 1174 len 24576
067/300 Write block 2890 len 24576 data: 3 3 2 2 2 3
068/300 Write block 111 len 4096 data: 3
069/300 Read block 1247 len 8192
070/300 Write block 1356 len 16384 data: 4 4 3 4
071/300 Flush
072/300 Flush
073/300 Read block 123 len 36864
074/300 Write block 4212 len 16384 data: 2 2 2 3
075/300 Read block 697 len 24576
076/300 Read block 3104 len 12288
077/300 Flush
078/300 Read block 2567 len 40960
079/300 Read block 1235 len 24576
080/300 Read block 247 len 40960
081/300 Read block 2163 len 24576
082/300 Write block 1442 len 36864 data: 3 3 3 2 2 2 2 3 3
083/300 Flush
084/300 Write block 1703 len 28672 data: 3 3 3 3 3 2 2
085/300 Write block 3809 len 4096 data: 4
086/300 Read block 3618 len 8192
087/300 Read block 4770 len 28672
088/300 Write block 3732 len 28672 data: 2 2 2 2 2 2 2
089/300 Flush
090/300 Read block 4390 len 28672
091/300 Read block 3588 len 40960
092/300 Read block 2730 len 28672
093/300 Write block 3644 len 16384 data: 3 3 3 4
094/300 Read block 3669 len 4096
095/300 Read block 4625 len 8192
096/300 Read block 2873 len 40960
097/300 Write block 2846 len 40960 data: 2 2 2 2 2 2 3 2 2 2
098/300 Write block 29 len 28672 data: 2 2 2 2 2 2 2
099/300 Write block 1819 len 20480 data: 3 3 3 3 3
100/300 Read block 1233 len 36864
101/300 Read block 2096 len 24576
102/300 Read block 4369 len 4096
103/300 Read block 2889 len 16384
104/300 Read block 4633 len 36864
105/300 Write block 4802 len 12288 data: 2 3 2
106/300 Write block 2036 len 4096 data: 2
107/300 Read block 1553 len 24576
108/300 Write block 4513 len 4096 data: 3
109/300 Read block 2907 len 28672
110/300 Write block 2407 len 40960 data: 3 3 3 3 3 2 2 3 3 3
111/300 Flush
112/300 Read block 1165 len 24576
113/300 Write block 4125 len 32768 data: 4 4 4 3 3 3 3 2
114/300 Read block 1841 len 16384
115/300 Read block 2840 len 36864
116/300 Write block 767 len 8192 data: 2 2
117/300 Write block 4486 len 16384 data: 4 5 6 6
118/300 Flush
119/300 Flush
120/300 Write block 101 len 4096 data: 2
121/300 Read block 3687 len 12288
122/300 Write block 899 len 28672 data: 2 3 3 3 3 3 3
123/300 Write block 4711 len 8192 data: 2 2
124/300 Write block 2012 len 24576 data: 4 4 4 3 3 4
125/300 Write block 2498 len 40960 data: 3 3 3 3 3 4 3 2 2 3
126/300 Read block 725 len 32768
127/300 Flush
128/300 Read block 875 len 4096
129/300 Write block 3224 len 40960 data: 5 6 6 5 5 4 3 3 2 2
130/300 Write block 161 len 4096 data: 4
131/300 Read block 1534 len 20480
132/300 Write block 1711 len 8192 data: 2 2
133/300 Flush
134/300 Flush
135/300 Write block 3167 len 12288 data: 2 2 2
136/300 Flush
137/300 Read block 3331 len 40960
138/300 Read block 4534 len 12288
139/300 Flush
140/300 Read block 2064 len 20480
141/300 Read block 1968 len 32768
142/300 Flush
143/300 Write block 2391 len 28672 data: 2 3 3 3 3 3 2
144/300 Read block 191 len 40960
145/300 Read block 73 len 4096
146/300 Write block 1830 len 20480 data: 2 2 2 2 2
147/300 Read block 3795 len 32768
148/300 Write block 3814 len 16384 data: 2 2 2 2
149/300 Read block 3879 len 32768
150/300 Read block 3468 len 28672
151/300 Write block 1889 len 8192 data: 3 3
152/300 Read block 4841 len 24576
153/300 Read block 142 len 20480
154/300 Read block 3371 len 40960
155/300 Write block 3758 len 24576 data: 2 2 2 2 2 2
156/300 Read block 1736 len 8192
157/300 Write block 3124 len 32768 data: 4 4 4 2 3 2 2 2
158/300 Read block 2820 len 28672
159/300 Flush
160/300 Read block 2624 len 8192
161/300 Read block 4140 len 20480
162/300 Read block 3647 len 4096
163/300 Read block 3596 len 20480
164/300 Read block 3500 len 40960
165/300 Read block 3763 len 40960
166/300 Read block 600 len 24576
167/300 Write block 558 len 4096 data: 2
168/300 Write block 3573 len 12288 data: 2 2 2
169/300 Read block 1780 len 8192
170/300 Read block 3285 len 12288
171/300 Read block 2688 len 4096
172/300 Flush
173/300 Read block 2829 len 32768
174/300 Write block 1025 len 16384 data: 2 2 2 2
175/300 Read block 3737 len 28672
176/300 Write block 2283 len 32768 data: 2 3 3 3 3 3 3 3
177/300 Read block 4455 len 4096
178/300 Write block 4401 len 4096 data: 3
179/300 Read block 3651 len 28672
180/300 Write block 2325 len 32768 data: 2 2 4 4 3 3 3 3
181/300 Read block 3712 len 16384
182/300 Read block 1016 len 4096
183/300 Read block 2674 len 32768
184/300 Read block 1915 len 24576
185/300 Write block 346 len 12288 data: 3 3 3
186/300 Read block 4520 len 12288
187/300 Read block 4226 len 20480
188/300 Read block 494 len 40960
189/300 Flush
190/300 Flush
191/300 Read block 968 len 32768
192/300 Read block 2764 len 36864
193/300 Read block 1552 len 4096
194/300 Read block 1326 len 20480
195/300 Read block 1625 len 28672
196/300 Write block 255 len 32768 data: 4 3 3 3 2 2 2 2
197/300 Read block 4159 len 36864
198/300 Read block 1408 len 36864
199/300 Read block 2144 len 32768
200/300 Read block 3401 len 28672
201/300 Write block 3209 len 28672 data: 3 3 3 3 3 2 2
202/300 Read block 4028 len 40960
203/300 Read block 157 len 36864
204/300 Read block 887 len 28672
205/300 Read block 2027 len 12288
206/300 Flush
207/300 Write block 2458 len 40960 data: 2 2 2 2 2 3 3 3 3 3
208/300 Write block 1136 len 4096 data: 4
209/300 Read block 647 len 32768
210/300 Write block 4230 len 4096 data: 3
211/300 Flush
212/300 Write block 462 len 36864 data: 4 3 3 4 4 4 3 2 2
213/300 Read block 3354 len 8192
214/300 Read block 1880 len 8192
215/300 Write block 3220 len 36864 data: 4 5 5 5 6 7 7 6 6
216/300 Write block 854 len 40960 data: 3 4 5 4 3 3 3 3 3 3
217/300 Read block 3083 len 12288
218/300 Write block 3978 len 40960 data: 3 3 3 2 2 2 2 2 2 2
219/300 Flush
220/300 Read block 2851 len 12288
221/300 Write block 1878 len 24576 data: 2 2 2 2 2 2
222/300 Read block 624 len 40960
223/300 Write block 3832 len 4096 data: 3
224/300 Read block 4897 len 8192
225/300 Write block 3709 len 4096 data: 2
226/300 Read block 3400 len 16384
227/300 Write block 1131 len 8192 data: 3 3
228/300 Write block 4901 len 28672 data: 2 2 2 2 2 2 3
229/300 Read block 2362 len 32768
230/300 Write block 884 len 40960 data: 3 3 4 3 3 2 2 2 2 3
231/300 Write block 918 len 8192 data: 2 2
232/300 Write block 267 len 28672 data: 2 3 3 3 4 4 2
233/300 Read block 4793 len 8192
234/300 Flush
235/300 Flush
236/300 Read block 1966 len 12288
237/300 Write block 3194 len 4096 data: 4
238/300 Read block 752 len 8192
239/300 Write block 3275 len 12288 data: 2 2 2
240/300 Write block 1594 len 8192 data: 4 4
241/300 Read block 474 len 28672
242/300 Read block 4609 len 8192
243/300 Flush
244/300 Flush
245/300 Write block 1985 len 28672 data: 2 2 2 2 3 3 3
246/300 Write block 1329 len 20480 data: 3 3 3 4 4
247/300 Read block 255 len 16384
248/300 Write block 1103 len 40960 data: 3 4 4 4 4 3 3 3 3 3
249/300 Write block 3736 len 28672 data: 3 3 3 2 3 3 3
250/300 Write block 4121 len 16384 data: 3 3 4 4
251/300 Write block 355 len 40960 data: 4 3 4 4 3 3 3 3 3 3
252/300 Write block 2203 len 4096 data: 3
253/300 Write block 1937 len 28672 data: 2 2 2 2 2 2 2
254/300 Write block 4238 len 12288 data: 2 2 2
255/300 Read block 3404 len 4096
256/300 Write block 3134 len 28672 data: 2 2 2 2 2 2 2
257/300 Write block 48 len 12288 data: 5 5 4
258/300 Write block 4110 len 32768 data: 2 3 3 3 4 4 4 3
259/300 Read block 649 len 36864
260/300 Read block 3932 len 20480
261/300 Read block 3987 len 40960
262/300 Flush
263/300 Write block 1911 len 32768 data: 3 4 4 3 3 2 2 2
264/300 Write block 1252 len 20480 data: 3 3 2 2 2
265/300 Read block 2470 len 28672
266/300 Write block 2294 len 16384 data: 2 2 2 2
267/300 Write block 858 len 24576 data: 4 4 4 4 4 4
268/300 Write block 2197 len 24576 data: 2 2 2 2 2 2
269/300 Read block 3920 len 4096
270/300 Write block 3033 len 16384 data: 2 2 2 2
271/300 Read block 1824 len 8192
272/300 Read block 2377 len 16384
273/300 Write block 872 len 28672 data: 3 2 2 2 2 2 2
274/300 Read block 850 len 28672
275/300 Write block 2120 len 24576 data: 2 2 2 2 2 2
276/300 Write block 685 len 28672 data: 5 5 5 5 5 4 3
277/300 Read block 4639 len 8192
278/300 Read block 2214 len 36864
279/300 Read block 1452 len 16384
280/300 Read block 3136 len 4096
281/300 Write block 4403 len 36864 data: 2 2 3 4 4 3 2 2 2
282/300 Write block 1072 len 12288 data: 3 2 2
283/300 Write block 3530 len 32768 data: 2 2 2 2 2 2 2 2
284/300 Write block 4770 len 28672 data: 3 3 3 2 2 3 3
285/300 Write block 1522 len 8192 data: 3 2
286/300 Read block 2887 len 40960
287/300 Read block 1438 len 20480
288/300 Write block 2549 len 12288 data: 2 2 2
289/300 Read block 3523 len 24576
290/300 Write block 884 len 16384 data: 4 4 5 4
291/300 Write block 1172 len 20480 data: 3 3 2 2 2
292/300 Read block 264 len 20480
293/300 Write block 1128 len 12288 data: 2 2 3
294/300 Read block 694 len 8192
295/300 Read block 4884 len 24576
296/300 Write block 1716 len 24576 data: 2 3 3 4 4 4
297/300 Write block 4117 len 12288 data: 4 3 3
298/300 Write block 3572 len 32768 data: 2 3 3 3 2 3 3 3
299/300 Write block 248 len 8192 data: 4 4
300/300 Read block 1445 len 32768
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 13 21:40:55.876 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/0, method: POST, req_id: 34165be1-ea6d-4b59-9a5a-79e169ccd307, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
1809 Acked 2808 Read 8 New Done Done true
1810 Acked 2809 Read 8 New Done Done true
1811 Acked 2810 Write 5 New Done Done false
1812 Acked 2811 Write 3 New Done Done false
1813 Acked 2812 Flush 0 New Done Done false
1814 Acked 2813 Read 9 New Done Done false
1815 Acked 2814 Write 3 New Done Done false
1816 Acked 2815 Write 7 New Done Done false
1817 Acked 2816 Write 10 New Done Done false
1818 Acked 2817 Read 2 New Done Done false
1819 Acked 2818 Write 7 New Done Done false
1820 Acked 2819 Read 1 New Done Done false
1821 Acked 2820 Read 5 New Done Done false
1822 Acked 2821 Write 2 New Done Done false
1823 Acked 2822 Read 7 New Done Done false
1824 Acked 2823 Read 7 New Done Done false
1825 Acked 2824 Write 2 New Done Done false
1826 Acked 2825 Read 10 New Done Done false
1827 Acked 2826 Read 7 New Done Done false
1828 Acked 2827 Read 9 New Done Done false
1829 Acked 2828 Read 7 New Done Done false
1830 Acked 2829 Flush 0 New Done Done false
1831 Acked 2830 Write 1 New Done Done false
1832 Acked 2831 Write 8 New Done Done false
1833 Acked 2832 Read 3 New Done Done false
1834 Acked 2833 Read 5 New Done Done false
1835 Acked 2834 Write 1 New Done Done false
1836 Acked 2835 Read 10 New Done Done false
1837 Acked 2836 Read 6 New Done Done false
1838 Acked 2837 Write 9 New Done Done false
1839 Acked 2838 Read 6 New Done Done false
1840 Acked 2839 Flush 0 New Done Done false
1841 Acked 2840 Flush 0 New Done Done false
1842 Acked 2841 Write 9 New Done Done false
1843 Acked 2842 Read 2 New Done Done false
1844 Acked 2843 Write 2 New Done Done false
1845 Acked 2844 Read 2 New Done Done false
1846 Acked 2845 Read 9 New Done Done false
1847 Acked 2846 Write 1 New Done Done false
1848 Acked 2847 Read 1 New Done Done false
1849 Acked 2848 Read 4 New Done Done false
1850 Acked 2849 Read 3 New Done Done false
1851 Acked 2850 Flush 0 New Done Done false
1852 Acked 2851 Write 8 New Done Done false
1853 Acked 2852 Read 7 New Done Done false
1854 Acked 2853 Read 7 New Done Done false
1855 Acked 2854 Write 1 New Done Done false
1856 Acked 2855 Flush 0 New Done Done false
1857 Acked 2856 Read 9 New Done Done false
1858 Acked 2857 Read 2 New Done Done false
1859 Acked 2858 Write 1 New Done Done false
1860 Acked 2859 Flush 0 New Done Done false
1861 Acked 2860 Read 8 New Done Done false
1862 Acked 2861 Write 9 New Done Done false
1863 Acked 2862 Read 8 New Done Done false
1864 Acked 2863 Read 2 New Done Done false
1865 Acked 2864 Write 9 New Done Done false
1866 Acked 2865 Read 1 New Done Done false
1867 Acked 2866 Write 8 New Done Done false
1868 Acked 2867 Write 4 New Done Done false
1869 Acked 2868 Read 10 New Done Done false
1870 Acked 2869 Read 8 New Done Done false
1871 Acked 2870 Write 7 New Done Done false
1872 Acked 2871 Read 4 New Done Done false
1873 Acked 2872 Read 8 New Done Done false
1874 Acked 2873 Read 6 New Done Done false
1875 Acked 2874 Write 6 New Done Done false
1876 Acked 2875 Write 1 New Done Done false
1877 Acked 2876 Read 2 New Done Done false
1878 Acked 2877 Write 4 New Done Done false
1879 Acked 2878 Flush 0 New Done Done false
1880 Acked 2879 Flush 0 New Done Done false
1881 Acked 2880 Read 9 New Done Done false
1882 Acked 2881 Write 4 New Done Done false
1883 Acked 2882 Read 6 New Done Done false
1884 Acked 2883 Read 3 New Done Done false
1885 Acked 2884 Flush 0 New Done Done false
1886 Acked 2885 Read 10 New Done Done false
1887 Acked 2886 Read 6 New Done Done false
1888 Acked 2887 Read 10 New Done Done false
1889 Acked 2888 Read 6 New Done Done false
1890 Acked 2889 Write 9 New Done Done false
1891 Acked 2890 Flush 0 New Done Done false
1892 Acked 2891 Write 7 New Done Done false
1893 Acked 2892 Write 1 New Done Done false
1894 Acked 2893 Read 2 New Done Done false
1895 Acked 2894 Read 7 New Done Done false
1896 Acked 2895 Write 7 New Done Done false
1897 Acked 2896 Flush 0 New Done Done false
1898 Acked 2897 Read 7 New Done Done false
1899 Acked 2898 Read 10 New Done Done false
1900 Acked 2899 Read 7 New Done Done false
1901 Acked 2900 Write 4 New Done Done false
1902 Acked 2901 Read 1 New Done Done false
1903 Acked 2902 Read 2 New Done Done false
1904 Acked 2903 Read 10 New Done Done false
1905 Acked 2904 Write 10 New Done Done false
1906 Acked 2905 Write 7 New Done Done false
1907 Acked 2906 Write 5 New Done Done false
1908 Acked 2907 Read 9 New Done Done false
1909 Acked 2908 Read 6 New Done Done false
1910 Acked 2909 Read 1 New Done Done false
1911 Acked 2910 Read 4 New Done Done false
1912 Acked 2911 Read 9 New Done Done false
1913 Acked 2912 Write 3 New Done Done false
1914 Acked 2913 Write 1 New Done Done false
1915 Acked 2914 Read 6 New Done Done false
1916 Acked 2915 Write 1 New Done Done false
1917 Acked 2916 Read 7 New Done Done false
1918 Acked 2917 Write 10 New Done Done false
1919 Acked 2918 Flush 0 New Done Done false
1920 Acked 2919 Read 6 New Done Done false
1921 Acked 2920 Write 8 New Done Done false
1922 Acked 2921 Read 4 New Done Done false
1923 Acked 2922 Read 9 New Done Done false
1924 Acked 2923 Write 2 New Done Done false
1925 Acked 2924 Write 4 New Done Done false
1926 Acked 2925 Flush 0 New Done Done false
1927 Acked 2926 Flush 0 New Done Done false
1928 Acked 2927 Write 1 New Done Done false
1929 Acked 2928 Read 3 New Done Done false
1930 Acked 2929 Write 7 New Done Done false
1931 Acked 2930 Write 2 New Done Done false
1932 Acked 2931 Write 6 New Done Done false
1933 Acked 2932 Write 10 New Done Done false
1934 Acked 2933 Read 8 New Done Done false
1935 Acked 2934 Flush 0 New Done Done false
1936 Acked 2935 Read 1 New Done Done false
1937 Acked 2936 Write 10 New Done Done false
1938 Acked 2937 Write 1 New Done Done false
1939 Acked 2938 Read 5 New Done Done false
1940 Acked 2939 Write 2 New Done Done false
1941 Acked 2940 Flush 0 New Done Done false
1942 Acked 2941 Flush 0 New Done Done false
1943 Acked 2942 Write 3 New Done Done false
1944 Acked 2943 Flush 0 New Done Done false
1945 Acked 2944 Read 10 New Done Done false
1946 Acked 2945 Read 3 New Done Done false
1947 Acked 2946 Flush 0 New Done Done false
1948 Acked 2947 Read 5 New Done Done false
1949 Acked 2948 Read 8 New Done Done false
1950 Acked 2949 Flush 0 New Done Done false
1951 Acked 2950 Write 7 New Done Done false
1952 Acked 2951 Read 10 New Done Done false
1953 Acked 2952 Read 1 New Done Done false
1954 Acked 2953 Write 5 New Done Done false
1955 Acked 2954 Read 8 New Done Done false
1956 Acked 2955 Write 4 New Done Done false
1957 Acked 2956 Read 8 New Done Done false
1958 Acked 2957 Read 7 New Done Done false
1959 Acked 2958 Write 2 New Done Done false
1960 Acked 2959 Read 6 New Done Done false
1961 Acked 2960 Read 5 New Done Done false
1962 Acked 2961 Read 10 New Done Done false
1963 Acked 2962 Write 6 New Done Done false
1964 Acked 2963 Read 2 New Done Done false
1965 Acked 2964 Write 8 New Done Done false
1966 Acked 2965 Read 7 New Done Done false
1967 Acked 2966 Flush 0 New Done Done false
1968 Acked 2967 Read 2 New Done Done false
1969 Acked 2968 Read 5 New Done Done false
1970 Acked 2969 Read 1 New Done Done false
1971 Acked 2970 Read 5 New Done Done false
1972 Acked 2971 Read 10 New Done Done false
1973 Acked 2972 Read 10 New Done Done false
1974 Acked 2973 Read 6 New Done Done false
1975 Acked 2974 Write 1 New Done Done false
1976 Acked 2975 Write 3 New Done Done false
1977 Acked 2976 Read 2 New Done Done false
1978 Acked 2977 Read 3 New Done Done false
1979 Acked 2978 Read 1 New Done Done false
1980 Acked 2979 Flush 0 New Done Done false
1981 Acked 2980 Read 8 New Done Done false
1982 Acked 2981 Write 4 New Done Done false
1983 Acked 2982 Read 7 New Done Done false
1984 Acked 2983 Write 8 New Done Done false
1985 Acked 2984 Read 1 New Done Done false
1986 Acked 2985 Write 1 New Done Done false
1987 Acked 2986 Read 7 New Done Done false
1988 Acked 2987 Write 8 New Done Done false
1989 Acked 2988 Read 4 New Done Done false
1990 Acked 2989 Read 1 New Done Done false
1991 Acked 2990 Read 8 New Done Done false
1992 Acked 2991 Read 6 New Done Done false
1993 Acked 2992 Write 3 New Done Done false
1994 Acked 2993 Read 3 New Done Done false
1995 Acked 2994 Read 5 New Done Done false
1996 Acked 2995 Read 10 New Done Done false
1997 Acked 2996 Flush 0 New Done Done false
1998 Acked 2997 Flush 0 New Done Done false
1999 Acked 2998 Read 8 New Done Done false
2000 Acked 2999 Read 9 New Done Done false
2001 Acked 3000 Read 1 New Done Done false
2002 Acked 3001 Read 5 New Done Done false
2003 Acked 3002 Read 7 New Done Done false
2004 Acked 3003 Write 8 New Done Done false
2005 Acked 3004 Read 9 New Done Done false
2006 Acked 3005 Read 9 New Done Done false
2007 Acked 3006 Read 8 New Done Done false
2008 Acked 3007 Read 7 New Done Done false
2009 Acked 3008 Write 7 New Done Done false
2010 Acked 3009 Read 10 New Done Done false
2011 Acked 3010 Read 9 New Done Done false
2012 Acked 3011 Read 7 New Done Done false
2013 Acked 3012 Read 3 New Done Done false
2014 Acked 3013 Flush 0 New Done Done false
2015 Acked 3014 Write 10 New Done Done false
2016 Acked 3015 Write 1 New Done Done false
2017 Acked 3016 Read 8 New Done Done false
2018 Acked 3017 Write 1 New Done Done false
2019 Acked 3018 Flush 0 New Done Done false
2020 Acked 3019 Write 9 New Done Done false
2021 Acked 3020 Read 2 New Done Done false
2022 Acked 3021 Read 2 New Done Done false
2023 Acked 3022 Write 9 New Done Done false
2024 Acked 3023 Write 10 New Done Done false
2025 Acked 3024 Read 3 New Done Done false
2026 Acked 3025 Write 10 New Done Done false
2027 Acked 3026 Flush 0 New Done Done false
2028 Acked 3027 Read 3 New Done Done false
2029 Acked 3028 Write 6 New Done Done false
2030 Acked 3029 Read 10 New Done Done false
2031 Acked 3030 Write 1 New Done Done false
2032 Acked 3031 Read 2 New Done Done false
2033 Acked 3032 Write 1 New Done Done false
2034 Acked 3033 Read 4 New Done Done false
2035 Acked 3034 Write 2 New Done Done false
2036 Acked 3035 Write 7 New Done Done false
2037 Acked 3036 Read 8 New Done Done false
2038 Acked 3037 Write 10 New Done Done false
2039 Acked 3038 Write 2 New Done Done false
2040 Acked 3039 Write 7 New Done Done false
2041 Acked 3040 Read 2 New Done Done false
2042 Acked 3041 Flush 0 New Done Done false
2043 Acked 3042 Flush 0 New Done Done false
2044 Acked 3043 Read 3 New Done Done false
2045 Acked 3044 Write 1 New Done Done false
2046 Acked 3045 Read 2 New Done Done false
2047 Acked 3046 Write 3 New Done Done false
2048 Acked 3047 Write 2 New Done Done false
2049 Acked 3048 Read 7 New Done Done false
2050 Acked 3049 Read 2 New Done Done false
2051 Acked 3050 Flush 0 New Done Done false
2052 Acked 3051 Flush 0 New Done Done false
2053 Acked 3052 Write 7 New Done Done false
2054 Acked 3053 Write 5 New Done Done false
2055 Acked 3054 Read 4 New Done Done false
2056 Acked 3055 Write 10 New Done Done false
2057 Acked 3056 Write 7 New Done Done false
2058 Acked 3057 Write 4 New Done Done false
2059 Acked 3058 Write 10 New Done Done false
2060 Acked 3059 Write 1 New Done Done false
2061 Acked 3060 Write 7 New Done Done false
2062 Acked 3061 Write 3 New Done Done false
2063 Acked 3062 Read 1 New Done Done false
2064 Acked 3063 Write 7 New Done Done false
2065 Acked 3064 Write 3 New Done Done false
2066 Acked 3065 Write 8 New Done Done false
2067 Acked 3066 Read 9 New Done Done false
2068 Acked 3067 Read 5 New Done Done false
2069 Acked 3068 Read 10 New Done Done false
2070 Acked 3069 Flush 0 New Done Done false
2071 Acked 3070 Write 8 New Done Done false
2072 Acked 3071 Write 5 New Done Done false
2073 Acked 3072 Read 7 New Done Done false
2074 Acked 3073 Write 4 New Done Done false
2075 Acked 3074 Write 6 New Done Done false
2076 Acked 3075 Write 6 New Done Done false
2077 Acked 3076 Read 1 New Done Done false
2078 Acked 3077 Write 4 New Done Done false
2079 Acked 3078 Read 2 New Done Done false
2080 Acked 3079 Read 4 New Done Done false
2081 Acked 3080 Write 7 New Done Done false
2082 Acked 3081 Read 7 New Done Done false
2083 Acked 3082 Write 6 New Done Done false
2084 Acked 3083 Write 7 New Done Done false
2085 Acked 3084 Read 2 New Done Done false
2086 Acked 3085 Read 9 New Done Done false
2087 Acked 3086 Read 4 New Done Done false
2088 Acked 3087 Read 1 New Done Done false
2089 Acked 3088 Write 9 New Done Done false
2090 Acked 3089 Write 3 New Done Done false
2091 Acked 3090 Write 8 New Done Done false
2092 Acked 3091 Write 7 New Done Done false
2093 Acked 3092 Write 2 New Done Done false
2094 Acked 3093 Read 10 New Done Done false
2095 Acked 3094 Read 5 New Done Done false
2096 Acked 3095 Write 3 New Done Done false
2097 Acked 3096 Read 6 New Done Done false
2098 Acked 3097 Write 4 New Done Done false
2099 Acked 3098 Write 5 New Done Done false
2100 Acked 3099 Read 5 New Done Done false
2101 Acked 3100 Write 3 New Done Done false
2102 Acked 3101 Read 2 New Done Done false
2103 Acked 3102 Read 6 New Done Done false
2104 Acked 3103 Write 6 New Done Done false
2105 Acked 3104 Write 3 New Done Done false
2106 Acked 3105 Write 8 New Done Done false
2107 Acked 3106 Write 2 New Done Done false
2108 Acked 3107 Read 8 New Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
New 300 0 0 300
Sent 0 0 0 0
Done 0 300 300 600
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 2807 3069 3069
Downstairs last five completed: 2807 2806 2805 2804 2803
Upstairs last five completed: 2108 2107 2106 2105 2104
CLIENT: Up:0 ds:300 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:56.1525136Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:56.152575758Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:56.152599667Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:56.152620896Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:56.152636675Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 2807","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:56.152654185Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:56.152700313Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:57.153705253Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:57.15376944Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:57.153791909Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:57.153812749Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:57.153828688Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 2807","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:57.153846167Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:57.153939534Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:58.154985622Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:58.155048429Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:58.155069958Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:58.155090328Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:58.155106027Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 2807","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:58.155137756Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:58.15529654Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:59.155889296Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:59.155957483Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:59.155986312Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:40:59.156031841Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:59.156057769Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 2807","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:59.156086228Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:40:59.156193834Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1171)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
1809 Acked 2808 Read 8 New Done Done true
1810 Acked 2809 Read 8 New Done Done true
1811 Acked 2810 Write 5 New Done Done true
1812 Acked 2811 Write 3 New Done Done true
1813 Acked 2812 Flush 0 New Done Done true
1814 Acked 2813 Read 9 New Done Done true
1815 Acked 2814 Write 3 New Done Done true
1816 Acked 2815 Write 7 New Done Done true
1817 Acked 2816 Write 10 New Done Done true
1818 Acked 2817 Read 2 New Done Done true
1819 Acked 2818 Write 7 New Done Done true
1820 Acked 2819 Read 1 New Done Done true
1821 Acked 2820 Read 5 New Done Done true
1822 Acked 2821 Write 2 New Done Done true
1823 Acked 2822 Read 7 New Done Done true
1824 Acked 2823 Read 7 New Done Done true
1825 Acked 2824 Write 2 New Done Done true
1826 Acked 2825 Read 10 New Done Done true
1827 Acked 2826 Read 7 New Done Done true
1828 Acked 2827 Read 9 New Done Done true
1829 Acked 2828 Read 7 New Done Done true
1830 Acked 2829 Flush 0 New Done Done true
1831 Acked 2830 Write 1 New Done Done true
1832 Acked 2831 Write 8 New Done Done true
1833 Acked 2832 Read 3 New Done Done true
1834 Acked 2833 Read 5 New Done Done true
1835 Acked 2834 Write 1 New Done Done true
1836 Acked 2835 Read 10 New Done Done true
1837 Acked 2836 Read 6 New Done Done true
1838 Acked 2837 Write 9 New Done Done true
1839 Acked 2838 Read 6 New Done Done true
1840 Acked 2839 Flush 0 New Done Done true
1841 Acked 2840 Flush 0 New Done Done true
1842 Acked 2841 Write 9 New Done Done true
1843 Acked 2842 Read 2 New Done Done true
1844 Acked 2843 Write 2 New Done Done true
1845 Acked 2844 Read 2 New Done Done true
1846 Acked 2845 Read 9 New Done Done true
1847 Acked 2846 Write 1 New Done Done true
1848 Acked 2847 Read 1 New Done Done true
1849 Acked 2848 Read 4 New Done Done true
1850 Acked 2849 Read 3 New Done Done true
1851 Acked 2850 Flush 0 New Done Done true
1852 Acked 2851 Write 8 New Done Done true
1853 Acked 2852 Read 7 New Done Done true
1854 Acked 2853 Read 7 New Done Done true
1855 Acked 2854 Write 1 New Done Done true
1856 Acked 2855 Flush 0 New Done Done true
1857 Acked 2856 Read 9 New Done Done true
1858 Acked 2857 Read 2 New Done Done true
1859 Acked 2858 Write 1 New Done Done true
1860 Acked 2859 Flush 0 New Done Done true
1861 Acked 2860 Read 8 New Done Done true
1862 Acked 2861 Write 9 New Done Done true
1863 Acked 2862 Read 8 New Done Done true
1864 Acked 2863 Read 2 New Done Done true
1865 Acked 2864 Write 9 New Done Done true
1866 Acked 2865 Read 1 New Done Done true
1867 Acked 2866 Write 8 New Done Done true
1868 Acked 2867 Write 4 New Done Done true
1869 Acked 2868 Read 10 New Done Done true
1870 Acked 2869 Read 8 New Done Done true
1871 Acked 2870 Write 7 New Done Done true
1872 Acked 2871 Read 4 New Done Done true
1873 Acked 2872 Read 8 New Done Done true
1874 Acked 2873 Read 6 New Done Done true
1875 Acked 2874 Write 6 New Done Done true
1876 Acked 2875 Write 1 New Done Done true
1877 Acked 2876 Read 2 New Done Done true
1878 Acked 2877 Write 4 New Done Done true
1879 Acked 2878 Flush 0 New Done Done true
1880 Acked 2879 Flush 0 New Done Done true
1881 Acked 2880 Read 9 New Done Done true
1882 Acked 2881 Write 4 New Done Done true
1883 Acked 2882 Read 6 New Done Done true
1884 Acked 2883 Read 3 New Done Done true
1885 Acked 2884 Flush 0 New Done Done true
1886 Acked 2885 Read 10 New Done Done true
1887 Acked 2886 Read 6 New Done Done true
1888 Acked 2887 Read 10 New Done Done true
1889 Acked 2888 Read 6 New Done Done true
1890 Acked 2889 Write 9 New Done Done true
1891 Acked 2890 Flush 0 New Done Done true
1892 Acked 2891 Write 7 New Done Done true
1893 Acked 2892 Write 1 New Done Done true
1894 Acked 2893 Read 2 New Done Done true
1895 Acked 2894 Read 7 New Done Done true
1896 Acked 2895 Write 7 New Done Done true
1897 Acked 2896 Flush 0 New Done Done true
1898 Acked 2897 Read 7 New Done Done true
1899 Acked 2898 Read 10 New Done Done true
1900 Acked 2899 Read 7 New Done Done true
1901 Acked 2900 Write 4 New Done Done true
1902 Acked 2901 Read 1 New Done Done true
1903 Acked 2902 Read 2 New Done Done true
1904 Acked 2903 Read 10 New Done Done true
1905 Acked 2904 Write 10 New Done Done true
1906 Acked 2905 Write 7 New Done Done true
1907 Acked 2906 Write 5 New Done Done true
1908 Acked 2907 Read 9 New Done Done true
1909 Acked 2908 Read 6 New Done Done true
1910 Acked 2909 Read 1 New Done Done true
1911 Acked 2910 Read 4 New Done Done true
1912 Acked 2911 Read 9 New Done Done true
1913 Acked 2912 Write 3 New Done Done true
1914 Acked 2913 Write 1 New Done Done true
1915 Acked 2914 Read 6 New Done Done true
1916 Acked 2915 Write 1 New Done Done true
1917 Acked 2916 Read 7 New Done Done true
1918 Acked 2917 Write 10 New Done Done true
1919 Acked 2918 Flush 0 New Done Done true
1920 Acked 2919 Read 6 New Done Done true
1921 Acked 2920 Write 8 New Done Done true
1922 Acked 2921 Read 4 New Done Done true
1923 Acked 2922 Read 9 New Done Done true
1924 Acked 2923 Write 2 New Done Done true
1925 Acked 2924 Write 4 New Done Done true
1926 Acked 2925 Flush 0 New Done Done true
1927 Acked 2926 Flush 0 New Done Done true
1928 Acked 2927 Write 1 New Done Done true
1929 Acked 2928 Read 3 New Done Done true
1930 Acked 2929 Write 7 New Done Done true
1931 Acked 2930 Write 2 New Done Done true
1932 Acked 2931 Write 6 New Done Done true
1933 Acked 2932 Write 10 New Done Done true
1934 Acked 2933 Read 8 New Done Done true
1935 Acked 2934 Flush 0 New Done Done true
1936 Acked 2935 Read 1 New Done Done true
1937 Acked 2936 Write 10 New Done Done true
1938 Acked 2937 Write 1 New Done Done true
1939 Acked 2938 Read 5 New Done Done true
1940 Acked 2939 Write 2 New Done Done true
1941 Acked 2940 Flush 0 New Done Done true
1942 Acked 2941 Flush 0 New Done Done true
1943 Acked 2942 Write 3 New Done Done true
1944 Acked 2943 Flush 0 New Done Done true
1945 Acked 2944 Read 10 New Done Done true
1946 Acked 2945 Read 3 New Done Done true
1947 Acked 2946 Flush 0 New Done Done true
1948 Acked 2947 Read 5 New Done Done true
1949 Acked 2948 Read 8 New Done Done true
1950 Acked 2949 Flush 0 New Done Done true
1951 Acked 2950 Write 7 New Done Done true
1952 Acked 2951 Read 10 New Done Done true
1953 Acked 2952 Read 1 New Done Done true
1954 Acked 2953 Write 5 New Done Done true
1955 Acked 2954 Read 8 New Done Done true
1956 Acked 2955 Write 4 New Done Done true
1957 Acked 2956 Read 8 New Done Done true
1958 Acked 2957 Read 7 New Done Done true
1959 Acked 2958 Write 2 New Done Done true
1960 Acked 2959 Read 6 New Done Done true
1961 Acked 2960 Read 5 New Done Done true
1962 Acked 2961 Read 10 New Done Done true
1963 Acked 2962 Write 6 New Done Done true
1964 Acked 2963 Read 2 New Done Done true
1965 Acked 2964 Write 8 New Done Done true
1966 Acked 2965 Read 7 New Done Done true
1967 Acked 2966 Flush 0 New Done Done true
1968 Acked 2967 Read 2 New Done Done true
1969 Acked 2968 Read 5 New Done Done true
1970 Acked 2969 Read 1 New Done Done true
1971 Acked 2970 Read 5 New Done Done true
1972 Acked 2971 Read 10 New Done Done true
1973 Acked 2972 Read 10 New Done Done true
1974 Acked 2973 Read 6 New Done Done true
1975 Acked 2974 Write 1 New Done Done true
1976 Acked 2975 Write 3 New Done Done true
1977 Acked 2976 Read 2 New Done Done true
1978 Acked 2977 Read 3 New Done Done true
1979 Acked 2978 Read 1 New Done Done true
1980 Acked 2979 Flush 0 New Done Done true
1981 Acked 2980 Read 8 New Done Done true
1982 Acked 2981 Write 4 New Done Done true
1983 Acked 2982 Read 7 New Done Done true
1984 Acked 2983 Write 8 New Done Done true
1985 Acked 2984 Read 1 New Done Done true
1986 Acked 2985 Write 1 New Done Done true
1987 Acked 2986 Read 7 New Done Done true
1988 Acked 2987 Write 8 New Done Done true
1989 Acked 2988 Read 4 New Done Done true
1990 Acked 2989 Read 1 New Done Done true
1991 Acked 2990 Read 8 New Done Done true
1992 Acked 2991 Read 6 New Done Done true
1993 Acked 2992 Write 3 New Done Done true
1994 Acked 2993 Read 3 New Done Done true
1995 Acked 2994 Read 5 New Done Done true
1996 Acked 2995 Read 10 New Done Done true
1997 Acked 2996 Flush 0 New Done Done true
1998 Acked 2997 Flush 0 New Done Done true
1999 Acked 2998 Read 8 New Done Done true
2000 Acked 2999 Read 9 New Done Done true
2001 Acked 3000 Read 1 New Done Done true
2002 Acked 3001 Read 5 New Done Done true
2003 Acked 3002 Read 7 New Done Done true
2004 Acked 3003 Write 8 New Done Done true
2005 Acked 3004 Read 9 New Done Done true
2006 Acked 3005 Read 9 New Done Done true
2007 Acked 3006 Read 8 New Done Done true
2008 Acked 3007 Read 7 New Done Done true
2009 Acked 3008 Write 7 New Done Done true
2010 Acked 3009 Read 10 New Done Done true
2011 Acked 3010 Read 9 New Done Done true
2012 Acked 3011 Read 7 New Done Done true
2013 Acked 3012 Read 3 New Done Done true
2014 Acked 3013 Flush 0 New Done Done true
2015 Acked 3014 Write 10 New Done Done true
2016 Acked 3015 Write 1 New Done Done true
2017 Acked 3016 Read 8 New Done Done true
2018 Acked 3017 Write 1 New Done Done true
2019 Acked 3018 Flush 0 New Done Done true
2020 Acked 3019 Write 9 New Done Done true
2021 Acked 3020 Read 2 New Done Done true
2022 Acked 3021 Read 2 New Done Done true
2023 Acked 3022 Write 9 New Done Done true
2024 Acked 3023 Write 10 New Done Done true
2025 Acked 3024 Read 3 New Done Done true
2026 Acked 3025 Write 10 New Done Done true
2027 Acked 3026 Flush 0 New Done Done true
2028 Acked 3027 Read 3 New Done Done true
2029 Acked 3028 Write 6 New Done Done true
2030 Acked 3029 Read 10 New Done Done true
2031 Acked 3030 Write 1 New Done Done true
2032 Acked 3031 Read 2 New Done Done true
2033 Acked 3032 Write 1 New Done Done true
2034 Acked 3033 Read 4 New Done Done true
2035 Acked 3034 Write 2 New Done Done true
2036 Acked 3035 Write 7 New Done Done true
2037 Acked 3036 Read 8 New Done Done true
2038 Acked 3037 Write 10 New Done Done true
2039 Acked 3038 Write 2 New Done Done true
2040 Acked 3039 Write 7 New Done Done true
2041 Acked 3040 Read 2 New Done Done true
2042 Acked 3041 Flush 0 New Done Done true
2043 Acked 3042 Flush 0 New Done Done true
2044 Acked 3043 Read 3 New Done Done true
2045 Acked 3044 Write 1 New Done Done true
2046 Acked 3045 Read 2 New Done Done true
2047 Acked 3046 Write 3 New Done Done true
2048 Acked 3047 Write 2 New Done Done true
2049 Acked 3048 Read 7 New Done Done true
2050 Acked 3049 Read 2 New Done Done true
2051 Acked 3050 Flush 0 New Done Done true
2052 Acked 3051 Flush 0 New Done Done true
2053 Acked 3052 Write 7 New Done Done true
2054 Acked 3053 Write 5 New Done Done true
2055 Acked 3054 Read 4 New Done Done true
2056 Acked 3055 Write 10 New Done Done true
2057 Acked 3056 Write 7 New Done Done true
2058 Acked 3057 Write 4 New Done Done true
2059 Acked 3058 Write 10 New Done Done true
2060 Acked 3059 Write 1 New Done Done true
2061 Acked 3060 Write 7 New Done Done true
2062 Acked 3061 Write 3 New Done Done true
2063 Acked 3062 Read 1 New Done Done true
2064 Acked 3063 Write 7 New Done Done true
2065 Acked 3064 Write 3 New Done Done true
2066 Acked 3065 Write 8 New Done Done true
2067 Acked 3066 Read 9 New Done Done true
2068 Acked 3067 Read 5 New Done Done true
2069 Acked 3068 Read 10 New Done Done true
2070 Acked 3069 Flush 0 New Done Done true
2071 Acked 3070 Write 8 New Done Done true
2072 Acked 3071 Write 5 New Done Done true
2073 Acked 3072 Read 7 New Done Done true
2074 Acked 3073 Write 4 New Done Done true
2075 Acked 3074 Write 6 New Done Done true
2076 Acked 3075 Write 6 New Done Done true
2077 Acked 3076 Read 1 New Done Done true
2078 Acked 3077 Write 4 New Done Done true
2079 Acked 3078 Read 2 New Done Done true
2080 Acked 3079 Read 4 New Done Done true
2081 Acked 3080 Write 7 New Done Done true
2082 Acked 3081 Read 7 New Done Done true
2083 Acked 3082 Write 6 New Done Done true
2084 Acked 3083 Write 7 New Done Done true
2085 Acked 3084 Read 2 New Done Done true
2086 Acked 3085 Read 9 New Done Done true
2087 Acked 3086 Read 4 New Done Done true
2088 Acked 3087 Read 1 New Done Done true
2089 Acked 3088 Write 9 New Done Done true
2090 Acked 3089 Write 3 New Done Done true
2091 Acked 3090 Write 8 New Done Done true
2092 Acked 3091 Write 7 New Done Done true
2093 Acked 3092 Write 2 New Done Done true
2094 Acked 3093 Read 10 New Done Done true
2095 Acked 3094 Read 5 New Done Done true
2096 Acked 3095 Write 3 New Done Done true
2097 Acked 3096 Read 6 New Done Done true
2098 Acked 3097 Write 4 New Done Done true
2099 Acked 3098 Write 5 New Done Done true
2100 Acked 3099 Read 5 New Done Done true
2101 Acked 3100 Write 3 New Done Done true
2102 Acked 3101 Read 2 New Done Done true
2103 Acked 3102 Read 6 New Done Done true
2104 Acked 3103 Write 6 New Done Done true
2105 Acked 3104 Write 3 New Done Done true
2106 Acked 3105 Write 8 New Done Done true
2107 Acked 3106 Write 2 New Done Done true
2108 Acked 3107 Read 8 New Done Done true
2109 Acked 3108 Flush 0 New Done Done true
STATES DS:0 DS:1 DS:2 TOTAL
New 301 0 0 301
Sent 0 0 0 0
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 2807 3108 3108
Downstairs last five completed: 2807 2806 2805 2804 2803
Upstairs last five completed: 2109 2108 2107 2106 2105
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:00.156552039Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:00.156712412Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:00.158045212Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:00.1580779Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 2807","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:00.15809765Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 2807","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:00.158270353Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:00.158313821Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:00.15834809Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:00.158381659Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:00.158404208Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 3108 3107 3106 3105 3104
Upstairs last five completed: 2109 2108 2107 2106 2105
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Feb 13 21:41:03.882 INFO request completed, [8810] Got stop action so:false kr:false
latency_us: 324, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 9b445278-ae12-41f8-9b6c-045c5166341f, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:03.882915068Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:03.882985785Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:03.883019804Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:03.883058912Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:41:03.883 INFO request completed, latency_us: {"msg":"[0] client re-new 0 jobs since flush 3108","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:03.883089821Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
160, response_code: 200, uri: /state/cid/0, method: GET, req_id: e57bc5ed-981f-4f82-8e22-3b3da15ce456, remote_addr:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:03.88311534Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
127.0.0.1:52124, local_addr: 127.0.0.1:9998
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:04.884383711Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:04.884439948Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:04.884464217Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:04.884485016Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:04.884506296Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 3108","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:04.884524415Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:04.884542254Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:05.886413551Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:05.886764217Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:05.886914522Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:05.886933771Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:05.88694715Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 3108","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:05.88696149Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:05.88697572Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:06.888024457Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:06.888083485Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:06.888105004Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:06.888133263Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:06.888149692Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 3108","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:06.888167052Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:06.888190121Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:41:07.884 INFO request completed, latency_us: 207, response_code: 200, uri: /state/cid/0, method: GET, req_id: d78723f2-804d-48dc-a145-e6c2b53fa708, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Read block 2028 len 20480
002/300 Read block 666 len 4096
003/300 Write block 3475 len 4096 data: 2
004/300 Flush
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:07.88844173Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:07.888510217Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:07.888547606Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:07.888580904Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:07.888597374Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 4 jobs since flush 3108","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:07.888611883Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:07.888625813Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
005/300 Read block 674 len 4096
006/300 Write block 1175 len 16384 data: 3 3 2 2
007/300 Read block 2799 len 32768
008/300 Write block 756 len 16384 data: 2 2 3 3
009/300 Write block 1567 len 8192 data: 2 3
010/300 Write block 459 len 32768 data: 2 2 4 5 4 4 5 5
011/300 Write block 3056 len 36864 data: 3 2 2 2 2 2 3 2 2
012/300 Read block 172 len 4096
013/300 Read block 1812 len 12288
014/300 Flush
015/300 Write block 34 len 4096 data: 3
016/300 Flush
017/300 Flush
018/300 Write block 2966 len 12288 data: 3 2 2
019/300 Write block 882 len 36864 data: 2 2 5 5 6 5 4 3 3
020/300 Write block 639 len 28672 data: 4 4 4 3 3 3 3
021/300 Read block 4643 len 20480
022/300 Read block 4436 len 8192
023/300 Write block 1773 len 32768 data: 4 4 4 4 4 4 2 2
024/300 Write block 3560 len 12288 data: 4 4 3
025/300 Read block 2845 len 4096
026/300 Read block 1972 len 16384
027/300 Read block 2593 len 4096
028/300 Write block 3530 len 20480 data: 3 3 3 3 3
029/300 Flush
030/300 Write block 2943 len 12288 data: 3 3 3
031/300 Write block 4129 len 16384 data: 4 4 4 3
032/300 Read block 4915 len 36864
033/300 Read block 1806 len 20480
034/300 Write block 2526 len 40960 data: 3 3 3 3 3 2 3 3 4 4
035/300 Write block 4876 len 16384 data: 2 2 2 2
036/300 Write block 2441 len 8192 data: 3 4
037/300 Flush
038/300 Read block 2179 len 4096
039/300 Flush
040/300 Read block 698 len 16384
041/300 Read block 1475 len 20480
042/300 Read block 2098 len 4096
043/300 Read block 749 len 16384
044/300 Write block 7 len 8192 data: 3 2
045/300 Read block 2427 len 40960
046/300 Write block 1118 len 24576 data: 2 2 2 3 3 3
047/300 Read block 999 len 32768
048/300 Write block 3368 len 24576 data: 3 3 3 3 3 3
049/300 Read block 194 len 24576
050/300 Write block 2142 len 8192 data: 4 3
051/300 Flush
052/300 Write block 4466 len 40960 data: 2 2 2 2 2 2 2 2 2 3
053/300 Read block 3037 len 8192
054/300 Write block 1101 len 24576 data: 2 2 4 5 5 5
055/300 Read block 2223 len 20480
056/300 Write block 3251 len 8192 data: 3 3
057/300 Read block 1320 len 28672
058/300 Write block 3027 len 24576 data: 2 2 2 2 2 2
059/300 Write block 1227 len 8192 data: 3 3
060/300 Read block 2903 len 28672
061/300 Read block 4518 len 28672
062/300 Write block 4897 len 8192 data: 2 2
063/300 Write block 2330 len 28672 data: 4 4 4 3 3 3 3
064/300 Write block 4961 len 28672 data: 2 2 2 2 2 2 2
065/300 Write block 4663 len 36864 data: 4 4 4 4 4 4 4 3 3
066/300 Write block 4535 len 4096 data: 4
067/300 Write block 1339 len 8192 data: 2 2
068/300 Read block 315 len 24576
069/300 Write block 1067 len 28672 data: 2 2 2 2 2 4 3
070/300 Flush
071/300 Read block 612 len 8192
072/300 Write block 1690 len 28672 data: 2 2 2 2 2 2 2
073/300 Write block 4008 len 32768 data: 3 3 2 2 2 2 3 3
074/300 Read block 3323 len 4096
075/300 Read block 559 len 32768
076/300 Read block 4830 len 40960
077/300 Read block 4092 len 16384
078/300 Write block 4600 len 12288 data: 3 2 2
079/300 Write block 951 len 32768 data: 2 2 2 2 2 2 2 2
080/300 Read block 162 len 20480
081/300 Read block 2408 len 24576
082/300 Flush
083/300 Read block 3980 len 40960
084/300 Flush
085/300 Write block 2855 len 32768 data: 3 2 2 2 2 2 2 2
086/300 Read block 4512 len 8192
087/300 Flush
088/300 Flush
089/300 Read block 4907 len 40960
090/300 Write block 4300 len 24576 data: 3 2 2 2 2 2
091/300 Write block 3525 len 36864 data: 2 2 2 2 2 4 4 4 4
092/300 Read block 1882 len 24576
093/300 Read block 4742 len 24576
094/300 Flush
095/300 Read block 3596 len 36864
096/300 Read block 2979 len 40960
097/300 Read block 4612 len 32768
098/300 Read block 2627 len 24576
099/300 Read block 4819 len 12288
100/300 Read block 2038 len 4096
101/300 Write block 3810 len 28672 data: 4 3 3 2 3 3 3
102/300 Read block 2020 len 28672
103/300 Read block 1663 len 16384
104/300 Read block 1870 len 24576
105/300 Write block 1638 len 36864 data: 2 2 3 3 3 3 5 4 4
106/300 Flush
107/300 Read block 1542 len 16384
108/300 Read block 1913 len 40960
109/300 Flush
110/300 Read block 3824 len 4096
111/300 Read block 427 len 4096
112/300 Write block 846 len 20480 data: 3 3 3 2 2
113/300 Write block 142 len 4096 data: 3
114/300 Read block 850 len 40960
115/300 Write block 1042 len 20480 data: 3 3 3 3 2
116/300 Write block 247 len 32768 data: 3 5 5 4 4 3 3 3
117/300 Read block 2751 len 28672
118/300 Write block 4623 len 8192 data: 3 3
119/300 Write block 3372 len 36864 data: 4 4 3 3 2 2 2 3 3
120/300 Read block 3310 len 36864
121/300 Read block 3352 len 24576
122/300 Write block 2284 len 8192 data: 4 4
123/300 Flush
124/300 Write block 3254 len 20480 data: 2 2 2 3 4
125/300 Write block 1998 len 32768 data: 4 4 4 3 2 3 3 4
126/300 Read block 3465 len 28672
127/300 Read block 2737 len 16384
128/300 Write block 1935 len 24576 data: 2 2 3 3 3 3
129/300 Read block 412 len 32768
130/300 Write block 1267 len 8192 data: 3 2
131/300 Write block 574 len 40960 data: 2 2 2 2 2 2 2 2 2 2
132/300 Write block 3256 len 12288 data: 3 4 5
133/300 Write block 1993 len 36864 data: 3 4 4 4 5 5 5 5 4
134/300 Flush
135/300 Write block 1120 len 8192 data: 3 4
136/300 Write block 2507 len 40960 data: 4 4 4 4 4 4 4 4 4 3
137/300 Read block 3756 len 4096
138/300 Write block 228 len 8192 data: 2 2
139/300 Write block 3850 len 40960 data: 2 3 3 3 3 3 3 3 3 3
140/300 Write block 1403 len 12288 data: 2 2 3
141/300 Write block 4098 len 12288 data: 2 2 2
142/300 Read block 4316 len 20480
143/300 Read block 1414 len 24576
144/300 Write block 2606 len 28672 data: 2 2 2 2 2 2 2
145/300 Write block 1045 len 8192 data: 4 3
146/300 Write block 2076 len 12288 data: 3 2 2
147/300 Read block 1652 len 28672
148/300 Write block 1574 len 12288 data: 3 3 3
149/300 Read block 487 len 24576
150/300 Read block 3185 len 32768
151/300 Read block 3836 len 28672
152/300 Write block 2934 len 36864 data: 2 2 2 3 3 3 3 3 3
153/300 Read block 2645 len 40960
154/300 Read block 1217 len 4096
155/300 Read block 4195 len 32768
156/300 Read block 2665 len 4096
157/300 Read block 4323 len 28672
158/300 Write block 2486 len 36864 data: 4 4 4 3 3 2 2 2 2
159/300 Read block 1127 len 16384
160/300 Write block 332 len 20480 data: 6 4 5 5 5
161/300 Read block 2047 len 4096
162/300 Read block 189 len 32768
163/300 Read block 2951 len 16384
164/300 Read block 3180 len 40960
165/300 Read block 4832 len 4096
166/300 Read block 2571 len 16384
167/300 Write block 2895 len 24576 data: 4 3 3 2 2 2
168/300 Write block 3033 len 24576 data: 3 3 3 3 3 3
169/300 Write block 4315 len 24576 data: 2 2 2 2 2 2
170/300 Read block 749 len 12288
171/300 Write block 1226 len 16384 data: 2 4 4 3
172/300 Write block 1354 len 12288 data: 4 4 5
173/300 Write block 4488 len 12288 data: 7 7 5
174/300 Read block 4750 len 12288
175/300 Read block 1647 len 24576
176/300 Write block 3106 len 40960 data: 4 4 4 3 3 3 2 2 2 2
177/300 Write block 1718 len 20480 data: 4 5 5 5 5
178/300 Write block 420 len 12288 data: 2 2 2
179/300 Flush
180/300 Write block 2836 len 20480 data: 5 5 5 4 4
181/300 Read block 89 len 36864
182/300 Write block 3537 len 28672 data: 3 3 2 2 2 2 2
183/300 Read block 3232 len 32768
184/300 Write block 4253 len 12288 data: 2 2 2
185/300 Read block 4592 len 24576
186/300 Write block 1468 len 4096 data: 2
187/300 Read block 1677 len 24576
188/300 Read block 2625 len 28672
189/300 Read block 3527 len 12288
190/300 Read block 4953 len 8192
191/300 Read block 694 len 8192
192/300 Write block 3895 len 28672 data: 4 4 3 3 3 3 3
193/300 Write block 1692 len 12288 data: 3 3 3
194/300 Read block 3376 len 32768
195/300 Read block 2768 len 4096
196/300 Read block 4948 len 12288
197/300 Write block 4943 len 24576 data: 4 4 4 5 4 5
198/300 Read block 145 len 20480
199/300 Read block 2385 len 32768
200/300 Flush
201/300 Write block 526 len 24576 data: 3 3 3 2 2 4
202/300 Write block 3557 len 4096 data: 4
203/300 Write block 35 len 12288 data: 3 3 3
204/300 Read block 1011 len 40960
205/300 Flush
206/300 Flush
207/300 Read block 1930 len 24576
208/300 Read block 2960 len 12288
209/300 Read block 2348 len 36864
210/300 Write block 2988 len 12288 data: 2 2 3
211/300 Read block 4323 len 8192
212/300 Write block 3276 len 24576 data: 3 3 3 3 3 4
213/300 Write block 1909 len 20480 data: 3 3 4 5 5
214/300 Read block 1555 len 40960
215/300 Read block 3514 len 36864
216/300 Write block 4659 len 12288 data: 3 3 3
217/300 Write block 4727 len 28672 data: 2 3 4 4 4 4 4
218/300 Read block 1850 len 8192
219/300 Write block 1943 len 28672 data: 3 2 2 3 3 3 3
220/300 Read block 4305 len 16384
221/300 Flush
222/300 Read block 2308 len 4096
223/300 Flush
224/300 Read block 3207 len 36864
225/300 Write block 3851 len 40960 data: 4 4 4 4 4 4 4 4 4 3
226/300 Write block 3604 len 24576 data: 2 2 2 3 3 3
227/300 Write block 2017 len 8192 data: 5 4
228/300 Read block 4050 len 12288
229/300 Write block 84 len 36864 data: 3 3 3 3 2 2 3 4 4
230/300 Read block 3173 len 20480
231/300 Write block 839 len 40960 data: 2 2 2 3 3 3 3 4 4 4
232/300 Read block 2995 len 12288
233/300 Read block 4962 len 12288
234/300 Write block 2946 len 12288 data: 2 2 2
235/300 Read block 4635 len 16384
236/300 Write block 1266 len 12288 data: 3 4 3
237/300 Write block 1250 len 12288 data: 4 4 4
238/300 Write block 4838 len 4096 data: 3
239/300 Flush
240/300 Write block 4228 len 24576 data: 3 3 4 3 3 2
241/300 Read block 3888 len 40960
242/300 Write block 709 len 32768 data: 3 4 4 5 5 5 4 4
243/300 Write block 4075 len 4096 data: 2
244/300 Read block 1682 len 24576
245/300 Write block 497 len 28672 data: 2 2 2 2 2 2 3
246/300 Read block 3840 len 32768
247/300 Flush
248/300 Read block 3216 len 20480
249/300 Read block 2626 len 8192
250/300 Write block 4390 len 20480 data: 3 3 2 2 2
251/300 Write block 764 len 16384 data: 2 2 2 3
252/300 Read block 470 len 4096
253/300 Write block 4646 len 28672 data: 2 2 2 2 3 3 2
254/300 Read block 2647 len 16384
255/300 Read block 304 len 24576
256/300 Read block 873 len 40960
257/300 Read block 1087 len 36864
258/300 Write block 2974 len 16384 data: 2 2 2 2
259/300 Read block 3264 len 40960
260/300 Write block 680 len 20480 data: 3 3 3 5 5
261/300 Write block 4040 len 20480 data: 4 4 3 3 3
262/300 Write block 1202 len 4096 data: 3
263/300 Read block 524 len 4096
264/300 Write block 875 len 4096 data: 3
265/300 Write block 4686 len 12288 data: 2 2 2
266/300 Read block 4465 len 36864
267/300 Write block 3119 len 24576 data: 4 4 4 4 4 5
268/300 Write block 2085 len 16384 data: 3 3 3 3
269/300 Read block 1992 len 28672
270/300 Write block 1820 len 28672 data: 4 4 4 4 3 3 3
271/300 Flush
272/300 Write block 2194 len 12288 data: 2 2 2
273/300 Flush
274/300 Read block 1043 len 36864
275/300 Write block 18 len 40960 data: 3 4 4 3 3 3 2 2 2 2
276/300 Read block 1405 len 12288
277/300 Read block 632 len 20480
278/300 Write block 3196 len 12288 data: 4 5 5
279/300 Read block 4464 len 16384
280/300 Read block 4039 len 20480
281/300 Read block 112 len 8192
282/300 Write block 2038 len 12288 data: 2 2 2
283/300 Write block 2211 len 4096 data: 2
284/300 Write block 4184 len 40960 data: 4 4 4 3 2 2 2 2 2 2
285/300 Write block 3068 len 16384 data: 2 2 2 2
286/300 Read block 842 len 24576
287/300 Read block 1048 len 40960
288/300 Write block 1019 len 4096 data: 2
289/300 Read block 2687 len 12288
290/300 Read block 4797 len 32768
291/300 Read block 2937 len 36864
292/300 Read block 2858 len 8192
293/300 Read block 620 len 32768
294/300 Flush
295/300 Write block 3027 len 36864 data: 3 3 3 3 3 3 4 4 4
296/300 Read block 2621 len 24576
297/300 Read block 931 len 36864
298/300 Flush
299/300 Write block 4420 len 20480 data: 3 3 3 2 2
300/300 Write block 2109 len 28672 data: 3 2 2 2 2 2 2
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 13 21:41:08.762 INFO request completed, latency_us: 204, response_code: 204, uri: /start/cid/0, method: POST, req_id: 1c2e4592-a6c6-4711-b91b-39db3af4e241, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
2110 Acked 3109 Read 5 New Done Done true
2111 Acked 3110 Read 1 New Done Done true
2112 Acked 3111 Write 1 New Done Done true
2113 Acked 3112 Flush 0 New Done Done true
2114 Acked 3113 Read 1 New Done Done false
2115 Acked 3114 Write 4 New Done Done false
2116 Acked 3115 Read 8 New Done Done false
2117 Acked 3116 Write 4 New Done Done false
2118 Acked 3117 Write 2 New Done Done false
2119 Acked 3118 Write 8 New Done Done false
2120 Acked 3119 Write 9 New Done Done false
2121 Acked 3120 Read 1 New Done Done false
2122 Acked 3121 Read 3 New Done Done false
2123 Acked 3122 Flush 0 New Done Done false
2124 Acked 3123 Write 1 New Done Done false
2125 Acked 3124 Flush 0 New Done Done false
2126 Acked 3125 Flush 0 New Done Done false
2127 Acked 3126 Write 3 New Done Done false
2128 Acked 3127 Write 9 New Done Done false
2129 Acked 3128 Write 7 New Done Done false
2130 Acked 3129 Read 5 New Done Done false
2131 Acked 3130 Read 2 New Done Done false
2132 Acked 3131 Write 8 New Done Done false
2133 Acked 3132 Write 3 New Done Done false
2134 Acked 3133 Read 1 New Done Done false
2135 Acked 3134 Read 4 New Done Done false
2136 Acked 3135 Read 1 New Done Done false
2137 Acked 3136 Write 5 New Done Done false
2138 Acked 3137 Flush 0 New Done Done false
2139 Acked 3138 Write 3 New Done Done false
2140 Acked 3139 Write 4 New Done Done false
2141 Acked 3140 Read 9 New Done Done false
2142 Acked 3141 Read 5 New Done Done false
2143 Acked 3142 Write 10 New Done Done false
2144 Acked 3143 Write 4 New Done Done false
2145 Acked 3144 Write 2 New Done Done false
2146 Acked 3145 Flush 0 New Done Done false
2147 Acked 3146 Read 1 New Done Done false
2148 Acked 3147 Flush 0 New Done Done false
2149 Acked 3148 Read 4 New Done Done false
2150 Acked 3149 Read 5 New Done Done false
2151 Acked 3150 Read 1 New Done Done false
2152 Acked 3151 Read 4 New Done Done false
2153 Acked 3152 Write 2 New Done Done false
2154 Acked 3153 Read 10 New Done Done false
2155 Acked 3154 Write 6 New Done Done false
2156 Acked 3155 Read 8 New Done Done false
2157 Acked 3156 Write 6 New Done Done false
2158 Acked 3157 Read 6 New Done Done false
2159 Acked 3158 Write 2 New Done Done false
2160 Acked 3159 Flush 0 New Done Done false
2161 Acked 3160 Write 10 New Done Done false
2162 Acked 3161 Read 2 New Done Done false
2163 Acked 3162 Write 6 New Done Done false
2164 Acked 3163 Read 5 New Done Done false
2165 Acked 3164 Write 2 New Done Done false
2166 Acked 3165 Read 7 New Done Done false
2167 Acked 3166 Write 6 New Done Done false
2168 Acked 3167 Write 2 New Done Done false
2169 Acked 3168 Read 7 New Done Done false
2170 Acked 3169 Read 7 New Done Done false
2171 Acked 3170 Write 2 New Done Done false
2172 Acked 3171 Write 7 New Done Done false
2173 Acked 3172 Write 7 New Done Done false
2174 Acked 3173 Write 9 New Done Done false
2175 Acked 3174 Write 1 New Done Done false
2176 Acked 3175 Write 2 New Done Done false
2177 Acked 3176 Read 6 New Done Done false
2178 Acked 3177 Write 7 New Done Done false
2179 Acked 3178 Flush 0 New Done Done false
2180 Acked 3179 Read 2 New Done Done false
2181 Acked 3180 Write 7 New Done Done false
2182 Acked 3181 Write 8 New Done Done false
2183 Acked 3182 Flush 0 New Done Done false
2184 Acked 3183 Read 1 New Done Done false
2185 Acked 3184 Read 8 New Done Done false
2186 Acked 3185 Read 10 New Done Done false
2187 Acked 3186 Read 4 New Done Done false
2188 Acked 3187 Write 3 New Done Done false
2189 Acked 3188 Write 8 New Done Done false
2190 Acked 3189 Read 5 New Done Done false
2191 Acked 3190 Read 6 New Done Done false
2192 Acked 3191 Flush 0 New Done Done false
2193 Acked 3192 Read 10 New Done Done false
2194 Acked 3193 Flush 0 New Done Done false
2195 Acked 3194 Write 8 New Done Done false
2196 Acked 3195 Read 2 New Done Done false
2197 Acked 3196 Flush 0 New Done Done false
2198 Acked 3197 Flush 0 New Done Done false
2199 Acked 3198 Read 10 New Done Done false
2200 Acked 3199 Write 6 New Done Done false
2201 Acked 3200 Write 9 New Done Done false
2202 Acked 3201 Read 6 New Done Done false
2203 Acked 3202 Read 6 New Done Done false
2204 Acked 3203 Flush 0 New Done Done false
2205 Acked 3204 Read 9 New Done Done false
2206 Acked 3205 Read 10 New Done Done false
2207 Acked 3206 Read 8 New Done Done false
2208 Acked 3207 Read 6 New Done Done false
2209 Acked 3208 Read 3 New Done Done false
2210 Acked 3209 Read 1 New Done Done false
2211 Acked 3210 Write 7 New Done Done false
2212 Acked 3211 Read 7 New Done Done false
2213 Acked 3212 Read 4 New Done Done false
2214 Acked 3213 Read 6 New Done Done false
2215 Acked 3214 Write 9 New Done Done false
2216 Acked 3215 Flush 0 New Done Done false
2217 Acked 3216 Read 4 New Done Done false
2218 Acked 3217 Read 10 New Done Done false
2219 Acked 3218 Flush 0 New Done Done false
2220 Acked 3219 Read 1 New Done Done false
2221 Acked 3220 Read 1 New Done Done false
2222 Acked 3221 Write 5 New Done Done false
2223 Acked 3222 Write 1 New Done Done false
2224 Acked 3223 Read 10 New Done Done false
2225 Acked 3224 Write 5 New Done Done false
2226 Acked 3225 Write 8 New Done Done false
2227 Acked 3226 Read 7 New Done Done false
2228 Acked 3227 Write 2 New Done Done false
2229 Acked 3228 Write 9 New Done Done false
2230 Acked 3229 Read 9 New Done Done false
2231 Acked 3230 Read 6 New Done Done false
2232 Acked 3231 Write 2 New Done Done false
2233 Acked 3232 Flush 0 New Done Done false
2234 Acked 3233 Write 5 New Done Done false
2235 Acked 3234 Write 8 New Done Done false
2236 Acked 3235 Read 7 New Done Done false
2237 Acked 3236 Read 4 New Done Done false
2238 Acked 3237 Write 6 New Done Done false
2239 Acked 3238 Read 8 New Done Done false
2240 Acked 3239 Write 2 New Done Done false
2241 Acked 3240 Write 10 New Done Done false
2242 Acked 3241 Write 3 New Done Done false
2243 Acked 3242 Write 9 New Done Done false
2244 Acked 3243 Flush 0 New Done Done false
2245 Acked 3244 Write 2 New Done Done false
2246 Acked 3245 Write 10 New Done Done false
2247 Acked 3246 Read 1 New Done Done false
2248 Acked 3247 Write 2 New Done Done false
2249 Acked 3248 Write 10 New Done Done false
2250 Acked 3249 Write 3 New Done Done false
2251 Acked 3250 Write 3 New Done Done false
2252 Acked 3251 Read 5 New Done Done false
2253 Acked 3252 Read 6 New Done Done false
2254 Acked 3253 Write 7 New Done Done false
2255 Acked 3254 Write 2 New Done Done false
2256 Acked 3255 Write 3 New Done Done false
2257 Acked 3256 Read 7 New Done Done false
2258 Acked 3257 Write 3 New Done Done false
2259 Acked 3258 Read 6 New Done Done false
2260 Acked 3259 Read 8 New Done Done false
2261 Acked 3260 Read 7 New Done Done false
2262 Acked 3261 Write 9 New Done Done false
2263 Acked 3262 Read 10 New Done Done false
2264 Acked 3263 Read 1 New Done Done false
2265 Acked 3264 Read 8 New Done Done false
2266 Acked 3265 Read 1 New Done Done false
2267 Acked 3266 Read 7 New Done Done false
2268 Acked 3267 Write 9 New Done Done false
2269 Acked 3268 Read 4 New Done Done false
2270 Acked 3269 Write 5 New Done Done false
2271 Acked 3270 Read 1 New Done Done false
2272 Acked 3271 Read 8 New Done Done false
2273 Acked 3272 Read 4 New Done Done false
2274 Acked 3273 Read 10 New Done Done false
2275 Acked 3274 Read 1 New Done Done false
2276 Acked 3275 Read 4 New Done Done false
2277 Acked 3276 Write 6 New Done Done false
2278 Acked 3277 Write 6 New Done Done false
2279 Acked 3278 Write 6 New Done Done false
2280 Acked 3279 Read 3 New Done Done false
2281 Acked 3280 Write 4 New Done Done false
2282 Acked 3281 Write 3 New Done Done false
2283 Acked 3282 Write 3 New Done Done false
2284 Acked 3283 Read 3 New Done Done false
2285 Acked 3284 Read 6 New Done Done false
2286 Acked 3285 Write 10 New Done Done false
2287 Acked 3286 Write 5 New Done Done false
2288 Acked 3287 Write 3 New Done Done false
2289 Acked 3288 Flush 0 New Done Done false
2290 Acked 3289 Write 5 New Done Done false
2291 Acked 3290 Read 9 New Done Done false
2292 Acked 3291 Write 7 New Done Done false
2293 Acked 3292 Read 8 New Done Done false
2294 Acked 3293 Write 3 New Done Done false
2295 Acked 3294 Read 6 New Done Done false
2296 Acked 3295 Write 1 New Done Done false
2297 Acked 3296 Read 6 New Done Done false
2298 Acked 3297 Read 7 New Done Done false
2299 Acked 3298 Read 3 New Done Done false
2300 Acked 3299 Read 2 New Done Done false
2301 Acked 3300 Read 2 New Done Done false
2302 Acked 3301 Write 7 New Done Done false
2303 Acked 3302 Write 3 New Done Done false
2304 Acked 3303 Read 8 New Done Done false
2305 Acked 3304 Read 1 New Done Done false
2306 Acked 3305 Read 3 New Done Done false
2307 Acked 3306 Write 6 New Done Done false
2308 Acked 3307 Read 5 New Done Done false
2309 Acked 3308 Read 8 New Done Done false
2310 Acked 3309 Flush 0 New Done Done false
2311 Acked 3310 Write 6 New Done Done false
2312 Acked 3311 Write 1 New Done Done false
2313 Acked 3312 Write 3 New Done Done false
2314 Acked 3313 Read 10 New Done Done false
2315 Acked 3314 Flush 0 New Done Done false
2316 Acked 3315 Flush 0 New Done Done false
2317 Acked 3316 Read 6 New Done Done false
2318 Acked 3317 Read 3 New Done Done false
2319 Acked 3318 Read 9 New Done Done false
2320 Acked 3319 Write 3 New Done Done false
2321 Acked 3320 Read 2 New Done Done false
2322 Acked 3321 Write 6 New Done Done false
2323 Acked 3322 Write 5 New Done Done false
2324 Acked 3323 Read 10 New Done Done false
2325 Acked 3324 Read 9 New Done Done false
2326 Acked 3325 Write 3 New Done Done false
2327 Acked 3326 Write 7 New Done Done false
2328 Acked 3327 Read 2 New Done Done false
2329 Acked 3328 Write 7 New Done Done false
2330 Acked 3329 Read 4 New Done Done false
2331 Acked 3330 Flush 0 New Done Done false
2332 Acked 3331 Read 1 New Done Done false
2333 Acked 3332 Flush 0 New Done Done false
2334 Acked 3333 Read 9 New Done Done false
2335 Acked 3334 Write 10 New Done Done false
2336 Acked 3335 Write 6 New Done Done false
2337 Acked 3336 Write 2 New Done Done false
2338 Acked 3337 Read 3 New Done Done false
2339 Acked 3338 Write 9 New Done Done false
2340 Acked 3339 Read 5 New Done Done false
2341 Acked 3340 Write 10 New Done Done false
2342 Acked 3341 Read 3 New Done Done false
2343 Acked 3342 Read 3 New Done Done false
2344 Acked 3343 Write 3 New Done Done false
2345 Acked 3344 Read 4 New Done Done false
2346 Acked 3345 Write 3 New Done Done false
2347 Acked 3346 Write 3 New Done Done false
2348 Acked 3347 Write 1 New Done Done false
2349 Acked 3348 Flush 0 New Done Done false
2350 Acked 3349 Write 6 New Done Done false
2351 Acked 3350 Read 10 New Done Done false
2352 Acked 3351 Write 8 New Done Done false
2353 Acked 3352 Write 1 New Done Done false
2354 Acked 3353 Read 6 New Done Done false
2355 Acked 3354 Write 7 New Done Done false
2356 Acked 3355 Read 8 New Done Done false
2357 Acked 3356 Flush 0 New Done Done false
2358 Acked 3357 Read 5 New Done Done false
2359 Acked 3358 Read 2 New Done Done false
2360 Acked 3359 Write 5 New Done Done false
2361 Acked 3360 Write 4 New Done Done false
2362 Acked 3361 Read 1 New Done Done false
2363 Acked 3362 Write 7 New Done Done false
2364 Acked 3363 Read 4 New Done Done false
2365 Acked 3364 Read 6 New Done Done false
2366 Acked 3365 Read 10 New Done Done false
2367 Acked 3366 Read 9 New Done Done false
2368 Acked 3367 Write 4 New Done Done false
2369 Acked 3368 Read 10 New Done Done false
2370 Acked 3369 Write 5 New Done Done false
2371 Acked 3370 Write 5 New Done Done false
2372 Acked 3371 Write 1 New Done Done false
2373 Acked 3372 Read 1 New Done Done false
2374 Acked 3373 Write 1 New Done Done false
2375 Acked 3374 Write 3 New Done Done false
2376 Acked 3375 Read 9 New Done Done false
2377 Acked 3376 Write 6 New Done Done false
2378 Acked 3377 Write 4 New Done Done false
2379 Acked 3378 Read 7 New Done Done false
2380 Acked 3379 Write 7 New Done Done false
2381 Acked 3380 Flush 0 New Done Done false
2382 Acked 3381 Write 3 New Done Done false
2383 Acked 3382 Flush 0 New Done Done false
2384 Acked 3383 Read 9 New Done Done false
2385 Acked 3384 Write 10 New Done Done false
2386 Acked 3385 Read 3 New Done Done false
2387 Acked 3386 Read 5 New Done Done false
2388 Acked 3387 Write 3 New Done Done false
2389 Acked 3388 Read 4 New Done Done false
2390 Acked 3389 Read 5 New Done Done false
2391 Acked 3390 Read 2 New Done Done false
2392 Acked 3391 Write 3 New Done Done false
2393 Acked 3392 Write 1 New Done Done false
2394 Acked 3393 Write 10 New Done Done false
2395 Acked 3394 Write 4 New Done Done false
2396 Acked 3395 Read 6 New Done Done false
2397 Acked 3396 Read 10 New Done Done false
2398 Acked 3397 Write 1 New Done Done false
2399 Acked 3398 Read 3 New Done Done false
2400 Acked 3399 Read 8 New Done Done false
2401 Acked 3400 Read 9 New Done Done false
2402 Acked 3401 Read 2 New Done Done false
2403 Acked 3402 Read 8 New Done Done false
2404 Acked 3403 Flush 0 New Done Done false
2405 Acked 3404 Write 9 New Done Done false
2406 Acked 3405 Read 6 New Done Done false
2407 Acked 3406 Read 9 New Done Done false
2408 Acked 3407 Flush 0 New Done Done false
2409 Acked 3408 Write 5 New Done Done false
2410 Acked 3409 Write 7 New Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
New 301 0 0 301
Sent 0 0 0 0
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 3108 3407 3407
Downstairs last five completed: 3108 3107 3106 3105 3104
Upstairs last five completed: 2410 2409 2408 2407 2406
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:08.889883383Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:08.889940581Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:08.88996439Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:08.889985859Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:08.890002099Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 3108","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:08.890019898Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:08.890124584Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:09.89097708Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:09.891051047Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:09.891075396Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:09.891096545Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:09.891112604Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 3108","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:09.891130294Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:09.891252869Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:10.891876843Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:10.89197096Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:10.892012188Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:10.892050197Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:10.892079795Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 3108","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:10.892112054Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:10.892354985Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] got start action
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:11.892917463Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] I am going to restart
Starting downstairs at port 8810
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:11.89297899Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Make output file at "/tmp/dsc/downstairs-8810.txt"
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:11.893001469Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:11.893021579Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810][0] reports Starting
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:11.893036948Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 3108","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:11.893059877Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:11.893158013Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1172)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
2110 Acked 3109 Read 5 New Done Done true
2111 Acked 3110 Read 1 New Done Done true
2112 Acked 3111 Write 1 New Done Done true
2113 Acked 3112 Flush 0 New Done Done true
2114 Acked 3113 Read 1 New Done Done true
2115 Acked 3114 Write 4 New Done Done true
2116 Acked 3115 Read 8 New Done Done true
2117 Acked 3116 Write 4 New Done Done true
2118 Acked 3117 Write 2 New Done Done true
2119 Acked 3118 Write 8 New Done Done true
2120 Acked 3119 Write 9 New Done Done true
2121 Acked 3120 Read 1 New Done Done true
2122 Acked 3121 Read 3 New Done Done true
2123 Acked 3122 Flush 0 New Done Done true
2124 Acked 3123 Write 1 New Done Done true
2125 Acked 3124 Flush 0 New Done Done true
2126 Acked 3125 Flush 0 New Done Done true
2127 Acked 3126 Write 3 New Done Done true
2128 Acked 3127 Write 9 New Done Done true
2129 Acked 3128 Write 7 New Done Done true
2130 Acked 3129 Read 5 New Done Done true
2131 Acked 3130 Read 2 New Done Done true
2132 Acked 3131 Write 8 New Done Done true
2133 Acked 3132 Write 3 New Done Done true
2134 Acked 3133 Read 1 New Done Done true
2135 Acked 3134 Read 4 New Done Done true
2136 Acked 3135 Read 1 New Done Done true
2137 Acked 3136 Write 5 New Done Done true
2138 Acked 3137 Flush 0 New Done Done true
2139 Acked 3138 Write 3 New Done Done true
2140 Acked 3139 Write 4 New Done Done true
2141 Acked 3140 Read 9 New Done Done true
2142 Acked 3141 Read 5 New Done Done true
2143 Acked 3142 Write 10 New Done Done true
2144 Acked 3143 Write 4 New Done Done true
2145 Acked 3144 Write 2 New Done Done true
2146 Acked 3145 Flush 0 New Done Done true
2147 Acked 3146 Read 1 New Done Done true
2148 Acked 3147 Flush 0 New Done Done true
2149 Acked 3148 Read 4 New Done Done true
2150 Acked 3149 Read 5 New Done Done true
2151 Acked 3150 Read 1 New Done Done true
2152 Acked 3151 Read 4 New Done Done true
2153 Acked 3152 Write 2 New Done Done true
2154 Acked 3153 Read 10 New Done Done true
2155 Acked 3154 Write 6 New Done Done true
2156 Acked 3155 Read 8 New Done Done true
2157 Acked 3156 Write 6 New Done Done true
2158 Acked 3157 Read 6 New Done Done true
2159 Acked 3158 Write 2 New Done Done true
2160 Acked 3159 Flush 0 New Done Done true
2161 Acked 3160 Write 10 New Done Done true
2162 Acked 3161 Read 2 New Done Done true
2163 Acked 3162 Write 6 New Done Done true
2164 Acked 3163 Read 5 New Done Done true
2165 Acked 3164 Write 2 New Done Done true
2166 Acked 3165 Read 7 New Done Done true
2167 Acked 3166 Write 6 New Done Done true
2168 Acked 3167 Write 2 New Done Done true
2169 Acked 3168 Read 7 New Done Done true
2170 Acked 3169 Read 7 New Done Done true
2171 Acked 3170 Write 2 New Done Done true
2172 Acked 3171 Write 7 New Done Done true
2173 Acked 3172 Write 7 New Done Done true
2174 Acked 3173 Write 9 New Done Done true
2175 Acked 3174 Write 1 New Done Done true
2176 Acked 3175 Write 2 New Done Done true
2177 Acked 3176 Read 6 New Done Done true
2178 Acked 3177 Write 7 New Done Done true
2179 Acked 3178 Flush 0 New Done Done true
2180 Acked 3179 Read 2 New Done Done true
2181 Acked 3180 Write 7 New Done Done true
2182 Acked 3181 Write 8 New Done Done true
2183 Acked 3182 Flush 0 New Done Done true
2184 Acked 3183 Read 1 New Done Done true
2185 Acked 3184 Read 8 New Done Done true
2186 Acked 3185 Read 10 New Done Done true
2187 Acked 3186 Read 4 New Done Done true
2188 Acked 3187 Write 3 New Done Done true
2189 Acked 3188 Write 8 New Done Done true
2190 Acked 3189 Read 5 New Done Done true
2191 Acked 3190 Read 6 New Done Done true
2192 Acked 3191 Flush 0 New Done Done true
2193 Acked 3192 Read 10 New Done Done true
2194 Acked 3193 Flush 0 New Done Done true
2195 Acked 3194 Write 8 New Done Done true
2196 Acked 3195 Read 2 New Done Done true
2197 Acked 3196 Flush 0 New Done Done true
2198 Acked 3197 Flush 0 New Done Done true
2199 Acked 3198 Read 10 New Done Done true
2200 Acked 3199 Write 6 New Done Done true
2201 Acked 3200 Write 9 New Done Done true
2202 Acked 3201 Read 6 New Done Done true
2203 Acked 3202 Read 6 New Done Done true
2204 Acked 3203 Flush 0 New Done Done true
2205 Acked 3204 Read 9 New Done Done true
2206 Acked 3205 Read 10 New Done Done true
2207 Acked 3206 Read 8 New Done Done true
2208 Acked 3207 Read 6 New Done Done true
2209 Acked 3208 Read 3 New Done Done true
2210 Acked 3209 Read 1 New Done Done true
2211 Acked 3210 Write 7 New Done Done true
2212 Acked 3211 Read 7 New Done Done true
2213 Acked 3212 Read 4 New Done Done true
2214 Acked 3213 Read 6 New Done Done true
2215 Acked 3214 Write 9 New Done Done true
2216 Acked 3215 Flush 0 New Done Done true
2217 Acked 3216 Read 4 New Done Done true
2218 Acked 3217 Read 10 New Done Done true
2219 Acked 3218 Flush 0 New Done Done true
2220 Acked 3219 Read 1 New Done Done true
2221 Acked 3220 Read 1 New Done Done true
2222 Acked 3221 Write 5 New Done Done true
2223 Acked 3222 Write 1 New Done Done true
2224 Acked 3223 Read 10 New Done Done true
2225 Acked 3224 Write 5 New Done Done true
2226 Acked 3225 Write 8 New Done Done true
2227 Acked 3226 Read 7 New Done Done true
2228 Acked 3227 Write 2 New Done Done true
2229 Acked 3228 Write 9 New Done Done true
2230 Acked 3229 Read 9 New Done Done true
2231 Acked 3230 Read 6 New Done Done true
2232 Acked 3231 Write 2 New Done Done true
2233 Acked 3232 Flush 0 New Done Done true
2234 Acked 3233 Write 5 New Done Done true
2235 Acked 3234 Write 8 New Done Done true
2236 Acked 3235 Read 7 New Done Done true
2237 Acked 3236 Read 4 New Done Done true
2238 Acked 3237 Write 6 New Done Done true
2239 Acked 3238 Read 8 New Done Done true
2240 Acked 3239 Write 2 New Done Done true
2241 Acked 3240 Write 10 New Done Done true
2242 Acked 3241 Write 3 New Done Done true
2243 Acked 3242 Write 9 New Done Done true
2244 Acked 3243 Flush 0 New Done Done true
2245 Acked 3244 Write 2 New Done Done true
2246 Acked 3245 Write 10 New Done Done true
2247 Acked 3246 Read 1 New Done Done true
2248 Acked 3247 Write 2 New Done Done true
2249 Acked 3248 Write 10 New Done Done true
2250 Acked 3249 Write 3 New Done Done true
2251 Acked 3250 Write 3 New Done Done true
2252 Acked 3251 Read 5 New Done Done true
2253 Acked 3252 Read 6 New Done Done true
2254 Acked 3253 Write 7 New Done Done true
2255 Acked 3254 Write 2 New Done Done true
2256 Acked 3255 Write 3 New Done Done true
2257 Acked 3256 Read 7 New Done Done true
2258 Acked 3257 Write 3 New Done Done true
2259 Acked 3258 Read 6 New Done Done true
2260 Acked 3259 Read 8 New Done Done true
2261 Acked 3260 Read 7 New Done Done true
2262 Acked 3261 Write 9 New Done Done true
2263 Acked 3262 Read 10 New Done Done true
2264 Acked 3263 Read 1 New Done Done true
2265 Acked 3264 Read 8 New Done Done true
2266 Acked 3265 Read 1 New Done Done true
2267 Acked 3266 Read 7 New Done Done true
2268 Acked 3267 Write 9 New Done Done true
2269 Acked 3268 Read 4 New Done Done true
2270 Acked 3269 Write 5 New Done Done true
2271 Acked 3270 Read 1 New Done Done true
2272 Acked 3271 Read 8 New Done Done true
2273 Acked 3272 Read 4 New Done Done true
2274 Acked 3273 Read 10 New Done Done true
2275 Acked 3274 Read 1 New Done Done true
2276 Acked 3275 Read 4 New Done Done true
2277 Acked 3276 Write 6 New Done Done true
2278 Acked 3277 Write 6 New Done Done true
2279 Acked 3278 Write 6 New Done Done true
2280 Acked 3279 Read 3 New Done Done true
2281 Acked 3280 Write 4 New Done Done true
2282 Acked 3281 Write 3 New Done Done true
2283 Acked 3282 Write 3 New Done Done true
2284 Acked 3283 Read 3 New Done Done true
2285 Acked 3284 Read 6 New Done Done true
2286 Acked 3285 Write 10 New Done Done true
2287 Acked 3286 Write 5 New Done Done true
2288 Acked 3287 Write 3 New Done Done true
2289 Acked 3288 Flush 0 New Done Done true
2290 Acked 3289 Write 5 New Done Done true
2291 Acked 3290 Read 9 New Done Done true
2292 Acked 3291 Write 7 New Done Done true
2293 Acked 3292 Read 8 New Done Done true
2294 Acked 3293 Write 3 New Done Done true
2295 Acked 3294 Read 6 New Done Done true
2296 Acked 3295 Write 1 New Done Done true
2297 Acked 3296 Read 6 New Done Done true
2298 Acked 3297 Read 7 New Done Done true
2299 Acked 3298 Read 3 New Done Done true
2300 Acked 3299 Read 2 New Done Done true
2301 Acked 3300 Read 2 New Done Done true
2302 Acked 3301 Write 7 New Done Done true
2303 Acked 3302 Write 3 New Done Done true
2304 Acked 3303 Read 8 New Done Done true
2305 Acked 3304 Read 1 New Done Done true
2306 Acked 3305 Read 3 New Done Done true
2307 Acked 3306 Write 6 New Done Done true
2308 Acked 3307 Read 5 New Done Done true
2309 Acked 3308 Read 8 New Done Done true
2310 Acked 3309 Flush 0 New Done Done true
2311 Acked 3310 Write 6 New Done Done true
2312 Acked 3311 Write 1 New Done Done true
2313 Acked 3312 Write 3 New Done Done true
2314 Acked 3313 Read 10 New Done Done true
2315 Acked 3314 Flush 0 New Done Done true
2316 Acked 3315 Flush 0 New Done Done true
2317 Acked 3316 Read 6 New Done Done true
2318 Acked 3317 Read 3 New Done Done true
2319 Acked 3318 Read 9 New Done Done true
2320 Acked 3319 Write 3 New Done Done true
2321 Acked 3320 Read 2 New Done Done true
2322 Acked 3321 Write 6 New Done Done true
2323 Acked 3322 Write 5 New Done Done true
2324 Acked 3323 Read 10 New Done Done true
2325 Acked 3324 Read 9 New Done Done true
2326 Acked 3325 Write 3 New Done Done true
2327 Acked 3326 Write 7 New Done Done true
2328 Acked 3327 Read 2 New Done Done true
2329 Acked 3328 Write 7 New Done Done true
2330 Acked 3329 Read 4 New Done Done true
2331 Acked 3330 Flush 0 New Done Done true
2332 Acked 3331 Read 1 New Done Done true
2333 Acked 3332 Flush 0 New Done Done true
2334 Acked 3333 Read 9 New Done Done true
2335 Acked 3334 Write 10 New Done Done true
2336 Acked 3335 Write 6 New Done Done true
2337 Acked 3336 Write 2 New Done Done true
2338 Acked 3337 Read 3 New Done Done true
2339 Acked 3338 Write 9 New Done Done true
2340 Acked 3339 Read 5 New Done Done true
2341 Acked 3340 Write 10 New Done Done true
2342 Acked 3341 Read 3 New Done Done true
2343 Acked 3342 Read 3 New Done Done true
2344 Acked 3343 Write 3 New Done Done true
2345 Acked 3344 Read 4 New Done Done true
2346 Acked 3345 Write 3 New Done Done true
2347 Acked 3346 Write 3 New Done Done true
2348 Acked 3347 Write 1 New Done Done true
2349 Acked 3348 Flush 0 New Done Done true
2350 Acked 3349 Write 6 New Done Done true
2351 Acked 3350 Read 10 New Done Done true
2352 Acked 3351 Write 8 New Done Done true
2353 Acked 3352 Write 1 New Done Done true
2354 Acked 3353 Read 6 New Done Done true
2355 Acked 3354 Write 7 New Done Done true
2356 Acked 3355 Read 8 New Done Done true
2357 Acked 3356 Flush 0 New Done Done true
2358 Acked 3357 Read 5 New Done Done true
2359 Acked 3358 Read 2 New Done Done true
2360 Acked 3359 Write 5 New Done Done true
2361 Acked 3360 Write 4 New Done Done true
2362 Acked 3361 Read 1 New Done Done true
2363 Acked 3362 Write 7 New Done Done true
2364 Acked 3363 Read 4 New Done Done true
2365 Acked 3364 Read 6 New Done Done true
2366 Acked 3365 Read 10 New Done Done true
2367 Acked 3366 Read 9 New Done Done true
2368 Acked 3367 Write 4 New Done Done true
2369 Acked 3368 Read 10 New Done Done true
2370 Acked 3369 Write 5 New Done Done true
2371 Acked 3370 Write 5 New Done Done true
2372 Acked 3371 Write 1 New Done Done true
2373 Acked 3372 Read 1 New Done Done true
2374 Acked 3373 Write 1 New Done Done true
2375 Acked 3374 Write 3 New Done Done true
2376 Acked 3375 Read 9 New Done Done true
2377 Acked 3376 Write 6 New Done Done true
2378 Acked 3377 Write 4 New Done Done true
2379 Acked 3378 Read 7 New Done Done true
2380 Acked 3379 Write 7 New Done Done true
2381 Acked 3380 Flush 0 New Done Done true
2382 Acked 3381 Write 3 New Done Done true
2383 Acked 3382 Flush 0 New Done Done true
2384 Acked 3383 Read 9 New Done Done true
2385 Acked 3384 Write 10 New Done Done true
2386 Acked 3385 Read 3 New Done Done true
2387 Acked 3386 Read 5 New Done Done true
2388 Acked 3387 Write 3 New Done Done true
2389 Acked 3388 Read 4 New Done Done true
2390 Acked 3389 Read 5 New Done Done true
2391 Acked 3390 Read 2 New Done Done true
2392 Acked 3391 Write 3 New Done Done true
2393 Acked 3392 Write 1 New Done Done true
2394 Acked 3393 Write 10 New Done Done true
2395 Acked 3394 Write 4 New Done Done true
2396 Acked 3395 Read 6 New Done Done true
2397 Acked 3396 Read 10 New Done Done true
2398 Acked 3397 Write 1 New Done Done true
2399 Acked 3398 Read 3 New Done Done true
2400 Acked 3399 Read 8 New Done Done true
2401 Acked 3400 Read 9 New Done Done true
2402 Acked 3401 Read 2 New Done Done true
2403 Acked 3402 Read 8 New Done Done true
2404 Acked 3403 Flush 0 New Done Done true
2405 Acked 3404 Write 9 New Done Done true
2406 Acked 3405 Read 6 New Done Done true
2407 Acked 3406 Read 9 New Done Done true
2408 Acked 3407 Flush 0 New Done Done true
2409 Acked 3408 Write 5 New Done Done true
2410 Acked 3409 Write 7 New Done Done true
2411 Acked 3410 Flush 0 New Done Done true
STATES DS:0 DS:1 DS:2 TOTAL
New 302 0 0 302
Sent 0 0 0 0
Done 0 302 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 3108 3410 3410
Downstairs last five completed: 3108 3107 3106 3105 3104
Upstairs last five completed: 2411 2410 2409 2408 2407
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:12.894298648Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:12.894357976Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:12.895766812Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:12.895794111Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 3108","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:12.89580944Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 3108","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:12.895994983Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:12.896018132Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:12.896032841Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:12.896047011Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:12.8960615Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 3410 3409 3408 3407 3406
Upstairs last five completed: 2411 2410 2409 2408 2407
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(2)
stop 2
Feb 13 21:41:16.768 INFO request completed, latency_us: 337, response_code: 204, uri: /stop/cid/2, method[8830] Got stop action so:false kr:false
: POST, req_id: dae6af46-cc51-4db2-99d1-9ed9b734dc4c, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:16.769182114Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:41:16.769 INFO request completed, latency_us: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:16.76926885Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
171, response_code: {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:16.769313489Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
200, uri: /state/cid/2, method: GET, req_id: a7cc9e0b-90b5-44be-9d5f-79d60f39235e, remote_addr{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:16.769336658Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
: 127.0.0.1:52124, local_addr: 127.{"msg":"[2] client re-new 0 jobs since flush 3410","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:16.769404465Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
0.0.1:9998
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:16.769440724Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:17.771242574Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:17.771296922Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:17.771315361Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:17.77133237Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:17.77134524Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 3410","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:17.771378428Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:17.771420097Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:18.7726102Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:18.772659498Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:18.772680937Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:18.772708146Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:18.772725785Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 3410","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:18.772753594Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:18.772788373Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:19.773718895Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:19.773781183Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:19.773802922Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:19.773823331Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:19.773839251Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 3410","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:19.77385659Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:19.77387506Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:41:20.771 INFO request completed, latency_us: 217, response_code: 200, uri: /state/cid/2, method: GET, req_id: d59479ce-1473-43fc-a8a9-cd371b36b636, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Read block 972 len 16384
002/300 Write block 4394 len 36864 data: 3 2 2 3 3 3 3 4 3
003/300 Read block 0 len 24576
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:20.774839591Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:20.774892099Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:20.774916438Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:20.774933898Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:20.774963947Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 3 jobs since flush 3410","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:20.774981936Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:20.774996485Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
004/300 Write block 1403 len 28672 data: 3 3 4 4 4 3 2
005/300 Write block 891 len 8192 data: 3 3
006/300 Read block 3914 len 36864
007/300 Write block 2538 len 4096 data: 3
008/300 Write block 4670 len 28672 data: 4 4 2 2 2 2 2
009/300 Write block 744 len 20480 data: 2 2 2 2 2
010/300 Read block 2333 len 16384
011/300 Read block 2730 len 28672
012/300 Write block 3805 len 4096 data: 3
013/300 Write block 518 len 16384 data: 3 4 4 4
014/300 Write block 136 len 16384 data: 3 3 4 4
015/300 Read block 3418 len 20480
016/300 Write block 1820 len 4096 data: 5
017/300 Read block 871 len 24576
018/300 Read block 169 len 4096
019/300 Write block 349 len 40960 data: 3 3 5 6 5 5 5 4 5 5
020/300 Write block 4450 len 8192 data: 4 5
021/300 Read block 1975 len 24576
022/300 Write block 3094 len 16384 data: 2 2 2 4
023/300 Write block 741 len 40960 data: 5 3 3 3 3 3 3 3 2 2
024/300 Write block 4812 len 32768 data: 2 2 2 2 2 2 2 2
025/300 Read block 1337 len 24576
026/300 Write block 4350 len 16384 data: 2 2 2 2
027/300 Write block 1643 len 16384 data: 4 6 5 5
028/300 Read block 2833 len 24576
029/300 Write block 1749 len 16384 data: 2 2 2 3
030/300 Read block 3009 len 36864
031/300 Write block 386 len 24576 data: 4 2 3 3 3 3
032/300 Read block 2921 len 24576
033/300 Read block 1316 len 20480
034/300 Read block 3718 len 4096
035/300 Read block 4387 len 20480
036/300 Read block 1048 len 32768
037/300 Write block 157 len 36864 data: 3 3 3 3 5 5 7 8 8
038/300 Write block 3276 len 20480 data: 4 4 4 4 4
039/300 Write block 4562 len 32768 data: 2 2 3 3 4 4 3 3
040/300 Write block 2079 len 20480 data: 2 2 2 2 2
041/300 Write block 2402 len 16384 data: 2 3 2 2
042/300 Write block 172 len 4096 data: 4
043/300 Read block 2010 len 4096
044/300 Write block 3737 len 16384 data: 4 4 3 4
045/300 Read block 552 len 24576
046/300 Read block 1644 len 16384
047/300 Write block 1220 len 32768 data: 2 2 2 2 2 2 3 5
048/300 Write block 3728 len 24576 data: 2 2 2 2 3 3
049/300 Read block 2098 len 16384
050/300 Flush
051/300 Read block 4102 len 16384
052/300 Write block 317 len 28672 data: 2 2 3 3 3 2 2
053/300 Read block 207 len 32768
054/300 Read block 2120 len 28672
055/300 Read block 555 len 32768
056/300 Flush
057/300 Read block 3986 len 4096
058/300 Read block 2326 len 36864
059/300 Read block 3127 len 8192
060/300 Write block 1192 len 24576 data: 3 3 3 2 2 2
061/300 Write block 1356 len 4096 data: 6
062/300 Flush
063/300 Read block 3185 len 28672
064/300 Read block 4210 len 28672
065/300 Read block 1146 len 24576
066/300 Read block 4796 len 32768
067/300 Read block 2976 len 36864
068/300 Flush
069/300 Write block 4795 len 36864 data: 4 5 5 5 5 4 3 3 4
070/300 Write block 87 len 16384 data: 4 3 3 4
071/300 Read block 4802 len 24576
072/300 Write block 4681 len 24576 data: 2 2 2 2 2 3
073/300 Read block 4657 len 40960
074/300 Read block 2902 len 16384
075/300 Read block 4562 len 40960
076/300 Write block 227 len 28672 data: 2 3 3 3 4 4 4
077/300 Read block 949 len 24576
078/300 Write block 3120 len 28672 data: 5 5 5 5 6 5 5
079/300 Read block 2957 len 36864
080/300 Read block 16 len 28672
081/300 Write block 599 len 36864 data: 4 2 2 2 2 3 3 3 3
082/300 Flush
083/300 Write block 4114 len 32768 data: 5 5 5 5 4 4 2 4
084/300 Write block 1346 len 12288 data: 2 2 2
085/300 Write block 2847 len 8192 data: 3 3
086/300 Flush
087/300 Read block 587 len 24576
088/300 Read block 4561 len 28672
089/300 Write block 4187 len 8192 data: 4 3
090/300 Read block 2540 len 24576
091/300 Write block 3006 len 4096 data: 4
092/300 Read block 355 len 28672
093/300 Read block 595 len 36864
094/300 Read block 587 len 8192
095/300 Read block 3754 len 12288
096/300 Write block 2458 len 40960 data: 3 3 3 3 3 4 4 4 4 4
097/300 Write block 3150 len 20480 data: 4 4 4 4 4
098/300 Flush
099/300 Read block 2728 len 28672
100/300 Write block 1110 len 40960 data: 4 4 4 2 2 2 2 2 3 3
101/300 Read block 4745 len 8192
102/300 Read block 4359 len 24576
103/300 Read block 2300 len 24576
104/300 Read block 2582 len 16384
105/300 Write block 2074 len 20480 data: 3 3 4 3 3
106/300 Read block 2729 len 36864
107/300 Read block 1508 len 4096
108/300 Flush
109/300 Write block 781 len 28672 data: 2 2 2 2 2 2 2
110/300 Write block 4458 len 36864 data: 3 3 3 3 3 2 2 2 3
111/300 Read block 3922 len 40960
112/300 Write block 359 len 24576 data: 4 4 4 4 4 4
113/300 Write block 2014 len 40960 data: 5 4 4 6 5 4 5 3 3 3
114/300 Read block 761 len 4096
115/300 Read block 703 len 4096
116/300 Write block 2436 len 32768 data: 2 3 3 3 3 4 5 4
117/300 Write block 4375 len 28672 data: 2 2 2 2 5 5 4
118/300 Read block 2040 len 8192
119/300 Write block 3840 len 36864 data: 2 2 2 2 2 2 2 2 2
120/300 Write block 3714 len 28672 data: 2 2 2 3 2 2 2
121/300 Read block 3933 len 32768
122/300 Write block 3817 len 40960 data: 3 3 3 3 4 4 4 4 4 3
123/300 Write block 2779 len 40960 data: 2 2 2 3 3 3 2 2 2 2
124/300 Write block 701 len 8192 data: 4 3
125/300 Flush
126/300 Read block 473 len 20480
127/300 Read block 4171 len 40960
128/300 Write block 749 len 36864 data: 3 3 2 2 2 2 2 3 3
129/300 Write block 4245 len 28672 data: 2 2 2 2 2 2 2
130/300 Read block 141 len 28672
131/300 Read block 2886 len 12288
132/300 Read block 2366 len 8192
133/300 Write block 1795 len 40960 data: 2 2 2 2 2 2 2 2 2 2
134/300 Write block 1364 len 28672 data: 2 2 2 2 2 2 2
135/300 Read block 854 len 24576
136/300 Read block 1096 len 32768
137/300 Write block 3833 len 16384 data: 3 3 3 3
138/300 Write block 4722 len 16384 data: 5 4 4 4
139/300 Read block 4240 len 20480
140/300 Read block 569 len 32768
141/300 Flush
142/300 Flush
143/300 Read block 199 len 16384
144/300 Write block 3451 len 4096 data: 2
145/300 Read block 1204 len 12288
146/300 Write block 2504 len 12288 data: 4 3 3
147/300 Write block 1003 len 4096 data: 3
148/300 Write block 543 len 20480 data: 2 2 2 2 2
149/300 Read block 1245 len 8192
150/300 Read block 4813 len 16384
151/300 Read block 3930 len 36864
152/300 Read block 4404 len 12288
153/300 Read block 4651 len 40960
154/300 Write block 234 len 36864 data: 4 4 4 3 4 4 4 4 4
155/300 Read block 1003 len 28672
156/300 Read block 2077 len 12288
157/300 Read block 4561 len 28672
158/300 Read block 4022 len 20480
159/300 Read block 2143 len 4096
160/300 Read block 905 len 16384
161/300 Read block 4888 len 8192
162/300 Read block 3647 len 36864
163/300 Write block 4782 len 32768 data: 2 2 2 2 2 2 2 2
164/300 Read block 1945 len 40960
165/300 Read block 1924 len 28672
166/300 Write block 3676 len 40960 data: 4 3 3 2 3 3 4 5 4 4
167/300 Write block 1965 len 28672 data: 4 4 4 4 3 2 2
168/300 Read block 1114 len 24576
169/300 Write block 2374 len 24576 data: 4 3 2 2 2 2
170/300 Write block 1154 len 20480 data: 2 2 2 2 3
171/300 Write block 3019 len 24576 data: 3 3 3 3 3 2
172/300 Write block 4573 len 8192 data: 4 4
173/300 Read block 4884 len 28672
174/300 Read block 4425 len 12288
175/300 Write block 3510 len 40960 data: 4 4 3 3 4 4 4 4 3 3
176/300 Write block 3587 len 8192 data: 6 6
177/300 Read block 1017 len 16384
178/300 Read block 4393 len 16384
179/300 Write block 1991 len 12288 data: 4 3 4
180/300 Write block 2883 len 16384 data: 3 3 2 2
181/300 Write block 3838 len 8192 data: 2 2
182/300 Flush
183/300 Write block 4106 len 8192 data: 2 2
184/300 Read block 1153 len 32768
185/300 Read block 2577 len 12288
186/300 Read block 1630 len 20480
187/300 Write block 1206 len 20480 data: 4 4 4 5 4
188/300 Write block 2628 len 24576 data: 2 2 2 2 2 2
189/300 Read block 2487 len 12288
190/300 Read block 791 len 32768
191/300 Write block 3282 len 32768 data: 3 5 5 5 5 5 4 4
192/300 Flush
193/300 Read block 2668 len 40960
194/300 Flush
195/300 Write block 2633 len 12288 data: 3 2 2
196/300 Read block 2916 len 4096
197/300 Read block 4845 len 4096
198/300 Read block 2365 len 12288
199/300 Read block 1522 len 32768
200/300 Read block 1922 len 24576
201/300 Read block 1040 len 4096
202/300 Write block 3553 len 32768 data: 3 3 3 3 5 4 4 5
203/300 Read block 1699 len 20480
204/300 Flush
205/300 Read block 2897 len 16384
206/300 Read block 3239 len 12288
207/300 Read block 4304 len 20480
208/300 Flush
209/300 Read block 3647 len 36864
210/300 Write block 3750 len 4096 data: 2
211/300 Read block 104 len 40960
212/300 Write block 526 len 40960 data: 4 4 4 3 3 5 3 3 3 3
213/300 Read block 3731 len 20480
214/300 Read block 2210 len 20480
215/300 Write block 1036 len 24576 data: 2 2 3 3 3 3
216/300 Read block 4705 len 36864
217/300 Read block 3006 len 12288
218/300 Read block 3769 len 20480
219/300 Flush
220/300 Write block 976 len 32768 data: 2 3 3 2 2 2 2 2
221/300 Read block 191 len 8192
222/300 Read block 611 len 8192
223/300 Flush
224/300 Write block 1104 len 36864 data: 6 6 6 5 4 4 5 5 5
225/300 Read block 3512 len 20480
226/300 Read block 1968 len 24576
227/300 Read block 139 len 12288
228/300 Read block 3630 len 28672
229/300 Write block 2303 len 40960 data: 4 4 3 3 3 3 3 4 4 4
230/300 Write block 4927 len 28672 data: 2 2 2 3 3 3 3
231/300 Read block 534 len 28672
232/300 Read block 6 len 32768
233/300 Read block 429 len 28672
234/300 Write block 4538 len 4096 data: 4
235/300 Read block 305 len 8192
236/300 Write block 1516 len 8192 data: 2 2
237/300 Read block 1731 len 4096
238/300 Write block 4485 len 24576 data: 3 5 6 8 8 6
239/300 Write block 354 len 40960 data: 6 6 5 6 6 5 5 5 5 5
240/300 Read block 3982 len 16384
241/300 Write block 581 len 40960 data: 3 3 3 2 2 2 2 2 2 4
242/300 Write block 3339 len 24576 data: 5 4 4 5 6 5
243/300 Read block 4118 len 36864
244/300 Read block 1152 len 12288
245/300 Read block 4269 len 40960
246/300 Read block 4935 len 4096
247/300 Write block 4738 len 40960 data: 2 2 3 3 3 3 3 3 3 3
248/300 Read block 18 len 20480
249/300 Write block 1885 len 8192 data: 2 4
250/300 Write block 1631 len 32768 data: 3 3 3 3 3 2 2 3
251/300 Read block 1853 len 16384
252/300 Write block 186 len 40960 data: 4 4 3 2 2 2 2 2 2 2
253/300 Write block 4771 len 8192 data: 4 4
254/300 Write block 4209 len 4096 data: 3
255/300 Read block 2244 len 16384
256/300 Read block 3381 len 40960
257/300 Write block 1639 len 24576 data: 3 4 4 4 5 7
258/300 Flush
259/300 Read block 1807 len 8192
260/300 Read block 653 len 28672
261/300 Read block 2956 len 20480
262/300 Write block 2582 len 16384 data: 2 3 3 3
263/300 Flush
264/300 Write block 3038 len 4096 data: 4
265/300 Write block 2625 len 20480 data: 2 2 3 3 3
266/300 Write block 1634 len 8192 data: 4 4
267/300 Flush
268/300 Write block 3970 len 4096 data: 3
269/300 Flush
270/300 Flush
271/300 Write block 2571 len 24576 data: 4 4 5 5 4 3
272/300 Write block 3304 len 32768 data: 3 3 4 3 3 5 5 5
273/300 Read block 3492 len 20480
274/300 Read block 4349 len 4096
275/300 Read block 1519 len 16384
276/300 Write block 1310 len 4096 data: 3
277/300 Read block 4328 len 24576
278/300 Write block 4605 len 8192 data: 2 2
279/300 Read block 2897 len 12288
280/300 Write block 1791 len 16384 data: 2 2 2 2
281/300 Read block 4302 len 20480
282/300 Write block 2236 len 8192 data: 2 3
283/300 Flush
284/300 Read block 341 len 32768
285/300 Read block 446 len 12288
286/300 Write block 1467 len 12288 data: 2 3 2
287/300 Write block 4440 len 20480 data: 2 3 3 3 3
288/300 Write block 894 len 36864 data: 3 3 2 2 2 3 4 4 4
289/300 Write block 622 len 16384 data: 3 3 2 2
290/300 Flush
291/300 Read block 2022 len 12288
292/300 Flush
293/300 Read block 4307 len 36864
294/300 Write block 806 len 4096 data: 2
295/300 Write block 1106 len 20480 data: 7 6 5 5 6
296/300 Write block 3216 len 36864 data: 2 3 2 4 5 6 6 6 7
297/300 Read block 3846 len 32768
298/300 Write block 2448 len 16384 data: 3 2 2 2
299/300 Read block 4063 len 4096
300/300 Write block 4298 len 16384 data: 4 4 4 3
Main task has work to do, go find it
got dsc Start(2)
start 2
Feb 13 21:41:21.461 INFO request completed, latency_us: 251, response_code: 204, uri: /start/cid/2, method: POST, req_id: 8d49a22a-ef0c-419e-b40d-ab682ddfd090, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
2412 Acked 3411 Read 4 Done Done New true
2413 Acked 3412 Write 9 Done Done New true
2414 Acked 3413 Read 6 Done Done New true
2415 Acked 3414 Write 7 Done Done New false
2416 Acked 3415 Write 2 Done Done New false
2417 Acked 3416 Read 9 Done Done New false
2418 Acked 3417 Write 1 Done Done New false
2419 Acked 3418 Write 7 Done Done New false
2420 Acked 3419 Write 5 Done Done New false
2421 Acked 3420 Read 4 Done Done New false
2422 Acked 3421 Read 7 Done Done New false
2423 Acked 3422 Write 1 Done Done New false
2424 Acked 3423 Write 4 Done Done New false
2425 Acked 3424 Write 4 Done Done New false
2426 Acked 3425 Read 5 Done Done New false
2427 Acked 3426 Write 1 Done Done New false
2428 Acked 3427 Read 6 Done Done New false
2429 Acked 3428 Read 1 Done Done New false
2430 Acked 3429 Write 10 Done Done New false
2431 Acked 3430 Write 2 Done Done New false
2432 Acked 3431 Read 6 Done Done New false
2433 Acked 3432 Write 4 Done Done New false
2434 Acked 3433 Write 10 Done Done New false
2435 Acked 3434 Write 8 Done Done New false
2436 Acked 3435 Read 6 Done Done New false
2437 Acked 3436 Write 4 Done Done New false
2438 Acked 3437 Write 4 Done Done New false
2439 Acked 3438 Read 6 Done Done New false
2440 Acked 3439 Write 4 Done Done New false
2441 Acked 3440 Read 9 Done Done New false
2442 Acked 3441 Write 6 Done Done New false
2443 Acked 3442 Read 6 Done Done New false
2444 Acked 3443 Read 5 Done Done New false
2445 Acked 3444 Read 1 Done Done New false
2446 Acked 3445 Read 5 Done Done New false
2447 Acked 3446 Read 8 Done Done New false
2448 Acked 3447 Write 9 Done Done New false
2449 Acked 3448 Write 5 Done Done New false
2450 Acked 3449 Write 8 Done Done New false
2451 Acked 3450 Write 5 Done Done New false
2452 Acked 3451 Write 4 Done Done New false
2453 Acked 3452 Write 1 Done Done New false
2454 Acked 3453 Read 1 Done Done New false
2455 Acked 3454 Write 4 Done Done New false
2456 Acked 3455 Read 6 Done Done New false
2457 Acked 3456 Read 4 Done Done New false
2458 Acked 3457 Write 8 Done Done New false
2459 Acked 3458 Write 6 Done Done New false
2460 Acked 3459 Read 4 Done Done New false
2461 Acked 3460 Flush 0 Done Done New false
2462 Acked 3461 Read 4 Done Done New false
2463 Acked 3462 Write 7 Done Done New false
2464 Acked 3463 Read 8 Done Done New false
2465 Acked 3464 Read 7 Done Done New false
2466 Acked 3465 Read 8 Done Done New false
2467 Acked 3466 Flush 0 Done Done New false
2468 Acked 3467 Read 1 Done Done New false
2469 Acked 3468 Read 9 Done Done New false
2470 Acked 3469 Read 2 Done Done New false
2471 Acked 3470 Write 6 Done Done New false
2472 Acked 3471 Write 1 Done Done New false
2473 Acked 3472 Flush 0 Done Done New false
2474 Acked 3473 Read 7 Done Done New false
2475 Acked 3474 Read 7 Done Done New false
2476 Acked 3475 Read 6 Done Done New false
2477 Acked 3476 Read 8 Done Done New false
2478 Acked 3477 Read 9 Done Done New false
2479 Acked 3478 Flush 0 Done Done New false
2480 Acked 3479 Write 9 Done Done New false
2481 Acked 3480 Write 4 Done Done New false
2482 Acked 3481 Read 6 Done Done New false
2483 Acked 3482 Write 6 Done Done New false
2484 Acked 3483 Read 10 Done Done New false
2485 Acked 3484 Read 4 Done Done New false
2486 Acked 3485 Read 10 Done Done New false
2487 Acked 3486 Write 7 Done Done New false
2488 Acked 3487 Read 6 Done Done New false
2489 Acked 3488 Write 7 Done Done New false
2490 Acked 3489 Read 9 Done Done New false
2491 Acked 3490 Read 7 Done Done New false
2492 Acked 3491 Write 9 Done Done New false
2493 Acked 3492 Flush 0 Done Done New false
2494 Acked 3493 Write 8 Done Done New false
2495 Acked 3494 Write 3 Done Done New false
2496 Acked 3495 Write 2 Done Done New false
2497 Acked 3496 Flush 0 Done Done New false
2498 Acked 3497 Read 6 Done Done New false
2499 Acked 3498 Read 7 Done Done New false
2500 Acked 3499 Write 2 Done Done New false
2501 Acked 3500 Read 6 Done Done New false
2502 Acked 3501 Write 1 Done Done New false
2503 Acked 3502 Read 7 Done Done New false
2504 Acked 3503 Read 9 Done Done New false
2505 Acked 3504 Read 2 Done Done New false
2506 Acked 3505 Read 3 Done Done New false
2507 Acked 3506 Write 10 Done Done New false
2508 Acked 3507 Write 5 Done Done New false
2509 Acked 3508 Flush 0 Done Done New false
2510 Acked 3509 Read 7 Done Done New false
2511 Acked 3510 Write 10 Done Done New false
2512 Acked 3511 Read 2 Done Done New false
2513 Acked 3512 Read 6 Done Done New false
2514 Acked 3513 Read 6 Done Done New false
2515 Acked 3514 Read 4 Done Done New false
2516 Acked 3515 Write 5 Done Done New false
2517 Acked 3516 Read 9 Done Done New false
2518 Acked 3517 Read 1 Done Done New false
2519 Acked 3518 Flush 0 Done Done New false
2520 Acked 3519 Write 7 Done Done New false
2521 Acked 3520 Write 9 Done Done New false
2522 Acked 3521 Read 10 Done Done New false
2523 Acked 3522 Write 6 Done Done New false
2524 Acked 3523 Write 10 Done Done New false
2525 Acked 3524 Read 1 Done Done New false
2526 Acked 3525 Read 1 Done Done New false
2527 Acked 3526 Write 8 Done Done New false
2528 Acked 3527 Write 7 Done Done New false
2529 Acked 3528 Read 2 Done Done New false
2530 Acked 3529 Write 9 Done Done New false
2531 Acked 3530 Write 7 Done Done New false
2532 Acked 3531 Read 8 Done Done New false
2533 Acked 3532 Write 10 Done Done New false
2534 Acked 3533 Write 10 Done Done New false
2535 Acked 3534 Write 2 Done Done New false
2536 Acked 3535 Flush 0 Done Done New false
2537 Acked 3536 Read 5 Done Done New false
2538 Acked 3537 Read 10 Done Done New false
2539 Acked 3538 Write 9 Done Done New false
2540 Acked 3539 Write 7 Done Done New false
2541 Acked 3540 Read 7 Done Done New false
2542 Acked 3541 Read 3 Done Done New false
2543 Acked 3542 Read 2 Done Done New false
2544 Acked 3543 Write 10 Done Done New false
2545 Acked 3544 Write 7 Done Done New false
2546 Acked 3545 Read 6 Done Done New false
2547 Acked 3546 Read 8 Done Done New false
2548 Acked 3547 Write 4 Done Done New false
2549 Acked 3548 Write 4 Done Done New false
2550 Acked 3549 Read 5 Done Done New false
2551 Acked 3550 Read 8 Done Done New false
2552 Acked 3551 Flush 0 Done Done New false
2553 Acked 3552 Flush 0 Done Done New false
2554 Acked 3553 Read 4 Done Done New false
2555 Acked 3554 Write 1 Done Done New false
2556 Acked 3555 Read 3 Done Done New false
2557 Acked 3556 Write 3 Done Done New false
2558 Acked 3557 Write 1 Done Done New false
2559 Acked 3558 Write 5 Done Done New false
2560 Acked 3559 Read 2 Done Done New false
2561 Acked 3560 Read 4 Done Done New false
2562 Acked 3561 Read 9 Done Done New false
2563 Acked 3562 Read 3 Done Done New false
2564 Acked 3563 Read 10 Done Done New false
2565 Acked 3564 Write 9 Done Done New false
2566 Acked 3565 Read 7 Done Done New false
2567 Acked 3566 Read 3 Done Done New false
2568 Acked 3567 Flush 0 Done Done New false
2569 Acked 3568 Read 7 Done Done New false
2570 Acked 3569 Read 5 Done Done New false
2571 Acked 3570 Read 1 Done Done New false
2572 Acked 3571 Read 4 Done Done New false
2573 Acked 3572 Read 2 Done Done New false
2574 Acked 3573 Read 9 Done Done New false
2575 Acked 3574 Write 8 Done Done New false
2576 Acked 3575 Read 10 Done Done New false
2577 Acked 3576 Read 7 Done Done New false
2578 Acked 3577 Write 10 Done Done New false
2579 Acked 3578 Write 7 Done Done New false
2580 Acked 3579 Read 6 Done Done New false
2581 Acked 3580 Write 6 Done Done New false
2582 Acked 3581 Write 5 Done Done New false
2583 Acked 3582 Write 6 Done Done New false
2584 Acked 3583 Write 2 Done Done New false
2585 Acked 3584 Read 7 Done Done New false
2586 Acked 3585 Read 3 Done Done New false
2587 Acked 3586 Write 10 Done Done New false
2588 Acked 3587 Write 2 Done Done New false
2589 Acked 3588 Read 4 Done Done New false
2590 Acked 3589 Read 4 Done Done New false
2591 Acked 3590 Write 3 Done Done New false
2592 Acked 3591 Write 4 Done Done New false
2593 Acked 3592 Write 2 Done Done New false
2594 Acked 3593 Flush 0 Done Done New false
2595 Acked 3594 Write 2 Done Done New false
2596 Acked 3595 Read 8 Done Done New false
2597 Acked 3596 Read 3 Done Done New false
2598 Acked 3597 Read 5 Done Done New false
2599 Acked 3598 Write 5 Done Done New false
2600 Acked 3599 Write 6 Done Done New false
2601 Acked 3600 Read 3 Done Done New false
2602 Acked 3601 Read 8 Done Done New false
2603 Acked 3602 Write 8 Done Done New false
2604 Acked 3603 Flush 0 Done Done New false
2605 Acked 3604 Read 10 Done Done New false
2606 Acked 3605 Flush 0 Done Done New false
2607 Acked 3606 Write 3 Done Done New false
2608 Acked 3607 Read 1 Done Done New false
2609 Acked 3608 Read 1 Done Done New false
2610 Acked 3609 Read 3 Done Done New false
2611 Acked 3610 Read 8 Done Done New false
2612 Acked 3611 Read 6 Done Done New false
2613 Acked 3612 Read 1 Done Done New false
2614 Acked 3613 Write 8 Done Done New false
2615 Acked 3614 Read 5 Done Done New false
2616 Acked 3615 Flush 0 Done Done New false
2617 Acked 3616 Read 4 Done Done New false
2618 Acked 3617 Read 3 Done Done New false
2619 Acked 3618 Read 5 Done Done New false
2620 Acked 3619 Flush 0 Done Done New false
2621 Acked 3620 Read 9 Done Done New false
2622 Acked 3621 Write 1 Done Done New false
2623 Acked 3622 Read 10 Done Done New false
2624 Acked 3623 Write 10 Done Done New false
2625 Acked 3624 Read 5 Done Done New false
2626 Acked 3625 Read 5 Done Done New false
2627 Acked 3626 Write 6 Done Done New false
2628 Acked 3627 Read 9 Done Done New false
2629 Acked 3628 Read 3 Done Done New false
2630 Acked 3629 Read 5 Done Done New false
2631 Acked 3630 Flush 0 Done Done New false
2632 Acked 3631 Write 8 Done Done New false
2633 Acked 3632 Read 2 Done Done New false
2634 Acked 3633 Read 2 Done Done New false
2635 Acked 3634 Flush 0 Done Done New false
2636 Acked 3635 Write 9 Done Done New false
2637 Acked 3636 Read 5 Done Done New false
2638 Acked 3637 Read 6 Done Done New false
2639 Acked 3638 Read 3 Done Done New false
2640 Acked 3639 Read 7 Done Done New false
2641 Acked 3640 Write 10 Done Done New false
2642 Acked 3641 Write 7 Done Done New false
2643 Acked 3642 Read 7 Done Done New false
2644 Acked 3643 Read 8 Done Done New false
2645 Acked 3644 Read 7 Done Done New false
2646 Acked 3645 Write 1 Done Done New false
2647 Acked 3646 Read 2 Done Done New false
2648 Acked 3647 Write 2 Done Done New false
2649 Acked 3648 Read 1 Done Done New false
2650 Acked 3649 Write 6 Done Done New false
2651 Acked 3650 Write 10 Done Done New false
2652 Acked 3651 Read 4 Done Done New false
2653 Acked 3652 Write 10 Done Done New false
2654 Acked 3653 Write 6 Done Done New false
2655 Acked 3654 Read 9 Done Done New false
2656 Acked 3655 Read 3 Done Done New false
2657 Acked 3656 Read 10 Done Done New false
2658 Acked 3657 Read 1 Done Done New false
2659 Acked 3658 Write 10 Done Done New false
2660 Acked 3659 Read 5 Done Done New false
2661 Acked 3660 Write 2 Done Done New false
2662 Acked 3661 Write 8 Done Done New false
2663 Acked 3662 Read 4 Done Done New false
2664 Acked 3663 Write 10 Done Done New false
2665 Acked 3664 Write 2 Done Done New false
2666 Acked 3665 Write 1 Done Done New false
2667 Acked 3666 Read 4 Done Done New false
2668 Acked 3667 Read 10 Done Done New false
2669 Acked 3668 Write 6 Done Done New false
2670 Acked 3669 Flush 0 Done Done New false
2671 Acked 3670 Read 2 Done Done New false
2672 Acked 3671 Read 7 Done Done New false
2673 Acked 3672 Read 5 Done Done New false
2674 Acked 3673 Write 4 Done Done New false
2675 Acked 3674 Flush 0 Done Done New false
2676 Acked 3675 Write 1 Done Done New false
2677 Acked 3676 Write 5 Done Done New false
2678 Acked 3677 Write 2 Done Done New false
2679 Acked 3678 Flush 0 Done Done New false
2680 Acked 3679 Write 1 Done Done New false
2681 Acked 3680 Flush 0 Done Done New false
2682 Acked 3681 Flush 0 Done Done New false
2683 Acked 3682 Write 6 Done Done New false
2684 Acked 3683 Write 8 Done Done New false
2685 Acked 3684 Read 5 Done Done New false
2686 Acked 3685 Read 1 Done Done New false
2687 Acked 3686 Read 4 Done Done New false
2688 Acked 3687 Write 1 Done Done New false
2689 Acked 3688 Read 6 Done Done New false
2690 Acked 3689 Write 2 Done Done New false
2691 Acked 3690 Read 3 Done Done New false
2692 Acked 3691 Write 4 Done Done New false
2693 Acked 3692 Read 5 Done Done New false
2694 Acked 3693 Write 2 Done Done New false
2695 Acked 3694 Flush 0 Done Done New false
2696 Acked 3695 Read 8 Done Done New false
2697 Acked 3696 Read 3 Done Done New false
2698 Acked 3697 Write 3 Done Done New false
2699 Acked 3698 Write 5 Done Done New false
2700 Acked 3699 Write 9 Done Done New false
2701 Acked 3700 Write 4 Done Done New false
2702 Acked 3701 Flush 0 Done Done New false
2703 Acked 3702 Read 3 Done Done New false
2704 Acked 3703 Flush 0 Done Done New false
2705 Acked 3704 Read 9 Done Done New false
2706 Acked 3705 Write 1 Done Done New false
2707 Acked 3706 Write 5 Done Done New false
2708 Acked 3707 Write 9 Done Done New false
2709 Acked 3708 Read 8 Done Done New false
2710 Acked 3709 Write 4 Done Done New false
2711 Acked 3710 Read 1 Done Done New false
2712 Acked 3711 Write 4 Done Done New false
STATES DS:0 DS:1 DS:2 TOTAL
New 0 0 301 301
Sent 0 0 0 0
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 3703 3703 3410
Downstairs last five completed: 3410 3409 3408 3407 3406
Upstairs last five completed: 2712 2711 2710 2709 2708
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:21.776251095Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:21.776347092Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:21.77639168Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:21.776430989Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:21.776461078Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 302 jobs since flush 3410","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:21.776493996Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:21.776690539Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:22.778380113Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:22.778441941Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:22.7784654Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:22.778487009Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:22.778503429Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 302 jobs since flush 3410","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:22.778524428Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:22.778619344Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1173)
[8830][2] reports Running
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:23.779602405Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:23.779660823Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:23.781016941Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:23.78104908Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 3410","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:23.781067949Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 3410","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:23.781263302Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:23.78130488Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:23.781331429Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:23.781348529Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:23.781365468Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 3712 3711 3710 3709 3708
Upstairs last five completed: 2713 2712 2711 2710 2709
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(1)
stop 1
Feb 13 21:41:25.463 INFO request completed, latency_us: [8820] Got stop action so:false kr:false
325, response_code: 204, uri: /stop/cid/1, method: POST, req_id: b9e9b419-3a4d-4d35-b557-f3f4ccd8f888, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:25.464716025Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:25.464765684Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:25.464782753Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:25.464795982Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:41:25.464 INFO request completed, latency_us: 144, response_code: 200, {"msg":"[1] client re-new 0 jobs since flush 3712","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:25.464814742Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
uri: /state/cid/1, method: GET, req_id: c1525792-c2f5-4f34-8306-c3de2eb021b9, remote_addr: 127.0.0.1:52124, local_addr: 127.{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:25.46485063Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
0.0.1:9998
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:26.46586261Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:26.465920368Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:26.465940857Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:26.465957956Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:26.465970516Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 0 jobs since flush 3712","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:26.465984635Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:26.466012414Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:27.466733426Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:27.466788393Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:27.466806313Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:27.466822962Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:27.466835451Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 0 jobs since flush 3712","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:27.466911089Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:27.466930288Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:28.468060073Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:28.468160689Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:28.468203407Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:28.468242216Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:28.468271615Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 0 jobs since flush 3712","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:28.468303454Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:28.468335402Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:41:29.466 INFO request completed, latency_us: 211, response_code: 200, uri: /state/cid/1, method: GET, req_id: 1bb39008-0df1-40be-bf1c-be5ea4deed1e, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Read block 4275 len 28672
002/300 Write block 665 len 24576 data: 3 3 3 3 3 3
003/300 Read block 3129 len 16384
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:29.469428639Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:29.469480067Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:29.469498857Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:29.469515636Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:29.469528545Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 3 jobs since flush 3712","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:29.469549975Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:29.469564414Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
004/300 Write block 859 len 36864 data: 5 5 5 5 5 3 4 4 4
005/300 Read block 1323 len 24576
006/300 Flush
007/300 Read block 1791 len 20480
008/300 Write block 2430 len 24576 data: 2 2 2 2 2 2
009/300 Read block 2207 len 12288
010/300 Write block 40 len 28672 data: 4 4 4 4 4 5 5
011/300 Write block 3365 len 24576 data: 2 2 3 4 4 4
012/300 Write block 2179 len 24576 data: 2 2 2 2 2 2
013/300 Flush
014/300 Write block 3145 len 12288 data: 3 3 4
015/300 Read block 2733 len 36864
016/300 Read block 387 len 24576
017/300 Flush
018/300 Write block 4239 len 8192 data: 3 3
019/300 Write block 2993 len 36864 data: 3 3 3 3 3 4 3 3 3
020/300 Read block 86 len 28672
021/300 Write block 2551 len 24576 data: 3 2 2 2 2 2
022/300 Write block 1569 len 16384 data: 3 3 3 3
023/300 Read block 3753 len 4096
024/300 Read block 4723 len 4096
025/300 Read block 2353 len 40960
026/300 Write block 1982 len 16384 data: 2 2 2 3
027/300 Write block 2239 len 28672 data: 3 3 3 3 3 4 3
028/300 Read block 1002 len 40960
029/300 Read block 4409 len 28672
030/300 Flush
031/300 Read block 1813 len 28672
032/300 Read block 1157 len 12288
033/300 Read block 561 len 4096
034/300 Read block 2110 len 32768
035/300 Flush
036/300 Write block 3022 len 32768 data: 4 4 3 2 2 4 4 4
037/300 Read block 1799 len 40960
038/300 Read block 2549 len 40960
039/300 Read block 3319 len 20480
040/300 Read block 1094 len 4096
041/300 Read block 4638 len 24576
042/300 Write block 2569 len 32768 data: 3 3 5 5 6 6 5 4
043/300 Read block 3006 len 16384
044/300 Write block 2499 len 24576 data: 4 4 4 4 5 5
045/300 Read block 1972 len 40960
046/300 Read block 3990 len 4096
047/300 Read block 75 len 4096
048/300 Flush
049/300 Read block 3964 len 40960
050/300 Read block 1361 len 36864
051/300 Flush
052/300 Write block 4243 len 24576 data: 2 2 3 3 3 3
053/300 Read block 3466 len 36864
054/300 Write block 4476 len 12288 data: 5 5 4
055/300 Read block 2759 len 40960
056/300 Write block 1989 len 16384 data: 4 4 5 4
057/300 Read block 488 len 16384
058/300 Read block 1767 len 32768
059/300 Flush
060/300 Read block 3360 len 40960
061/300 Write block 1653 len 28672 data: 2 2 3 4 4 4 5
062/300 Write block 2016 len 24576 data: 5 7 6 5 6 4
063/300 Read block 1967 len 24576
064/300 Read block 2569 len 40960
065/300 Write block 3556 len 32768 data: 4 6 5 5 6 5 4 3
066/300 Read block 3463 len 40960
067/300 Read block 3719 len 28672
068/300 Flush
069/300 Write block 872 len 40960 data: 4 3 3 4 3 3 3 2 2 2
070/300 Write block 2253 len 24576 data: 3 4 4 4 4 2
071/300 Write block 905 len 8192 data: 4 2
072/300 Read block 355 len 20480
073/300 Read block 2258 len 12288
074/300 Read block 2005 len 36864
075/300 Read block 3414 len 36864
076/300 Read block 2115 len 8192
077/300 Read block 1394 len 32768
078/300 Write block 2533 len 8192 data: 4 5
079/300 Read block 2321 len 8192
080/300 Read block 2206 len 28672
081/300 Read block 1556 len 20480
082/300 Write block 1496 len 20480 data: 3 3 3 3 3
083/300 Read block 3118 len 32768
084/300 Write block 4708 len 20480 data: 3 3 3 3 3
085/300 Write block 4633 len 28672 data: 2 2 2 3 3 3 3
086/300 Read block 2191 len 8192
087/300 Read block 310 len 20480
088/300 Write block 285 len 36864 data: 2 2 2 2 3 2 3 3 3
089/300 Write block 4048 len 20480 data: 2 2 2 3 2
090/300 Read block 3719 len 40960
091/300 Write block 2705 len 36864 data: 2 2 2 2 3 3 3 3 4
092/300 Flush
093/300 Write block 2625 len 12288 data: 3 3 4
094/300 Write block 4888 len 8192 data: 3 3
095/300 Read block 4097 len 20480
096/300 Read block 2871 len 28672
097/300 Write block 343 len 40960 data: 2 2 3 4 4 4 4 4 6 7
098/300 Write block 4448 len 16384 data: 4 4 5 6
099/300 Read block 2755 len 36864
100/300 Read block 3293 len 36864
101/300 Read block 2810 len 16384
102/300 Read block 42 len 28672
103/300 Read block 2757 len 20480
104/300 Read block 4972 len 40960
105/300 Read block 1768 len 20480
106/300 Write block 4621 len 20480 data: 2 2 4 4 3
107/300 Read block 3858 len 8192
108/300 Write block 1 len 28672 data: 2 3 3 3 2 3 4
109/300 Read block 2636 len 24576
110/300 Read block 1950 len 24576
111/300 Read block 1823 len 8192
112/300 Write block 4552 len 36864 data: 3 3 3 3 3 3 2 2 2
113/300 Read block 4603 len 4096
114/300 Write block 1713 len 32768 data: 2 2 2 3 4 5 6 6
115/300 Read block 1705 len 20480
116/300 Read block 758 len 24576
117/300 Read block 567 len 24576
118/300 Flush
119/300 Write block 1800 len 32768 data: 3 3 3 3 3 2 2 2
120/300 Read block 986 len 20480
121/300 Read block 3002 len 28672
122/300 Read block 4641 len 24576
123/300 Read block 2940 len 8192
124/300 Write block 317 len 20480 data: 3 3 4 4 4
125/300 Read block 201 len 40960
126/300 Read block 3969 len 36864
127/300 Read block 781 len 24576
128/300 Write block 2501 len 36864 data: 5 5 6 6 4 4 5 5 5
129/300 Write block 3844 len 8192 data: 3 3
130/300 Write block 2034 len 16384 data: 2 2 3 2
131/300 Write block 2464 len 24576 data: 5 5 5 5 3 3
132/300 Write block 4992 len 16384 data: 4 4 3 2
133/300 Read block 2247 len 24576
134/300 Write block 4320 len 32768 data: 3 3 4 5 5 5 5 5
135/300 Write block 865 len 16384 data: 5 5 5 3
136/300 Read block 349 len 4096
137/300 Write block 4721 len 28672 data: 4 6 5 5 5 4 3
138/300 Read block 842 len 32768
139/300 Read block 3951 len 28672
140/300 Read block 3952 len 28672
141/300 Read block 3382 len 20480
142/300 Write block 2178 len 16384 data: 2 3 3 3
143/300 Read block 4151 len 20480
144/300 Write block 573 len 4096 data: 3
145/300 Write block 1459 len 40960 data: 2 3 3 3 3 3 3 2 3 4
146/300 Write block 2718 len 36864 data: 3 3 3 3 3 3 3 3 3
147/300 Flush
148/300 Flush
149/300 Read block 4403 len 24576
150/300 Read block 3195 len 28672
151/300 Write block 562 len 36864 data: 3 3 3 5 5 5 4 4 4
152/300 Read block 1008 len 4096
153/300 Flush
154/300 Read block 1094 len 28672
155/300 Read block 1290 len 8192
156/300 Write block 4759 len 16384 data: 2 2 2 2
157/300 Read block 1277 len 20480
158/300 Read block 2059 len 20480
159/300 Write block 4044 len 24576 data: 4 3 3 3 3 3
160/300 Write block 4680 len 8192 data: 2 3
161/300 Write block 4468 len 12288 data: 3 3 3
162/300 Flush
163/300 Write block 1935 len 12288 data: 3 3 4
164/300 Write block 2049 len 28672 data: 2 2 2 2 2 2 2
165/300 Read block 2605 len 8192
166/300 Write block 1189 len 4096 data: 2
167/300 Read block 588 len 28672
168/300 Read block 2015 len 40960
169/300 Write block 518 len 32768 data: 4 5 5 5 3 3 3 3
170/300 Write block 4647 len 20480 data: 3 3 3 4 4
171/300 Write block 3830 len 16384 data: 3 3 4 4
172/300 Read block 4482 len 28672
173/300 Write block 1127 len 16384 data: 3 3 3 4
174/300 Write block 3797 len 20480 data: 2 2 2 2 2
175/300 Read block 897 len 4096
176/300 Write block 1215 len 8192 data: 3 2
177/300 Read block 3329 len 32768
178/300 Write block 3357 len 28672 data: 4 4 3 3 3 3 3
179/300 Write block 1581 len 40960 data: 2 2 2 2 2 3 4 4 4 4
180/300 Read block 2954 len 20480
181/300 Read block 4196 len 32768
182/300 Write block 4894 len 28672 data: 2 2 2 3 3 2 2
183/300 Read block 710 len 32768
184/300 Write block 2009 len 28672 data: 4 3 4 5 5 6 5
185/300 Write block 1537 len 4096 data: 3
186/300 Write block 650 len 32768 data: 2 2 2 2 2 3 3 3
187/300 Read block 4328 len 16384
188/300 Read block 664 len 12288
189/300 Write block 2300 len 12288 data: 3 3 4
190/300 Write block 4088 len 24576 data: 3 3 3 3 3 2
191/300 Read block 3868 len 4096
192/300 Read block 3356 len 32768
193/300 Read block 786 len 24576
194/300 Read block 2930 len 40960
195/300 Read block 2697 len 8192
196/300 Write block 2577 len 28672 data: 3 3 2 3 2 3 4
197/300 Write block 4268 len 16384 data: 3 2 2 2
198/300 Write block 300 len 28672 data: 3 3 3 3 3 3 2
199/300 Read block 2684 len 12288
200/300 Write block 220 len 16384 data: 2 2 2 2
201/300 Write block 2017 len 28672 data: 8 7 6 7 5 4 4
202/300 Read block 4690 len 40960
203/300 Read block 4390 len 32768
204/300 Write block 1962 len 4096 data: 3
205/300 Write block 2161 len 8192 data: 4 4
206/300 Write block 2879 len 12288 data: 3 3 3
207/300 Write block 1261 len 12288 data: 3 3 4
208/300 Read block 1413 len 4096
209/300 Flush
210/300 Write block 3549 len 8192 data: 3 3
211/300 Flush
212/300 Write block 1212 len 12288 data: 3 3 3
213/300 Write block 3979 len 4096 data: 4
214/300 Read block 4163 len 12288
215/300 Read block 766 len 24576
216/300 Flush
217/300 Read block 554 len 24576
218/300 Read block 480 len 24576
219/300 Read block 1063 len 16384
220/300 Write block 2298 len 36864 data: 3 3 4 4 5 5 5 4 4
221/300 Read block 4355 len 16384
222/300 Read block 3999 len 8192
223/300 Read block 2812 len 12288
224/300 Flush
225/300 Read block 1642 len 8192
226/300 Read block 4114 len 4096
227/300 Read block 4406 len 24576
228/300 Write block 4538 len 20480 data: 5 4 3 3 3
229/300 Flush
230/300 Read block 2316 len 20480
231/300 Read block 4154 len 24576
232/300 Read block 1186 len 12288
233/300 Read block 2563 len 16384
234/300 Flush
235/300 Read block 97 len 24576
236/300 Write block 1142 len 24576 data: 2 2 3 3 2 2
237/300 Write block 3980 len 32768 data: 4 3 3 3 3 3 3 3
238/300 Write block 3689 len 4096 data: 5
239/300 Read block 486 len 24576
240/300 Read block 704 len 8192
241/300 Read block 1162 len 20480
242/300 Write block 3816 len 32768 data: 4 4 4 4 4 5 5 5
243/300 Flush
244/300 Read block 1738 len 16384
245/300 Read block 4725 len 20480
246/300 Write block 718 len 16384 data: 4 4 4 6
247/300 Write block 3944 len 20480 data: 2 2 2 2 3
248/300 Read block 2996 len 28672
249/300 Write block 1810 len 4096 data: 2
250/300 Write block 2736 len 12288 data: 3 3 3
251/300 Read block 1777 len 8192
252/300 Read block 1352 len 28672
253/300 Read block 4731 len 32768
254/300 Read block 4058 len 28672
255/300 Write block 4645 len 40960 data: 2 3 4 4 4 5 5 3 2 2
256/300 Read block 3089 len 20480
257/300 Write block 2269 len 16384 data: 2 2 2 2
258/300 Write block 3952 len 28672 data: 4 4 4 4 3 3 2
259/300 Read block 888 len 40960
260/300 Read block 1723 len 28672
261/300 Write block 2911 len 40960 data: 3 3 3 2 3 3 3 3 3 3
262/300 Read block 2983 len 28672
263/300 Write block 4073 len 24576 data: 2 2 3 2 2 2
264/300 Flush
265/300 Read block 1481 len 40960
266/300 Write block 1131 len 20480 data: 4 4 3 3 4
267/300 Read block 3035 len 40960
268/300 Flush
269/300 Read block 1413 len 4096
270/300 Write block 3350 len 12288 data: 3 4 4
271/300 Write block 3196 len 12288 data: 5 6 6
272/300 Read block 679 len 36864
273/300 Read block 2968 len 16384
274/300 Read block 2570 len 4096
275/300 Write block 3589 len 20480 data: 5 4 4 4 4
276/300 Write block 451 len 36864 data: 3 3 3 2 2 2 2 2 3
277/300 Write block 2821 len 40960 data: 2 2 2 2 2 2 3 3 3 5
278/300 Write block 453 len 40960 data: 4 3 3 3 3 3 4 3 5 6
279/300 Write block 4341 len 4096 data: 2
280/300 Read block 4697 len 20480
281/300 Read block 1001 len 24576
282/300 Write block 973 len 36864 data: 2 2 2 3 4 4 3 3 3
283/300 Write block 3314 len 4096 data: 3
284/300 Read block 2645 len 36864
285/300 Write block 4299 len 24576 data: 5 5 4 3 3 3
286/300 Read block 1380 len 28672
287/300 Read block 1929 len 20480
288/300 Read block 3926 len 12288
289/300 Write block 1481 len 8192 data: 2 2
290/300 Write block 1769 len 8192 data: 3 2
291/300 Read block 4977 len 40960
292/300 Read block 568 len 32768
293/300 Read block 2841 len 12288
294/300 Read block 3569 len 8192
295/300 Read block 1103 len 32768
296/300 Read block 1400 len 8192
297/300 Read block 2968 len 20480
298/300 Read block 1803 len 28672
299/300 Write block 1763 len 36864 data: 4 4 2 3 3 3 4 3 3
300/300 Read block 2913 len 40960
Main task has work to do, go find it
got dsc Start(1)
start 1
Feb 13 21:41:30.200 INFO request completed, latency_us: 213, response_code: 204, uri: /start/cid/1, method: POST, req_id: 22825d11-f6e7-473a-82dd-3f24d7237864, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
2714 Acked 3713 Read 7 Done New Done true
2715 Acked 3714 Write 6 Done New Done true
2716 Acked 3715 Read 4 Done New Done true
2717 Acked 3716 Write 9 Done New Done false
2718 Acked 3717 Read 6 Done New Done false
2719 Acked 3718 Flush 0 Done New Done false
2720 Acked 3719 Read 5 Done New Done false
2721 Acked 3720 Write 6 Done New Done false
2722 Acked 3721 Read 3 Done New Done false
2723 Acked 3722 Write 7 Done New Done false
2724 Acked 3723 Write 6 Done New Done false
2725 Acked 3724 Write 6 Done New Done false
2726 Acked 3725 Flush 0 Done New Done false
2727 Acked 3726 Write 3 Done New Done false
2728 Acked 3727 Read 9 Done New Done false
2729 Acked 3728 Read 6 Done New Done false
2730 Acked 3729 Flush 0 Done New Done false
2731 Acked 3730 Write 2 Done New Done false
2732 Acked 3731 Write 9 Done New Done false
2733 Acked 3732 Read 7 Done New Done false
2734 Acked 3733 Write 6 Done New Done false
2735 Acked 3734 Write 4 Done New Done false
2736 Acked 3735 Read 1 Done New Done false
2737 Acked 3736 Read 1 Done New Done false
2738 Acked 3737 Read 10 Done New Done false
2739 Acked 3738 Write 4 Done New Done false
2740 Acked 3739 Write 7 Done New Done false
2741 Acked 3740 Read 10 Done New Done false
2742 Acked 3741 Read 7 Done New Done false
2743 Acked 3742 Flush 0 Done New Done false
2744 Acked 3743 Read 7 Done New Done false
2745 Acked 3744 Read 3 Done New Done false
2746 Acked 3745 Read 1 Done New Done false
2747 Acked 3746 Read 8 Done New Done false
2748 Acked 3747 Flush 0 Done New Done false
2749 Acked 3748 Write 8 Done New Done false
2750 Acked 3749 Read 10 Done New Done false
2751 Acked 3750 Read 10 Done New Done false
2752 Acked 3751 Read 5 Done New Done false
2753 Acked 3752 Read 1 Done New Done false
2754 Acked 3753 Read 6 Done New Done false
2755 Acked 3754 Write 8 Done New Done false
2756 Acked 3755 Read 4 Done New Done false
2757 Acked 3756 Write 6 Done New Done false
2758 Acked 3757 Read 10 Done New Done false
2759 Acked 3758 Read 1 Done New Done false
2760 Acked 3759 Read 1 Done New Done false
2761 Acked 3760 Flush 0 Done New Done false
2762 Acked 3761 Read 10 Done New Done false
2763 Acked 3762 Read 9 Done New Done false
2764 Acked 3763 Flush 0 Done New Done false
2765 Acked 3764 Write 6 Done New Done false
2766 Acked 3765 Read 9 Done New Done false
2767 Acked 3766 Write 3 Done New Done false
2768 Acked 3767 Read 10 Done New Done false
2769 Acked 3768 Write 4 Done New Done false
2770 Acked 3769 Read 4 Done New Done false
2771 Acked 3770 Read 8 Done New Done false
2772 Acked 3771 Flush 0 Done New Done false
2773 Acked 3772 Read 10 Done New Done false
2774 Acked 3773 Write 7 Done New Done false
2775 Acked 3774 Write 6 Done New Done false
2776 Acked 3775 Read 6 Done New Done false
2777 Acked 3776 Read 10 Done New Done false
2778 Acked 3777 Write 8 Done New Done false
2779 Acked 3778 Read 10 Done New Done false
2780 Acked 3779 Read 7 Done New Done false
2781 Acked 3780 Flush 0 Done New Done false
2782 Acked 3781 Write 10 Done New Done false
2783 Acked 3782 Write 6 Done New Done false
2784 Acked 3783 Write 2 Done New Done false
2785 Acked 3784 Read 5 Done New Done false
2786 Acked 3785 Read 3 Done New Done false
2787 Acked 3786 Read 9 Done New Done false
2788 Acked 3787 Read 9 Done New Done false
2789 Acked 3788 Read 2 Done New Done false
2790 Acked 3789 Read 8 Done New Done false
2791 Acked 3790 Write 2 Done New Done false
2792 Acked 3791 Read 2 Done New Done false
2793 Acked 3792 Read 7 Done New Done false
2794 Acked 3793 Read 5 Done New Done false
2795 Acked 3794 Write 5 Done New Done false
2796 Acked 3795 Read 8 Done New Done false
2797 Acked 3796 Write 5 Done New Done false
2798 Acked 3797 Write 7 Done New Done false
2799 Acked 3798 Read 2 Done New Done false
2800 Acked 3799 Read 5 Done New Done false
2801 Acked 3800 Write 9 Done New Done false
2802 Acked 3801 Write 5 Done New Done false
2803 Acked 3802 Read 10 Done New Done false
2804 Acked 3803 Write 9 Done New Done false
2805 Acked 3804 Flush 0 Done New Done false
2806 Acked 3805 Write 3 Done New Done false
2807 Acked 3806 Write 2 Done New Done false
2808 Acked 3807 Read 5 Done New Done false
2809 Acked 3808 Read 7 Done New Done false
2810 Acked 3809 Write 10 Done New Done false
2811 Acked 3810 Write 4 Done New Done false
2812 Acked 3811 Read 9 Done New Done false
2813 Acked 3812 Read 9 Done New Done false
2814 Acked 3813 Read 4 Done New Done false
2815 Acked 3814 Read 7 Done New Done false
2816 Acked 3815 Read 5 Done New Done false
2817 Acked 3816 Read 10 Done New Done false
2818 Acked 3817 Read 5 Done New Done false
2819 Acked 3818 Write 5 Done New Done false
2820 Acked 3819 Read 2 Done New Done false
2821 Acked 3820 Write 7 Done New Done false
2822 Acked 3821 Read 6 Done New Done false
2823 Acked 3822 Read 6 Done New Done false
2824 Acked 3823 Read 2 Done New Done false
2825 Acked 3824 Write 9 Done New Done false
2826 Acked 3825 Read 1 Done New Done false
2827 Acked 3826 Write 8 Done New Done false
2828 Acked 3827 Read 5 Done New Done false
2829 Acked 3828 Read 6 Done New Done false
2830 Acked 3829 Read 6 Done New Done false
2831 Acked 3830 Flush 0 Done New Done false
2832 Acked 3831 Write 8 Done New Done false
2833 Acked 3832 Read 5 Done New Done false
2834 Acked 3833 Read 7 Done New Done false
2835 Acked 3834 Read 6 Done New Done false
2836 Acked 3835 Read 2 Done New Done false
2837 Acked 3836 Write 5 Done New Done false
2838 Acked 3837 Read 10 Done New Done false
2839 Acked 3838 Read 9 Done New Done false
2840 Acked 3839 Read 6 Done New Done false
2841 Acked 3840 Write 9 Done New Done false
2842 Acked 3841 Write 2 Done New Done false
2843 Acked 3842 Write 4 Done New Done false
2844 Acked 3843 Write 6 Done New Done false
2845 Acked 3844 Write 4 Done New Done false
2846 Acked 3845 Read 6 Done New Done false
2847 Acked 3846 Write 8 Done New Done false
2848 Acked 3847 Write 4 Done New Done false
2849 Acked 3848 Read 1 Done New Done false
2850 Acked 3849 Write 7 Done New Done false
2851 Acked 3850 Read 8 Done New Done false
2852 Acked 3851 Read 7 Done New Done false
2853 Acked 3852 Read 7 Done New Done false
2854 Acked 3853 Read 5 Done New Done false
2855 Acked 3854 Write 4 Done New Done false
2856 Acked 3855 Read 5 Done New Done false
2857 Acked 3856 Write 1 Done New Done false
2858 Acked 3857 Write 10 Done New Done false
2859 Acked 3858 Write 9 Done New Done false
2860 Acked 3859 Flush 0 Done New Done false
2861 Acked 3860 Flush 0 Done New Done false
2862 Acked 3861 Read 6 Done New Done false
2863 Acked 3862 Read 7 Done New Done false
2864 Acked 3863 Write 9 Done New Done false
2865 Acked 3864 Read 1 Done New Done false
2866 Acked 3865 Flush 0 Done New Done false
2867 Acked 3866 Read 7 Done New Done false
2868 Acked 3867 Read 2 Done New Done false
2869 Acked 3868 Write 4 Done New Done false
2870 Acked 3869 Read 5 Done New Done false
2871 Acked 3870 Read 5 Done New Done false
2872 Acked 3871 Write 6 Done New Done false
2873 Acked 3872 Write 2 Done New Done false
2874 Acked 3873 Write 3 Done New Done false
2875 Acked 3874 Flush 0 Done New Done false
2876 Acked 3875 Write 3 Done New Done false
2877 Acked 3876 Write 7 Done New Done false
2878 Acked 3877 Read 2 Done New Done false
2879 Acked 3878 Write 1 Done New Done false
2880 Acked 3879 Read 7 Done New Done false
2881 Acked 3880 Read 10 Done New Done false
2882 Acked 3881 Write 8 Done New Done false
2883 Acked 3882 Write 5 Done New Done false
2884 Acked 3883 Write 4 Done New Done false
2885 Acked 3884 Read 7 Done New Done false
2886 Acked 3885 Write 4 Done New Done false
2887 Acked 3886 Write 5 Done New Done false
2888 Acked 3887 Read 1 Done New Done false
2889 Acked 3888 Write 2 Done New Done false
2890 Acked 3889 Read 8 Done New Done false
2891 Acked 3890 Write 7 Done New Done false
2892 Acked 3891 Write 10 Done New Done false
2893 Acked 3892 Read 5 Done New Done false
2894 Acked 3893 Read 8 Done New Done false
2895 Acked 3894 Write 7 Done New Done false
2896 Acked 3895 Read 8 Done New Done false
2897 Acked 3896 Write 7 Done New Done false
2898 Acked 3897 Write 1 Done New Done false
2899 Acked 3898 Write 8 Done New Done false
2900 Acked 3899 Read 4 Done New Done false
2901 Acked 3900 Read 3 Done New Done false
2902 Acked 3901 Write 3 Done New Done false
2903 Acked 3902 Write 6 Done New Done false
2904 Acked 3903 Read 1 Done New Done false
2905 Acked 3904 Read 8 Done New Done false
2906 Acked 3905 Read 6 Done New Done false
2907 Acked 3906 Read 10 Done New Done false
2908 Acked 3907 Read 2 Done New Done false
2909 Acked 3908 Write 7 Done New Done false
2910 Acked 3909 Write 4 Done New Done false
2911 Acked 3910 Write 7 Done New Done false
2912 Acked 3911 Read 3 Done New Done false
2913 Acked 3912 Write 4 Done New Done false
2914 Acked 3913 Write 7 Done New Done false
2915 Acked 3914 Read 10 Done New Done false
2916 Acked 3915 Read 8 Done New Done false
2917 Acked 3916 Write 1 Done New Done false
2918 Acked 3917 Write 2 Done New Done false
2919 Acked 3918 Write 3 Done New Done false
2920 Acked 3919 Write 3 Done New Done false
2921 Acked 3920 Read 1 Done New Done false
2922 Acked 3921 Flush 0 Done New Done false
2923 Acked 3922 Write 2 Done New Done false
2924 Acked 3923 Flush 0 Done New Done false
2925 Acked 3924 Write 3 Done New Done false
2926 Acked 3925 Write 1 Done New Done false
2927 Acked 3926 Read 3 Done New Done false
2928 Acked 3927 Read 6 Done New Done false
2929 Acked 3928 Flush 0 Done New Done false
2930 Acked 3929 Read 6 Done New Done false
2931 Acked 3930 Read 6 Done New Done false
2932 Acked 3931 Read 4 Done New Done false
2933 Acked 3932 Write 9 Done New Done false
2934 Acked 3933 Read 4 Done New Done false
2935 Acked 3934 Read 2 Done New Done false
2936 Acked 3935 Read 3 Done New Done false
2937 Acked 3936 Flush 0 Done New Done false
2938 Acked 3937 Read 2 Done New Done false
2939 Acked 3938 Read 1 Done New Done false
2940 Acked 3939 Read 6 Done New Done false
2941 Acked 3940 Write 5 Done New Done false
2942 Acked 3941 Flush 0 Done New Done false
2943 Acked 3942 Read 5 Done New Done false
2944 Acked 3943 Read 6 Done New Done false
2945 Acked 3944 Read 3 Done New Done false
2946 Acked 3945 Read 4 Done New Done false
2947 Acked 3946 Flush 0 Done New Done false
2948 Acked 3947 Read 6 Done New Done false
2949 Acked 3948 Write 6 Done New Done false
2950 Acked 3949 Write 8 Done New Done false
2951 Acked 3950 Write 1 Done New Done false
2952 Acked 3951 Read 6 Done New Done false
2953 Acked 3952 Read 2 Done New Done false
2954 Acked 3953 Read 5 Done New Done false
2955 Acked 3954 Write 8 Done New Done false
2956 Acked 3955 Flush 0 Done New Done false
2957 Acked 3956 Read 4 Done New Done false
2958 Acked 3957 Read 5 Done New Done false
2959 Acked 3958 Write 4 Done New Done false
2960 Acked 3959 Write 5 Done New Done false
2961 Acked 3960 Read 7 Done New Done false
2962 Acked 3961 Write 1 Done New Done false
2963 Acked 3962 Write 3 Done New Done false
2964 Acked 3963 Read 2 Done New Done false
2965 Acked 3964 Read 7 Done New Done false
2966 Acked 3965 Read 8 Done New Done false
2967 Acked 3966 Read 7 Done New Done false
2968 Acked 3967 Write 10 Done New Done false
2969 Acked 3968 Read 5 Done New Done false
2970 Acked 3969 Write 4 Done New Done false
2971 Acked 3970 Write 7 Done New Done false
2972 Acked 3971 Read 10 Done New Done false
2973 Acked 3972 Flush 0 Done New Done false
2974 Acked 3973 Read 7 Done New Done false
2975 Acked 3974 Write 10 Done New Done false
2976 Acked 3975 Read 7 Done New Done false
2977 Acked 3976 Write 6 Done New Done false
2978 Acked 3977 Flush 0 Done New Done false
2979 Acked 3978 Read 10 Done New Done false
2980 Acked 3979 Write 5 Done New Done false
2981 Acked 3980 Read 10 Done New Done false
2982 Acked 3981 Flush 0 Done New Done false
2983 Acked 3982 Read 1 Done New Done false
2984 Acked 3983 Write 3 Done New Done false
2985 Acked 3984 Write 3 Done New Done false
2986 Acked 3985 Read 9 Done New Done false
2987 Acked 3986 Read 4 Done New Done false
2988 Acked 3987 Read 1 Done New Done false
2989 Acked 3988 Write 5 Done New Done false
2990 Acked 3989 Write 9 Done New Done false
2991 Acked 3990 Write 10 Done New Done false
2992 Acked 3991 Write 10 Done New Done false
2993 Acked 3992 Write 1 Done New Done false
2994 Acked 3993 Read 5 Done New Done false
2995 Acked 3994 Read 6 Done New Done false
2996 Acked 3995 Write 9 Done New Done false
2997 Acked 3996 Write 1 Done New Done false
2998 Acked 3997 Read 9 Done New Done false
2999 Acked 3998 Write 6 Done New Done false
3000 Acked 3999 Read 7 Done New Done false
3001 Acked 4000 Read 5 Done New Done false
3002 Acked 4001 Read 3 Done New Done false
3003 Acked 4002 Write 2 Done New Done false
3004 Acked 4003 Write 2 Done New Done false
3005 Acked 4004 Read 10 Done New Done false
3006 Acked 4005 Read 8 Done New Done false
3007 Acked 4006 Read 3 Done New Done false
3008 Acked 4007 Read 2 Done New Done false
3009 Acked 4008 Read 8 Done New Done false
3010 Acked 4009 Read 2 Done New Done false
3011 Acked 4010 Read 5 Done New Done false
3012 Acked 4011 Read 7 Done New Done false
3013 Acked 4012 Write 9 Done New Done false
3014 Acked 4013 Read 10 Done New Done false
STATES DS:0 DS:1 DS:2 TOTAL
New 0 301 0 301
Sent 0 0 0 0
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 3981 3712 3981
Downstairs last five completed: 3712 3711 3710 3709 3708
Upstairs last five completed: 3014 3013 3012 3011 3010
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:30.470744527Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:30.470795585Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:30.470814314Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:30.470831434Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:30.470844403Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 301 jobs since flush 3712","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:30.470859373Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:30.471012607Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:31.471957069Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:31.472014366Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:31.472033626Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:31.472055725Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:31.472069124Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 302 jobs since flush 3712","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:31.472089783Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:31.472254657Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1174)
[8820][1] reports Running
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:32.472900611Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:32.473016946Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:32.474466231Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:32.47449455Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 3712","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:32.474510469Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 3712","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:32.474695772Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:32.474719871Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:32.474734511Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:32.47474884Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:32.47476276Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 4014 4013 4012 4011 4010
Upstairs last five completed: 3015 3014 3013 3012 3011
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(2)
stop 2
Feb 13 21:41:34.202 INFO request completed, latency_us: 378, [8830] Got stop action so:false kr:false
response_code: 204, uri: /stop/cid/2, method: POST, req_id: 8045cda9-65a3-4fbd-80eb-ef79edbdf52c, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:34.203512861Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:34.203562279Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:34.203583639Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:34.203597308Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 4014","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:34.203657856Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:34.203679755Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:41:34.203 INFO request completed, latency_us: 126, response_code: 200, uri: /state/cid/2, method: GET, req_id: 380fdfba-7758-465e-b855-84ebf9e813b4, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:35.204828829Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:35.204889077Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:35.204911556Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:35.204932425Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:35.204948304Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 4014","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:35.204974014Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:35.204991943Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:36.205498583Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:36.20555789Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:36.205579939Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:36.205600499Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:36.205616358Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 4014","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:36.205633517Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:36.205650527Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:37.207217235Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:37.207283922Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:37.207307621Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:37.20732881Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:37.207360699Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 4014","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:37.207393818Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:37.207428387Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:41:38.205 INFO request completed, latency_us: 204, response_code: 200, uri: /state/cid/2, method: GET, req_id: 5629aba4-d485-4a29-bf89-fbac075edd9e, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Write block 1899 len 12288 data: 2 2 2
002/300 Write block 3610 len 32768 data: 3 4 4 4 4 4 4 4
003/300 Read block 1105 len 16384
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:38.208411507Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:38.208476735Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:38.208507324Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:38.208534512Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:38.208554522Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 3 jobs since flush 4014","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:38.208576801Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:38.20859895Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
004/300 Read block 1449 len 40960
005/300 Read block 1157 len 8192
006/300 Read block 4195 len 12288
007/300 Read block 41 len 32768
008/300 Write block 3588 len 36864 data: 7 6 5 5 5 5 3 2 2
009/300 Write block 2349 len 12288 data: 2 3 3
010/300 Flush
011/300 Read block 3539 len 40960
012/300 Read block 4373 len 16384
013/300 Write block 472 len 36864 data: 2 2 2 2 2 2 2 2 2
014/300 Flush
015/300 Write block 423 len 4096 data: 2
016/300 Read block 866 len 40960
017/300 Read block 1678 len 28672
018/300 Write block 2976 len 12288 data: 3 3 3
019/300 Write block 856 len 20480 data: 6 5 5 6 6
020/300 Read block 2743 len 24576
021/300 Write block 4898 len 36864 data: 4 3 3 3 3 3 3 3 3
022/300 Read block 4671 len 8192
023/300 Read block 88 len 28672
024/300 Read block 721 len 32768
025/300 Read block 4078 len 20480
026/300 Read block 3944 len 16384
027/300 Read block 1288 len 12288
028/300 Write block 2147 len 28672 data: 3 3 3 2 2 2 2
029/300 Write block 2982 len 36864 data: 4 3 3 4 4 4 3 3 4
030/300 Read block 4106 len 28672
031/300 Write block 943 len 16384 data: 2 2 2 2
032/300 Read block 2750 len 40960
033/300 Read block 4794 len 24576
034/300 Read block 2705 len 4096
035/300 Read block 2375 len 28672
036/300 Read block 4586 len 24576
037/300 Write block 3320 len 40960 data: 2 2 2 2 3 3 2 2 2 2
038/300 Write block 3790 len 4096 data: 4
039/300 Read block 891 len 8192
040/300 Flush
041/300 Read block 4672 len 28672
042/300 Read block 4683 len 24576
043/300 Write block 1492 len 20480 data: 2 2 2 3 4
044/300 Write block 3886 len 12288 data: 4 2 2
045/300 Write block 4666 len 28672 data: 5 5 5 5 5 5 3
046/300 Write block 2127 len 4096 data: 2
047/300 Read block 4215 len 24576
048/300 Read block 4029 len 4096
049/300 Read block 3217 len 20480
050/300 Read block 2964 len 32768
051/300 Read block 1201 len 16384
052/300 Read block 2036 len 24576
053/300 Read block 4948 len 40960
054/300 Write block 4318 len 12288 data: 3 3 4
055/300 Write block 3221 len 12288 data: 7 7 7
056/300 Write block 4900 len 20480 data: 4 4 4 4 4
057/300 Read block 1566 len 8192
058/300 Write block 2207 len 24576 data: 2 2 2 2 3 2
059/300 Write block 2574 len 40960 data: 7 6 5 4 4 3 4 3 4 5
060/300 Write block 2892 len 16384 data: 3 3 3 5
061/300 Read block 4707 len 32768
062/300 Write block 1568 len 8192 data: 4 4
063/300 Flush
064/300 Read block 2932 len 16384
065/300 Read block 1973 len 32768
066/300 Read block 780 len 28672
067/300 Flush
068/300 Flush
069/300 Read block 1847 len 12288
070/300 Write block 263 len 36864 data: 2 2 2 2 3 4 4 4 5
071/300 Write block 4383 len 32768 data: 6 6 5 5 5 4 3 4
072/300 Read block 4471 len 36864
073/300 Read block 65 len 24576
074/300 Write block 1531 len 24576 data: 2 2 2 2 2 3
075/300 Read block 2349 len 4096
076/300 Write block 1196 len 28672 data: 3 3 2 2 3 3 4
077/300 Write block 4365 len 40960 data: 2 3 3 4 4 4 4 2 2 2
078/300 Write block 4316 len 32768 data: 3 3 4 4 5 4 5 6
079/300 Read block 2898 len 40960
080/300 Read block 927 len 24576
081/300 Flush
082/300 Write block 2286 len 32768 data: 4 4 4 4 4 3 3 3
083/300 Write block 991 len 36864 data: 5 6 4 4 3 3 4 4 4
084/300 Write block 692 len 8192 data: 3 2
085/300 Read block 3916 len 40960
086/300 Read block 1352 len 24576
087/300 Read block 3488 len 12288
088/300 Write block 1391 len 36864 data: 5 3 3 3 3 3 3 2 2
089/300 Write block 4696 len 12288 data: 3 2 2
090/300 Read block 246 len 16384
091/300 Write block 3459 len 20480 data: 2 2 2 2 2
092/300 Write block 3059 len 24576 data: 3 3 3 4 3 3
093/300 Read block 3441 len 32768
094/300 Read block 1753 len 12288
095/300 Write block 1960 len 12288 data: 2 2 4
096/300 Read block 4832 len 24576
097/300 Read block 4469 len 8192
098/300 Write block 2471 len 40960 data: 4 4 3 3 3 3 3 2 2 2
099/300 Read block 2752 len 36864
100/300 Read block 3073 len 8192
101/300 Write block 3796 len 12288 data: 2 3 3
102/300 Read block 900 len 12288
103/300 Read block 2095 len 8192
104/300 Read block 994 len 20480
105/300 Read block 1508 len 20480
106/300 Read block 811 len 8192
107/300 Write block 2760 len 28672 data: 3 3 3 2 2 2 2
108/300 Flush
109/300 Flush
110/300 Write block 984 len 24576 data: 2 2 4 4 3 3
111/300 Read block 4134 len 28672
112/300 Write block 4057 len 24576 data: 3 3 3 3 3 3
113/300 Read block 3758 len 40960
114/300 Read block 1807 len 12288
115/300 Read block 2219 len 16384
116/300 Read block 4315 len 4096
117/300 Read block 1561 len 20480
118/300 Flush
119/300 Write block 526 len 20480 data: 5 5 5 4 4
120/300 Read block 382 len 28672
121/300 Write block 219 len 8192 data: 2 3
122/300 Write block 2503 len 16384 data: 7 7 5 5
123/300 Write block 2182 len 40960 data: 3 3 3 2 3 3 2 2 2 2
124/300 Read block 1689 len 20480
125/300 Write block 4363 len 4096 data: 2
126/300 Read block 239 len 32768
127/300 Read block 72 len 36864
128/300 Write block 2261 len 28672 data: 2 2 2 2 2 2 2
129/300 Write block 428 len 12288 data: 3 3 2
130/300 Write block 2848 len 40960 data: 4 3 3 3 4 3 3 4 3 3
131/300 Read block 4886 len 24576
132/300 Read block 2201 len 40960
133/300 Write block 1067 len 20480 data: 3 3 3 3 3
134/300 Read block 3907 len 24576
135/300 Write block 3858 len 20480 data: 5 5 4 2 2
136/300 Read block 3698 len 28672
137/300 Flush
138/300 Write block 2392 len 24576 data: 4 4 4 4 4 3
139/300 Read block 4299 len 8192
140/300 Write block 3170 len 32768 data: 2 4 4 4 3 3 3 3
141/300 Read block 4080 len 32768
142/300 Write block 1056 len 8192 data: 2 3
143/300 Read block 2415 len 40960
144/300 Read block 3987 len 20480
145/300 Flush
146/300 Write block 816 len 12288 data: 3 3 3
147/300 Write block 3004 len 16384 data: 3 4 5 3
148/300 Write block 751 len 12288 data: 3 3 3
149/300 Read block 2139 len 36864
150/300 Read block 4087 len 32768
151/300 Read block 2588 len 40960
152/300 Read block 4908 len 24576
153/300 Read block 3453 len 4096
154/300 Read block 2439 len 16384
155/300 Write block 1760 len 28672 data: 3 5 4 5 5 3 4
156/300 Flush
157/300 Read block 2420 len 16384
158/300 Write block 306 len 24576 data: 3 2 2 3 3 3
159/300 Write block 4916 len 24576 data: 2 2 2 2 2 2
160/300 Write block 4322 len 36864 data: 6 7 6 6 6 6 4 4 4
161/300 Write block 880 len 32768 data: 3 3 3 3 6 6 7 6
162/300 Read block 3984 len 8192
163/300 Write block 3223 len 24576 data: 8 8 8 8 7 7
164/300 Write block 1803 len 32768 data: 4 4 3 3 3 2 2 3
165/300 Read block 66 len 40960
166/300 Read block 2671 len 28672
167/300 Read block 704 len 28672
168/300 Write block 1101 len 4096 data: 3
169/300 Flush
170/300 Write block 562 len 28672 data: 4 4 4 6 6 6 5
171/300 Write block 3921 len 20480 data: 3 3 4 4 4
172/300 Write block 3188 len 40960 data: 2 2 2 2 4 4 5 4 6 7
173/300 Write block 4079 len 40960 data: 2 2 2 2 2 2 2 2 2 4
174/300 Read block 3137 len 32768
175/300 Read block 2792 len 24576
176/300 Read block 2367 len 40960
177/300 Write block 2409 len 16384 data: 4 4 4 3
178/300 Read block 3531 len 20480
179/300 Write block 3191 len 24576 data: 3 5 5 6 5 7
180/300 Read block 2840 len 20480
181/300 Write block 374 len 20480 data: 3 3 2 2 2
182/300 Flush
183/300 Write block 4154 len 36864 data: 2 3 3 3 4 4 4 4 3
184/300 Read block 64 len 28672
185/300 Read block 2776 len 40960
186/300 Write block 2328 len 36864 data: 5 4 5 5 5 4 4 4 4
187/300 Read block 1052 len 4096
188/300 Read block 2268 len 8192
189/300 Write block 3135 len 24576 data: 3 3 3 3 3 3
190/300 Read block 876 len 20480
191/300 Read block 4447 len 12288
192/300 Write block 1 len 24576 data: 3 4 4 4 3 4
193/300 Write block 3295 len 16384 data: 2 3 3 3
194/300 Write block 4722 len 36864 data: 7 6 6 6 5 4 4 5 5
195/300 Flush
196/300 Read block 1352 len 16384
197/300 Read block 3638 len 32768
198/300 Read block 300 len 36864
199/300 Write block 2539 len 8192 data: 3 3
200/300 Read block 3893 len 4096
201/300 Read block 4921 len 32768
202/300 Read block 4442 len 8192
203/300 Read block 2822 len 20480
204/300 Write block 4985 len 12288 data: 3 3 4
205/300 Read block 3155 len 16384
206/300 Read block 3012 len 32768
207/300 Write block 3477 len 28672 data: 2 2 2 2 2 3 3
208/300 Write block 917 len 32768 data: 2 3 3 3 3 3 3 3
209/300 Write block 3404 len 40960 data: 5 5 5 3 4 4 4 5 4 3
210/300 Read block 2064 len 24576
211/300 Write block 744 len 24576 data: 4 4 4 4 4 4
212/300 Read block 3072 len 32768
213/300 Write block 2299 len 28672 data: 4 5 5 6 6 6 5
214/300 Write block 3349 len 12288 data: 3 4 5
215/300 Read block 109 len 24576
216/300 Write block 3225 len 36864 data: 9 9 8 8 5 4 4 3 3
217/300 Read block 2801 len 24576
218/300 Write block 518 len 4096 data: 5
219/300 Write block 3917 len 16384 data: 3 3 3 3
220/300 Write block 686 len 12288 data: 6 6 6
221/300 Write block 4712 len 8192 data: 4 3
222/300 Write block 4500 len 4096 data: 3
223/300 Write block 1060 len 4096 data: 3
224/300 Write block 3318 len 36864 data: 3 2 3 3 3 3 4 4 3
225/300 Write block 2374 len 20480 data: 5 4 3 3 3
226/300 Write block 381 len 36864 data: 3 4 4 4 4 5 3 4 4
227/300 Write block 163 len 16384 data: 8 9 9 6
228/300 Flush
229/300 Flush
230/300 Read block 2439 len 12288
231/300 Read block 2131 len 8192
232/300 Write block 191 len 28672 data: 3 3 3 3 3 2 2
233/300 Write block 151 len 40960 data: 4 4 4 3 3 2 4 4 4 4
234/300 Read block 3771 len 16384
235/300 Read block 3604 len 36864
236/300 Read block 1780 len 16384
237/300 Flush
238/300 Write block 2985 len 32768 data: 5 5 5 4 4 5 3 3
239/300 Write block 4664 len 36864 data: 5 5 6 6 6 6 6 6 4
240/300 Write block 2745 len 20480 data: 2 2 2 2 2
241/300 Flush
242/300 Write block 4077 len 36864 data: 3 3 3 3 3 3 3 3 3
243/300 Read block 873 len 12288
244/300 Write block 3668 len 4096 data: 4
245/300 Read block 3798 len 4096
246/300 Write block 4972 len 28672 data: 2 2 2 2 2 2 2
247/300 Write block 2210 len 24576 data: 3 4 3 2 2 2
248/300 Read block 1121 len 8192
249/300 Write block 2429 len 32768 data: 2 3 3 3 3 3 3 3
250/300 Flush
251/300 Write block 1813 len 12288 data: 2 2 2
252/300 Write block 3557 len 4096 data: 7
253/300 Read block 369 len 16384
254/300 Write block 4618 len 20480 data: 2 2 2 3 3
255/300 Read block 3016 len 32768
256/300 Read block 1261 len 32768
257/300 Write block 3902 len 40960 data: 4 4 4 3 3 3 3 3 3 3
258/300 Read block 3166 len 8192
259/300 Read block 4942 len 12288
260/300 Write block 1101 len 12288 data: 4 3 5
261/300 Flush
262/300 Write block 3041 len 20480 data: 2 3 3 3 2
263/300 Read block 2500 len 40960
264/300 Flush
265/300 Write block 1383 len 40960 data: 2 2 3 3 3 3 4 4 6 4
266/300 Read block 837 len 20480
267/300 Read block 2774 len 12288
268/300 Write block 2678 len 8192 data: 3 3
269/300 Read block 1246 len 28672
270/300 Read block 2120 len 8192
271/300 Write block 1893 len 32768 data: 2 2 3 3 3 3 3 3
272/300 Write block 1751 len 4096 data: 3
273/300 Write block 4169 len 32768 data: 3 3 2 2 2 2 2 3
274/300 Read block 1574 len 20480
275/300 Read block 67 len 40960
276/300 Read block 143 len 8192
277/300 Write block 1298 len 28672 data: 3 3 4 4 4 3 3
278/300 Read block 413 len 12288
279/300 Write block 4436 len 8192 data: 3 3
280/300 Read block 395 len 36864
281/300 Write block 2132 len 12288 data: 5 5 5
282/300 Read block 171 len 40960
283/300 Read block 2896 len 16384
284/300 Write block 641 len 16384 data: 5 4 4 4
285/300 Read block 3475 len 36864
286/300 Read block 752 len 20480
287/300 Read block 2043 len 40960
288/300 Read block 2105 len 12288
289/300 Read block 1311 len 8192
290/300 Write block 3076 len 28672 data: 4 4 4 3 3 3 3
291/300 Write block 2611 len 36864 data: 3 3 2 2 2 2 2 2 2
292/300 Write block 550 len 4096 data: 2
293/300 Write block 1658 len 36864 data: 5 6 4 4 3 3 2 2 2
294/300 Write block 4933 len 36864 data: 4 3 3 3 3 3 3 4 4
295/300 Read block 2764 len 4096
296/300 Write block 644 len 36864 data: 5 4 3 3 3 2 3 3 3
297/300 Flush
298/300 Read block 2849 len 8192
299/300 Flush
300/300 Flush
Main task has work to do, go find it
got dsc Start(2)
start 2
Feb 13 21:41:38.924 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/2, method: POST, req_id: 21ddecb1-883b-4ad4-a5e2-81618f461e27, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
3016 Acked 4015 Write 3 Done Done New true
3017 Acked 4016 Write 8 Done Done New true
3018 Acked 4017 Read 4 Done Done New true
3019 Acked 4018 Read 10 Done Done New false
3020 Acked 4019 Read 2 Done Done New false
3021 Acked 4020 Read 3 Done Done New false
3022 Acked 4021 Read 8 Done Done New false
3023 Acked 4022 Write 9 Done Done New false
3024 Acked 4023 Write 3 Done Done New false
3025 Acked 4024 Flush 0 Done Done New false
3026 Acked 4025 Read 10 Done Done New false
3027 Acked 4026 Read 4 Done Done New false
3028 Acked 4027 Write 9 Done Done New false
3029 Acked 4028 Flush 0 Done Done New false
3030 Acked 4029 Write 1 Done Done New false
3031 Acked 4030 Read 10 Done Done New false
3032 Acked 4031 Read 7 Done Done New false
3033 Acked 4032 Write 3 Done Done New false
3034 Acked 4033 Write 5 Done Done New false
3035 Acked 4034 Read 6 Done Done New false
3036 Acked 4035 Write 9 Done Done New false
3037 Acked 4036 Read 2 Done Done New false
3038 Acked 4037 Read 7 Done Done New false
3039 Acked 4038 Read 8 Done Done New false
3040 Acked 4039 Read 5 Done Done New false
3041 Acked 4040 Read 4 Done Done New false
3042 Acked 4041 Read 3 Done Done New false
3043 Acked 4042 Write 7 Done Done New false
3044 Acked 4043 Write 9 Done Done New false
3045 Acked 4044 Read 7 Done Done New false
3046 Acked 4045 Write 4 Done Done New false
3047 Acked 4046 Read 10 Done Done New false
3048 Acked 4047 Read 6 Done Done New false
3049 Acked 4048 Read 1 Done Done New false
3050 Acked 4049 Read 7 Done Done New false
3051 Acked 4050 Read 6 Done Done New false
3052 Acked 4051 Write 10 Done Done New false
3053 Acked 4052 Write 1 Done Done New false
3054 Acked 4053 Read 2 Done Done New false
3055 Acked 4054 Flush 0 Done Done New false
3056 Acked 4055 Read 7 Done Done New false
3057 Acked 4056 Read 6 Done Done New false
3058 Acked 4057 Write 5 Done Done New false
3059 Acked 4058 Write 3 Done Done New false
3060 Acked 4059 Write 7 Done Done New false
3061 Acked 4060 Write 1 Done Done New false
3062 Acked 4061 Read 6 Done Done New false
3063 Acked 4062 Read 1 Done Done New false
3064 Acked 4063 Read 5 Done Done New false
3065 Acked 4064 Read 8 Done Done New false
3066 Acked 4065 Read 4 Done Done New false
3067 Acked 4066 Read 6 Done Done New false
3068 Acked 4067 Read 10 Done Done New false
3069 Acked 4068 Write 3 Done Done New false
3070 Acked 4069 Write 3 Done Done New false
3071 Acked 4070 Write 5 Done Done New false
3072 Acked 4071 Read 2 Done Done New false
3073 Acked 4072 Write 6 Done Done New false
3074 Acked 4073 Write 10 Done Done New false
3075 Acked 4074 Write 4 Done Done New false
3076 Acked 4075 Read 8 Done Done New false
3077 Acked 4076 Write 2 Done Done New false
3078 Acked 4077 Flush 0 Done Done New false
3079 Acked 4078 Read 4 Done Done New false
3080 Acked 4079 Read 8 Done Done New false
3081 Acked 4080 Read 7 Done Done New false
3082 Acked 4081 Flush 0 Done Done New false
3083 Acked 4082 Flush 0 Done Done New false
3084 Acked 4083 Read 3 Done Done New false
3085 Acked 4084 Write 9 Done Done New false
3086 Acked 4085 Write 8 Done Done New false
3087 Acked 4086 Read 9 Done Done New false
3088 Acked 4087 Read 6 Done Done New false
3089 Acked 4088 Write 6 Done Done New false
3090 Acked 4089 Read 1 Done Done New false
3091 Acked 4090 Write 7 Done Done New false
3092 Acked 4091 Write 10 Done Done New false
3093 Acked 4092 Write 8 Done Done New false
3094 Acked 4093 Read 10 Done Done New false
3095 Acked 4094 Read 6 Done Done New false
3096 Acked 4095 Flush 0 Done Done New false
3097 Acked 4096 Write 8 Done Done New false
3098 Acked 4097 Write 9 Done Done New false
3099 Acked 4098 Write 2 Done Done New false
3100 Acked 4099 Read 10 Done Done New false
3101 Acked 4100 Read 6 Done Done New false
3102 Acked 4101 Read 3 Done Done New false
3103 Acked 4102 Write 9 Done Done New false
3104 Acked 4103 Write 3 Done Done New false
3105 Acked 4104 Read 4 Done Done New false
3106 Acked 4105 Write 5 Done Done New false
3107 Acked 4106 Write 6 Done Done New false
3108 Acked 4107 Read 8 Done Done New false
3109 Acked 4108 Read 3 Done Done New false
3110 Acked 4109 Write 3 Done Done New false
3111 Acked 4110 Read 6 Done Done New false
3112 Acked 4111 Read 2 Done Done New false
3113 Acked 4112 Write 10 Done Done New false
3114 Acked 4113 Read 9 Done Done New false
3115 Acked 4114 Read 2 Done Done New false
3116 Acked 4115 Write 3 Done Done New false
3117 Acked 4116 Read 3 Done Done New false
3118 Acked 4117 Read 2 Done Done New false
3119 Acked 4118 Read 5 Done Done New false
3120 Acked 4119 Read 5 Done Done New false
3121 Acked 4120 Read 2 Done Done New false
3122 Acked 4121 Write 7 Done Done New false
3123 Acked 4122 Flush 0 Done Done New false
3124 Acked 4123 Flush 0 Done Done New false
3125 Acked 4124 Write 6 Done Done New false
3126 Acked 4125 Read 7 Done Done New false
3127 Acked 4126 Write 6 Done Done New false
3128 Acked 4127 Read 10 Done Done New false
3129 Acked 4128 Read 3 Done Done New false
3130 Acked 4129 Read 4 Done Done New false
3131 Acked 4130 Read 1 Done Done New false
3132 Acked 4131 Read 5 Done Done New false
3133 Acked 4132 Flush 0 Done Done New false
3134 Acked 4133 Write 5 Done Done New false
3135 Acked 4134 Read 7 Done Done New false
3136 Acked 4135 Write 2 Done Done New false
3137 Acked 4136 Write 4 Done Done New false
3138 Acked 4137 Write 10 Done Done New false
3139 Acked 4138 Read 5 Done Done New false
3140 Acked 4139 Write 1 Done Done New false
3141 Acked 4140 Read 8 Done Done New false
3142 Acked 4141 Read 9 Done Done New false
3143 Acked 4142 Write 7 Done Done New false
3144 Acked 4143 Write 3 Done Done New false
3145 Acked 4144 Write 10 Done Done New false
3146 Acked 4145 Read 6 Done Done New false
3147 Acked 4146 Read 10 Done Done New false
3148 Acked 4147 Write 5 Done Done New false
3149 Acked 4148 Read 6 Done Done New false
3150 Acked 4149 Write 5 Done Done New false
3151 Acked 4150 Read 7 Done Done New false
3152 Acked 4151 Flush 0 Done Done New false
3153 Acked 4152 Write 6 Done Done New false
3154 Acked 4153 Read 2 Done Done New false
3155 Acked 4154 Write 8 Done Done New false
3156 Acked 4155 Read 8 Done Done New false
3157 Acked 4156 Write 2 Done Done New false
3158 Acked 4157 Read 10 Done Done New false
3159 Acked 4158 Read 5 Done Done New false
3160 Acked 4159 Flush 0 Done Done New false
3161 Acked 4160 Write 3 Done Done New false
3162 Acked 4161 Write 4 Done Done New false
3163 Acked 4162 Write 3 Done Done New false
3164 Acked 4163 Read 9 Done Done New false
3165 Acked 4164 Read 8 Done Done New false
3166 Acked 4165 Read 10 Done Done New false
3167 Acked 4166 Read 6 Done Done New false
3168 Acked 4167 Read 1 Done Done New false
3169 Acked 4168 Read 4 Done Done New false
3170 Acked 4169 Write 7 Done Done New false
3171 Acked 4170 Flush 0 Done Done New false
3172 Acked 4171 Read 4 Done Done New false
3173 Acked 4172 Write 6 Done Done New false
3174 Acked 4173 Write 6 Done Done New false
3175 Acked 4174 Write 9 Done Done New false
3176 Acked 4175 Write 8 Done Done New false
3177 Acked 4176 Read 2 Done Done New false
3178 Acked 4177 Write 6 Done Done New false
3179 Acked 4178 Write 8 Done Done New false
3180 Acked 4179 Read 10 Done Done New false
3181 Acked 4180 Read 7 Done Done New false
3182 Acked 4181 Read 7 Done Done New false
3183 Acked 4182 Write 1 Done Done New false
3184 Acked 4183 Flush 0 Done Done New false
3185 Acked 4184 Write 7 Done Done New false
3186 Acked 4185 Write 5 Done Done New false
3187 Acked 4186 Write 10 Done Done New false
3188 Acked 4187 Write 10 Done Done New false
3189 Acked 4188 Read 8 Done Done New false
3190 Acked 4189 Read 6 Done Done New false
3191 Acked 4190 Read 10 Done Done New false
3192 Acked 4191 Write 4 Done Done New false
3193 Acked 4192 Read 5 Done Done New false
3194 Acked 4193 Write 6 Done Done New false
3195 Acked 4194 Read 5 Done Done New false
3196 Acked 4195 Write 5 Done Done New false
3197 Acked 4196 Flush 0 Done Done New false
3198 Acked 4197 Write 9 Done Done New false
3199 Acked 4198 Read 7 Done Done New false
3200 Acked 4199 Read 10 Done Done New false
3201 Acked 4200 Write 9 Done Done New false
3202 Acked 4201 Read 1 Done Done New false
3203 Acked 4202 Read 2 Done Done New false
3204 Acked 4203 Write 6 Done Done New false
3205 Acked 4204 Read 5 Done Done New false
3206 Acked 4205 Read 3 Done Done New false
3207 Acked 4206 Write 6 Done Done New false
3208 Acked 4207 Write 4 Done Done New false
3209 Acked 4208 Write 9 Done Done New false
3210 Acked 4209 Flush 0 Done Done New false
3211 Acked 4210 Read 4 Done Done New false
3212 Acked 4211 Read 8 Done Done New false
3213 Acked 4212 Read 9 Done Done New false
3214 Acked 4213 Write 2 Done Done New false
3215 Acked 4214 Read 1 Done Done New false
3216 Acked 4215 Read 8 Done Done New false
3217 Acked 4216 Read 2 Done Done New false
3218 Acked 4217 Read 5 Done Done New false
3219 Acked 4218 Write 3 Done Done New false
3220 Acked 4219 Read 4 Done Done New false
3221 Acked 4220 Read 8 Done Done New false
3222 Acked 4221 Write 7 Done Done New false
3223 Acked 4222 Write 8 Done Done New false
3224 Acked 4223 Write 10 Done Done New false
3225 Acked 4224 Read 6 Done Done New false
3226 Acked 4225 Write 6 Done Done New false
3227 Acked 4226 Read 8 Done Done New false
3228 Acked 4227 Write 7 Done Done New false
3229 Acked 4228 Write 3 Done Done New false
3230 Acked 4229 Read 6 Done Done New false
3231 Acked 4230 Write 9 Done Done New false
3232 Acked 4231 Read 6 Done Done New false
3233 Acked 4232 Flush 0 Done Done New false
3234 Acked 4233 Write 1 Done Done New false
3235 Acked 4234 Write 4 Done Done New false
3236 Acked 4235 Write 3 Done Done New false
3237 Acked 4236 Write 2 Done Done New false
3238 Acked 4237 Write 1 Done Done New false
3239 Acked 4238 Write 1 Done Done New false
3240 Acked 4239 Write 9 Done Done New false
3241 Acked 4240 Write 5 Done Done New false
3242 Acked 4241 Write 9 Done Done New false
3243 Acked 4242 Write 4 Done Done New false
3244 Acked 4243 Flush 0 Done Done New false
3245 Acked 4244 Flush 0 Done Done New false
3246 Acked 4245 Read 3 Done Done New false
3247 Acked 4246 Read 2 Done Done New false
3248 Acked 4247 Write 7 Done Done New false
3249 Acked 4248 Write 10 Done Done New false
3250 Acked 4249 Read 4 Done Done New false
3251 Acked 4250 Read 9 Done Done New false
3252 Acked 4251 Read 4 Done Done New false
3253 Acked 4252 Flush 0 Done Done New false
3254 Acked 4253 Write 8 Done Done New false
3255 Acked 4254 Write 9 Done Done New false
3256 Acked 4255 Write 5 Done Done New false
3257 Acked 4256 Flush 0 Done Done New false
3258 Acked 4257 Write 9 Done Done New false
3259 Acked 4258 Read 3 Done Done New false
3260 Acked 4259 Write 1 Done Done New false
3261 Acked 4260 Read 1 Done Done New false
3262 Acked 4261 Write 7 Done Done New false
3263 Acked 4262 Write 6 Done Done New false
3264 Acked 4263 Read 2 Done Done New false
3265 Acked 4264 Write 8 Done Done New false
3266 Acked 4265 Flush 0 Done Done New false
3267 Acked 4266 Write 3 Done Done New false
3268 Acked 4267 Write 1 Done Done New false
3269 Acked 4268 Read 4 Done Done New false
3270 Acked 4269 Write 5 Done Done New false
3271 Acked 4270 Read 8 Done Done New false
3272 Acked 4271 Read 8 Done Done New false
3273 Acked 4272 Write 10 Done Done New false
3274 Acked 4273 Read 2 Done Done New false
3275 Acked 4274 Read 3 Done Done New false
3276 Acked 4275 Write 3 Done Done New false
3277 Acked 4276 Flush 0 Done Done New false
3278 Acked 4277 Write 5 Done Done New false
3279 Acked 4278 Read 10 Done Done New false
3280 Acked 4279 Flush 0 Done Done New false
3281 Acked 4280 Write 10 Done Done New false
3282 Acked 4281 Read 5 Done Done New false
3283 Acked 4282 Read 3 Done Done New false
3284 Acked 4283 Write 2 Done Done New false
3285 Acked 4284 Read 7 Done Done New false
3286 Acked 4285 Read 2 Done Done New false
3287 Acked 4286 Write 8 Done Done New false
3288 Acked 4287 Write 1 Done Done New false
3289 Acked 4288 Write 8 Done Done New false
3290 Acked 4289 Read 5 Done Done New false
3291 Acked 4290 Read 10 Done Done New false
3292 Acked 4291 Read 2 Done Done New false
3293 Acked 4292 Write 7 Done Done New false
3294 Acked 4293 Read 3 Done Done New false
3295 Acked 4294 Write 2 Done Done New false
3296 Acked 4295 Read 9 Done Done New false
3297 Acked 4296 Write 3 Done Done New false
3298 Acked 4297 Read 10 Done Done New false
3299 Acked 4298 Read 4 Done Done New false
3300 Acked 4299 Write 4 Done Done New false
3301 Acked 4300 Read 9 Done Done New false
3302 Acked 4301 Read 5 Done Done New false
3303 Acked 4302 Read 10 Done Done New false
3304 Acked 4303 Read 3 Done Done New false
3305 Acked 4304 Read 2 Done Done New false
3306 Acked 4305 Write 7 Done Done New false
3307 Acked 4306 Write 9 Done Done New false
3308 Acked 4307 Write 1 Done Done New false
3309 Acked 4308 Write 9 Done Done New false
3310 Acked 4309 Write 9 Done Done New false
3311 Acked 4310 Read 1 Done Done New false
3312 Acked 4311 Write 9 Done Done New false
3313 Acked 4312 Flush 0 Done Done New false
3314 Acked 4313 Read 2 Done Done New false
3315 Acked 4314 Flush 0 Done Done New false
3316 Acked 4315 Flush 0 Done Done New false
STATES DS:0 DS:1 DS:2 TOTAL
New 0 0 301 301
Sent 0 0 0 0
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 4315 4315 4014
Downstairs last five completed: 4014 4013 4012 4011 4010
Upstairs last five completed: 3316 3315 3314 3313 3312
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:39.209710626Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:39.209775594Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:39.209799693Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:39.209820502Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:39.209836622Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 301 jobs since flush 4014","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:39.209854031Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:39.209998375Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:40.210570212Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:40.210634219Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:40.210657048Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:40.210677668Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:40.210693817Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 301 jobs since flush 4014","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:40.210711436Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:40.210857981Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1175)
[8830][2] reports Running
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:41.21164209Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:41.211704557Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:41.212983258Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:41.213016607Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 4014","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:41.213035096Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 4014","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:41.213178241Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:41.21320276Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:41.213220659Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:41.213237829Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:41.213254778Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 4315 4314 4313 4312 4311
Upstairs last five completed: 3316 3315 3314 3313 3312
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Feb 13 21:41:42.927 INFO request completed, latency_us: 345, response_code: 204, uri: [8810] Got stop action so:false kr:false
/stop/cid/0, method: POST, req_id: 880f7746-a448-438c-aaf0-aee5533d11bb, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:42.928676929Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:41:42.928 INFO request completed, latency_us: 120, response_code: 200, uri: /state/cid/0, method: GET, req_id: fa819bc7-8100-49eb-a26b-1f4791bcc1da{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:42.928767336Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:42.928828223Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:42.928848112Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 4315","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:42.928863332Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:42.928877461Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:43.929566553Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:43.92963891Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:43.929664099Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:43.929685159Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:43.929701018Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 4315","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:43.929718447Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:43.929735627Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:44.931315715Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:44.931385213Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:44.931408972Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:44.931429811Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:44.93144615Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 4315","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:44.931463819Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:44.931481059Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:45.933133194Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:45.933205461Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:45.933247759Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:45.933278738Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:45.933302727Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 4315","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:45.933323667Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:45.933341236Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:41:46.930 INFO request completed, latency_us: 186, response_code: 200, uri: /state/cid/0, method: GET, req_id: 0bf7beff-d8fb-4951-92b7-c9b582f6bb21, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Write block 4396 len 28672 data: 3 4 4 4 4 5 4
002/300 Write block 3157 len 28672 data: 2 2 2 2 2 2 2
003/300 Write block 4292 len 24576 data: 4 4 3 3 3 3
004/300 Write block 3811 len 16384 data: 4 4 3 4
005/300 Flush
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:46.934396914Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:46.934468641Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:46.934503219Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:46.934532608Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:46.934554197Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 5 jobs since flush 4315","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:46.934579436Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:46.934603206Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
006/300 Read block 2315 len 16384
007/300 Write block 2944 len 24576 data: 4 4 3 3 3 2
008/300 Write block 1002 len 28672 data: 3 4 3 3 2 2 2
009/300 Read block 4932 len 12288
010/300 Read block 2360 len 12288
011/300 Read block 1862 len 20480
012/300 Write block 4151 len 24576 data: 3 2 2 3 4 4
013/300 Read block 1232 len 28672
014/300 Write block 409 len 8192 data: 3 2
015/300 Read block 2775 len 36864
016/300 Write block 3322 len 8192 data: 4 4
017/300 Write block 3961 len 24576 data: 2 2 2 3 3 3
018/300 Read block 2779 len 8192
019/300 Write block 1019 len 20480 data: 3 2 2 2 2
020/300 Write block 4113 len 20480 data: 4 6 6 6 6
021/300 Write block 893 len 24576 data: 4 4 4 3 3 3
022/300 Flush
023/300 Read block 1355 len 32768
024/300 Read block 4120 len 16384
025/300 Flush
026/300 Read block 1005 len 36864
027/300 Read block 4590 len 36864
028/300 Write block 4161 len 16384 data: 5 4 3 3
029/300 Flush
030/300 Flush
031/300 Write block 3430 len 40960 data: 5 4 4 3 4 4 4 3 3 2
032/300 Read block 4894 len 16384
033/300 Write block 4698 len 20480 data: 3 2 3 3 3
034/300 Read block 1302 len 20480
035/300 Flush
036/300 Read block 2831 len 36864
037/300 Read block 1482 len 36864
038/300 Read block 1034 len 4096
039/300 Read block 129 len 16384
040/300 Flush
041/300 Flush
042/300 Flush
043/300 Read block 4389 len 28672
044/300 Write block 2440 len 16384 data: 4 5 6 5
045/300 Write block 3547 len 8192 data: 3 3
046/300 Write block 2947 len 4096 data: 4
047/300 Read block 836 len 8192
048/300 Write block 4932 len 4096 data: 4
049/300 Read block 4137 len 28672
050/300 Flush
051/300 Write block 3183 len 16384 data: 2 2 2 2
052/300 Write block 38 len 8192 data: 3 4
053/300 Write block 4017 len 8192 data: 2 2
054/300 Write block 1409 len 28672 data: 3 3 3 3 3 2 2
055/300 Read block 3224 len 12288
056/300 Write block 1918 len 28672 data: 3 2 2 2 2 2 2
057/300 Flush
058/300 Read block 2291 len 28672
059/300 Read block 1821 len 4096
060/300 Write block 621 len 4096 data: 4
061/300 Write block 2663 len 24576 data: 4 3 3 3 3 3
062/300 Read block 4826 len 16384
063/300 Flush
064/300 Read block 1728 len 32768
065/300 Write block 3377 len 4096 data: 3
066/300 Write block 3373 len 8192 data: 5 4
067/300 Read block 4057 len 40960
068/300 Flush
069/300 Read block 3390 len 40960
070/300 Read block 1411 len 36864
071/300 Read block 1445 len 8192
072/300 Write block 1160 len 24576 data: 3 2 2 2 3 3
073/300 Read block 3280 len 12288
074/300 Flush
075/300 Write block 71 len 32768 data: 4 2 4 4 3 3 2 2
076/300 Write block 286 len 28672 data: 3 3 3 4 3 4 4
077/300 Read block 3572 len 16384
078/300 Read block 4440 len 36864
079/300 Write block 1668 len 4096 data: 2
080/300 Read block 455 len 32768
081/300 Write block 4020 len 20480 data: 2 2 3 3 3
082/300 Read block 2686 len 28672
083/300 Read block 4375 len 12288
084/300 Write block 1687 len 28672 data: 2 2 2 3 3 4 4
085/300 Write block 2256 len 32768 data: 5 5 3 2 2 3 3 3
086/300 Read block 4540 len 4096
087/300 Write block 446 len 16384 data: 3 4 4 3
088/300 Write block 1428 len 40960 data: 2 2 2 2 2 3 3 3 3 3
089/300 Flush
090/300 Write block 2825 len 40960 data: 3 3 4 4 4 6 3 4 4 4
091/300 Read block 3004 len 4096
092/300 Read block 10 len 32768
093/300 Read block 4640 len 32768
094/300 Write block 1202 len 28672 data: 5 4 4 4 5 5 5
095/300 Flush
096/300 Read block 3437 len 32768
097/300 Read block 4566 len 12288
098/300 Write block 962 len 40960 data: 2 2 2 2 3 3 3 3 3 3
099/300 Write block 2049 len 32768 data: 3 3 3 3 3 3 3 2
100/300 Write block 4623 len 12288 data: 5 5 4
101/300 Write block 2334 len 24576 data: 5 5 5 3 4 4
102/300 Write block 2925 len 16384 data: 2 2 2 2
103/300 Read block 2597 len 16384
104/300 Read block 540 len 12288
105/300 Read block 1356 len 20480
106/300 Write block 3797 len 28672 data: 4 4 3 3 3 2 2
107/300 Read block 4491 len 4096
108/300 Read block 2287 len 40960
109/300 Write block 3139 len 28672 data: 4 4 2 2 2 3 4
110/300 Read block 1641 len 20480
111/300 Write block 2304 len 16384 data: 7 6 5 4
112/300 Flush
113/300 Write block 3678 len 24576 data: 4 3 4 4 5 6
114/300 Write block 279 len 20480 data: 2 2 2 2 2
115/300 Read block 1322 len 24576
116/300 Read block 128 len 8192
117/300 Write block 4384 len 12288 data: 7 6 6
118/300 Write block 3593 len 20480 data: 6 4 3 3 2
119/300 Write block 1297 len 40960 data: 2 4 4 5 5 5 4 4 2 5
120/300 Write block 3567 len 12288 data: 2 2 2
121/300 Read block 1480 len 12288
122/300 Write block 3336 len 40960 data: 4 4 4 6 5 5 6 7 6 4
123/300 Write block 1312 len 40960 data: 3 3 3 2 2 2 2 2 2 2
124/300 Write block 1185 len 16384 data: 2 2 2 2
125/300 Read block 1370 len 8192
126/300 Write block 3176 len 40960 data: 4 4 3 2 2 2 2 3 3 3
127/300 Read block 641 len 12288
128/300 Write block 1111 len 8192 data: 6 6
129/300 Read block 906 len 20480
130/300 Flush
131/300 Read block 4353 len 8192
132/300 Read block 1234 len 16384
133/300 Write block 3806 len 8192 data: 3 4
134/300 Read block 4709 len 20480
135/300 Read block 4063 len 12288
136/300 Read block 650 len 4096
137/300 Write block 348 len 16384 data: 5 5 5 7
138/300 Write block 2751 len 4096 data: 2
139/300 Write block 1558 len 36864 data: 3 3 3 2 2 2 2 3 3
140/300 Read block 680 len 4096
141/300 Read block 2578 len 24576
142/300 Write block 489 len 28672 data: 2 2 2 2 2 2 2
143/300 Read block 4531 len 24576
144/300 Read block 3188 len 36864
145/300 Read block 2614 len 8192
146/300 Write block 3404 len 36864 data: 6 6 6 4 5 5 5 6 5
147/300 Read block 4689 len 32768
148/300 Write block 1194 len 20480 data: 4 3 4 4 3
149/300 Read block 1729 len 32768
150/300 Flush
151/300 Read block 962 len 24576
152/300 Read block 834 len 40960
153/300 Write block 492 len 16384 data: 3 3 3 3
154/300 Read block 4075 len 12288
155/300 Write block 58 len 28672 data: 4 4 4 5 5 4 5
156/300 Write block 488 len 28672 data: 2 3 3 3 4 4 4
157/300 Flush
158/300 Read block 3369 len 28672
159/300 Write block 958 len 40960 data: 3 2 2 2 3 3 3 3 4 4
160/300 Read block 2384 len 8192
161/300 Write block 4687 len 4096 data: 3
162/300 Read block 947 len 4096
163/300 Read block 1821 len 36864
164/300 Write block 3580 len 28672 data: 3 4 5 4 4 4 6
165/300 Read block 608 len 16384
166/300 Write block 4511 len 20480 data: 2 2 4 2 2
167/300 Flush
168/300 Read block 2070 len 36864
169/300 Read block 3726 len 36864
170/300 Write block 3697 len 4096 data: 2
171/300 Write block 1050 len 8192 data: 2 2
172/300 Flush
173/300 Flush
174/300 Read block 1222 len 4096
175/300 Write block 3467 len 4096 data: 2
176/300 Write block 368 len 28672 data: 2 2 2 2 3 4 4
177/300 Write block 2657 len 36864 data: 3 3 3 3 3 4 5 4 4
178/300 Flush
179/300 Read block 2205 len 16384
180/300 Read block 1492 len 12288
181/300 Read block 2659 len 40960
182/300 Write block 2650 len 12288 data: 5 5 5
183/300 Read block 1972 len 28672
184/300 Write block 1162 len 32768 data: 3 3 4 4 3 3 3 3
185/300 Write block 2338 len 12288 data: 5 5 4
186/300 Read block 3589 len 28672
187/300 Read block 3132 len 28672
188/300 Write block 3588 len 16384 data: 8 7 6 6
189/300 Read block 4903 len 8192
190/300 Read block 2704 len 8192
191/300 Write block 2426 len 24576 data: 2 2 2 3 4 4
192/300 Write block 2161 len 12288 data: 5 5 4
193/300 Read block 1242 len 24576
194/300 Read block 402 len 32768
195/300 Write block 3129 len 20480 data: 3 3 3 2 2
196/300 Read block 4387 len 40960
197/300 Flush
198/300 Write block 2569 len 36864 data: 4 4 6 6 7 8 7 6 5
199/300 Write block 1889 len 28672 data: 4 4 3 3 3 3 4
200/300 Read block 3657 len 36864
201/300 Write block 307 len 24576 data: 3 3 4 4 4 3
202/300 Write block 2525 len 20480 data: 3 4 4 4 4
203/300 Write block 922 len 20480 data: 4 4 4 3 3
204/300 Write block 4691 len 40960 data: 3 3 3 3 3 4 3 4 3 4
205/300 Write block 2625 len 16384 data: 4 4 5 4
206/300 Read block 4568 len 12288
207/300 Read block 2282 len 12288
208/300 Write block 2946 len 16384 data: 4 5 4 3
209/300 Write block 1611 len 4096 data: 5
210/300 Read block 1209 len 36864
211/300 Read block 4902 len 24576
212/300 Read block 1717 len 32768
213/300 Write block 1279 len 16384 data: 3 3 4 4
214/300 Read block 571 len 24576
215/300 Flush
216/300 Read block 435 len 32768
217/300 Write block 1042 len 12288 data: 4 4 4
218/300 Write block 1614 len 8192 data: 3 3
219/300 Read block 1609 len 28672
220/300 Write block 4234 len 12288 data: 2 2 2
221/300 Write block 3836 len 28672 data: 4 3 3 3 3 3 3
222/300 Write block 1482 len 40960 data: 3 2 3 2 2 2 2 2 2 2
223/300 Write block 2858 len 40960 data: 3 3 3 3 3 2 2 2 2 2
224/300 Write block 4461 len 28672 data: 4 4 3 3 3 4 3
225/300 Read block 3868 len 20480
226/300 Write block 2809 len 12288 data: 2 2 2
227/300 Write block 4827 len 20480 data: 3 3 3 2 3
228/300 Read block 83 len 12288
229/300 Read block 3871 len 32768
230/300 Write block 1625 len 16384 data: 2 2 2 3
231/300 Flush
232/300 Read block 993 len 36864
233/300 Read block 523 len 8192
234/300 Read block 1259 len 8192
235/300 Read block 3886 len 24576
236/300 Write block 148 len 4096 data: 3
237/300 Read block 988 len 12288
238/300 Read block 1042 len 8192
239/300 Read block 4473 len 24576
240/300 Write block 4705 len 32768 data: 3 3 3 4 4 4 4 5
241/300 Read block 2134 len 32768
242/300 Write block 330 len 12288 data: 4 6 7
243/300 Read block 1058 len 36864
244/300 Write block 2593 len 40960 data: 4 4 4 4 3 3 3 3 2 2
245/300 Read block 3450 len 8192
246/300 Write block 1131 len 12288 data: 5 5 4
247/300 Write block 2198 len 32768 data: 3 3 3 3 3 4 2 2
248/300 Write block 2350 len 36864 data: 4 4 4 5 5 5 5 4 4
249/300 Write block 1543 len 36864 data: 2 2 2 2 2 3 4 4 4
250/300 Write block 4221 len 20480 data: 4 4 3 3 2
251/300 Read block 4240 len 12288
252/300 Write block 4106 len 32768 data: 3 3 2 2 3 4 4 5
253/300 Read block 167 len 20480
254/300 Read block 1111 len 20480
255/300 Read block 4956 len 40960
256/300 Write block 4008 len 12288 data: 4 4 3
257/300 Read block 1011 len 24576
258/300 Write block 2456 len 40960 data: 2 2 4 4 4 4 4 5 6 6
259/300 Write block 4308 len 12288 data: 3 3 3
260/300 Write block 3766 len 32768 data: 2 2 3 3 4 4 4 3
261/300 Read block 4581 len 40960
262/300 Read block 3410 len 4096
263/300 Read block 1780 len 20480
264/300 Read block 886 len 8192
265/300 Write block 3251 len 24576 data: 4 4 3 3 3 4
266/300 Write block 1634 len 16384 data: 5 5 3 3
267/300 Write block 2114 len 36864 data: 3 3 2 2 2 2 3 3 3
268/300 Read block 1384 len 4096
269/300 Flush
270/300 Flush
271/300 Write block 4113 len 4096 data: 6
272/300 Write block 4903 len 4096 data: 5
273/300 Read block 3561 len 20480
274/300 Write block 4613 len 28672 data: 3 2 2 2 2 3 3
275/300 Write block 761 len 20480 data: 3 2 2 3 3
276/300 Read block 1734 len 32768
277/300 Read block 201 len 4096
278/300 Read block 3146 len 24576
279/300 Write block 4711 len 32768 data: 5 6 4 5 4 3 3 3
280/300 Read block 4327 len 32768
281/300 Write block 2046 len 28672 data: 2 2 2 4 4 4 4
282/300 Read block 4921 len 8192
283/300 Write block 1377 len 4096 data: 2
284/300 Read block 1060 len 16384
285/300 Write block 641 len 24576 data: 6 5 5 6 5 4
286/300 Write block 1616 len 16384 data: 3 3 4 4
287/300 Read block 4094 len 12288
288/300 Write block 1563 len 40960 data: 3 3 4 4 3 5 5 4 4 4
289/300 Flush
290/300 Flush
291/300 Flush
292/300 Write block 4763 len 16384 data: 3 4 3 2
293/300 Read block 2502 len 24576
294/300 Read block 4873 len 12288
295/300 Read block 2353 len 12288
296/300 Read block 4020 len 12288
297/300 Write block 4542 len 16384 data: 4 2 2 2
298/300 Flush
299/300 Read block 3220 len 36864
300/300 Read block 121 len 40960
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 13 21:41:47.849 INFO request completed, latency_us: 224, response_code: 204, uri: /start/cid/0, method: POST, req_id: a0e0e890-d791-49f5-a34b-f5f655766c39, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
3317 Acked 4316 Write 7 New Done Done true
3318 Acked 4317 Write 7 New Done Done true
3319 Acked 4318 Write 6 New Done Done true
3320 Acked 4319 Write 4 New Done Done true
3321 Acked 4320 Flush 0 New Done Done true
3322 Acked 4321 Read 4 New Done Done false
3323 Acked 4322 Write 6 New Done Done false
3324 Acked 4323 Write 7 New Done Done false
3325 Acked 4324 Read 3 New Done Done false
3326 Acked 4325 Read 3 New Done Done false
3327 Acked 4326 Read 5 New Done Done false
3328 Acked 4327 Write 6 New Done Done false
3329 Acked 4328 Read 7 New Done Done false
3330 Acked 4329 Write 2 New Done Done false
3331 Acked 4330 Read 9 New Done Done false
3332 Acked 4331 Write 2 New Done Done false
3333 Acked 4332 Write 6 New Done Done false
3334 Acked 4333 Read 2 New Done Done false
3335 Acked 4334 Write 5 New Done Done false
3336 Acked 4335 Write 5 New Done Done false
3337 Acked 4336 Write 6 New Done Done false
3338 Acked 4337 Flush 0 New Done Done false
3339 Acked 4338 Read 8 New Done Done false
3340 Acked 4339 Read 4 New Done Done false
3341 Acked 4340 Flush 0 New Done Done false
3342 Acked 4341 Read 9 New Done Done false
3343 Acked 4342 Read 9 New Done Done false
3344 Acked 4343 Write 4 New Done Done false
3345 Acked 4344 Flush 0 New Done Done false
3346 Acked 4345 Flush 0 New Done Done false
3347 Acked 4346 Write 10 New Done Done false
3348 Acked 4347 Read 4 New Done Done false
3349 Acked 4348 Write 5 New Done Done false
3350 Acked 4349 Read 5 New Done Done false
3351 Acked 4350 Flush 0 New Done Done false
3352 Acked 4351 Read 9 New Done Done false
3353 Acked 4352 Read 9 New Done Done false
3354 Acked 4353 Read 1 New Done Done false
3355 Acked 4354 Read 4 New Done Done false
3356 Acked 4355 Flush 0 New Done Done false
3357 Acked 4356 Flush 0 New Done Done false
3358 Acked 4357 Flush 0 New Done Done false
3359 Acked 4358 Read 7 New Done Done false
3360 Acked 4359 Write 4 New Done Done false
3361 Acked 4360 Write 2 New Done Done false
3362 Acked 4361 Write 1 New Done Done false
3363 Acked 4362 Read 2 New Done Done false
3364 Acked 4363 Write 1 New Done Done false
3365 Acked 4364 Read 7 New Done Done false
3366 Acked 4365 Flush 0 New Done Done false
3367 Acked 4366 Write 4 New Done Done false
3368 Acked 4367 Write 2 New Done Done false
3369 Acked 4368 Write 2 New Done Done false
3370 Acked 4369 Write 7 New Done Done false
3371 Acked 4370 Read 3 New Done Done false
3372 Acked 4371 Write 7 New Done Done false
3373 Acked 4372 Flush 0 New Done Done false
3374 Acked 4373 Read 7 New Done Done false
3375 Acked 4374 Read 1 New Done Done false
3376 Acked 4375 Write 1 New Done Done false
3377 Acked 4376 Write 6 New Done Done false
3378 Acked 4377 Read 4 New Done Done false
3379 Acked 4378 Flush 0 New Done Done false
3380 Acked 4379 Read 8 New Done Done false
3381 Acked 4380 Write 1 New Done Done false
3382 Acked 4381 Write 2 New Done Done false
3383 Acked 4382 Read 10 New Done Done false
3384 Acked 4383 Flush 0 New Done Done false
3385 Acked 4384 Read 10 New Done Done false
3386 Acked 4385 Read 9 New Done Done false
3387 Acked 4386 Read 2 New Done Done false
3388 Acked 4387 Write 6 New Done Done false
3389 Acked 4388 Read 3 New Done Done false
3390 Acked 4389 Flush 0 New Done Done false
3391 Acked 4390 Write 8 New Done Done false
3392 Acked 4391 Write 7 New Done Done false
3393 Acked 4392 Read 4 New Done Done false
3394 Acked 4393 Flush 0 New Done Done false
3395 Acked 4394 Read 9 New Done Done false
3396 Acked 4395 Write 1 New Done Done false
3397 Acked 4396 Read 8 New Done Done false
3398 Acked 4397 Write 5 New Done Done false
3399 Acked 4398 Read 7 New Done Done false
3400 Acked 4399 Read 3 New Done Done false
3401 Acked 4400 Write 7 New Done Done false
3402 Acked 4401 Write 8 New Done Done false
3403 Acked 4402 Read 1 New Done Done false
3404 Acked 4403 Write 4 New Done Done false
3405 Acked 4404 Write 10 New Done Done false
3406 Acked 4405 Flush 0 New Done Done false
3407 Acked 4406 Write 10 New Done Done false
3408 Acked 4407 Read 1 New Done Done false
3409 Acked 4408 Read 8 New Done Done false
3410 Acked 4409 Read 8 New Done Done false
3411 Acked 4410 Write 7 New Done Done false
3412 Acked 4411 Flush 0 New Done Done false
3413 Acked 4412 Read 8 New Done Done false
3414 Acked 4413 Read 3 New Done Done false
3415 Acked 4414 Write 10 New Done Done false
3416 Acked 4415 Write 8 New Done Done false
3417 Acked 4416 Write 3 New Done Done false
3418 Acked 4417 Write 6 New Done Done false
3419 Acked 4418 Write 4 New Done Done false
3420 Acked 4419 Read 4 New Done Done false
3421 Acked 4420 Read 3 New Done Done false
3422 Acked 4421 Read 5 New Done Done false
3423 Acked 4422 Write 7 New Done Done false
3424 Acked 4423 Read 1 New Done Done false
3425 Acked 4424 Read 10 New Done Done false
3426 Acked 4425 Write 7 New Done Done false
3427 Acked 4426 Read 5 New Done Done false
3428 Acked 4427 Write 4 New Done Done false
3429 Acked 4428 Flush 0 New Done Done false
3430 Acked 4429 Write 6 New Done Done false
3431 Acked 4430 Write 5 New Done Done false
3432 Acked 4431 Read 6 New Done Done false
3433 Acked 4432 Read 2 New Done Done false
3434 Acked 4433 Write 3 New Done Done false
3435 Acked 4434 Write 5 New Done Done false
3436 Acked 4435 Write 10 New Done Done false
3437 Acked 4436 Write 3 New Done Done false
3438 Acked 4437 Read 3 New Done Done false
3439 Acked 4438 Write 10 New Done Done false
3440 Acked 4439 Write 10 New Done Done false
3441 Acked 4440 Write 4 New Done Done false
3442 Acked 4441 Read 2 New Done Done false
3443 Acked 4442 Write 10 New Done Done false
3444 Acked 4443 Read 3 New Done Done false
3445 Acked 4444 Write 2 New Done Done false
3446 Acked 4445 Read 5 New Done Done false
3447 Acked 4446 Flush 0 New Done Done false
3448 Acked 4447 Read 2 New Done Done false
3449 Acked 4448 Read 4 New Done Done false
3450 Acked 4449 Write 2 New Done Done false
3451 Acked 4450 Read 5 New Done Done false
3452 Acked 4451 Read 3 New Done Done false
3453 Acked 4452 Read 1 New Done Done false
3454 Acked 4453 Write 4 New Done Done false
3455 Acked 4454 Write 1 New Done Done false
3456 Acked 4455 Write 9 New Done Done false
3457 Acked 4456 Read 1 New Done Done false
3458 Acked 4457 Read 6 New Done Done false
3459 Acked 4458 Write 7 New Done Done false
3460 Acked 4459 Read 6 New Done Done false
3461 Acked 4460 Read 9 New Done Done false
3462 Acked 4461 Read 2 New Done Done false
3463 Acked 4462 Write 9 New Done Done false
3464 Acked 4463 Read 8 New Done Done false
3465 Acked 4464 Write 5 New Done Done false
3466 Acked 4465 Read 8 New Done Done false
3467 Acked 4466 Flush 0 New Done Done false
3468 Acked 4467 Read 6 New Done Done false
3469 Acked 4468 Read 10 New Done Done false
3470 Acked 4469 Write 4 New Done Done false
3471 Acked 4470 Read 3 New Done Done false
3472 Acked 4471 Write 7 New Done Done false
3473 Acked 4472 Write 7 New Done Done false
3474 Acked 4473 Flush 0 New Done Done false
3475 Acked 4474 Read 7 New Done Done false
3476 Acked 4475 Write 10 New Done Done false
3477 Acked 4476 Read 2 New Done Done false
3478 Acked 4477 Write 1 New Done Done false
3479 Acked 4478 Read 1 New Done Done false
3480 Acked 4479 Read 9 New Done Done false
3481 Acked 4480 Write 7 New Done Done false
3482 Acked 4481 Read 4 New Done Done false
3483 Acked 4482 Write 5 New Done Done false
3484 Acked 4483 Flush 0 New Done Done false
3485 Acked 4484 Read 9 New Done Done false
3486 Acked 4485 Read 9 New Done Done false
3487 Acked 4486 Write 1 New Done Done false
3488 Acked 4487 Write 2 New Done Done false
3489 Acked 4488 Flush 0 New Done Done false
3490 Acked 4489 Flush 0 New Done Done false
3491 Acked 4490 Read 1 New Done Done false
3492 Acked 4491 Write 1 New Done Done false
3493 Acked 4492 Write 7 New Done Done false
3494 Acked 4493 Write 9 New Done Done false
3495 Acked 4494 Flush 0 New Done Done false
3496 Acked 4495 Read 4 New Done Done false
3497 Acked 4496 Read 3 New Done Done false
3498 Acked 4497 Read 10 New Done Done false
3499 Acked 4498 Write 3 New Done Done false
3500 Acked 4499 Read 7 New Done Done false
3501 Acked 4500 Write 8 New Done Done false
3502 Acked 4501 Write 3 New Done Done false
3503 Acked 4502 Read 7 New Done Done false
3504 Acked 4503 Read 7 New Done Done false
3505 Acked 4504 Write 4 New Done Done false
3506 Acked 4505 Read 2 New Done Done false
3507 Acked 4506 Read 2 New Done Done false
3508 Acked 4507 Write 6 New Done Done false
3509 Acked 4508 Write 3 New Done Done false
3510 Acked 4509 Read 6 New Done Done false
3511 Acked 4510 Read 8 New Done Done false
3512 Acked 4511 Write 5 New Done Done false
3513 Acked 4512 Read 10 New Done Done false
3514 Acked 4513 Flush 0 New Done Done false
3515 Acked 4514 Write 9 New Done Done false
3516 Acked 4515 Write 7 New Done Done false
3517 Acked 4516 Read 9 New Done Done false
3518 Acked 4517 Write 6 New Done Done false
3519 Acked 4518 Write 5 New Done Done false
3520 Acked 4519 Write 5 New Done Done false
3521 Acked 4520 Write 10 New Done Done false
3522 Acked 4521 Write 4 New Done Done false
3523 Acked 4522 Read 3 New Done Done false
3524 Acked 4523 Read 3 New Done Done false
3525 Acked 4524 Write 4 New Done Done false
3526 Acked 4525 Write 1 New Done Done false
3527 Acked 4526 Read 9 New Done Done false
3528 Acked 4527 Read 6 New Done Done false
3529 Acked 4528 Read 8 New Done Done false
3530 Acked 4529 Write 4 New Done Done false
3531 Acked 4530 Read 6 New Done Done false
3532 Acked 4531 Flush 0 New Done Done false
3533 Acked 4532 Read 8 New Done Done false
3534 Acked 4533 Write 3 New Done Done false
3535 Acked 4534 Write 2 New Done Done false
3536 Acked 4535 Read 7 New Done Done false
3537 Acked 4536 Write 3 New Done Done false
3538 Acked 4537 Flush 0 New Done Done false
3539 Acked 4538 Write 7 New Done Done false
3540 Acked 4539 Write 10 New Done Done false
3541 Acked 4540 Write 10 New Done Done false
3542 Acked 4541 Write 7 New Done Done false
3543 Acked 4542 Read 5 New Done Done false
3544 Acked 4543 Write 3 New Done Done false
3545 Acked 4544 Write 5 New Done Done false
3546 Acked 4545 Read 3 New Done Done false
3547 Acked 4546 Read 8 New Done Done false
3548 Acked 4547 Write 4 New Done Done false
3549 Acked 4548 Flush 0 New Done Done false
3550 Acked 4549 Read 9 New Done Done false
3551 Acked 4550 Read 2 New Done Done false
3552 Acked 4551 Read 2 New Done Done false
3553 Acked 4552 Read 6 New Done Done false
3554 Acked 4553 Write 1 New Done Done false
3555 Acked 4554 Read 3 New Done Done false
3556 Acked 4555 Read 2 New Done Done false
3557 Acked 4556 Read 6 New Done Done false
3558 Acked 4557 Write 8 New Done Done false
3559 Acked 4558 Read 8 New Done Done false
3560 Acked 4559 Write 3 New Done Done false
3561 Acked 4560 Read 9 New Done Done false
3562 Acked 4561 Write 10 New Done Done false
3563 Acked 4562 Read 2 New Done Done false
3564 Acked 4563 Write 3 New Done Done false
3565 Acked 4564 Write 8 New Done Done false
3566 Acked 4565 Write 9 New Done Done false
3567 Acked 4566 Write 9 New Done Done false
3568 Acked 4567 Write 5 New Done Done false
3569 Acked 4568 Read 3 New Done Done false
3570 Acked 4569 Write 8 New Done Done false
3571 Acked 4570 Read 5 New Done Done false
3572 Acked 4571 Read 5 New Done Done false
3573 Acked 4572 Read 10 New Done Done false
3574 Acked 4573 Write 3 New Done Done false
3575 Acked 4574 Read 6 New Done Done false
3576 Acked 4575 Write 10 New Done Done false
3577 Acked 4576 Write 3 New Done Done false
3578 Acked 4577 Write 8 New Done Done false
3579 Acked 4578 Read 10 New Done Done false
3580 Acked 4579 Read 1 New Done Done false
3581 Acked 4580 Read 5 New Done Done false
3582 Acked 4581 Read 2 New Done Done false
3583 Acked 4582 Write 6 New Done Done false
3584 Acked 4583 Write 4 New Done Done false
3585 Acked 4584 Write 9 New Done Done false
3586 Acked 4585 Read 1 New Done Done false
3587 Acked 4586 Flush 0 New Done Done false
3588 Acked 4587 Flush 0 New Done Done false
3589 Acked 4588 Write 1 New Done Done false
3590 Acked 4589 Write 1 New Done Done false
3591 Acked 4590 Read 5 New Done Done false
3592 Acked 4591 Write 7 New Done Done false
3593 Acked 4592 Write 5 New Done Done false
3594 Acked 4593 Read 8 New Done Done false
3595 Acked 4594 Read 1 New Done Done false
3596 Acked 4595 Read 6 New Done Done false
3597 Acked 4596 Write 8 New Done Done false
3598 Acked 4597 Read 8 New Done Done false
3599 Acked 4598 Write 7 New Done Done false
3600 Acked 4599 Read 2 New Done Done false
3601 Acked 4600 Write 1 New Done Done false
3602 Acked 4601 Read 4 New Done Done false
3603 Acked 4602 Write 6 New Done Done false
3604 Acked 4603 Write 4 New Done Done false
3605 Acked 4604 Read 3 New Done Done false
3606 Acked 4605 Write 10 New Done Done false
3607 Acked 4606 Flush 0 New Done Done false
3608 Acked 4607 Flush 0 New Done Done false
3609 Acked 4608 Flush 0 New Done Done false
3610 Acked 4609 Write 4 New Done Done false
3611 Acked 4610 Read 6 New Done Done false
3612 Acked 4611 Read 3 New Done Done false
3613 Acked 4612 Read 3 New Done Done false
3614 Acked 4613 Read 3 New Done Done false
3615 Acked 4614 Write 4 New Done Done false
3616 Acked 4615 Flush 0 New Done Done false
3617 Acked 4616 Read 9 New Done Done false
3618 Acked 4617 Read 10 New Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
New 302 0 0 302
Sent 0 0 0 0
Done 0 302 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 4315 4615 4615
Downstairs last five completed: 4315 4314 4313 4312 4311
Upstairs last five completed: 3618 3617 3616 3615 3614
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:47.936096158Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:47.936153375Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:47.936177084Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:47.936201554Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:47.936217483Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 4315","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:47.936234932Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:47.936343478Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:48.936743251Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:48.936837577Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:48.936880176Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:48.936918704Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:48.936948193Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 303 jobs since flush 4315","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:48.936980272Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:48.937188894Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1176)
[8810][0] reports Running
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:49.93779213Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:49.937870277Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:49.939256444Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:49.939281573Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 4315","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:49.939298462Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 4315","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:49.939467096Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:49.939495565Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:49.939520844Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:49.939539843Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:49.939554442Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 4618 4617 4616 4615 4614
Upstairs last five completed: 3619 3618 3617 3616 3615
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Feb 13 21:41:51.852 INFO request completed, latency_us: 358, response_code[8810] Got stop action so:false kr:false
: 204, uri: /stop/cid/0, method: POST, req_id: b4b0d19c-b915-4c8e-ae4e-9f9d5fa0d669, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Feb 13 21:41:51.853 INFO request completed, latency_us: 140, response_code: 200, uri: /state/cid/0, method: GET, req_id: 3b137331-a9a0-4e03-a89d-43ef2d95b526, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:51.853286316Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:51.853352043Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:51.853372482Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:51.853401611Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 4618","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:51.85343109Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:51.853457379Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:52.854584864Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:52.854648172Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:52.854670321Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:52.85469916Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:52.854727539Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 4618","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:52.854760737Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:52.854793996Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:53.856839987Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:53.857047609Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:53.857068198Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:53.857084608Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:53.857105377Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 4618","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:53.857131686Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:53.857156565Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:54.857672723Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:54.857732141Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:54.85775509Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:54.857776619Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:54.857792669Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 4618","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:54.857820698Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:54.857852027Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:41:55.855 INFO request completed, latency_us: 206, response_code: 200, uri: /state/cid/0, method: GET, req_id: d3f63cec-b089-4548-a1eb-f5d1597daa1b, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Read block 4281 len 8192
002/300 Read block 4800 len 20480
003/300 Read block 4661 len 32768
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:55.858914915Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:55.858985112Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:55.859065909Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:55.859106487Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:55.859123287Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 3 jobs since flush 4618","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:55.859138356Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:55.859154655Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
004/300 Flush
005/300 Write block 519 len 20480 data: 6 6 6 4 4
006/300 Read block 3587 len 28672
007/300 Write block 2048 len 36864 data: 3 5 5 5 5 4 4 4 3
008/300 Flush
009/300 Write block 3762 len 16384 data: 3 3 2 2
010/300 Write block 4813 len 24576 data: 3 3 3 3 3 3
011/300 Read block 4936 len 28672
012/300 Write block 2821 len 32768 data: 3 3 3 3 4 4 5 5
013/300 Read block 1731 len 36864
014/300 Read block 3779 len 28672
015/300 Write block 4585 len 8192 data: 2 2
016/300 Read block 3413 len 40960
017/300 Write block 3154 len 8192 data: 5 3
018/300 Read block 4167 len 32768
019/300 Read block 4696 len 16384
020/300 Write block 2205 len 36864 data: 3 2 3 3 3 4 5 4 3
021/300 Write block 4002 len 24576 data: 3 3 3 3 3 3
022/300 Write block 4419 len 20480 data: 3 4 4 4 3
023/300 Write block 3866 len 28672 data: 3 3 2 3 3 3 2
024/300 Read block 1600 len 20480
025/300 Flush
026/300 Write block 1429 len 20480 data: 3 3 3 3 4
027/300 Read block 4518 len 4096
028/300 Flush
029/300 Read block 4956 len 24576
030/300 Write block 3252 len 32768 data: 5 4 4 4 5 5 6 3
031/300 Write block 2074 len 8192 data: 4 4
032/300 Write block 1187 len 20480 data: 3 3 3 2 3
033/300 Flush
034/300 Write block 2512 len 28672 data: 5 5 5 5 4 4 4
035/300 Flush
036/300 Write block 3286 len 16384 data: 6 6 5 5
037/300 Write block 2980 len 24576 data: 3 4 5 4 4 6
038/300 Read block 1050 len 20480
039/300 Read block 4839 len 40960
040/300 Read block 571 len 8192
041/300 Read block 39 len 24576
042/300 Read block 4958 len 20480
043/300 Read block 3200 len 24576
044/300 Write block 3562 len 32768 data: 5 4 3 2 2 3 3 3
045/300 Write block 1208 len 16384 data: 6 6 5 4
046/300 Write block 391 len 24576 data: 4 3 3 3 4 4
047/300 Write block 1757 len 4096 data: 3
048/300 Flush
049/300 Write block 4908 len 12288 data: 3 3 3
050/300 Write block 4465 len 4096 data: 4
051/300 Write block 1149 len 8192 data: 2 2
052/300 Write block 1034 len 24576 data: 2 2 3 3 4 4
053/300 Write block 662 len 28672 data: 3 3 3 4 4 4 4
054/300 Read block 3342 len 28672
055/300 Write block 331 len 12288 data: 7 8 5
056/300 Write block 1494 len 12288 data: 3 4 5
057/300 Flush
058/300 Read block 1714 len 12288
059/300 Write block 1659 len 32768 data: 7 5 5 4 4 3 3 3
060/300 Read block 3602 len 28672
061/300 Read block 3544 len 16384
062/300 Write block 3750 len 4096 data: 3
063/300 Write block 4080 len 4096 data: 4
064/300 Read block 3650 len 36864
065/300 Flush
066/300 Read block 1673 len 16384
067/300 Read block 1102 len 8192
068/300 Write block 4191 len 36864 data: 3 3 3 2 3 3 4 4 4
069/300 Read block 503 len 4096
070/300 Read block 4597 len 36864
071/300 Write block 4927 len 12288 data: 3 3 3
072/300 Write block 1029 len 36864 data: 2 2 2 2 2 3 3 4 4
073/300 Read block 1583 len 4096
074/300 Read block 4007 len 8192
075/300 Read block 238 len 16384
076/300 Flush
077/300 Flush
078/300 Read block 1174 len 4096
079/300 Read block 635 len 4096
080/300 Read block 2951 len 40960
081/300 Write block 2383 len 16384 data: 2 2 2 2
082/300 Write block 92 len 32768 data: 5 4 4 4 4 3 3 2
083/300 Flush
084/300 Flush
085/300 Flush
086/300 Read block 4462 len 28672
087/300 Read block 2324 len 12288
088/300 Read block 2798 len 8192
089/300 Write block 4967 len 4096 data: 3
090/300 Read block 512 len 32768
091/300 Write block 4767 len 24576 data: 2 2 2 4 5 5
092/300 Read block 4754 len 16384
093/300 Write block 4545 len 4096 data: 3
094/300 Read block 2945 len 16384
095/300 Read block 942 len 8192
096/300 Read block 1274 len 32768
097/300 Read block 2113 len 36864
098/300 Write block 1509 len 28672 data: 2 2 2 2 2 2 2
099/300 Flush
100/300 Read block 2670 len 40960
101/300 Read block 547 len 36864
102/300 Read block 475 len 40960
103/300 Read block 1968 len 32768
104/300 Read block 2330 len 20480
105/300 Write block 1075 len 40960 data: 2 2 2 2 2 2 2 2 2 2
106/300 Read block 446 len 28672
107/300 Write block 2852 len 32768 data: 5 4 4 5 4 4 4 4
108/300 Read block 4181 len 36864
109/300 Write block 3852 len 20480 data: 5 5 5 5 5
110/300 Write block 1653 len 36864 data: 3 3 4 5 5 6 8 6 6
111/300 Read block 2974 len 12288
112/300 Read block 1879 len 12288
113/300 Write block 3650 len 36864 data: 4 3 3 3 3 4 4 3 3
114/300 Write block 3338 len 12288 data: 5 7 6
115/300 Read block 4650 len 16384
116/300 Read block 1821 len 20480
117/300 Read block 181 len 28672
118/300 Read block 1764 len 20480
119/300 Read block 818 len 8192
120/300 Write block 4494 len 40960 data: 2 2 2 2 2 2 4 3 3 4
121/300 Write block 4396 len 28672 data: 4 5 5 5 5 6 5
122/300 Flush
123/300 Write block 559 len 36864 data: 3 3 3 5 5 5 7 7 7
124/300 Write block 1923 len 24576 data: 3 3 3 3 3 3
125/300 Read block 1399 len 20480
126/300 Read block 4053 len 24576
127/300 Read block 80 len 24576
128/300 Write block 800 len 40960 data: 3 3 2 2 2 2 3 2 2 2
129/300 Write block 3852 len 28672 data: 6 6 6 6 6 5 6
130/300 Read block 951 len 12288
131/300 Write block 3362 len 32768 data: 4 4 3 3 3 4 5 5
132/300 Read block 335 len 40960
133/300 Read block 3034 len 8192
134/300 Read block 2077 len 16384
135/300 Flush
136/300 Write block 1916 len 36864 data: 3 3 4 3 3 3 3 4 4
137/300 Read block 3768 len 28672
138/300 Write block 1491 len 20480 data: 3 3 3 4 5
139/300 Read block 662 len 28672
140/300 Write block 427 len 28672 data: 2 4 4 3 2 2 2
141/300 Read block 3866 len 8192
142/300 Write block 1571 len 40960 data: 5 5 3 4 4 4 3 2 2 2
143/300 Flush
144/300 Read block 1218 len 12288
145/300 Read block 102 len 4096
146/300 Read block 1579 len 16384
147/300 Write block 829 len 8192 data: 4 2
148/300 Read block 3005 len 8192
149/300 Write block 3701 len 12288 data: 2 2 2
150/300 Flush
151/300 Read block 4116 len 20480
152/300 Flush
153/300 Read block 4214 len 28672
154/300 Write block 2437 len 28672 data: 4 4 4 5 6 7 6
155/300 Write block 4709 len 4096 data: 5
156/300 Write block 744 len 32768 data: 5 5 5 5 5 5 4 4
157/300 Read block 2901 len 4096
158/300 Read block 3997 len 20480
159/300 Write block 4915 len 20480 data: 2 3 3 3 3
160/300 Write block 4090 len 24576 data: 4 4 4 3 2 2
161/300 Read block 2224 len 4096
162/300 Read block 4473 len 16384
163/300 Write block 1869 len 36864 data: 2 2 2 2 2 2 2 2 2
164/300 Write block 4873 len 36864 data: 3 2 2 3 3 3 3 2 2
165/300 Read block 3446 len 12288
166/300 Write block 3390 len 24576 data: 3 2 2 2 2 2
167/300 Write block 1392 len 12288 data: 5 4 4
168/300 Read block 2608 len 36864
169/300 Write block 1431 len 12288 data: 4 4 5
170/300 Write block 2593 len 28672 data: 5 5 5 5 4 4 4
171/300 Read block 4606 len 40960
172/300 Read block 4228 len 32768
173/300 Read block 724 len 36864
174/300 Read block 1052 len 24576
175/300 Read block 2492 len 36864
176/300 Read block 1210 len 16384
177/300 Write block 3379 len 8192 data: 4 4
178/300 Read block 3342 len 4096
179/300 Write block 1634 len 24576 data: 6 6 4 4 4 4
180/300 Flush
181/300 Write block 3357 len 32768 data: 5 5 4 4 4 5 5 4
182/300 Write block 771 len 16384 data: 2 2 2 3
183/300 Write block 1802 len 8192 data: 4 5
184/300 Write block 1224 len 40960 data: 3 3 4 6 5 4 2 2 2 3
185/300 Write block 2824 len 20480 data: 4 5 5 6 6
186/300 Write block 3489 len 8192 data: 3 3
187/300 Write block 3495 len 36864 data: 2 2 2 3 3 3 3 2 3
188/300 Read block 940 len 24576
189/300 Read block 3042 len 16384
190/300 Write block 4203 len 16384 data: 3 2 2 2
191/300 Write block 1144 len 36864 data: 4 4 3 3 2 3 3 2 2
192/300 Read block 1434 len 4096
193/300 Read block 1509 len 24576
194/300 Read block 76 len 28672
195/300 Read block 2171 len 16384
196/300 Read block 4369 len 40960
197/300 Read block 3434 len 36864
198/300 Write block 4440 len 24576 data: 3 4 4 4 4 3
199/300 Write block 4551 len 20480 data: 2 4 4 4 4
200/300 Write block 4918 len 12288 data: 4 4 3
201/300 Read block 918 len 20480
202/300 Read block 2583 len 12288
203/300 Read block 3527 len 16384
204/300 Read block 2157 len 4096
205/300 Read block 519 len 20480
206/300 Flush
207/300 Flush
208/300 Write block 2200 len 28672 data: 4 4 4 5 3 4 3
209/300 Read block 3962 len 4096
210/300 Read block 625 len 20480
211/300 Write block 1807 len 4096 data: 4
212/300 Flush
213/300 Read block 4520 len 8192
214/300 Write block 2047 len 16384 data: 3 4 6 6
215/300 Read block 4907 len 24576
216/300 Flush
217/300 Flush
218/300 Read block 401 len 32768
219/300 Write block 3602 len 20480 data: 2 2 3 3 3
220/300 Read block 346 len 20480
221/300 Write block 3033 len 16384 data: 5 5 5 4
222/300 Write block 4292 len 40960 data: 5 5 4 4 4 4 5 6 6 5
223/300 Read block 513 len 8192
224/300 Read block 3912 len 20480
225/300 Read block 30 len 8192
226/300 Read block 2149 len 4096
227/300 Flush
228/300 Read block 1539 len 40960
229/300 Read block 1181 len 8192
230/300 Write block 4358 len 20480 data: 2 2 2 2 2
231/300 Write block 2837 len 16384 data: 6 6 5 5
232/300 Read block 3967 len 16384
233/300 Flush
234/300 Read block 1095 len 12288
235/300 Write block 3736 len 32768 data: 4 5 5 4 5 4 4 3
236/300 Read block 214 len 4096
237/300 Write block 1010 len 24576 data: 2 2 2 2 2 2
238/300 Read block 3285 len 36864
239/300 Write block 1820 len 4096 data: 6
240/300 Read block 1534 len 8192
241/300 Write block 4653 len 36864 data: 3 3 2 3 3 3 4 4 4
242/300 Write block 328 len 24576 data: 3 4 5 8 9 6
243/300 Write block 4535 len 36864 data: 5 5 4 6 5 4 4 5 3
244/300 Read block 713 len 32768
245/300 Write block 4714 len 20480 data: 6 5 4 4 4
246/300 Flush
247/300 Read block 1977 len 12288
248/300 Write block 2912 len 28672 data: 4 4 3 4 4 4 4
249/300 Read block 1419 len 28672
250/300 Write block 1184 len 36864 data: 2 3 3 4 4 4 3 4 4
251/300 Read block 4230 len 8192
252/300 Write block 1465 len 24576 data: 4 3 4 5 3 3
253/300 Read block 2205 len 36864
254/300 Read block 1161 len 12288
255/300 Flush
256/300 Read block 1565 len 36864
257/300 Write block 4779 len 4096 data: 3
258/300 Read block 1732 len 4096
259/300 Flush
260/300 Flush
261/300 Read block 4717 len 28672
262/300 Read block 1287 len 20480
263/300 Read block 2140 len 8192
264/300 Read block 1549 len 32768
265/300 Write block 4836 len 36864 data: 3 3 4 3 3 3 4 3 3
266/300 Write block 1323 len 28672 data: 3 3 2 2 2 3 4
267/300 Read block 3160 len 28672
268/300 Write block 2682 len 28672 data: 3 3 2 2 2 3 3
269/300 Write block 919 len 40960 data: 4 4 4 5 5 5 4 4 3 3
270/300 Read block 2811 len 36864
271/300 Read block 3778 len 32768
272/300 Read block 3418 len 12288
273/300 Read block 2251 len 4096
274/300 Write block 626 len 8192 data: 2 2
275/300 Read block 1160 len 40960
276/300 Read block 4294 len 20480
277/300 Write block 4336 len 36864 data: 3 3 3 3 3 3 2 2 2
278/300 Read block 96 len 16384
279/300 Read block 182 len 4096
280/300 Read block 58 len 32768
281/300 Write block 74 len 8192 data: 5 4
282/300 Write block 1904 len 12288 data: 2 2 2
283/300 Write block 4593 len 28672 data: 3 3 3 3 3 3 3
284/300 Read block 3924 len 40960
285/300 Flush
286/300 Read block 4007 len 12288
287/300 Read block 2524 len 28672
288/300 Write block 3197 len 24576 data: 8 7 5 4 4 3
289/300 Read block 4188 len 16384
290/300 Read block 831 len 12288
291/300 Read block 4946 len 36864
292/300 Write block 3434 len 40960 data: 5 5 5 4 4 3 2 2 2 3
293/300 Read block 851 len 36864
294/300 Read block 4279 len 8192
295/300 Write block 3940 len 20480 data: 3 3 2 2 3
296/300 Write block 2653 len 40960 data: 4 4 5 4 4 4 4 4 4 5
297/300 Read block 4179 len 32768
298/300 Write block 3054 len 8192 data: 3 3
299/300 Write block 269 len 16384 data: 5 5 6 5
300/300 Read block 1966 len 40960
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 13 21:41:56.695 INFO request completed, latency_us: 228, response_code: 204, uri: /start/cid/0, method: POST, req_id: e2b284be-ca4b-4cbc-9b5d-e7c9f1a76cf0, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
3620 Acked 4619 Read 2 New Done Done true
3621 Acked 4620 Read 5 New Done Done true
3622 Acked 4621 Read 8 New Done Done true
3623 Acked 4622 Flush 0 New Done Done false
3624 Acked 4623 Write 5 New Done Done false
3625 Acked 4624 Read 7 New Done Done false
3626 Acked 4625 Write 9 New Done Done false
3627 Acked 4626 Flush 0 New Done Done false
3628 Acked 4627 Write 4 New Done Done false
3629 Acked 4628 Write 6 New Done Done false
3630 Acked 4629 Read 7 New Done Done false
3631 Acked 4630 Write 8 New Done Done false
3632 Acked 4631 Read 9 New Done Done false
3633 Acked 4632 Read 7 New Done Done false
3634 Acked 4633 Write 2 New Done Done false
3635 Acked 4634 Read 10 New Done Done false
3636 Acked 4635 Write 2 New Done Done false
3637 Acked 4636 Read 8 New Done Done false
3638 Acked 4637 Read 4 New Done Done false
3639 Acked 4638 Write 9 New Done Done false
3640 Acked 4639 Write 6 New Done Done false
3641 Acked 4640 Write 5 New Done Done false
3642 Acked 4641 Write 7 New Done Done false
3643 Acked 4642 Read 5 New Done Done false
3644 Acked 4643 Flush 0 New Done Done false
3645 Acked 4644 Write 5 New Done Done false
3646 Acked 4645 Read 1 New Done Done false
3647 Acked 4646 Flush 0 New Done Done false
3648 Acked 4647 Read 6 New Done Done false
3649 Acked 4648 Write 8 New Done Done false
3650 Acked 4649 Write 2 New Done Done false
3651 Acked 4650 Write 5 New Done Done false
3652 Acked 4651 Flush 0 New Done Done false
3653 Acked 4652 Write 7 New Done Done false
3654 Acked 4653 Flush 0 New Done Done false
3655 Acked 4654 Write 4 New Done Done false
3656 Acked 4655 Write 6 New Done Done false
3657 Acked 4656 Read 5 New Done Done false
3658 Acked 4657 Read 10 New Done Done false
3659 Acked 4658 Read 2 New Done Done false
3660 Acked 4659 Read 6 New Done Done false
3661 Acked 4660 Read 5 New Done Done false
3662 Acked 4661 Read 6 New Done Done false
3663 Acked 4662 Write 8 New Done Done false
3664 Acked 4663 Write 4 New Done Done false
3665 Acked 4664 Write 6 New Done Done false
3666 Acked 4665 Write 1 New Done Done false
3667 Acked 4666 Flush 0 New Done Done false
3668 Acked 4667 Write 3 New Done Done false
3669 Acked 4668 Write 1 New Done Done false
3670 Acked 4669 Write 2 New Done Done false
3671 Acked 4670 Write 6 New Done Done false
3672 Acked 4671 Write 7 New Done Done false
3673 Acked 4672 Read 7 New Done Done false
3674 Acked 4673 Write 3 New Done Done false
3675 Acked 4674 Write 3 New Done Done false
3676 Acked 4675 Flush 0 New Done Done false
3677 Acked 4676 Read 3 New Done Done false
3678 Acked 4677 Write 8 New Done Done false
3679 Acked 4678 Read 7 New Done Done false
3680 Acked 4679 Read 4 New Done Done false
3681 Acked 4680 Write 1 New Done Done false
3682 Acked 4681 Write 1 New Done Done false
3683 Acked 4682 Read 9 New Done Done false
3684 Acked 4683 Flush 0 New Done Done false
3685 Acked 4684 Read 4 New Done Done false
3686 Acked 4685 Read 2 New Done Done false
3687 Acked 4686 Write 9 New Done Done false
3688 Acked 4687 Read 1 New Done Done false
3689 Acked 4688 Read 9 New Done Done false
3690 Acked 4689 Write 3 New Done Done false
3691 Acked 4690 Write 9 New Done Done false
3692 Acked 4691 Read 1 New Done Done false
3693 Acked 4692 Read 2 New Done Done false
3694 Acked 4693 Read 4 New Done Done false
3695 Acked 4694 Flush 0 New Done Done false
3696 Acked 4695 Flush 0 New Done Done false
3697 Acked 4696 Read 1 New Done Done false
3698 Acked 4697 Read 1 New Done Done false
3699 Acked 4698 Read 10 New Done Done false
3700 Acked 4699 Write 4 New Done Done false
3701 Acked 4700 Write 8 New Done Done false
3702 Acked 4701 Flush 0 New Done Done false
3703 Acked 4702 Flush 0 New Done Done false
3704 Acked 4703 Flush 0 New Done Done false
3705 Acked 4704 Read 7 New Done Done false
3706 Acked 4705 Read 3 New Done Done false
3707 Acked 4706 Read 2 New Done Done false
3708 Acked 4707 Write 1 New Done Done false
3709 Acked 4708 Read 8 New Done Done false
3710 Acked 4709 Write 6 New Done Done false
3711 Acked 4710 Read 4 New Done Done false
3712 Acked 4711 Write 1 New Done Done false
3713 Acked 4712 Read 4 New Done Done false
3714 Acked 4713 Read 2 New Done Done false
3715 Acked 4714 Read 8 New Done Done false
3716 Acked 4715 Read 9 New Done Done false
3717 Acked 4716 Write 7 New Done Done false
3718 Acked 4717 Flush 0 New Done Done false
3719 Acked 4718 Read 10 New Done Done false
3720 Acked 4719 Read 9 New Done Done false
3721 Acked 4720 Read 10 New Done Done false
3722 Acked 4721 Read 8 New Done Done false
3723 Acked 4722 Read 5 New Done Done false
3724 Acked 4723 Write 10 New Done Done false
3725 Acked 4724 Read 7 New Done Done false
3726 Acked 4725 Write 8 New Done Done false
3727 Acked 4726 Read 9 New Done Done false
3728 Acked 4727 Write 5 New Done Done false
3729 Acked 4728 Write 9 New Done Done false
3730 Acked 4729 Read 3 New Done Done false
3731 Acked 4730 Read 3 New Done Done false
3732 Acked 4731 Write 9 New Done Done false
3733 Acked 4732 Write 3 New Done Done false
3734 Acked 4733 Read 4 New Done Done false
3735 Acked 4734 Read 5 New Done Done false
3736 Acked 4735 Read 7 New Done Done false
3737 Acked 4736 Read 5 New Done Done false
3738 Acked 4737 Read 2 New Done Done false
3739 Acked 4738 Write 10 New Done Done false
3740 Acked 4739 Write 7 New Done Done false
3741 Acked 4740 Flush 0 New Done Done false
3742 Acked 4741 Write 9 New Done Done false
3743 Acked 4742 Write 6 New Done Done false
3744 Acked 4743 Read 5 New Done Done false
3745 Acked 4744 Read 6 New Done Done false
3746 Acked 4745 Read 6 New Done Done false
3747 Acked 4746 Write 10 New Done Done false
3748 Acked 4747 Write 7 New Done Done false
3749 Acked 4748 Read 3 New Done Done false
3750 Acked 4749 Write 8 New Done Done false
3751 Acked 4750 Read 10 New Done Done false
3752 Acked 4751 Read 2 New Done Done false
3753 Acked 4752 Read 4 New Done Done false
3754 Acked 4753 Flush 0 New Done Done false
3755 Acked 4754 Write 9 New Done Done false
3756 Acked 4755 Read 7 New Done Done false
3757 Acked 4756 Write 5 New Done Done false
3758 Acked 4757 Read 7 New Done Done false
3759 Acked 4758 Write 7 New Done Done false
3760 Acked 4759 Read 2 New Done Done false
3761 Acked 4760 Write 10 New Done Done false
3762 Acked 4761 Flush 0 New Done Done false
3763 Acked 4762 Read 3 New Done Done false
3764 Acked 4763 Read 1 New Done Done false
3765 Acked 4764 Read 4 New Done Done false
3766 Acked 4765 Write 2 New Done Done false
3767 Acked 4766 Read 2 New Done Done false
3768 Acked 4767 Write 3 New Done Done false
3769 Acked 4768 Flush 0 New Done Done false
3770 Acked 4769 Read 5 New Done Done false
3771 Acked 4770 Flush 0 New Done Done false
3772 Acked 4771 Read 7 New Done Done false
3773 Acked 4772 Write 7 New Done Done false
3774 Acked 4773 Write 1 New Done Done false
3775 Acked 4774 Write 8 New Done Done false
3776 Acked 4775 Read 1 New Done Done false
3777 Acked 4776 Read 5 New Done Done false
3778 Acked 4777 Write 5 New Done Done false
3779 Acked 4778 Write 6 New Done Done false
3780 Acked 4779 Read 1 New Done Done false
3781 Acked 4780 Read 4 New Done Done false
3782 Acked 4781 Write 9 New Done Done false
3783 Acked 4782 Write 9 New Done Done false
3784 Acked 4783 Read 3 New Done Done false
3785 Acked 4784 Write 6 New Done Done false
3786 Acked 4785 Write 3 New Done Done false
3787 Acked 4786 Read 9 New Done Done false
3788 Acked 4787 Write 3 New Done Done false
3789 Acked 4788 Write 7 New Done Done false
3790 Acked 4789 Read 10 New Done Done false
3791 Acked 4790 Read 8 New Done Done false
3792 Acked 4791 Read 9 New Done Done false
3793 Acked 4792 Read 6 New Done Done false
3794 Acked 4793 Read 9 New Done Done false
3795 Acked 4794 Read 4 New Done Done false
3796 Acked 4795 Write 2 New Done Done false
3797 Acked 4796 Read 1 New Done Done false
3798 Acked 4797 Write 6 New Done Done false
3799 Acked 4798 Flush 0 New Done Done false
3800 Acked 4799 Write 8 New Done Done false
3801 Acked 4800 Write 4 New Done Done false
3802 Acked 4801 Write 2 New Done Done false
3803 Acked 4802 Write 10 New Done Done false
3804 Acked 4803 Write 5 New Done Done false
3805 Acked 4804 Write 2 New Done Done false
3806 Acked 4805 Write 9 New Done Done false
3807 Acked 4806 Read 6 New Done Done false
3808 Acked 4807 Read 4 New Done Done false
3809 Acked 4808 Write 4 New Done Done false
3810 Acked 4809 Write 9 New Done Done false
3811 Acked 4810 Read 1 New Done Done false
3812 Acked 4811 Read 6 New Done Done false
3813 Acked 4812 Read 7 New Done Done false
3814 Acked 4813 Read 4 New Done Done false
3815 Acked 4814 Read 10 New Done Done false
3816 Acked 4815 Read 9 New Done Done false
3817 Acked 4816 Write 6 New Done Done false
3818 Acked 4817 Write 5 New Done Done false
3819 Acked 4818 Write 3 New Done Done false
3820 Acked 4819 Read 5 New Done Done false
3821 Acked 4820 Read 3 New Done Done false
3822 Acked 4821 Read 4 New Done Done false
3823 Acked 4822 Read 1 New Done Done false
3824 Acked 4823 Read 5 New Done Done false
3825 Acked 4824 Flush 0 New Done Done false
3826 Acked 4825 Flush 0 New Done Done false
3827 Acked 4826 Write 7 New Done Done false
3828 Acked 4827 Read 1 New Done Done false
3829 Acked 4828 Read 5 New Done Done false
3830 Acked 4829 Write 1 New Done Done false
3831 Acked 4830 Flush 0 New Done Done false
3832 Acked 4831 Read 2 New Done Done false
3833 Acked 4832 Write 4 New Done Done false
3834 Acked 4833 Read 6 New Done Done false
3835 Acked 4834 Flush 0 New Done Done false
3836 Acked 4835 Flush 0 New Done Done false
3837 Acked 4836 Read 8 New Done Done false
3838 Acked 4837 Write 5 New Done Done false
3839 Acked 4838 Read 5 New Done Done false
3840 Acked 4839 Write 4 New Done Done false
3841 Acked 4840 Write 10 New Done Done false
3842 Acked 4841 Read 2 New Done Done false
3843 Acked 4842 Read 5 New Done Done false
3844 Acked 4843 Read 2 New Done Done false
3845 Acked 4844 Read 1 New Done Done false
3846 Acked 4845 Flush 0 New Done Done false
3847 Acked 4846 Read 10 New Done Done false
3848 Acked 4847 Read 2 New Done Done false
3849 Acked 4848 Write 5 New Done Done false
3850 Acked 4849 Write 4 New Done Done false
3851 Acked 4850 Read 4 New Done Done false
3852 Acked 4851 Flush 0 New Done Done false
3853 Acked 4852 Read 3 New Done Done false
3854 Acked 4853 Write 8 New Done Done false
3855 Acked 4854 Read 1 New Done Done false
3856 Acked 4855 Write 6 New Done Done false
3857 Acked 4856 Read 9 New Done Done false
3858 Acked 4857 Write 1 New Done Done false
3859 Acked 4858 Read 2 New Done Done false
3860 Acked 4859 Write 9 New Done Done false
3861 Acked 4860 Write 6 New Done Done false
3862 Acked 4861 Write 9 New Done Done false
3863 Acked 4862 Read 8 New Done Done false
3864 Acked 4863 Write 5 New Done Done false
3865 Acked 4864 Flush 0 New Done Done false
3866 Acked 4865 Read 3 New Done Done false
3867 Acked 4866 Write 7 New Done Done false
3868 Acked 4867 Read 7 New Done Done false
3869 Acked 4868 Write 9 New Done Done false
3870 Acked 4869 Read 2 New Done Done false
3871 Acked 4870 Write 6 New Done Done false
3872 Acked 4871 Read 9 New Done Done false
3873 Acked 4872 Read 3 New Done Done false
3874 Acked 4873 Flush 0 New Done Done false
3875 Acked 4874 Read 9 New Done Done false
3876 Acked 4875 Write 1 New Done Done false
3877 Acked 4876 Read 1 New Done Done false
3878 Acked 4877 Flush 0 New Done Done false
3879 Acked 4878 Flush 0 New Done Done false
3880 Acked 4879 Read 7 New Done Done false
3881 Acked 4880 Read 5 New Done Done false
3882 Acked 4881 Read 2 New Done Done false
3883 Acked 4882 Read 8 New Done Done false
3884 Acked 4883 Write 9 New Done Done false
3885 Acked 4884 Write 7 New Done Done false
3886 Acked 4885 Read 7 New Done Done false
3887 Acked 4886 Write 7 New Done Done false
3888 Acked 4887 Write 10 New Done Done false
3889 Acked 4888 Read 9 New Done Done false
3890 Acked 4889 Read 8 New Done Done false
3891 Acked 4890 Read 3 New Done Done false
3892 Acked 4891 Read 1 New Done Done false
3893 Acked 4892 Write 2 New Done Done false
3894 Acked 4893 Read 10 New Done Done false
3895 Acked 4894 Read 5 New Done Done false
3896 Acked 4895 Write 9 New Done Done false
3897 Acked 4896 Read 4 New Done Done false
3898 Acked 4897 Read 1 New Done Done false
3899 Acked 4898 Read 8 New Done Done false
3900 Acked 4899 Write 2 New Done Done false
3901 Acked 4900 Write 3 New Done Done false
3902 Acked 4901 Write 7 New Done Done false
3903 Acked 4902 Read 10 New Done Done false
3904 Acked 4903 Flush 0 New Done Done false
3905 Acked 4904 Read 3 New Done Done false
3906 Acked 4905 Read 7 New Done Done false
3907 Acked 4906 Write 6 New Done Done false
3908 Acked 4907 Read 4 New Done Done false
3909 Acked 4908 Read 3 New Done Done false
3910 Acked 4909 Read 9 New Done Done false
3911 Acked 4910 Write 10 New Done Done false
3912 Acked 4911 Read 9 New Done Done false
3913 Acked 4912 Read 2 New Done Done false
3914 Acked 4913 Write 5 New Done Done false
3915 Acked 4914 Write 10 New Done Done false
3916 Acked 4915 Read 8 New Done Done false
3917 Acked 4916 Write 2 New Done Done false
3918 Acked 4917 Write 4 New Done Done false
3919 Acked 4918 Read 10 New Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
New 300 0 0 300
Sent 0 0 0 0
Done 0 300 300 600
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 4618 4903 4903
Downstairs last five completed: 4618 4617 4616 4615 4614
Upstairs last five completed: 3919 3918 3917 3916 3915
CLIENT: Up:0 ds:300 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:56.860139466Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:56.860198884Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:56.860222163Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:56.860242642Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:56.860258481Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 4618","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:56.860275671Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:56.860393706Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:57.862462905Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:57.862560032Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:57.8626023Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:41:57.862649368Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:57.862702266Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 4618","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:57.862759454Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:57.862969216Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1177)
[8810][0] reports Running
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:58.863541992Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:58.863657618Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:58.8649116Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:58.864938359Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 4618","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:58.864954538Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 4618","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:58.865115172Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:58.865147081Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:58.86517265Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:58.865196099Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:41:58.865216788Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 4919 4918 4917 4916 4915
Upstairs last five completed: 3920 3919 3918 3917 3916
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(2)
stop 2
Feb 13 21:42:00.699 INFO request completed, latency_us: 346, response_code: 204, uri: [8830] Got stop action so:false kr:false
/stop/cid/2, method: POST, req_id: d01ff782-ae8a-4a04-bda1-90e84cb1bf33, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:00.699810355Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:00.699880623Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:42:00.699 INFO request completed, latency_us: 142, response_code: 200, uri: /state/cid/2{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:00.699910801Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
, method: GET, req_id: 1b667188-8329-4ea9-b50b-d2614ad68fc6, remote_addr: 127.0.0.1:52124, local_addr: 127.0{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:00.699937991Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
.0.1:9998
{"msg":"[2] client re-new 0 jobs since flush 4919","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:00.699968279Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:00.699995998Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:01.700488578Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:01.700549805Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:01.700588644Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:01.700613063Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:01.700629852Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 4919","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:01.700647672Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:01.700665041Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:02.701671341Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:02.701735429Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:02.701759048Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:02.701779797Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:02.701795557Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 4919","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:02.701813026Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:02.701831155Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:03.703356155Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:03.703424122Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:03.703447131Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:03.703467501Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:03.70348304Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 4919","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:03.703500759Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:03.703527878Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:42:04.701 INFO request completed, latency_us: 211, response_code: 200, uri: /state/cid/2, method: GET, req_id: 52245db7-75fe-4435-b9ae-59e24a5cc3c7, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Write block 700 len 12288 data: 4 5 4
002/300 Write block 3192 len 40960 data: 6 6 7 6 8 9 8 6 5 5
003/300 Write block 1074 len 28672 data: 3 3 3 3 3 3 3
004/300 Read block 2022 len 40960
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:04.704368145Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:04.704468051Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:04.70450459Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:04.704536679Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:04.704557658Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 4 jobs since flush 4919","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:04.704573037Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:04.704595996Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
005/300 Read block 4198 len 40960
006/300 Write block 4334 len 8192 data: 4 4
007/300 Write block 749 len 20480 data: 6 5 5 4 4
008/300 Read block 4928 len 36864
009/300 Read block 2135 len 28672
010/300 Write block 2397 len 24576 data: 4 2 2 2 2 3
011/300 Read block 1911 len 40960
012/300 Write block 6 len 8192 data: 5 5
013/300 Write block 1054 len 4096 data: 2
014/300 Write block 985 len 12288 data: 3 5 5
015/300 Write block 4681 len 8192 data: 4 3
016/300 Read block 2284 len 32768
017/300 Flush
018/300 Flush
019/300 Read block 3997 len 24576
020/300 Write block 3826 len 20480 data: 4 3 4 4 4
021/300 Read block 3596 len 24576
022/300 Flush
023/300 Write block 3574 len 36864 data: 4 4 3 4 4 4 4 5 6
024/300 Write block 4651 len 4096 data: 6
025/300 Read block 3435 len 32768
026/300 Read block 2085 len 32768
027/300 Write block 194 len 36864 data: 4 4 3 3 2 2 2 2 2
028/300 Read block 4896 len 20480
029/300 Write block 4338 len 32768 data: 4 4 4 4 3 3 3 2
030/300 Write block 2171 len 36864 data: 4 3 2 2 2 2 2 3 4
031/300 Write block 59 len 28672 data: 5 5 6 6 5 6 5
032/300 Read block 41 len 8192
033/300 Read block 2850 len 36864
034/300 Read block 266 len 4096
035/300 Write block 1180 len 40960 data: 2 2 2 2 3 4 4 5 5 5
036/300 Write block 2730 len 16384 data: 2 2 3 3
037/300 Read block 3326 len 36864
038/300 Write block 2671 len 36864 data: 2 2 2 2 3 3 3 4 4
039/300 Read block 4424 len 32768
040/300 Write block 2482 len 20480 data: 2 2 3 4 5
041/300 Read block 4481 len 24576
042/300 Read block 3005 len 28672
043/300 Write block 863 len 32768 data: 6 4 6 6 6 4 4 4
044/300 Flush
045/300 Read block 1738 len 12288
046/300 Write block 3198 len 32768 data: 9 7 6 6 4 3 3 3
047/300 Read block 2259 len 4096
048/300 Read block 1551 len 28672
049/300 Write block 4877 len 24576 data: 4 4 4 3 3 2
050/300 Read block 3764 len 8192
051/300 Read block 3911 len 28672
052/300 Flush
053/300 Write block 2807 len 12288 data: 3 3 3
054/300 Read block 4260 len 24576
055/300 Write block 2178 len 32768 data: 4 5 4 4 4 4 4 3
056/300 Write block 1592 len 16384 data: 3 3 5 5
057/300 Write block 491 len 20480 data: 4 5 5 5 4
058/300 Read block 3826 len 4096
059/300 Write block 3293 len 4096 data: 2
060/300 Read block 1931 len 12288
061/300 Read block 44 len 36864
062/300 Write block 3739 len 16384 data: 5 6 5 5
063/300 Read block 3127 len 32768
064/300 Read block 485 len 8192
065/300 Write block 1151 len 32768 data: 3 3 2 3 3 3 3 4
066/300 Read block 3950 len 12288
067/300 Write block 125 len 4096 data: 5
068/300 Read block 1287 len 4096
069/300 Write block 1934 len 40960 data: 3 4 4 5 4 4 4 3 3 4
070/300 Flush
071/300 Read block 1067 len 40960
072/300 Read block 2785 len 28672
073/300 Read block 4402 len 40960
074/300 Write block 1274 len 12288 data: 2 2 2
075/300 Read block 2564 len 24576
076/300 Flush
077/300 Read block 4680 len 20480
078/300 Read block 2912 len 20480
079/300 Read block 1125 len 4096
080/300 Read block 2520 len 12288
081/300 Read block 3859 len 16384
082/300 Write block 238 len 8192 data: 5 5
083/300 Read block 4379 len 4096
084/300 Read block 1891 len 36864
085/300 Flush
086/300 Flush
087/300 Write block 4589 len 40960 data: 2 2 3 3 4 4 4 4 4 4
088/300 Write block 3554 len 40960 data: 4 4 5 8 6 6 7 6 6 5
089/300 Write block 3400 len 28672 data: 4 4 5 5 7 7 7
090/300 Read block 2087 len 28672
091/300 Write block 1665 len 4096 data: 4
092/300 Read block 1897 len 8192
093/300 Read block 3685 len 40960
094/300 Write block 3894 len 24576 data: 3 5 5 4 4 4
095/300 Write block 2402 len 28672 data: 4 4 3 3 3 4 4
096/300 Read block 552 len 16384
097/300 Read block 339 len 20480
098/300 Write block 3697 len 20480 data: 3 2 2 2 3
099/300 Read block 1406 len 32768
100/300 Write block 1256 len 40960 data: 3 2 2 2 3 4 4 5 3 3
101/300 Read block 4876 len 16384
102/300 Read block 3296 len 8192
103/300 Write block 4 len 32768 data: 5 4 6 6 3 2 2 2
104/300 Read block 4195 len 32768
105/300 Read block 4127 len 12288
106/300 Read block 172 len 28672
107/300 Read block 4205 len 8192
108/300 Write block 1522 len 36864 data: 4 3 2 2 2 2 2 2 2
109/300 Flush
110/300 Read block 1622 len 24576
111/300 Write block 3039 len 32768 data: 3 2 3 4 4 4 3 3
112/300 Write block 2642 len 28672 data: 2 2 3 3 3 3 3
113/300 Write block 3053 len 8192 data: 3 4
114/300 Read block 255 len 40960
115/300 Read block 2099 len 8192
116/300 Read block 1344 len 32768
117/300 Write block 172 len 12288 data: 5 4 3
118/300 Read block 81 len 28672
119/300 Read block 1500 len 4096
120/300 Write block 3610 len 28672 data: 4 5 5 5 5 5 5
121/300 Read block 4332 len 36864
122/300 Write block 2550 len 32768 data: 3 4 3 3 3 3 3 2
123/300 Write block 3675 len 28672 data: 5 5 4 5 4 5 5
124/300 Write block 3079 len 16384 data: 4 4 4 4
125/300 Read block 3841 len 20480
126/300 Flush
127/300 Read block 2089 len 16384
128/300 Write block 2749 len 12288 data: 3 2 3
129/300 Read block 186 len 28672
130/300 Write block 3675 len 40960 data: 6 6 5 6 5 6 6 6 7 5
131/300 Flush
132/300 Write block 4547 len 36864 data: 3 3 3 3 3 5 5 5 5
133/300 Read block 1079 len 12288
134/300 Read block 2660 len 4096
135/300 Write block 4529 len 40960 data: 2 3 4 4 4 4 6 6 5 7
136/300 Read block 3547 len 4096
137/300 Read block 4316 len 12288
138/300 Flush
139/300 Read block 3489 len 24576
140/300 Write block 3688 len 20480 data: 5 6 2 2 3
141/300 Flush
142/300 Write block 1570 len 20480 data: 5 6 6 4 5
143/300 Write block 3913 len 20480 data: 3 3 4 3 4
144/300 Write block 4844 len 8192 data: 4 3
145/300 Read block 4480 len 4096
146/300 Write block 4631 len 28672 data: 2 2 3 3 3 4 4
147/300 Flush
148/300 Write block 1384 len 20480 data: 3 4 4 4 4
149/300 Read block 2939 len 20480
150/300 Read block 2559 len 24576
151/300 Write block 1318 len 8192 data: 3 3
152/300 Read block 4467 len 8192
153/300 Read block 3730 len 20480
154/300 Flush
155/300 Read block 3571 len 36864
156/300 Write block 1189 len 24576 data: 6 4 5 5 4 5
157/300 Flush
158/300 Write block 3385 len 4096 data: 5
159/300 Read block 3558 len 32768
160/300 Read block 2212 len 36864
161/300 Write block 4063 len 40960 data: 3 3 3 3 3 3 3 3 2 2
162/300 Write block 3517 len 20480 data: 5 4 4 3 3
163/300 Write block 3407 len 8192 data: 5 6
164/300 Read block 4337 len 20480
165/300 Write block 1179 len 8192 data: 2 3
166/300 Read block 3387 len 36864
167/300 Write block 440 len 24576 data: 2 3 3 3 3 3
168/300 Write block 1226 len 20480 data: 5 7 6 5 3
169/300 Read block 4655 len 24576
170/300 Read block 4968 len 12288
171/300 Read block 103 len 36864
172/300 Write block 2632 len 4096 data: 3
173/300 Read block 573 len 8192
174/300 Read block 3021 len 16384
175/300 Read block 377 len 12288
176/300 Read block 2795 len 28672
177/300 Write block 1245 len 4096 data: 3
178/300 Read block 751 len 16384
179/300 Read block 1217 len 20480
180/300 Write block 842 len 40960 data: 4 4 4 4 5 5 5 3 3 2
181/300 Write block 4974 len 24576 data: 3 3 3 3 3 2
182/300 Flush
183/300 Write block 1991 len 8192 data: 6 5
184/300 Write block 1464 len 20480 data: 4 5 4 5 6
185/300 Write block 1256 len 20480 data: 4 3 3 3 4
186/300 Write block 1335 len 4096 data: 3
187/300 Write block 2882 len 8192 data: 3 4
188/300 Write block 1052 len 16384 data: 2 3 3 2
189/300 Read block 1867 len 8192
190/300 Flush
191/300 Flush
192/300 Flush
193/300 Write block 4323 len 12288 data: 8 7 7
194/300 Read block 415 len 12288
195/300 Read block 2654 len 28672
196/300 Read block 3089 len 4096
197/300 Flush
198/300 Write block 3218 len 24576 data: 3 5 6 8 8 9
199/300 Read block 4700 len 8192
200/300 Write block 620 len 32768 data: 3 5 4 4 3 3 3 3
201/300 Read block 4429 len 32768
202/300 Write block 1542 len 28672 data: 2 3 3 3 3 3 4
203/300 Read block 4245 len 32768
204/300 Flush
205/300 Write block 4608 len 32768 data: 3 3 3 3 3 4 3 3
206/300 Write block 2904 len 24576 data: 2 2 2 2 3 3
207/300 Flush
208/300 Read block 797 len 24576
209/300 Write block 935 len 16384 data: 2 2 2 2
210/300 Write block 792 len 36864 data: 3 3 3 3 3 4 4 4 4
211/300 Flush
212/300 Write block 4897 len 8192 data: 4 5
213/300 Read block 3326 len 32768
214/300 Write block 729 len 28672 data: 3 4 3 5 5 5 4
215/300 Read block 1895 len 20480
216/300 Write block 2302 len 32768 data: 7 7 8 7 6 5 4 4
217/300 Write block 4491 len 8192 data: 4 3
218/300 Write block 4099 len 32768 data: 3 3 2 2 2 2 2 4
219/300 Write block 1817 len 32768 data: 2 2 4 7 5 5 5 4
220/300 Write block 1295 len 28672 data: 3 2 3 5 5 6 6
221/300 Read block 3120 len 32768
222/300 Read block 11 len 12288
223/300 Read block 981 len 20480
224/300 Write block 907 len 40960 data: 2 2 2 2 2 2 2 2 2 2
225/300 Read block 204 len 32768
226/300 Read block 2999 len 4096
227/300 Flush
228/300 Flush
229/300 Read block 1849 len 32768
230/300 Read block 4753 len 16384
231/300 Read block 378 len 28672
232/300 Read block 51 len 36864
233/300 Write block 4254 len 28672 data: 3 3 2 2 2 2 3
234/300 Flush
235/300 Write block 1905 len 4096 data: 3
236/300 Write block 68 len 40960 data: 4 4 4 5 3 5 6 5 4 3
237/300 Write block 312 len 12288 data: 4 2 2
238/300 Read block 2616 len 28672
239/300 Read block 501 len 12288
240/300 Read block 849 len 32768
241/300 Read block 1392 len 36864
242/300 Write block 2165 len 24576 data: 5 4 3 3 3 5
243/300 Read block 1039 len 40960
244/300 Read block 2040 len 40960
245/300 Read block 2491 len 28672
246/300 Write block 770 len 16384 data: 2 3 3 3
247/300 Read block 3960 len 4096
248/300 Write block 1013 len 40960 data: 3 3 3 2 2 2 4 3 3 3
249/300 Read block 2801 len 24576
250/300 Write block 1639 len 28672 data: 5 5 5 5 6 8 6
251/300 Read block 2442 len 20480
252/300 Read block 4728 len 24576
253/300 Read block 24 len 24576
254/300 Read block 621 len 20480
255/300 Read block 1706 len 28672
256/300 Read block 2500 len 24576
257/300 Write block 4831 len 36864 data: 4 3 4 4 4 4 4 5 4
258/300 Flush
259/300 Read block 2305 len 20480
260/300 Write block 4062 len 36864 data: 4 4 4 4 4 4 4 4 4
261/300 Write block 3235 len 40960 data: 2 2 3 3 3 3 3 3 3 2
262/300 Write block 468 len 16384 data: 4 3 3 2
263/300 Write block 2894 len 12288 data: 4 6 4
264/300 Read block 3973 len 20480
265/300 Write block 639 len 40960 data: 5 5 7 6 6 7 6 5 4 4
266/300 Read block 431 len 40960
267/300 Read block 2705 len 12288
268/300 Read block 2277 len 28672
269/300 Read block 3249 len 4096
270/300 Write block 709 len 20480 data: 4 5 5 6 6
271/300 Write block 541 len 24576 data: 2 2 3 3 3 3
272/300 Read block 1554 len 32768
273/300 Read block 2826 len 20480
274/300 Read block 3025 len 32768
275/300 Write block 4840 len 12288 data: 4 4 5
276/300 Write block 3685 len 28672 data: 5 4 4 6 7 3 3
277/300 Write block 2522 len 8192 data: 4 4
278/300 Read block 877 len 16384
279/300 Read block 1710 len 36864
280/300 Read block 3733 len 28672
281/300 Read block 85 len 20480
282/300 Flush
283/300 Write block 849 len 40960 data: 4 4 3 2 2 4 5 7 6 6
284/300 Read block 2350 len 32768
285/300 Read block 582 len 16384
286/300 Write block 1969 len 16384 data: 4 3 3 2
287/300 Write block 175 len 8192 data: 3 3
288/300 Read block 909 len 12288
289/300 Write block 105 len 20480 data: 4 4 4 3 3
290/300 Read block 426 len 16384
291/300 Write block 157 len 12288 data: 5 5 5
292/300 Write block 2595 len 32768 data: 6 6 5 5 5 4 3 3
293/300 Read block 3338 len 4096
294/300 Read block 3208 len 36864
295/300 Read block 3103 len 8192
296/300 Read block 3713 len 12288
297/300 Write block 3265 len 8192 data: 3 2
298/300 Read block 1015 len 24576
299/300 Read block 2427 len 8192
300/300 Read block 151 len 24576
Main task has work to do, go find it
got dsc Start(2)
start 2
Feb 13 21:42:05.588 INFO request completed, latency_us: 238, response_code: 204, uri: /start/cid/2, method: POST, req_id: 885dc467-1262-4d64-8b6c-82a72769257d, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
3921 Acked 4920 Write 3 Done Done New true
3922 Acked 4921 Write 10 Done Done New true
3923 Acked 4922 Write 7 Done Done New true
3924 Acked 4923 Read 10 Done Done New true
3925 Acked 4924 Read 10 Done Done New false
3926 Acked 4925 Write 2 Done Done New false
3927 Acked 4926 Write 5 Done Done New false
3928 Acked 4927 Read 9 Done Done New false
3929 Acked 4928 Read 7 Done Done New false
3930 Acked 4929 Write 6 Done Done New false
3931 Acked 4930 Read 10 Done Done New false
3932 Acked 4931 Write 2 Done Done New false
3933 Acked 4932 Write 1 Done Done New false
3934 Acked 4933 Write 3 Done Done New false
3935 Acked 4934 Write 2 Done Done New false
3936 Acked 4935 Read 8 Done Done New false
3937 Acked 4936 Flush 0 Done Done New false
3938 Acked 4937 Flush 0 Done Done New false
3939 Acked 4938 Read 6 Done Done New false
3940 Acked 4939 Write 5 Done Done New false
3941 Acked 4940 Read 6 Done Done New false
3942 Acked 4941 Flush 0 Done Done New false
3943 Acked 4942 Write 9 Done Done New false
3944 Acked 4943 Write 1 Done Done New false
3945 Acked 4944 Read 8 Done Done New false
3946 Acked 4945 Read 8 Done Done New false
3947 Acked 4946 Write 9 Done Done New false
3948 Acked 4947 Read 5 Done Done New false
3949 Acked 4948 Write 8 Done Done New false
3950 Acked 4949 Write 9 Done Done New false
3951 Acked 4950 Write 7 Done Done New false
3952 Acked 4951 Read 2 Done Done New false
3953 Acked 4952 Read 9 Done Done New false
3954 Acked 4953 Read 1 Done Done New false
3955 Acked 4954 Write 10 Done Done New false
3956 Acked 4955 Write 4 Done Done New false
3957 Acked 4956 Read 9 Done Done New false
3958 Acked 4957 Write 9 Done Done New false
3959 Acked 4958 Read 8 Done Done New false
3960 Acked 4959 Write 5 Done Done New false
3961 Acked 4960 Read 6 Done Done New false
3962 Acked 4961 Read 7 Done Done New false
3963 Acked 4962 Write 8 Done Done New false
3964 Acked 4963 Flush 0 Done Done New false
3965 Acked 4964 Read 3 Done Done New false
3966 Acked 4965 Write 8 Done Done New false
3967 Acked 4966 Read 1 Done Done New false
3968 Acked 4967 Read 7 Done Done New false
3969 Acked 4968 Write 6 Done Done New false
3970 Acked 4969 Read 2 Done Done New false
3971 Acked 4970 Read 7 Done Done New false
3972 Acked 4971 Flush 0 Done Done New false
3973 Acked 4972 Write 3 Done Done New false
3974 Acked 4973 Read 6 Done Done New false
3975 Acked 4974 Write 8 Done Done New false
3976 Acked 4975 Write 4 Done Done New false
3977 Acked 4976 Write 5 Done Done New false
3978 Acked 4977 Read 1 Done Done New false
3979 Acked 4978 Write 1 Done Done New false
3980 Acked 4979 Read 3 Done Done New false
3981 Acked 4980 Read 9 Done Done New false
3982 Acked 4981 Write 4 Done Done New false
3983 Acked 4982 Read 8 Done Done New false
3984 Acked 4983 Read 2 Done Done New false
3985 Acked 4984 Write 8 Done Done New false
3986 Acked 4985 Read 3 Done Done New false
3987 Acked 4986 Write 1 Done Done New false
3988 Acked 4987 Read 1 Done Done New false
3989 Acked 4988 Write 10 Done Done New false
3990 Acked 4989 Flush 0 Done Done New false
3991 Acked 4990 Read 10 Done Done New false
3992 Acked 4991 Read 7 Done Done New false
3993 Acked 4992 Read 10 Done Done New false
3994 Acked 4993 Write 3 Done Done New false
3995 Acked 4994 Read 6 Done Done New false
3996 Acked 4995 Flush 0 Done Done New false
3997 Acked 4996 Read 5 Done Done New false
3998 Acked 4997 Read 5 Done Done New false
3999 Acked 4998 Read 1 Done Done New false
4000 Acked 4999 Read 3 Done Done New false
4001 Acked 5000 Read 4 Done Done New false
4002 Acked 5001 Write 2 Done Done New false
4003 Acked 5002 Read 1 Done Done New false
4004 Acked 5003 Read 9 Done Done New false
4005 Acked 5004 Flush 0 Done Done New false
4006 Acked 5005 Flush 0 Done Done New false
4007 Acked 5006 Write 10 Done Done New false
4008 Acked 5007 Write 10 Done Done New false
4009 Acked 5008 Write 7 Done Done New false
4010 Acked 5009 Read 7 Done Done New false
4011 Acked 5010 Write 1 Done Done New false
4012 Acked 5011 Read 2 Done Done New false
4013 Acked 5012 Read 10 Done Done New false
4014 Acked 5013 Write 6 Done Done New false
4015 Acked 5014 Write 7 Done Done New false
4016 Acked 5015 Read 4 Done Done New false
4017 Acked 5016 Read 5 Done Done New false
4018 Acked 5017 Write 5 Done Done New false
4019 Acked 5018 Read 8 Done Done New false
4020 Acked 5019 Write 10 Done Done New false
4021 Acked 5020 Read 4 Done Done New false
4022 Acked 5021 Read 2 Done Done New false
4023 Acked 5022 Write 8 Done Done New false
4024 Acked 5023 Read 8 Done Done New false
4025 Acked 5024 Read 3 Done Done New false
4026 Acked 5025 Read 7 Done Done New false
4027 Acked 5026 Read 2 Done Done New false
4028 Acked 5027 Write 9 Done Done New false
4029 Acked 5028 Flush 0 Done Done New false
4030 Acked 5029 Read 6 Done Done New false
4031 Acked 5030 Write 8 Done Done New false
4032 Acked 5031 Write 7 Done Done New false
4033 Acked 5032 Write 2 Done Done New false
4034 Acked 5033 Read 10 Done Done New false
4035 Acked 5034 Read 2 Done Done New false
4036 Acked 5035 Read 8 Done Done New false
4037 Acked 5036 Write 3 Done Done New false
4038 Acked 5037 Read 7 Done Done New false
4039 Acked 5038 Read 1 Done Done New false
4040 Acked 5039 Write 7 Done Done New false
4041 Acked 5040 Read 9 Done Done New false
4042 Acked 5041 Write 8 Done Done New false
4043 Acked 5042 Write 7 Done Done New false
4044 Acked 5043 Write 4 Done Done New false
4045 Acked 5044 Read 5 Done Done New false
4046 Acked 5045 Flush 0 Done Done New false
4047 Acked 5046 Read 4 Done Done New false
4048 Acked 5047 Write 3 Done Done New false
4049 Acked 5048 Read 7 Done Done New false
4050 Acked 5049 Write 10 Done Done New false
4051 Acked 5050 Flush 0 Done Done New false
4052 Acked 5051 Write 9 Done Done New false
4053 Acked 5052 Read 3 Done Done New false
4054 Acked 5053 Read 1 Done Done New false
4055 Acked 5054 Write 10 Done Done New false
4056 Acked 5055 Read 1 Done Done New false
4057 Acked 5056 Read 3 Done Done New false
4058 Acked 5057 Flush 0 Done Done New false
4059 Acked 5058 Read 6 Done Done New false
4060 Acked 5059 Write 5 Done Done New false
4061 Acked 5060 Flush 0 Done Done New false
4062 Acked 5061 Write 5 Done Done New false
4063 Acked 5062 Write 5 Done Done New false
4064 Acked 5063 Write 2 Done Done New false
4065 Acked 5064 Read 1 Done Done New false
4066 Acked 5065 Write 7 Done Done New false
4067 Acked 5066 Flush 0 Done Done New false
4068 Acked 5067 Write 5 Done Done New false
4069 Acked 5068 Read 5 Done Done New false
4070 Acked 5069 Read 6 Done Done New false
4071 Acked 5070 Write 2 Done Done New false
4072 Acked 5071 Read 2 Done Done New false
4073 Acked 5072 Read 5 Done Done New false
4074 Acked 5073 Flush 0 Done Done New false
4075 Acked 5074 Read 9 Done Done New false
4076 Acked 5075 Write 6 Done Done New false
4077 Acked 5076 Flush 0 Done Done New false
4078 Acked 5077 Write 1 Done Done New false
4079 Acked 5078 Read 8 Done Done New false
4080 Acked 5079 Read 9 Done Done New false
4081 Acked 5080 Write 10 Done Done New false
4082 Acked 5081 Write 5 Done Done New false
4083 Acked 5082 Write 2 Done Done New false
4084 Acked 5083 Read 5 Done Done New false
4085 Acked 5084 Write 2 Done Done New false
4086 Acked 5085 Read 9 Done Done New false
4087 Acked 5086 Write 6 Done Done New false
4088 Acked 5087 Write 5 Done Done New false
4089 Acked 5088 Read 6 Done Done New false
4090 Acked 5089 Read 3 Done Done New false
4091 Acked 5090 Read 9 Done Done New false
4092 Acked 5091 Write 1 Done Done New false
4093 Acked 5092 Read 2 Done Done New false
4094 Acked 5093 Read 4 Done Done New false
4095 Acked 5094 Read 3 Done Done New false
4096 Acked 5095 Read 7 Done Done New false
4097 Acked 5096 Write 1 Done Done New false
4098 Acked 5097 Read 4 Done Done New false
4099 Acked 5098 Read 5 Done Done New false
4100 Acked 5099 Write 10 Done Done New false
4101 Acked 5100 Write 6 Done Done New false
4102 Acked 5101 Flush 0 Done Done New false
4103 Acked 5102 Write 2 Done Done New false
4104 Acked 5103 Write 5 Done Done New false
4105 Acked 5104 Write 5 Done Done New false
4106 Acked 5105 Write 1 Done Done New false
4107 Acked 5106 Write 2 Done Done New false
4108 Acked 5107 Write 4 Done Done New false
4109 Acked 5108 Read 2 Done Done New false
4110 Acked 5109 Flush 0 Done Done New false
4111 Acked 5110 Flush 0 Done Done New false
4112 Acked 5111 Flush 0 Done Done New false
4113 Acked 5112 Write 3 Done Done New false
4114 Acked 5113 Read 3 Done Done New false
4115 Acked 5114 Read 7 Done Done New false
4116 Acked 5115 Read 1 Done Done New false
4117 Acked 5116 Flush 0 Done Done New false
4118 Acked 5117 Write 6 Done Done New false
4119 Acked 5118 Read 2 Done Done New false
4120 Acked 5119 Write 8 Done Done New false
4121 Acked 5120 Flush 0 Done Done New false
4122 Acked 5121 Read 8 Done Done New false
4123 Acked 5122 Write 7 Done Done New false
4124 Acked 5123 Read 8 Done Done New false
4125 Acked 5124 Flush 0 Done Done New false
4126 Acked 5125 Write 8 Done Done New false
4127 Acked 5126 Write 6 Done Done New false
4128 Acked 5127 Flush 0 Done Done New false
4129 Acked 5128 Read 6 Done Done New false
4130 Acked 5129 Write 4 Done Done New false
4131 Acked 5130 Write 9 Done Done New false
4132 Acked 5131 Flush 0 Done Done New false
4133 Acked 5132 Write 2 Done Done New false
4134 Acked 5133 Read 8 Done Done New false
4135 Acked 5134 Write 7 Done Done New false
4136 Acked 5135 Read 5 Done Done New false
4137 Acked 5136 Write 8 Done Done New false
4138 Acked 5137 Write 2 Done Done New false
4139 Acked 5138 Write 8 Done Done New false
4140 Acked 5139 Write 8 Done Done New false
4141 Acked 5140 Write 7 Done Done New false
4142 Acked 5141 Read 8 Done Done New false
4143 Acked 5142 Read 3 Done Done New false
4144 Acked 5143 Read 5 Done Done New false
4145 Acked 5144 Write 10 Done Done New false
4146 Acked 5145 Read 8 Done Done New false
4147 Acked 5146 Read 1 Done Done New false
4148 Acked 5147 Flush 0 Done Done New false
4149 Acked 5148 Flush 0 Done Done New false
4150 Acked 5149 Read 8 Done Done New false
4151 Acked 5150 Read 4 Done Done New false
4152 Acked 5151 Read 7 Done Done New false
4153 Acked 5152 Read 9 Done Done New false
4154 Acked 5153 Write 7 Done Done New false
4155 Acked 5154 Flush 0 Done Done New false
4156 Acked 5155 Write 1 Done Done New false
4157 Acked 5156 Write 10 Done Done New false
4158 Acked 5157 Write 3 Done Done New false
4159 Acked 5158 Read 7 Done Done New false
4160 Acked 5159 Read 3 Done Done New false
4161 Acked 5160 Read 8 Done Done New false
4162 Acked 5161 Read 9 Done Done New false
4163 Acked 5162 Write 6 Done Done New false
4164 Acked 5163 Read 10 Done Done New false
4165 Acked 5164 Read 10 Done Done New false
4166 Acked 5165 Read 7 Done Done New false
4167 Acked 5166 Write 4 Done Done New false
4168 Acked 5167 Read 1 Done Done New false
4169 Acked 5168 Write 10 Done Done New false
4170 Acked 5169 Read 6 Done Done New false
4171 Acked 5170 Write 7 Done Done New false
4172 Acked 5171 Read 5 Done Done New false
4173 Acked 5172 Read 6 Done Done New false
4174 Acked 5173 Read 6 Done Done New false
4175 Acked 5174 Read 5 Done Done New false
4176 Acked 5175 Read 7 Done Done New false
4177 Acked 5176 Read 6 Done Done New false
4178 Acked 5177 Write 9 Done Done New false
4179 Acked 5178 Flush 0 Done Done New false
4180 Acked 5179 Read 5 Done Done New false
4181 Acked 5180 Write 9 Done Done New false
4182 Acked 5181 Write 10 Done Done New false
4183 Acked 5182 Write 4 Done Done New false
4184 Acked 5183 Write 3 Done Done New false
4185 Acked 5184 Read 5 Done Done New false
4186 Acked 5185 Write 10 Done Done New false
4187 Acked 5186 Read 10 Done Done New false
4188 Acked 5187 Read 3 Done Done New false
4189 Acked 5188 Read 7 Done Done New false
4190 Acked 5189 Read 1 Done Done New false
4191 Acked 5190 Write 5 Done Done New false
4192 Acked 5191 Write 6 Done Done New false
4193 Acked 5192 Read 8 Done Done New false
4194 Acked 5193 Read 5 Done Done New false
4195 Acked 5194 Read 8 Done Done New false
4196 Acked 5195 Write 3 Done Done New false
4197 Acked 5196 Write 7 Done Done New false
4198 Acked 5197 Write 2 Done Done New false
4199 Acked 5198 Read 4 Done Done New false
4200 Acked 5199 Read 9 Done Done New false
4201 Acked 5200 Read 7 Done Done New false
4202 Acked 5201 Read 5 Done Done New false
4203 Acked 5202 Flush 0 Done Done New false
4204 Acked 5203 Write 10 Done Done New false
4205 Acked 5204 Read 8 Done Done New false
4206 Acked 5205 Read 4 Done Done New false
4207 Acked 5206 Write 4 Done Done New false
4208 Acked 5207 Write 2 Done Done New false
4209 Acked 5208 Read 3 Done Done New false
4210 Acked 5209 Write 5 Done Done New false
4211 Acked 5210 Read 4 Done Done New false
4212 Acked 5211 Write 3 Done Done New false
4213 Acked 5212 Write 8 Done Done New false
4214 Acked 5213 Read 1 Done Done New false
4215 Acked 5214 Read 9 Done Done New false
4216 Acked 5215 Read 2 Done Done New false
4217 Acked 5216 Read 3 Done Done New false
4218 Acked 5217 Write 2 Done Done New false
4219 Acked 5218 Read 6 Done Done New false
4220 Acked 5219 Read 2 Done Done New false
4221 Acked 5220 Read 6 Done Done New false
STATES DS:0 DS:1 DS:2 TOTAL
New 0 0 301 301
Sent 0 0 0 0
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 5202 5202 4919
Downstairs last five completed: 4919 4918 4917 4916 4915
Upstairs last five completed: 4221 4220 4219 4218 4217
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:05.705880196Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:05.705938184Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:05.705961913Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:05.705982932Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:05.705999181Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 301 jobs since flush 4919","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:05.70601705Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:05.706094918Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:06.706864477Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:06.706917855Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:06.706941464Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:06.706962333Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:06.706985092Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 302 jobs since flush 4919","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:06.707004041Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:06.707099278Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:07.708162296Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:07.708232533Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:07.708256493Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:07.708277152Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:07.708293081Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 302 jobs since flush 4919","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:07.70831057Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:07.708443415Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:08.709983625Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:08.710044832Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:08.710067612Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:08.71009646Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:08.71011321Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 302 jobs since flush 4919","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:08.710130699Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:08.710278344Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Active Active Offline
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
3921 Acked 4920 Write 3 Done Done New true
3922 Acked 4921 Write 10 Done Done New true
3923 Acked 4922 Write 7 Done Done New true
3924 Acked 4923 Read 10 Done Done New true
3925 Acked 4924 Read 10 Done Done New true
3926 Acked 4925 Write 2 Done Done New true
3927 Acked 4926 Write 5 Done Done New true
3928 Acked 4927 Read 9 Done Done New true
3929 Acked 4928 Read 7 Done Done New true
3930 Acked 4929 Write 6 Done Done New true
3931 Acked 4930 Read 10 Done Done New true
3932 Acked 4931 Write 2 Done Done New true
3933 Acked 4932 Write 1 Done Done New true
3934 Acked 4933 Write 3 Done Done New true
3935 Acked 4934 Write 2 Done Done New true
3936 Acked 4935 Read 8 Done Done New true
3937 Acked 4936 Flush 0 Done Done New true
3938 Acked 4937 Flush 0 Done Done New true
3939 Acked 4938 Read 6 Done Done New true
3940 Acked 4939 Write 5 Done Done New true
3941 Acked 4940 Read 6 Done Done New true
3942 Acked 4941 Flush 0 Done Done New true
3943 Acked 4942 Write 9 Done Done New true
3944 Acked 4943 Write 1 Done Done New true
3945 Acked 4944 Read 8 Done Done New true
3946 Acked 4945 Read 8 Done Done New true
3947 Acked 4946 Write 9 Done Done New true
3948 Acked 4947 Read 5 Done Done New true
3949 Acked 4948 Write 8 Done Done New true
3950 Acked 4949 Write 9 Done Done New true
3951 Acked 4950 Write 7 Done Done New true
3952 Acked 4951 Read 2 Done Done New true
3953 Acked 4952 Read 9 Done Done New true
3954 Acked 4953 Read 1 Done Done New true
3955 Acked 4954 Write 10 Done Done New true
3956 Acked 4955 Write 4 Done Done New true
3957 Acked 4956 Read 9 Done Done New true
3958 Acked 4957 Write 9 Done Done New true
3959 Acked 4958 Read 8 Done Done New true
3960 Acked 4959 Write 5 Done Done New true
3961 Acked 4960 Read 6 Done Done New true
3962 Acked 4961 Read 7 Done Done New true
3963 Acked 4962 Write 8 Done Done New true
3964 Acked 4963 Flush 0 Done Done New true
3965 Acked 4964 Read 3 Done Done New true
3966 Acked 4965 Write 8 Done Done New true
3967 Acked 4966 Read 1 Done Done New true
3968 Acked 4967 Read 7 Done Done New true
3969 Acked 4968 Write 6 Done Done New true
3970 Acked 4969 Read 2 Done Done New true
3971 Acked 4970 Read 7 Done Done New true
3972 Acked 4971 Flush 0 Done Done New true
3973 Acked 4972 Write 3 Done Done New true
3974 Acked 4973 Read 6 Done Done New true
3975 Acked 4974 Write 8 Done Done New true
3976 Acked 4975 Write 4 Done Done New true
3977 Acked 4976 Write 5 Done Done New true
3978 Acked 4977 Read 1 Done Done New true
3979 Acked 4978 Write 1 Done Done New true
3980 Acked 4979 Read 3 Done Done New true
3981 Acked 4980 Read 9 Done Done New true
3982 Acked 4981 Write 4 Done Done New true
3983 Acked 4982 Read 8 Done Done New true
3984 Acked 4983 Read 2 Done Done New true
3985 Acked 4984 Write 8 Done Done New true
3986 Acked 4985 Read 3 Done Done New true
3987 Acked 4986 Write 1 Done Done New true
3988 Acked 4987 Read 1 Done Done New true
3989 Acked 4988 Write 10 Done Done New true
3990 Acked 4989 Flush 0 Done Done New true
3991 Acked 4990 Read 10 Done Done New true
3992 Acked 4991 Read 7 Done Done New true
3993 Acked 4992 Read 10 Done Done New true
3994 Acked 4993 Write 3 Done Done New true
3995 Acked 4994 Read 6 Done Done New true
3996 Acked 4995 Flush 0 Done Done New true
3997 Acked 4996 Read 5 Done Done New true
3998 Acked 4997 Read 5 Done Done New true
3999 Acked 4998 Read 1 Done Done New true
4000 Acked 4999 Read 3 Done Done New true
4001 Acked 5000 Read 4 Done Done New true
4002 Acked 5001 Write 2 Done Done New true
4003 Acked 5002 Read 1 Done Done New true
4004 Acked 5003 Read 9 Done Done New true
4005 Acked 5004 Flush 0 Done Done New true
4006 Acked 5005 Flush 0 Done Done New true
4007 Acked 5006 Write 10 Done Done New true
4008 Acked 5007 Write 10 Done Done New true
4009 Acked 5008 Write 7 Done Done New true
4010 Acked 5009 Read 7 Done Done New true
4011 Acked 5010 Write 1 Done Done New true
4012 Acked 5011 Read 2 Done Done New true
4013 Acked 5012 Read 10 Done Done New true
4014 Acked 5013 Write 6 Done Done New true
4015 Acked 5014 Write 7 Done Done New true
4016 Acked 5015 Read 4 Done Done New true
4017 Acked 5016 Read 5 Done Done New true
4018 Acked 5017 Write 5 Done Done New true
4019 Acked 5018 Read 8 Done Done New true
4020 Acked 5019 Write 10 Done Done New true
4021 Acked 5020 Read 4 Done Done New true
4022 Acked 5021 Read 2 Done Done New true
4023 Acked 5022 Write 8 Done Done New true
4024 Acked 5023 Read 8 Done Done New true
4025 Acked 5024 Read 3 Done Done New true
4026 Acked 5025 Read 7 Done Done New true
4027 Acked 5026 Read 2 Done Done New true
4028 Acked 5027 Write 9 Done Done New true
4029 Acked 5028 Flush 0 Done Done New true
4030 Acked 5029 Read 6 Done Done New true
4031 Acked 5030 Write 8 Done Done New true
4032 Acked 5031 Write 7 Done Done New true
4033 Acked 5032 Write 2 Done Done New true
4034 Acked 5033 Read 10 Done Done New true
4035 Acked 5034 Read 2 Done Done New true
4036 Acked 5035 Read 8 Done Done New true
4037 Acked 5036 Write 3 Done Done New true
4038 Acked 5037 Read 7 Done Done New true
4039 Acked 5038 Read 1 Done Done New true
4040 Acked 5039 Write 7 Done Done New true
4041 Acked 5040 Read 9 Done Done New true
4042 Acked 5041 Write 8 Done Done New true
4043 Acked 5042 Write 7 Done Done New true
4044 Acked 5043 Write 4 Done Done New true
4045 Acked 5044 Read 5 Done Done New true
4046 Acked 5045 Flush 0 Done Done New true
4047 Acked 5046 Read 4 Done Done New true
4048 Acked 5047 Write 3 Done Done New true
4049 Acked 5048 Read 7 Done Done New true
4050 Acked 5049 Write 10 Done Done New true
4051 Acked 5050 Flush 0 Done Done New true
4052 Acked 5051 Write 9 Done Done New true
4053 Acked 5052 Read 3 Done Done New true
4054 Acked 5053 Read 1 Done Done New true
4055 Acked 5054 Write 10 Done Done New true
4056 Acked 5055 Read 1 Done Done New true
4057 Acked 5056 Read 3 Done Done New true
4058 Acked 5057 Flush 0 Done Done New true
4059 Acked 5058 Read 6 Done Done New true
4060 Acked 5059 Write 5 Done Done New true
4061 Acked 5060 Flush 0 Done Done New true
4062 Acked 5061 Write 5 Done Done New true
4063 Acked 5062 Write 5 Done Done New true
4064 Acked 5063 Write 2 Done Done New true
4065 Acked 5064 Read 1 Done Done New true
4066 Acked 5065 Write 7 Done Done New true
4067 Acked 5066 Flush 0 Done Done New true
4068 Acked 5067 Write 5 Done Done New true
4069 Acked 5068 Read 5 Done Done New true
4070 Acked 5069 Read 6 Done Done New true
4071 Acked 5070 Write 2 Done Done New true
4072 Acked 5071 Read 2 Done Done New true
4073 Acked 5072 Read 5 Done Done New true
4074 Acked 5073 Flush 0 Done Done New true
4075 Acked 5074 Read 9 Done Done New true
4076 Acked 5075 Write 6 Done Done New true
4077 Acked 5076 Flush 0 Done Done New true
4078 Acked 5077 Write 1 Done Done New true
4079 Acked 5078 Read 8 Done Done New true
4080 Acked 5079 Read 9 Done Done New true
4081 Acked 5080 Write 10 Done Done New true
4082 Acked 5081 Write 5 Done Done New true
4083 Acked 5082 Write 2 Done Done New true
4084 Acked 5083 Read 5 Done Done New true
4085 Acked 5084 Write 2 Done Done New true
4086 Acked 5085 Read 9 Done Done New true
4087 Acked 5086 Write 6 Done Done New true
4088 Acked 5087 Write 5 Done Done New true
4089 Acked 5088 Read 6 Done Done New true
4090 Acked 5089 Read 3 Done Done New true
4091 Acked 5090 Read 9 Done Done New true
4092 Acked 5091 Write 1 Done Done New true
4093 Acked 5092 Read 2 Done Done New true
4094 Acked 5093 Read 4 Done Done New true
4095 Acked 5094 Read 3 Done Done New true
4096 Acked 5095 Read 7 Done Done New true
4097 Acked 5096 Write 1 Done Done New true
4098 Acked 5097 Read 4 Done Done New true
4099 Acked 5098 Read 5 Done Done New true
4100 Acked 5099 Write 10 Done Done New true
4101 Acked 5100 Write 6 Done Done New true
4102 Acked 5101 Flush 0 Done Done New true
4103 Acked 5102 Write 2 Done Done New true
4104 Acked 5103 Write 5 Done Done New true
4105 Acked 5104 Write 5 Done Done New true
4106 Acked 5105 Write 1 Done Done New true
4107 Acked 5106 Write 2 Done Done New true
4108 Acked 5107 Write 4 Done Done New true
4109 Acked 5108 Read 2 Done Done New true
4110 Acked 5109 Flush 0 Done Done New true
4111 Acked 5110 Flush 0 Done Done New true
4112 Acked 5111 Flush 0 Done Done New true
4113 Acked 5112 Write 3 Done Done New true
4114 Acked 5113 Read 3 Done Done New true
4115 Acked 5114 Read 7 Done Done New true
4116 Acked 5115 Read 1 Done Done New true
4117 Acked 5116 Flush 0 Done Done New true
4118 Acked 5117 Write 6 Done Done New true
4119 Acked 5118 Read 2 Done Done New true
4120 Acked 5119 Write 8 Done Done New true
4121 Acked 5120 Flush 0 Done Done New true
4122 Acked 5121 Read 8 Done Done New true
4123 Acked 5122 Write 7 Done Done New true
4124 Acked 5123 Read 8 Done Done New true
4125 Acked 5124 Flush 0 Done Done New true
4126 Acked 5125 Write 8 Done Done New true
4127 Acked 5126 Write 6 Done Done New true
4128 Acked 5127 Flush 0 Done Done New true
4129 Acked 5128 Read 6 Done Done New true
4130 Acked 5129 Write 4 Done Done New true
4131 Acked 5130 Write 9 Done Done New true
4132 Acked 5131 Flush 0 Done Done New true
4133 Acked 5132 Write 2 Done Done New true
4134 Acked 5133 Read 8 Done Done New true
4135 Acked 5134 Write 7 Done Done New true
4136 Acked 5135 Read 5 Done Done New true
4137 Acked 5136 Write 8 Done Done New true
4138 Acked 5137 Write 2 Done Done New true
4139 Acked 5138 Write 8 Done Done New true
4140 Acked 5139 Write 8 Done Done New true
4141 Acked 5140 Write 7 Done Done New true
4142 Acked 5141 Read 8 Done Done New true
4143 Acked 5142 Read 3 Done Done New true
4144 Acked 5143 Read 5 Done Done New true
4145 Acked 5144 Write 10 Done Done New true
4146 Acked 5145 Read 8 Done Done New true
4147 Acked 5146 Read 1 Done Done New true
4148 Acked 5147 Flush 0 Done Done New true
4149 Acked 5148 Flush 0 Done Done New true
4150 Acked 5149 Read 8 Done Done New true
4151 Acked 5150 Read 4 Done Done New true
4152 Acked 5151 Read 7 Done Done New true
4153 Acked 5152 Read 9 Done Done New true
4154 Acked 5153 Write 7 Done Done New true
4155 Acked 5154 Flush 0 Done Done New true
4156 Acked 5155 Write 1 Done Done New true
4157 Acked 5156 Write 10 Done Done New true
4158 Acked 5157 Write 3 Done Done New true
4159 Acked 5158 Read 7 Done Done New true
4160 Acked 5159 Read 3 Done Done New true
4161 Acked 5160 Read 8 Done Done New true
4162 Acked 5161 Read 9 Done Done New true
4163 Acked 5162 Write 6 Done Done New true
4164 Acked 5163 Read 10 Done Done New true
4165 Acked 5164 Read 10 Done Done New true
4166 Acked 5165 Read 7 Done Done New true
4167 Acked 5166 Write 4 Done Done New true
4168 Acked 5167 Read 1 Done Done New true
4169 Acked 5168 Write 10 Done Done New true
4170 Acked 5169 Read 6 Done Done New true
4171 Acked 5170 Write 7 Done Done New true
4172 Acked 5171 Read 5 Done Done New true
4173 Acked 5172 Read 6 Done Done New true
4174 Acked 5173 Read 6 Done Done New true
4175 Acked 5174 Read 5 Done Done New true
4176 Acked 5175 Read 7 Done Done New true
4177 Acked 5176 Read 6 Done Done New true
4178 Acked 5177 Write 9 Done Done New true
4179 Acked 5178 Flush 0 Done Done New true
4180 Acked 5179 Read 5 Done Done New true
4181 Acked 5180 Write 9 Done Done New true
4182 Acked 5181 Write 10 Done Done New true
4183 Acked 5182 Write 4 Done Done New true
4184 Acked 5183 Write 3 Done Done New true
4185 Acked 5184 Read 5 Done Done New true
4186 Acked 5185 Write 10 Done Done New true
4187 Acked 5186 Read 10 Done Done New true
4188 Acked 5187 Read 3 Done Done New true
4189 Acked 5188 Read 7 Done Done New true
4190 Acked 5189 Read 1 Done Done New true
4191 Acked 5190 Write 5 Done Done New true
4192 Acked 5191 Write 6 Done Done New true
4193 Acked 5192 Read 8 Done Done New true
4194 Acked 5193 Read 5 Done Done New true
4195 Acked 5194 Read 8 Done Done New true
4196 Acked 5195 Write 3 Done Done New true
4197 Acked 5196 Write 7 Done Done New true
4198 Acked 5197 Write 2 Done Done New true
4199 Acked 5198 Read 4 Done Done New true
4200 Acked 5199 Read 9 Done Done New true
4201 Acked 5200 Read 7 Done Done New true
4202 Acked 5201 Read 5 Done Done New true
4203 Acked 5202 Flush 0 Done Done New true
4204 Acked 5203 Write 10 Done Done New true
4205 Acked 5204 Read 8 Done Done New true
4206 Acked 5205 Read 4 Done Done New true
4207 Acked 5206 Write 4 Done Done New true
4208 Acked 5207 Write 2 Done Done New true
4209 Acked 5208 Read 3 Done Done New true
4210 Acked 5209 Write 5 Done Done New true
4211 Acked 5210 Read 4 Done Done New true
4212 Acked 5211 Write 3 Done Done New true
4213 Acked 5212 Write 8 Done Done New true
4214 Acked 5213 Read 1 Done Done New true
4215 Acked 5214 Read 9 Done Done New true
4216 Acked 5215 Read 2 Done Done New true
4217 Acked 5216 Read 3 Done Done New true
4218 Acked 5217 Write 2 Done Done New true
4219 Acked 5218 Read 6 Done Done New true
4220 Acked 5219 Read 2 Done Done New true
4221 Acked 5220 Read 6 Done Done New true
4222 Acked 5221 Flush 0 Done Done New true
STATES DS:0 DS:1 DS:2 TOTAL
New 0 0 302 302
Sent 0 0 0 0
Done 302 302 0 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 5221 5221 4919
Downstairs last five completed: 4919 4918 4917 4916 4915
Upstairs last five completed: 4222 4221 4220 4219 4218
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:09.711237745Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:09.711297082Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:09.711321962Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:09.711343671Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:09.71135984Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 302 jobs since flush 4919","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:09.71137775Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:09.711495705Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1178)
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:10.713007046Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:10.713077383Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8830][2] reports Running
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:10.713110352Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:10.713142051Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:10.71316111Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 302 jobs since flush 4919","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:10.713180829Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:10.713303465Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:11.713857872Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:11.713932699Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:11.715320836Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:11.715349645Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 4919","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:11.715367704Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 4919","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:11.715519998Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:11.715545947Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:11.715570586Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:11.715596005Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:11.715617724Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 5221 5220 5219 5218 5217
Upstairs last five completed: 4222 4221 4220 4219 4218
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Feb 13 21:42:13.594 INFO request completed, latency_us: 363, response_code: 204, uri:[8810] Got stop action so:false kr:false
/stop/cid/0, method: POST, req_id: 167e07b6-0708-43fb-be0f-0a33e18543f2, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:13.595164633Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:42:13.595 INFO request completed, latency_us: 173, response_code: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:13.595257169Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
200, uri: /state/cid/0, method: GET, req_id: 8c38d58c-17cb-4804-90fe-977de631584e, remote_addr: 127.{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:13.595302838Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
0.0.{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:13.595348786Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
1:52124, local_addr: 127.0.0.1:9998
{"msg":"[0] client re-new 0 jobs since flush 5221","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:13.595386295Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:13.595423043Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:14.596298909Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:14.596359946Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:14.596383565Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:14.596404905Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:14.596421264Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 5221","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:14.596438703Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:14.596455863Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:15.598133267Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:15.598260422Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:15.5983148Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:15.598367028Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:15.598411526Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 5221","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:15.598458104Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:15.598478833Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:16.600040332Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:16.600097Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:16.600118999Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:16.600139578Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:16.600155957Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 5221","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:16.600173297Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:16.600191016Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:42:17.597 INFO request completed, latency_us: 221, response_code: 200, uri: /state/cid/0, method: GET, req_id: 1d7bc2dd-77af-48bf-a4ed-7bf1b5133ede, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Read block 1025 len 16384
002/300 Read block 2583 len 4096
003/300 Read block 1392 len 16384
004/300 Write block 2914 len 32768 data: 4 5 5 5 5 4 4 3
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:17.60132611Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:17.601395958Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:17.601432137Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:17.601467015Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:17.601494084Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 4 jobs since flush 5221","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:17.601513363Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{005/300" Write block 3679 len 24576 data:msg 6" 7:" 7 7 8client task is exiting 6"
,"v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:17.601534393Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
006/300 Read block 3296 len 28672
007/300 Read block 3673 len 32768
008/300 Read block 4577 len 32768
009/300 Write block 2378 len 32768 data: 4 3 2 2 2 3 3 3
010/300 Read block 1538 len 16384
011/300 Read block 2782 len 28672
012/300 Read block 3108 len 40960
013/300 Read block 2907 len 28672
014/300 Write block 3090 len 12288 data: 2 2 2
015/300 Read block 2282 len 16384
016/300 Write block 3264 len 16384 data: 4 4 3 2
017/300 Write block 3320 len 32768 data: 4 4 5 5 5 5 4 3
018/300 Flush
019/300 Write block 2570 len 24576 data: 5 7 7 8 9 8
020/300 Write block 1707 len 16384 data: 4 3 3 2
021/300 Write block 2105 len 8192 data: 4 4
022/300 Read block 4533 len 28672
023/300 Read block 2332 len 16384
024/300 Write block 2842 len 8192 data: 3 2
025/300 Read block 4955 len 24576
026/300 Flush
027/300 Write block 2871 len 28672 data: 2 2 2 2 3 3 3
028/300 Write block 3956 len 32768 data: 4 4 3 2 2 3 3 3
029/300 Read block 2291 len 4096
030/300 Flush
031/300 Read block 4536 len 12288
032/300 Read block 3774 len 4096
033/300 Read block 4032 len 20480
034/300 Read block 4855 len 8192
035/300 Read block 2395 len 40960
036/300 Write block 2556 len 32768 data: 4 3 2 3 3 3 3 4
037/300 Write block 4038 len 20480 data: 4 4 5 5 4
038/300 Read block 1228 len 36864
039/300 Read block 2927 len 4096
040/300 Read block 2422 len 24576
041/300 Write block 3225 len 16384 data: 10 10 9 9
042/300 Write block 3484 len 12288 data: 3 4 4
043/300 Write block 4251 len 12288 data: 3 2 3
044/300 Read block 4028 len 40960
045/300 Write block 3456 len 12288 data: 2 2 2
046/300 Read block 2811 len 4096
047/300 Read block 551 len 12288
048/300 Write block 2675 len 36864 data: 4 4 4 5 5 3 3 4 4
049/300 Read block 2319 len 20480
050/300 Read block 1541 len 28672
051/300 Write block 728 len 28672 data: 3 4 5 4 6 6 6
052/300 Write block 3369 len 32768 data: 6 5 4 5 6 5 4 3
053/300 Write block 4321 len 36864 data: 5 7 9 8 8 7 7 5 5
054/300 Write block 614 len 32768 data: 2 2 2 3 3 3 4 6
055/300 Write block 1532 len 12288 data: 3 3 3
056/300 Read block 354 len 36864
057/300 Read block 1190 len 8192
058/300 Write block 1353 len 40960 data: 4 5 5 7 5 4 5 4 4 3
059/300 Write block 3094 len 4096 data: 3
060/300 Write block 1426 len 4096 data: 2
061/300 Read block 4091 len 40960
062/300 Read block 388 len 28672
063/300 Write block 3788 len 36864 data: 4 4 5 3 3 2 2 2 3
064/300 Write block 1705 len 12288 data: 4 4 5
065/300 Read block 735 len 32768
066/300 Write block 731 len 16384 data: 5 7 7 7
067/300 Write block 1598 len 8192 data: 4 4
068/300 Write block 1053 len 40960 data: 4 4 3 3 4 3 3 4 4 4
069/300 Read block 527 len 28672
070/300 Read block 4188 len 32768
071/300 Read block 142 len 8192
072/300 Read block 4680 len 8192
073/300 Write block 51 len 32768 data: 3 3 3 3 3 3 4 5
074/300 Write block 1890 len 36864 data: 5 4 4 4 4 5 4 4 4
075/300 Read block 164 len 8192
076/300 Read block 3539 len 40960
077/300 Write block 3980 len 32768 data: 5 4 4 4 4 4 4 4
078/300 Write block 2482 len 36864 data: 3 3 4 5 6 5 5 4 4
079/300 Read block 354 len 8192
080/300 Write block 1625 len 20480 data: 3 3 3 4 3
081/300 Read block 645 len 32768
082/300 Read block 907 len 20480
083/300 Read block 3533 len 4096
084/300 Read block 2902 len 16384
085/300 Flush
086/300 Write block 958 len 32768 data: 4 3 3 3 4 4 4 4
087/300 Read block 3012 len 32768
088/300 Write block 2432 len 8192 data: 4 4
089/300 Read block 4041 len 16384
090/300 Read block 4163 len 16384
091/300 Write block 2708 len 32768 data: 3 4 4 4 4 5 3 3
092/300 Read block 133 len 8192
093/300 Read block 3335 len 36864
094/300 Read block 4564 len 12288
095/300 Write block 4282 len 24576 data: 2 2 3 4 4 6
096/300 Write block 4722 len 4096 data: 8
097/300 Read block 1224 len 32768
098/300 Read block 4496 len 36864
099/300 Write block 359 len 12288 data: 6 6 6
100/300 Flush
101/300 Read block 147 len 32768
102/300 Write block 3749 len 32768 data: 2 4 2 3 3 3 2 2
103/300 Write block 3180 len 8192 data: 3 3
104/300 Read block 4829 len 4096
105/300 Read block 4620 len 28672
106/300 Write block 4520 len 36864 data: 2 2 2 2 2 2 2 2 2
107/300 Read block 82 len 40960
108/300 Write block 4502 len 36864 data: 4 5 4 4 4 4 3 3 2
109/300 Read block 182 len 32768
110/300 Read block 3923 len 40960
111/300 Write block 114 len 8192 data: 2 2
112/300 Write block 3557 len 8192 data: 9 7
113/300 Read block 3405 len 16384
114/300 Read block 2143 len 20480
115/300 Flush
116/300 Write block 4334 len 20480 data: 5 5 4 4 5
117/300 Read block 3884 len 32768
118/300 Read block 2236 len 8192
119/300 Write block 2843 len 24576 data: 3 2 2 3 4 5
120/300 Write block 2698 len 28672 data: 2 2 2 2 2 2 2
121/300 Read block 4583 len 12288
122/300 Read block 482 len 12288
123/300 Read block 4903 len 36864
124/300 Write block 2111 len 32768 data: 3 3 3 4 4 3 3 3
125/300 Read block 361 len 12288
126/300 Read block 2203 len 24576
127/300 Read block 1439 len 32768
128/300 Write block 1822 len 4096 data: 6
129/300 Read block 2912 len 40960
130/300 Write block 4148 len 32768 data: 3 3 3 4 3 3 4 5
131/300 Read block 1959 len 28672
132/300 Read block 884 len 24576
133/300 Read block 918 len 8192
134/300 Read block 915 len 8192
135/300 Read block 2140 len 8192
136/300 Write block 296 len 28672 data: 3 3 3 3 4 4 4
137/300 Write block 3131 len 24576 data: 4 3 3 3 4 4
138/300 Write block 3661 len 20480 data: 2 3 3 3 4
139/300 Read block 4333 len 36864
140/300 Read block 4375 len 20480
141/300 Read block 804 len 40960
142/300 Write block 884 len 20480 data: 7 7 8 7 5
143/300 Read block 3458 len 28672
144/300 Read block 1669 len 8192
145/300 Write block 4083 len 12288 data: 4 4 4
146/300 Read block 1184 len 12288
147/300 Read block 1422 len 8192
148/300 Write block 1476 len 4096 data: 3
149/300 Write block 1474 len 16384 data: 3 4 4 2
150/300 Read block 2231 len 40960
151/300 Write block 3620 len 36864 data: 4 3 3 4 3 3 2 2 2
152/300 Write block 1763 len 4096 data: 6
153/300 Read block 2728 len 32768
154/300 Write block 3509 len 40960 data: 4 5 5 4 4 5 5 5 6 5
155/300 Write block 4378 len 16384 data: 3 6 6 5
156/300 Read block 4802 len 24576
157/300 Read block 4929 len 24576
158/300 Write block 1388 len 8192 data: 5 5
159/300 Read block 980 len 24576
160/300 Write block 3270 len 36864 data: 2 2 2 2 2 3 5 5 5
161/300 Write block 3861 len 28672 data: 3 3 2 2 2 4 4
162/300 Read block 4100 len 36864
163/300 Flush
164/300 Flush
165/300 Write block 634 len 36864 data: 3 3 3 3 3 6 6 8 7
166/300 Write block 493 len 28672 data: 6 6 5 2 3 3 3
167/300 Read block 2519 len 20480
168/300 Read block 3359 len 16384
169/300 Read block 3257 len 16384
170/300 Flush
171/300 Write block 121 len 20480 data: 4 4 6 6 6
172/300 Write block 4491 len 24576 data: 5 4 2 3 3 3
173/300 Read block 1412 len 12288
174/300 Read block 4591 len 12288
175/300 Read block 1922 len 24576
176/300 Read block 48 len 4096
177/300 Read block 2562 len 4096
178/300 Flush
179/300 Write block 1369 len 20480 data: 3 3 2 2 2
180/300 Write block 2239 len 36864 data: 4 4 4 4 4 5 4 3 3
181/300 Write block 4660 len 4096 data: 5
182/300 Read block 1985 len 4096
183/300 Write block 2404 len 8192 data: 4 4
184/300 Read block 4473 len 36864
185/300 Read block 2514 len 16384
186/300 Write block 649 len 28672 data: 3 4 4 4 3 3 4
187/300 Read block 4852 len 24576
188/300 Write block 32 len 4096 data: 3
189/300 Write block 4296 len 4096 data: 5
190/300 Read block 26 len 20480
191/300 Read block 4056 len 16384
192/300 Write block 4956 len 4096 data: 2
193/300 Write block 1673 len 28672 data: 2 2 2 2 2 2 2
194/300 Read block 2447 len 32768
195/300 Write block 828 len 8192 data: 4 5
196/300 Write block 294 len 28672 data: 3 3 4 4 4 4 5
197/300 Read block 2410 len 40960
198/300 Read block 4981 len 4096
199/300 Read block 1999 len 8192
200/300 Read block 2886 len 36864
201/300 Write block 304 len 20480 data: 4 4 4 4 4
202/300 Write block 1368 len 32768 data: 3 4 4 3 3 3 2 2
203/300 Write block 1058 len 36864 data: 4 4 5 5 5 4 4 4 2
204/300 Flush
205/300 Write block 1828 len 24576 data: 2 2 3 3 3 3
206/300 Flush
207/300 Read block 1355 len 8192
208/300 Read block 3696 len 8192
209/300 Flush
210/300 Read block 291 len 12288
211/300 Read block 502 len 8192
212/300 Read block 4619 len 12288
213/300 Write block 1942 len 20480 data: 4 5 3 3 4
214/300 Read block 1834 len 36864
215/300 Read block 4898 len 4096
216/300 Write block 3837 len 28672 data: 4 4 4 4 4 4 3
217/300 Flush
218/300 Flush
219/300 Read block 1377 len 36864
220/300 Write block 2254 len 36864 data: 5 5 6 6 4 3 3 4 4
221/300 Write block 1937 len 16384 data: 6 5 5 5
222/300 Write block 4326 len 12288 data: 8 8 6
223/300 Read block 503 len 12288
224/300 Read block 3776 len 4096
225/300 Write block 1965 len 24576 data: 5 5 5 5 5 4
226/300 Flush
227/300 Flush
228/300 Write block 431 len 36864 data: 3 3 3 2 3 2 2 2 2
229/300 Write block 2496 len 28672 data: 2 3 4 5 5 6 6
230/300 Write block 1475 len 12288 data: 5 5 3
231/300 Write block 1844 len 12288 data: 2 2 3
232/300 Write block 188 len 16384 data: 4 3 3 4
233/300 Flush
234/300 Flush
235/300 Write block 4622 len 28672 data: 4 6 6 5 3 3 3
236/300 Flush
237/300 Read block 4945 len 12288
238/300 Write block 4864 len 20480 data: 3 3 3 3 3
239/300 Flush
240/300 Read block 3950 len 16384
241/300 Write block 3023 len 40960 data: 5 4 3 3 5 5 5 4 4 4
242/300 Write block 3882 len 40960 data: 3 4 5 4 5 3 3 2 2 2
243/300 Write block 1570 len 32768 data: 6 7 7 5 6 5 5 4
244/300 Write block 3779 len 40960 data: 3 3 3 3 3 3 4 4 4 5
245/300 Read block 2405 len 20480
246/300 Read block 4300 len 4096
247/300 Read block 1113 len 40960
248/300 Write block 2902 len 40960 data: 2 2 3 3 3 3 4 4 3 4
249/300 Read block 2888 len 24576
250/300 Read block 3384 len 12288
251/300 Read block 3971 len 4096
252/300 Write block 2005 len 12288 data: 5 3 3
253/300 Write block 3687 len 36864 data: 5 7 8 4 4 4 2 2 2
254/300 Read block 4818 len 20480
255/300 Write block 3284 len 4096 data: 6
256/300 Read block 3046 len 40960
257/300 Flush
258/300 Write block 881 len 4096 data: 4
259/300 Write block 1965 len 4096 data: 6
260/300 Flush
261/300 Read block 574 len 28672
262/300 Write block 3241 len 32768 data: 4 4 4 3 2 3 4 4
263/300 Read block 2139 len 28672
264/300 Read block 1709 len 8192
265/300 Write block 4528 len 4096 data: 3
266/300 Write block 2651 len 20480 data: 6 6 5 5 6
267/300 Read block 2646 len 24576
268/300 Read block 4256 len 8192
269/300 Write block 376 len 40960 data: 3 3 3 2 3 4 5 5 5 5
270/300 Flush
271/300 Write block 1029 len 12288 data: 3 3 3
272/300 Write block 2445 len 12288 data: 3 3 3
273/300 Flush
274/300 Read block 128 len 16384
275/300 Flush
276/300 Write block 3662 len 20480 data: 4 4 4 5 4
277/300 Write block 3920 len 4096 data: 4
278/300 Read block 732 len 28672
279/300 Write block 3136 len 40960 data: 5 4 4 5 5 3 3 3 4 5
280/300 Write block 1677 len 12288 data: 3 3 3
281/300 Flush
282/300 Write block 885 len 36864 data: 8 9 8 6 4 4 4 4 5
283/300 Read block 4975 len 24576
284/300 Write block 2896 len 4096 data: 5
285/300 Read block 3749 len 8192
286/300 Read block 2760 len 20480
287/300 Write block 347 len 40960 data: 5 6 6 6 8 8 6 7 7 6
288/300 Read block 4775 len 24576
289/300 Flush
290/300 Read block 1993 len 24576
291/300 Write block 1975 len 12288 data: 2 2 2
292/300 Read block 2983 len 32768
293/300 Write block 1939 len 12288 data: 6 6 4
294/300 Write block 1995 len 32768 data: 5 5 6 6 6 6 5 3
295/300 Read block 3656 len 4096
296/300 Write block 2085 len 36864 data: 4 4 4 4 3 4 4 4 4
297/300 Write block 2193 len 20480 data: 2 3 3 3 3
298/300 Flush
299/300 Read block 2699 len 24576
300/300 Read block 1383 len 8192
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 13 21:42:18.332 INFO request completed, latency_us: 235, response_code: 204, uri: /start/cid/0, method: POST, req_id: 5ef995a9-03f1-49bd-b277-ebda4271be62, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
4223 Acked 5222 Read 4 New Done Done true
4224 Acked 5223 Read 1 New Done Done true
4225 Acked 5224 Read 4 New Done Done true
4226 Acked 5225 Write 8 New Done Done true
4227 Acked 5226 Write 6 New Done Done false
4228 Acked 5227 Read 7 New Done Done false
4229 Acked 5228 Read 8 New Done Done false
4230 Acked 5229 Read 8 New Done Done false
4231 Acked 5230 Write 8 New Done Done false
4232 Acked 5231 Read 4 New Done Done false
4233 Acked 5232 Read 7 New Done Done false
4234 Acked 5233 Read 10 New Done Done false
4235 Acked 5234 Read 7 New Done Done false
4236 Acked 5235 Write 3 New Done Done false
4237 Acked 5236 Read 4 New Done Done false
4238 Acked 5237 Write 4 New Done Done false
4239 Acked 5238 Write 8 New Done Done false
4240 Acked 5239 Flush 0 New Done Done false
4241 Acked 5240 Write 6 New Done Done false
4242 Acked 5241 Write 4 New Done Done false
4243 Acked 5242 Write 2 New Done Done false
4244 Acked 5243 Read 7 New Done Done false
4245 Acked 5244 Read 4 New Done Done false
4246 Acked 5245 Write 2 New Done Done false
4247 Acked 5246 Read 6 New Done Done false
4248 Acked 5247 Flush 0 New Done Done false
4249 Acked 5248 Write 7 New Done Done false
4250 Acked 5249 Write 8 New Done Done false
4251 Acked 5250 Read 1 New Done Done false
4252 Acked 5251 Flush 0 New Done Done false
4253 Acked 5252 Read 3 New Done Done false
4254 Acked 5253 Read 1 New Done Done false
4255 Acked 5254 Read 5 New Done Done false
4256 Acked 5255 Read 2 New Done Done false
4257 Acked 5256 Read 10 New Done Done false
4258 Acked 5257 Write 8 New Done Done false
4259 Acked 5258 Write 5 New Done Done false
4260 Acked 5259 Read 9 New Done Done false
4261 Acked 5260 Read 1 New Done Done false
4262 Acked 5261 Read 6 New Done Done false
4263 Acked 5262 Write 4 New Done Done false
4264 Acked 5263 Write 3 New Done Done false
4265 Acked 5264 Write 3 New Done Done false
4266 Acked 5265 Read 10 New Done Done false
4267 Acked 5266 Write 3 New Done Done false
4268 Acked 5267 Read 1 New Done Done false
4269 Acked 5268 Read 3 New Done Done false
4270 Acked 5269 Write 9 New Done Done false
4271 Acked 5270 Read 5 New Done Done false
4272 Acked 5271 Read 7 New Done Done false
4273 Acked 5272 Write 7 New Done Done false
4274 Acked 5273 Write 8 New Done Done false
4275 Acked 5274 Write 9 New Done Done false
4276 Acked 5275 Write 8 New Done Done false
4277 Acked 5276 Write 3 New Done Done false
4278 Acked 5277 Read 9 New Done Done false
4279 Acked 5278 Read 2 New Done Done false
4280 Acked 5279 Write 10 New Done Done false
4281 Acked 5280 Write 1 New Done Done false
4282 Acked 5281 Write 1 New Done Done false
4283 Acked 5282 Read 10 New Done Done false
4284 Acked 5283 Read 7 New Done Done false
4285 Acked 5284 Write 9 New Done Done false
4286 Acked 5285 Write 3 New Done Done false
4287 Acked 5286 Read 8 New Done Done false
4288 Acked 5287 Write 4 New Done Done false
4289 Acked 5288 Write 2 New Done Done false
4290 Acked 5289 Write 10 New Done Done false
4291 Acked 5290 Read 7 New Done Done false
4292 Acked 5291 Read 8 New Done Done false
4293 Acked 5292 Read 2 New Done Done false
4294 Acked 5293 Read 2 New Done Done false
4295 Acked 5294 Write 8 New Done Done false
4296 Acked 5295 Write 9 New Done Done false
4297 Acked 5296 Read 2 New Done Done false
4298 Acked 5297 Read 10 New Done Done false
4299 Acked 5298 Write 8 New Done Done false
4300 Acked 5299 Write 9 New Done Done false
4301 Acked 5300 Read 2 New Done Done false
4302 Acked 5301 Write 5 New Done Done false
4303 Acked 5302 Read 8 New Done Done false
4304 Acked 5303 Read 5 New Done Done false
4305 Acked 5304 Read 1 New Done Done false
4306 Acked 5305 Read 4 New Done Done false
4307 Acked 5306 Flush 0 New Done Done false
4308 Acked 5307 Write 8 New Done Done false
4309 Acked 5308 Read 8 New Done Done false
4310 Acked 5309 Write 2 New Done Done false
4311 Acked 5310 Read 4 New Done Done false
4312 Acked 5311 Read 4 New Done Done false
4313 Acked 5312 Write 8 New Done Done false
4314 Acked 5313 Read 2 New Done Done false
4315 Acked 5314 Read 9 New Done Done false
4316 Acked 5315 Read 3 New Done Done false
4317 Acked 5316 Write 6 New Done Done false
4318 Acked 5317 Write 1 New Done Done false
4319 Acked 5318 Read 8 New Done Done false
4320 Acked 5319 Read 9 New Done Done false
4321 Acked 5320 Write 3 New Done Done false
4322 Acked 5321 Flush 0 New Done Done false
4323 Acked 5322 Read 8 New Done Done false
4324 Acked 5323 Write 8 New Done Done false
4325 Acked 5324 Write 2 New Done Done false
4326 Acked 5325 Read 1 New Done Done false
4327 Acked 5326 Read 7 New Done Done false
4328 Acked 5327 Write 9 New Done Done false
4329 Acked 5328 Read 10 New Done Done false
4330 Acked 5329 Write 9 New Done Done false
4331 Acked 5330 Read 8 New Done Done false
4332 Acked 5331 Read 10 New Done Done false
4333 Acked 5332 Write 2 New Done Done false
4334 Acked 5333 Write 2 New Done Done false
4335 Acked 5334 Read 4 New Done Done false
4336 Acked 5335 Read 5 New Done Done false
4337 Acked 5336 Flush 0 New Done Done false
4338 Acked 5337 Write 5 New Done Done false
4339 Acked 5338 Read 8 New Done Done false
4340 Acked 5339 Read 2 New Done Done false
4341 Acked 5340 Write 6 New Done Done false
4342 Acked 5341 Write 7 New Done Done false
4343 Acked 5342 Read 3 New Done Done false
4344 Acked 5343 Read 3 New Done Done false
4345 Acked 5344 Read 9 New Done Done false
4346 Acked 5345 Write 8 New Done Done false
4347 Acked 5346 Read 3 New Done Done false
4348 Acked 5347 Read 6 New Done Done false
4349 Acked 5348 Read 8 New Done Done false
4350 Acked 5349 Write 1 New Done Done false
4351 Acked 5350 Read 10 New Done Done false
4352 Acked 5351 Write 8 New Done Done false
4353 Acked 5352 Read 7 New Done Done false
4354 Acked 5353 Read 6 New Done Done false
4355 Acked 5354 Read 2 New Done Done false
4356 Acked 5355 Read 2 New Done Done false
4357 Acked 5356 Read 2 New Done Done false
4358 Acked 5357 Write 7 New Done Done false
4359 Acked 5358 Write 6 New Done Done false
4360 Acked 5359 Write 5 New Done Done false
4361 Acked 5360 Read 9 New Done Done false
4362 Acked 5361 Read 5 New Done Done false
4363 Acked 5362 Read 10 New Done Done false
4364 Acked 5363 Write 5 New Done Done false
4365 Acked 5364 Read 7 New Done Done false
4366 Acked 5365 Read 2 New Done Done false
4367 Acked 5366 Write 3 New Done Done false
4368 Acked 5367 Read 3 New Done Done false
4369 Acked 5368 Read 2 New Done Done false
4370 Acked 5369 Write 1 New Done Done false
4371 Acked 5370 Write 4 New Done Done false
4372 Acked 5371 Read 10 New Done Done false
4373 Acked 5372 Write 9 New Done Done false
4374 Acked 5373 Write 1 New Done Done false
4375 Acked 5374 Read 8 New Done Done false
4376 Acked 5375 Write 10 New Done Done false
4377 Acked 5376 Write 4 New Done Done false
4378 Acked 5377 Read 6 New Done Done false
4379 Acked 5378 Read 6 New Done Done false
4380 Acked 5379 Write 2 New Done Done false
4381 Acked 5380 Read 6 New Done Done false
4382 Acked 5381 Write 9 New Done Done false
4383 Acked 5382 Write 7 New Done Done false
4384 Acked 5383 Read 9 New Done Done false
4385 Acked 5384 Flush 0 New Done Done false
4386 Acked 5385 Flush 0 New Done Done false
4387 Acked 5386 Write 9 New Done Done false
4388 Acked 5387 Write 7 New Done Done false
4389 Acked 5388 Read 5 New Done Done false
4390 Acked 5389 Read 4 New Done Done false
4391 Acked 5390 Read 4 New Done Done false
4392 Acked 5391 Flush 0 New Done Done false
4393 Acked 5392 Write 5 New Done Done false
4394 Acked 5393 Write 6 New Done Done false
4395 Acked 5394 Read 3 New Done Done false
4396 Acked 5395 Read 3 New Done Done false
4397 Acked 5396 Read 6 New Done Done false
4398 Acked 5397 Read 1 New Done Done false
4399 Acked 5398 Read 1 New Done Done false
4400 Acked 5399 Flush 0 New Done Done false
4401 Acked 5400 Write 5 New Done Done false
4402 Acked 5401 Write 9 New Done Done false
4403 Acked 5402 Write 1 New Done Done false
4404 Acked 5403 Read 1 New Done Done false
4405 Acked 5404 Write 2 New Done Done false
4406 Acked 5405 Read 9 New Done Done false
4407 Acked 5406 Read 4 New Done Done false
4408 Acked 5407 Write 7 New Done Done false
4409 Acked 5408 Read 6 New Done Done false
4410 Acked 5409 Write 1 New Done Done false
4411 Acked 5410 Write 1 New Done Done false
4412 Acked 5411 Read 5 New Done Done false
4413 Acked 5412 Read 4 New Done Done false
4414 Acked 5413 Write 1 New Done Done false
4415 Acked 5414 Write 7 New Done Done false
4416 Acked 5415 Read 8 New Done Done false
4417 Acked 5416 Write 2 New Done Done false
4418 Acked 5417 Write 7 New Done Done false
4419 Acked 5418 Read 10 New Done Done false
4420 Acked 5419 Read 1 New Done Done false
4421 Acked 5420 Read 2 New Done Done false
4422 Acked 5421 Read 9 New Done Done false
4423 Acked 5422 Write 5 New Done Done false
4424 Acked 5423 Write 8 New Done Done false
4425 Acked 5424 Write 9 New Done Done false
4426 Acked 5425 Flush 0 New Done Done false
4427 Acked 5426 Write 6 New Done Done false
4428 Acked 5427 Flush 0 New Done Done false
4429 Acked 5428 Read 2 New Done Done false
4430 Acked 5429 Read 2 New Done Done false
4431 Acked 5430 Flush 0 New Done Done false
4432 Acked 5431 Read 3 New Done Done false
4433 Acked 5432 Read 2 New Done Done false
4434 Acked 5433 Read 3 New Done Done false
4435 Acked 5434 Write 5 New Done Done false
4436 Acked 5435 Read 9 New Done Done false
4437 Acked 5436 Read 1 New Done Done false
4438 Acked 5437 Write 7 New Done Done false
4439 Acked 5438 Flush 0 New Done Done false
4440 Acked 5439 Flush 0 New Done Done false
4441 Acked 5440 Read 9 New Done Done false
4442 Acked 5441 Write 9 New Done Done false
4443 Acked 5442 Write 4 New Done Done false
4444 Acked 5443 Write 3 New Done Done false
4445 Acked 5444 Read 3 New Done Done false
4446 Acked 5445 Read 1 New Done Done false
4447 Acked 5446 Write 6 New Done Done false
4448 Acked 5447 Flush 0 New Done Done false
4449 Acked 5448 Flush 0 New Done Done false
4450 Acked 5449 Write 9 New Done Done false
4451 Acked 5450 Write 7 New Done Done false
4452 Acked 5451 Write 3 New Done Done false
4453 Acked 5452 Write 3 New Done Done false
4454 Acked 5453 Write 4 New Done Done false
4455 Acked 5454 Flush 0 New Done Done false
4456 Acked 5455 Flush 0 New Done Done false
4457 Acked 5456 Write 7 New Done Done false
4458 Acked 5457 Flush 0 New Done Done false
4459 Acked 5458 Read 3 New Done Done false
4460 Acked 5459 Write 5 New Done Done false
4461 Acked 5460 Flush 0 New Done Done false
4462 Acked 5461 Read 4 New Done Done false
4463 Acked 5462 Write 10 New Done Done false
4464 Acked 5463 Write 10 New Done Done false
4465 Acked 5464 Write 8 New Done Done false
4466 Acked 5465 Write 10 New Done Done false
4467 Acked 5466 Read 5 New Done Done false
4468 Acked 5467 Read 1 New Done Done false
4469 Acked 5468 Read 10 New Done Done false
4470 Acked 5469 Write 10 New Done Done false
4471 Acked 5470 Read 6 New Done Done false
4472 Acked 5471 Read 3 New Done Done false
4473 Acked 5472 Read 1 New Done Done false
4474 Acked 5473 Write 3 New Done Done false
4475 Acked 5474 Write 9 New Done Done false
4476 Acked 5475 Read 5 New Done Done false
4477 Acked 5476 Write 1 New Done Done false
4478 Acked 5477 Read 10 New Done Done false
4479 Acked 5478 Flush 0 New Done Done false
4480 Acked 5479 Write 1 New Done Done false
4481 Acked 5480 Write 1 New Done Done false
4482 Acked 5481 Flush 0 New Done Done false
4483 Acked 5482 Read 7 New Done Done false
4484 Acked 5483 Write 8 New Done Done false
4485 Acked 5484 Read 7 New Done Done false
4486 Acked 5485 Read 2 New Done Done false
4487 Acked 5486 Write 1 New Done Done false
4488 Acked 5487 Write 5 New Done Done false
4489 Acked 5488 Read 6 New Done Done false
4490 Acked 5489 Read 2 New Done Done false
4491 Acked 5490 Write 10 New Done Done false
4492 Acked 5491 Flush 0 New Done Done false
4493 Acked 5492 Write 3 New Done Done false
4494 Acked 5493 Write 3 New Done Done false
4495 Acked 5494 Flush 0 New Done Done false
4496 Acked 5495 Read 4 New Done Done false
4497 Acked 5496 Flush 0 New Done Done false
4498 Acked 5497 Write 5 New Done Done false
4499 Acked 5498 Write 1 New Done Done false
4500 Acked 5499 Read 7 New Done Done false
4501 Acked 5500 Write 10 New Done Done false
4502 Acked 5501 Write 3 New Done Done false
4503 Acked 5502 Flush 0 New Done Done false
4504 Acked 5503 Write 9 New Done Done false
4505 Acked 5504 Read 6 New Done Done false
4506 Acked 5505 Write 1 New Done Done false
4507 Acked 5506 Read 2 New Done Done false
4508 Acked 5507 Read 5 New Done Done false
4509 Acked 5508 Write 10 New Done Done false
4510 Acked 5509 Read 6 New Done Done false
4511 Acked 5510 Flush 0 New Done Done false
4512 Acked 5511 Read 6 New Done Done false
4513 Acked 5512 Write 3 New Done Done false
4514 Acked 5513 Read 8 New Done Done false
4515 Acked 5514 Write 3 New Done Done false
4516 Acked 5515 Write 8 New Done Done false
4517 Acked 5516 Read 1 New Done Done false
4518 Acked 5517 Write 9 New Done Done false
4519 Acked 5518 Write 5 New Done Done false
4520 Acked 5519 Flush 0 New Done Done false
4521 Acked 5520 Read 6 New Done Done false
4522 Acked 5521 Read 2 New Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
New 300 0 0 300
Sent 0 0 0 0
Done 0 300 300 600
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 5221 5519 5519
Downstairs last five completed: 5221 5220 5219 5218 5217
Upstairs last five completed: 4522 4521 4520 4519 4518
CLIENT: Up:0 ds:300 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:18.602431906Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:18.602483744Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:18.602502794Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:18.602523343Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:18.602536622Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 300 jobs since flush 5221","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:18.602551542Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:18.602684287Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:19.603906679Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:19.603968297Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:19.603991785Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:19.604012995Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:19.604041914Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 5221","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:19.604065663Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:19.604168459Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1179)
[8810][0] reports Running
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:20.604800033Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:20.60487894Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:20.606135652Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:20.606159131Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 5221","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:20.60617377Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 5221","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:20.606383542Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:20.60642984Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:20.606462979Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:20.606494698Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:20.606526267Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 5522 5521 5520 5519 5518
Upstairs last five completed: 4523 4522 4521 4520 4519
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(2)
stop 2
Feb 13 21:42:22.336 INFO request completed, latency_us: 355[8830] Got stop action so:false kr:false
, response_code: 204, uri: /stop/cid/2, method: POST, req_id: a2d89306-ed32-4a3b-8d1d-6a2a594dae34, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:22.336987114Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:22.337036992Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:22.337070851Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:22.3370855Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 5522","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:22.337158387Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:22.337195126Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:42:22.337 INFO request completed, latency_us: 134, response_code: 200, uri: /state/cid/2, method: GET, req_id: 5fea4cef-4f52-460f-a99f-29e316d01822, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:23.337811841Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:23.337868769Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:23.337888468Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:23.337905217Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:23.337918067Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 5522","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:23.337932096Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:23.337954585Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:24.338673097Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:24.338736414Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:24.338759263Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:24.338779983Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:24.338803872Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 5522","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:24.338822291Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:24.33883979Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:25.339501323Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:25.339554521Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:25.33957238Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:25.33958892Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:25.339601589Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 5522","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:25.339622909Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:25.339650808Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:42:26.339 INFO request completed, latency_us: 211, response_code: 200, uri: /state/cid/2, method: GET, req_id: a51d6731-a21b-468e-9adc-532456957cff, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Read block 3019 len 24576
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:26.340418066Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:26.340467804Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:26.340486373Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:26.340508193Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:26.340521552Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 1 jobs since flush 5522","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:26.340535542Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:26.340549421Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
002/300 Write block 2548 len 8192 data: 2 3
003/300 Write block 2464 len 4096 data: 7
004/300 Write block 585 len 8192 data: 3 3
005/300 Write block 2276 len 32768 data: 2 2 2 2 2 2 2 3
006/300 Read block 4318 len 16384
007/300 Read block 4453 len 28672
008/300 Read block 4858 len 24576
009/300 Read block 4322 len 28672
010/300 Write block 1731 len 32768 data: 3 3 4 3 3 3 3 3
011/300 Write block 1591 len 24576 data: 4 4 4 6 6 5
012/300 Flush
013/300 Read block 4284 len 8192
014/300 Read block 3690 len 4096
015/300 Read block 1415 len 36864
016/300 Read block 4788 len 16384
017/300 Read block 4256 len 16384
018/300 Write block 1773 len 40960 data: 5 5 5 5 5 5 3 3 2 2
019/300 Read block 2847 len 24576
020/300 Read block 3984 len 8192
021/300 Write block 1453 len 40960 data: 4 4 4 4 3 3 3 4 4 4
022/300 Write block 2190 len 36864 data: 3 3 2 3 4 4 4 4 4
023/300 Flush
024/300 Write block 1345 len 36864 data: 2 3 3 3 2 3 3 4 5
025/300 Read block 1219 len 12288
026/300 Read block 4828 len 32768
027/300 Read block 1678 len 8192
028/300 Read block 3422 len 12288
029/300 Write block 3369 len 8192 data: 7 6
030/300 Write block 1904 len 28672 data: 3 4 3 2 2 4 4
031/300 Read block 3678 len 12288
032/300 Write block 4280 len 28672 data: 2 2 3 3 4 5 5
033/300 Write block 3933 len 20480 data: 2 2 2 2 2
034/300 Flush
035/300 Read block 2110 len 32768
036/300 Write block 4678 len 24576 data: 2 2 3 5 4 3
037/300 Read block 2873 len 20480
038/300 Read block 9 len 40960
039/300 Write block 3313 len 20480 data: 5 4 3 3 3
040/300 Read block 2905 len 20480
041/300 Write block 4283 len 4096 data: 4
042/300 Write block 519 len 12288 data: 7 7 7
043/300 Read block 25 len 4096
044/300 Flush
045/300 Write block 4972 len 20480 data: 3 3 4 4 4
046/300 Read block 2513 len 24576
047/300 Read block 3938 len 20480
048/300 Flush
049/300 Flush
050/300 Read block 569 len 4096
051/300 Write block 2558 len 40960 data: 3 4 4 4 4 5 4 4 5 4
052/300 Write block 2931 len 12288 data: 2 2 2
053/300 Read block 3664 len 36864
054/300 Read block 932 len 28672
055/300 Write block 2731 len 12288 data: 3 4 4
056/300 Write block 3283 len 4096 data: 6
057/300 Read block 1240 len 8192
058/300 Read block 297 len 20480
059/300 Read block 3592 len 24576
060/300 Write block 2773 len 8192 data: 2 2
061/300 Read block 73 len 4096
062/300 Read block 3325 len 32768
063/300 Read block 1205 len 20480
064/300 Read block 2301 len 8192
065/300 Write block 1260 len 20480 data: 5 5 5 6 4
066/300 Read block 1253 len 8192
067/300 Read block 1775 len 40960
068/300 Flush
069/300 Read block 1485 len 24576
070/300 Write block 2068 len 40960 data: 3 3 3 3 2 3 5 5 5 4
071/300 Read block 238 len 36864
072/300 Read block 4020 len 8192
073/300 Write block 4418 len 24576 data: 3 4 5 5 5 4
074/300 Flush
075/300 Read block 4674 len 12288
076/300 Read block 3069 len 28672
077/300 Write block 1246 len 28672 data: 3 3 4 4 5 5 5
078/300 Flush
079/300 Flush
080/300 Write block 88 len 24576 data: 4 4 5 5 6 5
081/300 Read block 3213 len 4096
082/300 Read block 1256 len 28672
083/300 Write block 1671 len 24576 data: 2 2 3 3 3 3
084/300 Flush
085/300 Write block 2460 len 16384 data: 5 5 5 6
086/300 Write block 216 len 16384 data: 2 2 2 3
087/300 Write block 383 len 16384 data: 6 6 6 6
088/300 Read block 1948 len 8192
089/300 Write block 892 len 28672 data: 5 6 5 5 4 4 4
090/300 Write block 432 len 24576 data: 4 4 3 4 3 3
091/300 Read block 2026 len 20480
092/300 Flush
093/300 Read block 4453 len 32768
094/300 Read block 3823 len 8192
095/300 Read block 4898 len 36864
096/300 Write block 32 len 32768 data: 4 3 4 4 4 4 4 5
097/300 Read block 4265 len 24576
098/300 Read block 3939 len 20480
099/300 Read block 1381 len 16384
100/300 Write block 246 len 16384 data: 5 4 6 6
101/300 Read block 1175 len 28672
102/300 Write block 2595 len 28672 data: 7 7 6 6 6 5 4
103/300 Flush
104/300 Read block 4332 len 16384
105/300 Write block 3090 len 20480 data: 3 3 3 3 4
106/300 Read block 3755 len 16384
107/300 Write block 2576 len 12288 data: 7 6 5
108/300 Read block 2162 len 8192
109/300 Read block 46 len 4096
110/300 Write block 3350 len 8192 data: 5 6
111/300 Read block 4186 len 28672
112/300 Read block 2800 len 40960
113/300 Write block 4263 len 8192 data: 4 4
114/300 Read block 3326 len 32768
115/300 Write block 2104 len 32768 data: 4 5 5 4 4 4 3 4
116/300 Write block 2889 len 36864 data: 3 4 4 4 4 5 7 6 4
117/300 Write block 1367 len 20480 data: 3 4 5 5 4
118/300 Write block 747 len 4096 data: 6
119/300 Write block 5 len 24576 data: 5 7 7 4 3 3
120/300 Write block 59 len 20480 data: 6 6 7 7 6
121/300 Write block 4524 len 32768 data: 3 3 3 3 4 3 4 5
122/300 Write block 2036 len 32768 data: 4 3 3 3 3 2 2 2
123/300 Read block 1396 len 8192
124/300 Read block 4030 len 8192
125/300 Flush
126/300 Read block 1767 len 12288
127/300 Write block 3309 len 12288 data: 6 6 6
128/300 Read block 4111 len 36864
129/300 Write block 4377 len 12288 data: 3 4 7
130/300 Write block 2107 len 32768 data: 5 5 5 4 5 4 4 5
131/300 Read block 1002 len 28672
132/300 Read block 500 len 28672
133/300 Read block 3823 len 4096
134/300 Write block 129 len 32768 data: 5 6 6 6 4 4 3 4
135/300 Write block 4700 len 24576 data: 5 4 4 2 2 4
136/300 Flush
137/300 Read block 1675 len 12288
138/300 Write block 3109 len 12288 data: 4 4 4
139/300 Write block 238 len 28672 data: 6 6 5 5 5 5 6
140/300 Write block 206 len 36864 data: 2 2 2 2 2 2 2 2 2
141/300 Read block 1742 len 28672
142/300 Read block 972 len 36864
143/300 Read block 3766 len 32768
144/300 Read block 1866 len 16384
145/300 Read block 1015 len 36864
146/300 Read block 2296 len 32768
147/300 Read block 3375 len 32768
148/300 Read block 4094 len 36864
149/300 Read block 3579 len 36864
150/300 Read block 3098 len 20480
151/300 Read block 4942 len 20480
152/300 Read block 4782 len 12288
153/300 Read block 4100 len 32768
154/300 Read block 3907 len 28672
155/300 Read block 4974 len 8192
156/300 Read block 1769 len 32768
157/300 Write block 3334 len 4096 data: 3
158/300 Read block 1678 len 16384
159/300 Read block 496 len 40960
160/300 Write block 1941 len 32768 data: 5 5 6 4 4 5 4 4
161/300 Read block 4240 len 16384
162/300 Read block 1637 len 28672
163/300 Write block 4359 len 20480 data: 3 3 3 3 3
164/300 Write block 1070 len 16384 data: 4 4 5 4
165/300 Read block 2977 len 36864
166/300 Write block 3679 len 40960 data: 7 8 8 8 9 7 6 5 6 8
167/300 Read block 4871 len 40960
168/300 Write block 2303 len 32768 data: 8 9 8 7 6 5 5 5
169/300 Write block 1167 len 16384 data: 4 4 4 3
170/300 Flush
171/300 Read block 1189 len 20480
172/300 Write block 3685 len 40960 data: 7 6 7 9 9 5 5 5 3 3
173/300 Read block 3955 len 40960
174/300 Read block 3032 len 40960
175/300 Read block 304 len 16384
176/300 Write block 3169 len 32768 data: 3 3 5 5 5 4 4 5
177/300 Read block 782 len 12288
178/300 Write block 1998 len 28672 data: 7 7 7 6 4 4 4
179/300 Read block 1706 len 28672
180/300 Write block 302 len 36864 data: 5 4 5 5 5 5 5 5 5
181/300 Read block 2625 len 32768
182/300 Read block 3891 len 4096
183/300 Read block 1086 len 4096
184/300 Read block 2965 len 40960
185/300 Read block 4623 len 40960
186/300 Write block 4089 len 28672 data: 4 5 5 5 4 3 3
187/300 Read block 754 len 20480
188/300 Write block 3985 len 36864 data: 5 5 5 2 2 2 2 2 2
189/300 Read block 4982 len 32768
190/300 Read block 3011 len 24576
191/300 Read block 3239 len 16384
192/300 Write block 4620 len 12288 data: 3 4 5
193/300 Read block 1816 len 28672
194/300 Write block 581 len 4096 data: 4
195/300 Flush
196/300 Write block 3825 len 36864 data: 5 5 4 5 5 5 4 5 5
197/300 Read block 4934 len 28672
198/300 Read block 100 len 40960
199/300 Write block 4912 len 36864 data: 2 2 2 3 4 4 5 5 4
200/300 Read block 3728 len 40960
201/300 Read block 2876 len 12288
202/300 Write block 4512 len 24576 data: 3 5 3 3 2 2
203/300 Flush
204/300 Write block 2981 len 16384 data: 5 6 5 5
205/300 Read block 3533 len 32768
206/300 Write block 799 len 28672 data: 5 5 4 3 3 3 3
207/300 Read block 2258 len 8192
208/300 Flush
209/300 Read block 3123 len 24576
210/300 Write block 3504 len 20480 data: 3 3 3 3 3
211/300 Write block 3257 len 24576 data: 6 7 4 3 3 3
212/300 Read block 3554 len 40960
213/300 Write block 3597 len 32768 data: 3 2 2 3 3 3 3 4
214/300 Read block 3649 len 16384
215/300 Read block 2843 len 40960
216/300 Read block 2256 len 24576
217/300 Write block 2039 len 20480 data: 4 4 3 3 3
218/300 Flush
219/300 Write block 923 len 4096 data: 6
220/300 Read block 3668 len 28672
221/300 Read block 1149 len 28672
222/300 Read block 3902 len 24576
223/300 Read block 1114 len 4096
224/300 Write block 1348 len 24576 data: 4 3 4 4 5 6
225/300 Read block 1728 len 40960
226/300 Write block 1733 len 40960 data: 5 4 4 4 4 4 2 2 2 3
227/300 Read block 2928 len 32768
228/300 Write block 1508 len 8192 data: 2 3
229/300 Write block 353 len 12288 data: 7 8 8
230/300 Read block 1019 len 20480
231/300 Read block 3642 len 40960
232/300 Write block 1153 len 8192 data: 3 4
233/300 Read block 48 len 40960
234/300 Flush
235/300 Write block 1348 len 24576 data: 5 4 5 5 6 7
236/300 Write block 1360 len 20480 data: 5 5 4 3 3
237/300 Write block 4772 len 40960 data: 6 3 3 4 4 3 3 4 3 2
238/300 Flush
239/300 Write block 3734 len 20480 data: 3 3 5 6 6
240/300 Read block 3724 len 40960
241/300 Write block 550 len 24576 data: 3 2 3 2 2 3
242/300 Read block 3687 len 28672
243/300 Write block 232 len 8192 data: 5 5
244/300 Read block 3405 len 40960
245/300 Read block 2119 len 20480
246/300 Flush
247/300 Write block 2926 len 24576 data: 3 3 3 2 2 3
248/300 Read block 1685 len 36864
249/300 Write block 1067 len 28672 data: 4 4 4 5 5 6 5
250/300 Read block 4971 len 8192
251/300 Read block 4086 len 20480
252/300 Read block 2487 len 20480
253/300 Read block 4263 len 8192
254/300 Write block 604 len 12288 data: 4 4 4
255/300 Read block 2210 len 32768
256/300 Write block 0 len 28672 data: 2 4 5 5 6 6 8
257/300 Read block 192 len 32768
258/300 Read block 4319 len 40960
259/300 Write block 4627 len 32768 data: 4 4 2 2 3 3 4 4
260/300 Read block 4220 len 12288
261/300 Read block 2486 len 32768
262/300 Write block 187 len 36864 data: 5 5 4 4 5 4 4 5 5
263/300 Flush
264/300 Read block 2724 len 20480
265/300 Flush
266/300 Write block 1843 len 12288 data: 2 3 3
267/300 Read block 3207 len 8192
268/300 Write block 2555 len 36864 data: 4 5 4 4 5 5 5 5 6
269/300 Write block 4286 len 32768 data: 6 7 6 6 5 5 6 6
270/300 Write block 231 len 32768 data: 5 6 6 5 5 5 4 7
271/300 Read block 1341 len 24576
272/300 Read block 3002 len 4096
273/300 Flush
274/300 Read block 960 len 20480
275/300 Flush
276/300 Read block 3616 len 32768
277/300 Write block 1070 len 8192 data: 6 6
278/300 Flush
279/300 Flush
280/300 Read block 999 len 20480
281/300 Write block 4994 len 20480 data: 4 3 2 2 2
282/300 Write block 1258 len 40960 data: 4 4 6 6 6 7 5 4 4 5
283/300 Flush
284/300 Flush
285/300 Flush
286/300 Read block 4919 len 28672
287/300 Read block 3914 len 4096
288/300 Write block 3529 len 40960 data: 3 5 5 5 5 4 3 3 4 4
289/300 Write block 2281 len 32768 data: 3 3 4 5 5 5 5 5
290/300 Write block 1727 len 8192 data: 4 2
291/300 Write block 3854 len 40960 data: 7 7 7 6 7 6 5 4 4 3
292/300 Read block 4250 len 8192
293/300 Read block 2522 len 16384
294/300 Read block 4912 len 24576
295/300 Read block 3251 len 20480
296/300 Read block 3645 len 4096
297/300 Read block 3742 len 8192
298/300 Flush
299/300 Write block 4726 len 36864 data: 6 5 5 6 6 5 5 5 4
300/300 Flush
Main task has work to do, go find it
got dsc Start(2)
start 2
Feb 13 21:42:27.148 INFO request completed, latency_us: 207, response_code: 204, uri: /start/cid/2, method: POST, req_id: c3d1ffb7-cfa5-4702-a7a5-0a75ec01cb7b, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
States: Active Active Offline
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
4524 Acked 5523 Read 6 Done Done New true
4525 Acked 5524 Write 2 Done Done New false
4526 Acked 5525 Write 1 Done Done New false
4527 Acked 5526 Write 2 Done Done New false
4528 Acked 5527 Write 8 Done Done New false
4529 Acked 5528 Read 4 Done Done New false
4530 Acked 5529 Read 7 Done Done New false
4531 Acked 5530 Read 6 Done Done New false
4532 Acked 5531 Read 7 Done Done New false
4533 Acked 5532 Write 8 Done Done New false
4534 Acked 5533 Write 6 Done Done New false
4535 Acked 5534 Flush 0 Done Done New false
4536 Acked 5535 Read 2 Done Done New false
4537 Acked 5536 Read 1 Done Done New false
4538 Acked 5537 Read 9 Done Done New false
4539 Acked 5538 Read 4 Done Done New false
4540 Acked 5539 Read 4 Done Done New false
4541 Acked 5540 Write 10 Done Done New false
4542 Acked 5541 Read 6 Done Done New false
4543 Acked 5542 Read 2 Done Done New false
4544 Acked 5543 Write 10 Done Done New false
4545 Acked 5544 Write 9 Done Done New false
4546 Acked 5545 Flush 0 Done Done New false
4547 Acked 5546 Write 9 Done Done New false
4548 Acked 5547 Read 3 Done Done New false
4549 Acked 5548 Read 8 Done Done New false
4550 Acked 5549 Read 2 Done Done New false
4551 Acked 5550 Read 3 Done Done New false
4552 Acked 5551 Write 2 Done Done New false
4553 Acked 5552 Write 7 Done Done New false
4554 Acked 5553 Read 3 Done Done New false
4555 Acked 5554 Write 7 Done Done New false
4556 Acked 5555 Write 5 Done Done New false
4557 Acked 5556 Flush 0 Done Done New false
4558 Acked 5557 Read 8 Done Done New false
4559 Acked 5558 Write 6 Done Done New false
4560 Acked 5559 Read 5 Done Done New false
4561 Acked 5560 Read 10 Done Done New false
4562 Acked 5561 Write 5 Done Done New false
4563 Acked 5562 Read 5 Done Done New false
4564 Acked 5563 Write 1 Done Done New false
4565 Acked 5564 Write 3 Done Done New false
4566 Acked 5565 Read 1 Done Done New false
4567 Acked 5566 Flush 0 Done Done New false
4568 Acked 5567 Write 5 Done Done New false
4569 Acked 5568 Read 6 Done Done New false
4570 Acked 5569 Read 5 Done Done New false
4571 Acked 5570 Flush 0 Done Done New false
4572 Acked 5571 Flush 0 Done Done New false
4573 Acked 5572 Read 1 Done Done New false
4574 Acked 5573 Write 10 Done Done New false
4575 Acked 5574 Write 3 Done Done New false
4576 Acked 5575 Read 9 Done Done New false
4577 Acked 5576 Read 7 Done Done New false
4578 Acked 5577 Write 3 Done Done New false
4579 Acked 5578 Write 1 Done Done New false
4580 Acked 5579 Read 2 Done Done New false
4581 Acked 5580 Read 5 Done Done New false
4582 Acked 5581 Read 6 Done Done New false
4583 Acked 5582 Write 2 Done Done New false
4584 Acked 5583 Read 1 Done Done New false
4585 Acked 5584 Read 8 Done Done New false
4586 Acked 5585 Read 5 Done Done New false
4587 Acked 5586 Read 2 Done Done New false
4588 Acked 5587 Write 5 Done Done New false
4589 Acked 5588 Read 2 Done Done New false
4590 Acked 5589 Read 10 Done Done New false
4591 Acked 5590 Flush 0 Done Done New false
4592 Acked 5591 Read 6 Done Done New false
4593 Acked 5592 Write 10 Done Done New false
4594 Acked 5593 Read 9 Done Done New false
4595 Acked 5594 Read 2 Done Done New false
4596 Acked 5595 Write 6 Done Done New false
4597 Acked 5596 Flush 0 Done Done New false
4598 Acked 5597 Read 3 Done Done New false
4599 Acked 5598 Read 7 Done Done New false
4600 Acked 5599 Write 7 Done Done New false
4601 Acked 5600 Flush 0 Done Done New false
4602 Acked 5601 Flush 0 Done Done New false
4603 Acked 5602 Write 6 Done Done New false
4604 Acked 5603 Read 1 Done Done New false
4605 Acked 5604 Read 7 Done Done New false
4606 Acked 5605 Write 6 Done Done New false
4607 Acked 5606 Flush 0 Done Done New false
4608 Acked 5607 Write 4 Done Done New false
4609 Acked 5608 Write 4 Done Done New false
4610 Acked 5609 Write 4 Done Done New false
4611 Acked 5610 Read 2 Done Done New false
4612 Acked 5611 Write 7 Done Done New false
4613 Acked 5612 Write 6 Done Done New false
4614 Acked 5613 Read 5 Done Done New false
4615 Acked 5614 Flush 0 Done Done New false
4616 Acked 5615 Read 8 Done Done New false
4617 Acked 5616 Read 2 Done Done New false
4618 Acked 5617 Read 9 Done Done New false
4619 Acked 5618 Write 8 Done Done New false
4620 Acked 5619 Read 6 Done Done New false
4621 Acked 5620 Read 5 Done Done New false
4622 Acked 5621 Read 4 Done Done New false
4623 Acked 5622 Write 4 Done Done New false
4624 Acked 5623 Read 7 Done Done New false
4625 Acked 5624 Write 7 Done Done New false
4626 Acked 5625 Flush 0 Done Done New false
4627 Acked 5626 Read 4 Done Done New false
4628 Acked 5627 Write 5 Done Done New false
4629 Acked 5628 Read 4 Done Done New false
4630 Acked 5629 Write 3 Done Done New false
4631 Acked 5630 Read 2 Done Done New false
4632 Acked 5631 Read 1 Done Done New false
4633 Acked 5632 Write 2 Done Done New false
4634 Acked 5633 Read 7 Done Done New false
4635 Acked 5634 Read 10 Done Done New false
4636 Acked 5635 Write 2 Done Done New false
4637 Acked 5636 Read 8 Done Done New false
4638 Acked 5637 Write 8 Done Done New false
4639 Acked 5638 Write 9 Done Done New false
4640 Acked 5639 Write 5 Done Done New false
4641 Acked 5640 Write 1 Done Done New false
4642 Acked 5641 Write 6 Done Done New false
4643 Acked 5642 Write 5 Done Done New false
4644 Acked 5643 Write 8 Done Done New false
4645 Acked 5644 Write 8 Done Done New false
4646 Acked 5645 Read 2 Done Done New false
4647 Acked 5646 Read 2 Done Done New false
4648 Acked 5647 Flush 0 Done Done New false
4649 Acked 5648 Read 3 Done Done New false
4650 Acked 5649 Write 3 Done Done New false
4651 Acked 5650 Read 9 Done Done New false
4652 Acked 5651 Write 3 Done Done New false
4653 Acked 5652 Write 8 Done Done New false
4654 Acked 5653 Read 7 Done Done New false
4655 Acked 5654 Read 7 Done Done New false
4656 Acked 5655 Read 1 Done Done New false
4657 Acked 5656 Write 8 Done Done New false
4658 Acked 5657 Write 6 Done Done New false
4659 Acked 5658 Flush 0 Done Done New false
4660 Acked 5659 Read 3 Done Done New false
4661 Acked 5660 Write 3 Done Done New false
4662 Acked 5661 Write 7 Done Done New false
4663 Acked 5662 Write 9 Done Done New false
4664 Acked 5663 Read 7 Done Done New false
4665 Acked 5664 Read 9 Done Done New false
4666 Acked 5665 Read 8 Done Done New false
4667 Acked 5666 Read 4 Done Done New false
4668 Acked 5667 Read 9 Done Done New false
4669 Acked 5668 Read 8 Done Done New false
4670 Acked 5669 Read 8 Done Done New false
4671 Acked 5670 Read 9 Done Done New false
4672 Acked 5671 Read 9 Done Done New false
4673 Acked 5672 Read 5 Done Done New false
4674 Acked 5673 Read 5 Done Done New false
4675 Acked 5674 Read 3 Done Done New false
4676 Acked 5675 Read 8 Done Done New false
4677 Acked 5676 Read 7 Done Done New false
4678 Acked 5677 Read 2 Done Done New false
4679 Acked 5678 Read 8 Done Done New false
4680 Acked 5679 Write 1 Done Done New false
4681 Acked 5680 Read 4 Done Done New false
4682 Acked 5681 Read 10 Done Done New false
4683 Acked 5682 Write 8 Done Done New false
4684 Acked 5683 Read 4 Done Done New false
4685 Acked 5684 Read 7 Done Done New false
4686 Acked 5685 Write 5 Done Done New false
4687 Acked 5686 Write 4 Done Done New false
4688 Acked 5687 Read 9 Done Done New false
4689 Acked 5688 Write 10 Done Done New false
4690 Acked 5689 Read 10 Done Done New false
4691 Acked 5690 Write 8 Done Done New false
4692 Acked 5691 Write 4 Done Done New false
4693 Acked 5692 Flush 0 Done Done New false
4694 Acked 5693 Read 5 Done Done New false
4695 Acked 5694 Write 10 Done Done New false
4696 Acked 5695 Read 10 Done Done New false
4697 Acked 5696 Read 10 Done Done New false
4698 Acked 5697 Read 4 Done Done New false
4699 Acked 5698 Write 8 Done Done New false
4700 Acked 5699 Read 3 Done Done New false
4701 Acked 5700 Write 7 Done Done New false
4702 Acked 5701 Read 7 Done Done New false
4703 Acked 5702 Write 9 Done Done New false
4704 Acked 5703 Read 8 Done Done New false
4705 Acked 5704 Read 1 Done Done New false
4706 Acked 5705 Read 1 Done Done New false
4707 Acked 5706 Read 10 Done Done New false
4708 Acked 5707 Read 10 Done Done New false
4709 Acked 5708 Write 7 Done Done New false
4710 Acked 5709 Read 5 Done Done New false
4711 Acked 5710 Write 9 Done Done New false
4712 Acked 5711 Read 8 Done Done New false
4713 Acked 5712 Read 6 Done Done New false
4714 Acked 5713 Read 4 Done Done New false
4715 Acked 5714 Write 3 Done Done New false
4716 Acked 5715 Read 7 Done Done New false
4717 Acked 5716 Write 1 Done Done New false
4718 Acked 5717 Flush 0 Done Done New false
4719 Acked 5718 Write 9 Done Done New false
4720 Acked 5719 Read 7 Done Done New false
4721 Acked 5720 Read 10 Done Done New false
4722 Acked 5721 Write 9 Done Done New false
4723 Acked 5722 Read 10 Done Done New false
4724 Acked 5723 Read 3 Done Done New false
4725 Acked 5724 Write 6 Done Done New false
4726 Acked 5725 Flush 0 Done Done New false
4727 Acked 5726 Write 4 Done Done New false
4728 Acked 5727 Read 8 Done Done New false
4729 Acked 5728 Write 7 Done Done New false
4730 Acked 5729 Read 2 Done Done New false
4731 Acked 5730 Flush 0 Done Done New false
4732 Acked 5731 Read 6 Done Done New false
4733 Acked 5732 Write 5 Done Done New false
4734 Acked 5733 Write 6 Done Done New false
4735 Acked 5734 Read 10 Done Done New false
4736 Acked 5735 Write 8 Done Done New false
4737 Acked 5736 Read 4 Done Done New false
4738 Acked 5737 Read 10 Done Done New false
4739 Acked 5738 Read 6 Done Done New false
4740 Acked 5739 Write 5 Done Done New false
4741 Acked 5740 Flush 0 Done Done New false
4742 Acked 5741 Write 1 Done Done New false
4743 Acked 5742 Read 7 Done Done New false
4744 Acked 5743 Read 7 Done Done New false
4745 Acked 5744 Read 6 Done Done New false
4746 Acked 5745 Read 1 Done Done New false
4747 Acked 5746 Write 6 Done Done New false
4748 Acked 5747 Read 10 Done Done New false
4749 Acked 5748 Write 10 Done Done New false
4750 Acked 5749 Read 8 Done Done New false
4751 Acked 5750 Write 2 Done Done New false
4752 Acked 5751 Write 3 Done Done New false
4753 Acked 5752 Read 5 Done Done New false
4754 Acked 5753 Read 10 Done Done New false
4755 Acked 5754 Write 2 Done Done New false
4756 Acked 5755 Read 10 Done Done New false
4757 Acked 5756 Flush 0 Done Done New false
4758 Acked 5757 Write 6 Done Done New false
4759 Acked 5758 Write 5 Done Done New false
4760 Acked 5759 Write 10 Done Done New false
4761 Acked 5760 Flush 0 Done Done New false
4762 Acked 5761 Write 5 Done Done New false
4763 Acked 5762 Read 10 Done Done New false
4764 Acked 5763 Write 6 Done Done New false
4765 Acked 5764 Read 7 Done Done New false
4766 Acked 5765 Write 2 Done Done New false
4767 Acked 5766 Read 10 Done Done New false
4768 Acked 5767 Read 5 Done Done New false
4769 Acked 5768 Flush 0 Done Done New false
4770 Acked 5769 Write 6 Done Done New false
4771 Acked 5770 Read 9 Done Done New false
4772 Acked 5771 Write 7 Done Done New false
4773 Acked 5772 Read 2 Done Done New false
4774 Acked 5773 Read 5 Done Done New false
4775 Acked 5774 Read 5 Done Done New false
4776 Acked 5775 Read 2 Done Done New false
4777 Acked 5776 Write 3 Done Done New false
4778 Acked 5777 Read 8 Done Done New false
4779 Acked 5778 Write 7 Done Done New false
4780 Acked 5779 Read 8 Done Done New false
4781 Acked 5780 Read 10 Done Done New false
4782 Acked 5781 Write 8 Done Done New false
4783 Acked 5782 Read 3 Done Done New false
4784 Acked 5783 Read 8 Done Done New false
4785 Acked 5784 Write 9 Done Done New false
4786 Acked 5785 Flush 0 Done Done New false
4787 Acked 5786 Read 5 Done Done New false
4788 Acked 5787 Flush 0 Done Done New false
4789 Acked 5788 Write 3 Done Done New false
4790 Acked 5789 Read 2 Done Done New false
4791 Acked 5790 Write 9 Done Done New false
4792 Acked 5791 Write 8 Done Done New false
4793 Acked 5792 Write 8 Done Done New false
4794 Acked 5793 Read 6 Done Done New false
4795 Acked 5794 Read 1 Done Done New false
4796 Acked 5795 Flush 0 Done Done New false
4797 Acked 5796 Read 5 Done Done New false
4798 Acked 5797 Flush 0 Done Done New false
4799 Acked 5798 Read 8 Done Done New false
4800 Acked 5799 Write 2 Done Done New false
4801 Acked 5800 Flush 0 Done Done New false
4802 Acked 5801 Flush 0 Done Done New false
4803 Acked 5802 Read 5 Done Done New false
4804 Acked 5803 Write 5 Done Done New false
4805 Acked 5804 Write 10 Done Done New false
4806 Acked 5805 Flush 0 Done Done New false
4807 Acked 5806 Flush 0 Done Done New false
4808 Acked 5807 Flush 0 Done Done New false
4809 Acked 5808 Read 7 Done Done New false
4810 Acked 5809 Read 1 Done Done New false
4811 Acked 5810 Write 10 Done Done New false
4812 Acked 5811 Write 8 Done Done New false
4813 Acked 5812 Write 2 Done Done New false
4814 Acked 5813 Write 10 Done Done New false
4815 Acked 5814 Read 2 Done Done New false
4816 Acked 5815 Read 4 Done Done New false
4817 Acked 5816 Read 6 Done Done New false
4818 Acked 5817 Read 5 Done Done New false
4819 Acked 5818 Read 1 Done Done New false
4820 Acked 5819 Read 2 Done Done New false
4821 Acked 5820 Flush 0 Done Done New false
4822 Acked 5821 Write 9 Done Done New false
4823 Acked 5822 Flush 0 Done Done New false
STATES DS:0 DS:1 DS:2 TOTAL
New 0 0 300 300
Sent 0 0 0 0
Done 300 300 0 600
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 5822 5822 5522
Downstairs last five completed: 5522 5521 5520 5519 5518
Upstairs last five completed: 4823 4822 4821 4820 4819
CLIENT: Up:0 ds:300 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:27.341618099Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:27.341680587Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:27.341703776Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:27.341724925Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:27.341740974Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 300 jobs since flush 5522","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:27.341758334Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:27.34184789Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:28.342719305Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:28.342779913Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:28.342807972Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:28.342830031Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:28.34284612Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 300 jobs since flush 5522","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:28.34286488Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:28.343029873Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1180)
[8830][2] reports Running
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:29.343818652Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:29.343953027Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:29.345384592Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:29.345417611Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 5522","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:29.34543623Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 5522","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:29.345619294Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:29.345648382Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:29.345666252Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:29.345687821Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:29.34570513Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 5822 5821 5820 5819 5818
Upstairs last five completed: 4823 4822 4821 4820 4819
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(1)
stop 1
Feb 13 21:42:31.152 INFO request completed, latency_us: 383, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 6f680c5c-1c0d-45d4-b502-972e91abf378, remote_addr: 127[8820] Got stop action so:false kr:false
.0.0.1:52124, local_addr: 127.0.0.1:9998
Feb 13 21:42:31.153 INFO request completed, latency_us: 162, response_code: 200, uri: /state/cid/1, method: GET, req_id: aba791d2-4836-414b-9dd4-ce91820c7ba6, remote_addr: 127.0.0{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:31.153313499Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
.1:52124, local_addr: 127.0.0.1:9998
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:31.153388446Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:31.153416995Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:31.153439354Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 0 jobs since flush 5822","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:31.153458653Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:31.153472963Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:32.155249834Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:32.155311741Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:32.15533578Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:32.155364169Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:32.155380929Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 0 jobs since flush 5822","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:32.155398828Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:32.155416527Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:33.157295504Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:33.15739284Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:33.157436288Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:33.157475817Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:33.157506345Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 0 jobs since flush 5822","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:33.157539504Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:33.157571803Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:34.159277717Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:34.159331895Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:34.159352494Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:34.159374513Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:34.159387562Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 0 jobs since flush 5822","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:34.159401592Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:34.159418131Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:42:35.155 INFO request completed, latency_us: 265, response_code: 200, uri: /state/cid/1, method: GET, req_id: 241a4456-eda0-4fbc-ae81-d28646bb0653, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Read block 1728 len 12288
002/300 Read block 4255 len 32768
003/300 Write block 2009 len 36864 data: 5 4 5 6 6 7 6 6 9
004/300 Write block 4203 len 28672 data: 4 3 3 3 2 2 4
005/300 Write block 1963 len 4096 data: 3
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:35.160350634Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:35.160421451Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:35.160457309Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:35.160490778Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:35.160517337Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 5 jobs since flush 5822","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:35.160534866Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:35.160549346Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
006/300 Write block 4616 len 36864 data: 3 3 4 4 4 5 6 7 7
007/300 Write block 1899 len 12288 data: 4 4 3
008/300 Read block 22 len 32768
009/300 Write block 831 len 28672 data: 2 2 2 2 3 3 2
010/300 Read block 730 len 24576
011/300 Read block 3041 len 32768
012/300 Write block 3493 len 16384 data: 2 2 3 3
013/300 Write block 3327 len 12288 data: 4 3 3
014/300 Write block 4352 len 36864 data: 3 3 2 2 2 3 3 4 4
015/300 Write block 60 len 28672 data: 7 8 8 7 7 6 5
016/300 Read block 3479 len 20480
017/300 Write block 1059 len 36864 data: 5 6 6 6 5 5 5 3 5
018/300 Write block 2914 len 12288 data: 5 6 6
019/300 Read block 4433 len 32768
020/300 Read block 1559 len 28672
021/300 Write block 4038 len 4096 data: 5
022/300 Read block 2401 len 8192
023/300 Write block 4406 len 12288 data: 5 5 4
024/300 Write block 4707 len 8192 data: 4 5
025/300 Write block 3817 len 8192 data: 5 5
026/300 Write block 4722 len 4096 data: 9
027/300 Read block 1161 len 4096
028/300 Write block 4016 len 28672 data: 2 3 3 2 3 3 4
029/300 Flush
030/300 Read block 4033 len 8192
031/300 Write block 4240 len 24576 data: 4 2 2 3 3 4
032/300 Write block 1922 len 4096 data: 4
033/300 Read block 3758 len 28672
034/300 Read block 1270 len 24576
035/300 Read block 1924 len 24576
036/300 Flush
037/300 Flush
038/300 Read block 2698 len 40960
039/300 Write block 2123 len 24576 data: 3 3 3 2 3 3
040/300 Flush
041/300 Write block 2435 len 32768 data: 4 4 5 5 5 6 7 8
042/300 Read block 2213 len 20480
043/300 Read block 410 len 36864
044/300 Flush
045/300 Write block 3643 len 36864 data: 4 4 4 4 5 4 4 5 4
046/300 Read block 4211 len 4096
047/300 Write block 786 len 24576 data: 3 3 2 2 2 2
048/300 Write block 2625 len 36864 data: 5 5 6 5 4 3 3 4 4
049/300 Write block 2662 len 20480 data: 6 6 5 5 4
050/300 Read block 3735 len 24576
051/300 Write block 1077 len 24576 data: 4 4 4 4 3 3
052/300 Write block 1297 len 32768 data: 4 6 6 7 7 6 5 5
053/300 Read block 4719 len 4096
054/300 Read block 2398 len 4096
055/300 Write block 1631 len 16384 data: 4 4 4 7
056/300 Read block 2001 len 32768
057/300 Read block 3063 len 32768
058/300 Write block 3523 len 40960 data: 2 2 3 3 3 3 4 6 6 6
059/300 Write block 2500 len 32768 data: 6 7 7 8 8 6 6 6
060/300 Read block 2803 len 40960
061/300 Read block 510 len 24576
062/300 Write block 2110 len 12288 data: 5 6 5
063/300 Read block 511 len 12288
064/300 Read block 2831 len 4096
065/300 Write block 2150 len 24576 data: 3 3 3 3 2 2
066/300 Read block 2040 len 40960
067/300 Read block 3602 len 24576
068/300 Read block 3764 len 28672
069/300 Read block 1922 len 24576
070/300 Read block 3222 len 20480
071/300 Flush
072/300 Write block 2941 len 12288 data: 4 4 4
073/300 Flush
074/300 Write block 994 len 8192 data: 5 4
075/300 Write block 1514 len 16384 data: 3 3 3 3
076/300 Read block 1166 len 40960
077/300 Read block 1897 len 20480
078/300 Write block 4033 len 16384 data: 2 3 3 3
079/300 Write block 2878 len 20480 data: 3 4 4 4 4
080/300 Write block 4869 len 8192 data: 2 3
081/300 Write block 2529 len 40960 data: 5 4 3 4 5 6 5 3 3 4
082/300 Read block 924 len 40960
083/300 Write block 4956 len 28672 data: 3 2 3 3 2 3 3
084/300 Read block 3108 len 24576
085/300 Read block 1822 len 4096
086/300 Write block 4501 len 16384 data: 4 5 6 5
087/300 Read block 1462 len 16384
088/300 Write block 1772 len 36864 data: 3 6 6 6 6 6 6 4 4
089/300 Read block 360 len 16384
090/300 Read block 3554 len 28672
091/300 Write block 4328 len 40960 data: 7 6 5 4 4 4 6 6 5 5
092/300 Write block 4748 len 36864 data: 3 3 3 3 3 2 2 2 2
093/300 Read block 4543 len 28672
094/300 Read block 3313 len 32768
095/300 Flush
096/300 Read block 2660 len 20480
097/300 Read block 2013 len 40960
098/300 Read block 800 len 32768
099/300 Read block 2431 len 32768
100/300 Write block 849 len 40960 data: 5 5 4 3 3 5 6 8 7 7
101/300 Write block 3445 len 8192 data: 3 3
102/300 Write block 4756 len 24576 data: 3 2 2 3 3 3
103/300 Write block 1686 len 4096 data: 2
104/300 Write block 1910 len 32768 data: 5 5 6 6 4 4 4 4
105/300 Flush
106/300 Flush
107/300 Write block 3015 len 12288 data: 4 5 5
108/300 Read block 3614 len 36864
109/300 Write block 211 len 32768 data: 3 3 3 3 2 3 3 3
110/300 Flush
111/300 Write block 4693 len 20480 data: 4 4 4 5 4
112/300 Flush
113/300 Flush
114/300 Write block 1028 len 36864 data: 3 4 4 4 3 3 4 4 5
115/300 Write block 858 len 28672 data: 8 7 7 6 6 7 5
116/300 Write block 1235 len 16384 data: 3 2 2 2
117/300 Write block 3155 len 32768 data: 4 3 3 3 3 3 3 3
118/300 Write block 414 len 8192 data: 4 4
119/300 Read block 3625 len 36864
120/300 Write block 3624 len 36864 data: 4 4 3 3 3 2 2 2 2
121/300 Read block 3126 len 20480
122/300 Write block 2731 len 24576 data: 4 5 5 3 3 4
123/300 Write block 3951 len 4096 data: 4
124/300 Read block 1359 len 40960
125/300 Write block 2497 len 12288 data: 4 5 6
126/300 Read block 2426 len 28672
127/300 Read block 4038 len 20480
128/300 Read block 1579 len 20480
129/300 Read block 3280 len 32768
130/300 Flush
131/300 Write block 2235 len 4096 data: 2
132/300 Flush
133/300 Write block 4788 len 12288 data: 3 3 2
134/300 Read block 3906 len 36864
135/300 Write block 4855 len 40960 data: 2 2 2 2 3 3 3 3 3 4
136/300 Read block 1605 len 16384
137/300 Write block 4455 len 12288 data: 3 3 3
138/300 Read block 3805 len 4096
139/300 Read block 3158 len 32768
140/300 Read block 1553 len 8192
141/300 Write block 4876 len 36864 data: 4 5 5 5 4 4 3 3 3
142/300 Read block 4667 len 40960
143/300 Read block 4135 len 32768
144/300 Flush
145/300 Read block 1129 len 16384
146/300 Read block 3032 len 16384
147/300 Read block 2181 len 16384
148/300 Flush
149/300 Write block 78 len 4096 data: 3
150/300 Read block 4217 len 24576
151/300 Read block 1201 len 36864
152/300 Write block 4233 len 28672 data: 3 3 3 3 2 3 4
153/300 Read block 4231 len 40960
154/300 Read block 3625 len 40960
155/300 Read block 3311 len 20480
156/300 Read block 4083 len 4096
157/300 Read block 2699 len 8192
158/300 Read block 3725 len 40960
159/300 Write block 2800 len 40960 data: 4 3 3 3 3 3 3 4 4 4
160/300 Read block 3894 len 4096
161/300 Flush
162/300 Flush
163/300 Write block 3296 len 20480 data: 4 4 4 3 3
164/300 Write block 344 len 32768 data: 3 4 5 6 7 7 7 9
165/300 Read block 3514 len 28672
166/300 Write block 3724 len 28672 data: 2 2 2 2 3 3 3
167/300 Read block 2491 len 20480
168/300 Read block 3673 len 20480
169/300 Write block 951 len 4096 data: 3
170/300 Read block 2209 len 28672
171/300 Write block 1697 len 8192 data: 2 3
172/300 Write block 4727 len 4096 data: 6
173/300 Read block 782 len 36864
174/300 Read block 2533 len 24576
175/300 Read block 2654 len 36864
176/300 Read block 2119 len 32768
177/300 Write block 4827 len 28672 data: 4 4 4 3 5 4 5
178/300 Flush
179/300 Read block 2522 len 32768
180/300 Write block 2179 len 28672 data: 6 5 5 5 5 5 4
181/300 Write block 2914 len 16384 data: 6 7 7 6
182/300 Flush
183/300 Read block 2023 len 20480
184/300 Write block 3484 len 4096 data: 4
185/300 Read block 4363 len 36864
186/300 Read block 2272 len 4096
187/300 Write block 1434 len 20480 data: 4 4 4 4 3
188/300 Write block 4350 len 40960 data: 3 3 4 4 3 3 3 4 4 5
189/300 Read block 3378 len 4096
190/300 Read block 88 len 12288
191/300 Write block 4462 len 40960 data: 5 4 4 5 5 4 4 4 4 3
192/300 Flush
193/300 Read block 1525 len 24576
194/300 Read block 4554 len 24576
195/300 Flush
196/300 Flush
197/300 Read block 382 len 4096
198/300 Read block 324 len 36864
199/300 Flush
200/300 Write block 2853 len 32768 data: 5 5 6 5 5 5 5 4
201/300 Read block 735 len 28672
202/300 Write block 3392 len 12288 data: 3 3 3
203/300 Write block 4795 len 36864 data: 5 6 6 6 6 5 4 4 5
204/300 Write block 998 len 12288 data: 5 5 4
205/300 Read block 1462 len 24576
206/300 Write block 1360 len 4096 data: 6
207/300 Write block 4881 len 4096 data: 5
208/300 Read block 662 len 4096
209/300 Read block 1216 len 8192
210/300 Read block 1104 len 28672
211/300 Read block 1857 len 4096
212/300 Read block 3888 len 20480
213/300 Read block 2993 len 8192
214/300 Read block 1733 len 16384
215/300 Read block 878 len 40960
216/300 Flush
217/300 Read block 1494 len 28672
218/300 Read block 2225 len 8192
219/300 Read block 553 len 20480
220/300 Read block 3440 len 32768
221/300 Write block 4391 len 20480 data: 4 3 3 4 3
222/300 Read block 3665 len 4096
223/300 Read block 3236 len 20480
224/300 Write block 1533 len 24576 data: 4 4 3 4 4 2
225/300 Read block 1998 len 28672
226/300 Flush
227/300 Read block 676 len 28672
228/300 Read block 2678 len 4096
229/300 Write block 2706 len 32768 data: 3 3 4 5 5 5 5 6
230/300 Read block 1798 len 8192
231/300 Write block 3179 len 8192 data: 3 4
232/300 Read block 3278 len 28672
233/300 Write block 2673 len 8192 data: 3 3
234/300 Flush
235/300 Write block 4927 len 12288 data: 4 4 4
236/300 Write block 1355 len 32768 data: 6 8 6 5 6 7 6 5
237/300 Write block 3369 len 36864 data: 8 7 5 6 7 6 5 4 4
238/300 Write block 604 len 8192 data: 5 5
239/300 Write block 4730 len 8192 data: 7 6
240/300 Read block 4625 len 24576
241/300 Read block 3233 len 32768
242/300 Read block 4735 len 8192
243/300 Read block 3722 len 32768
244/300 Read block 1953 len 8192
245/300 Write block 499 len 32768 data: 4 3 3 3 4 3 3 3
246/300 Write block 3325 len 16384 data: 6 5 5 4
247/300 Flush
248/300 Read block 624 len 40960
249/300 Write block 3109 len 16384 data: 5 5 5 3
250/300 Read block 3837 len 4096
251/300 Flush
252/300 Read block 341 len 36864
253/300 Write block 986 len 36864 data: 6 6 4 4 3 6 7 5 6
254/300 Write block 3549 len 12288 data: 4 4 3
255/300 Write block 473 len 4096 data: 3
256/300 Write block 442 len 20480 data: 4 4 4 4 4
257/300 Write block 2689 len 28672 data: 3 2 2 2 2 2 2
258/300 Read block 1129 len 32768
259/300 Write block 1874 len 36864 data: 3 3 3 3 3 3 3 3 3
260/300 Read block 1663 len 24576
261/300 Read block 2107 len 8192
262/300 Write block 3353 len 28672 data: 4 4 4 4 6 6 5
263/300 Read block 4668 len 28672
264/300 Write block 4792 len 24576 data: 2 3 4 6 7 7
265/300 Read block 2495 len 36864
266/300 Read block 221 len 40960
267/300 Read block 3128 len 20480
268/300 Write block 4544 len 16384 data: 3 4 2 4
269/300 Read block 4742 len 4096
270/300 Read block 1623 len 40960
271/300 Read block 4926 len 32768
272/300 Read block 4800 len 24576
273/300 Write block 2716 len 40960 data: 2 3 4 4 4 4 4 4 4 4
274/300 Read block 4067 len 32768
275/300 Read block 4787 len 32768
276/300 Read block 4161 len 8192
277/300 Write block 4809 len 16384 data: 3 3 2 3
278/300 Read block 4825 len 4096
279/300 Read block 3838 len 4096
280/300 Write block 3393 len 4096 data: 4
281/300 Read block 2507 len 8192
282/300 Read block 4655 len 24576
283/300 Write block 3752 len 36864 data: 4 4 4 3 3 2 3 3 3
284/300 Write block 1075 len 12288 data: 4 4 5
285/300 Write block 635 len 16384 data: 4 4 4 4
286/300 Read block 1830 len 4096
287/300 Write block 4409 len 8192 data: 3 3
288/300 Read block 1200 len 20480
289/300 Read block 2269 len 4096
290/300 Write block 4440 len 28672 data: 4 5 5 5 5 4 3
291/300 Flush
292/300 Read block 340 len 20480
293/300 Read block 1491 len 36864
294/300 Read block 1755 len 12288
295/300 Read block 3837 len 16384
296/300 Read block 3845 len 36864
297/300 Read block 117 len 40960
298/300 Write block 1493 len 32768 data: 4 5 6 6 4 4 4 4
299/300 Write block 741 len 28672 data: 6 4 4 6 6 6 7
300/300 Write block 4984 len 32768 data: 2 4 4 5 4 4 4 4
Main task has work to do, go find it
got dsc Start(1)
start 1
Feb 13 21:42:35.965 INFO request completed, latency_us: 220, response_code: 204, uri: /start/cid/1, method: POST, req_id: 1d4ea4e3-1f2f-4b64-9c8e-9c317f203eeb, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
States: Active Offline Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
4824 Acked 5823 Read 3 Done New Done true
4825 Acked 5824 Read 8 Done New Done true
4826 Acked 5825 Write 9 Done New Done true
4827 Acked 5826 Write 7 Done New Done true
4828 Acked 5827 Write 1 Done New Done true
4829 Acked 5828 Write 9 Done New Done false
4830 Acked 5829 Write 3 Done New Done false
4831 Acked 5830 Read 8 Done New Done false
4832 Acked 5831 Write 7 Done New Done false
4833 Acked 5832 Read 6 Done New Done false
4834 Acked 5833 Read 8 Done New Done false
4835 Acked 5834 Write 4 Done New Done false
4836 Acked 5835 Write 3 Done New Done false
4837 Acked 5836 Write 9 Done New Done false
4838 Acked 5837 Write 7 Done New Done false
4839 Acked 5838 Read 5 Done New Done false
4840 Acked 5839 Write 9 Done New Done false
4841 Acked 5840 Write 3 Done New Done false
4842 Acked 5841 Read 8 Done New Done false
4843 Acked 5842 Read 7 Done New Done false
4844 Acked 5843 Write 1 Done New Done false
4845 Acked 5844 Read 2 Done New Done false
4846 Acked 5845 Write 3 Done New Done false
4847 Acked 5846 Write 2 Done New Done false
4848 Acked 5847 Write 2 Done New Done false
4849 Acked 5848 Write 1 Done New Done false
4850 Acked 5849 Read 1 Done New Done false
4851 Acked 5850 Write 7 Done New Done false
4852 Acked 5851 Flush 0 Done New Done false
4853 Acked 5852 Read 2 Done New Done false
4854 Acked 5853 Write 6 Done New Done false
4855 Acked 5854 Write 1 Done New Done false
4856 Acked 5855 Read 7 Done New Done false
4857 Acked 5856 Read 6 Done New Done false
4858 Acked 5857 Read 6 Done New Done false
4859 Acked 5858 Flush 0 Done New Done false
4860 Acked 5859 Flush 0 Done New Done false
4861 Acked 5860 Read 10 Done New Done false
4862 Acked 5861 Write 6 Done New Done false
4863 Acked 5862 Flush 0 Done New Done false
4864 Acked 5863 Write 8 Done New Done false
4865 Acked 5864 Read 5 Done New Done false
4866 Acked 5865 Read 9 Done New Done false
4867 Acked 5866 Flush 0 Done New Done false
4868 Acked 5867 Write 9 Done New Done false
4869 Acked 5868 Read 1 Done New Done false
4870 Acked 5869 Write 6 Done New Done false
4871 Acked 5870 Write 9 Done New Done false
4872 Acked 5871 Write 5 Done New Done false
4873 Acked 5872 Read 6 Done New Done false
4874 Acked 5873 Write 6 Done New Done false
4875 Acked 5874 Write 8 Done New Done false
4876 Acked 5875 Read 1 Done New Done false
4877 Acked 5876 Read 1 Done New Done false
4878 Acked 5877 Write 4 Done New Done false
4879 Acked 5878 Read 8 Done New Done false
4880 Acked 5879 Read 8 Done New Done false
4881 Acked 5880 Write 10 Done New Done false
4882 Acked 5881 Write 8 Done New Done false
4883 Acked 5882 Read 10 Done New Done false
4884 Acked 5883 Read 6 Done New Done false
4885 Acked 5884 Write 3 Done New Done false
4886 Acked 5885 Read 3 Done New Done false
4887 Acked 5886 Read 1 Done New Done false
4888 Acked 5887 Write 6 Done New Done false
4889 Acked 5888 Read 10 Done New Done false
4890 Acked 5889 Read 6 Done New Done false
4891 Acked 5890 Read 7 Done New Done false
4892 Acked 5891 Read 6 Done New Done false
4893 Acked 5892 Read 5 Done New Done false
4894 Acked 5893 Flush 0 Done New Done false
4895 Acked 5894 Write 3 Done New Done false
4896 Acked 5895 Flush 0 Done New Done false
4897 Acked 5896 Write 2 Done New Done false
4898 Acked 5897 Write 4 Done New Done false
4899 Acked 5898 Read 10 Done New Done false
4900 Acked 5899 Read 5 Done New Done false
4901 Acked 5900 Write 4 Done New Done false
4902 Acked 5901 Write 5 Done New Done false
4903 Acked 5902 Write 2 Done New Done false
4904 Acked 5903 Write 10 Done New Done false
4905 Acked 5904 Read 10 Done New Done false
4906 Acked 5905 Write 7 Done New Done false
4907 Acked 5906 Read 6 Done New Done false
4908 Acked 5907 Read 1 Done New Done false
4909 Acked 5908 Write 4 Done New Done false
4910 Acked 5909 Read 4 Done New Done false
4911 Acked 5910 Write 9 Done New Done false
4912 Acked 5911 Read 4 Done New Done false
4913 Acked 5912 Read 7 Done New Done false
4914 Acked 5913 Write 10 Done New Done false
4915 Acked 5914 Write 9 Done New Done false
4916 Acked 5915 Read 7 Done New Done false
4917 Acked 5916 Read 8 Done New Done false
4918 Acked 5917 Flush 0 Done New Done false
4919 Acked 5918 Read 5 Done New Done false
4920 Acked 5919 Read 10 Done New Done false
4921 Acked 5920 Read 8 Done New Done false
4922 Acked 5921 Read 8 Done New Done false
4923 Acked 5922 Write 10 Done New Done false
4924 Acked 5923 Write 2 Done New Done false
4925 Acked 5924 Write 6 Done New Done false
4926 Acked 5925 Write 1 Done New Done false
4927 Acked 5926 Write 8 Done New Done false
4928 Acked 5927 Flush 0 Done New Done false
4929 Acked 5928 Flush 0 Done New Done false
4930 Acked 5929 Write 3 Done New Done false
4931 Acked 5930 Read 9 Done New Done false
4932 Acked 5931 Write 8 Done New Done false
4933 Acked 5932 Flush 0 Done New Done false
4934 Acked 5933 Write 5 Done New Done false
4935 Acked 5934 Flush 0 Done New Done false
4936 Acked 5935 Flush 0 Done New Done false
4937 Acked 5936 Write 9 Done New Done false
4938 Acked 5937 Write 7 Done New Done false
4939 Acked 5938 Write 4 Done New Done false
4940 Acked 5939 Write 8 Done New Done false
4941 Acked 5940 Write 2 Done New Done false
4942 Acked 5941 Read 9 Done New Done false
4943 Acked 5942 Write 9 Done New Done false
4944 Acked 5943 Read 5 Done New Done false
4945 Acked 5944 Write 6 Done New Done false
4946 Acked 5945 Write 1 Done New Done false
4947 Acked 5946 Read 10 Done New Done false
4948 Acked 5947 Write 3 Done New Done false
4949 Acked 5948 Read 7 Done New Done false
4950 Acked 5949 Read 5 Done New Done false
4951 Acked 5950 Read 5 Done New Done false
4952 Acked 5951 Read 8 Done New Done false
4953 Acked 5952 Flush 0 Done New Done false
4954 Acked 5953 Write 1 Done New Done false
4955 Acked 5954 Flush 0 Done New Done false
4956 Acked 5955 Write 3 Done New Done false
4957 Acked 5956 Read 9 Done New Done false
4958 Acked 5957 Write 10 Done New Done false
4959 Acked 5958 Read 4 Done New Done false
4960 Acked 5959 Write 3 Done New Done false
4961 Acked 5960 Read 1 Done New Done false
4962 Acked 5961 Read 8 Done New Done false
4963 Acked 5962 Read 2 Done New Done false
4964 Acked 5963 Write 9 Done New Done false
4965 Acked 5964 Read 10 Done New Done false
4966 Acked 5965 Read 8 Done New Done false
4967 Acked 5966 Flush 0 Done New Done false
4968 Acked 5967 Read 4 Done New Done false
4969 Acked 5968 Read 4 Done New Done false
4970 Acked 5969 Read 4 Done New Done false
4971 Acked 5970 Flush 0 Done New Done false
4972 Acked 5971 Write 1 Done New Done false
4973 Acked 5972 Read 6 Done New Done false
4974 Acked 5973 Read 9 Done New Done false
4975 Acked 5974 Write 7 Done New Done false
4976 Acked 5975 Read 10 Done New Done false
4977 Acked 5976 Read 10 Done New Done false
4978 Acked 5977 Read 5 Done New Done false
4979 Acked 5978 Read 1 Done New Done false
4980 Acked 5979 Read 2 Done New Done false
4981 Acked 5980 Read 10 Done New Done false
4982 Acked 5981 Write 10 Done New Done false
4983 Acked 5982 Read 1 Done New Done false
4984 Acked 5983 Flush 0 Done New Done false
4985 Acked 5984 Flush 0 Done New Done false
4986 Acked 5985 Write 5 Done New Done false
4987 Acked 5986 Write 8 Done New Done false
4988 Acked 5987 Read 7 Done New Done false
4989 Acked 5988 Write 7 Done New Done false
4990 Acked 5989 Read 5 Done New Done false
4991 Acked 5990 Read 5 Done New Done false
4992 Acked 5991 Write 1 Done New Done false
4993 Acked 5992 Read 7 Done New Done false
4994 Acked 5993 Write 2 Done New Done false
4995 Acked 5994 Write 1 Done New Done false
4996 Acked 5995 Read 9 Done New Done false
4997 Acked 5996 Read 6 Done New Done false
4998 Acked 5997 Read 9 Done New Done false
4999 Acked 5998 Read 8 Done New Done false
5000 Acked 5999 Write 7 Done New Done false
5001 Acked 6000 Flush 0 Done New Done false
5002 Acked 6001 Read 8 Done New Done false
5003 Acked 6002 Write 7 Done New Done false
5004 Acked 6003 Write 4 Done New Done false
5005 Acked 6004 Flush 0 Done New Done false
5006 Acked 6005 Read 5 Done New Done false
5007 Acked 6006 Write 1 Done New Done false
5008 Acked 6007 Read 9 Done New Done false
5009 Acked 6008 Read 1 Done New Done false
5010 Acked 6009 Write 5 Done New Done false
5011 Acked 6010 Write 10 Done New Done false
5012 Acked 6011 Read 1 Done New Done false
5013 Acked 6012 Read 3 Done New Done false
5014 Acked 6013 Write 10 Done New Done false
5015 Acked 6014 Flush 0 Done New Done false
5016 Acked 6015 Read 6 Done New Done false
5017 Acked 6016 Read 6 Done New Done false
5018 Acked 6017 Flush 0 Done New Done false
5019 Acked 6018 Flush 0 Done New Done false
5020 Acked 6019 Read 1 Done New Done false
5021 Acked 6020 Read 9 Done New Done false
5022 Acked 6021 Flush 0 Done New Done false
5023 Acked 6022 Write 8 Done New Done false
5024 Acked 6023 Read 7 Done New Done false
5025 Acked 6024 Write 3 Done New Done false
5026 Acked 6025 Write 9 Done New Done false
5027 Acked 6026 Write 3 Done New Done false
5028 Acked 6027 Read 6 Done New Done false
5029 Acked 6028 Write 1 Done New Done false
5030 Acked 6029 Write 1 Done New Done false
5031 Acked 6030 Read 1 Done New Done false
5032 Acked 6031 Read 2 Done New Done false
5033 Acked 6032 Read 7 Done New Done false
5034 Acked 6033 Read 1 Done New Done false
5035 Acked 6034 Read 5 Done New Done false
5036 Acked 6035 Read 2 Done New Done false
5037 Acked 6036 Read 4 Done New Done false
5038 Acked 6037 Read 10 Done New Done false
5039 Acked 6038 Flush 0 Done New Done false
5040 Acked 6039 Read 7 Done New Done false
5041 Acked 6040 Read 2 Done New Done false
5042 Acked 6041 Read 5 Done New Done false
5043 Acked 6042 Read 8 Done New Done false
5044 Acked 6043 Write 5 Done New Done false
5045 Acked 6044 Read 1 Done New Done false
5046 Acked 6045 Read 5 Done New Done false
5047 Acked 6046 Write 6 Done New Done false
5048 Acked 6047 Read 7 Done New Done false
5049 Acked 6048 Flush 0 Done New Done false
5050 Acked 6049 Read 7 Done New Done false
5051 Acked 6050 Read 1 Done New Done false
5052 Acked 6051 Write 8 Done New Done false
5053 Acked 6052 Read 2 Done New Done false
5054 Acked 6053 Write 2 Done New Done false
5055 Acked 6054 Read 7 Done New Done false
5056 Acked 6055 Write 2 Done New Done false
5057 Acked 6056 Flush 0 Done New Done false
5058 Acked 6057 Write 3 Done New Done false
5059 Acked 6058 Write 8 Done New Done false
5060 Acked 6059 Write 9 Done New Done false
5061 Acked 6060 Write 2 Done New Done false
5062 Acked 6061 Write 2 Done New Done false
5063 Acked 6062 Read 6 Done New Done false
5064 Acked 6063 Read 8 Done New Done false
5065 Acked 6064 Read 2 Done New Done false
5066 Acked 6065 Read 8 Done New Done false
5067 Acked 6066 Read 2 Done New Done false
5068 Acked 6067 Write 8 Done New Done false
5069 Acked 6068 Write 4 Done New Done false
5070 Acked 6069 Flush 0 Done New Done false
5071 Acked 6070 Read 10 Done New Done false
5072 Acked 6071 Write 4 Done New Done false
5073 Acked 6072 Read 1 Done New Done false
5074 Acked 6073 Flush 0 Done New Done false
5075 Acked 6074 Read 9 Done New Done false
5076 Acked 6075 Write 9 Done New Done false
5077 Acked 6076 Write 3 Done New Done false
5078 Acked 6077 Write 1 Done New Done false
5079 Acked 6078 Write 5 Done New Done false
5080 Acked 6079 Write 7 Done New Done false
5081 Acked 6080 Read 8 Done New Done false
5082 Acked 6081 Write 9 Done New Done false
5083 Acked 6082 Read 6 Done New Done false
5084 Acked 6083 Read 2 Done New Done false
5085 Acked 6084 Write 7 Done New Done false
5086 Acked 6085 Read 7 Done New Done false
5087 Acked 6086 Write 6 Done New Done false
5088 Acked 6087 Read 9 Done New Done false
5089 Acked 6088 Read 10 Done New Done false
5090 Acked 6089 Read 5 Done New Done false
5091 Acked 6090 Write 4 Done New Done false
5092 Acked 6091 Read 1 Done New Done false
5093 Acked 6092 Read 10 Done New Done false
5094 Acked 6093 Read 8 Done New Done false
5095 Acked 6094 Read 6 Done New Done false
5096 Acked 6095 Write 10 Done New Done false
5097 Acked 6096 Read 8 Done New Done false
5098 Acked 6097 Read 8 Done New Done false
5099 Acked 6098 Read 2 Done New Done false
5100 Acked 6099 Write 4 Done New Done false
5101 Acked 6100 Read 1 Done New Done false
5102 Acked 6101 Read 1 Done New Done false
5103 Acked 6102 Write 1 Done New Done false
5104 Acked 6103 Read 2 Done New Done false
5105 Acked 6104 Read 6 Done New Done false
5106 Acked 6105 Write 9 Done New Done false
5107 Acked 6106 Write 3 Done New Done false
5108 Acked 6107 Write 4 Done New Done false
5109 Acked 6108 Read 1 Done New Done false
5110 Acked 6109 Write 2 Done New Done false
5111 Acked 6110 Read 5 Done New Done false
5112 Acked 6111 Read 1 Done New Done false
5113 Acked 6112 Write 7 Done New Done false
5114 Acked 6113 Flush 0 Done New Done false
5115 Acked 6114 Read 5 Done New Done false
5116 Acked 6115 Read 9 Done New Done false
5117 Acked 6116 Read 3 Done New Done false
5118 Acked 6117 Read 4 Done New Done false
5119 Acked 6118 Read 9 Done New Done false
5120 Acked 6119 Read 10 Done New Done false
5121 Acked 6120 Write 8 Done New Done false
5122 Acked 6121 Write 7 Done New Done false
5123 Acked 6122 Write 8 Sent New Sent false
STATES DS:0 DS:1 DS:2 TOTAL
New 0 300 0 300
Sent 1 0 1 2
Done 299 0 299 598
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 6113 5822 6113
Downstairs last five completed: 5822 5821 5820 5819 5818
Upstairs last five completed: 5123 5122 5121 5120 5119
CLIENT: Up:0 ds:300 act:2
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:36.161933431Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:36.162022288Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:36.162045877Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:36.162066816Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:36.162083155Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 300 jobs since flush 5822","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:36.162101265Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:36.16221831Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:37.162641743Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:37.162697461Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:37.16271589Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:37.16273315Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:37.162745899Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 301 jobs since flush 5822","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:37.162769488Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:37.162933812Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1181)
[8820][1] reports Running
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:38.163552277Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:38.163620665Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:38.164910805Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:38.164939564Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 5822","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:38.164955414Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 5822","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:38.165133437Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:38.165158846Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:38.165173825Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:38.165188075Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:38.165202454Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 6123 6122 6121 6120 6119
Upstairs last five completed: 5124 5123 5122 5121 5120
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Feb 13 21:42:39.967 INFO request completed, latency_us: 374, response_code: 204[8810] Got stop action so:false kr:false
, uri: /stop/cid/0, method: POST, req_id: dd6410c0-9522-4075-9013-b7c1a2ee17da, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:39.968577744Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:39.968644732Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:42:39.968 {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:39.968667501Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
INFO request completed{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:39.96868386Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
, latency_us: 118, response_code: 200, uri: /state/cid/0, method: GET, req_id: fa49ce58-239d-45c1-8792-055238015656, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"[0] client re-new 0 jobs since flush 6123","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:39.968758897Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:39.968787536Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:40.969492348Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:40.969554285Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:40.969577124Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:40.969598004Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:40.969614053Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 6123","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:40.969631432Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:40.969648482Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:41.971140333Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:41.97120534Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:41.971228399Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:41.971249118Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:41.971265008Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 6123","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:41.971283687Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:41.971301176Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:42.972932683Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:42.9730106Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:42.973032129Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:42.973049808Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:42.973062648Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 6123","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:42.973082257Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:42.973114006Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:42:43.970 INFO request completed, latency_us: 209, response_code: 200, uri: /state/cid/0, method: GET, req_id: 596f67e4-8588-4d6e-a534-45f8c54a8acf, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Write block 2665 len 12288 data: 6 5 4
002/300 Write block 3610 len 12288 data: 5 6 6
003/300 Write block 4440 len 12288 data: 5 6 6
004/300 Read block 2009 len 36864
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:43.973387494Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:43.973435272Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:43.973453671Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:43.97347061Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:43.97348346Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 4 jobs since flush 6123","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:43.973497229Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:43.973511129Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
005/300 Flush
006/300 Read block 1996 len 36864
007/300 Read block 348 len 28672
008/300 Write block 4625 len 12288 data: 6 4 5
009/300 Read block 3236 len 8192
010/300 Read block 374 len 4096
011/300 Read block 65 len 40960
012/300 Read block 3618 len 24576
013/300 Write block 1960 len 28672 data: 3 3 5 4 4 7 6
014/300 Write block 1208 len 32768 data: 7 7 6 5 4 4 4 4
015/300 Write block 1371 len 16384 data: 5 4 4 3
016/300 Write block 100 len 40960 data: 2 3 2 3 3 5 5 5 4 4
017/300 Write block 763 len 28672 data: 3 4 4 3 4 3 2
018/300 Write block 4724 len 12288 data: 7 7 7
019/300 Write block 1408 len 40960 data: 4 4 4 4 4 4 3 3 2 2
020/300 Read block 57 len 28672
021/300 Read block 501 len 20480
022/300 Read block 3477 len 12288
023/300 Read block 449 len 36864
024/300 Write block 3112 len 32768 data: 4 3 3 3 2 3 3 5
025/300 Write block 3093 len 20480 data: 4 5 3 3 5
026/300 Write block 24 len 16384 data: 3 3 3 3
027/300 Read block 2759 len 16384
028/300 Read block 3123 len 28672
029/300 Read block 1358 len 32768
030/300 Write block 4259 len 8192 data: 3 4
031/300 Flush
032/300 Read block 439 len 24576
033/300 Read block 3842 len 8192
034/300 Write block 3802 len 12288 data: 3 3 3
035/300 Write block 1064 len 8192 data: 6 6
036/300 Read block 1746 len 36864
037/300 Read block 1927 len 32768
038/300 Write block 999 len 4096 data: 6
039/300 Write block 3505 len 12288 data: 4 4 4
040/300 Write block 3645 len 20480 data: 5 5 6 5 5
041/300 Read block 3039 len 32768
042/300 Read block 4103 len 36864
043/300 Write block 2875 len 8192 data: 4 4
044/300 Write block 2930 len 32768 data: 3 4 3 3 3 3 3 4
045/300 Flush
046/300 Read block 512 len 12288
047/300 Read block 2255 len 12288
048/300 Write block 3124 len 8192 data: 7 6
049/300 Read block 824 len 8192
050/300 Read block 3288 len 8192
051/300 Flush
052/300 Write block 3303 len 28672 data: 2 4 4 5 4 4 7
053/300 Write block 1032 len 24576 data: 4 4 5 5 6 5
054/300 Write block 4636 len 36864 data: 5 5 4 4 3 3 4 2 2
055/300 Read block 1643 len 20480
056/300 Read block 4214 len 36864
057/300 Write block 2783 len 24576 data: 4 4 3 3 3 3
058/300 Read block 4855 len 24576
059/300 Read block 2466 len 28672
060/300 Write block 2136 len 36864 data: 5 5 5 5 5 5 5 4 3
061/300 Write block 3867 len 24576 data: 5 3 4 4 4 3
062/300 Read block 3136 len 28672
063/300 Read block 4847 len 40960
064/300 Read block 4154 len 36864
065/300 Read block 383 len 12288
066/300 Flush
067/300 Flush
068/300 Write block 2506 len 28672 data: 7 7 6 6 5 5 6
069/300 Read block 286 len 4096
070/300 Read block 3416 len 24576
071/300 Flush
072/300 Read block 4533 len 24576
073/300 Read block 3195 len 24576
074/300 Flush
075/300 Read block 819 len 20480
076/300 Flush
077/300 Read block 1208 len 32768
078/300 Write block 675 len 4096 data: 3
079/300 Read block 201 len 4096
080/300 Flush
081/300 Read block 265 len 32768
082/300 Write block 1640 len 12288 data: 6 6 6
083/300 Flush
084/300 Write block 3426 len 40960 data: 3 4 4 5 6 5 5 4 6 6
085/300 Write block 2705 len 4096 data: 3
086/300 Write block 2973 len 24576 data: 2 3 3 4 4 4
087/300 Write block 1856 len 32768 data: 3 2 2 2 2 2 2 3
088/300 Read block 2974 len 32768
089/300 Read block 4427 len 40960
090/300 Write block 2896 len 12288 data: 7 5 3
091/300 Read block 882 len 40960
092/300 Read block 4008 len 12288
093/300 Flush
094/300 Write block 4407 len 20480 data: 6 5 4 4 3
095/300 Write block 3722 len 4096 data: 2
096/300 Read block 1047 len 32768
097/300 Read block 187 len 24576
098/300 Write block 3797 len 40960 data: 5 5 4 4 4 4 4 4 4 4
099/300 Read block 3071 len 36864
100/300 Read block 2423 len 24576
101/300 Write block 470 len 16384 data: 4 3 3 4
102/300 Read block 4215 len 4096
103/300 Write block 2209 len 16384 data: 4 5 6 5
104/300 Read block 1949 len 32768
105/300 Read block 1593 len 32768
106/300 Read block 2796 len 12288
107/300 Write block 805 len 24576 data: 4 4 3 3 3 3
108/300 Write block 287 len 4096 data: 4
109/300 Flush
110/300 Read block 701 len 8192
111/300 Write block 3909 len 36864 data: 4 4 4 3 4 4 5 4 5
112/300 Flush
113/300 Write block 930 len 32768 data: 2 2 2 2 2 3 3 3
114/300 Flush
115/300 Read block 4330 len 36864
116/300 Flush
117/300 Write block 2505 len 24576 data: 7 8 8 7 7 6
118/300 Read block 2409 len 20480
119/300 Read block 1467 len 20480
120/300 Flush
121/300 Write block 4875 len 12288 data: 3 5 6
122/300 Write block 140 len 32768 data: 4 5 4 3 3 2 3 3
123/300 Read block 4165 len 40960
124/300 Write block 670 len 28672 data: 4 3 3 2 3 4 3
125/300 Flush
126/300 Write block 1111 len 12288 data: 7 7 3
127/300 Read block 3148 len 16384
128/300 Write block 637 len 32768 data: 5 5 7 7 9 8 7 8
129/300 Read block 2807 len 40960
130/300 Flush
131/300 Read block 4389 len 36864
132/300 Write block 3041 len 16384 data: 4 5 5 5
133/300 Read block 2091 len 16384
134/300 Read block 3665 len 28672
135/300 Write block 4358 len 40960 data: 5 6 5 4 4 4 2 3 4 4
136/300 Read block 2016 len 28672
137/300 Write block 4480 len 16384 data: 3 3 2 2
138/300 Write block 406 len 20480 data: 3 3 3 4 3
139/300 Read block 673 len 12288
140/300 Flush
141/300 Read block 2230 len 12288
142/300 Read block 498 len 32768
143/300 Write block 3819 len 40960 data: 5 5 6 6 6 5 6 6 5 6
144/300 Read block 3050 len 28672
145/300 Read block 4406 len 20480
146/300 Write block 3720 len 36864 data: 3 3 3 2 3 3 3 3 4
147/300 Write block 169 len 36864 data: 6 5 5 6 5 4 4 4 3
148/300 Read block 488 len 36864
149/300 Write block 4106 len 40960 data: 5 4 3 3 4 5 5 7 7 7
150/300 Write block 2268 len 24576 data: 2 3 3 3 3 2
151/300 Write block 4498 len 28672 data: 3 3 5 5 6 7 6
152/300 Read block 3653 len 40960
153/300 Write block 3370 len 20480 data: 8 6 7 8 7
154/300 Read block 4220 len 16384
155/300 Read block 328 len 28672
156/300 Write block 1302 len 24576 data: 7 6 6 3 6 5
157/300 Write block 4668 len 32768 data: 7 7 7 7 5 3 3 3
158/300 Write block 2479 len 16384 data: 3 3 2 4
159/300 Write block 3885 len 20480 data: 5 6 4 4 3
160/300 Read block 183 len 28672
161/300 Read block 1722 len 20480
162/300 Read block 4908 len 28672
163/300 Write block 4488 len 28672 data: 9 9 7 6 5 3 4
164/300 Read block 796 len 20480
165/300 Read block 1886 len 4096
166/300 Write block 795 len 12288 data: 4 4 5
167/300 Write block 3520 len 20480 data: 4 4 3 3 3
168/300 Read block 2055 len 20480
169/300 Read block 3650 len 28672
170/300 Read block 3656 len 8192
171/300 Read block 1409 len 16384
172/300 Read block 3753 len 4096
173/300 Read block 4120 len 24576
174/300 Write block 3661 len 24576 data: 3 5 5 5 6 5
175/300 Write block 448 len 40960 data: 5 4 3 4 4 5 4 4 4 4
176/300 Read block 612 len 24576
177/300 Read block 3905 len 12288
178/300 Read block 3295 len 4096
179/300 Read block 4257 len 12288
180/300 Write block 864 len 8192 data: 6 7
181/300 Write block 1464 len 32768 data: 5 6 5 6 7 4 4 3
182/300 Read block 1744 len 16384
183/300 Write block 441 len 16384 data: 4 5 5 5
184/300 Flush
185/300 Write block 4147 len 36864 data: 2 4 4 4 5 4 4 5 6
186/300 Read block 2525 len 16384
187/300 Write block 1283 len 16384 data: 3 3 3 3
188/300 Read block 419 len 28672
189/300 Write block 4496 len 16384 data: 4 3 4 4
190/300 Read block 2779 len 40960
191/300 Write block 3888 len 36864 data: 5 4 3 3 2 2 4 6 6
192/300 Write block 303 len 24576 data: 5 6 6 6 6 6
193/300 Flush
194/300 Write block 3602 len 24576 data: 4 4 5 4 4 4
195/300 Write block 3512 len 24576 data: 5 5 6 6 6 7
196/300 Flush
197/300 Read block 1761 len 28672
198/300 Write block 2848 len 28672 data: 6 4 4 4 6 6 6
199/300 Write block 172 len 32768 data: 7 6 5 5 5 4 3 3
200/300 Write block 4463 len 20480 data: 5 5 6 6 5
201/300 Read block 2737 len 24576
202/300 Flush
203/300 Read block 3517 len 20480
204/300 Read block 2553 len 32768
205/300 Read block 389 len 16384
206/300 Read block 2518 len 32768
207/300 Read block 660 len 36864
208/300 Read block 1825 len 8192
209/300 Read block 1526 len 40960
210/300 Read block 3949 len 12288
211/300 Write block 1533 len 12288 data: 5 5 4
212/300 Write block 2244 len 16384 data: 6 5 4 4
213/300 Write block 1601 len 24576 data: 4 5 5 3 4 4
214/300 Read block 3748 len 20480
215/300 Write block 384 len 20480 data: 7 7 7 4 5
216/300 Write block 939 len 28672 data: 2 2 2 2 3 3 3
217/300 Read block 3233 len 20480
218/300 Read block 2732 len 20480
219/300 Write block 4688 len 40960 data: 3 2 2 4 4 5 5 5 6 5
220/300 Write block 90 len 28672 data: 6 6 7 6 5 5 5
221/300 Read block 1422 len 32768
222/300 Read block 3990 len 12288
223/300 Write block 1411 len 28672 data: 5 5 5 4 4 3 3
224/300 Read block 721 len 12288
225/300 Read block 186 len 20480
226/300 Write block 1160 len 28672 data: 4 3 4 4 5 5 4
227/300 Write block 101 len 20480 data: 4 3 4 4 6
228/300 Read block 2284 len 24576
229/300 Read block 2800 len 16384
230/300 Write block 1447 len 8192 data: 3 3
231/300 Read block 4123 len 4096
232/300 Write block 2016 len 4096 data: 7
233/300 Write block 1317 len 4096 data: 3
234/300 Write block 4347 len 8192 data: 2 2
235/300 Write block 3227 len 28672 data: 10 10 6 5 5 4 4
236/300 Write block 3935 len 12288 data: 3 3 3
237/300 Read block 834 len 40960
238/300 Flush
239/300 Read block 1453 len 4096
240/300 Write block 1478 len 8192 data: 2 2
241/300 Read block 4173 len 16384
242/300 Read block 4491 len 40960
243/300 Read block 599 len 24576
244/300 Write block 4705 len 4096 data: 5
245/300 Write block 801 len 32768 data: 5 4 4 4 5 5 4 4
246/300 Flush
247/300 Read block 3922 len 16384
248/300 Read block 4842 len 12288
249/300 Write block 1760 len 36864 data: 4 6 5 7 6 4 5 4 4
250/300 Write block 40 len 8192 data: 5 5
251/300 Read block 937 len 24576
252/300 Read block 3900 len 16384
253/300 Read block 440 len 28672
254/300 Read block 229 len 24576
255/300 Read block 1944 len 32768
256/300 Read block 1464 len 28672
257/300 Write block 4287 len 24576 data: 8 7 7 6 6 7
258/300 Write block 3817 len 28672 data: 6 6 6 6 7 7 7
259/300 Read block 251 len 40960
260/300 Read block 2140 len 12288
261/300 Write block 3220 len 12288 data: 7 9 9
262/300 Read block 1037 len 32768
263/300 Write block 2743 len 24576 data: 2 2 3 3 3 3
264/300 Read block 3468 len 20480
265/300 Write block 1419 len 4096 data: 3
266/300 Read block 1439 len 8192
267/300 Write block 2815 len 20480 data: 2 2 2 2 2
268/300 Read block 1618 len 12288
269/300 Write block 2960 len 32768 data: 4 3 3 2 2 3 4 3
270/300 Read block 766 len 40960
271/300 Write block 3247 len 8192 data: 5 5
272/300 Write block 1435 len 32768 data: 5 5 5 4 3 3 3 4
273/300 Read block 2429 len 28672
274/300 Read block 1702 len 16384
275/300 Read block 745 len 12288
276/300 Read block 3215 len 8192
277/300 Read block 1217 len 8192
278/300 Flush
279/300 Write block 1765 len 16384 data: 5 6 5 5
280/300 Read block 3139 len 28672
281/300 Write block 4041 len 36864 data: 6 5 4 5 4 4 4 4 4
282/300 Write block 4937 len 24576 data: 4 4 4 5 5 5
283/300 Write block 826 len 24576 data: 4 4 5 6 3 3
284/300 Write block 2651 len 8192 data: 7 7
285/300 Write block 3998 len 32768 data: 3 3 3 3 4 4 4 4
286/300 Read block 3043 len 16384
287/300 Read block 2415 len 40960
288/300 Read block 1157 len 4096
289/300 Flush
290/300 Write block 709 len 20480 data: 5 6 6 7 7
291/300 Read block 1088 len 4096
292/300 Write block 3679 len 36864 data: 8 9 9 9 10 8 8 7 8
293/300 Write block 0 len 24576 data: 3 5 6 6 7 7
294/300 Read block 4488 len 8192
295/300 Read block 3066 len 28672
296/300 Read block 343 len 24576
297/300 Read block 2812 len 4096
298/300 Write block 4234 len 24576 data: 4 4 4 3 4 5
299/300 Read block 1866 len 8192
300/300 Write block 3306 len 12288 data: 6 5 5
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 13 21:42:44.793 INFO request completed, latency_us: 224, response_code: 204, uri: /start/cid/0, method: POST, req_id: 805f93b2-7778-4da0-ac00-9f5586879ef8, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
5125 Acked 6124 Write 3 New Done Done true
5126 Acked 6125 Write 3 New Done Done true
5127 Acked 6126 Write 3 New Done Done true
5128 Acked 6127 Read 9 New Done Done true
5129 Acked 6128 Flush 0 New Done Done false
5130 Acked 6129 Read 9 New Done Done false
5131 Acked 6130 Read 7 New Done Done false
5132 Acked 6131 Write 3 New Done Done false
5133 Acked 6132 Read 2 New Done Done false
5134 Acked 6133 Read 1 New Done Done false
5135 Acked 6134 Read 10 New Done Done false
5136 Acked 6135 Read 6 New Done Done false
5137 Acked 6136 Write 7 New Done Done false
5138 Acked 6137 Write 8 New Done Done false
5139 Acked 6138 Write 4 New Done Done false
5140 Acked 6139 Write 10 New Done Done false
5141 Acked 6140 Write 7 New Done Done false
5142 Acked 6141 Write 3 New Done Done false
5143 Acked 6142 Write 10 New Done Done false
5144 Acked 6143 Read 7 New Done Done false
5145 Acked 6144 Read 5 New Done Done false
5146 Acked 6145 Read 3 New Done Done false
5147 Acked 6146 Read 9 New Done Done false
5148 Acked 6147 Write 8 New Done Done false
5149 Acked 6148 Write 5 New Done Done false
5150 Acked 6149 Write 4 New Done Done false
5151 Acked 6150 Read 4 New Done Done false
5152 Acked 6151 Read 7 New Done Done false
5153 Acked 6152 Read 8 New Done Done false
5154 Acked 6153 Write 2 New Done Done false
5155 Acked 6154 Flush 0 New Done Done false
5156 Acked 6155 Read 6 New Done Done false
5157 Acked 6156 Read 2 New Done Done false
5158 Acked 6157 Write 3 New Done Done false
5159 Acked 6158 Write 2 New Done Done false
5160 Acked 6159 Read 9 New Done Done false
5161 Acked 6160 Read 8 New Done Done false
5162 Acked 6161 Write 1 New Done Done false
5163 Acked 6162 Write 3 New Done Done false
5164 Acked 6163 Write 5 New Done Done false
5165 Acked 6164 Read 8 New Done Done false
5166 Acked 6165 Read 9 New Done Done false
5167 Acked 6166 Write 2 New Done Done false
5168 Acked 6167 Write 8 New Done Done false
5169 Acked 6168 Flush 0 New Done Done false
5170 Acked 6169 Read 3 New Done Done false
5171 Acked 6170 Read 3 New Done Done false
5172 Acked 6171 Write 2 New Done Done false
5173 Acked 6172 Read 2 New Done Done false
5174 Acked 6173 Read 2 New Done Done false
5175 Acked 6174 Flush 0 New Done Done false
5176 Acked 6175 Write 7 New Done Done false
5177 Acked 6176 Write 6 New Done Done false
5178 Acked 6177 Write 9 New Done Done false
5179 Acked 6178 Read 5 New Done Done false
5180 Acked 6179 Read 9 New Done Done false
5181 Acked 6180 Write 6 New Done Done false
5182 Acked 6181 Read 6 New Done Done false
5183 Acked 6182 Read 7 New Done Done false
5184 Acked 6183 Write 9 New Done Done false
5185 Acked 6184 Write 6 New Done Done false
5186 Acked 6185 Read 7 New Done Done false
5187 Acked 6186 Read 10 New Done Done false
5188 Acked 6187 Read 9 New Done Done false
5189 Acked 6188 Read 3 New Done Done false
5190 Acked 6189 Flush 0 New Done Done false
5191 Acked 6190 Flush 0 New Done Done false
5192 Acked 6191 Write 7 New Done Done false
5193 Acked 6192 Read 1 New Done Done false
5194 Acked 6193 Read 6 New Done Done false
5195 Acked 6194 Flush 0 New Done Done false
5196 Acked 6195 Read 6 New Done Done false
5197 Acked 6196 Read 6 New Done Done false
5198 Acked 6197 Flush 0 New Done Done false
5199 Acked 6198 Read 5 New Done Done false
5200 Acked 6199 Flush 0 New Done Done false
5201 Acked 6200 Read 8 New Done Done false
5202 Acked 6201 Write 1 New Done Done false
5203 Acked 6202 Read 1 New Done Done false
5204 Acked 6203 Flush 0 New Done Done false
5205 Acked 6204 Read 8 New Done Done false
5206 Acked 6205 Write 3 New Done Done false
5207 Acked 6206 Flush 0 New Done Done false
5208 Acked 6207 Write 10 New Done Done false
5209 Acked 6208 Write 1 New Done Done false
5210 Acked 6209 Write 6 New Done Done false
5211 Acked 6210 Write 8 New Done Done false
5212 Acked 6211 Read 8 New Done Done false
5213 Acked 6212 Read 10 New Done Done false
5214 Acked 6213 Write 3 New Done Done false
5215 Acked 6214 Read 10 New Done Done false
5216 Acked 6215 Read 3 New Done Done false
5217 Acked 6216 Flush 0 New Done Done false
5218 Acked 6217 Write 5 New Done Done false
5219 Acked 6218 Write 1 New Done Done false
5220 Acked 6219 Read 8 New Done Done false
5221 Acked 6220 Read 6 New Done Done false
5222 Acked 6221 Write 10 New Done Done false
5223 Acked 6222 Read 9 New Done Done false
5224 Acked 6223 Read 6 New Done Done false
5225 Acked 6224 Write 4 New Done Done false
5226 Acked 6225 Read 1 New Done Done false
5227 Acked 6226 Write 4 New Done Done false
5228 Acked 6227 Read 8 New Done Done false
5229 Acked 6228 Read 8 New Done Done false
5230 Acked 6229 Read 3 New Done Done false
5231 Acked 6230 Write 6 New Done Done false
5232 Acked 6231 Write 1 New Done Done false
5233 Acked 6232 Flush 0 New Done Done false
5234 Acked 6233 Read 2 New Done Done false
5235 Acked 6234 Write 9 New Done Done false
5236 Acked 6235 Flush 0 New Done Done false
5237 Acked 6236 Write 8 New Done Done false
5238 Acked 6237 Flush 0 New Done Done false
5239 Acked 6238 Read 9 New Done Done false
5240 Acked 6239 Flush 0 New Done Done false
5241 Acked 6240 Write 6 New Done Done false
5242 Acked 6241 Read 5 New Done Done false
5243 Acked 6242 Read 5 New Done Done false
5244 Acked 6243 Flush 0 New Done Done false
5245 Acked 6244 Write 3 New Done Done false
5246 Acked 6245 Write 8 New Done Done false
5247 Acked 6246 Read 10 New Done Done false
5248 Acked 6247 Write 7 New Done Done false
5249 Acked 6248 Flush 0 New Done Done false
5250 Acked 6249 Write 3 New Done Done false
5251 Acked 6250 Read 4 New Done Done false
5252 Acked 6251 Write 8 New Done Done false
5253 Acked 6252 Read 10 New Done Done false
5254 Acked 6253 Flush 0 New Done Done false
5255 Acked 6254 Read 9 New Done Done false
5256 Acked 6255 Flush 0 New Done Done false
5257 Acked 6256 Write 4 New Done Done false
5258 Acked 6257 Read 4 New Done Done false
5259 Acked 6258 Read 7 New Done Done false
5260 Acked 6259 Write 10 New Done Done false
5261 Acked 6260 Read 7 New Done Done false
5262 Acked 6261 Write 4 New Done Done false
5263 Acked 6262 Write 5 New Done Done false
5264 Acked 6263 Read 3 New Done Done false
5265 Acked 6264 Flush 0 New Done Done false
5266 Acked 6265 Read 3 New Done Done false
5267 Acked 6266 Read 8 New Done Done false
5268 Acked 6267 Write 10 New Done Done false
5269 Acked 6268 Read 7 New Done Done false
5270 Acked 6269 Read 5 New Done Done false
5271 Acked 6270 Write 9 New Done Done false
5272 Acked 6271 Write 9 New Done Done false
5273 Acked 6272 Read 9 New Done Done false
5274 Acked 6273 Write 10 New Done Done false
5275 Acked 6274 Write 6 New Done Done false
5276 Acked 6275 Write 7 New Done Done false
5277 Acked 6276 Read 10 New Done Done false
5278 Acked 6277 Write 5 New Done Done false
5279 Acked 6278 Read 4 New Done Done false
5280 Acked 6279 Read 7 New Done Done false
5281 Acked 6280 Write 6 New Done Done false
5282 Acked 6281 Write 8 New Done Done false
5283 Acked 6282 Write 4 New Done Done false
5284 Acked 6283 Write 5 New Done Done false
5285 Acked 6284 Read 7 New Done Done false
5286 Acked 6285 Read 5 New Done Done false
5287 Acked 6286 Read 7 New Done Done false
5288 Acked 6287 Write 7 New Done Done false
5289 Acked 6288 Read 5 New Done Done false
5290 Acked 6289 Read 1 New Done Done false
5291 Acked 6290 Write 3 New Done Done false
5292 Acked 6291 Write 5 New Done Done false
5293 Acked 6292 Read 5 New Done Done false
5294 Acked 6293 Read 7 New Done Done false
5295 Acked 6294 Read 2 New Done Done false
5296 Acked 6295 Read 4 New Done Done false
5297 Acked 6296 Read 1 New Done Done false
5298 Acked 6297 Read 6 New Done Done false
5299 Acked 6298 Write 6 New Done Done false
5300 Acked 6299 Write 10 New Done Done false
5301 Acked 6300 Read 6 New Done Done false
5302 Acked 6301 Read 3 New Done Done false
5303 Acked 6302 Read 1 New Done Done false
5304 Acked 6303 Read 3 New Done Done false
5305 Acked 6304 Write 2 New Done Done false
5306 Acked 6305 Write 8 New Done Done false
5307 Acked 6306 Read 4 New Done Done false
5308 Acked 6307 Write 4 New Done Done false
5309 Acked 6308 Flush 0 New Done Done false
5310 Acked 6309 Write 9 New Done Done false
5311 Acked 6310 Read 4 New Done Done false
5312 Acked 6311 Write 4 New Done Done false
5313 Acked 6312 Read 7 New Done Done false
5314 Acked 6313 Write 4 New Done Done false
5315 Acked 6314 Read 10 New Done Done false
5316 Acked 6315 Write 9 New Done Done false
5317 Acked 6316 Write 6 New Done Done false
5318 Acked 6317 Flush 0 New Done Done false
5319 Acked 6318 Write 6 New Done Done false
5320 Acked 6319 Write 6 New Done Done false
5321 Acked 6320 Flush 0 New Done Done false
5322 Acked 6321 Read 7 New Done Done false
5323 Acked 6322 Write 7 New Done Done false
5324 Acked 6323 Write 8 New Done Done false
5325 Acked 6324 Write 5 New Done Done false
5326 Acked 6325 Read 6 New Done Done false
5327 Acked 6326 Flush 0 New Done Done false
5328 Acked 6327 Read 5 New Done Done false
5329 Acked 6328 Read 8 New Done Done false
5330 Acked 6329 Read 4 New Done Done false
5331 Acked 6330 Read 8 New Done Done false
5332 Acked 6331 Read 9 New Done Done false
5333 Acked 6332 Read 2 New Done Done false
5334 Acked 6333 Read 10 New Done Done false
5335 Acked 6334 Read 3 New Done Done false
5336 Acked 6335 Write 3 New Done Done false
5337 Acked 6336 Write 4 New Done Done false
5338 Acked 6337 Write 6 New Done Done false
5339 Acked 6338 Read 5 New Done Done false
5340 Acked 6339 Write 5 New Done Done false
5341 Acked 6340 Write 7 New Done Done false
5342 Acked 6341 Read 5 New Done Done false
5343 Acked 6342 Read 5 New Done Done false
5344 Acked 6343 Write 10 New Done Done false
5345 Acked 6344 Write 7 New Done Done false
5346 Acked 6345 Read 8 New Done Done false
5347 Acked 6346 Read 3 New Done Done false
5348 Acked 6347 Write 7 New Done Done false
5349 Acked 6348 Read 3 New Done Done false
5350 Acked 6349 Read 5 New Done Done false
5351 Acked 6350 Write 7 New Done Done false
5352 Acked 6351 Write 5 New Done Done false
5353 Acked 6352 Read 6 New Done Done false
5354 Acked 6353 Read 4 New Done Done false
5355 Acked 6354 Write 2 New Done Done false
5356 Acked 6355 Read 1 New Done Done false
5357 Acked 6356 Write 1 New Done Done false
5358 Acked 6357 Write 1 New Done Done false
5359 Acked 6358 Write 2 New Done Done false
5360 Acked 6359 Write 7 New Done Done false
5361 Acked 6360 Write 3 New Done Done false
5362 Acked 6361 Read 10 New Done Done false
5363 Acked 6362 Flush 0 New Done Done false
5364 Acked 6363 Read 1 New Done Done false
5365 Acked 6364 Write 2 New Done Done false
5366 Acked 6365 Read 4 New Done Done false
5367 Acked 6366 Read 10 New Done Done false
5368 Acked 6367 Read 6 New Done Done false
5369 Acked 6368 Write 1 New Done Done false
5370 Acked 6369 Write 8 New Done Done false
5371 Acked 6370 Flush 0 New Done Done false
5372 Acked 6371 Read 4 New Done Done false
5373 Acked 6372 Read 3 New Done Done false
5374 Acked 6373 Write 9 New Done Done false
5375 Acked 6374 Write 2 New Done Done false
5376 Acked 6375 Read 6 New Done Done false
5377 Acked 6376 Read 4 New Done Done false
5378 Acked 6377 Read 7 New Done Done false
5379 Acked 6378 Read 6 New Done Done false
5380 Acked 6379 Read 8 New Done Done false
5381 Acked 6380 Read 7 New Done Done false
5382 Acked 6381 Write 6 New Done Done false
5383 Acked 6382 Write 7 New Done Done false
5384 Acked 6383 Read 10 New Done Done false
5385 Acked 6384 Read 3 New Done Done false
5386 Acked 6385 Write 3 New Done Done false
5387 Acked 6386 Read 8 New Done Done false
5388 Acked 6387 Write 6 New Done Done false
5389 Acked 6388 Read 5 New Done Done false
5390 Acked 6389 Write 1 New Done Done false
5391 Acked 6390 Read 2 New Done Done false
5392 Acked 6391 Write 5 New Done Done false
5393 Acked 6392 Read 3 New Done Done false
5394 Acked 6393 Write 8 New Done Done false
5395 Acked 6394 Read 10 New Done Done false
5396 Acked 6395 Write 2 New Done Done false
5397 Acked 6396 Write 8 New Done Done false
5398 Acked 6397 Read 7 New Done Done false
5399 Acked 6398 Read 4 New Done Done false
5400 Acked 6399 Read 3 New Done Done false
5401 Acked 6400 Read 2 New Done Done false
5402 Acked 6401 Read 2 New Done Done false
5403 Acked 6402 Flush 0 New Done Done false
5404 Acked 6403 Write 4 New Done Done false
5405 Acked 6404 Read 7 New Done Done false
5406 Acked 6405 Write 9 New Done Done false
5407 Acked 6406 Write 6 New Done Done false
5408 Acked 6407 Write 6 New Done Done false
5409 Acked 6408 Write 2 New Done Done false
5410 Acked 6409 Write 8 New Done Done false
5411 Acked 6410 Read 4 New Done Done false
5412 Acked 6411 Read 10 New Done Done false
5413 Acked 6412 Read 1 New Done Done false
5414 Acked 6413 Flush 0 New Done Done false
5415 Acked 6414 Write 5 New Done Done false
5416 Acked 6415 Read 1 New Done Done false
5417 Acked 6416 Write 9 New Done Done false
5418 Acked 6417 Write 6 New Done Done false
5419 Acked 6418 Read 2 New Done Done false
5420 Acked 6419 Read 7 New Done Done false
5421 Acked 6420 Read 6 New Done Done false
5422 Acked 6421 Read 1 New Done Done false
5423 Acked 6422 Write 6 New Done Done false
5424 Acked 6423 Read 2 New Done Done false
5425 Acked 6424 Write 3 New Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
New 301 0 0 301
Sent 0 0 0 0
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 6123 6413 6413
Downstairs last five completed: 6123 6122 6121 6120 6119
Upstairs last five completed: 5425 5424 5423 5422 5421
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:44.975120335Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:44.975172813Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:44.975191573Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:44.975223852Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:44.975250041Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 6123","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:44.975278489Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:44.975305908Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:45.975953172Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:45.976041499Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:45.976066588Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:45.976087607Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:45.976103407Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 6123","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:45.976123006Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:45.976248371Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:46.976819678Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:46.976884025Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:46.976907654Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:46.976937103Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:46.976953462Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 6123","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:46.976971312Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:46.977088117Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:47.977621706Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:47.977671404Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:47.977689993Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810][0] reports Starting
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:47.977706903Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:47.977720032Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 6123","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:47.977807019Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:47.977954503Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1182)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
5125 Acked 6124 Write 3 New Done Done true
5126 Acked 6125 Write 3 New Done Done true
5127 Acked 6126 Write 3 New Done Done true
5128 Acked 6127 Read 9 New Done Done true
5129 Acked 6128 Flush 0 New Done Done true
5130 Acked 6129 Read 9 New Done Done true
5131 Acked 6130 Read 7 New Done Done true
5132 Acked 6131 Write 3 New Done Done true
5133 Acked 6132 Read 2 New Done Done true
5134 Acked 6133 Read 1 New Done Done true
5135 Acked 6134 Read 10 New Done Done true
5136 Acked 6135 Read 6 New Done Done true
5137 Acked 6136 Write 7 New Done Done true
5138 Acked 6137 Write 8 New Done Done true
5139 Acked 6138 Write 4 New Done Done true
5140 Acked 6139 Write 10 New Done Done true
5141 Acked 6140 Write 7 New Done Done true
5142 Acked 6141 Write 3 New Done Done true
5143 Acked 6142 Write 10 New Done Done true
5144 Acked 6143 Read 7 New Done Done true
5145 Acked 6144 Read 5 New Done Done true
5146 Acked 6145 Read 3 New Done Done true
5147 Acked 6146 Read 9 New Done Done true
5148 Acked 6147 Write 8 New Done Done true
5149 Acked 6148 Write 5 New Done Done true
5150 Acked 6149 Write 4 New Done Done true
5151 Acked 6150 Read 4 New Done Done true
5152 Acked 6151 Read 7 New Done Done true
5153 Acked 6152 Read 8 New Done Done true
5154 Acked 6153 Write 2 New Done Done true
5155 Acked 6154 Flush 0 New Done Done true
5156 Acked 6155 Read 6 New Done Done true
5157 Acked 6156 Read 2 New Done Done true
5158 Acked 6157 Write 3 New Done Done true
5159 Acked 6158 Write 2 New Done Done true
5160 Acked 6159 Read 9 New Done Done true
5161 Acked 6160 Read 8 New Done Done true
5162 Acked 6161 Write 1 New Done Done true
5163 Acked 6162 Write 3 New Done Done true
5164 Acked 6163 Write 5 New Done Done true
5165 Acked 6164 Read 8 New Done Done true
5166 Acked 6165 Read 9 New Done Done true
5167 Acked 6166 Write 2 New Done Done true
5168 Acked 6167 Write 8 New Done Done true
5169 Acked 6168 Flush 0 New Done Done true
5170 Acked 6169 Read 3 New Done Done true
5171 Acked 6170 Read 3 New Done Done true
5172 Acked 6171 Write 2 New Done Done true
5173 Acked 6172 Read 2 New Done Done true
5174 Acked 6173 Read 2 New Done Done true
5175 Acked 6174 Flush 0 New Done Done true
5176 Acked 6175 Write 7 New Done Done true
5177 Acked 6176 Write 6 New Done Done true
5178 Acked 6177 Write 9 New Done Done true
5179 Acked 6178 Read 5 New Done Done true
5180 Acked 6179 Read 9 New Done Done true
5181 Acked 6180 Write 6 New Done Done true
5182 Acked 6181 Read 6 New Done Done true
5183 Acked 6182 Read 7 New Done Done true
5184 Acked 6183 Write 9 New Done Done true
5185 Acked 6184 Write 6 New Done Done true
5186 Acked 6185 Read 7 New Done Done true
5187 Acked 6186 Read 10 New Done Done true
5188 Acked 6187 Read 9 New Done Done true
5189 Acked 6188 Read 3 New Done Done true
5190 Acked 6189 Flush 0 New Done Done true
5191 Acked 6190 Flush 0 New Done Done true
5192 Acked 6191 Write 7 New Done Done true
5193 Acked 6192 Read 1 New Done Done true
5194 Acked 6193 Read 6 New Done Done true
5195 Acked 6194 Flush 0 New Done Done true
5196 Acked 6195 Read 6 New Done Done true
5197 Acked 6196 Read 6 New Done Done true
5198 Acked 6197 Flush 0 New Done Done true
5199 Acked 6198 Read 5 New Done Done true
5200 Acked 6199 Flush 0 New Done Done true
5201 Acked 6200 Read 8 New Done Done true
5202 Acked 6201 Write 1 New Done Done true
5203 Acked 6202 Read 1 New Done Done true
5204 Acked 6203 Flush 0 New Done Done true
5205 Acked 6204 Read 8 New Done Done true
5206 Acked 6205 Write 3 New Done Done true
5207 Acked 6206 Flush 0 New Done Done true
5208 Acked 6207 Write 10 New Done Done true
5209 Acked 6208 Write 1 New Done Done true
5210 Acked 6209 Write 6 New Done Done true
5211 Acked 6210 Write 8 New Done Done true
5212 Acked 6211 Read 8 New Done Done true
5213 Acked 6212 Read 10 New Done Done true
5214 Acked 6213 Write 3 New Done Done true
5215 Acked 6214 Read 10 New Done Done true
5216 Acked 6215 Read 3 New Done Done true
5217 Acked 6216 Flush 0 New Done Done true
5218 Acked 6217 Write 5 New Done Done true
5219 Acked 6218 Write 1 New Done Done true
5220 Acked 6219 Read 8 New Done Done true
5221 Acked 6220 Read 6 New Done Done true
5222 Acked 6221 Write 10 New Done Done true
5223 Acked 6222 Read 9 New Done Done true
5224 Acked 6223 Read 6 New Done Done true
5225 Acked 6224 Write 4 New Done Done true
5226 Acked 6225 Read 1 New Done Done true
5227 Acked 6226 Write 4 New Done Done true
5228 Acked 6227 Read 8 New Done Done true
5229 Acked 6228 Read 8 New Done Done true
5230 Acked 6229 Read 3 New Done Done true
5231 Acked 6230 Write 6 New Done Done true
5232 Acked 6231 Write 1 New Done Done true
5233 Acked 6232 Flush 0 New Done Done true
5234 Acked 6233 Read 2 New Done Done true
5235 Acked 6234 Write 9 New Done Done true
5236 Acked 6235 Flush 0 New Done Done true
5237 Acked 6236 Write 8 New Done Done true
5238 Acked 6237 Flush 0 New Done Done true
5239 Acked 6238 Read 9 New Done Done true
5240 Acked 6239 Flush 0 New Done Done true
5241 Acked 6240 Write 6 New Done Done true
5242 Acked 6241 Read 5 New Done Done true
5243 Acked 6242 Read 5 New Done Done true
5244 Acked 6243 Flush 0 New Done Done true
5245 Acked 6244 Write 3 New Done Done true
5246 Acked 6245 Write 8 New Done Done true
5247 Acked 6246 Read 10 New Done Done true
5248 Acked 6247 Write 7 New Done Done true
5249 Acked 6248 Flush 0 New Done Done true
5250 Acked 6249 Write 3 New Done Done true
5251 Acked 6250 Read 4 New Done Done true
5252 Acked 6251 Write 8 New Done Done true
5253 Acked 6252 Read 10 New Done Done true
5254 Acked 6253 Flush 0 New Done Done true
5255 Acked 6254 Read 9 New Done Done true
5256 Acked 6255 Flush 0 New Done Done true
5257 Acked 6256 Write 4 New Done Done true
5258 Acked 6257 Read 4 New Done Done true
5259 Acked 6258 Read 7 New Done Done true
5260 Acked 6259 Write 10 New Done Done true
5261 Acked 6260 Read 7 New Done Done true
5262 Acked 6261 Write 4 New Done Done true
5263 Acked 6262 Write 5 New Done Done true
5264 Acked 6263 Read 3 New Done Done true
5265 Acked 6264 Flush 0 New Done Done true
5266 Acked 6265 Read 3 New Done Done true
5267 Acked 6266 Read 8 New Done Done true
5268 Acked 6267 Write 10 New Done Done true
5269 Acked 6268 Read 7 New Done Done true
5270 Acked 6269 Read 5 New Done Done true
5271 Acked 6270 Write 9 New Done Done true
5272 Acked 6271 Write 9 New Done Done true
5273 Acked 6272 Read 9 New Done Done true
5274 Acked 6273 Write 10 New Done Done true
5275 Acked 6274 Write 6 New Done Done true
5276 Acked 6275 Write 7 New Done Done true
5277 Acked 6276 Read 10 New Done Done true
5278 Acked 6277 Write 5 New Done Done true
5279 Acked 6278 Read 4 New Done Done true
5280 Acked 6279 Read 7 New Done Done true
5281 Acked 6280 Write 6 New Done Done true
5282 Acked 6281 Write 8 New Done Done true
5283 Acked 6282 Write 4 New Done Done true
5284 Acked 6283 Write 5 New Done Done true
5285 Acked 6284 Read 7 New Done Done true
5286 Acked 6285 Read 5 New Done Done true
5287 Acked 6286 Read 7 New Done Done true
5288 Acked 6287 Write 7 New Done Done true
5289 Acked 6288 Read 5 New Done Done true
5290 Acked 6289 Read 1 New Done Done true
5291 Acked 6290 Write 3 New Done Done true
5292 Acked 6291 Write 5 New Done Done true
5293 Acked 6292 Read 5 New Done Done true
5294 Acked 6293 Read 7 New Done Done true
5295 Acked 6294 Read 2 New Done Done true
5296 Acked 6295 Read 4 New Done Done true
5297 Acked 6296 Read 1 New Done Done true
5298 Acked 6297 Read 6 New Done Done true
5299 Acked 6298 Write 6 New Done Done true
5300 Acked 6299 Write 10 New Done Done true
5301 Acked 6300 Read 6 New Done Done true
5302 Acked 6301 Read 3 New Done Done true
5303 Acked 6302 Read 1 New Done Done true
5304 Acked 6303 Read 3 New Done Done true
5305 Acked 6304 Write 2 New Done Done true
5306 Acked 6305 Write 8 New Done Done true
5307 Acked 6306 Read 4 New Done Done true
5308 Acked 6307 Write 4 New Done Done true
5309 Acked 6308 Flush 0 New Done Done true
5310 Acked 6309 Write 9 New Done Done true
5311 Acked 6310 Read 4 New Done Done true
5312 Acked 6311 Write 4 New Done Done true
5313 Acked 6312 Read 7 New Done Done true
5314 Acked 6313 Write 4 New Done Done true
5315 Acked 6314 Read 10 New Done Done true
5316 Acked 6315 Write 9 New Done Done true
5317 Acked 6316 Write 6 New Done Done true
5318 Acked 6317 Flush 0 New Done Done true
5319 Acked 6318 Write 6 New Done Done true
5320 Acked 6319 Write 6 New Done Done true
5321 Acked 6320 Flush 0 New Done Done true
5322 Acked 6321 Read 7 New Done Done true
5323 Acked 6322 Write 7 New Done Done true
5324 Acked 6323 Write 8 New Done Done true
5325 Acked 6324 Write 5 New Done Done true
5326 Acked 6325 Read 6 New Done Done true
5327 Acked 6326 Flush 0 New Done Done true
5328 Acked 6327 Read 5 New Done Done true
5329 Acked 6328 Read 8 New Done Done true
5330 Acked 6329 Read 4 New Done Done true
5331 Acked 6330 Read 8 New Done Done true
5332 Acked 6331 Read 9 New Done Done true
5333 Acked 6332 Read 2 New Done Done true
5334 Acked 6333 Read 10 New Done Done true
5335 Acked 6334 Read 3 New Done Done true
5336 Acked 6335 Write 3 New Done Done true
5337 Acked 6336 Write 4 New Done Done true
5338 Acked 6337 Write 6 New Done Done true
5339 Acked 6338 Read 5 New Done Done true
5340 Acked 6339 Write 5 New Done Done true
5341 Acked 6340 Write 7 New Done Done true
5342 Acked 6341 Read 5 New Done Done true
5343 Acked 6342 Read 5 New Done Done true
5344 Acked 6343 Write 10 New Done Done true
5345 Acked 6344 Write 7 New Done Done true
5346 Acked 6345 Read 8 New Done Done true
5347 Acked 6346 Read 3 New Done Done true
5348 Acked 6347 Write 7 New Done Done true
5349 Acked 6348 Read 3 New Done Done true
5350 Acked 6349 Read 5 New Done Done true
5351 Acked 6350 Write 7 New Done Done true
5352 Acked 6351 Write 5 New Done Done true
5353 Acked 6352 Read 6 New Done Done true
5354 Acked 6353 Read 4 New Done Done true
5355 Acked 6354 Write 2 New Done Done true
5356 Acked 6355 Read 1 New Done Done true
5357 Acked 6356 Write 1 New Done Done true
5358 Acked 6357 Write 1 New Done Done true
5359 Acked 6358 Write 2 New Done Done true
5360 Acked 6359 Write 7 New Done Done true
5361 Acked 6360 Write 3 New Done Done true
5362 Acked 6361 Read 10 New Done Done true
5363 Acked 6362 Flush 0 New Done Done true
5364 Acked 6363 Read 1 New Done Done true
5365 Acked 6364 Write 2 New Done Done true
5366 Acked 6365 Read 4 New Done Done true
5367 Acked 6366 Read 10 New Done Done true
5368 Acked 6367 Read 6 New Done Done true
5369 Acked 6368 Write 1 New Done Done true
5370 Acked 6369 Write 8 New Done Done true
5371 Acked 6370 Flush 0 New Done Done true
5372 Acked 6371 Read 4 New Done Done true
5373 Acked 6372 Read 3 New Done Done true
5374 Acked 6373 Write 9 New Done Done true
5375 Acked 6374 Write 2 New Done Done true
5376 Acked 6375 Read 6 New Done Done true
5377 Acked 6376 Read 4 New Done Done true
5378 Acked 6377 Read 7 New Done Done true
5379 Acked 6378 Read 6 New Done Done true
5380 Acked 6379 Read 8 New Done Done true
5381 Acked 6380 Read 7 New Done Done true
5382 Acked 6381 Write 6 New Done Done true
5383 Acked 6382 Write 7 New Done Done true
5384 Acked 6383 Read 10 New Done Done true
5385 Acked 6384 Read 3 New Done Done true
5386 Acked 6385 Write 3 New Done Done true
5387 Acked 6386 Read 8 New Done Done true
5388 Acked 6387 Write 6 New Done Done true
5389 Acked 6388 Read 5 New Done Done true
5390 Acked 6389 Write 1 New Done Done true
5391 Acked 6390 Read 2 New Done Done true
5392 Acked 6391 Write 5 New Done Done true
5393 Acked 6392 Read 3 New Done Done true
5394 Acked 6393 Write 8 New Done Done true
5395 Acked 6394 Read 10 New Done Done true
5396 Acked 6395 Write 2 New Done Done true
5397 Acked 6396 Write 8 New Done Done true
5398 Acked 6397 Read 7 New Done Done true
5399 Acked 6398 Read 4 New Done Done true
5400 Acked 6399 Read 3 New Done Done true
5401 Acked 6400 Read 2 New Done Done true
5402 Acked 6401 Read 2 New Done Done true
5403 Acked 6402 Flush 0 New Done Done true
5404 Acked 6403 Write 4 New Done Done true
5405 Acked 6404 Read 7 New Done Done true
5406 Acked 6405 Write 9 New Done Done true
5407 Acked 6406 Write 6 New Done Done true
5408 Acked 6407 Write 6 New Done Done true
5409 Acked 6408 Write 2 New Done Done true
5410 Acked 6409 Write 8 New Done Done true
5411 Acked 6410 Read 4 New Done Done true
5412 Acked 6411 Read 10 New Done Done true
5413 Acked 6412 Read 1 New Done Done true
5414 Acked 6413 Flush 0 New Done Done true
5415 Acked 6414 Write 5 New Done Done true
5416 Acked 6415 Read 1 New Done Done true
5417 Acked 6416 Write 9 New Done Done true
5418 Acked 6417 Write 6 New Done Done true
5419 Acked 6418 Read 2 New Done Done true
5420 Acked 6419 Read 7 New Done Done true
5421 Acked 6420 Read 6 New Done Done true
5422 Acked 6421 Read 1 New Done Done true
5423 Acked 6422 Write 6 New Done Done true
5424 Acked 6423 Read 2 New Done Done true
5425 Acked 6424 Write 3 New Done Done true
5426 Acked 6425 Flush 0 New Done Done true
STATES DS:0 DS:1 DS:2 TOTAL
New 302 0 0 302
Sent 0 0 0 0
Done 0 302 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 6123 6425 6425
Downstairs last five completed: 6123 6122 6121 6120 6119
Upstairs last five completed: 5426 5425 5424 5423 5422
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:48.97930844Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:48.979401966Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:48.980571411Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:48.980593771Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 6123","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:48.98060861Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 6123","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:48.980772184Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:48.980805322Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:48.980832231Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:48.980847401Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:48.98086145Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 6425 6424 6423 6422 6421
Upstairs last five completed: 5426 5425 5424 5423 5422
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(1)
stop 1
Feb 13 21:42:52.798 INFO request completed, latency_us: 322, response_code: [8820] Got stop action so:false kr:false
204, uri: /stop/cid/1, method: POST, req_id: 061a6c04-0daf-4976-8d21-c5afa799c37f, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Feb 13 21:42:52.799 INFO request completed, latency_us: 131, response_code: 200, uri: /state/cid/1, method: GET, req_id: ec987820-ef27-416b-9888-46cc7126cf9a, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:52.799485235Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:52.799574181Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:52.79960975Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:52.799636449Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 0 jobs since flush 6425","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:52.799663208Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:52.799689917Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:53.800533203Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:53.800600431Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:53.80062545Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:53.800647109Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:53.800663318Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 0 jobs since flush 6425","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:53.800680748Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:53.800698007Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:54.802022415Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:54.802084172Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:54.802106991Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:54.80212813Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:54.80214435Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 0 jobs since flush 6425","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:54.802161489Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:54.802178659Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:55.802665519Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:55.802727467Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:55.802749596Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:55.802770095Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:55.802785944Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 0 jobs since flush 6425","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:55.802803894Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:55.802821253Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:42:56.801 INFO request completed, latency_us: 203, response_code: 200, uri: /state/cid/1, method: GET, req_id: 5ed54e1e-660e-4f2e-a559-bd172d9aabae, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Write block 1887 len 36864 data: 4 3 5 6 5 5 5 5 6
002/300 Write block 1948 len 20480 data: 5 4 3 3 4
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:56.80391511Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:56.803967088Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40003/300 Write block 2335 len 12288 data: 6," 6time" 4:
"2024-02-13T21:42:56.803987287Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:56.804027555Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:56.804042014Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 2 jobs since flush 6425","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:56.804068644Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:56.804096012Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
004/300 Read block 4461 len 24576
005/300 Write block 3622 len 8192 data: 4 5
006/300 Read block 2276 len 20480
007/300 Read block 2207 len 4096
008/300 Read block 367 len 32768
009/300 Write block 3569 len 16384 data: 4 2 2 3
010/300 Flush
011/300 Read block 1133 len 28672
012/300 Read block 2969 len 32768
013/300 Read block 1479 len 24576
014/300 Read block 4731 len 24576
015/300 Write block 3623 len 28672 data: 6 5 5 4 4 4 3
016/300 Read block 839 len 36864
017/300 Flush
018/300 Read block 506 len 24576
019/300 Write block 3847 len 40960 data: 3 3 2 3 5 7 7 8 8 8
020/300 Read block 4502 len 20480
021/300 Write block 1273 len 40960 data: 2 3 3 3 2 2 4 4 5 5
022/300 Write block 4644 len 16384 data: 3 3 4 5
023/300 Read block 1783 len 40960
024/300 Write block 25 len 4096 data: 4
025/300 Write block 3776 len 36864 data: 3 3 3 4 4 4 4 4 4
026/300 Read block 2771 len 12288
027/300 Write block 1904 len 4096 data: 4
028/300 Read block 605 len 40960
029/300 Write block 2978 len 40960 data: 5 3 4 6 7 6 6 7 6 6
030/300 Flush
031/300 Read block 1730 len 36864
032/300 Write block 3654 len 20480 data: 4 5 5 4 4
033/300 Write block 4108 len 12288 data: 4 4 5
034/300 Read block 4591 len 8192
035/300 Write block 4161 len 36864 data: 6 5 4 4 3 3 3 3 4
036/300 Write block 330 len 8192 data: 6 9
037/300 Write block 2742 len 24576 data: 2 3 3 4 4 4
038/300 Flush
039/300 Write block 27 len 24576 data: 4 2 3 3 3 5
040/300 Read block 2362 len 16384
041/300 Flush
042/300 Flush
043/300 Flush
044/300 Read block 504 len 32768
045/300 Write block 1416 len 16384 data: 4 4 3 4
046/300 Write block 3475 len 16384 data: 3 2 3 3
047/300 Write block 1698 len 8192 data: 4 3
048/300 Write block 806 len 20480 data: 6 5 5 4 4
049/300 Read block 2185 len 12288
050/300 Read block 224 len 16384
051/300 Read block 4509 len 8192
052/300 Flush
053/300 Read block 123 len 36864
054/300 Read block 430 len 28672
055/300 Flush
056/300 Read block 3626 len 32768
057/300 Write block 3811 len 12288 data: 5 5 4
058/300 Write block 4849 len 36864 data: 2 2 2 2 2 2 3 3 3
059/300 Read block 531 len 28672
060/300 Write block 219 len 24576 data: 4 4 3 3 3 2
061/300 Write block 1578 len 32768 data: 3 3 3 3 3 3 3 3
062/300 Write block 3141 len 4096 data: 4
063/300 Write block 706 len 24576 data: 2 2 2 6 7 7
064/300 Flush
065/300 Read block 1204 len 36864
066/300 Write block 2009 len 24576 data: 6 5 6 7 7 8
067/300 Read block 4069 len 40960
068/300 Write block 2560 len 40960 data: 6 6 6 7 5 5 6 5 4 5
069/300 Write block 4831 len 24576 data: 6 5 6 5 5 5
070/300 Read block 3890 len 4096
071/300 Read block 3749 len 8192
072/300 Write block 3273 len 40960 data: 3 3 4 6 6 6 5 5 5 4
073/300 Flush
074/300 Read block 358 len 32768
075/300 Read block 1003 len 20480
076/300 Read block 1234 len 24576
077/300 Write block 1178 len 8192 data: 3 3
078/300 Write block 2367 len 28672 data: 4 4 4 4 4 3 3
079/300 Write block 2521 len 40960 data: 4 5 5 3 4 5 5 5 6 5
080/300 Write block 2913 len 36864 data: 5 7 8 8 7 6 5 5 4
081/300 Write block 4847 len 4096 data: 2
082/300 Write block 1147 len 40960 data: 4 3 4 4 4 4 4 5 4 4
083/300 Write block 682 len 40960 data: 4 6 6 6 7 7 7 6 5 4
084/300 Write block 4487 len 40960 data: 7 10 10 8 7 6 4 5 4 5
085/300 Write block 1080 len 32768 data: 5 4 4 3 3 2 2 2
086/300 Write block 4787 len 32768 data: 3 4 4 3 2 3 4 5
087/300 Write block 50 len 8192 data: 5 4
088/300 Write block 3740 len 12288 data: 7 6 6
089/300 Flush
090/300 Read block 423 len 20480
091/300 Read block 3645 len 20480
092/300 Write block 3489 len 12288 data: 4 4 3
093/300 Read block 3031 len 4096
094/300 Read block 4697 len 16384
095/300 Read block 2713 len 32768
096/300 Write block 4191 len 40960 data: 4 4 4 3 4 4 5 5 5 4
097/300 Read block 875 len 28672
098/300 Write block 1264 len 40960 data: 6 5 5 6 4 2 2 2 2 3
099/300 Read block 351 len 36864
100/300 Read block 2768 len 40960
101/300 Write block 325 len 16384 data: 2 3 3 4
102/300 Write block 2414 len 32768 data: 4 4 4 3 3 3 2 2
103/300 Read block 2121 len 16384
104/300 Write block 972 len 28672 data: 2 3 3 3 4 5 5
105/300 Read block 270 len 4096
106/300 Write block 1765 len 36864 data: 6 7 6 6 5 4 4 4 7
107/300 Flush
108/300 Write block 121 len 32768 data: 5 5 7 7 7 5 5 4
109/300 Flush
110/300 Write block 3402 len 36864 data: 6 6 8 8 8 6 7 6 6
111/300 Read block 3896 len 4096
112/300 Write block 4079 len 20480 data: 4 5 4 4 5
113/300 Write block 407 len 20480 data: 4 4 5 4 4
114/300 Read block 4413 len 4096
115/300 Read block 3628 len 16384
116/300 Read block 3838 len 4096
117/300 Write block 4917 len 24576 data: 5 6 6 5 3 2
118/300 Write block 3664 len 28672 data: 6 7 6 4 5 5 6
119/300 Write block 3995 len 28672 data: 3 3 3 4 4 4 4
120/300 Write block 3625 len 28672 data: 6 5 5 5 4 3 3
121/300 Read block 960 len 28672
122/300 Write block 1688 len 28672 data: 3 3 4 4 5 5 4
123/300 Read block 1892 len 28672
124/300 Read block 2223 len 12288
125/300 Read block 1158 len 20480
126/300 Read block 2504 len 40960
127/300 Write block 1999 len 20480 data: 8 8 7 5 5
128/300 Read block 3053 len 32768
129/300 Read block 4369 len 12288
130/300 Read block 4758 len 12288
131/300 Write block 4662 len 32768 data: 4 5 6 6 7 7 8 8
132/300 Write block 1999 len 24576 data: 9 9 8 6 6 5
133/300 Write block 4572 len 28672 data: 4 5 5 4 4 4 5
134/300 Write block 1440 len 8192 data: 4 4
135/300 Read block 3821 len 12288
136/300 Read block 3642 len 28672
137/300 Write block 826 len 20480 data: 5 5 6 7 4
138/300 Write block 3182 len 32768 data: 3 4 4 4 3 2 3 3
139/300 Read block 1890 len 28672
140/300 Write block 815 len 40960 data: 3 4 4 4 2 2 3 3 3 3
141/300 Flush
142/300 Read block 3519 len 24576
143/300 Write block 4572 len 8192 data: 5 6
144/300 Read block 318 len 28672
145/300 Write block 2711 len 12288 data: 6 6 7
146/300 Flush
147/300 Read block 2115 len 4096
148/300 Write block 1139 len 40960 data: 4 3 3 3 3 5 5 4 5 4
149/300 Write block 2913 len 20480 data: 6 8 9 9 8
150/300 Write block 4571 len 32768 data: 3 6 7 6 5 5 5 6
151/300 Write block 4009 len 28672 data: 5 4 3 3 3 4 4
152/300 Read block 4486 len 40960
153/300 Write block 2743 len 20480 data: 4 4 5 5 5
154/300 Read block 4803 len 8192
155/300 Read block 3302 len 24576
156/300 Read block 2680 len 24576
157/300 Read block 4359 len 32768
158/300 Write block 4090 len 12288 data: 6 6 6
159/300 Read block 4995 len 20480
160/300 Write block 2703 len 4096 data: 3
161/300 Read block 299 len 8192
162/300 Read block 1567 len 40960
163/300 Write block 3129 len 36864 data: 4 4 5 4 4 4 5 6 5
164/300 Read block 2493 len 28672
165/300 Read block 3678 len 8192
166/300 Read block 2157 len 8192
167/300 Write block 4744 len 8192 data: 4 4
168/300 Write block 2250 len 4096 data: 3
169/300 Read block 4948 len 24576
170/300 Read block 2260 len 4096
171/300 Read block 3654 len 24576
172/300 Read block 610 len 28672
173/300 Flush
174/300 Write block 4277 len 24576 data: 4 4 3 3 3 4
175/300 Read block 4974 len 24576
176/300 Read block 3759 len 4096
177/300 Read block 544 len 28672
178/300 Read block 501 len 36864
179/300 Read block 1714 len 20480
180/300 Write block 3004 len 24576 data: 4 5 6 4 4 4
181/300 Read block 3300 len 28672
182/300 Read block 860 len 32768
183/300 Read block 1450 len 4096
184/300 Write block 1948 len 12288 data: 6 5 4
185/300 Read block 3103 len 20480
186/300 Read block 2458 len 36864
187/300 Flush
188/300 Write block 3813 len 12288 data: 5 5 4
189/300 Write block 4210 len 36864 data: 2 2 3 3 3 4 3 3 4
190/300 Read block 3160 len 16384
191/300 Write block 4801 len 12288 data: 5 5 6
192/300 Flush
193/300 Read block 3050 len 20480
194/300 Write block 1210 len 20480 data: 7 6 5 5 5
195/300 Write block 4963 len 16384 data: 3 3 3 3
196/300 Read block 4968 len 36864
197/300 Flush
198/300 Write block 3776 len 28672 data: 4 4 4 5 5 5 5
199/300 Flush
200/300 Write block 4344 len 8192 data: 4 3
201/300 Read block 1530 len 40960
202/300 Write block 340 len 8192 data: 3 3
203/300 Read block 1988 len 20480
204/300 Write block 2492 len 16384 data: 3 3 3 2
205/300 Read block 108 len 28672
206/300 Write block 1654 len 28672 data: 4 5 6 6 7 9 7
207/300 Write block 4194 len 32768 data: 4 5 5 6 6 6 5 4
208/300 Write block 3317 len 24576 data: 4 4 3 5 5 6
209/300 Flush
210/300 Write block 432 len 24576 data: 5 5 4 5 4 4
211/300 Write block 2422 len 8192 data: 2 2
212/300 Write block 4003 len 8192 data: 5 5
213/300 Write block 4213 len 40960 data: 4 4 5 4 4 5 4 4 5 5
214/300 Write block 4430 len 40960 data: 3 3 3 3 4 3 4 4 3 2
215/300 Read block 2093 len 8192
216/300 Write block 112 len 28672 data: 2 2 3 3 2 3 3
217/300 Read block 3437 len 28672
218/300 Write block 2968 len 32768 data: 3 2 2 2 2 3 4 4
219/300 Read block 4837 len 28672
220/300 Write block 2062 len 28672 data: 3 3 3 3 2 3 4
221/300 Write block 3368 len 32768 data: 6 9 9 7 8 9 8 6
222/300 Write block 1009 len 4096 data: 2
223/300 Read block 3199 len 16384
224/300 Write block 4118 len 8192 data: 5 5
225/300 Read block 2611 len 24576
226/300 Read block 235 len 32768
227/300 Flush
228/300 Flush
229/300 Read block 4065 len 20480
230/300 Flush
231/300 Read block 3239 len 28672
232/300 Write block 1736 len 36864 data: 5 5 5 3 3 3 4 3 4
233/300 Read block 2216 len 8192
234/300 Read block 37 len 4096
235/300 Read block 2694 len 40960
236/300 Read block 3454 len 40960
237/300 Flush
238/300 Read block 4087 len 20480
239/300 Write block 3827 len 4096 data: 6
240/300 Read block 4025 len 36864
241/300 Write block 226 len 4096 data: 2
242/300 Write block 725 len 28672 data: 4 4 3 4 5 6 6
243/300 Write block 1626 len 16384 data: 4 4 5 4
244/300 Read block 177 len 12288
245/300 Read block 2225 len 4096
246/300 Read block 2539 len 8192
247/300 Write block 818 len 36864 data: 5 3 3 4 4 4 4 3 6
248/300 Read block 2996 len 28672
249/300 Read block 1505 len 8192
250/300 Write block 3093 len 16384 data: 5 6 4 4
251/300 Read block 726 len 36864
252/300 Read block 340 len 32768
253/300 Read block 4988 len 28672
254/300 Read block 1255 len 16384
255/300 Read block 4550 len 8192
256/300 Write block 4205 len 16384 data: 4 4 3 3
257/300 Write block 78 len 36864 data: 4 2 2 2 2 3 4 4 4
258/300 Write block 3522 len 8192 data: 4 4
259/300 Write block 273 len 8192 data: 3 2
260/300 Read block 4574 len 16384
261/300 Write block 3111 len 20480 data: 6 5 4 4 4
262/300 Write block 1984 len 4096 data: 3
263/300 Write block 96 len 24576 data: 6 4 4 3 3 5
264/300 Read block 2115 len 36864
265/300 Read block 1525 len 32768
266/300 Read block 4368 len 16384
267/300 Flush
268/300 Read block 1830 len 28672
269/300 Read block 1930 len 28672
270/300 Write block 3838 len 8192 data: 5 5
271/300 Read block 4194 len 40960
272/300 Read block 1400 len 20480
273/300 Write block 3707 len 4096 data: 2
274/300 Read block 3401 len 36864
275/300 Write block 2196 len 20480 data: 5 5 5 4 5
276/300 Read block 4295 len 32768
277/300 Write block 382 len 16384 data: 6 7 8 8
278/300 Write block 3013 len 8192 data: 3 4
279/300 Write block 3428 len 28672 data: 5 6 7 6 6 5 7
280/300 Write block 50 len 28672 data: 6 5 4 4 4 4 4
281/300 Write block 2732 len 4096 data: 6
282/300 Write block 1274 len 32768 data: 4 4 4 3 3 5 5 6
283/300 Read block 4027 len 4096
284/300 Write block 761 len 40960 data: 4 3 4 5 5 4 5 4 3 3
285/300 Write block 3285 len 20480 data: 6 7 7 6 6
286/300 Read block 222 len 8192
287/300 Write block 374 len 16384 data: 5 4 4 4
288/300 Write block 3082 len 40960 data: 5 3 2 2 2 2 2 2 4 4
289/300 Flush
290/300 Write block 3512 len 12288 data: 6 6 7
291/300 Read block 2142 len 28672
292/300 Read block 2583 len 40960
293/300 Read block 1346 len 8192
294/300 Write block 4998 len 4096 data: 3
295/300 Read block 1765 len 24576
296/300 Write block 293 len 32768 data: 4 4 4 5 5 5 5 6
297/300 Flush
298/300 Read block 2684 len 24576
299/300 Read block 791 len 32768
300/300 Write block 2921 len 36864 data: 5 3 3 2 3 4 4 4 3
Main task has work to do, go find it
got dsc Start(1)
start 1
Feb 13 21:42:57.609 INFO request completed, latency_us: 264, response_code: 204, uri: /start/cid/1, method: POST, req_id: 6b694db7-b681-48f3-a3ec-41b498f7844e, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
States: Active Offline Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
5427 Acked 6426 Write 9 Done New Done true
5428 Acked 6427 Write 5 Done New Done true
5429 Acked 6428 Write 3 Done New Done false
5430 Acked 6429 Read 6 Done New Done false
5431 Acked 6430 Write 2 Done New Done false
5432 Acked 6431 Read 5 Done New Done false
5433 Acked 6432 Read 1 Done New Done false
5434 Acked 6433 Read 8 Done New Done false
5435 Acked 6434 Write 4 Done New Done false
5436 Acked 6435 Flush 0 Done New Done false
5437 Acked 6436 Read 7 Done New Done false
5438 Acked 6437 Read 8 Done New Done false
5439 Acked 6438 Read 6 Done New Done false
5440 Acked 6439 Read 6 Done New Done false
5441 Acked 6440 Write 7 Done New Done false
5442 Acked 6441 Read 9 Done New Done false
5443 Acked 6442 Flush 0 Done New Done false
5444 Acked 6443 Read 6 Done New Done false
5445 Acked 6444 Write 10 Done New Done false
5446 Acked 6445 Read 5 Done New Done false
5447 Acked 6446 Write 10 Done New Done false
5448 Acked 6447 Write 4 Done New Done false
5449 Acked 6448 Read 10 Done New Done false
5450 Acked 6449 Write 1 Done New Done false
5451 Acked 6450 Write 9 Done New Done false
5452 Acked 6451 Read 3 Done New Done false
5453 Acked 6452 Write 1 Done New Done false
5454 Acked 6453 Read 10 Done New Done false
5455 Acked 6454 Write 10 Done New Done false
5456 Acked 6455 Flush 0 Done New Done false
5457 Acked 6456 Read 9 Done New Done false
5458 Acked 6457 Write 5 Done New Done false
5459 Acked 6458 Write 3 Done New Done false
5460 Acked 6459 Read 2 Done New Done false
5461 Acked 6460 Write 9 Done New Done false
5462 Acked 6461 Write 2 Done New Done false
5463 Acked 6462 Write 6 Done New Done false
5464 Acked 6463 Flush 0 Done New Done false
5465 Acked 6464 Write 6 Done New Done false
5466 Acked 6465 Read 4 Done New Done false
5467 Acked 6466 Flush 0 Done New Done false
5468 Acked 6467 Flush 0 Done New Done false
5469 Acked 6468 Flush 0 Done New Done false
5470 Acked 6469 Read 8 Done New Done false
5471 Acked 6470 Write 4 Done New Done false
5472 Acked 6471 Write 4 Done New Done false
5473 Acked 6472 Write 2 Done New Done false
5474 Acked 6473 Write 5 Done New Done false
5475 Acked 6474 Read 3 Done New Done false
5476 Acked 6475 Read 4 Done New Done false
5477 Acked 6476 Read 2 Done New Done false
5478 Acked 6477 Flush 0 Done New Done false
5479 Acked 6478 Read 9 Done New Done false
5480 Acked 6479 Read 7 Done New Done false
5481 Acked 6480 Flush 0 Done New Done false
5482 Acked 6481 Read 8 Done New Done false
5483 Acked 6482 Write 3 Done New Done false
5484 Acked 6483 Write 9 Done New Done false
5485 Acked 6484 Read 7 Done New Done false
5486 Acked 6485 Write 6 Done New Done false
5487 Acked 6486 Write 8 Done New Done false
5488 Acked 6487 Write 1 Done New Done false
5489 Acked 6488 Write 6 Done New Done false
5490 Acked 6489 Flush 0 Done New Done false
5491 Acked 6490 Read 9 Done New Done false
5492 Acked 6491 Write 6 Done New Done false
5493 Acked 6492 Read 10 Done New Done false
5494 Acked 6493 Write 10 Done New Done false
5495 Acked 6494 Write 6 Done New Done false
5496 Acked 6495 Read 1 Done New Done false
5497 Acked 6496 Read 2 Done New Done false
5498 Acked 6497 Write 10 Done New Done false
5499 Acked 6498 Flush 0 Done New Done false
5500 Acked 6499 Read 8 Done New Done false
5501 Acked 6500 Read 5 Done New Done false
5502 Acked 6501 Read 6 Done New Done false
5503 Acked 6502 Write 2 Done New Done false
5504 Acked 6503 Write 7 Done New Done false
5505 Acked 6504 Write 10 Done New Done false
5506 Acked 6505 Write 9 Done New Done false
5507 Acked 6506 Write 1 Done New Done false
5508 Acked 6507 Write 10 Done New Done false
5509 Acked 6508 Write 10 Done New Done false
5510 Acked 6509 Write 10 Done New Done false
5511 Acked 6510 Write 8 Done New Done false
5512 Acked 6511 Write 8 Done New Done false
5513 Acked 6512 Write 2 Done New Done false
5514 Acked 6513 Write 3 Done New Done false
5515 Acked 6514 Flush 0 Done New Done false
5516 Acked 6515 Read 5 Done New Done false
5517 Acked 6516 Read 5 Done New Done false
5518 Acked 6517 Write 3 Done New Done false
5519 Acked 6518 Read 1 Done New Done false
5520 Acked 6519 Read 4 Done New Done false
5521 Acked 6520 Read 8 Done New Done false
5522 Acked 6521 Write 10 Done New Done false
5523 Acked 6522 Read 7 Done New Done false
5524 Acked 6523 Write 10 Done New Done false
5525 Acked 6524 Read 9 Done New Done false
5526 Acked 6525 Read 10 Done New Done false
5527 Acked 6526 Write 4 Done New Done false
5528 Acked 6527 Write 8 Done New Done false
5529 Acked 6528 Read 4 Done New Done false
5530 Acked 6529 Write 7 Done New Done false
5531 Acked 6530 Read 1 Done New Done false
5532 Acked 6531 Write 9 Done New Done false
5533 Acked 6532 Flush 0 Done New Done false
5534 Acked 6533 Write 8 Done New Done false
5535 Acked 6534 Flush 0 Done New Done false
5536 Acked 6535 Write 9 Done New Done false
5537 Acked 6536 Read 1 Done New Done false
5538 Acked 6537 Write 5 Done New Done false
5539 Acked 6538 Write 5 Done New Done false
5540 Acked 6539 Read 1 Done New Done false
5541 Acked 6540 Read 4 Done New Done false
5542 Acked 6541 Read 1 Done New Done false
5543 Acked 6542 Write 6 Done New Done false
5544 Acked 6543 Write 7 Done New Done false
5545 Acked 6544 Write 7 Done New Done false
5546 Acked 6545 Write 7 Done New Done false
5547 Acked 6546 Read 7 Done New Done false
5548 Acked 6547 Write 7 Done New Done false
5549 Acked 6548 Read 7 Done New Done false
5550 Acked 6549 Read 3 Done New Done false
5551 Acked 6550 Read 5 Done New Done false
5552 Acked 6551 Read 10 Done New Done false
5553 Acked 6552 Write 5 Done New Done false
5554 Acked 6553 Read 8 Done New Done false
5555 Acked 6554 Read 3 Done New Done false
5556 Acked 6555 Read 3 Done New Done false
5557 Acked 6556 Write 8 Done New Done false
5558 Acked 6557 Write 6 Done New Done false
5559 Acked 6558 Write 7 Done New Done false
5560 Acked 6559 Write 2 Done New Done false
5561 Acked 6560 Read 3 Done New Done false
5562 Acked 6561 Read 7 Done New Done false
5563 Acked 6562 Write 5 Done New Done false
5564 Acked 6563 Write 8 Done New Done false
5565 Acked 6564 Read 7 Done New Done false
5566 Acked 6565 Write 10 Done New Done false
5567 Acked 6566 Flush 0 Done New Done false
5568 Acked 6567 Read 6 Done New Done false
5569 Acked 6568 Write 2 Done New Done false
5570 Acked 6569 Read 7 Done New Done false
5571 Acked 6570 Write 3 Done New Done false
5572 Acked 6571 Flush 0 Done New Done false
5573 Acked 6572 Read 1 Done New Done false
5574 Acked 6573 Write 10 Done New Done false
5575 Acked 6574 Write 5 Done New Done false
5576 Acked 6575 Write 8 Done New Done false
5577 Acked 6576 Write 7 Done New Done false
5578 Acked 6577 Read 10 Done New Done false
5579 Acked 6578 Write 5 Done New Done false
5580 Acked 6579 Read 2 Done New Done false
5581 Acked 6580 Read 6 Done New Done false
5582 Acked 6581 Read 6 Done New Done false
5583 Acked 6582 Read 8 Done New Done false
5584 Acked 6583 Write 3 Done New Done false
5585 Acked 6584 Read 5 Done New Done false
5586 Acked 6585 Write 1 Done New Done false
5587 Acked 6586 Read 2 Done New Done false
5588 Acked 6587 Read 10 Done New Done false
5589 Acked 6588 Write 9 Done New Done false
5590 Acked 6589 Read 7 Done New Done false
5591 Acked 6590 Read 2 Done New Done false
5592 Acked 6591 Read 2 Done New Done false
5593 Acked 6592 Write 2 Done New Done false
5594 Acked 6593 Write 1 Done New Done false
5595 Acked 6594 Read 6 Done New Done false
5596 Acked 6595 Read 1 Done New Done false
5597 Acked 6596 Read 6 Done New Done false
5598 Acked 6597 Read 7 Done New Done false
5599 Acked 6598 Flush 0 Done New Done false
5600 Acked 6599 Write 6 Done New Done false
5601 Acked 6600 Read 6 Done New Done false
5602 Acked 6601 Read 1 Done New Done false
5603 Acked 6602 Read 7 Done New Done false
5604 Acked 6603 Read 9 Done New Done false
5605 Acked 6604 Read 5 Done New Done false
5606 Acked 6605 Write 6 Done New Done false
5607 Acked 6606 Read 7 Done New Done false
5608 Acked 6607 Read 8 Done New Done false
5609 Acked 6608 Read 1 Done New Done false
5610 Acked 6609 Write 3 Done New Done false
5611 Acked 6610 Read 5 Done New Done false
5612 Acked 6611 Read 9 Done New Done false
5613 Acked 6612 Flush 0 Done New Done false
5614 Acked 6613 Write 3 Done New Done false
5615 Acked 6614 Write 9 Done New Done false
5616 Acked 6615 Read 4 Done New Done false
5617 Acked 6616 Write 3 Done New Done false
5618 Acked 6617 Flush 0 Done New Done false
5619 Acked 6618 Read 5 Done New Done false
5620 Acked 6619 Write 5 Done New Done false
5621 Acked 6620 Write 4 Done New Done false
5622 Acked 6621 Read 9 Done New Done false
5623 Acked 6622 Flush 0 Done New Done false
5624 Acked 6623 Write 7 Done New Done false
5625 Acked 6624 Flush 0 Done New Done false
5626 Acked 6625 Write 2 Done New Done false
5627 Acked 6626 Read 10 Done New Done false
5628 Acked 6627 Write 2 Done New Done false
5629 Acked 6628 Read 5 Done New Done false
5630 Acked 6629 Write 4 Done New Done false
5631 Acked 6630 Read 7 Done New Done false
5632 Acked 6631 Write 7 Done New Done false
5633 Acked 6632 Write 8 Done New Done false
5634 Acked 6633 Write 6 Done New Done false
5635 Acked 6634 Flush 0 Done New Done false
5636 Acked 6635 Write 6 Done New Done false
5637 Acked 6636 Write 2 Done New Done false
5638 Acked 6637 Write 2 Done New Done false
5639 Acked 6638 Write 10 Done New Done false
5640 Acked 6639 Write 10 Done New Done false
5641 Acked 6640 Read 2 Done New Done false
5642 Acked 6641 Write 7 Done New Done false
5643 Acked 6642 Read 7 Done New Done false
5644 Acked 6643 Write 8 Done New Done false
5645 Acked 6644 Read 7 Done New Done false
5646 Acked 6645 Write 7 Done New Done false
5647 Acked 6646 Write 8 Done New Done false
5648 Acked 6647 Write 1 Done New Done false
5649 Acked 6648 Read 4 Done New Done false
5650 Acked 6649 Write 2 Done New Done false
5651 Acked 6650 Read 6 Done New Done false
5652 Acked 6651 Read 8 Done New Done false
5653 Acked 6652 Flush 0 Done New Done false
5654 Acked 6653 Flush 0 Done New Done false
5655 Acked 6654 Read 5 Done New Done false
5656 Acked 6655 Flush 0 Done New Done false
5657 Acked 6656 Read 7 Done New Done false
5658 Acked 6657 Write 9 Done New Done false
5659 Acked 6658 Read 2 Done New Done false
5660 Acked 6659 Read 1 Done New Done false
5661 Acked 6660 Read 10 Done New Done false
5662 Acked 6661 Read 10 Done New Done false
5663 Acked 6662 Flush 0 Done New Done false
5664 Acked 6663 Read 5 Done New Done false
5665 Acked 6664 Write 1 Done New Done false
5666 Acked 6665 Read 9 Done New Done false
5667 Acked 6666 Write 1 Done New Done false
5668 Acked 6667 Write 7 Done New Done false
5669 Acked 6668 Write 4 Done New Done false
5670 Acked 6669 Read 3 Done New Done false
5671 Acked 6670 Read 1 Done New Done false
5672 Acked 6671 Read 2 Done New Done false
5673 Acked 6672 Write 9 Done New Done false
5674 Acked 6673 Read 7 Done New Done false
5675 Acked 6674 Read 2 Done New Done false
5676 Acked 6675 Write 4 Done New Done false
5677 Acked 6676 Read 9 Done New Done false
5678 Acked 6677 Read 8 Done New Done false
5679 Acked 6678 Read 7 Done New Done false
5680 Acked 6679 Read 4 Done New Done false
5681 Acked 6680 Read 2 Done New Done false
5682 Acked 6681 Write 4 Done New Done false
5683 Acked 6682 Write 9 Done New Done false
5684 Acked 6683 Write 2 Done New Done false
5685 Acked 6684 Write 2 Done New Done false
5686 Acked 6685 Read 4 Done New Done false
5687 Acked 6686 Write 5 Done New Done false
5688 Acked 6687 Write 1 Done New Done false
5689 Acked 6688 Write 6 Done New Done false
5690 Acked 6689 Read 9 Done New Done false
5691 Acked 6690 Read 8 Done New Done false
5692 Acked 6691 Read 4 Done New Done false
5693 Acked 6692 Flush 0 Done New Done false
5694 Acked 6693 Read 7 Done New Done false
5695 Acked 6694 Read 7 Done New Done false
5696 Acked 6695 Write 2 Done New Done false
5697 Acked 6696 Read 10 Done New Done false
5698 Acked 6697 Read 5 Done New Done false
5699 Acked 6698 Write 1 Done New Done false
5700 Acked 6699 Read 9 Done New Done false
5701 Acked 6700 Write 5 Done New Done false
5702 Acked 6701 Read 8 Done New Done false
5703 Acked 6702 Write 4 Done New Done false
5704 Acked 6703 Write 2 Done New Done false
5705 Acked 6704 Write 7 Done New Done false
5706 Acked 6705 Write 7 Done New Done false
5707 Acked 6706 Write 1 Done New Done false
5708 Acked 6707 Write 8 Done New Done false
5709 Acked 6708 Read 1 Done New Done false
5710 Acked 6709 Write 10 Done New Done false
5711 Acked 6710 Write 5 Done New Done false
5712 Acked 6711 Read 2 Done New Done false
5713 Acked 6712 Write 4 Done New Done false
5714 Acked 6713 Write 10 Done New Done false
5715 Acked 6714 Flush 0 Done New Done false
5716 Acked 6715 Write 3 Done New Done false
5717 Acked 6716 Read 7 Done New Done false
5718 Acked 6717 Read 10 Done New Done false
5719 Acked 6718 Read 2 Done New Done false
5720 Acked 6719 Write 1 Done New Done false
5721 Acked 6720 Read 6 Done New Done false
5722 Acked 6721 Write 8 Done New Done false
5723 Acked 6722 Flush 0 Done New Done false
5724 Acked 6723 Read 6 Done New Done false
5725 Acked 6724 Read 8 Done New Done false
5726 Acked 6725 Write 9 Done New Done false
STATES DS:0 DS:1 DS:2 TOTAL
New 0 300 0 300
Sent 0 0 0 0
Done 300 0 300 600
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 6722 6425 6722
Downstairs last five completed: 6425 6424 6423 6422 6421
Upstairs last five completed: 5726 5725 5724 5723 5722
CLIENT: Up:0 ds:300 act:2
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:57.804903901Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:57.805026036Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:57.805072524Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:57.805114002Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:57.805143591Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 300 jobs since flush 6425","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:57.805195369Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:57.805367343Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:58.805606362Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:58.805669039Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:58.805703528Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:58.805741197Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:58.805772816Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 301 jobs since flush 6425","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:58.805807704Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:58.805840683Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:59.8071773Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:59.807240868Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:59.807263357Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:42:59.807284507Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:59.807300436Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 301 jobs since flush 6425","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:59.807317945Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:42:59.807441741Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:00.809099865Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:00.809159053Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:00.809189852Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:00.809211061Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:00.80922699Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 301 jobs since flush 6425","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:00.809244689Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:00.809388374Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
5427 Acked 6426 Write 9 Done New Done true
5428 Acked 6427 Write 5 Done New Done true
5429 Acked 6428 Write 3 Done New Done true
5430 Acked 6429 Read 6 Done New Done true
5431 Acked 6430 Write 2 Done New Done true
5432 Acked 6431 Read 5 Done New Done true
5433 Acked 6432 Read 1 Done New Done true
5434 Acked 6433 Read 8 Done New Done true
5435 Acked 6434 Write 4 Done New Done true
5436 Acked 6435 Flush 0 Done New Done true
5437 Acked 6436 Read 7 Done New Done true
5438 Acked 6437 Read 8 Done New Done true
5439 Acked 6438 Read 6 Done New Done true
5440 Acked 6439 Read 6 Done New Done true
5441 Acked 6440 Write 7 Done New Done true
5442 Acked 6441 Read 9 Done New Done true
5443 Acked 6442 Flush 0 Done New Done true
5444 Acked 6443 Read 6 Done New Done true
5445 Acked 6444 Write 10 Done New Done true
5446 Acked 6445 Read 5 Done New Done true
5447 Acked 6446 Write 10 Done New Done true
5448 Acked 6447 Write 4 Done New Done true
5449 Acked 6448 Read 10 Done New Done true
5450 Acked 6449 Write 1 Done New Done true
5451 Acked 6450 Write 9 Done New Done true
5452 Acked 6451 Read 3 Done New Done true
5453 Acked 6452 Write 1 Done New Done true
5454 Acked 6453 Read 10 Done New Done true
5455 Acked 6454 Write 10 Done New Done true
5456 Acked 6455 Flush 0 Done New Done true
5457 Acked 6456 Read 9 Done New Done true
5458 Acked 6457 Write 5 Done New Done true
5459 Acked 6458 Write 3 Done New Done true
5460 Acked 6459 Read 2 Done New Done true
5461 Acked 6460 Write 9 Done New Done true
5462 Acked 6461 Write 2 Done New Done true
5463 Acked 6462 Write 6 Done New Done true
5464 Acked 6463 Flush 0 Done New Done true
5465 Acked 6464 Write 6 Done New Done true
5466 Acked 6465 Read 4 Done New Done true
5467 Acked 6466 Flush 0 Done New Done true
5468 Acked 6467 Flush 0 Done New Done true
5469 Acked 6468 Flush 0 Done New Done true
5470 Acked 6469 Read 8 Done New Done true
5471 Acked 6470 Write 4 Done New Done true
5472 Acked 6471 Write 4 Done New Done true
5473 Acked 6472 Write 2 Done New Done true
5474 Acked 6473 Write 5 Done New Done true
5475 Acked 6474 Read 3 Done New Done true
5476 Acked 6475 Read 4 Done New Done true
5477 Acked 6476 Read 2 Done New Done true
5478 Acked 6477 Flush 0 Done New Done true
5479 Acked 6478 Read 9 Done New Done true
5480 Acked 6479 Read 7 Done New Done true
5481 Acked 6480 Flush 0 Done New Done true
5482 Acked 6481 Read 8 Done New Done true
5483 Acked 6482 Write 3 Done New Done true
5484 Acked 6483 Write 9 Done New Done true
5485 Acked 6484 Read 7 Done New Done true
5486 Acked 6485 Write 6 Done New Done true
5487 Acked 6486 Write 8 Done New Done true
5488 Acked 6487 Write 1 Done New Done true
5489 Acked 6488 Write 6 Done New Done true
5490 Acked 6489 Flush 0 Done New Done true
5491 Acked 6490 Read 9 Done New Done true
5492 Acked 6491 Write 6 Done New Done true
5493 Acked 6492 Read 10 Done New Done true
5494 Acked 6493 Write 10 Done New Done true
5495 Acked 6494 Write 6 Done New Done true
5496 Acked 6495 Read 1 Done New Done true
5497 Acked 6496 Read 2 Done New Done true
5498 Acked 6497 Write 10 Done New Done true
5499 Acked 6498 Flush 0 Done New Done true
5500 Acked 6499 Read 8 Done New Done true
5501 Acked 6500 Read 5 Done New Done true
5502 Acked 6501 Read 6 Done New Done true
5503 Acked 6502 Write 2 Done New Done true
5504 Acked 6503 Write 7 Done New Done true
5505 Acked 6504 Write 10 Done New Done true
5506 Acked 6505 Write 9 Done New Done true
5507 Acked 6506 Write 1 Done New Done true
5508 Acked 6507 Write 10 Done New Done true
5509 Acked 6508 Write 10 Done New Done true
5510 Acked 6509 Write 10 Done New Done true
5511 Acked 6510 Write 8 Done New Done true
5512 Acked 6511 Write 8 Done New Done true
5513 Acked 6512 Write 2 Done New Done true
5514 Acked 6513 Write 3 Done New Done true
5515 Acked 6514 Flush 0 Done New Done true
5516 Acked 6515 Read 5 Done New Done true
5517 Acked 6516 Read 5 Done New Done true
5518 Acked 6517 Write 3 Done New Done true
5519 Acked 6518 Read 1 Done New Done true
5520 Acked 6519 Read 4 Done New Done true
5521 Acked 6520 Read 8 Done New Done true
5522 Acked 6521 Write 10 Done New Done true
5523 Acked 6522 Read 7 Done New Done true
5524 Acked 6523 Write 10 Done New Done true
5525 Acked 6524 Read 9 Done New Done true
5526 Acked 6525 Read 10 Done New Done true
5527 Acked 6526 Write 4 Done New Done true
5528 Acked 6527 Write 8 Done New Done true
5529 Acked 6528 Read 4 Done New Done true
5530 Acked 6529 Write 7 Done New Done true
5531 Acked 6530 Read 1 Done New Done true
5532 Acked 6531 Write 9 Done New Done true
5533 Acked 6532 Flush 0 Done New Done true
5534 Acked 6533 Write 8 Done New Done true
5535 Acked 6534 Flush 0 Done New Done true
5536 Acked 6535 Write 9 Done New Done true
5537 Acked 6536 Read 1 Done New Done true
5538 Acked 6537 Write 5 Done New Done true
5539 Acked 6538 Write 5 Done New Done true
5540 Acked 6539 Read 1 Done New Done true
5541 Acked 6540 Read 4 Done New Done true
5542 Acked 6541 Read 1 Done New Done true
5543 Acked 6542 Write 6 Done New Done true
5544 Acked 6543 Write 7 Done New Done true
5545 Acked 6544 Write 7 Done New Done true
5546 Acked 6545 Write 7 Done New Done true
5547 Acked 6546 Read 7 Done New Done true
5548 Acked 6547 Write 7 Done New Done true
5549 Acked 6548 Read 7 Done New Done true
5550 Acked 6549 Read 3 Done New Done true
5551 Acked 6550 Read 5 Done New Done true
5552 Acked 6551 Read 10 Done New Done true
5553 Acked 6552 Write 5 Done New Done true
5554 Acked 6553 Read 8 Done New Done true
5555 Acked 6554 Read 3 Done New Done true
5556 Acked 6555 Read 3 Done New Done true
5557 Acked 6556 Write 8 Done New Done true
5558 Acked 6557 Write 6 Done New Done true
5559 Acked 6558 Write 7 Done New Done true
5560 Acked 6559 Write 2 Done New Done true
5561 Acked 6560 Read 3 Done New Done true
5562 Acked 6561 Read 7 Done New Done true
5563 Acked 6562 Write 5 Done New Done true
5564 Acked 6563 Write 8 Done New Done true
5565 Acked 6564 Read 7 Done New Done true
5566 Acked 6565 Write 10 Done New Done true
5567 Acked 6566 Flush 0 Done New Done true
5568 Acked 6567 Read 6 Done New Done true
5569 Acked 6568 Write 2 Done New Done true
5570 Acked 6569 Read 7 Done New Done true
5571 Acked 6570 Write 3 Done New Done true
5572 Acked 6571 Flush 0 Done New Done true
5573 Acked 6572 Read 1 Done New Done true
5574 Acked 6573 Write 10 Done New Done true
5575 Acked 6574 Write 5 Done New Done true
5576 Acked 6575 Write 8 Done New Done true
5577 Acked 6576 Write 7 Done New Done true
5578 Acked 6577 Read 10 Done New Done true
5579 Acked 6578 Write 5 Done New Done true
5580 Acked 6579 Read 2 Done New Done true
5581 Acked 6580 Read 6 Done New Done true
5582 Acked 6581 Read 6 Done New Done true
5583 Acked 6582 Read 8 Done New Done true
5584 Acked 6583 Write 3 Done New Done true
5585 Acked 6584 Read 5 Done New Done true
5586 Acked 6585 Write 1 Done New Done true
5587 Acked 6586 Read 2 Done New Done true
5588 Acked 6587 Read 10 Done New Done true
5589 Acked 6588 Write 9 Done New Done true
5590 Acked 6589 Read 7 Done New Done true
5591 Acked 6590 Read 2 Done New Done true
5592 Acked 6591 Read 2 Done New Done true
5593 Acked 6592 Write 2 Done New Done true
5594 Acked 6593 Write 1 Done New Done true
5595 Acked 6594 Read 6 Done New Done true
5596 Acked 6595 Read 1 Done New Done true
5597 Acked 6596 Read 6 Done New Done true
5598 Acked 6597 Read 7 Done New Done true
5599 Acked 6598 Flush 0 Done New Done true
5600 Acked 6599 Write 6 Done New Done true
5601 Acked 6600 Read 6 Done New Done true
5602 Acked 6601 Read 1 Done New Done true
5603 Acked 6602 Read 7 Done New Done true
5604 Acked 6603 Read 9 Done New Done true
5605 Acked 6604 Read 5 Done New Done true
5606 Acked 6605 Write 6 Done New Done true
5607 Acked 6606 Read 7 Done New Done true
5608 Acked 6607 Read 8 Done New Done true
5609 Acked 6608 Read 1 Done New Done true
5610 Acked 6609 Write 3 Done New Done true
5611 Acked 6610 Read 5 Done New Done true
5612 Acked 6611 Read 9 Done New Done true
5613 Acked 6612 Flush 0 Done New Done true
5614 Acked 6613 Write 3 Done New Done true
5615 Acked 6614 Write 9 Done New Done true
5616 Acked 6615 Read 4 Done New Done true
5617 Acked 6616 Write 3 Done New Done true
5618 Acked 6617 Flush 0 Done New Done true
5619 Acked 6618 Read 5 Done New Done true
5620 Acked 6619 Write 5 Done New Done true
5621 Acked 6620 Write 4 Done New Done true
5622 Acked 6621 Read 9 Done New Done true
5623 Acked 6622 Flush 0 Done New Done true
5624 Acked 6623 Write 7 Done New Done true
5625 Acked 6624 Flush 0 Done New Done true
5626 Acked 6625 Write 2 Done New Done true
5627 Acked 6626 Read 10 Done New Done true
5628 Acked 6627 Write 2 Done New Done true
5629 Acked 6628 Read 5 Done New Done true
5630 Acked 6629 Write 4 Done New Done true
5631 Acked 6630 Read 7 Done New Done true
5632 Acked 6631 Write 7 Done New Done true
5633 Acked 6632 Write 8 Done New Done true
5634 Acked 6633 Write 6 Done New Done true
5635 Acked 6634 Flush 0 Done New Done true
5636 Acked 6635 Write 6 Done New Done true
5637 Acked 6636 Write 2 Done New Done true
5638 Acked 6637 Write 2 Done New Done true
5639 Acked 6638 Write 10 Done New Done true
5640 Acked 6639 Write 10 Done New Done true
5641 Acked 6640 Read 2 Done New Done true
5642 Acked 6641 Write 7 Done New Done true
5643 Acked 6642 Read 7 Done New Done true
5644 Acked 6643 Write 8 Done New Done true
5645 Acked 6644 Read 7 Done New Done true
5646 Acked 6645 Write 7 Done New Done true
5647 Acked 6646 Write 8 Done New Done true
5648 Acked 6647 Write 1 Done New Done true
5649 Acked 6648 Read 4 Done New Done true
5650 Acked 6649 Write 2 Done New Done true
5651 Acked 6650 Read 6 Done New Done true
5652 Acked 6651 Read 8 Done New Done true
5653 Acked 6652 Flush 0 Done New Done true
5654 Acked 6653 Flush 0 Done New Done true
5655 Acked 6654 Read 5 Done New Done true
5656 Acked 6655 Flush 0 Done New Done true
5657 Acked 6656 Read 7 Done New Done true
5658 Acked 6657 Write 9 Done New Done true
5659 Acked 6658 Read 2 Done New Done true
5660 Acked 6659 Read 1 Done New Done true
5661 Acked 6660 Read 10 Done New Done true
5662 Acked 6661 Read 10 Done New Done true
5663 Acked 6662 Flush 0 Done New Done true
5664 Acked 6663 Read 5 Done New Done true
5665 Acked 6664 Write 1 Done New Done true
5666 Acked 6665 Read 9 Done New Done true
5667 Acked 6666 Write 1 Done New Done true
5668 Acked 6667 Write 7 Done New Done true
5669 Acked 6668 Write 4 Done New Done true
5670 Acked 6669 Read 3 Done New Done true
5671 Acked 6670 Read 1 Done New Done true
5672 Acked 6671 Read 2 Done New Done true
5673 Acked 6672 Write 9 Done New Done true
5674 Acked 6673 Read 7 Done New Done true
5675 Acked 6674 Read 2 Done New Done true
5676 Acked 6675 Write 4 Done New Done true
5677 Acked 6676 Read 9 Done New Done true
5678 Acked 6677 Read 8 Done New Done true
5679 Acked 6678 Read 7 Done New Done true
5680 Acked 6679 Read 4 Done New Done true
5681 Acked 6680 Read 2 Done New Done true
5682 Acked 6681 Write 4 Done New Done true
5683 Acked 6682 Write 9 Done New Done true
5684 Acked 6683 Write 2 Done New Done true
5685 Acked 6684 Write 2 Done New Done true
5686 Acked 6685 Read 4 Done New Done true
5687 Acked 6686 Write 5 Done New Done true
5688 Acked 6687 Write 1 Done New Done true
5689 Acked 6688 Write 6 Done New Done true
5690 Acked 6689 Read 9 Done New Done true
5691 Acked 6690 Read 8 Done New Done true
5692 Acked 6691 Read 4 Done New Done true
5693 Acked 6692 Flush 0 Done New Done true
5694 Acked 6693 Read 7 Done New Done true
5695 Acked 6694 Read 7 Done New Done true
5696 Acked 6695 Write 2 Done New Done true
5697 Acked 6696 Read 10 Done New Done true
5698 Acked 6697 Read 5 Done New Done true
5699 Acked 6698 Write 1 Done New Done true
5700 Acked 6699 Read 9 Done New Done true
5701 Acked 6700 Write 5 Done New Done true
5702 Acked 6701 Read 8 Done New Done true
5703 Acked 6702 Write 4 Done New Done true
5704 Acked 6703 Write 2 Done New Done true
5705 Acked 6704 Write 7 Done New Done true
5706 Acked 6705 Write 7 Done New Done true
5707 Acked 6706 Write 1 Done New Done true
5708 Acked 6707 Write 8 Done New Done true
5709 Acked 6708 Read 1 Done New Done true
5710 Acked 6709 Write 10 Done New Done true
5711 Acked 6710 Write 5 Done New Done true
5712 Acked 6711 Read 2 Done New Done true
5713 Acked 6712 Write 4 Done New Done true
5714 Acked 6713 Write 10 Done New Done true
5715 Acked 6714 Flush 0 Done New Done true
5716 Acked 6715 Write 3 Done New Done true
5717 Acked 6716 Read 7 Done New Done true
5718 Acked 6717 Read 10 Done New Done true
5719 Acked 6718 Read 2 Done New Done true
5720 Acked 6719 Write 1 Done New Done true
5721 Acked 6720 Read 6 Done New Done true
5722 Acked 6721 Write 8 Done New Done true
5723 Acked 6722 Flush 0 Done New Done true
5724 Acked 6723 Read 6 Done New Done true
5725 Acked 6724 Read 8 Done New Done true
5726 Acked 6725 Write 9 Done New Done true
5727 Acked 6726 Flush 0 Done New Done true
STATES DS:0 DS:1 DS:2 TOTAL
New 0 301 0 301
Sent 0 0 0 0
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 6726 6425 6726
Downstairs last five completed: 6425 6424 6423 6422 6421
Upstairs last five completed: 5727 5726 5725 5724 5723
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:01.810031088Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:01.810082076Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:01.810104295Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:01.810125084Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:01.810141064Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 301 jobs since flush 6425","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:01.810159293Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:01.810263469Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:02.810891554Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:02.810952551Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:02.81097474Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:02.81099564Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:02.811018289Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 301 jobs since flush 6425","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:02.811036588Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:02.811149344Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:03.811699911Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:03.811743059Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:03.811764469Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:03.811784828Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:03.811800947Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 301 jobs since flush 6425","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:03.811818456Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:03.811916113Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:04.813035268Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:04.813095566Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:04.813118955Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:04.813139824Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:04.813163773Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 301 jobs since flush 6425","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:04.813181952Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:04.813277009Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
5427 Acked 6426 Write 9 Done New Done true
5428 Acked 6427 Write 5 Done New Done true
5429 Acked 6428 Write 3 Done New Done true
5430 Acked 6429 Read 6 Done New Done true
5431 Acked 6430 Write 2 Done New Done true
5432 Acked 6431 Read 5 Done New Done true
5433 Acked 6432 Read 1 Done New Done true
5434 Acked 6433 Read 8 Done New Done true
5435 Acked 6434 Write 4 Done New Done true
5436 Acked 6435 Flush 0 Done New Done true
5437 Acked 6436 Read 7 Done New Done true
5438 Acked 6437 Read 8 Done New Done true
5439 Acked 6438 Read 6 Done New Done true
5440 Acked 6439 Read 6 Done New Done true
5441 Acked 6440 Write 7 Done New Done true
5442 Acked 6441 Read 9 Done New Done true
5443 Acked 6442 Flush 0 Done New Done true
5444 Acked 6443 Read 6 Done New Done true
5445 Acked 6444 Write 10 Done New Done true
5446 Acked 6445 Read 5 Done New Done true
5447 Acked 6446 Write 10 Done New Done true
5448 Acked 6447 Write 4 Done New Done true
5449 Acked 6448 Read 10 Done New Done true
5450 Acked 6449 Write 1 Done New Done true
5451 Acked 6450 Write 9 Done New Done true
5452 Acked 6451 Read 3 Done New Done true
5453 Acked 6452 Write 1 Done New Done true
5454 Acked 6453 Read 10 Done New Done true
5455 Acked 6454 Write 10 Done New Done true
5456 Acked 6455 Flush 0 Done New Done true
5457 Acked 6456 Read 9 Done New Done true
5458 Acked 6457 Write 5 Done New Done true
5459 Acked 6458 Write 3 Done New Done true
5460 Acked 6459 Read 2 Done New Done true
5461 Acked 6460 Write 9 Done New Done true
5462 Acked 6461 Write 2 Done New Done true
5463 Acked 6462 Write 6 Done New Done true
5464 Acked 6463 Flush 0 Done New Done true
5465 Acked 6464 Write 6 Done New Done true
5466 Acked 6465 Read 4 Done New Done true
5467 Acked 6466 Flush 0 Done New Done true
5468 Acked 6467 Flush 0 Done New Done true
5469 Acked 6468 Flush 0 Done New Done true
5470 Acked 6469 Read 8 Done New Done true
5471 Acked 6470 Write 4 Done New Done true
5472 Acked 6471 Write 4 Done New Done true
5473 Acked 6472 Write 2 Done New Done true
5474 Acked 6473 Write 5 Done New Done true
5475 Acked 6474 Read 3 Done New Done true
5476 Acked 6475 Read 4 Done New Done true
5477 Acked 6476 Read 2 Done New Done true
5478 Acked 6477 Flush 0 Done New Done true
5479 Acked 6478 Read 9 Done New Done true
5480 Acked 6479 Read 7 Done New Done true
5481 Acked 6480 Flush 0 Done New Done true
5482 Acked 6481 Read 8 Done New Done true
5483 Acked 6482 Write 3 Done New Done true
5484 Acked 6483 Write 9 Done New Done true
5485 Acked 6484 Read 7 Done New Done true
5486 Acked 6485 Write 6 Done New Done true
5487 Acked 6486 Write 8 Done New Done true
5488 Acked 6487 Write 1 Done New Done true
5489 Acked 6488 Write 6 Done New Done true
5490 Acked 6489 Flush 0 Done New Done true
5491 Acked 6490 Read 9 Done New Done true
5492 Acked 6491 Write 6 Done New Done true
5493 Acked 6492 Read 10 Done New Done true
5494 Acked 6493 Write 10 Done New Done true
5495 Acked 6494 Write 6 Done New Done true
5496 Acked 6495 Read 1 Done New Done true
5497 Acked 6496 Read 2 Done New Done true
5498 Acked 6497 Write 10 Done New Done true
5499 Acked 6498 Flush 0 Done New Done true
5500 Acked 6499 Read 8 Done New Done true
5501 Acked 6500 Read 5 Done New Done true
5502 Acked 6501 Read 6 Done New Done true
5503 Acked 6502 Write 2 Done New Done true
5504 Acked 6503 Write 7 Done New Done true
5505 Acked 6504 Write 10 Done New Done true
5506 Acked 6505 Write 9 Done New Done true
5507 Acked 6506 Write 1 Done New Done true
5508 Acked 6507 Write 10 Done New Done true
5509 Acked 6508 Write 10 Done New Done true
5510 Acked 6509 Write 10 Done New Done true
5511 Acked 6510 Write 8 Done New Done true
5512 Acked 6511 Write 8 Done New Done true
5513 Acked 6512 Write 2 Done New Done true
5514 Acked 6513 Write 3 Done New Done true
5515 Acked 6514 Flush 0 Done New Done true
5516 Acked 6515 Read 5 Done New Done true
5517 Acked 6516 Read 5 Done New Done true
5518 Acked 6517 Write 3 Done New Done true
5519 Acked 6518 Read 1 Done New Done true
5520 Acked 6519 Read 4 Done New Done true
5521 Acked 6520 Read 8 Done New Done true
5522 Acked 6521 Write 10 Done New Done true
5523 Acked 6522 Read 7 Done New Done true
5524 Acked 6523 Write 10 Done New Done true
5525 Acked 6524 Read 9 Done New Done true
5526 Acked 6525 Read 10 Done New Done true
5527 Acked 6526 Write 4 Done New Done true
5528 Acked 6527 Write 8 Done New Done true
5529 Acked 6528 Read 4 Done New Done true
5530 Acked 6529 Write 7 Done New Done true
5531 Acked 6530 Read 1 Done New Done true
5532 Acked 6531 Write 9 Done New Done true
5533 Acked 6532 Flush 0 Done New Done true
5534 Acked 6533 Write 8 Done New Done true
5535 Acked 6534 Flush 0 Done New Done true
5536 Acked 6535 Write 9 Done New Done true
5537 Acked 6536 Read 1 Done New Done true
5538 Acked 6537 Write 5 Done New Done true
5539 Acked 6538 Write 5 Done New Done true
5540 Acked 6539 Read 1 Done New Done true
5541 Acked 6540 Read 4 Done New Done true
5542 Acked 6541 Read 1 Done New Done true
5543 Acked 6542 Write 6 Done New Done true
5544 Acked 6543 Write 7 Done New Done true
5545 Acked 6544 Write 7 Done New Done true
5546 Acked 6545 Write 7 Done New Done true
5547 Acked 6546 Read 7 Done New Done true
5548 Acked 6547 Write 7 Done New Done true
5549 Acked 6548 Read 7 Done New Done true
5550 Acked 6549 Read 3 Done New Done true
5551 Acked 6550 Read 5 Done New Done true
5552 Acked 6551 Read 10 Done New Done true
5553 Acked 6552 Write 5 Done New Done true
5554 Acked 6553 Read 8 Done New Done true
5555 Acked 6554 Read 3 Done New Done true
5556 Acked 6555 Read 3 Done New Done true
5557 Acked 6556 Write 8 Done New Done true
5558 Acked 6557 Write 6 Done New Done true
5559 Acked 6558 Write 7 Done New Done true
5560 Acked 6559 Write 2 Done New Done true
5561 Acked 6560 Read 3 Done New Done true
5562 Acked 6561 Read 7 Done New Done true
5563 Acked 6562 Write 5 Done New Done true
5564 Acked 6563 Write 8 Done New Done true
5565 Acked 6564 Read 7 Done New Done true
5566 Acked 6565 Write 10 Done New Done true
5567 Acked 6566 Flush 0 Done New Done true
5568 Acked 6567 Read 6 Done New Done true
5569 Acked 6568 Write 2 Done New Done true
5570 Acked 6569 Read 7 Done New Done true
5571 Acked 6570 Write 3 Done New Done true
5572 Acked 6571 Flush 0 Done New Done true
5573 Acked 6572 Read 1 Done New Done true
5574 Acked 6573 Write 10 Done New Done true
5575 Acked 6574 Write 5 Done New Done true
5576 Acked 6575 Write 8 Done New Done true
5577 Acked 6576 Write 7 Done New Done true
5578 Acked 6577 Read 10 Done New Done true
5579 Acked 6578 Write 5 Done New Done true
5580 Acked 6579 Read 2 Done New Done true
5581 Acked 6580 Read 6 Done New Done true
5582 Acked 6581 Read 6 Done New Done true
5583 Acked 6582 Read 8 Done New Done true
5584 Acked 6583 Write 3 Done New Done true
5585 Acked 6584 Read 5 Done New Done true
5586 Acked 6585 Write 1 Done New Done true
5587 Acked 6586 Read 2 Done New Done true
5588 Acked 6587 Read 10 Done New Done true
5589 Acked 6588 Write 9 Done New Done true
5590 Acked 6589 Read 7 Done New Done true
5591 Acked 6590 Read 2 Done New Done true
5592 Acked 6591 Read 2 Done New Done true
5593 Acked 6592 Write 2 Done New Done true
5594 Acked 6593 Write 1 Done New Done true
5595 Acked 6594 Read 6 Done New Done true
5596 Acked 6595 Read 1 Done New Done true
5597 Acked 6596 Read 6 Done New Done true
5598 Acked 6597 Read 7 Done New Done true
5599 Acked 6598 Flush 0 Done New Done true
5600 Acked 6599 Write 6 Done New Done true
5601 Acked 6600 Read 6 Done New Done true
5602 Acked 6601 Read 1 Done New Done true
5603 Acked 6602 Read 7 Done New Done true
5604 Acked 6603 Read 9 Done New Done true
5605 Acked 6604 Read 5 Done New Done true
5606 Acked 6605 Write 6 Done New Done true
5607 Acked 6606 Read 7 Done New Done true
5608 Acked 6607 Read 8 Done New Done true
5609 Acked 6608 Read 1 Done New Done true
5610 Acked 6609 Write 3 Done New Done true
5611 Acked 6610 Read 5 Done New Done true
5612 Acked 6611 Read 9 Done New Done true
5613 Acked 6612 Flush 0 Done New Done true
5614 Acked 6613 Write 3 Done New Done true
5615 Acked 6614 Write 9 Done New Done true
5616 Acked 6615 Read 4 Done New Done true
5617 Acked 6616 Write 3 Done New Done true
5618 Acked 6617 Flush 0 Done New Done true
5619 Acked 6618 Read 5 Done New Done true
5620 Acked 6619 Write 5 Done New Done true
5621 Acked 6620 Write 4 Done New Done true
5622 Acked 6621 Read 9 Done New Done true
5623 Acked 6622 Flush 0 Done New Done true
5624 Acked 6623 Write 7 Done New Done true
5625 Acked 6624 Flush 0 Done New Done true
5626 Acked 6625 Write 2 Done New Done true
5627 Acked 6626 Read 10 Done New Done true
5628 Acked 6627 Write 2 Done New Done true
5629 Acked 6628 Read 5 Done New Done true
5630 Acked 6629 Write 4 Done New Done true
5631 Acked 6630 Read 7 Done New Done true
5632 Acked 6631 Write 7 Done New Done true
5633 Acked 6632 Write 8 Done New Done true
5634 Acked 6633 Write 6 Done New Done true
5635 Acked 6634 Flush 0 Done New Done true
5636 Acked 6635 Write 6 Done New Done true
5637 Acked 6636 Write 2 Done New Done true
5638 Acked 6637 Write 2 Done New Done true
5639 Acked 6638 Write 10 Done New Done true
5640 Acked 6639 Write 10 Done New Done true
5641 Acked 6640 Read 2 Done New Done true
5642 Acked 6641 Write 7 Done New Done true
5643 Acked 6642 Read 7 Done New Done true
5644 Acked 6643 Write 8 Done New Done true
5645 Acked 6644 Read 7 Done New Done true
5646 Acked 6645 Write 7 Done New Done true
5647 Acked 6646 Write 8 Done New Done true
5648 Acked 6647 Write 1 Done New Done true
5649 Acked 6648 Read 4 Done New Done true
5650 Acked 6649 Write 2 Done New Done true
5651 Acked 6650 Read 6 Done New Done true
5652 Acked 6651 Read 8 Done New Done true
5653 Acked 6652 Flush 0 Done New Done true
5654 Acked 6653 Flush 0 Done New Done true
5655 Acked 6654 Read 5 Done New Done true
5656 Acked 6655 Flush 0 Done New Done true
5657 Acked 6656 Read 7 Done New Done true
5658 Acked 6657 Write 9 Done New Done true
5659 Acked 6658 Read 2 Done New Done true
5660 Acked 6659 Read 1 Done New Done true
5661 Acked 6660 Read 10 Done New Done true
5662 Acked 6661 Read 10 Done New Done true
5663 Acked 6662 Flush 0 Done New Done true
5664 Acked 6663 Read 5 Done New Done true
5665 Acked 6664 Write 1 Done New Done true
5666 Acked 6665 Read 9 Done New Done true
5667 Acked 6666 Write 1 Done New Done true
5668 Acked 6667 Write 7 Done New Done true
5669 Acked 6668 Write 4 Done New Done true
5670 Acked 6669 Read 3 Done New Done true
5671 Acked 6670 Read 1 Done New Done true
5672 Acked 6671 Read 2 Done New Done true
5673 Acked 6672 Write 9 Done New Done true
5674 Acked 6673 Read 7 Done New Done true
5675 Acked 6674 Read 2 Done New Done true
5676 Acked 6675 Write 4 Done New Done true
5677 Acked 6676 Read 9 Done New Done true
5678 Acked 6677 Read 8 Done New Done true
5679 Acked 6678 Read 7 Done New Done true
5680 Acked 6679 Read 4 Done New Done true
5681 Acked 6680 Read 2 Done New Done true
5682 Acked 6681 Write 4 Done New Done true
5683 Acked 6682 Write 9 Done New Done true
5684 Acked 6683 Write 2 Done New Done true
5685 Acked 6684 Write 2 Done New Done true
5686 Acked 6685 Read 4 Done New Done true
5687 Acked 6686 Write 5 Done New Done true
5688 Acked 6687 Write 1 Done New Done true
5689 Acked 6688 Write 6 Done New Done true
5690 Acked 6689 Read 9 Done New Done true
5691 Acked 6690 Read 8 Done New Done true
5692 Acked 6691 Read 4 Done New Done true
5693 Acked 6692 Flush 0 Done New Done true
5694 Acked 6693 Read 7 Done New Done true
5695 Acked 6694 Read 7 Done New Done true
5696 Acked 6695 Write 2 Done New Done true
5697 Acked 6696 Read 10 Done New Done true
5698 Acked 6697 Read 5 Done New Done true
5699 Acked 6698 Write 1 Done New Done true
5700 Acked 6699 Read 9 Done New Done true
5701 Acked 6700 Write 5 Done New Done true
5702 Acked 6701 Read 8 Done New Done true
5703 Acked 6702 Write 4 Done New Done true
5704 Acked 6703 Write 2 Done New Done true
5705 Acked 6704 Write 7 Done New Done true
5706 Acked 6705 Write 7 Done New Done true
5707 Acked 6706 Write 1 Done New Done true
5708 Acked 6707 Write 8 Done New Done true
5709 Acked 6708 Read 1 Done New Done true
5710 Acked 6709 Write 10 Done New Done true
5711 Acked 6710 Write 5 Done New Done true
5712 Acked 6711 Read 2 Done New Done true
5713 Acked 6712 Write 4 Done New Done true
5714 Acked 6713 Write 10 Done New Done true
5715 Acked 6714 Flush 0 Done New Done true
5716 Acked 6715 Write 3 Done New Done true
5717 Acked 6716 Read 7 Done New Done true
5718 Acked 6717 Read 10 Done New Done true
5719 Acked 6718 Read 2 Done New Done true
5720 Acked 6719 Write 1 Done New Done true
5721 Acked 6720 Read 6 Done New Done true
5722 Acked 6721 Write 8 Done New Done true
5723 Acked 6722 Flush 0 Done New Done true
5724 Acked 6723 Read 6 Done New Done true
5725 Acked 6724 Read 8 Done New Done true
5726 Acked 6725 Write 9 Done New Done true
5727 Acked 6726 Flush 0 Done New Done true
STATES DS:0 DS:1 DS:2 TOTAL
New 0 301 0 301
Sent 0 0 0 0
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 6726 6425 6726
Downstairs last five completed: 6425 6424 6423 6422 6421
Upstairs last five completed: 5727 5726 5725 5724 5723
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:05.813870855Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:05.813934152Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:05.813956691Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:05.813977761Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:05.81399373Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 301 jobs since flush 6425","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:05.814011749Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:05.814135185Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1183)
[8820][1] reports Running
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:06.814648443Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:06.814705251Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:06.81473191Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:06.814757479Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:06.814776558Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 301 jobs since flush 6425","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:06.814797898Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:06.814859815Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:07.815443321Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:07.815502469Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:07.816905995Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:07.816933324Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 6425","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:07.816948253Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 6425","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:07.817195374Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:07.817232293Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:07.817252242Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:07.817267261Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:07.817281521Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 6726 6725 6724 6723 6722
Upstairs last five completed: 5727 5726 5725 5724 5723
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Feb 13 21:43:09.616 INFO request completed, latency_us: [8810] Got stop action so:false kr:false
368, response_code: 204, uri: /stop/cid/0, method: POST, req_id: bb2d49d6-edf8-4840-b316-8681fdd3d4ae, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:09.617400015Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:09.617450633Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:09.617474383Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:09.617488472Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 6726","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:09.617510311Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:43:09.617 INFO request completed, latency_us: 131, response_code: 200, uri: /state/cid/0, method: GET, req_id: 4203e3ee-19c2-470a-83a1-fc579b29c91f, remote_addr:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:09.61754169Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
127.0.0.1:52124, local_addr: 127.0.0.1:9998
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:10.618421355Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:10.618483113Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:10.618505522Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:10.618526281Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:10.618542191Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 6726","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:10.61855964Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:10.618596149Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:11.620166638Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:11.620223165Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:11.620252594Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:11.620273094Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:11.620289493Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 6726","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:11.620307352Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:11.620331361Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:12.622018155Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:12.622086042Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:12.62215641Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:12.622181669Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:12.622198298Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 6726","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:12.622216117Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:12.622249106Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:43:13.618 INFO request completed, latency_us: 200, response_code: 200, uri: /state/cid/0, method: GET, req_id: 09f3b83f-cc81-4cde-b234-d70244067da1, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Read block 3288 len 12288
002/300 Read block 2426 len 4096
003/300 Flush
004/300 Read block 650 len 24576
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:13.622419078Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:13.622463917Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:13.622482286Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:13.622502815Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:13.622526264Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 4 jobs since flush 6726","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:13.622554013Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:13.622581532Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
005/300 Read block 2364 len 28672
006/300 Read block 4559 len 40960
007/300 Write block 51 len 12288 data: 6 5 5
008/300 Read block 2724 len 8192
009/300 Write block 2401 len 24576 data: 3 5 5 5 5 4
010/300 Write block 3498 len 24576 data: 4 4 4 4 3 4
011/300 Flush
012/300 Write block 3360 len 36864 data: 5 5 6 6 5 4 4 5 7
013/300 Write block 2249 len 40960 data: 3 4 3 3 4 6 6 7 7 5
014/300 Read block 3865 len 12288
015/300 Write block 2833 len 40960 data: 5 5 4 6 7 7 6 6 3 4
016/300 Read block 4718 len 20480
017/300 Write block 253 len 28672 data: 4 4 5 4 4 4 3
018/300 Read block 3874 len 24576
019/300 Read block 1989 len 16384
020/300 Read block 3960 len 16384
021/300 Read block 2502 len 16384
022/300 Write block 3334 len 28672 data: 4 3 5 5 6 8 7
023/300 Read block 3241 len 12288
024/300 Flush
025/300 Write block 3772 len 28672 data: 5 4 3 4 5 5 5
026/300 Read block 452 len 20480
027/300 Flush
028/300 Read block 1470 len 32768
029/300 Read block 3190 len 16384
030/300 Read block 1561 len 24576
031/300 Write block 800 len 28672 data: 6 6 5 5 5 6 7
032/300 Write block 2965 len 36864 data: 4 5 4 4 3 3 3 3 4
033/300 Read block 1119 len 40960
034/300 Write block 1846 len 4096 data: 4
035/300 Write block 969 len 16384 data: 4 4 4 3
036/300 Write block 2565 len 12288 data: 6 7 6
037/300 Write block 1726 len 4096 data: 4
038/300 Read block 1831 len 12288
039/300 Read block 3535 len 24576
040/300 Read block 4047 len 24576
041/300 Read block 4270 len 32768
042/300 Read block 3258 len 8192
043/300 Write block 2103 len 20480 data: 4 5 6 6 6
044/300 Flush
045/300 Read block 3614 len 40960
046/300 Write block 2185 len 40960 data: 5 4 4 3 3 4 4 3 4 5
047/300 Read block 4531 len 28672
048/300 Read block 3463 len 36864
049/300 Write block 3834 len 28672 data: 4 4 5 5 6 6 5
050/300 Write block 3428 len 20480 data: 6 7 8 7 7
051/300 Write block 389 len 4096 data: 5
052/300 Write block 4738 len 20480 data: 3 3 4 4 4
053/300 Write block 13 len 16384 data: 2 2 2 2
054/300 Flush
055/300 Write block 3361 len 4096 data: 6
056/300 Read block 3237 len 8192
057/300 Write block 3097 len 40960 data: 6 4 4 3 5 4 4 3 4 5
058/300 Read block 3092 len 16384
059/300 Read block 4050 len 12288
060/300 Read block 1517 len 36864
061/300 Read block 1115 len 20480
062/300 Read block 4043 len 28672
063/300 Write block 358 len 36864 data: 7 7 7 7 6 6 5 3 2
064/300 Read block 2880 len 32768
065/300 Write block 4206 len 24576 data: 5 4 4 5 3 3
066/300 Read block 3232 len 16384
067/300 Write block 532 len 4096 data: 4
068/300 Read block 1250 len 16384
069/300 Read block 3071 len 24576
070/300 Read block 85 len 40960
071/300 Read block 2352 len 40960
072/300 Write block 4041 len 8192 data: 7 6
073/300 Read block 3164 len 28672
074/300 Read block 4757 len 16384
075/300 Read block 2471 len 40960
076/300 Read block 4421 len 24576
077/300 Flush
078/300 Write block 1259 len 16384 data: 5 7 7 7
079/300 Read block 460 len 28672
080/300 Write block 2529 len 40960 data: 7 6 4 5 6 7 6 4 4 5
081/300 Read block 3558 len 36864
082/300 Read block 4646 len 36864
083/300 Write block 365 len 28672 data: 4 3 2 3 3 3 3
084/300 Read block 2795 len 32768
085/300 Write block 4260 len 16384 data: 5 3 3 5
086/300 Flush
087/300 Read block 1840 len 40960
088/300 Read block 2695 len 28672
089/300 Write block 1087 len 32768 data: 3 2 2 2 2 2 2 2
090/300 Write block 738 len 28672 data: 4 5 5 7 5 5 7
091/300 Write block 463 len 16384 data: 5 5 6 6
092/300 Read block 80 len 28672
093/300 Write block 2629 len 4096 data: 5
094/300 Write block 1494 len 16384 data: 6 7 7 5
095/300 Read block 265 len 12288
096/300 Read block 1168 len 24576
097/300 Read block 1804 len 32768
098/300 Write block 2691 len 12288 data: 3 3 3
099/300 Write block 2522 len 40960 data: 6 6 4 5 6 6 6 8 7 5
100/300 Read block 3867 len 16384
101/300 Read block 2427 len 8192
102/300 Read block 2510 len 8192
103/300 Write block 3379 len 12288 data: 5 5 4
104/300 Write block 3819 len 4096 data: 7
105/300 Write block 790 len 32768 data: 3 3 4 4 4 5 5 6
106/300 Flush
107/300 Write block 837 len 36864 data: 3 2 3 3 3 5 5 5 5
108/300 Read block 3257 len 12288
109/300 Write block 1639 len 40960 data: 6 7 7 7 7 9 7 6 4 4
110/300 Read block 1198 len 20480
111/300 Read block 1016 len 16384
112/300 Write block 4164 len 28672 data: 5 4 4 4 4 5 4
113/300 Read block 3318 len 36864
114/300 Write block 446 len 36864 data: 5 5 6 5 4 5 5 6 5
115/300 Flush
116/300 Write block 3751 len 8192 data: 3 5
117/300 Read block 3938 len 24576
118/300 Read block 4462 len 16384
119/300 Write block 1524 len 8192 data: 3 3
120/300 Read block 4854 len 24576
121/300 Write block 3445 len 32768 data: 4 4 3 3 3 3 3 2
122/300 Read block 3715 len 32768
123/300 Write block 2000 len 8192 data: 10 9
124/300 Read block 3102 len 40960
125/300 Read block 2998 len 36864
126/300 Write block 2077 len 20480 data: 5 4 3 3 3
127/300 Write block 303 len 24576 data: 6 7 7 7 7 7
128/300 Read block 629 len 28672
129/300 Write block 1466 len 16384 data: 6 7 8 5
130/300 Read block 990 len 24576
131/300 Write block 1840 len 4096 data: 3
132/300 Read block 2818 len 4096
133/300 Write block 554 len 36864 data: 3 4 3 2 3 4 4 4 6
134/300 Read block 4561 len 36864
135/300 Write block 2237 len 8192 data: 4 3
136/300 Read block 2710 len 16384
137/300 Read block 4531 len 8192
138/300 Read block 436 len 4096
139/300 Read block 2303 len 24576
140/300 Write block 4959 len 16384 data: 4 3 4 4
141/300 Write block 1079 len 12288 data: 5 6 5
142/300 Write block 4298 len 40960 data: 6 7 7 6 4 4 4 3 2 2
143/300 Read block 1196 len 8192
144/300 Read block 4172 len 20480
145/300 Write block 2258 len 40960 data: 6 4 4 5 5 4 3 3 3 3
146/300 Read block 3658 len 36864
147/300 Write block 976 len 24576 data: 5 6 6 4 4 4
148/300 Write block 461 len 12288 data: 6 7 6
149/300 Read block 3304 len 16384
150/300 Write block 2702 len 32768 data: 3 4 3 4 4 4 5 6
151/300 Read block 1898 len 12288
152/300 Read block 1162 len 36864
153/300 Read block 4750 len 40960
154/300 Write block 768 len 36864 data: 5 4 4 4 4 4 4 3 4
155/300 Flush
156/300 Read block 2919 len 28672
157/300 Flush
158/300 Read block 2768 len 4096
159/300 Read block 2094 len 28672
160/300 Read block 3286 len 28672
161/300 Flush
162/300 Write block 4264 len 4096 data: 5
163/300 Read block 849 len 36864
164/300 Read block 1645 len 8192
165/300 Read block 1710 len 4096
166/300 Read block 2343 len 16384
167/300 Read block 2297 len 16384
168/300 Read block 2395 len 32768
169/300 Write block 4601 len 40960 data: 3 3 2 2 3 3 3 4 4 4
170/300 Read block 4151 len 36864
171/300 Read block 538 len 8192
172/300 Write block 485 len 4096 data: 2
173/300 Read block 777 len 40960
174/300 Flush
175/300 Write block 596 len 20480 data: 4 4 4 5 3
176/300 Read block 4625 len 8192
177/300 Write block 4002 len 24576 data: 5 6 6 5 4 4
178/300 Read block 2289 len 36864
179/300 Write block 965 len 16384 data: 5 5 5 4
180/300 Write block 2 len 24576 data: 7 7 8 8 9 8
181/300 Read block 3561 len 32768
182/300 Write block 3510 len 4096 data: 6
183/300 Read block 3351 len 24576
184/300 Read block 148 len 4096
185/300 Read block 2081 len 12288
186/300 Read block 2045 len 12288
187/300 Write block 2423 len 20480 data: 3 2 2 3 3
188/300 Read block 2996 len 40960
189/300 Flush
190/300 Read block 3329 len 28672
191/300 Write block 3777 len 8192 data: 6 6
192/300 Write block 928 len 8192 data: 4 2
193/300 Write block 2976 len 16384 data: 5 5 6 4
194/300 Write block 446 len 24576 data: 6 6 7 6 5 6
195/300 Write block 4189 len 12288 data: 3 3 5
196/300 Write block 2981 len 4096 data: 7
197/300 Write block 1248 len 24576 data: 5 5 6 6 6 4
198/300 Write block 3708 len 20480 data: 2 3 2 2 3
199/300 Read block 3624 len 12288
200/300 Read block 1038 len 24576
201/300 Read block 1683 len 24576
202/300 Write block 1109 len 16384 data: 6 7 8 8
203/300 Write block 1432 len 12288 data: 5 6 5
204/300 Write block 4856 len 8192 data: 4 4
205/300 Read block 2697 len 16384
206/300 Read block 2504 len 12288
207/300 Read block 2328 len 24576
208/300 Read block 809 len 28672
209/300 Flush
210/300 Read block 1917 len 12288
211/300 Read block 2271 len 36864
212/300 Read block 3417 len 28672
213/300 Read block 4272 len 40960
214/300 Read block 2030 len 24576
215/300 Read block 3353 len 28672
216/300 Read block 2283 len 12288
217/300 Write block 4470 len 16384 data: 5 4 3 3
218/300 Read block 4456 len 40960
219/300 Read block 3466 len 40960
220/300 Write block 3541 len 8192 data: 3 3
221/300 Write block 4316 len 8192 data: 4 4
222/300 Flush
223/300 Read block 4438 len 8192
224/300 Write block 4756 len 24576 data: 4 3 3 4 4 4
225/300 Read block 4417 len 32768
226/300 Flush
227/300 Write block 1033 len 16384 data: 5 6 6 7
228/300 Write block 3663 len 16384 data: 6 7 8 7
229/300 Write block 2748 len 36864 data: 4 4 3 4 2 2 2 2 3
230/300 Write block 2483 len 36864 data: 4 5 6 7 6 6 5 5 3
231/300 Write block 4830 len 40960 data: 4 7 6 7 6 6 6 5 6 5
232/300 Read block 4583 len 36864
233/300 Read block 74 len 36864
234/300 Read block 889 len 12288
235/300 Flush
236/300 Write block 2245 len 20480 data: 6 5 5 2 4
237/300 Write block 3824 len 12288 data: 6 7 7
238/300 Read block 4653 len 24576
239/300 Write block 3011 len 16384 data: 3 3 4 5
240/300 Write block 4491 len 20480 data: 8 7 5 6 5
241/300 Read block 4748 len 40960
242/300 Read block 3896 len 12288
243/300 Flush
244/300 Write block 2348 len 36864 data: 2 3 5 5 5 6 6 6 6
245/300 Read block 1672 len 20480
246/300 Write block 4301 len 36864 data: 7 5 5 5 4 3 3 4 4
247/300 Write block 1532 len 28672 data: 4 6 6 5 5 5 3
248/300 Write block 3731 len 16384 data: 3 4 4 4
249/300 Read block 4425 len 8192
250/300 Read block 1623 len 12288
251/300 Read block 3644 len 20480
252/300 Write block 1082 len 20480 data: 5 4 4 3 3
253/300 Read block 2965 len 4096
254/300 Write block 2926 len 36864 data: 5 5 5 4 4 5 4 4 4
255/300 Read block 2920 len 8192
256/300 Read block 298 len 20480
257/300 Write block 3211 len 40960 data: 4 4 4 3 3 3 4 4 6 8
258/300 Write block 4944 len 28672 data: 5 5 6 5 6 5 4
259/300 Write block 4167 len 4096 data: 5
260/300 Read block 4040 len 40960
261/300 Write block 2685 len 36864 data: 3 3 4 4 4 3 4 4 4
262/300 Read block 3137 len 16384
263/300 Write block 869 len 40960 data: 5 5 3 5 4 4 5 4 4 4
264/300 Write block 920 len 36864 data: 5 5 6 7 6 5 5 4 5
265/300 Write block 4619 len 8192 data: 5 5
266/300 Write block 547 len 28672 data: 3 2 2 4 3 4 3
267/300 Write block 1207 len 12288 data: 6 8 8
268/300 Read block 1754 len 12288
269/300 Write block 3146 len 24576 data: 4 5 4 4 5 5
270/300 Read block 2886 len 24576
271/300 Write block 2929 len 24576 data: 5 5 6 5 5 5
272/300 Write block 2074 len 32768 data: 6 6 6 6 5 4 4 4
273/300 Read block 359 len 36864
274/300 Read block 21 len 8192
275/300 Flush
276/300 Read block 2887 len 4096
277/300 Read block 709 len 16384
278/300 Write block 294 len 24576 data: 5 5 6 6 6 6
279/300 Write block 543 len 32768 data: 4 4 4 4 4 3 3 5
280/300 Read block 4286 len 24576
281/300 Write block 4362 len 32768 data: 5 5 3 4 5 5 5 5
282/300 Write block 2702 len 40960 data: 4 5 4 5 5 5 6 7 6 7
283/300 Write block 2968 len 32768 data: 5 4 4 4 4 5 5 5
284/300 Read block 4837 len 8192
285/300 Write block 4838 len 4096 data: 7
286/300 Read block 4348 len 20480
287/300 Write block 4203 len 12288 data: 5 4 5
288/300 Flush
289/300 Read block 2158 len 8192
290/300 Read block 3595 len 24576
291/300 Flush
292/300 Write block 68 len 40960 data: 5 5 5 6 4 6 7 6 5 4
293/300 Flush
294/300 Read block 971 len 32768
295/300 Read block 1380 len 24576
296/300 Read block 215 len 16384
297/300 Read block 1579 len 40960
298/300 Flush
299/300 Write block 666 len 32768 data: 5 5 5 4 5 4 4 3
300/300 Read block 2684 len 8192
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 13 21:43:14.412 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/0, method: POST, req_id: 0732c77e-e6b8-4cc5-998a-089475a13b01, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
5728 Acked 6727 Read 3 New Done Done true
5729 Acked 6728 Read 1 New Done Done true
5730 Acked 6729 Flush 0 New Done Done true
5731 Acked 6730 Read 6 New Done Done true
5732 Acked 6731 Read 7 New Done Done false
5733 Acked 6732 Read 10 New Done Done false
5734 Acked 6733 Write 3 New Done Done false
5735 Acked 6734 Read 2 New Done Done false
5736 Acked 6735 Write 6 New Done Done false
5737 Acked 6736 Write 6 New Done Done false
5738 Acked 6737 Flush 0 New Done Done false
5739 Acked 6738 Write 9 New Done Done false
5740 Acked 6739 Write 10 New Done Done false
5741 Acked 6740 Read 3 New Done Done false
5742 Acked 6741 Write 10 New Done Done false
5743 Acked 6742 Read 5 New Done Done false
5744 Acked 6743 Write 7 New Done Done false
5745 Acked 6744 Read 6 New Done Done false
5746 Acked 6745 Read 4 New Done Done false
5747 Acked 6746 Read 4 New Done Done false
5748 Acked 6747 Read 4 New Done Done false
5749 Acked 6748 Write 7 New Done Done false
5750 Acked 6749 Read 3 New Done Done false
5751 Acked 6750 Flush 0 New Done Done false
5752 Acked 6751 Write 7 New Done Done false
5753 Acked 6752 Read 5 New Done Done false
5754 Acked 6753 Flush 0 New Done Done false
5755 Acked 6754 Read 8 New Done Done false
5756 Acked 6755 Read 4 New Done Done false
5757 Acked 6756 Read 6 New Done Done false
5758 Acked 6757 Write 7 New Done Done false
5759 Acked 6758 Write 9 New Done Done false
5760 Acked 6759 Read 10 New Done Done false
5761 Acked 6760 Write 1 New Done Done false
5762 Acked 6761 Write 4 New Done Done false
5763 Acked 6762 Write 3 New Done Done false
5764 Acked 6763 Write 1 New Done Done false
5765 Acked 6764 Read 3 New Done Done false
5766 Acked 6765 Read 6 New Done Done false
5767 Acked 6766 Read 6 New Done Done false
5768 Acked 6767 Read 8 New Done Done false
5769 Acked 6768 Read 2 New Done Done false
5770 Acked 6769 Write 5 New Done Done false
5771 Acked 6770 Flush 0 New Done Done false
5772 Acked 6771 Read 10 New Done Done false
5773 Acked 6772 Write 10 New Done Done false
5774 Acked 6773 Read 7 New Done Done false
5775 Acked 6774 Read 9 New Done Done false
5776 Acked 6775 Write 7 New Done Done false
5777 Acked 6776 Write 5 New Done Done false
5778 Acked 6777 Write 1 New Done Done false
5779 Acked 6778 Write 5 New Done Done false
5780 Acked 6779 Write 4 New Done Done false
5781 Acked 6780 Flush 0 New Done Done false
5782 Acked 6781 Write 1 New Done Done false
5783 Acked 6782 Read 2 New Done Done false
5784 Acked 6783 Write 10 New Done Done false
5785 Acked 6784 Read 4 New Done Done false
5786 Acked 6785 Read 3 New Done Done false
5787 Acked 6786 Read 9 New Done Done false
5788 Acked 6787 Read 5 New Done Done false
5789 Acked 6788 Read 7 New Done Done false
5790 Acked 6789 Write 9 New Done Done false
5791 Acked 6790 Read 8 New Done Done false
5792 Acked 6791 Write 6 New Done Done false
5793 Acked 6792 Read 4 New Done Done false
5794 Acked 6793 Write 1 New Done Done false
5795 Acked 6794 Read 4 New Done Done false
5796 Acked 6795 Read 6 New Done Done false
5797 Acked 6796 Read 10 New Done Done false
5798 Acked 6797 Read 10 New Done Done false
5799 Acked 6798 Write 2 New Done Done false
5800 Acked 6799 Read 7 New Done Done false
5801 Acked 6800 Read 4 New Done Done false
5802 Acked 6801 Read 10 New Done Done false
5803 Acked 6802 Read 6 New Done Done false
5804 Acked 6803 Flush 0 New Done Done false
5805 Acked 6804 Write 4 New Done Done false
5806 Acked 6805 Read 7 New Done Done false
5807 Acked 6806 Write 10 New Done Done false
5808 Acked 6807 Read 9 New Done Done false
5809 Acked 6808 Read 9 New Done Done false
5810 Acked 6809 Write 7 New Done Done false
5811 Acked 6810 Read 8 New Done Done false
5812 Acked 6811 Write 4 New Done Done false
5813 Acked 6812 Flush 0 New Done Done false
5814 Acked 6813 Read 10 New Done Done false
5815 Acked 6814 Read 7 New Done Done false
5816 Acked 6815 Write 8 New Done Done false
5817 Acked 6816 Write 7 New Done Done false
5818 Acked 6817 Write 4 New Done Done false
5819 Acked 6818 Read 7 New Done Done false
5820 Acked 6819 Write 1 New Done Done false
5821 Acked 6820 Write 4 New Done Done false
5822 Acked 6821 Read 3 New Done Done false
5823 Acked 6822 Read 6 New Done Done false
5824 Acked 6823 Read 8 New Done Done false
5825 Acked 6824 Write 3 New Done Done false
5826 Acked 6825 Write 10 New Done Done false
5827 Acked 6826 Read 4 New Done Done false
5828 Acked 6827 Read 2 New Done Done false
5829 Acked 6828 Read 2 New Done Done false
5830 Acked 6829 Write 3 New Done Done false
5831 Acked 6830 Write 1 New Done Done false
5832 Acked 6831 Write 8 New Done Done false
5833 Acked 6832 Flush 0 New Done Done false
5834 Acked 6833 Write 9 New Done Done false
5835 Acked 6834 Read 3 New Done Done false
5836 Acked 6835 Write 10 New Done Done false
5837 Acked 6836 Read 5 New Done Done false
5838 Acked 6837 Read 4 New Done Done false
5839 Acked 6838 Write 7 New Done Done false
5840 Acked 6839 Read 9 New Done Done false
5841 Acked 6840 Write 9 New Done Done false
5842 Acked 6841 Flush 0 New Done Done false
5843 Acked 6842 Write 2 New Done Done false
5844 Acked 6843 Read 6 New Done Done false
5845 Acked 6844 Read 4 New Done Done false
5846 Acked 6845 Write 2 New Done Done false
5847 Acked 6846 Read 6 New Done Done false
5848 Acked 6847 Write 8 New Done Done false
5849 Acked 6848 Read 8 New Done Done false
5850 Acked 6849 Write 2 New Done Done false
5851 Acked 6850 Read 10 New Done Done false
5852 Acked 6851 Read 9 New Done Done false
5853 Acked 6852 Write 5 New Done Done false
5854 Acked 6853 Write 6 New Done Done false
5855 Acked 6854 Read 7 New Done Done false
5856 Acked 6855 Write 4 New Done Done false
5857 Acked 6856 Read 6 New Done Done false
5858 Acked 6857 Write 1 New Done Done false
5859 Acked 6858 Read 1 New Done Done false
5860 Acked 6859 Write 9 New Done Done false
5861 Acked 6860 Read 9 New Done Done false
5862 Acked 6861 Write 2 New Done Done false
5863 Acked 6862 Read 4 New Done Done false
5864 Acked 6863 Read 2 New Done Done false
5865 Acked 6864 Read 1 New Done Done false
5866 Acked 6865 Read 6 New Done Done false
5867 Acked 6866 Write 4 New Done Done false
5868 Acked 6867 Write 3 New Done Done false
5869 Acked 6868 Write 10 New Done Done false
5870 Acked 6869 Read 2 New Done Done false
5871 Acked 6870 Read 5 New Done Done false
5872 Acked 6871 Write 10 New Done Done false
5873 Acked 6872 Read 9 New Done Done false
5874 Acked 6873 Write 6 New Done Done false
5875 Acked 6874 Write 3 New Done Done false
5876 Acked 6875 Read 4 New Done Done false
5877 Acked 6876 Write 8 New Done Done false
5878 Acked 6877 Read 3 New Done Done false
5879 Acked 6878 Read 9 New Done Done false
5880 Acked 6879 Read 10 New Done Done false
5881 Acked 6880 Write 9 New Done Done false
5882 Acked 6881 Flush 0 New Done Done false
5883 Acked 6882 Read 7 New Done Done false
5884 Acked 6883 Flush 0 New Done Done false
5885 Acked 6884 Read 1 New Done Done false
5886 Acked 6885 Read 7 New Done Done false
5887 Acked 6886 Read 7 New Done Done false
5888 Acked 6887 Flush 0 New Done Done false
5889 Acked 6888 Write 1 New Done Done false
5890 Acked 6889 Read 9 New Done Done false
5891 Acked 6890 Read 2 New Done Done false
5892 Acked 6891 Read 1 New Done Done false
5893 Acked 6892 Read 4 New Done Done false
5894 Acked 6893 Read 4 New Done Done false
5895 Acked 6894 Read 8 New Done Done false
5896 Acked 6895 Write 10 New Done Done false
5897 Acked 6896 Read 9 New Done Done false
5898 Acked 6897 Read 2 New Done Done false
5899 Acked 6898 Write 1 New Done Done false
5900 Acked 6899 Read 10 New Done Done false
5901 Acked 6900 Flush 0 New Done Done false
5902 Acked 6901 Write 5 New Done Done false
5903 Acked 6902 Read 2 New Done Done false
5904 Acked 6903 Write 6 New Done Done false
5905 Acked 6904 Read 9 New Done Done false
5906 Acked 6905 Write 4 New Done Done false
5907 Acked 6906 Write 6 New Done Done false
5908 Acked 6907 Read 8 New Done Done false
5909 Acked 6908 Write 1 New Done Done false
5910 Acked 6909 Read 6 New Done Done false
5911 Acked 6910 Read 1 New Done Done false
5912 Acked 6911 Read 3 New Done Done false
5913 Acked 6912 Read 3 New Done Done false
5914 Acked 6913 Write 5 New Done Done false
5915 Acked 6914 Read 10 New Done Done false
5916 Acked 6915 Flush 0 New Done Done false
5917 Acked 6916 Read 7 New Done Done false
5918 Acked 6917 Write 2 New Done Done false
5919 Acked 6918 Write 2 New Done Done false
5920 Acked 6919 Write 4 New Done Done false
5921 Acked 6920 Write 6 New Done Done false
5922 Acked 6921 Write 3 New Done Done false
5923 Acked 6922 Write 1 New Done Done false
5924 Acked 6923 Write 6 New Done Done false
5925 Acked 6924 Write 5 New Done Done false
5926 Acked 6925 Read 3 New Done Done false
5927 Acked 6926 Read 6 New Done Done false
5928 Acked 6927 Read 6 New Done Done false
5929 Acked 6928 Write 4 New Done Done false
5930 Acked 6929 Write 3 New Done Done false
5931 Acked 6930 Write 2 New Done Done false
5932 Acked 6931 Read 4 New Done Done false
5933 Acked 6932 Read 3 New Done Done false
5934 Acked 6933 Read 6 New Done Done false
5935 Acked 6934 Read 7 New Done Done false
5936 Acked 6935 Flush 0 New Done Done false
5937 Acked 6936 Read 3 New Done Done false
5938 Acked 6937 Read 9 New Done Done false
5939 Acked 6938 Read 7 New Done Done false
5940 Acked 6939 Read 10 New Done Done false
5941 Acked 6940 Read 6 New Done Done false
5942 Acked 6941 Read 7 New Done Done false
5943 Acked 6942 Read 3 New Done Done false
5944 Acked 6943 Write 4 New Done Done false
5945 Acked 6944 Read 10 New Done Done false
5946 Acked 6945 Read 10 New Done Done false
5947 Acked 6946 Write 2 New Done Done false
5948 Acked 6947 Write 2 New Done Done false
5949 Acked 6948 Flush 0 New Done Done false
5950 Acked 6949 Read 2 New Done Done false
5951 Acked 6950 Write 6 New Done Done false
5952 Acked 6951 Read 8 New Done Done false
5953 Acked 6952 Flush 0 New Done Done false
5954 Acked 6953 Write 4 New Done Done false
5955 Acked 6954 Write 4 New Done Done false
5956 Acked 6955 Write 9 New Done Done false
5957 Acked 6956 Write 9 New Done Done false
5958 Acked 6957 Write 10 New Done Done false
5959 Acked 6958 Read 9 New Done Done false
5960 Acked 6959 Read 9 New Done Done false
5961 Acked 6960 Read 3 New Done Done false
5962 Acked 6961 Flush 0 New Done Done false
5963 Acked 6962 Write 5 New Done Done false
5964 Acked 6963 Write 3 New Done Done false
5965 Acked 6964 Read 6 New Done Done false
5966 Acked 6965 Write 4 New Done Done false
5967 Acked 6966 Write 5 New Done Done false
5968 Acked 6967 Read 10 New Done Done false
5969 Acked 6968 Read 3 New Done Done false
5970 Acked 6969 Flush 0 New Done Done false
5971 Acked 6970 Write 9 New Done Done false
5972 Acked 6971 Read 5 New Done Done false
5973 Acked 6972 Write 9 New Done Done false
5974 Acked 6973 Write 7 New Done Done false
5975 Acked 6974 Write 4 New Done Done false
5976 Acked 6975 Read 2 New Done Done false
5977 Acked 6976 Read 3 New Done Done false
5978 Acked 6977 Read 5 New Done Done false
5979 Acked 6978 Write 5 New Done Done false
5980 Acked 6979 Read 1 New Done Done false
5981 Acked 6980 Write 9 New Done Done false
5982 Acked 6981 Read 2 New Done Done false
5983 Acked 6982 Read 5 New Done Done false
5984 Acked 6983 Write 10 New Done Done false
5985 Acked 6984 Write 7 New Done Done false
5986 Acked 6985 Write 1 New Done Done false
5987 Acked 6986 Read 10 New Done Done false
5988 Acked 6987 Write 9 New Done Done false
5989 Acked 6988 Read 4 New Done Done false
5990 Acked 6989 Write 10 New Done Done false
5991 Acked 6990 Write 9 New Done Done false
5992 Acked 6991 Write 2 New Done Done false
5993 Acked 6992 Write 7 New Done Done false
5994 Acked 6993 Write 3 New Done Done false
5995 Acked 6994 Read 3 New Done Done false
5996 Acked 6995 Write 6 New Done Done false
5997 Acked 6996 Read 6 New Done Done false
5998 Acked 6997 Write 6 New Done Done false
5999 Acked 6998 Write 8 New Done Done false
6000 Acked 6999 Read 9 New Done Done false
6001 Acked 7000 Read 2 New Done Done false
6002 Acked 7001 Flush 0 New Done Done false
6003 Acked 7002 Read 1 New Done Done false
6004 Acked 7003 Read 4 New Done Done false
6005 Acked 7004 Write 6 New Done Done false
6006 Acked 7005 Write 8 New Done Done false
6007 Acked 7006 Read 6 New Done Done false
6008 Acked 7007 Write 8 New Done Done false
6009 Acked 7008 Write 10 New Done Done false
6010 Acked 7009 Write 8 New Done Done false
6011 Acked 7010 Read 2 New Done Done false
6012 Acked 7011 Write 1 New Done Done false
6013 Acked 7012 Read 5 New Done Done false
6014 Acked 7013 Write 3 New Done Done false
6015 Acked 7014 Flush 0 New Done Done false
6016 Acked 7015 Read 2 New Done Done false
6017 Acked 7016 Read 6 New Done Done false
6018 Acked 7017 Flush 0 New Done Done false
6019 Acked 7018 Write 10 New Done Done false
6020 Acked 7019 Flush 0 New Done Done false
6021 Acked 7020 Read 8 New Done Done false
6022 Acked 7021 Read 6 New Done Done false
6023 Acked 7022 Read 4 New Done Done false
6024 Acked 7023 Read 10 New Done Done false
6025 Acked 7024 Flush 0 New Done Done false
6026 Acked 7025 Write 8 New Done Done false
6027 Acked 7026 Read 2 New Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
New 300 0 0 300
Sent 0 0 0 0
Done 0 300 300 600
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 6726 7024 7024
Downstairs last five completed: 6726 6725 6724 6723 6722
Upstairs last five completed: 6027 6026 6025 6024 6023
CLIENT: Up:0 ds:300 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:14.623762046Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:14.623832273Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:14.623857592Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:14.623878672Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:14.623899271Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 300 jobs since flush 6726","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:14.62391676Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:14.624010717Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:15.625321705Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Make output file at "/tmp/dsc/downstairs-8810.txt"
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:15.625383213Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810][0] reports Starting
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:15.625406182Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:15.625427831Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:15.62544377Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 6726","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:15.62546159Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:15.625593925Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1184)
[8810][0] reports Running
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:16.627080316Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:16.627152054Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:16.62854913Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:16.628580449Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 6726","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:16.628596478Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 6726","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:16.628762712Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:16.628796021Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:16.62882233Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:16.628849059Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:16.628865858Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 7027 7026 7025 7024 7023
Upstairs last five completed: 6028 6027 6026 6025 6024
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(0)
stop 0
[8810] Got stop action so:false kr:false
Feb 13 21:43:18.415 INFO request completed, latency_us: 338, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 50e80a32-f65c-40ed-843c-a82a02adc723, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:18.415808466Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:18.415879574Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:18.415896913Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:18.415919462Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:43:18.415 INFO request completed{"msg":"[0] client re-new 0 jobs since flush 7027","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:18.415946411Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
, latency_us: 113, response_code: 200, uri: /state/cid/0{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:18.41597529Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
, method: GET, req_id: 785cf66f-c5f8-444c-846f-d6ca429e44ec, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:19.41722359Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:19.417283168Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:19.417307227Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:19.417329116Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:19.417345496Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 7027","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:19.417363175Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:19.417380604Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:20.419331218Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:20.41954898Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:20.419655245Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:20.419716093Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:20.419770451Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 7027","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:20.419868787Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:20.419933245Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:21.421173776Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:21.421219214Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:21.421240364Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:21.421260673Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:21.421276412Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 7027","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:21.421293611Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:21.421311121Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:43:22.417 INFO request completed, latency_us: 230, response_code: 200, uri: /state/cid/0, method: GET, req_id: 1e046c6f-2627-410d-9c11-828ec9578230, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Read block 4756 len 28672
002/300 Read block 869 len 32768
003/300 Write block 736 len 4096 data: 4
004/300 Read block 2716 len 40960
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:22.422421857Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:22.422472555Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:22.422490944Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:22.422513803Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:22.422526923Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 4 jobs since flush 7027","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:22.422540972Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:22.422554752Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
005/300 Read block 2203 len 4096
006/300 Read block 4743 len 28672
007/300 Write block 3750 len 36864 data: 5 4 6 5 5 4 4 3 4
008/300 Write block 4279 len 20480 data: 4 4 4 5 5
009/300 Read block 399 len 36864
010/300 Read block 4950 len 32768
011/300 Write block 2048 len 12288 data: 5 7 7
012/300 Write block 4270 len 12288 data: 3 3 3
013/300 Flush
014/300 Write block 1809 len 32768 data: 3 4 2 2 3 3 3 2
015/300 Read block 1883 len 32768
016/300 Write block 4222 len 36864 data: 6 4 4 3 2 2 4 4 5
017/300 Read block 3509 len 20480
018/300 Read block 1115 len 36864
019/300 Read block 2716 len 12288
020/300 Flush
021/300 Read block 3726 len 28672
022/300 Read block 2942 len 24576
023/300 Write block 2871 len 4096 data: 3
024/300 Read block 648 len 20480
025/300 Read block 3457 len 4096
026/300 Read block 3977 len 32768
027/300 Write block 190 len 32768 data: 5 6 5 5 6 6 4 4
028/300 Read block 3486 len 32768
029/300 Flush
030/300 Write block 3455 len 12288 data: 3 3 3
031/300 Read block 4970 len 12288
032/300 Read block 4452 len 28672
033/300 Read block 614 len 8192
034/300 Write block 1900 len 32768 data: 5 4 2 2 5 5 4 3
035/300 Write block 885 len 20480 data: 9 10 9 7 5
036/300 Read block 592 len 16384
037/300 Flush
038/300 Write block 1012 len 4096 data: 3
039/300 Write block 1617 len 40960 data: 4 5 5 3 3 2 2 2 4 5
040/300 Read block 1345 len 20480
041/300 Write block 3161 len 12288 data: 4 4 3
042/300 Write block 1958 len 12288 data: 3 2 4
043/300 Write block 4843 len 28672 data: 4 5 4 2 3 2 3
044/300 Read block 4118 len 36864
045/300 Read block 3641 len 28672
046/300 Write block 3372 len 8192 data: 9 10
047/300 Read block 4892 len 12288
048/300 Read block 4765 len 16384
049/300 Read block 4365 len 40960
050/300 Flush
051/300 Write block 692 len 8192 data: 4 3
052/300 Read block 2020 len 16384
053/300 Read block 3011 len 24576
054/300 Read block 3402 len 40960
055/300 Read block 4022 len 4096
056/300 Write block 2744 len 20480 data: 5 6 6 6 5
057/300 Read block 2058 len 40960
058/300 Read block 1679 len 12288
059/300 Write block 2859 len 32768 data: 6 5 4 4 3 3 3 3
060/300 Write block 3454 len 36864 data: 2 4 4 4 3 3 3 3 3
061/300 Write block 1601 len 24576 data: 5 6 6 4 5 5
062/300 Read block 24 len 20480
063/300 Read block 1636 len 16384
064/300 Write block 4493 len 20480 data: 6 7 6 6 4
065/300 Write block 2566 len 20480 data: 8 7 5 6 6
066/300 Read block 1181 len 16384
067/300 Flush
068/300 Read block 3174 len 12288
069/300 Read block 3507 len 4096
070/300 Write block 4591 len 40960 data: 4 4 5 5 5 5 5 5 4 4
071/300 Read block 1425 len 16384
072/300 Write block 2989 len 20480 data: 5 6 4 4 4
073/300 Read block 481 len 8192
074/300 Write block 2401 len 36864 data: 4 6 6 6 6 5 5 5 5
075/300 Read block 18 len 40960
076/300 Write block 3982 len 36864 data: 5 5 5 6 6 6 3 3 3
077/300 Read block 3659 len 20480
078/300 Write block 621 len 16384 data: 7 5 5 4
079/300 Write block 2489 len 24576 data: 6 6 4 4 4 4
080/300 Read block 809 len 24576
081/300 Read block 2301 len 20480
082/300 Write block 3959 len 4096 data: 3
083/300 Read block 3922 len 40960
084/300 Write block 3314 len 24576 data: 5 4 4 5 5 4
085/300 Flush
086/300 Read block 3596 len 12288
087/300 Write block 4858 len 36864 data: 3 4 4 4 4 4 5 4 4
088/300 Write block 3128 len 36864 data: 4 5 5 6 5 5 5 6 7
089/300 Read block 3088 len 8192
090/300 Write block 2567 len 12288 data: 8 6 7
091/300 Write block 557 len 16384 data: 3 4 5 5
092/300 Read block 2567 len 36864
093/300 Read block 3510 len 16384
094/300 Flush
095/300 Write block 4399 len 12288 data: 6 6 7
096/300 Read block 4391 len 20480
097/300 Read block 4351 len 8192
098/300 Read block 4666 len 4096
099/300 Read block 3664 len 8192
100/300 Read block 349 len 4096
101/300 Read block 4746 len 8192
102/300 Read block 138 len 8192
103/300 Read block 139 len 36864
104/300 Write block 601 len 20480 data: 3 3 3 6 6
105/300 Read block 4198 len 16384
106/300 Write block 3590 len 4096 data: 7
107/300 Read block 2069 len 36864
108/300 Read block 3520 len 28672
109/300 Read block 4161 len 24576
110/300 Read block 1089 len 32768
111/300 Read block 3832 len 32768
112/300 Read block 4079 len 20480
113/300 Write block 101 len 28672 data: 6 4 5 5 7 6 6
114/300 Read block 1027 len 28672
115/300 Read block 2092 len 28672
116/300 Write block 1523 len 12288 data: 4 4 4
117/300 Read block 4483 len 32768
118/300 Read block 1081 len 28672
119/300 Read block 719 len 32768
120/300 Read block 4706 len 20480
121/300 Read block 4236 len 8192
122/300 Read block 2456 len 12288
123/300 Read block 3647 len 8192
124/300 Write block 3194 len 4096 data: 8
125/300 Write block 4146 len 4096 data: 2
126/300 Write block 4097 len 20480 data: 2 3 4 4 3
127/300 Read block 3064 len 28672
128/300 Flush
129/300 Write block 4425 len 36864 data: 2 2 2 3 3 4 4 4 4
130/300 Read block 1669 len 8192
131/300 Write block 4835 len 4096 data: 7
132/300 Write block 3041 len 20480 data: 5 6 6 6 4
133/300 Write block 4333 len 12288 data: 5 7 7
134/300 Read block 749 len 36864
135/300 Write block 2932 len 32768 data: 6 6 6 4 4 5 4 4
136/300 Read block 1589 len 32768
137/300 Read block 4571 len 32768
138/300 Read block 4194 len 36864
139/300 Write block 2007 len 16384 data: 4 3 7 6
140/300 Flush
141/300 Flush
142/300 Read block 1056 len 40960
143/300 Write block 2982 len 24576 data: 8 7 7 8 7 7
144/300 Read block 4460 len 40960
145/300 Read block 68 len 20480
146/300 Read block 3024 len 12288
147/300 Write block 1907 len 32768 data: 4 3 5 6 6 7 7 5
148/300 Read block 2672 len 8192
149/300 Read block 4000 len 4096
150/300 Read block 3085 len 24576
151/300 Read block 3842 len 40960
152/300 Read block 4022 len 8192
153/300 Read block 3559 len 24576
154/300 Write block 3292 len 28672 data: 2 3 2 3 5 5 5
155/300 Write block 4634 len 40960 data: 5 4 6 6 5 5 4 4 5 3
156/300 Write block 3975 len 24576 data: 3 3 3 4 5 6
157/300 Read block 3036 len 12288
158/300 Read block 2677 len 4096
159/300 Flush
160/300 Read block 3457 len 28672
161/300 Read block 3805 len 28672
162/300 Write block 4135 len 28672 data: 2 2 2 2 2 2 2
163/300 Write block 2025 len 4096 data: 2
164/300 Write block 967 len 36864 data: 6 5 5 5 5 4 4 4 4
165/300 Write block 2109 len 4096 data: 6
166/300 Write block 3104 len 24576 data: 4 5 6 5 5 6
167/300 Read block 4706 len 28672
168/300 Write block 587 len 28672 data: 3 3 3 5 4 4 4
169/300 Read block 4283 len 16384
170/300 Read block 2913 len 36864
171/300 Write block 748 len 8192 data: 6 7
172/300 Read block 3716 len 36864
173/300 Read block 2292 len 32768
174/300 Write block 3195 len 36864 data: 7 9 10 10 8 7 7 5 4
175/300 Read block 455 len 40960
176/300 Read block 1681 len 32768
177/300 Write block 1180 len 36864 data: 4 3 3 3 4 5 5 6 6
178/300 Write block 1639 len 8192 data: 7 8
179/300 Write block 2569 len 8192 data: 8 7
180/300 Read block 830 len 36864
181/300 Write block 3044 len 12288 data: 7 5 4
182/300 Write block 3848 len 32768 data: 4 3 4 6 8 8 9 9
183/300 Read block 610 len 36864
184/300 Write block 60 len 12288 data: 8 9 9
185/300 Read block 2075 len 36864
186/300 Read block 2987 len 32768
187/300 Read block 988 len 16384
188/300 Write block 741 len 8192 data: 8 6
189/300 Write block 3404 len 20480 data: 9 9 9 7 8
190/300 Write block 2124 len 8192 data: 4 4
191/300 Write block 67 len 24576 data: 4 6 6 6 7 5
192/300 Write block 3198 len 16384 data: 11 9 8 8
193/300 Flush
194/300 Read block 2747 len 28672
195/300 Read block 2966 len 20480
196/300 Read block 3775 len 40960
197/300 Write block 4800 len 12288 data: 6 6 6
198/300 Write block 2853 len 12288 data: 7 7 7
199/300 Read block 1099 len 28672
200/300 Write block 1477 len 32768 data: 4 3 3 2 3 4 3 4
201/300 Flush
202/300 Flush
203/300 Read block 3884 len 36864
204/300 Read block 468 len 28672
205/300 Write block 2384 len 12288 data: 4 4 3
206/300 Read block 3348 len 4096
207/300 Write block 588 len 40960 data: 4 4 6 5 5 5 4 4 5 5
208/300 Write block 4808 len 36864 data: 3 4 4 3 4 4 4 4 4
209/300 Write block 945 len 12288 data: 4 3 2
210/300 Read block 659 len 28672
211/300 Read block 2367 len 12288
212/300 Write block 886 len 4096 data: 11
213/300 Write block 2664 len 12288 data: 6 7 6
214/300 Write block 2405 len 4096 data: 7
215/300 Write block 2140 len 28672 data: 6 6 6 5 4 3 4
216/300 Read block 4161 len 32768
217/300 Write block 2555 len 36864 data: 5 6 5 5 6 7 7 7 8
218/300 Write block 904 len 4096 data: 4
219/300 Read block 575 len 32768
220/300 Write block 2563 len 28672 data: 9 6 7 9 9 7 9
221/300 Read block 796 len 20480
222/300 Read block 4251 len 40960
223/300 Write block 4510 len 32768 data: 3 3 4 6 4 4 3 3
224/300 Read block 2543 len 32768
225/300 Write block 54 len 36864 data: 5 5 5 5 6 7 9 10 10
226/300 Read block 3401 len 24576
227/300 Read block 3907 len 12288
228/300 Read block 1721 len 36864
229/300 Write block 850 len 4096 data: 6
230/300 Read block 3438 len 4096
231/300 Read block 2756 len 24576
232/300 Write block 4038 len 4096 data: 6
233/300 Flush
234/300 Flush
235/300 Write block 1830 len 24576 data: 4 4 4 4 3 2
236/300 Read block 3963 len 24576
237/300 Write block 464 len 36864 data: 6 7 7 5 5 4 5 4 4
238/300 Write block 3021 len 16384 data: 4 5 6 5
239/300 Write block 480 len 4096 data: 3
240/300 Read block 2491 len 36864
241/300 Write block 1538 len 32768 data: 4 2 2 2 3 4 4 4
242/300 Write block 1236 len 24576 data: 3 3 3 2 2 2
243/300 Read block 2387 len 20480
244/300 Read block 4811 len 20480
245/300 Write block 4388 len 16384 data: 5 4 5 5
246/300 Read block 2466 len 36864
247/300 Read block 982 len 28672
248/300 Read block 256 len 28672
249/300 Read block 4568 len 40960
250/300 Write block 2769 len 20480 data: 2 2 2 2 3
251/300 Read block 4298 len 16384
252/300 Read block 494 len 36864
253/300 Read block 2984 len 28672
254/300 Read block 4406 len 8192
255/300 Read block 2487 len 24576
256/300 Write block 866 len 4096 data: 7
257/300 Write block 1146 len 20480 data: 5 6 5 5 5
258/300 Write block 4893 len 16384 data: 2 3 3 3
259/300 Read block 1646 len 12288
260/300 Read block 4879 len 16384
261/300 Write block 2060 len 20480 data: 3 3 4 4 4
262/300 Read block 3059 len 4096
263/300 Read block 3456 len 20480
264/300 Read block 1283 len 32768
265/300 Read block 619 len 12288
266/300 Read block 1315 len 32768
267/300 Read block 2916 len 8192
268/300 Read block 3020 len 40960
269/300 Read block 4810 len 16384
270/300 Write block 1171 len 32768 data: 3 4 4 3 4 4 3 4
271/300 Read block 2139 len 40960
272/300 Write block 2547 len 12288 data: 2 3 4
273/300 Flush
274/300 Read block 3145 len 40960
275/300 Read block 1384 len 20480
276/300 Flush
277/300 Flush
278/300 Write block 616 len 8192 data: 3 4
279/300 Write block 1448 len 24576 data: 4 4 4 3 4 5
280/300 Write block 2644 len 40960 data: 4 4 4 4 4 5 6 8 8 6
281/300 Read block 3462 len 12288
282/300 Write block 1470 len 40960 data: 5 4 3 3 4 6 6 5 4 4
283/300 Read block 2989 len 16384
284/300 Read block 1242 len 8192
285/300 Read block 1418 len 28672
286/300 Read block 3922 len 24576
287/300 Write block 4148 len 16384 data: 5 5 5 6
288/300 Read block 305 len 4096
289/300 Flush
290/300 Write block 129 len 20480 data: 6 7 7 7 5
291/300 Read block 4391 len 12288
292/300 Read block 2902 len 40960
293/300 Write block 1145 len 40960 data: 6 6 7 6 6 6 5 5 5 6
294/300 Write block 3902 len 32768 data: 5 5 5 4 4 4 4 5
295/300 Write block 4116 len 36864 data: 7 7 6 6 3 5 4 5 5
296/300 Write block 2043 len 24576 data: 4 2 2 3 4 6
297/300 Write block 2642 len 8192 data: 3 3
298/300 Read block 224 len 40960
299/300 Read block 218 len 40960
300/300 Read block 255 len 12288
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 13 21:43:23.029 INFO request completed, latency_us: 214, response_code: 204, uri: /start/cid/0, method: POST, req_id: 40f9d6c9-6e7a-40a2-a245-244b810f5804, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
6029 Acked 7028 Read 7 New Done Done true
6030 Acked 7029 Read 8 New Done Done true
6031 Acked 7030 Write 1 New Done Done true
6032 Acked 7031 Read 10 New Done Done true
6033 Acked 7032 Read 1 New Done Done false
6034 Acked 7033 Read 7 New Done Done false
6035 Acked 7034 Write 9 New Done Done false
6036 Acked 7035 Write 5 New Done Done false
6037 Acked 7036 Read 9 New Done Done false
6038 Acked 7037 Read 8 New Done Done false
6039 Acked 7038 Write 3 New Done Done false
6040 Acked 7039 Write 3 New Done Done false
6041 Acked 7040 Flush 0 New Done Done false
6042 Acked 7041 Write 8 New Done Done false
6043 Acked 7042 Read 8 New Done Done false
6044 Acked 7043 Write 9 New Done Done false
6045 Acked 7044 Read 5 New Done Done false
6046 Acked 7045 Read 9 New Done Done false
6047 Acked 7046 Read 3 New Done Done false
6048 Acked 7047 Flush 0 New Done Done false
6049 Acked 7048 Read 7 New Done Done false
6050 Acked 7049 Read 6 New Done Done false
6051 Acked 7050 Write 1 New Done Done false
6052 Acked 7051 Read 5 New Done Done false
6053 Acked 7052 Read 1 New Done Done false
6054 Acked 7053 Read 8 New Done Done false
6055 Acked 7054 Write 8 New Done Done false
6056 Acked 7055 Read 8 New Done Done false
6057 Acked 7056 Flush 0 New Done Done false
6058 Acked 7057 Write 3 New Done Done false
6059 Acked 7058 Read 3 New Done Done false
6060 Acked 7059 Read 7 New Done Done false
6061 Acked 7060 Read 2 New Done Done false
6062 Acked 7061 Write 8 New Done Done false
6063 Acked 7062 Write 5 New Done Done false
6064 Acked 7063 Read 4 New Done Done false
6065 Acked 7064 Flush 0 New Done Done false
6066 Acked 7065 Write 1 New Done Done false
6067 Acked 7066 Write 10 New Done Done false
6068 Acked 7067 Read 5 New Done Done false
6069 Acked 7068 Write 3 New Done Done false
6070 Acked 7069 Write 3 New Done Done false
6071 Acked 7070 Write 7 New Done Done false
6072 Acked 7071 Read 9 New Done Done false
6073 Acked 7072 Read 7 New Done Done false
6074 Acked 7073 Write 2 New Done Done false
6075 Acked 7074 Read 3 New Done Done false
6076 Acked 7075 Read 4 New Done Done false
6077 Acked 7076 Read 10 New Done Done false
6078 Acked 7077 Flush 0 New Done Done false
6079 Acked 7078 Write 2 New Done Done false
6080 Acked 7079 Read 4 New Done Done false
6081 Acked 7080 Read 6 New Done Done false
6082 Acked 7081 Read 10 New Done Done false
6083 Acked 7082 Read 1 New Done Done false
6084 Acked 7083 Write 5 New Done Done false
6085 Acked 7084 Read 10 New Done Done false
6086 Acked 7085 Read 3 New Done Done false
6087 Acked 7086 Write 8 New Done Done false
6088 Acked 7087 Write 9 New Done Done false
6089 Acked 7088 Write 6 New Done Done false
6090 Acked 7089 Read 5 New Done Done false
6091 Acked 7090 Read 4 New Done Done false
6092 Acked 7091 Write 5 New Done Done false
6093 Acked 7092 Write 5 New Done Done false
6094 Acked 7093 Read 4 New Done Done false
6095 Acked 7094 Flush 0 New Done Done false
6096 Acked 7095 Read 3 New Done Done false
6097 Acked 7096 Read 1 New Done Done false
6098 Acked 7097 Write 10 New Done Done false
6099 Acked 7098 Read 4 New Done Done false
6100 Acked 7099 Write 5 New Done Done false
6101 Acked 7100 Read 2 New Done Done false
6102 Acked 7101 Write 9 New Done Done false
6103 Acked 7102 Read 10 New Done Done false
6104 Acked 7103 Write 9 New Done Done false
6105 Acked 7104 Read 5 New Done Done false
6106 Acked 7105 Write 4 New Done Done false
6107 Acked 7106 Write 6 New Done Done false
6108 Acked 7107 Read 6 New Done Done false
6109 Acked 7108 Read 5 New Done Done false
6110 Acked 7109 Write 1 New Done Done false
6111 Acked 7110 Read 10 New Done Done false
6112 Acked 7111 Write 6 New Done Done false
6113 Acked 7112 Flush 0 New Done Done false
6114 Acked 7113 Read 3 New Done Done false
6115 Acked 7114 Write 9 New Done Done false
6116 Acked 7115 Write 9 New Done Done false
6117 Acked 7116 Read 2 New Done Done false
6118 Acked 7117 Write 3 New Done Done false
6119 Acked 7118 Write 4 New Done Done false
6120 Acked 7119 Read 9 New Done Done false
6121 Acked 7120 Read 4 New Done Done false
6122 Acked 7121 Flush 0 New Done Done false
6123 Acked 7122 Write 3 New Done Done false
6124 Acked 7123 Read 5 New Done Done false
6125 Acked 7124 Read 2 New Done Done false
6126 Acked 7125 Read 1 New Done Done false
6127 Acked 7126 Read 2 New Done Done false
6128 Acked 7127 Read 1 New Done Done false
6129 Acked 7128 Read 2 New Done Done false
6130 Acked 7129 Read 2 New Done Done false
6131 Acked 7130 Read 9 New Done Done false
6132 Acked 7131 Write 5 New Done Done false
6133 Acked 7132 Read 4 New Done Done false
6134 Acked 7133 Write 1 New Done Done false
6135 Acked 7134 Read 9 New Done Done false
6136 Acked 7135 Read 7 New Done Done false
6137 Acked 7136 Read 6 New Done Done false
6138 Acked 7137 Read 8 New Done Done false
6139 Acked 7138 Read 8 New Done Done false
6140 Acked 7139 Read 5 New Done Done false
6141 Acked 7140 Write 7 New Done Done false
6142 Acked 7141 Read 7 New Done Done false
6143 Acked 7142 Read 7 New Done Done false
6144 Acked 7143 Write 3 New Done Done false
6145 Acked 7144 Read 8 New Done Done false
6146 Acked 7145 Read 7 New Done Done false
6147 Acked 7146 Read 8 New Done Done false
6148 Acked 7147 Read 5 New Done Done false
6149 Acked 7148 Read 2 New Done Done false
6150 Acked 7149 Read 3 New Done Done false
6151 Acked 7150 Read 2 New Done Done false
6152 Acked 7151 Write 1 New Done Done false
6153 Acked 7152 Write 1 New Done Done false
6154 Acked 7153 Write 5 New Done Done false
6155 Acked 7154 Read 7 New Done Done false
6156 Acked 7155 Flush 0 New Done Done false
6157 Acked 7156 Write 9 New Done Done false
6158 Acked 7157 Read 2 New Done Done false
6159 Acked 7158 Write 1 New Done Done false
6160 Acked 7159 Write 5 New Done Done false
6161 Acked 7160 Write 3 New Done Done false
6162 Acked 7161 Read 9 New Done Done false
6163 Acked 7162 Write 8 New Done Done false
6164 Acked 7163 Read 8 New Done Done false
6165 Acked 7164 Read 8 New Done Done false
6166 Acked 7165 Read 9 New Done Done false
6167 Acked 7166 Write 4 New Done Done false
6168 Acked 7167 Flush 0 New Done Done false
6169 Acked 7168 Flush 0 New Done Done false
6170 Acked 7169 Read 10 New Done Done false
6171 Acked 7170 Write 6 New Done Done false
6172 Acked 7171 Read 10 New Done Done false
6173 Acked 7172 Read 5 New Done Done false
6174 Acked 7173 Read 3 New Done Done false
6175 Acked 7174 Write 8 New Done Done false
6176 Acked 7175 Read 2 New Done Done false
6177 Acked 7176 Read 1 New Done Done false
6178 Acked 7177 Read 6 New Done Done false
6179 Acked 7178 Read 10 New Done Done false
6180 Acked 7179 Read 2 New Done Done false
6181 Acked 7180 Read 6 New Done Done false
6182 Acked 7181 Write 7 New Done Done false
6183 Acked 7182 Write 10 New Done Done false
6184 Acked 7183 Write 6 New Done Done false
6185 Acked 7184 Read 3 New Done Done false
6186 Acked 7185 Read 1 New Done Done false
6187 Acked 7186 Flush 0 New Done Done false
6188 Acked 7187 Read 7 New Done Done false
6189 Acked 7188 Read 7 New Done Done false
6190 Acked 7189 Write 7 New Done Done false
6191 Acked 7190 Write 1 New Done Done false
6192 Acked 7191 Write 9 New Done Done false
6193 Acked 7192 Write 1 New Done Done false
6194 Acked 7193 Write 6 New Done Done false
6195 Acked 7194 Read 7 New Done Done false
6196 Acked 7195 Write 7 New Done Done false
6197 Acked 7196 Read 4 New Done Done false
6198 Acked 7197 Read 9 New Done Done false
6199 Acked 7198 Write 2 New Done Done false
6200 Acked 7199 Read 9 New Done Done false
6201 Acked 7200 Read 8 New Done Done false
6202 Acked 7201 Write 9 New Done Done false
6203 Acked 7202 Read 10 New Done Done false
6204 Acked 7203 Read 8 New Done Done false
6205 Acked 7204 Write 9 New Done Done false
6206 Acked 7205 Write 2 New Done Done false
6207 Acked 7206 Write 2 New Done Done false
6208 Acked 7207 Read 9 New Done Done false
6209 Acked 7208 Write 3 New Done Done false
6210 Acked 7209 Write 8 New Done Done false
6211 Acked 7210 Read 9 New Done Done false
6212 Acked 7211 Write 3 New Done Done false
6213 Acked 7212 Read 9 New Done Done false
6214 Acked 7213 Read 8 New Done Done false
6215 Acked 7214 Read 4 New Done Done false
6216 Acked 7215 Write 2 New Done Done false
6217 Acked 7216 Write 5 New Done Done false
6218 Acked 7217 Write 2 New Done Done false
6219 Acked 7218 Write 6 New Done Done false
6220 Acked 7219 Write 4 New Done Done false
6221 Acked 7220 Flush 0 New Done Done false
6222 Acked 7221 Read 7 New Done Done false
6223 Acked 7222 Read 5 New Done Done false
6224 Acked 7223 Read 10 New Done Done false
6225 Acked 7224 Write 3 New Done Done false
6226 Acked 7225 Write 3 New Done Done false
6227 Acked 7226 Read 7 New Done Done false
6228 Acked 7227 Write 8 New Done Done false
6229 Acked 7228 Flush 0 New Done Done false
6230 Acked 7229 Flush 0 New Done Done false
6231 Acked 7230 Read 9 New Done Done false
6232 Acked 7231 Read 7 New Done Done false
6233 Acked 7232 Write 3 New Done Done false
6234 Acked 7233 Read 1 New Done Done false
6235 Acked 7234 Write 10 New Done Done false
6236 Acked 7235 Write 9 New Done Done false
6237 Acked 7236 Write 3 New Done Done false
6238 Acked 7237 Read 7 New Done Done false
6239 Acked 7238 Read 3 New Done Done false
6240 Acked 7239 Write 1 New Done Done false
6241 Acked 7240 Write 3 New Done Done false
6242 Acked 7241 Write 1 New Done Done false
6243 Acked 7242 Write 7 New Done Done false
6244 Acked 7243 Read 8 New Done Done false
6245 Acked 7244 Write 9 New Done Done false
6246 Acked 7245 Write 1 New Done Done false
6247 Acked 7246 Read 8 New Done Done false
6248 Acked 7247 Write 7 New Done Done false
6249 Acked 7248 Read 5 New Done Done false
6250 Acked 7249 Read 10 New Done Done false
6251 Acked 7250 Write 8 New Done Done false
6252 Acked 7251 Read 8 New Done Done false
6253 Acked 7252 Write 9 New Done Done false
6254 Acked 7253 Read 6 New Done Done false
6255 Acked 7254 Read 3 New Done Done false
6256 Acked 7255 Read 9 New Done Done false
6257 Acked 7256 Write 1 New Done Done false
6258 Acked 7257 Read 1 New Done Done false
6259 Acked 7258 Read 6 New Done Done false
6260 Acked 7259 Write 1 New Done Done false
6261 Acked 7260 Flush 0 New Done Done false
6262 Acked 7261 Flush 0 New Done Done false
6263 Acked 7262 Write 6 New Done Done false
6264 Acked 7263 Read 6 New Done Done false
6265 Acked 7264 Write 9 New Done Done false
6266 Acked 7265 Write 4 New Done Done false
6267 Acked 7266 Write 1 New Done Done false
6268 Acked 7267 Read 9 New Done Done false
6269 Acked 7268 Write 8 New Done Done false
6270 Acked 7269 Write 6 New Done Done false
6271 Acked 7270 Read 5 New Done Done false
6272 Acked 7271 Read 5 New Done Done false
6273 Acked 7272 Write 4 New Done Done false
6274 Acked 7273 Read 9 New Done Done false
6275 Acked 7274 Read 7 New Done Done false
6276 Acked 7275 Read 7 New Done Done false
6277 Acked 7276 Read 10 New Done Done false
6278 Acked 7277 Write 5 New Done Done false
6279 Acked 7278 Read 4 New Done Done false
6280 Acked 7279 Read 9 New Done Done false
6281 Acked 7280 Read 7 New Done Done false
6282 Acked 7281 Read 2 New Done Done false
6283 Acked 7282 Read 6 New Done Done false
6284 Acked 7283 Write 1 New Done Done false
6285 Acked 7284 Write 5 New Done Done false
6286 Acked 7285 Write 4 New Done Done false
6287 Acked 7286 Read 3 New Done Done false
6288 Acked 7287 Read 4 New Done Done false
6289 Acked 7288 Write 5 New Done Done false
6290 Acked 7289 Read 1 New Done Done false
6291 Acked 7290 Read 5 New Done Done false
6292 Acked 7291 Read 8 New Done Done false
6293 Acked 7292 Read 3 New Done Done false
6294 Acked 7293 Read 8 New Done Done false
6295 Acked 7294 Read 2 New Done Done false
6296 Acked 7295 Read 10 New Done Done false
6297 Acked 7296 Read 4 New Done Done false
6298 Acked 7297 Write 8 New Done Done false
6299 Acked 7298 Read 10 New Done Done false
6300 Acked 7299 Write 3 New Done Done false
6301 Acked 7300 Flush 0 New Done Done false
6302 Acked 7301 Read 10 New Done Done false
6303 Acked 7302 Read 5 New Done Done false
6304 Acked 7303 Flush 0 New Done Done false
6305 Acked 7304 Flush 0 New Done Done false
6306 Acked 7305 Write 2 New Done Done false
6307 Acked 7306 Write 6 New Done Done false
6308 Acked 7307 Write 10 New Done Done false
6309 Acked 7308 Read 3 New Done Done false
6310 Acked 7309 Write 10 New Done Done false
6311 Acked 7310 Read 4 New Done Done false
6312 Acked 7311 Read 2 New Done Done false
6313 Acked 7312 Read 7 New Done Done false
6314 Acked 7313 Read 6 New Done Done false
6315 Acked 7314 Write 4 New Done Done false
6316 Acked 7315 Read 1 New Done Done false
6317 Acked 7316 Flush 0 New Done Done false
6318 Acked 7317 Write 5 New Done Done false
6319 Acked 7318 Read 3 New Done Done false
6320 Acked 7319 Read 10 New Done Done false
6321 Acked 7320 Write 10 New Done Done false
6322 Acked 7321 Write 8 New Done Done false
6323 Acked 7322 Write 9 New Done Done false
6324 Acked 7323 Write 6 New Done Done false
6325 Acked 7324 Write 2 New Done Done false
6326 Acked 7325 Read 10 New Done Done false
6327 Acked 7326 Read 10 New Done Done false
6328 Acked 7327 Read 3 New Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
New 300 0 0 300
Sent 0 0 0 0
Done 0 300 300 600
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 7027 7316 7316
Downstairs last five completed: 7027 7026 7025 7024 7023
Upstairs last five completed: 6328 6327 6326 6325 6324
CLIENT: Up:0 ds:300 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:23.423960016Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:23.424029733Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:23.424053942Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:23.424074972Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:23.424091111Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 7027","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:23.4241087Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:23.424214626Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:24.424682587Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:24.424741705Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:24.424768384Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:24.424789183Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:24.424804853Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 7027","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:24.424821712Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:24.424918158Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1185)
[8810][0] reports Running
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:25.425500515Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:25.425573012Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:25.426982318Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:25.427013377Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 7027","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:25.427032956Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 7027","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:25.42720947Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:25.427238609Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:25.427256818Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:25.427277717Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:25.427295286Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 7328 7327 7326 7325 7324
Upstairs last five completed: 6329 6328 6327 6326 6325
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(2)
stop 2
Feb 13 21:43:27.032 INFO request completed, latency_us: 394, response_code[8830] Got stop action so:false kr:false
: 204, uri: /stop/cid/2, method: POST, req_id: 2435a4f1-8848-476f-9002-e83513564c8e, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Feb 13 21:43:27.032 INFO request completed, latency_us: 117{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:27.032862155Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
, response_code: 200, uri: /state/cid/2, method: GET, req_id: 77df59bc-a9f8-4a99-aa65-370434e86d3c, remote_addr: 127.0.0.1:52124, {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:27.032961181Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
local_addr: 127.0.0.1:9998
{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:27.03299633Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:27.033013409Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 7328","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:27.033034569Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:27.033053518Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:28.034133165Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:28.034197723Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:28.034220742Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:28.034250591Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:28.03426723Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 7328","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:28.03428505Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:28.034302669Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:29.035917086Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:29.035979864Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:29.036001903Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:29.036022262Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:29.036038052Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 7328","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:29.036055711Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:29.03607288Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:30.036796341Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:30.036859198Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:30.036882767Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:30.036911876Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:30.036928236Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 7328","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:30.036945735Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:30.036964364Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:43:31.034 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/2, method: GET, req_id: 8b63485b-2cd2-4482-8133-e5122ed2b776, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Write block 4719 len 12288 data: 3 3 5
002/300 Read block 1470 len 28672
003/300 Read block 4615 len 12288
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:31.037616447Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:31.037667795Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:31.037686545Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:31.037703354Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:31.037720523Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 3 jobs since flush 7328","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:31.037746602Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:31.037775151Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
004/300 Write block 186 len 32768 data: 5 6 6 5 6 7 6 6
005/300 Read block 4184 len 32768
006/300 Write block 2508 len 32768 data: 8 8 7 6 7 6 6 6
007/300 Flush
008/300 Read block 513 len 28672
009/300 Write block 4510 len 16384 data: 4 4 5 7
010/300 Read block 301 len 40960
011/300 Write block 3984 len 4096 data: 6
012/300 Read block 965 len 32768
013/300 Read block 4246 len 16384
014/300 Read block 3314 len 12288
015/300 Write block 2656 len 20480 data: 5 5 5 5 5
016/300 Write block 729 len 20480 data: 6 7 7 8 8
017/300 Read block 481 len 36864
018/300 Read block 4925 len 4096
019/300 Read block 3109 len 4096
020/300 Read block 1457 len 36864
021/300 Write block 120 len 40960 data: 3 6 6 8 8 8 6 6 5 7
022/300 Write block 2051 len 28672 data: 6 6 5 5 5 4 3
023/300 Flush
024/300 Read block 1054 len 8192
025/300 Read block 1814 len 28672
026/300 Write block 272 len 36864 data: 6 4 3 2 2 2 2 3 3
027/300 Write block 3773 len 28672 data: 5 4 5 6 7 7 6
028/300 Write block 4860 len 8192 data: 5 5
029/300 Flush
030/300 Write block 3898 len 8192 data: 5 5
031/300 Write block 3651 len 36864 data: 5 4 4 5 6 6 5 5 2
032/300 Read block 481 len 12288
033/300 Flush
034/300 Read block 46 len 16384
035/300 Read block 3382 len 24576
036/300 Write block 2201 len 36864 data: 5 5 6 4 5 4 4 4 5
037/300 Write block 3990 len 8192 data: 4 3
038/300 Write block 1382 len 20480 data: 2 3 4 5 5
039/300 Write block 3918 len 20480 data: 4 4 5 4 4
040/300 Read block 3405 len 20480
041/300 Read block 1639 len 16384
042/300 Write block 4862 len 32768 data: 5 5 6 5 5 4 4 3
043/300 Read block 885 len 24576
044/300 Write block 4263 len 20480 data: 6 6 4 3 3
045/300 Write block 4360 len 20480 data: 6 5 6 6 4
046/300 Read block 2898 len 40960
047/300 Write block 223 len 4096 data: 4
048/300 Read block 3180 len 36864
049/300 Read block 3457 len 20480
050/300 Read block 1628 len 12288
051/300 Read block 2141 len 8192
052/300 Write block 3007 len 32768 data: 5 5 5 3 4 4 5 6
053/300 Flush
054/300 Read block 2445 len 36864
055/300 Write block 3257 len 36864 data: 7 8 5 4 4 4 3 5 5
056/300 Read block 89 len 16384
057/300 Write block 4109 len 40960 data: 5 6 6 6 8 8 8 8 8 7
058/300 Write block 3277 len 40960 data: 7 7 6 6 6 5 7 7 7 8
059/300 Read block 3688 len 4096
060/300 Read block 3773 len 40960
061/300 Write block 2734 len 40960 data: 4 4 5 4 4 3 2 2 3 5
062/300 Read block 4573 len 40960
063/300 Flush
064/300 Read block 2637 len 16384
065/300 Read block 3992 len 16384
066/300 Read block 3807 len 16384
067/300 Write block 451 len 32768 data: 7 6 7 6 5 5 5 4
068/300 Write block 1294 len 4096 data: 3
069/300 Write block 1404 len 28672 data: 4 5 5 5 5 5 5
070/300 Write block 4595 len 24576 data: 6 6 6 6 5 5
071/300 Write block 58 len 4096 data: 7
072/300 Write block 686 len 8192 data: 8 8
073/300 Read block 666 len 12288
074/300 Read block 1626 len 16384
075/300 Write block 932 len 28672 data: 3 3 3 4 4 4 3
076/300 Write block 1716 len 8192 data: 4 5
077/300 Read block 93 len 36864
078/300 Read block 3853 len 8192
079/300 Write block 4770 len 12288 data: 5 6 7
080/300 Read block 1240 len 40960
081/300 Read block 1269 len 24576
082/300 Read block 681 len 32768
083/300 Write block 1904 len 16384 data: 6 6 5 5
084/300 Write block 1592 len 40960 data: 5 5 7 7 6 5 5 5 4 6
085/300 Read block 4465 len 4096
086/300 Write block 1497 len 28672 data: 6 5 5 5 3 3 3
087/300 Read block 4792 len 16384
088/300 Write block 3624 len 12288 data: 6 7 6
089/300 Read block 778 len 40960
090/300 Write block 1817 len 20480 data: 3 3 5 8 6
091/300 Write block 4327 len 20480 data: 9 8 7 6 5
092/300 Read block 4918 len 28672
093/300 Write block 2657 len 36864 data: 6 6 6 6 5 7 7 7 8
094/300 Read block 1723 len 32768
095/300 Read block 147 len 16384
096/300 Read block 4599 len 28672
097/300 Write block 2310 len 24576 data: 6 5 5 4 3 3
098/300 Read block 208 len 40960
099/300 Write block 1537 len 20480 data: 6 5 3 3 3
100/300 Write block 1282 len 32768 data: 6 4 4 4 4 3 3 2
101/300 Write block 84 len 32768 data: 5 5 5 5 5 5 7 7
102/300 Write block 3306 len 36864 data: 7 6 6 8 7 7 4 6 6
103/300 Read block 365 len 20480
104/300 Read block 4083 len 12288
105/300 Read block 3901 len 8192
106/300 Read block 814 len 24576
107/300 Flush
108/300 Read block 227 len 24576
109/300 Read block 562 len 28672
110/300 Write block 2272 len 16384 data: 4 3 2 2
111/300 Flush
112/300 Write block 2317 len 16384 data: 3 3 3 3
113/300 Flush
114/300 Flush
115/300 Write block 2295 len 36864 data: 3 3 3 4 5 6 6 8 9
116/300 Flush
117/300 Write block 4268 len 40960 data: 4 3 4 4 4 3 3 3 4 5
118/300 Read block 4857 len 16384
119/300 Read block 2162 len 8192
120/300 Write block 1365 len 24576 data: 3 3 4 5 6 6
121/300 Read block 2243 len 8192
122/300 Write block 2392 len 12288 data: 5 5 5
123/300 Write block 3807 len 36864 data: 5 4 5 5 6 6 6 6 5
124/300 Flush
125/300 Read block 3907 len 32768
126/300 Write block 1883 len 12288 data: 3 2 3
127/300 Write block 4922 len 12288 data: 3 2 2
128/300 Read block 1113 len 4096
129/300 Write block 1216 len 28672 data: 3 2 2 2 3 3 3
130/300 Write block 1408 len 16384 data: 6 6 6 6
131/300 Write block 1747 len 32768 data: 2 2 3 3 4 4 3 2
132/300 Write block 3865 len 4096 data: 3
133/300 Write block 2257 len 28672 data: 8 7 5 5 6 6 5
134/300 Write block 1508 len 4096 data: 3
135/300 Write block 4826 len 32768 data: 3 5 5 5 5 8 7 8
136/300 Write block 3198 len 32768 data: 12 10 9 9 6 5 4 4
137/300 Write block 126 len 24576 data: 7 7 6 8 8 8
138/300 Read block 3433 len 32768
139/300 Write block 4662 len 20480 data: 5 6 7 7 8
140/300 Flush
141/300 Read block 2445 len 32768
142/300 Read block 4090 len 20480
143/300 Write block 2275 len 8192 data: 3 3
144/300 Read block 4015 len 12288
145/300 Flush
146/300 Read block 2747 len 28672
147/300 Read block 347 len 8192
148/300 Write block 3296 len 16384 data: 6 6 6 4
149/300 Read block 3540 len 40960
150/300 Write block 4992 len 16384 data: 5 5 5 4
151/300 Flush
152/300 Read block 4068 len 40960
153/300 Flush
154/300 Read block 3279 len 4096
155/300 Flush
156/300 Write block 4332 len 24576 data: 5 6 8 8 6 6
157/300 Read block 1374 len 28672
158/300 Read block 3598 len 40960
159/300 Read block 43 len 20480
160/300 Read block 3266 len 8192
161/300 Read block 1521 len 24576
162/300 Write block 4082 len 16384 data: 5 6 5 5
163/300 Read block 1783 len 32768
164/300 Write block 4982 len 16384 data: 2 2 3 5
165/300 Read block 2774 len 20480
166/300 Flush
167/300 Write block 4281 len 24576 data: 5 6 6 5 6 7
168/300 Read block 4822 len 28672
169/300 Write block 2619 len 4096 data: 3
170/300 Read block 70 len 32768
171/300 Read block 2975 len 24576
172/300 Read block 1126 len 40960
173/300 Read block 1627 len 16384
174/300 Read block 1846 len 28672
175/300 Write block 4727 len 36864 data: 7 6 7 8 7 6 6 5 4
176/300 Read block 3690 len 16384
177/300 Flush
178/300 Write block 1457 len 16384 data: 4 4 4 5
179/300 Read block 425 len 32768
180/300 Write block 1069 len 32768 data: 5 7 7 7 6 4 5 5
181/300 Read block 1548 len 32768
182/300 Flush
183/300 Read block 1955 len 40960
184/300 Read block 4297 len 12288
185/300 Write block 3950 len 20480 data: 4 5 5 5 5
186/300 Read block 1180 len 12288
187/300 Write block 3990 len 12288 data: 5 4 3
188/300 Write block 1681 len 36864 data: 2 2 2 2 2 3 3 4 4
189/300 Write block 4803 len 36864 data: 7 3 2 3 3 4 5 5 4
190/300 Write block 1630 len 12288 data: 3 5 5
191/300 Read block 2130 len 32768
192/300 Write block 4499 len 8192 data: 5 6
193/300 Write block 1414 len 40960 data: 5 5 5 5 4 5 3 3 2 2
194/300 Read block 832 len 32768
195/300 Write block 888 len 12288 data: 8 6 5
196/300 Read block 1080 len 8192
197/300 Read block 3530 len 28672
198/300 Write block 99 len 12288 data: 4 4 7
199/300 Read block 3247 len 40960
200/300 Read block 2941 len 12288
201/300 Write block 127 len 20480 data: 8 7 9 9 9
202/300 Write block 4444 len 20480 data: 6 5 4 4 5
203/300 Read block 174 len 20480
204/300 Flush
205/300 Write block 2805 len 20480 data: 4 4 5 5 5
206/300 Read block 3385 len 36864
207/300 Read block 536 len 4096
208/300 Write block 1268 len 12288 data: 5 3 3
209/300 Read block 148 len 12288
210/300 Read block 1924 len 16384
211/300 Read block 2117 len 16384
212/300 Write block 2442 len 4096 data: 9
213/300 Write block 492 len 32768 data: 6 7 7 6 3 4 4 5
214/300 Write block 2445 len 40960 data: 4 4 4 4 3 3 3 2 2 2
215/300 Read block 2784 len 8192
216/300 Read block 685 len 36864
217/300 Write block 2814 len 40960 data: 2 3 3 3 3 3 2 4 4 4
218/300 Write block 2444 len 24576 data: 4 5 5 5 5 4
219/300 Read block 3078 len 40960
220/300 Write block 3888 len 28672 data: 6 5 4 4 3 3 5
221/300 Read block 2689 len 36864
222/300 Read block 2518 len 4096
223/300 Read block 3473 len 20480
224/300 Read block 4076 len 8192
225/300 Read block 1093 len 20480
226/300 Read block 3683 len 32768
227/300 Write block 425 len 20480 data: 2 2 3 5 5
228/300 Read block 718 len 8192
229/300 Write block 4119 len 4096 data: 7
230/300 Write block 3992 len 20480 data: 4 3 3 4 4
231/300 Write block 2599 len 4096 data: 7
232/300 Read block 745 len 40960
233/300 Read block 3191 len 20480
234/300 Write block 4817 len 16384 data: 4 4 3 2
235/300 Read block 4425 len 32768
236/300 Read block 2142 len 8192
237/300 Write block 1397 len 8192 data: 4 3
238/300 Read block 2730 len 36864
239/300 Write block 4902 len 4096 data: 5
240/300 Read block 3435 len 4096
241/300 Read block 1340 len 8192
242/300 Read block 1139 len 40960
243/300 Read block 4705 len 40960
244/300 Write block 4145 len 4096 data: 2
245/300 Read block 2429 len 24576
246/300 Flush
247/300 Read block 1644 len 24576
248/300 Read block 3874 len 40960
249/300 Write block 3188 len 16384 data: 4 4 3 4
250/300 Read block 845 len 32768
251/300 Read block 3518 len 40960
252/300 Read block 3959 len 40960
253/300 Write block 1537 len 16384 data: 7 6 4 4
254/300 Read block 191 len 24576
255/300 Flush
256/300 Read block 4354 len 4096
257/300 Read block 3459 len 40960
258/300 Write block 3650 len 8192 data: 6 6
259/300 Read block 2950 len 8192
260/300 Flush
261/300 Write block 2026 len 4096 data: 2
262/300 Write block 3475 len 40960 data: 4 3 4 4 3 3 3 4 4 5
263/300 Read block 2449 len 28672
264/300 Write block 4421 len 4096 data: 6
265/300 Read block 3612 len 40960
266/300 Write block 4304 len 24576 data: 6 5 4 4 5 5
267/300 Read block 1663 len 32768
268/300 Read block 756 len 24576
269/300 Read block 2128 len 4096
270/300 Read block 1682 len 24576
271/300 Read block 498 len 32768
272/300 Write block 839 len 36864 data: 4 4 4 6 6 6 6 6 6
273/300 Write block 1854 len 28672 data: 4 3 4 3 3 3 3
274/300 Read block 2892 len 40960
275/300 Write block 572 len 32768 data: 3 4 3 3 3 3 3 3
276/300 Read block 2627 len 12288
277/300 Read block 3834 len 20480
278/300 Read block 2679 len 24576
279/300 Write block 896 len 16384 data: 5 5 5 4
280/300 Write block 2480 len 28672 data: 4 3 5 5 6 7 8
281/300 Write block 1608 len 12288 data: 4 4 4
282/300 Write block 42 len 16384 data: 5 5 5 6
283/300 Read block 1650 len 32768
284/300 Read block 805 len 36864
285/300 Read block 4011 len 24576
286/300 Write block 4600 len 32768 data: 6 4 4 3 3 4 4 4
287/300 Write block 1524 len 24576 data: 5 5 3 3 3 3
288/300 Write block 3885 len 20480 data: 6 7 5 7 6
289/300 Write block 876 len 24576 data: 5 5 5 3 4 5
290/300 Read block 130 len 24576
291/300 Write block 1625 len 12288 data: 5 6 5
292/300 Read block 3771 len 4096
293/300 Write block 924 len 4096 data: 7
294/300 Write block 362 len 24576 data: 7 7 6 5 4 3
295/300 Read block 1623 len 36864
296/300 Flush
297/300 Read block 629 len 24576
298/300 Read block 4492 len 20480
299/300 Read block 3551 len 4096
300/300 Write block 147 len 8192 data: 4 4
Main task has work to do, go find it
got dsc Start(2)
start 2
Feb 13 21:43:31.813 INFO request completed, latency_us: 210, response_code: 204, uri: /start/cid/2, method: POST, req_id: 79057975-2076-4007-9416-c0a9a12be3e1, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
States: Active Active Offline
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
6330 Acked 7329 Write 3 Done Done New true
6331 Acked 7330 Read 7 Done Done New true
6332 Acked 7331 Read 3 Done Done New true
6333 Acked 7332 Write 8 Done Done New false
6334 Acked 7333 Read 8 Done Done New false
6335 Acked 7334 Write 8 Done Done New false
6336 Acked 7335 Flush 0 Done Done New false
6337 Acked 7336 Read 7 Done Done New false
6338 Acked 7337 Write 4 Done Done New false
6339 Acked 7338 Read 10 Done Done New false
6340 Acked 7339 Write 1 Done Done New false
6341 Acked 7340 Read 8 Done Done New false
6342 Acked 7341 Read 4 Done Done New false
6343 Acked 7342 Read 3 Done Done New false
6344 Acked 7343 Write 5 Done Done New false
6345 Acked 7344 Write 5 Done Done New false
6346 Acked 7345 Read 9 Done Done New false
6347 Acked 7346 Read 1 Done Done New false
6348 Acked 7347 Read 1 Done Done New false
6349 Acked 7348 Read 9 Done Done New false
6350 Acked 7349 Write 10 Done Done New false
6351 Acked 7350 Write 7 Done Done New false
6352 Acked 7351 Flush 0 Done Done New false
6353 Acked 7352 Read 2 Done Done New false
6354 Acked 7353 Read 7 Done Done New false
6355 Acked 7354 Write 9 Done Done New false
6356 Acked 7355 Write 7 Done Done New false
6357 Acked 7356 Write 2 Done Done New false
6358 Acked 7357 Flush 0 Done Done New false
6359 Acked 7358 Write 2 Done Done New false
6360 Acked 7359 Write 9 Done Done New false
6361 Acked 7360 Read 3 Done Done New false
6362 Acked 7361 Flush 0 Done Done New false
6363 Acked 7362 Read 4 Done Done New false
6364 Acked 7363 Read 6 Done Done New false
6365 Acked 7364 Write 9 Done Done New false
6366 Acked 7365 Write 2 Done Done New false
6367 Acked 7366 Write 5 Done Done New false
6368 Acked 7367 Write 5 Done Done New false
6369 Acked 7368 Read 5 Done Done New false
6370 Acked 7369 Read 4 Done Done New false
6371 Acked 7370 Write 8 Done Done New false
6372 Acked 7371 Read 6 Done Done New false
6373 Acked 7372 Write 5 Done Done New false
6374 Acked 7373 Write 5 Done Done New false
6375 Acked 7374 Read 10 Done Done New false
6376 Acked 7375 Write 1 Done Done New false
6377 Acked 7376 Read 9 Done Done New false
6378 Acked 7377 Read 5 Done Done New false
6379 Acked 7378 Read 3 Done Done New false
6380 Acked 7379 Read 2 Done Done New false
6381 Acked 7380 Write 8 Done Done New false
6382 Acked 7381 Flush 0 Done Done New false
6383 Acked 7382 Read 9 Done Done New false
6384 Acked 7383 Write 9 Done Done New false
6385 Acked 7384 Read 4 Done Done New false
6386 Acked 7385 Write 10 Done Done New false
6387 Acked 7386 Write 10 Done Done New false
6388 Acked 7387 Read 1 Done Done New false
6389 Acked 7388 Read 10 Done Done New false
6390 Acked 7389 Write 10 Done Done New false
6391 Acked 7390 Read 10 Done Done New false
6392 Acked 7391 Flush 0 Done Done New false
6393 Acked 7392 Read 4 Done Done New false
6394 Acked 7393 Read 4 Done Done New false
6395 Acked 7394 Read 4 Done Done New false
6396 Acked 7395 Write 8 Done Done New false
6397 Acked 7396 Write 1 Done Done New false
6398 Acked 7397 Write 7 Done Done New false
6399 Acked 7398 Write 6 Done Done New false
6400 Acked 7399 Write 1 Done Done New false
6401 Acked 7400 Write 2 Done Done New false
6402 Acked 7401 Read 3 Done Done New false
6403 Acked 7402 Read 4 Done Done New false
6404 Acked 7403 Write 7 Done Done New false
6405 Acked 7404 Write 2 Done Done New false
6406 Acked 7405 Read 9 Done Done New false
6407 Acked 7406 Read 2 Done Done New false
6408 Acked 7407 Write 3 Done Done New false
6409 Acked 7408 Read 10 Done Done New false
6410 Acked 7409 Read 6 Done Done New false
6411 Acked 7410 Read 8 Done Done New false
6412 Acked 7411 Write 4 Done Done New false
6413 Acked 7412 Write 10 Done Done New false
6414 Acked 7413 Read 1 Done Done New false
6415 Acked 7414 Write 7 Done Done New false
6416 Acked 7415 Read 4 Done Done New false
6417 Acked 7416 Write 3 Done Done New false
6418 Acked 7417 Read 10 Done Done New false
6419 Acked 7418 Write 5 Done Done New false
6420 Acked 7419 Write 5 Done Done New false
6421 Acked 7420 Read 7 Done Done New false
6422 Acked 7421 Write 9 Done Done New false
6423 Acked 7422 Read 8 Done Done New false
6424 Acked 7423 Read 4 Done Done New false
6425 Acked 7424 Read 7 Done Done New false
6426 Acked 7425 Write 6 Done Done New false
6427 Acked 7426 Read 10 Done Done New false
6428 Acked 7427 Write 5 Done Done New false
6429 Acked 7428 Write 8 Done Done New false
6430 Acked 7429 Write 8 Done Done New false
6431 Acked 7430 Write 9 Done Done New false
6432 Acked 7431 Read 5 Done Done New false
6433 Acked 7432 Read 3 Done Done New false
6434 Acked 7433 Read 2 Done Done New false
6435 Acked 7434 Read 6 Done Done New false
6436 Acked 7435 Flush 0 Done Done New false
6437 Acked 7436 Read 6 Done Done New false
6438 Acked 7437 Read 7 Done Done New false
6439 Acked 7438 Write 4 Done Done New false
6440 Acked 7439 Flush 0 Done Done New false
6441 Acked 7440 Write 4 Done Done New false
6442 Acked 7441 Flush 0 Done Done New false
6443 Acked 7442 Flush 0 Done Done New false
6444 Acked 7443 Write 9 Done Done New false
6445 Acked 7444 Flush 0 Done Done New false
6446 Acked 7445 Write 10 Done Done New false
6447 Acked 7446 Read 4 Done Done New false
6448 Acked 7447 Read 2 Done Done New false
6449 Acked 7448 Write 6 Done Done New false
6450 Acked 7449 Read 2 Done Done New false
6451 Acked 7450 Write 3 Done Done New false
6452 Acked 7451 Write 9 Done Done New false
6453 Acked 7452 Flush 0 Done Done New false
6454 Acked 7453 Read 8 Done Done New false
6455 Acked 7454 Write 3 Done Done New false
6456 Acked 7455 Write 3 Done Done New false
6457 Acked 7456 Read 1 Done Done New false
6458 Acked 7457 Write 7 Done Done New false
6459 Acked 7458 Write 4 Done Done New false
6460 Acked 7459 Write 8 Done Done New false
6461 Acked 7460 Write 1 Done Done New false
6462 Acked 7461 Write 7 Done Done New false
6463 Acked 7462 Write 1 Done Done New false
6464 Acked 7463 Write 8 Done Done New false
6465 Acked 7464 Write 8 Done Done New false
6466 Acked 7465 Write 6 Done Done New false
6467 Acked 7466 Read 8 Done Done New false
6468 Acked 7467 Write 5 Done Done New false
6469 Acked 7468 Flush 0 Done Done New false
6470 Acked 7469 Read 8 Done Done New false
6471 Acked 7470 Read 5 Done Done New false
6472 Acked 7471 Write 2 Done Done New false
6473 Acked 7472 Read 3 Done Done New false
6474 Acked 7473 Flush 0 Done Done New false
6475 Acked 7474 Read 7 Done Done New false
6476 Acked 7475 Read 2 Done Done New false
6477 Acked 7476 Write 4 Done Done New false
6478 Acked 7477 Read 10 Done Done New false
6479 Acked 7478 Write 4 Done Done New false
6480 Acked 7479 Flush 0 Done Done New false
6481 Acked 7480 Read 10 Done Done New false
6482 Acked 7481 Flush 0 Done Done New false
6483 Acked 7482 Read 1 Done Done New false
6484 Acked 7483 Flush 0 Done Done New false
6485 Acked 7484 Write 6 Done Done New false
6486 Acked 7485 Read 7 Done Done New false
6487 Acked 7486 Read 10 Done Done New false
6488 Acked 7487 Read 5 Done Done New false
6489 Acked 7488 Read 2 Done Done New false
6490 Acked 7489 Read 6 Done Done New false
6491 Acked 7490 Write 4 Done Done New false
6492 Acked 7491 Read 8 Done Done New false
6493 Acked 7492 Write 4 Done Done New false
6494 Acked 7493 Read 5 Done Done New false
6495 Acked 7494 Flush 0 Done Done New false
6496 Acked 7495 Write 6 Done Done New false
6497 Acked 7496 Read 7 Done Done New false
6498 Acked 7497 Write 1 Done Done New false
6499 Acked 7498 Read 8 Done Done New false
6500 Acked 7499 Read 6 Done Done New false
6501 Acked 7500 Read 10 Done Done New false
6502 Acked 7501 Read 4 Done Done New false
6503 Acked 7502 Read 7 Done Done New false
6504 Acked 7503 Write 9 Done Done New false
6505 Acked 7504 Read 4 Done Done New false
6506 Acked 7505 Flush 0 Done Done New false
6507 Acked 7506 Write 4 Done Done New false
6508 Acked 7507 Read 8 Done Done New false
6509 Acked 7508 Write 8 Done Done New false
6510 Acked 7509 Read 8 Done Done New false
6511 Acked 7510 Flush 0 Done Done New false
6512 Acked 7511 Read 10 Done Done New false
6513 Acked 7512 Read 3 Done Done New false
6514 Acked 7513 Write 5 Done Done New false
6515 Acked 7514 Read 3 Done Done New false
6516 Acked 7515 Write 3 Done Done New false
6517 Acked 7516 Write 9 Done Done New false
6518 Acked 7517 Write 9 Done Done New false
6519 Acked 7518 Write 3 Done Done New false
6520 Acked 7519 Read 8 Done Done New false
6521 Acked 7520 Write 2 Done Done New false
6522 Acked 7521 Write 10 Done Done New false
6523 Acked 7522 Read 8 Done Done New false
6524 Acked 7523 Write 3 Done Done New false
6525 Acked 7524 Read 2 Done Done New false
6526 Acked 7525 Read 7 Done Done New false
6527 Acked 7526 Write 3 Done Done New false
6528 Acked 7527 Read 10 Done Done New false
6529 Acked 7528 Read 3 Done Done New false
6530 Acked 7529 Write 5 Done Done New false
6531 Acked 7530 Write 5 Done Done New false
6532 Acked 7531 Read 5 Done Done New false
6533 Acked 7532 Flush 0 Done Done New false
6534 Acked 7533 Write 5 Done Done New false
6535 Acked 7534 Read 9 Done Done New false
6536 Acked 7535 Read 1 Done Done New false
6537 Acked 7536 Write 3 Done Done New false
6538 Acked 7537 Read 3 Done Done New false
6539 Acked 7538 Read 4 Done Done New false
6540 Acked 7539 Read 4 Done Done New false
6541 Acked 7540 Write 1 Done Done New false
6542 Acked 7541 Write 8 Done Done New false
6543 Acked 7542 Write 10 Done Done New false
6544 Acked 7543 Read 2 Done Done New false
6545 Acked 7544 Read 9 Done Done New false
6546 Acked 7545 Write 10 Done Done New false
6547 Acked 7546 Write 6 Done Done New false
6548 Acked 7547 Read 10 Done Done New false
6549 Acked 7548 Write 7 Done Done New false
6550 Acked 7549 Read 9 Done Done New false
6551 Acked 7550 Read 1 Done Done New false
6552 Acked 7551 Read 5 Done Done New false
6553 Acked 7552 Read 2 Done Done New false
6554 Acked 7553 Read 5 Done Done New false
6555 Acked 7554 Read 8 Done Done New false
6556 Acked 7555 Write 5 Done Done New false
6557 Acked 7556 Read 2 Done Done New false
6558 Acked 7557 Write 1 Done Done New false
6559 Acked 7558 Write 5 Done Done New false
6560 Acked 7559 Write 1 Done Done New false
6561 Acked 7560 Read 10 Done Done New false
6562 Acked 7561 Read 5 Done Done New false
6563 Acked 7562 Write 4 Done Done New false
6564 Acked 7563 Read 8 Done Done New false
6565 Acked 7564 Read 2 Done Done New false
6566 Acked 7565 Write 2 Done Done New false
6567 Acked 7566 Read 9 Done Done New false
6568 Acked 7567 Write 1 Done Done New false
6569 Acked 7568 Read 1 Done Done New false
6570 Acked 7569 Read 2 Done Done New false
6571 Acked 7570 Read 10 Done Done New false
6572 Acked 7571 Read 10 Done Done New false
6573 Acked 7572 Write 1 Done Done New false
6574 Acked 7573 Read 6 Done Done New false
6575 Acked 7574 Flush 0 Done Done New false
6576 Acked 7575 Read 6 Done Done New false
6577 Acked 7576 Read 10 Done Done New false
6578 Acked 7577 Write 4 Done Done New false
6579 Acked 7578 Read 8 Done Done New false
6580 Acked 7579 Read 10 Done Done New false
6581 Acked 7580 Read 10 Done Done New false
6582 Acked 7581 Write 4 Done Done New false
6583 Acked 7582 Read 6 Done Done New false
6584 Acked 7583 Flush 0 Done Done New false
6585 Acked 7584 Read 1 Done Done New false
6586 Acked 7585 Read 10 Done Done New false
6587 Acked 7586 Write 2 Done Done New false
6588 Acked 7587 Read 2 Done Done New false
6589 Acked 7588 Flush 0 Done Done New false
6590 Acked 7589 Write 1 Done Done New false
6591 Acked 7590 Write 10 Done Done New false
6592 Acked 7591 Read 7 Done Done New false
6593 Acked 7592 Write 1 Done Done New false
6594 Acked 7593 Read 10 Done Done New false
6595 Acked 7594 Write 6 Done Done New false
6596 Acked 7595 Read 8 Done Done New false
6597 Acked 7596 Read 6 Done Done New false
6598 Acked 7597 Read 1 Done Done New false
6599 Acked 7598 Read 6 Done Done New false
6600 Acked 7599 Read 8 Done Done New false
6601 Acked 7600 Write 9 Done Done New false
6602 Acked 7601 Write 7 Done Done New false
6603 Acked 7602 Read 10 Done Done New false
6604 Acked 7603 Write 8 Done Done New false
6605 Acked 7604 Read 3 Done Done New false
6606 Acked 7605 Read 5 Done Done New false
6607 Acked 7606 Read 6 Done Done New false
6608 Acked 7607 Write 4 Done Done New false
6609 Acked 7608 Write 7 Done Done New false
6610 Acked 7609 Write 3 Done Done New false
6611 Acked 7610 Write 4 Done Done New false
6612 Acked 7611 Read 8 Done Done New false
6613 Acked 7612 Read 9 Done Done New false
6614 Acked 7613 Read 6 Done Done New false
6615 Acked 7614 Write 8 Done Done New false
6616 Acked 7615 Write 6 Done Done New false
6617 Acked 7616 Write 5 Done Done New false
6618 Acked 7617 Write 6 Done Done New false
6619 Acked 7618 Read 6 Done Done New false
6620 Acked 7619 Write 3 Done Done New false
6621 Acked 7620 Read 1 Done Done New false
6622 Acked 7621 Write 1 Done Done New false
6623 Acked 7622 Write 6 Done Done New false
6624 Acked 7623 Read 9 Done Done New false
6625 Acked 7624 Flush 0 Done Done New false
6626 Acked 7625 Read 6 Done Done New false
6627 Acked 7626 Read 5 Done Done New false
6628 Acked 7627 Read 1 Done Done New false
6629 Acked 7628 Write 2 Done Done New false
STATES DS:0 DS:1 DS:2 TOTAL
New 0 0 300 300
Sent 0 0 0 0
Done 300 300 0 600
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 7624 7624 7328
Downstairs last five completed: 7328 7327 7326 7325 7324
Upstairs last five completed: 6629 6628 6627 6626 6625
CLIENT: Up:0 ds:300 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:32.038422355Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:32.038480072Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:32.038503331Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:32.038524151Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:32.03854018Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 301 jobs since flush 7328","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:32.038558149Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:32.038613727Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:33.039710664Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:33.039770002Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:33.039792191Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:33.03981245Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:33.03982884Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 301 jobs since flush 7328","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:33.039846329Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:33.039978344Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:34.040545081Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:34.040593089Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:34.040614359Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:34.040634778Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:34.040650597Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 301 jobs since flush 7328","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:34.040667937Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:34.040823651Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:35.042380949Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:35.042442787Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:35.042466776Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:35.042487635Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:35.042511144Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 301 jobs since flush 7328","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:35.042530124Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:35.04264099Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1186)
[8830][2] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
6330 Acked 7329 Write 3 Done Done New true
6331 Acked 7330 Read 7 Done Done New true
6332 Acked 7331 Read 3 Done Done New true
6333 Acked 7332 Write 8 Done Done New true
6334 Acked 7333 Read 8 Done Done New true
6335 Acked 7334 Write 8 Done Done New true
6336 Acked 7335 Flush 0 Done Done New true
6337 Acked 7336 Read 7 Done Done New true
6338 Acked 7337 Write 4 Done Done New true
6339 Acked 7338 Read 10 Done Done New true
6340 Acked 7339 Write 1 Done Done New true
6341 Acked 7340 Read 8 Done Done New true
6342 Acked 7341 Read 4 Done Done New true
6343 Acked 7342 Read 3 Done Done New true
6344 Acked 7343 Write 5 Done Done New true
6345 Acked 7344 Write 5 Done Done New true
6346 Acked 7345 Read 9 Done Done New true
6347 Acked 7346 Read 1 Done Done New true
6348 Acked 7347 Read 1 Done Done New true
6349 Acked 7348 Read 9 Done Done New true
6350 Acked 7349 Write 10 Done Done New true
6351 Acked 7350 Write 7 Done Done New true
6352 Acked 7351 Flush 0 Done Done New true
6353 Acked 7352 Read 2 Done Done New true
6354 Acked 7353 Read 7 Done Done New true
6355 Acked 7354 Write 9 Done Done New true
6356 Acked 7355 Write 7 Done Done New true
6357 Acked 7356 Write 2 Done Done New true
6358 Acked 7357 Flush 0 Done Done New true
6359 Acked 7358 Write 2 Done Done New true
6360 Acked 7359 Write 9 Done Done New true
6361 Acked 7360 Read 3 Done Done New true
6362 Acked 7361 Flush 0 Done Done New true
6363 Acked 7362 Read 4 Done Done New true
6364 Acked 7363 Read 6 Done Done New true
6365 Acked 7364 Write 9 Done Done New true
6366 Acked 7365 Write 2 Done Done New true
6367 Acked 7366 Write 5 Done Done New true
6368 Acked 7367 Write 5 Done Done New true
6369 Acked 7368 Read 5 Done Done New true
6370 Acked 7369 Read 4 Done Done New true
6371 Acked 7370 Write 8 Done Done New true
6372 Acked 7371 Read 6 Done Done New true
6373 Acked 7372 Write 5 Done Done New true
6374 Acked 7373 Write 5 Done Done New true
6375 Acked 7374 Read 10 Done Done New true
6376 Acked 7375 Write 1 Done Done New true
6377 Acked 7376 Read 9 Done Done New true
6378 Acked 7377 Read 5 Done Done New true
6379 Acked 7378 Read 3 Done Done New true
6380 Acked 7379 Read 2 Done Done New true
6381 Acked 7380 Write 8 Done Done New true
6382 Acked 7381 Flush 0 Done Done New true
6383 Acked 7382 Read 9 Done Done New true
6384 Acked 7383 Write 9 Done Done New true
6385 Acked 7384 Read 4 Done Done New true
6386 Acked 7385 Write 10 Done Done New true
6387 Acked 7386 Write 10 Done Done New true
6388 Acked 7387 Read 1 Done Done New true
6389 Acked 7388 Read 10 Done Done New true
6390 Acked 7389 Write 10 Done Done New true
6391 Acked 7390 Read 10 Done Done New true
6392 Acked 7391 Flush 0 Done Done New true
6393 Acked 7392 Read 4 Done Done New true
6394 Acked 7393 Read 4 Done Done New true
6395 Acked 7394 Read 4 Done Done New true
6396 Acked 7395 Write 8 Done Done New true
6397 Acked 7396 Write 1 Done Done New true
6398 Acked 7397 Write 7 Done Done New true
6399 Acked 7398 Write 6 Done Done New true
6400 Acked 7399 Write 1 Done Done New true
6401 Acked 7400 Write 2 Done Done New true
6402 Acked 7401 Read 3 Done Done New true
6403 Acked 7402 Read 4 Done Done New true
6404 Acked 7403 Write 7 Done Done New true
6405 Acked 7404 Write 2 Done Done New true
6406 Acked 7405 Read 9 Done Done New true
6407 Acked 7406 Read 2 Done Done New true
6408 Acked 7407 Write 3 Done Done New true
6409 Acked 7408 Read 10 Done Done New true
6410 Acked 7409 Read 6 Done Done New true
6411 Acked 7410 Read 8 Done Done New true
6412 Acked 7411 Write 4 Done Done New true
6413 Acked 7412 Write 10 Done Done New true
6414 Acked 7413 Read 1 Done Done New true
6415 Acked 7414 Write 7 Done Done New true
6416 Acked 7415 Read 4 Done Done New true
6417 Acked 7416 Write 3 Done Done New true
6418 Acked 7417 Read 10 Done Done New true
6419 Acked 7418 Write 5 Done Done New true
6420 Acked 7419 Write 5 Done Done New true
6421 Acked 7420 Read 7 Done Done New true
6422 Acked 7421 Write 9 Done Done New true
6423 Acked 7422 Read 8 Done Done New true
6424 Acked 7423 Read 4 Done Done New true
6425 Acked 7424 Read 7 Done Done New true
6426 Acked 7425 Write 6 Done Done New true
6427 Acked 7426 Read 10 Done Done New true
6428 Acked 7427 Write 5 Done Done New true
6429 Acked 7428 Write 8 Done Done New true
6430 Acked 7429 Write 8 Done Done New true
6431 Acked 7430 Write 9 Done Done New true
6432 Acked 7431 Read 5 Done Done New true
6433 Acked 7432 Read 3 Done Done New true
6434 Acked 7433 Read 2 Done Done New true
6435 Acked 7434 Read 6 Done Done New true
6436 Acked 7435 Flush 0 Done Done New true
6437 Acked 7436 Read 6 Done Done New true
6438 Acked 7437 Read 7 Done Done New true
6439 Acked 7438 Write 4 Done Done New true
6440 Acked 7439 Flush 0 Done Done New true
6441 Acked 7440 Write 4 Done Done New true
6442 Acked 7441 Flush 0 Done Done New true
6443 Acked 7442 Flush 0 Done Done New true
6444 Acked 7443 Write 9 Done Done New true
6445 Acked 7444 Flush 0 Done Done New true
6446 Acked 7445 Write 10 Done Done New true
6447 Acked 7446 Read 4 Done Done New true
6448 Acked 7447 Read 2 Done Done New true
6449 Acked 7448 Write 6 Done Done New true
6450 Acked 7449 Read 2 Done Done New true
6451 Acked 7450 Write 3 Done Done New true
6452 Acked 7451 Write 9 Done Done New true
6453 Acked 7452 Flush 0 Done Done New true
6454 Acked 7453 Read 8 Done Done New true
6455 Acked 7454 Write 3 Done Done New true
6456 Acked 7455 Write 3 Done Done New true
6457 Acked 7456 Read 1 Done Done New true
6458 Acked 7457 Write 7 Done Done New true
6459 Acked 7458 Write 4 Done Done New true
6460 Acked 7459 Write 8 Done Done New true
6461 Acked 7460 Write 1 Done Done New true
6462 Acked 7461 Write 7 Done Done New true
6463 Acked 7462 Write 1 Done Done New true
6464 Acked 7463 Write 8 Done Done New true
6465 Acked 7464 Write 8 Done Done New true
6466 Acked 7465 Write 6 Done Done New true
6467 Acked 7466 Read 8 Done Done New true
6468 Acked 7467 Write 5 Done Done New true
6469 Acked 7468 Flush 0 Done Done New true
6470 Acked 7469 Read 8 Done Done New true
6471 Acked 7470 Read 5 Done Done New true
6472 Acked 7471 Write 2 Done Done New true
6473 Acked 7472 Read 3 Done Done New true
6474 Acked 7473 Flush 0 Done Done New true
6475 Acked 7474 Read 7 Done Done New true
6476 Acked 7475 Read 2 Done Done New true
6477 Acked 7476 Write 4 Done Done New true
6478 Acked 7477 Read 10 Done Done New true
6479 Acked 7478 Write 4 Done Done New true
6480 Acked 7479 Flush 0 Done Done New true
6481 Acked 7480 Read 10 Done Done New true
6482 Acked 7481 Flush 0 Done Done New true
6483 Acked 7482 Read 1 Done Done New true
6484 Acked 7483 Flush 0 Done Done New true
6485 Acked 7484 Write 6 Done Done New true
6486 Acked 7485 Read 7 Done Done New true
6487 Acked 7486 Read 10 Done Done New true
6488 Acked 7487 Read 5 Done Done New true
6489 Acked 7488 Read 2 Done Done New true
6490 Acked 7489 Read 6 Done Done New true
6491 Acked 7490 Write 4 Done Done New true
6492 Acked 7491 Read 8 Done Done New true
6493 Acked 7492 Write 4 Done Done New true
6494 Acked 7493 Read 5 Done Done New true
6495 Acked 7494 Flush 0 Done Done New true
6496 Acked 7495 Write 6 Done Done New true
6497 Acked 7496 Read 7 Done Done New true
6498 Acked 7497 Write 1 Done Done New true
6499 Acked 7498 Read 8 Done Done New true
6500 Acked 7499 Read 6 Done Done New true
6501 Acked 7500 Read 10 Done Done New true
6502 Acked 7501 Read 4 Done Done New true
6503 Acked 7502 Read 7 Done Done New true
6504 Acked 7503 Write 9 Done Done New true
6505 Acked 7504 Read 4 Done Done New true
6506 Acked 7505 Flush 0 Done Done New true
6507 Acked 7506 Write 4 Done Done New true
6508 Acked 7507 Read 8 Done Done New true
6509 Acked 7508 Write 8 Done Done New true
6510 Acked 7509 Read 8 Done Done New true
6511 Acked 7510 Flush 0 Done Done New true
6512 Acked 7511 Read 10 Done Done New true
6513 Acked 7512 Read 3 Done Done New true
6514 Acked 7513 Write 5 Done Done New true
6515 Acked 7514 Read 3 Done Done New true
6516 Acked 7515 Write 3 Done Done New true
6517 Acked 7516 Write 9 Done Done New true
6518 Acked 7517 Write 9 Done Done New true
6519 Acked 7518 Write 3 Done Done New true
6520 Acked 7519 Read 8 Done Done New true
6521 Acked 7520 Write 2 Done Done New true
6522 Acked 7521 Write 10 Done Done New true
6523 Acked 7522 Read 8 Done Done New true
6524 Acked 7523 Write 3 Done Done New true
6525 Acked 7524 Read 2 Done Done New true
6526 Acked 7525 Read 7 Done Done New true
6527 Acked 7526 Write 3 Done Done New true
6528 Acked 7527 Read 10 Done Done New true
6529 Acked 7528 Read 3 Done Done New true
6530 Acked 7529 Write 5 Done Done New true
6531 Acked 7530 Write 5 Done Done New true
6532 Acked 7531 Read 5 Done Done New true
6533 Acked 7532 Flush 0 Done Done New true
6534 Acked 7533 Write 5 Done Done New true
6535 Acked 7534 Read 9 Done Done New true
6536 Acked 7535 Read 1 Done Done New true
6537 Acked 7536 Write 3 Done Done New true
6538 Acked 7537 Read 3 Done Done New true
6539 Acked 7538 Read 4 Done Done New true
6540 Acked 7539 Read 4 Done Done New true
6541 Acked 7540 Write 1 Done Done New true
6542 Acked 7541 Write 8 Done Done New true
6543 Acked 7542 Write 10 Done Done New true
6544 Acked 7543 Read 2 Done Done New true
6545 Acked 7544 Read 9 Done Done New true
6546 Acked 7545 Write 10 Done Done New true
6547 Acked 7546 Write 6 Done Done New true
6548 Acked 7547 Read 10 Done Done New true
6549 Acked 7548 Write 7 Done Done New true
6550 Acked 7549 Read 9 Done Done New true
6551 Acked 7550 Read 1 Done Done New true
6552 Acked 7551 Read 5 Done Done New true
6553 Acked 7552 Read 2 Done Done New true
6554 Acked 7553 Read 5 Done Done New true
6555 Acked 7554 Read 8 Done Done New true
6556 Acked 7555 Write 5 Done Done New true
6557 Acked 7556 Read 2 Done Done New true
6558 Acked 7557 Write 1 Done Done New true
6559 Acked 7558 Write 5 Done Done New true
6560 Acked 7559 Write 1 Done Done New true
6561 Acked 7560 Read 10 Done Done New true
6562 Acked 7561 Read 5 Done Done New true
6563 Acked 7562 Write 4 Done Done New true
6564 Acked 7563 Read 8 Done Done New true
6565 Acked 7564 Read 2 Done Done New true
6566 Acked 7565 Write 2 Done Done New true
6567 Acked 7566 Read 9 Done Done New true
6568 Acked 7567 Write 1 Done Done New true
6569 Acked 7568 Read 1 Done Done New true
6570 Acked 7569 Read 2 Done Done New true
6571 Acked 7570 Read 10 Done Done New true
6572 Acked 7571 Read 10 Done Done New true
6573 Acked 7572 Write 1 Done Done New true
6574 Acked 7573 Read 6 Done Done New true
6575 Acked 7574 Flush 0 Done Done New true
6576 Acked 7575 Read 6 Done Done New true
6577 Acked 7576 Read 10 Done Done New true
6578 Acked 7577 Write 4 Done Done New true
6579 Acked 7578 Read 8 Done Done New true
6580 Acked 7579 Read 10 Done Done New true
6581 Acked 7580 Read 10 Done Done New true
6582 Acked 7581 Write 4 Done Done New true
6583 Acked 7582 Read 6 Done Done New true
6584 Acked 7583 Flush 0 Done Done New true
6585 Acked 7584 Read 1 Done Done New true
6586 Acked 7585 Read 10 Done Done New true
6587 Acked 7586 Write 2 Done Done New true
6588 Acked 7587 Read 2 Done Done New true
6589 Acked 7588 Flush 0 Done Done New true
6590 Acked 7589 Write 1 Done Done New true
6591 Acked 7590 Write 10 Done Done New true
6592 Acked 7591 Read 7 Done Done New true
6593 Acked 7592 Write 1 Done Done New true
6594 Acked 7593 Read 10 Done Done New true
6595 Acked 7594 Write 6 Done Done New true
6596 Acked 7595 Read 8 Done Done New true
6597 Acked 7596 Read 6 Done Done New true
6598 Acked 7597 Read 1 Done Done New true
6599 Acked 7598 Read 6 Done Done New true
6600 Acked 7599 Read 8 Done Done New true
6601 Acked 7600 Write 9 Done Done New true
6602 Acked 7601 Write 7 Done Done New true
6603 Acked 7602 Read 10 Done Done New true
6604 Acked 7603 Write 8 Done Done New true
6605 Acked 7604 Read 3 Done Done New true
6606 Acked 7605 Read 5 Done Done New true
6607 Acked 7606 Read 6 Done Done New true
6608 Acked 7607 Write 4 Done Done New true
6609 Acked 7608 Write 7 Done Done New true
6610 Acked 7609 Write 3 Done Done New true
6611 Acked 7610 Write 4 Done Done New true
6612 Acked 7611 Read 8 Done Done New true
6613 Acked 7612 Read 9 Done Done New true
6614 Acked 7613 Read 6 Done Done New true
6615 Acked 7614 Write 8 Done Done New true
6616 Acked 7615 Write 6 Done Done New true
6617 Acked 7616 Write 5 Done Done New true
6618 Acked 7617 Write 6 Done Done New true
6619 Acked 7618 Read 6 Done Done New true
6620 Acked 7619 Write 3 Done Done New true
6621 Acked 7620 Read 1 Done Done New true
6622 Acked 7621 Write 1 Done Done New true
6623 Acked 7622 Write 6 Done Done New true
6624 Acked 7623 Read 9 Done Done New true
6625 Acked 7624 Flush 0 Done Done New true
6626 Acked 7625 Read 6 Done Done New true
6627 Acked 7626 Read 5 Done Done New true
6628 Acked 7627 Read 1 Done Done New true
6629 Acked 7628 Write 2 Done Done New true
6630 Acked 7629 Flush 0 Done Done New true
STATES DS:0 DS:1 DS:2 TOTAL
New 0 0 301 301
Sent 0 0 0 0
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 7629 7629 7328
Downstairs last five completed: 7328 7327 7326 7325 7324
Upstairs last five completed: 6630 6629 6628 6627 6626
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:36.043692597Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:36.043834002Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:36.044966999Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:36.044992998Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 7328","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:36.045008437Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 7328","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:36.0452033Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:36.045243528Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:36.045263487Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:36.045281657Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:36.045299196Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 7629 7628 7627 7626 7625
Upstairs last five completed: 6630 6629 6628 6627 6626
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Feb 13 21:43:39.819 INFO[8810] Got stop action so:false kr:false
request completed, latency_us: 333, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 414412ca-dff3-4553-86bc-09d5d584140f, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:39.820183961Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:39.820254619Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:39.820285788Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:43:39.820 INFO request completed, latency_us: 128{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:39.820311417Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
, response_code: 200, uri: /state/cid/0, method: GET, req_id: 21060dc2-4461-4c64-9085-40e160f8b22e, remote_addr: {"msg":"[0] client re-new 0 jobs since flush 7629","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:39.820364145Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:39.820398783Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:40.822319038Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:40.822380606Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:40.822404375Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:40.822425534Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:40.822441313Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 7629","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:40.822458593Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:40.822475712Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:41.824325399Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:41.824372948Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:41.824390737Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:41.824411996Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:41.824425366Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 7629","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:41.824439525Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:41.824453414Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:42.8255736Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:42.825631398Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:42.825652847Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:42.825673136Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:42.825688715Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 7629","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:42.825706025Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:42.825723094Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:43:43.822 INFO request completed, latency_us: 204, response_code: 200, uri: /state/cid/0, method: GET, req_id: f2db0e70-d6eb-4875-bcd6-3881dfdcb84a, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Read block 621 len 32768
002/300 Flush
003/300 Flush
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:43.826371177Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:43.826421606Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:43.826445845Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:43.826463644Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:43.826476693Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 3 jobs since flush 7629","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:43.826490673Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:43.826504512Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
004/300 Read block 4034 len 20480
005/300 Write block 2998 len 32768 data: 5 4 4 4 4 4 5 6
006/300 Read block 2256 len 32768
007/300 Write block 2433 len 12288 data: 5 4 5
008/300 Write block 704 len 12288 data: 2 2 3
009/300 Read block 1756 len 40960
010/300 Read block 3892 len 8192
011/300 Write block 2110 len 28672 data: 6 7 6 5 6 5 4
012/300 Read block 3764 len 8192
013/300 Read block 1782 len 16384
014/300 Read block 494 len 24576
015/300 Read block 3382 len 4096
016/300 Write block 3234 len 20480 data: 2 3 3 4 4
017/300 Flush
018/300 Read block 2550 len 32768
019/300 Write block 3408 len 24576 data: 9 7 7 7 6 4
020/300 Read block 1892 len 16384
021/300 Read block 1769 len 32768
022/300 Read block 2602 len 32768
023/300 Write block 3348 len 16384 data: 3 4 6 7
024/300 Read block 802 len 16384
025/300 Write block 4320 len 4096 data: 6
026/300 Write block 778 len 28672 data: 2 2 2 3 3 3 3
027/300 Write block 2969 len 40960 data: 5 5 5 5 6 6 6 6 6 7
028/300 Read block 3207 len 32768
029/300 Read block 2872 len 40960
030/300 Read block 2082 len 24576
031/300 Read block 403 len 12288
032/300 Read block 2158 len 12288
033/300 Write block 463 len 8192 data: 7 7
034/300 Flush
035/300 Read block 1922 len 32768
036/300 Flush
037/300 Read block 2077 len 12288
038/300 Flush
039/300 Write block 4722 len 20480 data: 10 7 8 8 8
040/300 Write block 4668 len 32768 data: 9 9 8 8 6 4 4 4
041/300 Write block 3023 len 24576 data: 7 6 4 4 6 6
042/300 Read block 136 len 32768
043/300 Write block 3882 len 20480 data: 4 5 6 7 8
044/300 Read block 4258 len 28672
045/300 Read block 390 len 20480
046/300 Flush
047/300 Read block 454 len 16384
048/300 Write block 2441 len 24576 data: 8 10 7 5 6 6
049/300 Write block 3844 len 36864 data: 4 4 3 4 5 4 5 7 9
050/300 Read block 132 len 28672
051/300 Read block 2396 len 32768
052/300 Read block 3247 len 28672
053/300 Write block 1184 len 12288 data: 5 6 6
054/300 Read block 3091 len 32768
055/300 Write block 4385 len 16384 data: 7 7 6 6
056/300 Read block 1908 len 12288
057/300 Read block 3796 len 32768
058/300 Write block 1015 len 36864 data: 4 3 3 3 5 4 4 4 3
059/300 Flush
060/300 Write block 4166 len 16384 data: 5 6 5 6
061/300 Write block 1360 len 12288 data: 8 7 6
062/300 Read block 1378 len 4096
063/300 Read block 619 len 40960
064/300 Write block 1533 len 36864 data: 7 7 6 6 8 7 5 5 4
065/300 Write block 2800 len 12288 data: 5 4 4
066/300 Read block 4904 len 40960
067/300 Read block 4896 len 12288
068/300 Write block 1373 len 32768 data: 5 4 3 2 3 2 2 2
069/300 Write block 4671 len 16384 data: 9 7 5 5
070/300 Flush
071/300 Write block 822 len 12288 data: 5 5 5
072/300 Write block 3408 len 24576 data: 10 8 8 8 7 5
073/300 Write block 4232 len 16384 data: 4 4 5 5
074/300 Read block 3974 len 28672
075/300 Read block 3871 len 20480
076/300 Flush
077/300 Write block 720 len 12288 data: 5 7 4
078/300 Write block 2546 len 40960 data: 2 3 4 5 4 5 4 4 4 6
079/300 Write block 1617 len 32768 data: 5 6 6 4 4 3 3 3
080/300 Read block 2694 len 32768
081/300 Write block 1909 len 12288 data: 6 7 7
082/300 Read block 56 len 4096
083/300 Write block 1269 len 8192 data: 4 4
084/300 Flush
085/300 Read block 2555 len 32768
086/300 Write block 0 len 8192 data: 4 6
087/300 Write block 4904 len 12288 data: 5 4 4
088/300 Read block 3437 len 32768
089/300 Write block 1471 len 24576 data: 5 4 4 5 7 7
090/300 Write block 2562 len 4096 data: 8
091/300 Read block 1224 len 36864
092/300 Read block 1602 len 4096
093/300 Flush
094/300 Write block 659 len 20480 data: 3 3 3 4 4
095/300 Flush
096/300 Write block 1719 len 36864 data: 7 7 6 6 5 5 4 5 5
097/300 Read block 3357 len 4096
098/300 Read block 650 len 20480
099/300 Read block 3908 len 16384
100/300 Read block 3873 len 20480
101/300 Read block 55 len 20480
102/300 Read block 4832 len 12288
103/300 Read block 2449 len 20480
104/300 Write block 2992 len 8192 data: 5 5
105/300 Flush
106/300 Read block 1661 len 12288
107/300 Read block 1654 len 20480
108/300 Write block 3681 len 8192 data: 10 10
109/300 Read block 1285 len 12288
110/300 Write block 4216 len 12288 data: 5 5 6
111/300 Write block 3042 len 4096 data: 7
112/300 Write block 4515 len 20480 data: 5 4 4 2 2
113/300 Read block 1199 len 12288
114/300 Write block 1668 len 20480 data: 3 2 2 3 3
115/300 Read block 534 len 20480
116/300 Read block 1557 len 40960
117/300 Read block 3721 len 36864
118/300 Write block 987 len 12288 data: 7 5 5
119/300 Write block 3197 len 24576 data: 11 13 11 10 10 7
120/300 Read block 162 len 12288
121/300 Write block 3573 len 8192 data: 4 5
122/300 Read block 4962 len 8192
123/300 Read block 2158 len 28672
124/300 Flush
125/300 Read block 2507 len 32768
126/300 Read block 3604 len 28672
127/300 Flush
128/300 Read block 1293 len 16384
129/300 Write block 1971 len 36864 data: 4 3 2 2 3 3 3 3 3
130/300 Read block 3434 len 4096
131/300 Write block 830 len 12288 data: 5 4 3
132/300 Write block 1240 len 36864 data: 3 3 2 2 2 4 4 4 6
133/300 Write block 4308 len 24576 data: 6 6 4 3 3 3
134/300 Write block 3651 len 40960 data: 7 5 5 6 7 7 6 6 3 2
135/300 Read block 3508 len 40960
136/300 Read block 686 len 28672
137/300 Write block 4968 len 24576 data: 2 2 2 2 4 4
138/300 Write block 3985 len 32768 data: 7 7 7 4 4 6 5 5
139/300 Write block 2526 len 20480 data: 7 7 7 9 8
140/300 Read block 364 len 36864
141/300 Read block 1530 len 32768
142/300 Read block 309 len 12288
143/300 Read block 1666 len 20480
144/300 Read block 2006 len 12288
145/300 Write block 3189 len 8192 data: 5 4
146/300 Read block 1806 len 40960
147/300 Write block 1522 len 32768 data: 5 5 6 6 4 4 4 4
148/300 Read block 2113 len 4096
149/300 Write block 3032 len 16384 data: 5 6 6 6
150/300 Write block 2784 len 12288 data: 5 4 4
151/300 Flush
152/300 Write block 4225 len 24576 data: 4 3 3 5 5 6
153/300 Write block 209 len 40960 data: 3 3 4 4 4 4 3 4 4 4
154/300 Read block 4704 len 28672
155/300 Write block 2572 len 36864 data: 8 9 10 9 8 7 6 4 5
156/300 Write block 1228 len 12288 data: 7 6 4
157/300 Write block 4712 len 20480 data: 7 5 7 6 5
158/300 Write block 3207 len 24576 data: 2 2 4 4 5 5
159/300 Read block 4734 len 16384
160/300 Write block 3984 len 24576 data: 7 8 8 8 5 5
161/300 Write block 4468 len 32768 data: 5 5 6 5 4 4 3 4
162/300 Read block 3626 len 28672
163/300 Write block 4823 len 20480 data: 2 2 2 4 6
164/300 Write block 4403 len 8192 data: 3 3
165/300 Read block 2656 len 16384
166/300 Flush
167/300 Write block 208 len 40960 data: 3 4 4 5 5 5 5 4 5 5
168/300 Flush
169/300 Write block 4248 len 24576 data: 4 3 3 4 3 4
170/300 Read block 3202 len 28672
171/300 Write block 2248 len 12288 data: 3 5 5
172/300 Flush
173/300 Write block 572 len 12288 data: 4 5 4
174/300 Read block 2346 len 28672
175/300 Read block 1953 len 36864
176/300 Flush
177/300 Read block 1346 len 24576
178/300 Read block 658 len 32768
179/300 Read block 3109 len 28672
180/300 Read block 632 len 4096
181/300 Flush
182/300 Read block 3224 len 4096
183/300 Read block 1926 len 20480
184/300 Read block 1370 len 40960
185/300 Write block 929 len 4096 data: 3
186/300 Read block 805 len 8192
187/300 Write block 1352 len 16384 data: 7 8 6 7
188/300 Read block 3400 len 8192
189/300 Write block 813 len 20480 data: 3 3 4 5 5
190/300 Write block 4483 len 40960 data: 3 3 4 6 8 11 11 9 9 8
191/300 Read block 1447 len 28672
192/300 Write block 331 len 4096 data: 10
193/300 Read block 4423 len 40960
194/300 Read block 4137 len 24576
195/300 Write block 3585 len 8192 data: 5 7
196/300 Write block 3141 len 20480 data: 5 4 4 5 6
197/300 Write block 3259 len 8192 data: 6 5
198/300 Write block 4232 len 16384 data: 5 5 6 6
199/300 Write block 1109 len 4096 data: 7
200/300 Flush
201/300 Read block 3660 len 16384
202/300 Write block 2602 len 4096 data: 4
203/300 Write block 948 len 8192 data: 2 2
204/300 Flush
205/300 Read block 4966 len 16384
206/300 Read block 4272 len 20480
207/300 Read block 3712 len 16384
208/300 Write block 4335 len 8192 data: 9 7
209/300 Read block 2612 len 4096
210/300 Read block 2363 len 36864
211/300 Write block 996 len 36864 data: 4 5 6 7 5 3 4 5 4
212/300 Read block 1939 len 12288
213/300 Read block 3605 len 36864
214/300 Read block 2981 len 28672
215/300 Read block 999 len 8192
216/300 Read block 4615 len 4096
217/300 Write block 640 len 28672 data: 8 10 9 8 9 7 6
218/300 Write block 4677 len 28672 data: 2 3 3 4 6 5 4
219/300 Write block 4939 len 28672 data: 5 6 6 6 5 6 6
220/300 Write block 2648 len 4096 data: 5
221/300 Read block 3147 len 8192
222/300 Write block 384 len 28672 data: 9 9 8 5 6 6 4
223/300 Read block 4233 len 8192
224/300 Write block 2533 len 40960 data: 7 8 7 5 5 6 4 4 3 3
225/300 Read block 728 len 4096
226/300 Read block 2481 len 4096
227/300 Flush
228/300 Read block 4451 len 12288
229/300 Flush
230/300 Read block 4640 len 32768
231/300 Read block 2945 len 28672
232/300 Write block 4319 len 36864 data: 5 7 6 8 10 9 9 9 10
233/300 Read block 68 len 20480
234/300 Write block 3223 len 8192 data: 10 9
235/300 Read block 3159 len 12288
236/300 Write block 2080 len 20480 data: 5 5 3 3 2
237/300 Write block 4002 len 20480 data: 6 7 7 6 5
238/300 Read block 1302 len 24576
239/300 Write block 152 len 28672 data: 5 5 4 4 3 6 6
240/300 Write block 4589 len 12288 data: 3 3 5
241/300 Write block 451 len 8192 data: 8 7
242/300 Read block 3803 len 16384
243/300 Write block 3860 len 16384 data: 6 5 5 4
244/300 Write block 3187 len 36864 data: 3 5 6 5 5 7 7 9 8
245/300 Write block 3753 len 40960 data: 6 6 5 5 4 5 4 4 3 4
246/300 Write block 396 len 24576 data: 5 3 3 3 4 4
247/300 Write block 3751 len 24576 data: 5 7 7 7 6 6
248/300 Read block 2064 len 4096
249/300 Read block 951 len 12288
250/300 Write block 202 len 16384 data: 3 2 2 2
251/300 Write block 4810 len 12288 data: 6 5 5
252/300 Flush
253/300 Write block 1604 len 28672 data: 5 6 6 4 5 5 5
254/300 Write block 3420 len 4096 data: 3
255/300 Write block 1668 len 16384 data: 4 3 3 4
256/300 Read block 2824 len 24576
257/300 Read block 1457 len 36864
258/300 Write block 2504 len 12288 data: 9 8 9
259/300 Write block 36 len 20480 data: 5 5 5 6 6
260/300 Read block 4405 len 8192
261/300 Read block 2211 len 40960
262/300 Read block 2798 len 16384
263/300 Write block 4481 len 28672 data: 4 3 4 4 5 7 9
264/300 Read block 1027 len 28672
265/300 Write block 2012 len 12288 data: 8 8 9
266/300 Write block 1110 len 12288 data: 8 9 9
267/300 Read block 4554 len 8192
268/300 Flush
269/300 Write block 3484 len 20480 data: 6 5 5 3 3
270/300 Write block 97 len 4096 data: 5
271/300 Write block 61 len 40960 data: 11 11 8 8 7 6 5 7 7 7
272/300 Read block 4404 len 32768
273/300 Read block 2264 len 40960
274/300 Write block 2944 len 4096 data: 5
275/300 Write block 3569 len 8192 data: 5 3
276/300 Read block 876 len 36864
277/300 Write block 218 len 32768 data: 5 5 5 4 4 5 3 2
278/300 Read block 4091 len 36864
279/300 Read block 2674 len 8192
280/300 Write block 1027 len 32768 data: 3 4 5 5 5 5 6 7
281/300 Write block 3495 len 4096 data: 4
282/300 Write block 3751 len 8192 data: 6 8
283/300 Read block 476 len 20480
284/300 Read block 3617 len 40960
285/300 Write block 2114 len 12288 data: 7 6 5
286/300 Read block 4844 len 12288
287/300 Write block 3562 len 12288 data: 7 6 4
288/300 Read block 4626 len 24576
289/300 Write block 1627 len 4096 data: 6
290/300 Flush
291/300 Read block 2975 len 40960
292/300 Read block 4679 len 28672
293/300 Read block 1592 len 16384
294/300 Write block 2153 len 20480 data: 4 3 3 2 3
295/300 Read block 3556 len 24576
296/300 Write block 1289 len 12288 data: 3 2 2
297/300 Write block 4827 len 36864 data: 7 6 6 6 9 8 9 7 8
298/300 Write block 3023 len 24576 data: 8 7 5 5 7 7
299/300 Write block 2987 len 8192 data: 8 5
300/300 Write block 1339 len 16384 data: 3 3 2 2
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 13 21:43:44.600 INFO request completed, latency_us: 272, response_code: 204, uri: /start/cid/0, method: POST, req_id: 3aa5c363-9cc5-40c7-b799-5d40c1cb36da, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
6631 Acked 7630 Read 8 New Done Done true
6632 Acked 7631 Flush 0 New Done Done true
6633 Acked 7632 Flush 0 New Done Done true
6634 Acked 7633 Read 5 New Done Done false
6635 Acked 7634 Write 8 New Done Done false
6636 Acked 7635 Read 8 New Done Done false
6637 Acked 7636 Write 3 New Done Done false
6638 Acked 7637 Write 3 New Done Done false
6639 Acked 7638 Read 10 New Done Done false
6640 Acked 7639 Read 2 New Done Done false
6641 Acked 7640 Write 7 New Done Done false
6642 Acked 7641 Read 2 New Done Done false
6643 Acked 7642 Read 4 New Done Done false
6644 Acked 7643 Read 6 New Done Done false
6645 Acked 7644 Read 1 New Done Done false
6646 Acked 7645 Write 5 New Done Done false
6647 Acked 7646 Flush 0 New Done Done false
6648 Acked 7647 Read 8 New Done Done false
6649 Acked 7648 Write 6 New Done Done false
6650 Acked 7649 Read 4 New Done Done false
6651 Acked 7650 Read 8 New Done Done false
6652 Acked 7651 Read 8 New Done Done false
6653 Acked 7652 Write 4 New Done Done false
6654 Acked 7653 Read 4 New Done Done false
6655 Acked 7654 Write 1 New Done Done false
6656 Acked 7655 Write 7 New Done Done false
6657 Acked 7656 Write 10 New Done Done false
6658 Acked 7657 Read 8 New Done Done false
6659 Acked 7658 Read 10 New Done Done false
6660 Acked 7659 Read 6 New Done Done false
6661 Acked 7660 Read 3 New Done Done false
6662 Acked 7661 Read 3 New Done Done false
6663 Acked 7662 Write 2 New Done Done false
6664 Acked 7663 Flush 0 New Done Done false
6665 Acked 7664 Read 8 New Done Done false
6666 Acked 7665 Flush 0 New Done Done false
6667 Acked 7666 Read 3 New Done Done false
6668 Acked 7667 Flush 0 New Done Done false
6669 Acked 7668 Write 5 New Done Done false
6670 Acked 7669 Write 8 New Done Done false
6671 Acked 7670 Write 6 New Done Done false
6672 Acked 7671 Read 8 New Done Done false
6673 Acked 7672 Write 5 New Done Done false
6674 Acked 7673 Read 7 New Done Done false
6675 Acked 7674 Read 5 New Done Done false
6676 Acked 7675 Flush 0 New Done Done false
6677 Acked 7676 Read 4 New Done Done false
6678 Acked 7677 Write 6 New Done Done false
6679 Acked 7678 Write 9 New Done Done false
6680 Acked 7679 Read 7 New Done Done false
6681 Acked 7680 Read 8 New Done Done false
6682 Acked 7681 Read 7 New Done Done false
6683 Acked 7682 Write 3 New Done Done false
6684 Acked 7683 Read 8 New Done Done false
6685 Acked 7684 Write 4 New Done Done false
6686 Acked 7685 Read 3 New Done Done false
6687 Acked 7686 Read 8 New Done Done false
6688 Acked 7687 Write 9 New Done Done false
6689 Acked 7688 Flush 0 New Done Done false
6690 Acked 7689 Write 4 New Done Done false
6691 Acked 7690 Write 3 New Done Done false
6692 Acked 7691 Read 1 New Done Done false
6693 Acked 7692 Read 10 New Done Done false
6694 Acked 7693 Write 9 New Done Done false
6695 Acked 7694 Write 3 New Done Done false
6696 Acked 7695 Read 10 New Done Done false
6697 Acked 7696 Flush 0 New Done Done false
6698 Acked 7697 Read 3 New Done Done false
6699 Acked 7698 Write 8 New Done Done false
6700 Acked 7699 Write 4 New Done Done false
6701 Acked 7700 Flush 0 New Done Done false
6702 Acked 7701 Write 3 New Done Done false
6703 Acked 7702 Write 6 New Done Done false
6704 Acked 7703 Write 4 New Done Done false
6705 Acked 7704 Read 7 New Done Done false
6706 Acked 7705 Read 5 New Done Done false
6707 Acked 7706 Flush 0 New Done Done false
6708 Acked 7707 Write 3 New Done Done false
6709 Acked 7708 Write 10 New Done Done false
6710 Acked 7709 Write 8 New Done Done false
6711 Acked 7710 Read 8 New Done Done false
6712 Acked 7711 Write 3 New Done Done false
6713 Acked 7712 Read 1 New Done Done false
6714 Acked 7713 Write 2 New Done Done false
6715 Acked 7714 Flush 0 New Done Done false
6716 Acked 7715 Read 8 New Done Done false
6717 Acked 7716 Write 2 New Done Done false
6718 Acked 7717 Write 3 New Done Done false
6719 Acked 7718 Read 8 New Done Done false
6720 Acked 7719 Write 6 New Done Done false
6721 Acked 7720 Write 1 New Done Done false
6722 Acked 7721 Read 9 New Done Done false
6723 Acked 7722 Read 1 New Done Done false
6724 Acked 7723 Flush 0 New Done Done false
6725 Acked 7724 Write 5 New Done Done false
6726 Acked 7725 Flush 0 New Done Done false
6727 Acked 7726 Write 9 New Done Done false
6728 Acked 7727 Read 1 New Done Done false
6729 Acked 7728 Read 5 New Done Done false
6730 Acked 7729 Read 4 New Done Done false
6731 Acked 7730 Read 5 New Done Done false
6732 Acked 7731 Read 5 New Done Done false
6733 Acked 7732 Read 3 New Done Done false
6734 Acked 7733 Read 5 New Done Done false
6735 Acked 7734 Write 2 New Done Done false
6736 Acked 7735 Flush 0 New Done Done false
6737 Acked 7736 Read 3 New Done Done false
6738 Acked 7737 Read 5 New Done Done false
6739 Acked 7738 Write 2 New Done Done false
6740 Acked 7739 Read 3 New Done Done false
6741 Acked 7740 Write 3 New Done Done false
6742 Acked 7741 Write 1 New Done Done false
6743 Acked 7742 Write 5 New Done Done false
6744 Acked 7743 Read 3 New Done Done false
6745 Acked 7744 Write 5 New Done Done false
6746 Acked 7745 Read 5 New Done Done false
6747 Acked 7746 Read 10 New Done Done false
6748 Acked 7747 Read 9 New Done Done false
6749 Acked 7748 Write 3 New Done Done false
6750 Acked 7749 Write 6 New Done Done false
6751 Acked 7750 Read 3 New Done Done false
6752 Acked 7751 Write 2 New Done Done false
6753 Acked 7752 Read 2 New Done Done false
6754 Acked 7753 Read 7 New Done Done false
6755 Acked 7754 Flush 0 New Done Done false
6756 Acked 7755 Read 8 New Done Done false
6757 Acked 7756 Read 7 New Done Done false
6758 Acked 7757 Flush 0 New Done Done false
6759 Acked 7758 Read 4 New Done Done false
6760 Acked 7759 Write 9 New Done Done false
6761 Acked 7760 Read 1 New Done Done false
6762 Acked 7761 Write 3 New Done Done false
6763 Acked 7762 Write 9 New Done Done false
6764 Acked 7763 Write 6 New Done Done false
6765 Acked 7764 Write 10 New Done Done false
6766 Acked 7765 Read 10 New Done Done false
6767 Acked 7766 Read 7 New Done Done false
6768 Acked 7767 Write 6 New Done Done false
6769 Acked 7768 Write 8 New Done Done false
6770 Acked 7769 Write 5 New Done Done false
6771 Acked 7770 Read 9 New Done Done false
6772 Acked 7771 Read 8 New Done Done false
6773 Acked 7772 Read 3 New Done Done false
6774 Acked 7773 Read 5 New Done Done false
6775 Acked 7774 Read 3 New Done Done false
6776 Acked 7775 Write 2 New Done Done false
6777 Acked 7776 Read 10 New Done Done false
6778 Acked 7777 Write 8 New Done Done false
6779 Acked 7778 Read 1 New Done Done false
6780 Acked 7779 Write 4 New Done Done false
6781 Acked 7780 Write 3 New Done Done false
6782 Acked 7781 Flush 0 New Done Done false
6783 Acked 7782 Write 6 New Done Done false
6784 Acked 7783 Write 10 New Done Done false
6785 Acked 7784 Read 7 New Done Done false
6786 Acked 7785 Write 9 New Done Done false
6787 Acked 7786 Write 3 New Done Done false
6788 Acked 7787 Write 5 New Done Done false
6789 Acked 7788 Write 6 New Done Done false
6790 Acked 7789 Read 4 New Done Done false
6791 Acked 7790 Write 6 New Done Done false
6792 Acked 7791 Write 8 New Done Done false
6793 Acked 7792 Read 7 New Done Done false
6794 Acked 7793 Write 5 New Done Done false
6795 Acked 7794 Write 2 New Done Done false
6796 Acked 7795 Read 4 New Done Done false
6797 Acked 7796 Flush 0 New Done Done false
6798 Acked 7797 Write 10 New Done Done false
6799 Acked 7798 Flush 0 New Done Done false
6800 Acked 7799 Write 6 New Done Done false
6801 Acked 7800 Read 7 New Done Done false
6802 Acked 7801 Write 3 New Done Done false
6803 Acked 7802 Flush 0 New Done Done false
6804 Acked 7803 Write 3 New Done Done false
6805 Acked 7804 Read 7 New Done Done false
6806 Acked 7805 Read 9 New Done Done false
6807 Acked 7806 Flush 0 New Done Done false
6808 Acked 7807 Read 6 New Done Done false
6809 Acked 7808 Read 8 New Done Done false
6810 Acked 7809 Read 7 New Done Done false
6811 Acked 7810 Read 1 New Done Done false
6812 Acked 7811 Flush 0 New Done Done false
6813 Acked 7812 Read 1 New Done Done false
6814 Acked 7813 Read 5 New Done Done false
6815 Acked 7814 Read 10 New Done Done false
6816 Acked 7815 Write 1 New Done Done false
6817 Acked 7816 Read 2 New Done Done false
6818 Acked 7817 Write 4 New Done Done false
6819 Acked 7818 Read 2 New Done Done false
6820 Acked 7819 Write 5 New Done Done false
6821 Acked 7820 Write 10 New Done Done false
6822 Acked 7821 Read 7 New Done Done false
6823 Acked 7822 Write 1 New Done Done false
6824 Acked 7823 Read 10 New Done Done false
6825 Acked 7824 Read 6 New Done Done false
6826 Acked 7825 Write 2 New Done Done false
6827 Acked 7826 Write 5 New Done Done false
6828 Acked 7827 Write 2 New Done Done false
6829 Acked 7828 Write 4 New Done Done false
6830 Acked 7829 Write 1 New Done Done false
6831 Acked 7830 Flush 0 New Done Done false
6832 Acked 7831 Read 4 New Done Done false
6833 Acked 7832 Write 1 New Done Done false
6834 Acked 7833 Write 2 New Done Done false
6835 Acked 7834 Flush 0 New Done Done false
6836 Acked 7835 Read 4 New Done Done false
6837 Acked 7836 Read 5 New Done Done false
6838 Acked 7837 Read 4 New Done Done false
6839 Acked 7838 Write 2 New Done Done false
6840 Acked 7839 Read 1 New Done Done false
6841 Acked 7840 Read 9 New Done Done false
6842 Acked 7841 Write 9 New Done Done false
6843 Acked 7842 Read 3 New Done Done false
6844 Acked 7843 Read 9 New Done Done false
6845 Acked 7844 Read 7 New Done Done false
6846 Acked 7845 Read 2 New Done Done false
6847 Acked 7846 Read 1 New Done Done false
6848 Acked 7847 Write 7 New Done Done false
6849 Acked 7848 Write 7 New Done Done false
6850 Acked 7849 Write 7 New Done Done false
6851 Acked 7850 Write 1 New Done Done false
6852 Acked 7851 Read 2 New Done Done false
6853 Acked 7852 Write 7 New Done Done false
6854 Acked 7853 Read 2 New Done Done false
6855 Acked 7854 Write 10 New Done Done false
6856 Acked 7855 Read 1 New Done Done false
6857 Acked 7856 Read 1 New Done Done false
6858 Acked 7857 Flush 0 New Done Done false
6859 Acked 7858 Read 3 New Done Done false
6860 Acked 7859 Flush 0 New Done Done false
6861 Acked 7860 Read 8 New Done Done false
6862 Acked 7861 Read 7 New Done Done false
6863 Acked 7862 Write 9 New Done Done false
6864 Acked 7863 Read 5 New Done Done false
6865 Acked 7864 Write 2 New Done Done false
6866 Acked 7865 Read 3 New Done Done false
6867 Acked 7866 Write 5 New Done Done false
6868 Acked 7867 Write 5 New Done Done false
6869 Acked 7868 Read 6 New Done Done false
6870 Acked 7869 Write 7 New Done Done false
6871 Acked 7870 Write 3 New Done Done false
6872 Acked 7871 Write 2 New Done Done false
6873 Acked 7872 Read 4 New Done Done false
6874 Acked 7873 Write 4 New Done Done false
6875 Acked 7874 Write 9 New Done Done false
6876 Acked 7875 Write 10 New Done Done false
6877 Acked 7876 Write 6 New Done Done false
6878 Acked 7877 Write 6 New Done Done false
6879 Acked 7878 Read 1 New Done Done false
6880 Acked 7879 Read 3 New Done Done false
6881 Acked 7880 Write 4 New Done Done false
6882 Acked 7881 Write 3 New Done Done false
6883 Acked 7882 Flush 0 New Done Done false
6884 Acked 7883 Write 7 New Done Done false
6885 Acked 7884 Write 1 New Done Done false
6886 Acked 7885 Write 4 New Done Done false
6887 Acked 7886 Read 6 New Done Done false
6888 Acked 7887 Read 9 New Done Done false
6889 Acked 7888 Write 3 New Done Done false
6890 Acked 7889 Write 5 New Done Done false
6891 Acked 7890 Read 2 New Done Done false
6892 Acked 7891 Read 10 New Done Done false
6893 Acked 7892 Read 4 New Done Done false
6894 Acked 7893 Write 7 New Done Done false
6895 Acked 7894 Read 7 New Done Done false
6896 Acked 7895 Write 3 New Done Done false
6897 Acked 7896 Write 3 New Done Done false
6898 Acked 7897 Read 2 New Done Done false
6899 Acked 7898 Flush 0 New Done Done false
6900 Acked 7899 Write 5 New Done Done false
6901 Acked 7900 Write 1 New Done Done false
6902 Acked 7901 Write 10 New Done Done false
6903 Acked 7902 Read 8 New Done Done false
6904 Acked 7903 Read 10 New Done Done false
6905 Acked 7904 Write 1 New Done Done false
6906 Acked 7905 Write 2 New Done Done false
6907 Acked 7906 Read 9 New Done Done false
6908 Acked 7907 Write 8 New Done Done false
6909 Acked 7908 Read 9 New Done Done false
6910 Acked 7909 Read 2 New Done Done false
6911 Acked 7910 Write 8 New Done Done false
6912 Acked 7911 Write 1 New Done Done false
6913 Acked 7912 Write 2 New Done Done false
6914 Acked 7913 Read 5 New Done Done false
6915 Acked 7914 Read 10 New Done Done false
6916 Acked 7915 Write 3 New Done Done false
6917 Acked 7916 Read 3 New Done Done false
6918 Acked 7917 Write 3 New Done Done false
6919 Acked 7918 Read 6 New Done Done false
6920 Acked 7919 Write 1 New Done Done false
6921 Acked 7920 Flush 0 New Done Done false
6922 Acked 7921 Read 10 New Done Done false
6923 Acked 7922 Read 7 New Done Done false
6924 Acked 7923 Read 4 New Done Done false
6925 Acked 7924 Write 5 New Done Done false
6926 Acked 7925 Read 6 New Done Done false
6927 Acked 7926 Write 3 New Done Done false
6928 Acked 7927 Write 9 New Done Done false
6929 Acked 7928 Write 6 New Done Done false
6930 Acked 7929 Write 2 New Done Done false
6931 Acked 7930 Write 4 New Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
New 301 0 0 301
Sent 0 0 0 0
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 7629 7920 7920
Downstairs last five completed: 7629 7628 7627 7626 7625
Upstairs last five completed: 6931 6930 6929 6928 6927
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:44.828057182Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:44.828115899Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:44.828139199Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:44.828159938Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:44.828182517Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 7629","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:44.828214156Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:44.828271424Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:45.8291174Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:45.829177008Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:45.829200037Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:45.829220606Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:45.829236566Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 7629","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:45.829254045Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:45.829351491Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:46.831326604Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:46.831382162Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:46.831404291Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:46.83142554Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:46.83144192Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 7629","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:46.831459469Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:46.831562165Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:47.832546206Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:47.832605544Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:47.832637933Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:47.832668271Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:47.832691441Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 7629","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:47.83271429Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:47.832843085Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1187)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
6631 Acked 7630 Read 8 New Done Done true
6632 Acked 7631 Flush 0 New Done Done true
6633 Acked 7632 Flush 0 New Done Done true
6634 Acked 7633 Read 5 New Done Done true
6635 Acked 7634 Write 8 New Done Done true
6636 Acked 7635 Read 8 New Done Done true
6637 Acked 7636 Write 3 New Done Done true
6638 Acked 7637 Write 3 New Done Done true
6639 Acked 7638 Read 10 New Done Done true
6640 Acked 7639 Read 2 New Done Done true
6641 Acked 7640 Write 7 New Done Done true
6642 Acked 7641 Read 2 New Done Done true
6643 Acked 7642 Read 4 New Done Done true
6644 Acked 7643 Read 6 New Done Done true
6645 Acked 7644 Read 1 New Done Done true
6646 Acked 7645 Write 5 New Done Done true
6647 Acked 7646 Flush 0 New Done Done true
6648 Acked 7647 Read 8 New Done Done true
6649 Acked 7648 Write 6 New Done Done true
6650 Acked 7649 Read 4 New Done Done true
6651 Acked 7650 Read 8 New Done Done true
6652 Acked 7651 Read 8 New Done Done true
6653 Acked 7652 Write 4 New Done Done true
6654 Acked 7653 Read 4 New Done Done true
6655 Acked 7654 Write 1 New Done Done true
6656 Acked 7655 Write 7 New Done Done true
6657 Acked 7656 Write 10 New Done Done true
6658 Acked 7657 Read 8 New Done Done true
6659 Acked 7658 Read 10 New Done Done true
6660 Acked 7659 Read 6 New Done Done true
6661 Acked 7660 Read 3 New Done Done true
6662 Acked 7661 Read 3 New Done Done true
6663 Acked 7662 Write 2 New Done Done true
6664 Acked 7663 Flush 0 New Done Done true
6665 Acked 7664 Read 8 New Done Done true
6666 Acked 7665 Flush 0 New Done Done true
6667 Acked 7666 Read 3 New Done Done true
6668 Acked 7667 Flush 0 New Done Done true
6669 Acked 7668 Write 5 New Done Done true
6670 Acked 7669 Write 8 New Done Done true
6671 Acked 7670 Write 6 New Done Done true
6672 Acked 7671 Read 8 New Done Done true
6673 Acked 7672 Write 5 New Done Done true
6674 Acked 7673 Read 7 New Done Done true
6675 Acked 7674 Read 5 New Done Done true
6676 Acked 7675 Flush 0 New Done Done true
6677 Acked 7676 Read 4 New Done Done true
6678 Acked 7677 Write 6 New Done Done true
6679 Acked 7678 Write 9 New Done Done true
6680 Acked 7679 Read 7 New Done Done true
6681 Acked 7680 Read 8 New Done Done true
6682 Acked 7681 Read 7 New Done Done true
6683 Acked 7682 Write 3 New Done Done true
6684 Acked 7683 Read 8 New Done Done true
6685 Acked 7684 Write 4 New Done Done true
6686 Acked 7685 Read 3 New Done Done true
6687 Acked 7686 Read 8 New Done Done true
6688 Acked 7687 Write 9 New Done Done true
6689 Acked 7688 Flush 0 New Done Done true
6690 Acked 7689 Write 4 New Done Done true
6691 Acked 7690 Write 3 New Done Done true
6692 Acked 7691 Read 1 New Done Done true
6693 Acked 7692 Read 10 New Done Done true
6694 Acked 7693 Write 9 New Done Done true
6695 Acked 7694 Write 3 New Done Done true
6696 Acked 7695 Read 10 New Done Done true
6697 Acked 7696 Flush 0 New Done Done true
6698 Acked 7697 Read 3 New Done Done true
6699 Acked 7698 Write 8 New Done Done true
6700 Acked 7699 Write 4 New Done Done true
6701 Acked 7700 Flush 0 New Done Done true
6702 Acked 7701 Write 3 New Done Done true
6703 Acked 7702 Write 6 New Done Done true
6704 Acked 7703 Write 4 New Done Done true
6705 Acked 7704 Read 7 New Done Done true
6706 Acked 7705 Read 5 New Done Done true
6707 Acked 7706 Flush 0 New Done Done true
6708 Acked 7707 Write 3 New Done Done true
6709 Acked 7708 Write 10 New Done Done true
6710 Acked 7709 Write 8 New Done Done true
6711 Acked 7710 Read 8 New Done Done true
6712 Acked 7711 Write 3 New Done Done true
6713 Acked 7712 Read 1 New Done Done true
6714 Acked 7713 Write 2 New Done Done true
6715 Acked 7714 Flush 0 New Done Done true
6716 Acked 7715 Read 8 New Done Done true
6717 Acked 7716 Write 2 New Done Done true
6718 Acked 7717 Write 3 New Done Done true
6719 Acked 7718 Read 8 New Done Done true
6720 Acked 7719 Write 6 New Done Done true
6721 Acked 7720 Write 1 New Done Done true
6722 Acked 7721 Read 9 New Done Done true
6723 Acked 7722 Read 1 New Done Done true
6724 Acked 7723 Flush 0 New Done Done true
6725 Acked 7724 Write 5 New Done Done true
6726 Acked 7725 Flush 0 New Done Done true
6727 Acked 7726 Write 9 New Done Done true
6728 Acked 7727 Read 1 New Done Done true
6729 Acked 7728 Read 5 New Done Done true
6730 Acked 7729 Read 4 New Done Done true
6731 Acked 7730 Read 5 New Done Done true
6732 Acked 7731 Read 5 New Done Done true
6733 Acked 7732 Read 3 New Done Done true
6734 Acked 7733 Read 5 New Done Done true
6735 Acked 7734 Write 2 New Done Done true
6736 Acked 7735 Flush 0 New Done Done true
6737 Acked 7736 Read 3 New Done Done true
6738 Acked 7737 Read 5 New Done Done true
6739 Acked 7738 Write 2 New Done Done true
6740 Acked 7739 Read 3 New Done Done true
6741 Acked 7740 Write 3 New Done Done true
6742 Acked 7741 Write 1 New Done Done true
6743 Acked 7742 Write 5 New Done Done true
6744 Acked 7743 Read 3 New Done Done true
6745 Acked 7744 Write 5 New Done Done true
6746 Acked 7745 Read 5 New Done Done true
6747 Acked 7746 Read 10 New Done Done true
6748 Acked 7747 Read 9 New Done Done true
6749 Acked 7748 Write 3 New Done Done true
6750 Acked 7749 Write 6 New Done Done true
6751 Acked 7750 Read 3 New Done Done true
6752 Acked 7751 Write 2 New Done Done true
6753 Acked 7752 Read 2 New Done Done true
6754 Acked 7753 Read 7 New Done Done true
6755 Acked 7754 Flush 0 New Done Done true
6756 Acked 7755 Read 8 New Done Done true
6757 Acked 7756 Read 7 New Done Done true
6758 Acked 7757 Flush 0 New Done Done true
6759 Acked 7758 Read 4 New Done Done true
6760 Acked 7759 Write 9 New Done Done true
6761 Acked 7760 Read 1 New Done Done true
6762 Acked 7761 Write 3 New Done Done true
6763 Acked 7762 Write 9 New Done Done true
6764 Acked 7763 Write 6 New Done Done true
6765 Acked 7764 Write 10 New Done Done true
6766 Acked 7765 Read 10 New Done Done true
6767 Acked 7766 Read 7 New Done Done true
6768 Acked 7767 Write 6 New Done Done true
6769 Acked 7768 Write 8 New Done Done true
6770 Acked 7769 Write 5 New Done Done true
6771 Acked 7770 Read 9 New Done Done true
6772 Acked 7771 Read 8 New Done Done true
6773 Acked 7772 Read 3 New Done Done true
6774 Acked 7773 Read 5 New Done Done true
6775 Acked 7774 Read 3 New Done Done true
6776 Acked 7775 Write 2 New Done Done true
6777 Acked 7776 Read 10 New Done Done true
6778 Acked 7777 Write 8 New Done Done true
6779 Acked 7778 Read 1 New Done Done true
6780 Acked 7779 Write 4 New Done Done true
6781 Acked 7780 Write 3 New Done Done true
6782 Acked 7781 Flush 0 New Done Done true
6783 Acked 7782 Write 6 New Done Done true
6784 Acked 7783 Write 10 New Done Done true
6785 Acked 7784 Read 7 New Done Done true
6786 Acked 7785 Write 9 New Done Done true
6787 Acked 7786 Write 3 New Done Done true
6788 Acked 7787 Write 5 New Done Done true
6789 Acked 7788 Write 6 New Done Done true
6790 Acked 7789 Read 4 New Done Done true
6791 Acked 7790 Write 6 New Done Done true
6792 Acked 7791 Write 8 New Done Done true
6793 Acked 7792 Read 7 New Done Done true
6794 Acked 7793 Write 5 New Done Done true
6795 Acked 7794 Write 2 New Done Done true
6796 Acked 7795 Read 4 New Done Done true
6797 Acked 7796 Flush 0 New Done Done true
6798 Acked 7797 Write 10 New Done Done true
6799 Acked 7798 Flush 0 New Done Done true
6800 Acked 7799 Write 6 New Done Done true
6801 Acked 7800 Read 7 New Done Done true
6802 Acked 7801 Write 3 New Done Done true
6803 Acked 7802 Flush 0 New Done Done true
6804 Acked 7803 Write 3 New Done Done true
6805 Acked 7804 Read 7 New Done Done true
6806 Acked 7805 Read 9 New Done Done true
6807 Acked 7806 Flush 0 New Done Done true
6808 Acked 7807 Read 6 New Done Done true
6809 Acked 7808 Read 8 New Done Done true
6810 Acked 7809 Read 7 New Done Done true
6811 Acked 7810 Read 1 New Done Done true
6812 Acked 7811 Flush 0 New Done Done true
6813 Acked 7812 Read 1 New Done Done true
6814 Acked 7813 Read 5 New Done Done true
6815 Acked 7814 Read 10 New Done Done true
6816 Acked 7815 Write 1 New Done Done true
6817 Acked 7816 Read 2 New Done Done true
6818 Acked 7817 Write 4 New Done Done true
6819 Acked 7818 Read 2 New Done Done true
6820 Acked 7819 Write 5 New Done Done true
6821 Acked 7820 Write 10 New Done Done true
6822 Acked 7821 Read 7 New Done Done true
6823 Acked 7822 Write 1 New Done Done true
6824 Acked 7823 Read 10 New Done Done true
6825 Acked 7824 Read 6 New Done Done true
6826 Acked 7825 Write 2 New Done Done true
6827 Acked 7826 Write 5 New Done Done true
6828 Acked 7827 Write 2 New Done Done true
6829 Acked 7828 Write 4 New Done Done true
6830 Acked 7829 Write 1 New Done Done true
6831 Acked 7830 Flush 0 New Done Done true
6832 Acked 7831 Read 4 New Done Done true
6833 Acked 7832 Write 1 New Done Done true
6834 Acked 7833 Write 2 New Done Done true
6835 Acked 7834 Flush 0 New Done Done true
6836 Acked 7835 Read 4 New Done Done true
6837 Acked 7836 Read 5 New Done Done true
6838 Acked 7837 Read 4 New Done Done true
6839 Acked 7838 Write 2 New Done Done true
6840 Acked 7839 Read 1 New Done Done true
6841 Acked 7840 Read 9 New Done Done true
6842 Acked 7841 Write 9 New Done Done true
6843 Acked 7842 Read 3 New Done Done true
6844 Acked 7843 Read 9 New Done Done true
6845 Acked 7844 Read 7 New Done Done true
6846 Acked 7845 Read 2 New Done Done true
6847 Acked 7846 Read 1 New Done Done true
6848 Acked 7847 Write 7 New Done Done true
6849 Acked 7848 Write 7 New Done Done true
6850 Acked 7849 Write 7 New Done Done true
6851 Acked 7850 Write 1 New Done Done true
6852 Acked 7851 Read 2 New Done Done true
6853 Acked 7852 Write 7 New Done Done true
6854 Acked 7853 Read 2 New Done Done true
6855 Acked 7854 Write 10 New Done Done true
6856 Acked 7855 Read 1 New Done Done true
6857 Acked 7856 Read 1 New Done Done true
6858 Acked 7857 Flush 0 New Done Done true
6859 Acked 7858 Read 3 New Done Done true
6860 Acked 7859 Flush 0 New Done Done true
6861 Acked 7860 Read 8 New Done Done true
6862 Acked 7861 Read 7 New Done Done true
6863 Acked 7862 Write 9 New Done Done true
6864 Acked 7863 Read 5 New Done Done true
6865 Acked 7864 Write 2 New Done Done true
6866 Acked 7865 Read 3 New Done Done true
6867 Acked 7866 Write 5 New Done Done true
6868 Acked 7867 Write 5 New Done Done true
6869 Acked 7868 Read 6 New Done Done true
6870 Acked 7869 Write 7 New Done Done true
6871 Acked 7870 Write 3 New Done Done true
6872 Acked 7871 Write 2 New Done Done true
6873 Acked 7872 Read 4 New Done Done true
6874 Acked 7873 Write 4 New Done Done true
6875 Acked 7874 Write 9 New Done Done true
6876 Acked 7875 Write 10 New Done Done true
6877 Acked 7876 Write 6 New Done Done true
6878 Acked 7877 Write 6 New Done Done true
6879 Acked 7878 Read 1 New Done Done true
6880 Acked 7879 Read 3 New Done Done true
6881 Acked 7880 Write 4 New Done Done true
6882 Acked 7881 Write 3 New Done Done true
6883 Acked 7882 Flush 0 New Done Done true
6884 Acked 7883 Write 7 New Done Done true
6885 Acked 7884 Write 1 New Done Done true
6886 Acked 7885 Write 4 New Done Done true
6887 Acked 7886 Read 6 New Done Done true
6888 Acked 7887 Read 9 New Done Done true
6889 Acked 7888 Write 3 New Done Done true
6890 Acked 7889 Write 5 New Done Done true
6891 Acked 7890 Read 2 New Done Done true
6892 Acked 7891 Read 10 New Done Done true
6893 Acked 7892 Read 4 New Done Done true
6894 Acked 7893 Write 7 New Done Done true
6895 Acked 7894 Read 7 New Done Done true
6896 Acked 7895 Write 3 New Done Done true
6897 Acked 7896 Write 3 New Done Done true
6898 Acked 7897 Read 2 New Done Done true
6899 Acked 7898 Flush 0 New Done Done true
6900 Acked 7899 Write 5 New Done Done true
6901 Acked 7900 Write 1 New Done Done true
6902 Acked 7901 Write 10 New Done Done true
6903 Acked 7902 Read 8 New Done Done true
6904 Acked 7903 Read 10 New Done Done true
6905 Acked 7904 Write 1 New Done Done true
6906 Acked 7905 Write 2 New Done Done true
6907 Acked 7906 Read 9 New Done Done true
6908 Acked 7907 Write 8 New Done Done true
6909 Acked 7908 Read 9 New Done Done true
6910 Acked 7909 Read 2 New Done Done true
6911 Acked 7910 Write 8 New Done Done true
6912 Acked 7911 Write 1 New Done Done true
6913 Acked 7912 Write 2 New Done Done true
6914 Acked 7913 Read 5 New Done Done true
6915 Acked 7914 Read 10 New Done Done true
6916 Acked 7915 Write 3 New Done Done true
6917 Acked 7916 Read 3 New Done Done true
6918 Acked 7917 Write 3 New Done Done true
6919 Acked 7918 Read 6 New Done Done true
6920 Acked 7919 Write 1 New Done Done true
6921 Acked 7920 Flush 0 New Done Done true
6922 Acked 7921 Read 10 New Done Done true
6923 Acked 7922 Read 7 New Done Done true
6924 Acked 7923 Read 4 New Done Done true
6925 Acked 7924 Write 5 New Done Done true
6926 Acked 7925 Read 6 New Done Done true
6927 Acked 7926 Write 3 New Done Done true
6928 Acked 7927 Write 9 New Done Done true
6929 Acked 7928 Write 6 New Done Done true
6930 Acked 7929 Write 2 New Done Done true
6931 Acked 7930 Write 4 New Done Done true
6932 Acked 7931 Flush 0 New Done Done true
STATES DS:0 DS:1 DS:2 TOTAL
New 302 0 0 302
Sent 0 0 0 0
Done 0 302 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 7629 7931 7931
Downstairs last five completed: 7629 7628 7627 7626 7625
Upstairs last five completed: 6932 6931 6930 6929 6928
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:48.833554826Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:48.833652622Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:48.834979251Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:48.8350182Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 7629","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:48.835035389Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 7629","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:48.835206773Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:48.835230702Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:48.835245171Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:48.83525912Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:48.83527702Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 7931 7930 7929 7928 7927
Upstairs last five completed: 6932 6931 6930 6929 6928
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Feb 13 21:43:52.606 INFO request completed, latency_us:[8810] Got stop action so:false kr:false
370, response_code: 204, uri: /stop/cid/0, method: POST, req_id: a9c1b708-657f-43ad-959b-4f1ebcc9cea4, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Feb 13 21:43:52.607 INFO request completed, latency_us: 127, response_code: 200, uri: /state/cid/0, method: GET, req_id: c2a954b8-d310-43f2-b4f9-21d5d091eb6e, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:52.607307203Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:52.607413609Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:52.607434798Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:52.607449248Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 7931","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:52.607467417Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:52.607493196Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:53.609348684Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:53.609406082Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:53.609428931Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:53.6094497Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:53.609471959Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 7931","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:53.609489839Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:53.609507218Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:54.611102825Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:54.611162632Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:54.611181372Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:54.611197621Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:54.611210201Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 7931","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:54.6112264Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:54.611256759Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:55.612960193Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:55.613015241Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:55.61303714Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:55.613064829Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:55.613081348Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 7931","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:55.613098647Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:55.613116007Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:43:56.610 INFO request completed, latency_us: 214, response_code: 200, uri: /state/cid/0, method: GET, req_id: 9dd912e2-7d91-464d-b572-085b1efe5cd6, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Write block 4187 len 12288 data: 5 4 4
002/300 Write block 978 len 40960 data: 7 5 5 5 3 3 3 4 7 8
003/300 Read block 2307 len 36864
004/300 Read block 1090 len 28672
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:56.61429413Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:56.614345938Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:56.614365587Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:56.614382937Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:56.614395916Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 3 jobs since flush 7931","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:56.614410006Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:56.614523811Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
005/300 Flush
006/300 Read block 2891 len 24576
007/300 Read block 3628 len 8192
008/300 Write block 4346 len 8192 data: 2 3
009/300 Read block 2742 len 40960
010/300 Write block 2043 len 24576 data: 5 3 3 4 5 7
011/300 Write block 4914 len 8192 data: 3 4
012/300 Read block 3871 len 12288
013/300 Read block 1529 len 20480
014/300 Write block 996 len 4096 data: 5
015/300 Flush
016/300 Write block 4720 len 32768 data: 4 6 11 8 9 9 9 8
017/300 Read block 613 len 12288
018/300 Flush
019/300 Read block 92 len 28672
020/300 Read block 3389 len 28672
021/300 Read block 2123 len 8192
022/300 Read block 2711 len 4096
023/300 Write block 1005 len 4096 data: 4
024/300 Write block 2672 len 12288 data: 3 4 4
025/300 Write block 3303 len 16384 data: 3 5 5 8
026/300 Write block 908 len 36864 data: 3 3 3 3 3 3 3 3 3
027/300 Write block 4845 len 24576 data: 5 3 4 3 4 3
028/300 Read block 3447 len 12288
029/300 Write block 3806 len 12288 data: 5 6 5
030/300 Write block 4939 len 12288 data: 6 7 7
031/300 Read block 3055 len 16384
032/300 Read block 4882 len 12288
033/300 Read block 3717 len 20480
034/300 Write block 4289 len 40960 data: 8 7 7 8 7 5 5 6 5 7
035/300 Read block 1089 len 40960
036/300 Read block 4928 len 20480
037/300 Read block 3927 len 16384
038/300 Write block 531 len 32768 data: 6 5 4 4 4 3 3 3
039/300 Read block 3935 len 8192
040/300 Write block 2496 len 36864 data: 3 5 6 7 7 8 8 9 10
041/300 Flush
042/300 Read block 2477 len 24576
043/300 Read block 544 len 28672
044/300 Read block 4927 len 32768
045/300 Write block 2628 len 12288 data: 6 6 4
046/300 Write block 129 len 28672 data: 10 10 10 8 6 5 4
047/300 Read block 4805 len 40960
048/300 Read block 863 len 20480
049/300 Read block 366 len 4096
050/300 Read block 406 len 16384
051/300 Read block 1180 len 36864
052/300 Read block 3617 len 16384
053/300 Write block 1770 len 4096 data: 5
054/300 Write block 4392 len 16384 data: 4 4 5 4
055/300 Read block 4324 len 12288
056/300 Read block 325 len 20480
057/300 Write block 72 len 28672 data: 6 7 8 7 6 5 5
058/300 Write block 4466 len 28672 data: 7 6 6 6 7 6 5
059/300 Write block 4658 len 36864 data: 4 5 6 5 6 7 8 8 9
060/300 Read block 3861 len 12288
061/300 Write block 2162 len 16384 data: 6 5 5 6
062/300 Read block 3641 len 40960
063/300 Read block 3540 len 8192
064/300 Write block 971 len 28672 data: 6 5 5 5 5 6 7
065/300 Write block 1831 len 12288 data: 5 5 5
066/300 Write block 4739 len 32768 data: 4 5 5 5 4 5 5 4
067/300 Read block 3679 len 40960
068/300 Flush
069/300 Read block 3306 len 40960
070/300 Write block 1895 len 16384 data: 7 5 5 5
071/300 Read block 4204 len 36864
072/300 Write block 3987 len 20480 data: 9 6 6 7 6
073/300 Write block 153 len 16384 data: 6 5 5 4
074/300 Read block 3312 len 28672
075/300 Flush
076/300 Write block 2938 len 20480 data: 5 5 4 5 5
077/300 Write block 931 len 32768 data: 3 4 4 4 5 5 5 4
078/300 Read block 1017 len 40960
079/300 Read block 2371 len 36864
080/300 Read block 4725 len 32768
081/300 Read block 2666 len 36864
082/300 Read block 4193 len 16384
083/300 Write block 2835 len 12288 data: 5 7 8
084/300 Write block 4651 len 28672 data: 7 4 4 4 3 4 4
085/300 Flush
086/300 Write block 1399 len 12288 data: 3 2 2
087/300 Read block 1559 len 28672
088/300 Write block 1816 len 4096 data: 3
089/300 Read block 1902 len 16384
090/300 Read block 4648 len 12288
091/300 Read block 337 len 8192
092/300 Write block 2829 len 20480 data: 5 7 4 5 6
093/300 Flush
094/300 Read block 2457 len 28672
095/300 Write block 3993 len 8192 data: 4 4
096/300 Write block 344 len 4096 data: 4
097/300 Read block 886 len 36864
098/300 Write block 4681 len 40960 data: 7 6 5 3 3 4 4 4 3 3
099/300 Read block 2759 len 24576
100/300 Write block 2311 len 12288 data: 6 6 5
101/300 Read block 4942 len 32768
102/300 Flush
103/300 Write block 1669 len 36864 data: 4 4 5 4 4 4 4 4 4
104/300 Write block 1136 len 32768 data: 5 3 3 5 4 4 4 4
105/300 Write block 1970 len 16384 data: 5 5 4 3
106/300 Write block 3929 len 12288 data: 3 3 3
107/300 Write block 3422 len 24576 data: 3 4 3 3 4 5
108/300 Read block 1845 len 16384
109/300 Read block 3430 len 8192
110/300 Read block 1738 len 36864
111/300 Read block 4072 len 20480
112/300 Write block 2185 len 40960 data: 6 5 5 4 4 5 5 4 5 6
113/300 Write block 2404 len 32768 data: 7 8 6 6 6 6 5 5
114/300 Read block 2257 len 16384
115/300 Flush
116/300 Read block 2245 len 16384
117/300 Write block 2458 len 8192 data: 5 5
118/300 Write block 1108 len 20480 data: 6 8 9 10 10
119/300 Write block 2968 len 40960 data: 6 6 6 6 6 7 7 7 7 7
120/300 Read block 82 len 24576
121/300 Read block 1317 len 32768
122/300 Read block 126 len 32768
123/300 Write block 4110 len 20480 data: 7 7 7 9 9
124/300 Write block 1026 len 8192 data: 3 4
125/300 Read block 2137 len 20480
126/300 Read block 2923 len 8192
127/300 Read block 53 len 36864
128/300 Write block 4417 len 12288 data: 3 4 5
129/300 Write block 3756 len 28672 data: 7 5 6 5 5 4 5
130/300 Write block 1878 len 12288 data: 4 4 4
131/300 Write block 1490 len 4096 data: 3
132/300 Write block 167 len 24576 data: 6 6 7 6 6 8
133/300 Write block 3464 len 40960 data: 2 2 2 3 2 2 2 2 2 2
134/300 Write block 849 len 28672 data: 6 7 5 4 4 6 7
135/300 Read block 1582 len 8192
136/300 Write block 2881 len 40960 data: 5 5 5 4 3 3 3 3 4 5
137/300 Flush
138/300 Read block 3785 len 24576
139/300 Read block 610 len 8192
140/300 Read block 1839 len 12288
141/300 Write block 512 len 20480 data: 3 3 3 3 3
142/300 Write block 733 len 12288 data: 9 8 5
143/300 Flush
144/300 Read block 4601 len 24576
145/300 Write block 3554 len 20480 data: 5 5 6 10 8
146/300 Read block 4515 len 36864
147/300 Write block 3571 len 4096 data: 3
148/300 Read block 2181 len 32768
149/300 Write block 1884 len 16384 data: 3 4 5 5
150/300 Read block 1659 len 8192
151/300 Write block 3137 len 36864 data: 6 5 6 6 6 5 5 6 7
152/300 Read block 1483 len 4096
153/300 Read block 4603 len 28672
154/300 Read block 4899 len 16384
155/300 Write block 1533 len 4096 data: 8
156/300 Write block 1872 len 24576 data: 3 3 4 4 4 4
157/300 Write block 676 len 4096 data: 4
158/300 Read block 1194 len 20480
159/300 Read block 729 len 40960
160/300 Write block 1273 len 16384 data: 4 5 5 5
161/300 Read block 2003 len 36864
162/300 Write block 1397 len 40960 data: 5 4 4 3 3 3 4 5 6 6
163/300 Write block 3909 len 40960 data: 6 5 5 4 5 5 6 5 6 5
164/300 Read block 2918 len 36864
165/300 Read block 2213 len 36864
166/300 Write block 3112 len 40960 data: 6 5 5 5 3 4 4 6 6 6
167/300 Read block 1736 len 12288
168/300 Flush
169/300 Write block 2816 len 8192 data: 4 4
170/300 Read block 3519 len 20480
171/300 Read block 3704 len 8192
172/300 Write block 2320 len 40960 data: 4 3 3 3 2 3 3 5 6 5
173/300 Read block 2107 len 24576
174/300 Write block 4009 len 32768 data: 6 5 4 4 4 5 5 3
175/300 Read block 84 len 8192
176/300 Write block 416 len 4096 data: 4
177/300 Read block 4803 len 36864
178/300 Read block 4861 len 32768
179/300 Read block 4038 len 16384
180/300 Read block 1178 len 16384
181/300 Write block 199 len 24576 data: 3 3 3 4 3 3
182/300 Read block 2485 len 8192
183/300 Write block 712 len 4096 data: 8
184/300 Read block 4926 len 20480
185/300 Read block 4706 len 20480
186/300 Read block 1406 len 28672
187/300 Read block 409 len 36864
188/300 Write block 2482 len 20480 data: 6 6 7 8 9
189/300 Write block 750 len 12288 data: 6 6 5
190/300 Write block 1639 len 40960 data: 8 9 8 8 8 10 8 7 5 5
191/300 Write block 2500 len 40960 data: 8 9 9 10 11 9 10 9 9 9
192/300 Write block 4704 len 36864 data: 3 6 4 5 6 6 5 6 8
193/300 Read block 3660 len 16384
194/300 Write block 3212 len 36864 data: 6 5 4 4 4 5 5 7 9
195/300 Write block 4924 len 4096 data: 3
196/300 Read block 2869 len 4096
197/300 Read block 4431 len 16384
198/300 Read block 4217 len 24576
199/300 Read block 2046 len 4096
200/300 Write block 2952 len 12288 data: 3 3 3
201/300 Write block 3299 len 40960 data: 5 4 3 3 4 6 6 9 7 7
202/300 Read block 507 len 28672
203/300 Write block 60 len 40960 data: 10 12 12 9 9 8 7 6 8 8
204/300 Read block 3162 len 16384
205/300 Write block 4319 len 40960 data: 6 8 7 9 11 10 10 10 11 9
206/300 Write block 2406 len 32768 data: 7 7 7 7 6 6 4 3
207/300 Write block 2898 len 8192 data: 4 3
208/300 Flush
209/300 Write block 4860 len 20480 data: 6 6 6 6 7
210/300 Read block 2483 len 32768
211/300 Read block 2763 len 40960
212/300 Read block 2199 len 36864
213/300 Read block 2289 len 24576
214/300 Write block 3497 len 4096 data: 3
215/300 Read block 73 len 16384
216/300 Flush
217/300 Read block 4963 len 4096
218/300 Write block 453 len 24576 data: 8 7 6 6 6 5
219/300 Read block 2852 len 16384
220/300 Read block 1595 len 20480
221/300 Read block 2829 len 40960
222/300 Write block 1169 len 12288 data: 5 4 4
223/300 Write block 3616 len 40960 data: 6 5 3 3 5 4 5 7 7 8
224/300 Read block 3297 len 12288
225/300 Flush
226/300 Read block 2232 len 8192
227/300 Read block 1950 len 16384
228/300 Write block 3158 len 4096 data: 4
229/300 Write block 4852 len 28672 data: 3 3 3 4 5 5 4
230/300 Read block 1979 len 12288
231/300 Read block 2714 len 8192
232/300 Write block 1659 len 36864 data: 10 8 7 5 5 4 5 4 2
233/300 Flush
234/300 Read block 386 len 36864
235/300 Write block 3404 len 8192 data: 10 10
236/300 Flush
237/300 Write block 3346 len 20480 data: 3 3 4 5 7
238/300 Read block 421 len 20480
239/300 Read block 419 len 16384
240/300 Write block 4377 len 36864 data: 4 5 8 7 6 4 7 8 8
241/300 Read block 1759 len 36864
242/300 Write block 4712 len 16384 data: 9 6 8 7
243/300 Write block 4741 len 8192 data: 6 6
244/300 Read block 852 len 36864
245/300 Write block 1825 len 12288 data: 4 4 2
246/300 Write block 1608 len 36864 data: 6 6 6 6 4 4 4 4 4
247/300 Read block 1376 len 28672
248/300 Read block 1726 len 4096
249/300 Flush
250/300 Read block 3977 len 40960
251/300 Read block 1406 len 20480
252/300 Read block 2160 len 36864
253/300 Flush
254/300 Flush
255/300 Read block 666 len 4096
256/300 Write block 1634 len 16384 data: 8 7 5 5
257/300 Write block 1939 len 16384 data: 7 7 6 6
258/300 Write block 4689 len 20480 data: 4 4 5 5 6
259/300 Write block 1804 len 40960 data: 5 4 4 5 3 4 5 3 3 4
260/300 Write block 95 len 20480 data: 6 7 6 5 5
261/300 Write block 1789 len 8192 data: 2 2
262/300 Write block 596 len 4096 data: 6
263/300 Read block 3743 len 28672
264/300 Write block 3367 len 36864 data: 6 8 10 10 8 10 11 9 7
265/300 Write block 4812 len 32768 data: 6 5 5 5 5 5 5 4
266/300 Write block 3620 len 4096 data: 6
267/300 Write block 4980 len 8192 data: 2 2
268/300 Write block 1047 len 32768 data: 2 2 2 3 3 3 5 5
269/300 Flush
270/300 Read block 851 len 12288
271/300 Write block 2579 len 24576 data: 5 6 4 5 6 4
272/300 Write block 4563 len 36864 data: 3 4 4 5 5 4 4 3 4
273/300 Read block 1112 len 28672
274/300 Write block 3957 len 28672 data: 5 4 4 3 4 4 4
275/300 Write block 3417 len 4096 data: 2
276/300 Write block 3799 len 4096 data: 5
277/300 Read block 2439 len 20480
278/300 Write block 892 len 40960 data: 6 7 6 6 6 6 6 5 5 5
279/300 Write block 3513 len 40960 data: 7 8 7 7 8 6 5 5 5 5
280/300 Read block 1104 len 12288
281/300 Read block 4087 len 8192
282/300 Flush
283/300 Write block 3491 len 40960 data: 4 2 3 3 5 4 4 5 5 5
284/300 Write block 2605 len 24576 data: 2 3 3 3 3 3
285/300 Read block 3820 len 4096
286/300 Read block 1202 len 16384
287/300 Read block 2886 len 36864
288/300 Flush
289/300 Read block 1417 len 4096
290/300 Read block 3884 len 28672
291/300 Read block 1164 len 36864
292/300 Read block 2680 len 8192
293/300 Read block 1286 len 20480
294/300 Read block 486 len 36864
295/300 Read block 2645 len 28672
296/300 Read block 1383 len 16384
297/300 Write block 787 len 4096 data: 4
298/300 Read block 2747 len 40960
299/300 Read block 2995 len 16384
300/300 Write block 1464 len 24576 data: 6 7 7 8 9 6
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 13 21:43:57.389 INFO request completed, latency_us: 232, response_code: 204, uri: /start/cid/0, method: POST, req_id: 9d21c6fe-32bc-4075-a049-818e2a1a26ef, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
6933 Acked 7932 Write 3 New Done Done true
6934 Acked 7933 Write 10 New Done Done true
6935 Acked 7934 Read 9 New Done Done true
6936 Acked 7935 Read 7 New Done Done false
6937 Acked 7936 Flush 0 New Done Done false
6938 Acked 7937 Read 6 New Done Done false
6939 Acked 7938 Read 2 New Done Done false
6940 Acked 7939 Write 2 New Done Done false
6941 Acked 7940 Read 10 New Done Done false
6942 Acked 7941 Write 6 New Done Done false
6943 Acked 7942 Write 2 New Done Done false
6944 Acked 7943 Read 3 New Done Done false
6945 Acked 7944 Read 5 New Done Done false
6946 Acked 7945 Write 1 New Done Done false
6947 Acked 7946 Flush 0 New Done Done false
6948 Acked 7947 Write 8 New Done Done false
6949 Acked 7948 Read 3 New Done Done false
6950 Acked 7949 Flush 0 New Done Done false
6951 Acked 7950 Read 7 New Done Done false
6952 Acked 7951 Read 7 New Done Done false
6953 Acked 7952 Read 2 New Done Done false
6954 Acked 7953 Read 1 New Done Done false
6955 Acked 7954 Write 1 New Done Done false
6956 Acked 7955 Write 3 New Done Done false
6957 Acked 7956 Write 4 New Done Done false
6958 Acked 7957 Write 9 New Done Done false
6959 Acked 7958 Write 6 New Done Done false
6960 Acked 7959 Read 3 New Done Done false
6961 Acked 7960 Write 3 New Done Done false
6962 Acked 7961 Write 3 New Done Done false
6963 Acked 7962 Read 4 New Done Done false
6964 Acked 7963 Read 3 New Done Done false
6965 Acked 7964 Read 5 New Done Done false
6966 Acked 7965 Write 10 New Done Done false
6967 Acked 7966 Read 10 New Done Done false
6968 Acked 7967 Read 5 New Done Done false
6969 Acked 7968 Read 4 New Done Done false
6970 Acked 7969 Write 8 New Done Done false
6971 Acked 7970 Read 2 New Done Done false
6972 Acked 7971 Write 9 New Done Done false
6973 Acked 7972 Flush 0 New Done Done false
6974 Acked 7973 Read 6 New Done Done false
6975 Acked 7974 Read 7 New Done Done false
6976 Acked 7975 Read 8 New Done Done false
6977 Acked 7976 Write 3 New Done Done false
6978 Acked 7977 Write 7 New Done Done false
6979 Acked 7978 Read 10 New Done Done false
6980 Acked 7979 Read 5 New Done Done false
6981 Acked 7980 Read 1 New Done Done false
6982 Acked 7981 Read 4 New Done Done false
6983 Acked 7982 Read 9 New Done Done false
6984 Acked 7983 Read 4 New Done Done false
6985 Acked 7984 Write 1 New Done Done false
6986 Acked 7985 Write 4 New Done Done false
6987 Acked 7986 Read 3 New Done Done false
6988 Acked 7987 Read 5 New Done Done false
6989 Acked 7988 Write 7 New Done Done false
6990 Acked 7989 Write 7 New Done Done false
6991 Acked 7990 Write 9 New Done Done false
6992 Acked 7991 Read 3 New Done Done false
6993 Acked 7992 Write 4 New Done Done false
6994 Acked 7993 Read 10 New Done Done false
6995 Acked 7994 Read 2 New Done Done false
6996 Acked 7995 Write 7 New Done Done false
6997 Acked 7996 Write 3 New Done Done false
6998 Acked 7997 Write 8 New Done Done false
6999 Acked 7998 Read 10 New Done Done false
7000 Acked 7999 Flush 0 New Done Done false
7001 Acked 8000 Read 10 New Done Done false
7002 Acked 8001 Write 4 New Done Done false
7003 Acked 8002 Read 9 New Done Done false
7004 Acked 8003 Write 5 New Done Done false
7005 Acked 8004 Write 4 New Done Done false
7006 Acked 8005 Read 7 New Done Done false
7007 Acked 8006 Flush 0 New Done Done false
7008 Acked 8007 Write 5 New Done Done false
7009 Acked 8008 Write 8 New Done Done false
7010 Acked 8009 Read 10 New Done Done false
7011 Acked 8010 Read 9 New Done Done false
7012 Acked 8011 Read 8 New Done Done false
7013 Acked 8012 Read 9 New Done Done false
7014 Acked 8013 Read 4 New Done Done false
7015 Acked 8014 Write 3 New Done Done false
7016 Acked 8015 Write 7 New Done Done false
7017 Acked 8016 Flush 0 New Done Done false
7018 Acked 8017 Write 3 New Done Done false
7019 Acked 8018 Read 7 New Done Done false
7020 Acked 8019 Write 1 New Done Done false
7021 Acked 8020 Read 4 New Done Done false
7022 Acked 8021 Read 3 New Done Done false
7023 Acked 8022 Read 2 New Done Done false
7024 Acked 8023 Write 5 New Done Done false
7025 Acked 8024 Flush 0 New Done Done false
7026 Acked 8025 Read 7 New Done Done false
7027 Acked 8026 Write 2 New Done Done false
7028 Acked 8027 Write 1 New Done Done false
7029 Acked 8028 Read 9 New Done Done false
7030 Acked 8029 Write 10 New Done Done false
7031 Acked 8030 Read 6 New Done Done false
7032 Acked 8031 Write 3 New Done Done false
7033 Acked 8032 Read 8 New Done Done false
7034 Acked 8033 Flush 0 New Done Done false
7035 Acked 8034 Write 9 New Done Done false
7036 Acked 8035 Write 8 New Done Done false
7037 Acked 8036 Write 4 New Done Done false
7038 Acked 8037 Write 3 New Done Done false
7039 Acked 8038 Write 6 New Done Done false
7040 Acked 8039 Read 4 New Done Done false
7041 Acked 8040 Read 2 New Done Done false
7042 Acked 8041 Read 9 New Done Done false
7043 Acked 8042 Read 5 New Done Done false
7044 Acked 8043 Write 10 New Done Done false
7045 Acked 8044 Write 8 New Done Done false
7046 Acked 8045 Read 4 New Done Done false
7047 Acked 8046 Flush 0 New Done Done false
7048 Acked 8047 Read 4 New Done Done false
7049 Acked 8048 Write 2 New Done Done false
7050 Acked 8049 Write 5 New Done Done false
7051 Acked 8050 Write 10 New Done Done false
7052 Acked 8051 Read 6 New Done Done false
7053 Acked 8052 Read 8 New Done Done false
7054 Acked 8053 Read 8 New Done Done false
7055 Acked 8054 Write 5 New Done Done false
7056 Acked 8055 Write 2 New Done Done false
7057 Acked 8056 Read 5 New Done Done false
7058 Acked 8057 Read 2 New Done Done false
7059 Acked 8058 Read 9 New Done Done false
7060 Acked 8059 Write 3 New Done Done false
7061 Acked 8060 Write 7 New Done Done false
7062 Acked 8061 Write 3 New Done Done false
7063 Acked 8062 Write 1 New Done Done false
7064 Acked 8063 Write 6 New Done Done false
7065 Acked 8064 Write 10 New Done Done false
7066 Acked 8065 Write 7 New Done Done false
7067 Acked 8066 Read 2 New Done Done false
7068 Acked 8067 Write 10 New Done Done false
7069 Acked 8068 Flush 0 New Done Done false
7070 Acked 8069 Read 6 New Done Done false
7071 Acked 8070 Read 2 New Done Done false
7072 Acked 8071 Read 3 New Done Done false
7073 Acked 8072 Write 5 New Done Done false
7074 Acked 8073 Write 3 New Done Done false
7075 Acked 8074 Flush 0 New Done Done false
7076 Acked 8075 Read 6 New Done Done false
7077 Acked 8076 Write 5 New Done Done false
7078 Acked 8077 Read 9 New Done Done false
7079 Acked 8078 Write 1 New Done Done false
7080 Acked 8079 Read 8 New Done Done false
7081 Acked 8080 Write 4 New Done Done false
7082 Acked 8081 Read 2 New Done Done false
7083 Acked 8082 Flush 0 New Done Done false
7084 Acked 8083 Write 9 New Done Done false
7085 Acked 8084 Read 1 New Done Done false
7086 Acked 8085 Read 7 New Done Done false
7087 Acked 8086 Read 4 New Done Done false
7088 Acked 8087 Write 1 New Done Done false
7089 Acked 8088 Write 6 New Done Done false
7090 Acked 8089 Write 1 New Done Done false
7091 Acked 8090 Read 5 New Done Done false
7092 Acked 8091 Read 10 New Done Done false
7093 Acked 8092 Write 4 New Done Done false
7094 Acked 8093 Read 9 New Done Done false
7095 Acked 8094 Write 10 New Done Done false
7096 Acked 8095 Write 10 New Done Done false
7097 Acked 8096 Read 9 New Done Done false
7098 Acked 8097 Read 9 New Done Done false
7099 Acked 8098 Write 10 New Done Done false
7100 Acked 8099 Read 3 New Done Done false
7101 Acked 8100 Flush 0 New Done Done false
7102 Acked 8101 Write 2 New Done Done false
7103 Acked 8102 Read 5 New Done Done false
7104 Acked 8103 Read 2 New Done Done false
7105 Acked 8104 Write 10 New Done Done false
7106 Acked 8105 Read 6 New Done Done false
7107 Acked 8106 Write 8 New Done Done false
7108 Acked 8107 Read 2 New Done Done false
7109 Acked 8108 Write 1 New Done Done false
7110 Acked 8109 Read 9 New Done Done false
7111 Acked 8110 Read 8 New Done Done false
7112 Acked 8111 Read 4 New Done Done false
7113 Acked 8112 Read 4 New Done Done false
7114 Acked 8113 Write 6 New Done Done false
7115 Acked 8114 Read 2 New Done Done false
7116 Acked 8115 Write 1 New Done Done false
7117 Acked 8116 Read 5 New Done Done false
7118 Acked 8117 Read 5 New Done Done false
7119 Acked 8118 Read 7 New Done Done false
7120 Acked 8119 Read 9 New Done Done false
7121 Acked 8120 Write 5 New Done Done false
7122 Acked 8121 Write 3 New Done Done false
7123 Acked 8122 Write 10 New Done Done false
7124 Acked 8123 Write 10 New Done Done false
7125 Acked 8124 Write 9 New Done Done false
7126 Acked 8125 Read 4 New Done Done false
7127 Acked 8126 Write 9 New Done Done false
7128 Acked 8127 Write 1 New Done Done false
7129 Acked 8128 Read 1 New Done Done false
7130 Acked 8129 Read 4 New Done Done false
7131 Acked 8130 Read 6 New Done Done false
7132 Acked 8131 Read 1 New Done Done false
7133 Acked 8132 Write 3 New Done Done false
7134 Acked 8133 Write 10 New Done Done false
7135 Acked 8134 Read 7 New Done Done false
7136 Acked 8135 Write 10 New Done Done false
7137 Acked 8136 Read 4 New Done Done false
7138 Acked 8137 Write 10 New Done Done false
7139 Acked 8138 Write 8 New Done Done false
7140 Acked 8139 Write 2 New Done Done false
7141 Acked 8140 Flush 0 New Done Done false
7142 Acked 8141 Write 5 New Done Done false
7143 Acked 8142 Read 8 New Done Done false
7144 Acked 8143 Read 10 New Done Done false
7145 Acked 8144 Read 9 New Done Done false
7146 Acked 8145 Read 6 New Done Done false
7147 Acked 8146 Write 1 New Done Done false
7148 Acked 8147 Read 4 New Done Done false
7149 Acked 8148 Flush 0 New Done Done false
7150 Acked 8149 Read 1 New Done Done false
7151 Acked 8150 Write 6 New Done Done false
7152 Acked 8151 Read 4 New Done Done false
7153 Acked 8152 Read 5 New Done Done false
7154 Acked 8153 Read 10 New Done Done false
7155 Acked 8154 Write 3 New Done Done false
7156 Acked 8155 Write 10 New Done Done false
7157 Acked 8156 Read 3 New Done Done false
7158 Acked 8157 Flush 0 New Done Done false
7159 Acked 8158 Read 2 New Done Done false
7160 Acked 8159 Read 4 New Done Done false
7161 Acked 8160 Write 1 New Done Done false
7162 Acked 8161 Write 7 New Done Done false
7163 Acked 8162 Read 3 New Done Done false
7164 Acked 8163 Read 2 New Done Done false
7165 Acked 8164 Write 9 New Done Done false
7166 Acked 8165 Flush 0 New Done Done false
7167 Acked 8166 Read 9 New Done Done false
7168 Acked 8167 Write 2 New Done Done false
7169 Acked 8168 Flush 0 New Done Done false
7170 Acked 8169 Write 5 New Done Done false
7171 Acked 8170 Read 5 New Done Done false
7172 Acked 8171 Read 4 New Done Done false
7173 Acked 8172 Write 9 New Done Done false
7174 Acked 8173 Read 9 New Done Done false
7175 Acked 8174 Write 4 New Done Done false
7176 Acked 8175 Write 2 New Done Done false
7177 Acked 8176 Read 9 New Done Done false
7178 Acked 8177 Write 3 New Done Done false
7179 Acked 8178 Write 9 New Done Done false
7180 Acked 8179 Read 7 New Done Done false
7181 Acked 8180 Read 1 New Done Done false
7182 Acked 8181 Flush 0 New Done Done false
7183 Acked 8182 Read 10 New Done Done false
7184 Acked 8183 Read 5 New Done Done false
7185 Acked 8184 Read 9 New Done Done false
7186 Acked 8185 Flush 0 New Done Done false
7187 Acked 8186 Flush 0 New Done Done false
7188 Acked 8187 Read 1 New Done Done false
7189 Acked 8188 Write 4 New Done Done false
7190 Acked 8189 Write 4 New Done Done false
7191 Acked 8190 Write 5 New Done Done false
7192 Acked 8191 Write 10 New Done Done false
7193 Acked 8192 Write 5 New Done Done false
7194 Acked 8193 Write 2 New Done Done false
7195 Acked 8194 Write 1 New Done Done false
7196 Acked 8195 Read 7 New Done Done false
7197 Acked 8196 Write 9 New Done Done false
7198 Acked 8197 Write 8 New Done Done false
7199 Acked 8198 Write 1 New Done Done false
7200 Acked 8199 Write 2 New Done Done false
7201 Acked 8200 Write 8 New Done Done false
7202 Acked 8201 Flush 0 New Done Done false
7203 Acked 8202 Read 3 New Done Done false
7204 Acked 8203 Write 6 New Done Done false
7205 Acked 8204 Write 9 New Done Done false
7206 Acked 8205 Read 7 New Done Done false
7207 Acked 8206 Write 7 New Done Done false
7208 Acked 8207 Write 1 New Done Done false
7209 Acked 8208 Write 1 New Done Done false
7210 Acked 8209 Read 5 New Done Done false
7211 Acked 8210 Write 10 New Done Done false
7212 Acked 8211 Write 10 New Done Done false
7213 Acked 8212 Read 3 New Done Done false
7214 Acked 8213 Read 2 New Done Done false
7215 Acked 8214 Flush 0 New Done Done false
7216 Acked 8215 Write 10 New Done Done false
7217 Acked 8216 Write 6 New Done Done false
7218 Acked 8217 Read 1 New Done Done false
7219 Acked 8218 Read 4 New Done Done false
7220 Acked 8219 Read 9 New Done Done false
7221 Acked 8220 Flush 0 New Done Done false
7222 Acked 8221 Read 1 New Done Done false
7223 Acked 8222 Read 7 New Done Done false
7224 Acked 8223 Read 9 New Done Done false
7225 Acked 8224 Read 2 New Done Done false
7226 Acked 8225 Read 5 New Done Done false
7227 Acked 8226 Read 9 New Done Done false
7228 Acked 8227 Read 7 New Done Done false
7229 Acked 8228 Read 4 New Done Done false
7230 Acked 8229 Write 1 New Done Done false
7231 Acked 8230 Read 10 New Done Done false
7232 Acked 8231 Read 4 New Done Done false
7233 Acked 8232 Write 6 New Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
New 301 0 0 301
Sent 0 0 0 0
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 7931 8220 8220
Downstairs last five completed: 7931 7930 7929 7928 7927
Upstairs last five completed: 7233 7232 7231 7230 7229
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:57.61633543Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:57.616393438Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:57.616425947Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:57.616467065Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:57.616500844Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 7931","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:57.616532823Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:57.616560262Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:58.617422207Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:58.617476875Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:58.617496335Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:58.617513314Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:58.617526154Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 7931","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:58.617547903Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:58.617673068Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:59.619325133Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:59.61938702Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:59.61941046Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:43:59.619431219Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:59.619447018Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 7931","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:59.619464408Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:43:59.619601942Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:00.621115194Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:00.621175621Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:00.62119754Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:00.621218039Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:00.621241958Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 7931","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:00.621260018Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:00.621342865Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
6933 Acked 7932 Write 3 New Done Done true
6934 Acked 7933 Write 10 New Done Done true
6935 Acked 7934 Read 9 New Done Done true
6936 Acked 7935 Read 7 New Done Done true
6937 Acked 7936 Flush 0 New Done Done true
6938 Acked 7937 Read 6 New Done Done true
6939 Acked 7938 Read 2 New Done Done true
6940 Acked 7939 Write 2 New Done Done true
6941 Acked 7940 Read 10 New Done Done true
6942 Acked 7941 Write 6 New Done Done true
6943 Acked 7942 Write 2 New Done Done true
6944 Acked 7943 Read 3 New Done Done true
6945 Acked 7944 Read 5 New Done Done true
6946 Acked 7945 Write 1 New Done Done true
6947 Acked 7946 Flush 0 New Done Done true
6948 Acked 7947 Write 8 New Done Done true
6949 Acked 7948 Read 3 New Done Done true
6950 Acked 7949 Flush 0 New Done Done true
6951 Acked 7950 Read 7 New Done Done true
6952 Acked 7951 Read 7 New Done Done true
6953 Acked 7952 Read 2 New Done Done true
6954 Acked 7953 Read 1 New Done Done true
6955 Acked 7954 Write 1 New Done Done true
6956 Acked 7955 Write 3 New Done Done true
6957 Acked 7956 Write 4 New Done Done true
6958 Acked 7957 Write 9 New Done Done true
6959 Acked 7958 Write 6 New Done Done true
6960 Acked 7959 Read 3 New Done Done true
6961 Acked 7960 Write 3 New Done Done true
6962 Acked 7961 Write 3 New Done Done true
6963 Acked 7962 Read 4 New Done Done true
6964 Acked 7963 Read 3 New Done Done true
6965 Acked 7964 Read 5 New Done Done true
6966 Acked 7965 Write 10 New Done Done true
6967 Acked 7966 Read 10 New Done Done true
6968 Acked 7967 Read 5 New Done Done true
6969 Acked 7968 Read 4 New Done Done true
6970 Acked 7969 Write 8 New Done Done true
6971 Acked 7970 Read 2 New Done Done true
6972 Acked 7971 Write 9 New Done Done true
6973 Acked 7972 Flush 0 New Done Done true
6974 Acked 7973 Read 6 New Done Done true
6975 Acked 7974 Read 7 New Done Done true
6976 Acked 7975 Read 8 New Done Done true
6977 Acked 7976 Write 3 New Done Done true
6978 Acked 7977 Write 7 New Done Done true
6979 Acked 7978 Read 10 New Done Done true
6980 Acked 7979 Read 5 New Done Done true
6981 Acked 7980 Read 1 New Done Done true
6982 Acked 7981 Read 4 New Done Done true
6983 Acked 7982 Read 9 New Done Done true
6984 Acked 7983 Read 4 New Done Done true
6985 Acked 7984 Write 1 New Done Done true
6986 Acked 7985 Write 4 New Done Done true
6987 Acked 7986 Read 3 New Done Done true
6988 Acked 7987 Read 5 New Done Done true
6989 Acked 7988 Write 7 New Done Done true
6990 Acked 7989 Write 7 New Done Done true
6991 Acked 7990 Write 9 New Done Done true
6992 Acked 7991 Read 3 New Done Done true
6993 Acked 7992 Write 4 New Done Done true
6994 Acked 7993 Read 10 New Done Done true
6995 Acked 7994 Read 2 New Done Done true
6996 Acked 7995 Write 7 New Done Done true
6997 Acked 7996 Write 3 New Done Done true
6998 Acked 7997 Write 8 New Done Done true
6999 Acked 7998 Read 10 New Done Done true
7000 Acked 7999 Flush 0 New Done Done true
7001 Acked 8000 Read 10 New Done Done true
7002 Acked 8001 Write 4 New Done Done true
7003 Acked 8002 Read 9 New Done Done true
7004 Acked 8003 Write 5 New Done Done true
7005 Acked 8004 Write 4 New Done Done true
7006 Acked 8005 Read 7 New Done Done true
7007 Acked 8006 Flush 0 New Done Done true
7008 Acked 8007 Write 5 New Done Done true
7009 Acked 8008 Write 8 New Done Done true
7010 Acked 8009 Read 10 New Done Done true
7011 Acked 8010 Read 9 New Done Done true
7012 Acked 8011 Read 8 New Done Done true
7013 Acked 8012 Read 9 New Done Done true
7014 Acked 8013 Read 4 New Done Done true
7015 Acked 8014 Write 3 New Done Done true
7016 Acked 8015 Write 7 New Done Done true
7017 Acked 8016 Flush 0 New Done Done true
7018 Acked 8017 Write 3 New Done Done true
7019 Acked 8018 Read 7 New Done Done true
7020 Acked 8019 Write 1 New Done Done true
7021 Acked 8020 Read 4 New Done Done true
7022 Acked 8021 Read 3 New Done Done true
7023 Acked 8022 Read 2 New Done Done true
7024 Acked 8023 Write 5 New Done Done true
7025 Acked 8024 Flush 0 New Done Done true
7026 Acked 8025 Read 7 New Done Done true
7027 Acked 8026 Write 2 New Done Done true
7028 Acked 8027 Write 1 New Done Done true
7029 Acked 8028 Read 9 New Done Done true
7030 Acked 8029 Write 10 New Done Done true
7031 Acked 8030 Read 6 New Done Done true
7032 Acked 8031 Write 3 New Done Done true
7033 Acked 8032 Read 8 New Done Done true
7034 Acked 8033 Flush 0 New Done Done true
7035 Acked 8034 Write 9 New Done Done true
7036 Acked 8035 Write 8 New Done Done true
7037 Acked 8036 Write 4 New Done Done true
7038 Acked 8037 Write 3 New Done Done true
7039 Acked 8038 Write 6 New Done Done true
7040 Acked 8039 Read 4 New Done Done true
7041 Acked 8040 Read 2 New Done Done true
7042 Acked 8041 Read 9 New Done Done true
7043 Acked 8042 Read 5 New Done Done true
7044 Acked 8043 Write 10 New Done Done true
7045 Acked 8044 Write 8 New Done Done true
7046 Acked 8045 Read 4 New Done Done true
7047 Acked 8046 Flush 0 New Done Done true
7048 Acked 8047 Read 4 New Done Done true
7049 Acked 8048 Write 2 New Done Done true
7050 Acked 8049 Write 5 New Done Done true
7051 Acked 8050 Write 10 New Done Done true
7052 Acked 8051 Read 6 New Done Done true
7053 Acked 8052 Read 8 New Done Done true
7054 Acked 8053 Read 8 New Done Done true
7055 Acked 8054 Write 5 New Done Done true
7056 Acked 8055 Write 2 New Done Done true
7057 Acked 8056 Read 5 New Done Done true
7058 Acked 8057 Read 2 New Done Done true
7059 Acked 8058 Read 9 New Done Done true
7060 Acked 8059 Write 3 New Done Done true
7061 Acked 8060 Write 7 New Done Done true
7062 Acked 8061 Write 3 New Done Done true
7063 Acked 8062 Write 1 New Done Done true
7064 Acked 8063 Write 6 New Done Done true
7065 Acked 8064 Write 10 New Done Done true
7066 Acked 8065 Write 7 New Done Done true
7067 Acked 8066 Read 2 New Done Done true
7068 Acked 8067 Write 10 New Done Done true
7069 Acked 8068 Flush 0 New Done Done true
7070 Acked 8069 Read 6 New Done Done true
7071 Acked 8070 Read 2 New Done Done true
7072 Acked 8071 Read 3 New Done Done true
7073 Acked 8072 Write 5 New Done Done true
7074 Acked 8073 Write 3 New Done Done true
7075 Acked 8074 Flush 0 New Done Done true
7076 Acked 8075 Read 6 New Done Done true
7077 Acked 8076 Write 5 New Done Done true
7078 Acked 8077 Read 9 New Done Done true
7079 Acked 8078 Write 1 New Done Done true
7080 Acked 8079 Read 8 New Done Done true
7081 Acked 8080 Write 4 New Done Done true
7082 Acked 8081 Read 2 New Done Done true
7083 Acked 8082 Flush 0 New Done Done true
7084 Acked 8083 Write 9 New Done Done true
7085 Acked 8084 Read 1 New Done Done true
7086 Acked 8085 Read 7 New Done Done true
7087 Acked 8086 Read 4 New Done Done true
7088 Acked 8087 Write 1 New Done Done true
7089 Acked 8088 Write 6 New Done Done true
7090 Acked 8089 Write 1 New Done Done true
7091 Acked 8090 Read 5 New Done Done true
7092 Acked 8091 Read 10 New Done Done true
7093 Acked 8092 Write 4 New Done Done true
7094 Acked 8093 Read 9 New Done Done true
7095 Acked 8094 Write 10 New Done Done true
7096 Acked 8095 Write 10 New Done Done true
7097 Acked 8096 Read 9 New Done Done true
7098 Acked 8097 Read 9 New Done Done true
7099 Acked 8098 Write 10 New Done Done true
7100 Acked 8099 Read 3 New Done Done true
7101 Acked 8100 Flush 0 New Done Done true
7102 Acked 8101 Write 2 New Done Done true
7103 Acked 8102 Read 5 New Done Done true
7104 Acked 8103 Read 2 New Done Done true
7105 Acked 8104 Write 10 New Done Done true
7106 Acked 8105 Read 6 New Done Done true
7107 Acked 8106 Write 8 New Done Done true
7108 Acked 8107 Read 2 New Done Done true
7109 Acked 8108 Write 1 New Done Done true
7110 Acked 8109 Read 9 New Done Done true
7111 Acked 8110 Read 8 New Done Done true
7112 Acked 8111 Read 4 New Done Done true
7113 Acked 8112 Read 4 New Done Done true
7114 Acked 8113 Write 6 New Done Done true
7115 Acked 8114 Read 2 New Done Done true
7116 Acked 8115 Write 1 New Done Done true
7117 Acked 8116 Read 5 New Done Done true
7118 Acked 8117 Read 5 New Done Done true
7119 Acked 8118 Read 7 New Done Done true
7120 Acked 8119 Read 9 New Done Done true
7121 Acked 8120 Write 5 New Done Done true
7122 Acked 8121 Write 3 New Done Done true
7123 Acked 8122 Write 10 New Done Done true
7124 Acked 8123 Write 10 New Done Done true
7125 Acked 8124 Write 9 New Done Done true
7126 Acked 8125 Read 4 New Done Done true
7127 Acked 8126 Write 9 New Done Done true
7128 Acked 8127 Write 1 New Done Done true
7129 Acked 8128 Read 1 New Done Done true
7130 Acked 8129 Read 4 New Done Done true
7131 Acked 8130 Read 6 New Done Done true
7132 Acked 8131 Read 1 New Done Done true
7133 Acked 8132 Write 3 New Done Done true
7134 Acked 8133 Write 10 New Done Done true
7135 Acked 8134 Read 7 New Done Done true
7136 Acked 8135 Write 10 New Done Done true
7137 Acked 8136 Read 4 New Done Done true
7138 Acked 8137 Write 10 New Done Done true
7139 Acked 8138 Write 8 New Done Done true
7140 Acked 8139 Write 2 New Done Done true
7141 Acked 8140 Flush 0 New Done Done true
7142 Acked 8141 Write 5 New Done Done true
7143 Acked 8142 Read 8 New Done Done true
7144 Acked 8143 Read 10 New Done Done true
7145 Acked 8144 Read 9 New Done Done true
7146 Acked 8145 Read 6 New Done Done true
7147 Acked 8146 Write 1 New Done Done true
7148 Acked 8147 Read 4 New Done Done true
7149 Acked 8148 Flush 0 New Done Done true
7150 Acked 8149 Read 1 New Done Done true
7151 Acked 8150 Write 6 New Done Done true
7152 Acked 8151 Read 4 New Done Done true
7153 Acked 8152 Read 5 New Done Done true
7154 Acked 8153 Read 10 New Done Done true
7155 Acked 8154 Write 3 New Done Done true
7156 Acked 8155 Write 10 New Done Done true
7157 Acked 8156 Read 3 New Done Done true
7158 Acked 8157 Flush 0 New Done Done true
7159 Acked 8158 Read 2 New Done Done true
7160 Acked 8159 Read 4 New Done Done true
7161 Acked 8160 Write 1 New Done Done true
7162 Acked 8161 Write 7 New Done Done true
7163 Acked 8162 Read 3 New Done Done true
7164 Acked 8163 Read 2 New Done Done true
7165 Acked 8164 Write 9 New Done Done true
7166 Acked 8165 Flush 0 New Done Done true
7167 Acked 8166 Read 9 New Done Done true
7168 Acked 8167 Write 2 New Done Done true
7169 Acked 8168 Flush 0 New Done Done true
7170 Acked 8169 Write 5 New Done Done true
7171 Acked 8170 Read 5 New Done Done true
7172 Acked 8171 Read 4 New Done Done true
7173 Acked 8172 Write 9 New Done Done true
7174 Acked 8173 Read 9 New Done Done true
7175 Acked 8174 Write 4 New Done Done true
7176 Acked 8175 Write 2 New Done Done true
7177 Acked 8176 Read 9 New Done Done true
7178 Acked 8177 Write 3 New Done Done true
7179 Acked 8178 Write 9 New Done Done true
7180 Acked 8179 Read 7 New Done Done true
7181 Acked 8180 Read 1 New Done Done true
7182 Acked 8181 Flush 0 New Done Done true
7183 Acked 8182 Read 10 New Done Done true
7184 Acked 8183 Read 5 New Done Done true
7185 Acked 8184 Read 9 New Done Done true
7186 Acked 8185 Flush 0 New Done Done true
7187 Acked 8186 Flush 0 New Done Done true
7188 Acked 8187 Read 1 New Done Done true
7189 Acked 8188 Write 4 New Done Done true
7190 Acked 8189 Write 4 New Done Done true
7191 Acked 8190 Write 5 New Done Done true
7192 Acked 8191 Write 10 New Done Done true
7193 Acked 8192 Write 5 New Done Done true
7194 Acked 8193 Write 2 New Done Done true
7195 Acked 8194 Write 1 New Done Done true
7196 Acked 8195 Read 7 New Done Done true
7197 Acked 8196 Write 9 New Done Done true
7198 Acked 8197 Write 8 New Done Done true
7199 Acked 8198 Write 1 New Done Done true
7200 Acked 8199 Write 2 New Done Done true
7201 Acked 8200 Write 8 New Done Done true
7202 Acked 8201 Flush 0 New Done Done true
7203 Acked 8202 Read 3 New Done Done true
7204 Acked 8203 Write 6 New Done Done true
7205 Acked 8204 Write 9 New Done Done true
7206 Acked 8205 Read 7 New Done Done true
7207 Acked 8206 Write 7 New Done Done true
7208 Acked 8207 Write 1 New Done Done true
7209 Acked 8208 Write 1 New Done Done true
7210 Acked 8209 Read 5 New Done Done true
7211 Acked 8210 Write 10 New Done Done true
7212 Acked 8211 Write 10 New Done Done true
7213 Acked 8212 Read 3 New Done Done true
7214 Acked 8213 Read 2 New Done Done true
7215 Acked 8214 Flush 0 New Done Done true
7216 Acked 8215 Write 10 New Done Done true
7217 Acked 8216 Write 6 New Done Done true
7218 Acked 8217 Read 1 New Done Done true
7219 Acked 8218 Read 4 New Done Done true
7220 Acked 8219 Read 9 New Done Done true
7221 Acked 8220 Flush 0 New Done Done true
7222 Acked 8221 Read 1 New Done Done true
7223 Acked 8222 Read 7 New Done Done true
7224 Acked 8223 Read 9 New Done Done true
7225 Acked 8224 Read 2 New Done Done true
7226 Acked 8225 Read 5 New Done Done true
7227 Acked 8226 Read 9 New Done Done true
7228 Acked 8227 Read 7 New Done Done true
7229 Acked 8228 Read 4 New Done Done true
7230 Acked 8229 Write 1 New Done Done true
7231 Acked 8230 Read 10 New Done Done true
7232 Acked 8231 Read 4 New Done Done true
7233 Acked 8232 Write 6 New Done Done true
7234 Acked 8233 Flush 0 New Done Done true
STATES DS:0 DS:1 DS:2 TOTAL
New 302 0 0 302
Sent 0 0 0 0
Done 0 302 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 7931 8233 8233
Downstairs last five completed: 7931 7930 7929 7928 7927
Upstairs last five completed: 7234 7233 7232 7231 7230
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:01.62326045Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:01.623315718Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:01.623340077Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:01.623360886Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:01.623377356Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 7931","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:01.623395245Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:01.623505371Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:02.625105458Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:02.625163916Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:02.625185675Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:02.625205794Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:02.625221514Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 7931","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:02.625238833Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:02.625403996Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:03.627168538Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:03.627218146Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:03.627239225Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:03.627259345Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:03.627275064Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 7931","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:03.627292343Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:03.627409759Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:04.628979778Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:04.629029406Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:04.629050145Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:04.629070574Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:04.629086204Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 7931","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:04.629105413Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:04.629261527Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
6933 Acked 7932 Write 3 New Done Done true
6934 Acked 7933 Write 10 New Done Done true
6935 Acked 7934 Read 9 New Done Done true
6936 Acked 7935 Read 7 New Done Done true
6937 Acked 7936 Flush 0 New Done Done true
6938 Acked 7937 Read 6 New Done Done true
6939 Acked 7938 Read 2 New Done Done true
6940 Acked 7939 Write 2 New Done Done true
6941 Acked 7940 Read 10 New Done Done true
6942 Acked 7941 Write 6 New Done Done true
6943 Acked 7942 Write 2 New Done Done true
6944 Acked 7943 Read 3 New Done Done true
6945 Acked 7944 Read 5 New Done Done true
6946 Acked 7945 Write 1 New Done Done true
6947 Acked 7946 Flush 0 New Done Done true
6948 Acked 7947 Write 8 New Done Done true
6949 Acked 7948 Read 3 New Done Done true
6950 Acked 7949 Flush 0 New Done Done true
6951 Acked 7950 Read 7 New Done Done true
6952 Acked 7951 Read 7 New Done Done true
6953 Acked 7952 Read 2 New Done Done true
6954 Acked 7953 Read 1 New Done Done true
6955 Acked 7954 Write 1 New Done Done true
6956 Acked 7955 Write 3 New Done Done true
6957 Acked 7956 Write 4 New Done Done true
6958 Acked 7957 Write 9 New Done Done true
6959 Acked 7958 Write 6 New Done Done true
6960 Acked 7959 Read 3 New Done Done true
6961 Acked 7960 Write 3 New Done Done true
6962 Acked 7961 Write 3 New Done Done true
6963 Acked 7962 Read 4 New Done Done true
6964 Acked 7963 Read 3 New Done Done true
6965 Acked 7964 Read 5 New Done Done true
6966 Acked 7965 Write 10 New Done Done true
6967 Acked 7966 Read 10 New Done Done true
6968 Acked 7967 Read 5 New Done Done true
6969 Acked 7968 Read 4 New Done Done true
6970 Acked 7969 Write 8 New Done Done true
6971 Acked 7970 Read 2 New Done Done true
6972 Acked 7971 Write 9 New Done Done true
6973 Acked 7972 Flush 0 New Done Done true
6974 Acked 7973 Read 6 New Done Done true
6975 Acked 7974 Read 7 New Done Done true
6976 Acked 7975 Read 8 New Done Done true
6977 Acked 7976 Write 3 New Done Done true
6978 Acked 7977 Write 7 New Done Done true
6979 Acked 7978 Read 10 New Done Done true
6980 Acked 7979 Read 5 New Done Done true
6981 Acked 7980 Read 1 New Done Done true
6982 Acked 7981 Read 4 New Done Done true
6983 Acked 7982 Read 9 New Done Done true
6984 Acked 7983 Read 4 New Done Done true
6985 Acked 7984 Write 1 New Done Done true
6986 Acked 7985 Write 4 New Done Done true
6987 Acked 7986 Read 3 New Done Done true
6988 Acked 7987 Read 5 New Done Done true
6989 Acked 7988 Write 7 New Done Done true
6990 Acked 7989 Write 7 New Done Done true
6991 Acked 7990 Write 9 New Done Done true
6992 Acked 7991 Read 3 New Done Done true
6993 Acked 7992 Write 4 New Done Done true
6994 Acked 7993 Read 10 New Done Done true
6995 Acked 7994 Read 2 New Done Done true
6996 Acked 7995 Write 7 New Done Done true
6997 Acked 7996 Write 3 New Done Done true
6998 Acked 7997 Write 8 New Done Done true
6999 Acked 7998 Read 10 New Done Done true
7000 Acked 7999 Flush 0 New Done Done true
7001 Acked 8000 Read 10 New Done Done true
7002 Acked 8001 Write 4 New Done Done true
7003 Acked 8002 Read 9 New Done Done true
7004 Acked 8003 Write 5 New Done Done true
7005 Acked 8004 Write 4 New Done Done true
7006 Acked 8005 Read 7 New Done Done true
7007 Acked 8006 Flush 0 New Done Done true
7008 Acked 8007 Write 5 New Done Done true
7009 Acked 8008 Write 8 New Done Done true
7010 Acked 8009 Read 10 New Done Done true
7011 Acked 8010 Read 9 New Done Done true
7012 Acked 8011 Read 8 New Done Done true
7013 Acked 8012 Read 9 New Done Done true
7014 Acked 8013 Read 4 New Done Done true
7015 Acked 8014 Write 3 New Done Done true
7016 Acked 8015 Write 7 New Done Done true
7017 Acked 8016 Flush 0 New Done Done true
7018 Acked 8017 Write 3 New Done Done true
7019 Acked 8018 Read 7 New Done Done true
7020 Acked 8019 Write 1 New Done Done true
7021 Acked 8020 Read 4 New Done Done true
7022 Acked 8021 Read 3 New Done Done true
7023 Acked 8022 Read 2 New Done Done true
7024 Acked 8023 Write 5 New Done Done true
7025 Acked 8024 Flush 0 New Done Done true
7026 Acked 8025 Read 7 New Done Done true
7027 Acked 8026 Write 2 New Done Done true
7028 Acked 8027 Write 1 New Done Done true
7029 Acked 8028 Read 9 New Done Done true
7030 Acked 8029 Write 10 New Done Done true
7031 Acked 8030 Read 6 New Done Done true
7032 Acked 8031 Write 3 New Done Done true
7033 Acked 8032 Read 8 New Done Done true
7034 Acked 8033 Flush 0 New Done Done true
7035 Acked 8034 Write 9 New Done Done true
7036 Acked 8035 Write 8 New Done Done true
7037 Acked 8036 Write 4 New Done Done true
7038 Acked 8037 Write 3 New Done Done true
7039 Acked 8038 Write 6 New Done Done true
7040 Acked 8039 Read 4 New Done Done true
7041 Acked 8040 Read 2 New Done Done true
7042 Acked 8041 Read 9 New Done Done true
7043 Acked 8042 Read 5 New Done Done true
7044 Acked 8043 Write 10 New Done Done true
7045 Acked 8044 Write 8 New Done Done true
7046 Acked 8045 Read 4 New Done Done true
7047 Acked 8046 Flush 0 New Done Done true
7048 Acked 8047 Read 4 New Done Done true
7049 Acked 8048 Write 2 New Done Done true
7050 Acked 8049 Write 5 New Done Done true
7051 Acked 8050 Write 10 New Done Done true
7052 Acked 8051 Read 6 New Done Done true
7053 Acked 8052 Read 8 New Done Done true
7054 Acked 8053 Read 8 New Done Done true
7055 Acked 8054 Write 5 New Done Done true
7056 Acked 8055 Write 2 New Done Done true
7057 Acked 8056 Read 5 New Done Done true
7058 Acked 8057 Read 2 New Done Done true
7059 Acked 8058 Read 9 New Done Done true
7060 Acked 8059 Write 3 New Done Done true
7061 Acked 8060 Write 7 New Done Done true
7062 Acked 8061 Write 3 New Done Done true
7063 Acked 8062 Write 1 New Done Done true
7064 Acked 8063 Write 6 New Done Done true
7065 Acked 8064 Write 10 New Done Done true
7066 Acked 8065 Write 7 New Done Done true
7067 Acked 8066 Read 2 New Done Done true
7068 Acked 8067 Write 10 New Done Done true
7069 Acked 8068 Flush 0 New Done Done true
7070 Acked 8069 Read 6 New Done Done true
7071 Acked 8070 Read 2 New Done Done true
7072 Acked 8071 Read 3 New Done Done true
7073 Acked 8072 Write 5 New Done Done true
7074 Acked 8073 Write 3 New Done Done true
7075 Acked 8074 Flush 0 New Done Done true
7076 Acked 8075 Read 6 New Done Done true
7077 Acked 8076 Write 5 New Done Done true
7078 Acked 8077 Read 9 New Done Done true
7079 Acked 8078 Write 1 New Done Done true
7080 Acked 8079 Read 8 New Done Done true
7081 Acked 8080 Write 4 New Done Done true
7082 Acked 8081 Read 2 New Done Done true
7083 Acked 8082 Flush 0 New Done Done true
7084 Acked 8083 Write 9 New Done Done true
7085 Acked 8084 Read 1 New Done Done true
7086 Acked 8085 Read 7 New Done Done true
7087 Acked 8086 Read 4 New Done Done true
7088 Acked 8087 Write 1 New Done Done true
7089 Acked 8088 Write 6 New Done Done true
7090 Acked 8089 Write 1 New Done Done true
7091 Acked 8090 Read 5 New Done Done true
7092 Acked 8091 Read 10 New Done Done true
7093 Acked 8092 Write 4 New Done Done true
7094 Acked 8093 Read 9 New Done Done true
7095 Acked 8094 Write 10 New Done Done true
7096 Acked 8095 Write 10 New Done Done true
7097 Acked 8096 Read 9 New Done Done true
7098 Acked 8097 Read 9 New Done Done true
7099 Acked 8098 Write 10 New Done Done true
7100 Acked 8099 Read 3 New Done Done true
7101 Acked 8100 Flush 0 New Done Done true
7102 Acked 8101 Write 2 New Done Done true
7103 Acked 8102 Read 5 New Done Done true
7104 Acked 8103 Read 2 New Done Done true
7105 Acked 8104 Write 10 New Done Done true
7106 Acked 8105 Read 6 New Done Done true
7107 Acked 8106 Write 8 New Done Done true
7108 Acked 8107 Read 2 New Done Done true
7109 Acked 8108 Write 1 New Done Done true
7110 Acked 8109 Read 9 New Done Done true
7111 Acked 8110 Read 8 New Done Done true
7112 Acked 8111 Read 4 New Done Done true
7113 Acked 8112 Read 4 New Done Done true
7114 Acked 8113 Write 6 New Done Done true
7115 Acked 8114 Read 2 New Done Done true
7116 Acked 8115 Write 1 New Done Done true
7117 Acked 8116 Read 5 New Done Done true
7118 Acked 8117 Read 5 New Done Done true
7119 Acked 8118 Read 7 New Done Done true
7120 Acked 8119 Read 9 New Done Done true
7121 Acked 8120 Write 5 New Done Done true
7122 Acked 8121 Write 3 New Done Done true
7123 Acked 8122 Write 10 New Done Done true
7124 Acked 8123 Write 10 New Done Done true
7125 Acked 8124 Write 9 New Done Done true
7126 Acked 8125 Read 4 New Done Done true
7127 Acked 8126 Write 9 New Done Done true
7128 Acked 8127 Write 1 New Done Done true
7129 Acked 8128 Read 1 New Done Done true
7130 Acked 8129 Read 4 New Done Done true
7131 Acked 8130 Read 6 New Done Done true
7132 Acked 8131 Read 1 New Done Done true
7133 Acked 8132 Write 3 New Done Done true
7134 Acked 8133 Write 10 New Done Done true
7135 Acked 8134 Read 7 New Done Done true
7136 Acked 8135 Write 10 New Done Done true
7137 Acked 8136 Read 4 New Done Done true
7138 Acked 8137 Write 10 New Done Done true
7139 Acked 8138 Write 8 New Done Done true
7140 Acked 8139 Write 2 New Done Done true
7141 Acked 8140 Flush 0 New Done Done true
7142 Acked 8141 Write 5 New Done Done true
7143 Acked 8142 Read 8 New Done Done true
7144 Acked 8143 Read 10 New Done Done true
7145 Acked 8144 Read 9 New Done Done true
7146 Acked 8145 Read 6 New Done Done true
7147 Acked 8146 Write 1 New Done Done true
7148 Acked 8147 Read 4 New Done Done true
7149 Acked 8148 Flush 0 New Done Done true
7150 Acked 8149 Read 1 New Done Done true
7151 Acked 8150 Write 6 New Done Done true
7152 Acked 8151 Read 4 New Done Done true
7153 Acked 8152 Read 5 New Done Done true
7154 Acked 8153 Read 10 New Done Done true
7155 Acked 8154 Write 3 New Done Done true
7156 Acked 8155 Write 10 New Done Done true
7157 Acked 8156 Read 3 New Done Done true
7158 Acked 8157 Flush 0 New Done Done true
7159 Acked 8158 Read 2 New Done Done true
7160 Acked 8159 Read 4 New Done Done true
7161 Acked 8160 Write 1 New Done Done true
7162 Acked 8161 Write 7 New Done Done true
7163 Acked 8162 Read 3 New Done Done true
7164 Acked 8163 Read 2 New Done Done true
7165 Acked 8164 Write 9 New Done Done true
7166 Acked 8165 Flush 0 New Done Done true
7167 Acked 8166 Read 9 New Done Done true
7168 Acked 8167 Write 2 New Done Done true
7169 Acked 8168 Flush 0 New Done Done true
7170 Acked 8169 Write 5 New Done Done true
7171 Acked 8170 Read 5 New Done Done true
7172 Acked 8171 Read 4 New Done Done true
7173 Acked 8172 Write 9 New Done Done true
7174 Acked 8173 Read 9 New Done Done true
7175 Acked 8174 Write 4 New Done Done true
7176 Acked 8175 Write 2 New Done Done true
7177 Acked 8176 Read 9 New Done Done true
7178 Acked 8177 Write 3 New Done Done true
7179 Acked 8178 Write 9 New Done Done true
7180 Acked 8179 Read 7 New Done Done true
7181 Acked 8180 Read 1 New Done Done true
7182 Acked 8181 Flush 0 New Done Done true
7183 Acked 8182 Read 10 New Done Done true
7184 Acked 8183 Read 5 New Done Done true
7185 Acked 8184 Read 9 New Done Done true
7186 Acked 8185 Flush 0 New Done Done true
7187 Acked 8186 Flush 0 New Done Done true
7188 Acked 8187 Read 1 New Done Done true
7189 Acked 8188 Write 4 New Done Done true
7190 Acked 8189 Write 4 New Done Done true
7191 Acked 8190 Write 5 New Done Done true
7192 Acked 8191 Write 10 New Done Done true
7193 Acked 8192 Write 5 New Done Done true
7194 Acked 8193 Write 2 New Done Done true
7195 Acked 8194 Write 1 New Done Done true
7196 Acked 8195 Read 7 New Done Done true
7197 Acked 8196 Write 9 New Done Done true
7198 Acked 8197 Write 8 New Done Done true
7199 Acked 8198 Write 1 New Done Done true
7200 Acked 8199 Write 2 New Done Done true
7201 Acked 8200 Write 8 New Done Done true
7202 Acked 8201 Flush 0 New Done Done true
7203 Acked 8202 Read 3 New Done Done true
7204 Acked 8203 Write 6 New Done Done true
7205 Acked 8204 Write 9 New Done Done true
7206 Acked 8205 Read 7 New Done Done true
7207 Acked 8206 Write 7 New Done Done true
7208 Acked 8207 Write 1 New Done Done true
7209 Acked 8208 Write 1 New Done Done true
7210 Acked 8209 Read 5 New Done Done true
7211 Acked 8210 Write 10 New Done Done true
7212 Acked 8211 Write 10 New Done Done true
7213 Acked 8212 Read 3 New Done Done true
7214 Acked 8213 Read 2 New Done Done true
7215 Acked 8214 Flush 0 New Done Done true
7216 Acked 8215 Write 10 New Done Done true
7217 Acked 8216 Write 6 New Done Done true
7218 Acked 8217 Read 1 New Done Done true
7219 Acked 8218 Read 4 New Done Done true
7220 Acked 8219 Read 9 New Done Done true
7221 Acked 8220 Flush 0 New Done Done true
7222 Acked 8221 Read 1 New Done Done true
7223 Acked 8222 Read 7 New Done Done true
7224 Acked 8223 Read 9 New Done Done true
7225 Acked 8224 Read 2 New Done Done true
7226 Acked 8225 Read 5 New Done Done true
7227 Acked 8226 Read 9 New Done Done true
7228 Acked 8227 Read 7 New Done Done true
7229 Acked 8228 Read 4 New Done Done true
7230 Acked 8229 Write 1 New Done Done true
7231 Acked 8230 Read 10 New Done Done true
7232 Acked 8231 Read 4 New Done Done true
7233 Acked 8232 Write 6 New Done Done true
7234 Acked 8233 Flush 0 New Done Done true
STATES DS:0 DS:1 DS:2 TOTAL
New 302 0 0 302
Sent 0 0 0 0
Done 0 302 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 7931 8233 8233
Downstairs last five completed: 7931 7930 7929 7928 7927
Upstairs last five completed: 7234 7233 7232 7231 7230
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:05.629726628Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:05.629778916Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:05.629800255Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:05.629824904Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:05.629841164Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 7931","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:05.629858613Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:05.629977188Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:06.63092675Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:06.630982218Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:06.631003527Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:06.631024307Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:06.631040146Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 7931","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:06.631057475Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:06.631164821Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:07.631846053Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:07.631901671Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:07.63192338Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:07.631950019Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:07.631966149Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 7931","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:07.631983958Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:07.632139822Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1188)
[8810][0] reports Running
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:08.633050835Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:08.633134082Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:08.633162211Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:08.63318017Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:08.63319372Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 302 jobs since flush 7931","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:08.633210509Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:08.633348634Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
6933 Acked 7932 Write 3 New Done Done true
6934 Acked 7933 Write 10 New Done Done true
6935 Acked 7934 Read 9 New Done Done true
6936 Acked 7935 Read 7 New Done Done true
6937 Acked 7936 Flush 0 New Done Done true
6938 Acked 7937 Read 6 New Done Done true
6939 Acked 7938 Read 2 New Done Done true
6940 Acked 7939 Write 2 New Done Done true
6941 Acked 7940 Read 10 New Done Done true
6942 Acked 7941 Write 6 New Done Done true
6943 Acked 7942 Write 2 New Done Done true
6944 Acked 7943 Read 3 New Done Done true
6945 Acked 7944 Read 5 New Done Done true
6946 Acked 7945 Write 1 New Done Done true
6947 Acked 7946 Flush 0 New Done Done true
6948 Acked 7947 Write 8 New Done Done true
6949 Acked 7948 Read 3 New Done Done true
6950 Acked 7949 Flush 0 New Done Done true
6951 Acked 7950 Read 7 New Done Done true
6952 Acked 7951 Read 7 New Done Done true
6953 Acked 7952 Read 2 New Done Done true
6954 Acked 7953 Read 1 New Done Done true
6955 Acked 7954 Write 1 New Done Done true
6956 Acked 7955 Write 3 New Done Done true
6957 Acked 7956 Write 4 New Done Done true
6958 Acked 7957 Write 9 New Done Done true
6959 Acked 7958 Write 6 New Done Done true
6960 Acked 7959 Read 3 New Done Done true
6961 Acked 7960 Write 3 New Done Done true
6962 Acked 7961 Write 3 New Done Done true
6963 Acked 7962 Read 4 New Done Done true
6964 Acked 7963 Read 3 New Done Done true
6965 Acked 7964 Read 5 New Done Done true
6966 Acked 7965 Write 10 New Done Done true
6967 Acked 7966 Read 10 New Done Done true
6968 Acked 7967 Read 5 New Done Done true
6969 Acked 7968 Read 4 New Done Done true
6970 Acked 7969 Write 8 New Done Done true
6971 Acked 7970 Read 2 New Done Done true
6972 Acked 7971 Write 9 New Done Done true
6973 Acked 7972 Flush 0 New Done Done true
6974 Acked 7973 Read 6 New Done Done true
6975 Acked 7974 Read 7 New Done Done true
6976 Acked 7975 Read 8 New Done Done true
6977 Acked 7976 Write 3 New Done Done true
6978 Acked 7977 Write 7 New Done Done true
6979 Acked 7978 Read 10 New Done Done true
6980 Acked 7979 Read 5 New Done Done true
6981 Acked 7980 Read 1 New Done Done true
6982 Acked 7981 Read 4 New Done Done true
6983 Acked 7982 Read 9 New Done Done true
6984 Acked 7983 Read 4 New Done Done true
6985 Acked 7984 Write 1 New Done Done true
6986 Acked 7985 Write 4 New Done Done true
6987 Acked 7986 Read 3 New Done Done true
6988 Acked 7987 Read 5 New Done Done true
6989 Acked 7988 Write 7 New Done Done true
6990 Acked 7989 Write 7 New Done Done true
6991 Acked 7990 Write 9 New Done Done true
6992 Acked 7991 Read 3 New Done Done true
6993 Acked 7992 Write 4 New Done Done true
6994 Acked 7993 Read 10 New Done Done true
6995 Acked 7994 Read 2 New Done Done true
6996 Acked 7995 Write 7 New Done Done true
6997 Acked 7996 Write 3 New Done Done true
6998 Acked 7997 Write 8 New Done Done true
6999 Acked 7998 Read 10 New Done Done true
7000 Acked 7999 Flush 0 New Done Done true
7001 Acked 8000 Read 10 New Done Done true
7002 Acked 8001 Write 4 New Done Done true
7003 Acked 8002 Read 9 New Done Done true
7004 Acked 8003 Write 5 New Done Done true
7005 Acked 8004 Write 4 New Done Done true
7006 Acked 8005 Read 7 New Done Done true
7007 Acked 8006 Flush 0 New Done Done true
7008 Acked 8007 Write 5 New Done Done true
7009 Acked 8008 Write 8 New Done Done true
7010 Acked 8009 Read 10 New Done Done true
7011 Acked 8010 Read 9 New Done Done true
7012 Acked 8011 Read 8 New Done Done true
7013 Acked 8012 Read 9 New Done Done true
7014 Acked 8013 Read 4 New Done Done true
7015 Acked 8014 Write 3 New Done Done true
7016 Acked 8015 Write 7 New Done Done true
7017 Acked 8016 Flush 0 New Done Done true
7018 Acked 8017 Write 3 New Done Done true
7019 Acked 8018 Read 7 New Done Done true
7020 Acked 8019 Write 1 New Done Done true
7021 Acked 8020 Read 4 New Done Done true
7022 Acked 8021 Read 3 New Done Done true
7023 Acked 8022 Read 2 New Done Done true
7024 Acked 8023 Write 5 New Done Done true
7025 Acked 8024 Flush 0 New Done Done true
7026 Acked 8025 Read 7 New Done Done true
7027 Acked 8026 Write 2 New Done Done true
7028 Acked 8027 Write 1 New Done Done true
7029 Acked 8028 Read 9 New Done Done true
7030 Acked 8029 Write 10 New Done Done true
7031 Acked 8030 Read 6 New Done Done true
7032 Acked 8031 Write 3 New Done Done true
7033 Acked 8032 Read 8 New Done Done true
7034 Acked 8033 Flush 0 New Done Done true
7035 Acked 8034 Write 9 New Done Done true
7036 Acked 8035 Write 8 New Done Done true
7037 Acked 8036 Write 4 New Done Done true
7038 Acked 8037 Write 3 New Done Done true
7039 Acked 8038 Write 6 New Done Done true
7040 Acked 8039 Read 4 New Done Done true
7041 Acked 8040 Read 2 New Done Done true
7042 Acked 8041 Read 9 New Done Done true
7043 Acked 8042 Read 5 New Done Done true
7044 Acked 8043 Write 10 New Done Done true
7045 Acked 8044 Write 8 New Done Done true
7046 Acked 8045 Read 4 New Done Done true
7047 Acked 8046 Flush 0 New Done Done true
7048 Acked 8047 Read 4 New Done Done true
7049 Acked 8048 Write 2 New Done Done true
7050 Acked 8049 Write 5 New Done Done true
7051 Acked 8050 Write 10 New Done Done true
7052 Acked 8051 Read 6 New Done Done true
7053 Acked 8052 Read 8 New Done Done true
7054 Acked 8053 Read 8 New Done Done true
7055 Acked 8054 Write 5 New Done Done true
7056 Acked 8055 Write 2 New Done Done true
7057 Acked 8056 Read 5 New Done Done true
7058 Acked 8057 Read 2 New Done Done true
7059 Acked 8058 Read 9 New Done Done true
7060 Acked 8059 Write 3 New Done Done true
7061 Acked 8060 Write 7 New Done Done true
7062 Acked 8061 Write 3 New Done Done true
7063 Acked 8062 Write 1 New Done Done true
7064 Acked 8063 Write 6 New Done Done true
7065 Acked 8064 Write 10 New Done Done true
7066 Acked 8065 Write 7 New Done Done true
7067 Acked 8066 Read 2 New Done Done true
7068 Acked 8067 Write 10 New Done Done true
7069 Acked 8068 Flush 0 New Done Done true
7070 Acked 8069 Read 6 New Done Done true
7071 Acked 8070 Read 2 New Done Done true
7072 Acked 8071 Read 3 New Done Done true
7073 Acked 8072 Write 5 New Done Done true
7074 Acked 8073 Write 3 New Done Done true
7075 Acked 8074 Flush 0 New Done Done true
7076 Acked 8075 Read 6 New Done Done true
7077 Acked 8076 Write 5 New Done Done true
7078 Acked 8077 Read 9 New Done Done true
7079 Acked 8078 Write 1 New Done Done true
7080 Acked 8079 Read 8 New Done Done true
7081 Acked 8080 Write 4 New Done Done true
7082 Acked 8081 Read 2 New Done Done true
7083 Acked 8082 Flush 0 New Done Done true
7084 Acked 8083 Write 9 New Done Done true
7085 Acked 8084 Read 1 New Done Done true
7086 Acked 8085 Read 7 New Done Done true
7087 Acked 8086 Read 4 New Done Done true
7088 Acked 8087 Write 1 New Done Done true
7089 Acked 8088 Write 6 New Done Done true
7090 Acked 8089 Write 1 New Done Done true
7091 Acked 8090 Read 5 New Done Done true
7092 Acked 8091 Read 10 New Done Done true
7093 Acked 8092 Write 4 New Done Done true
7094 Acked 8093 Read 9 New Done Done true
7095 Acked 8094 Write 10 New Done Done true
7096 Acked 8095 Write 10 New Done Done true
7097 Acked 8096 Read 9 New Done Done true
7098 Acked 8097 Read 9 New Done Done true
7099 Acked 8098 Write 10 New Done Done true
7100 Acked 8099 Read 3 New Done Done true
7101 Acked 8100 Flush 0 New Done Done true
7102 Acked 8101 Write 2 New Done Done true
7103 Acked 8102 Read 5 New Done Done true
7104 Acked 8103 Read 2 New Done Done true
7105 Acked 8104 Write 10 New Done Done true
7106 Acked 8105 Read 6 New Done Done true
7107 Acked 8106 Write 8 New Done Done true
7108 Acked 8107 Read 2 New Done Done true
7109 Acked 8108 Write 1 New Done Done true
7110 Acked 8109 Read 9 New Done Done true
7111 Acked 8110 Read 8 New Done Done true
7112 Acked 8111 Read 4 New Done Done true
7113 Acked 8112 Read 4 New Done Done true
7114 Acked 8113 Write 6 New Done Done true
7115 Acked 8114 Read 2 New Done Done true
7116 Acked 8115 Write 1 New Done Done true
7117 Acked 8116 Read 5 New Done Done true
7118 Acked 8117 Read 5 New Done Done true
7119 Acked 8118 Read 7 New Done Done true
7120 Acked 8119 Read 9 New Done Done true
7121 Acked 8120 Write 5 New Done Done true
7122 Acked 8121 Write 3 New Done Done true
7123 Acked 8122 Write 10 New Done Done true
7124 Acked 8123 Write 10 New Done Done true
7125 Acked 8124 Write 9 New Done Done true
7126 Acked 8125 Read 4 New Done Done true
7127 Acked 8126 Write 9 New Done Done true
7128 Acked 8127 Write 1 New Done Done true
7129 Acked 8128 Read 1 New Done Done true
7130 Acked 8129 Read 4 New Done Done true
7131 Acked 8130 Read 6 New Done Done true
7132 Acked 8131 Read 1 New Done Done true
7133 Acked 8132 Write 3 New Done Done true
7134 Acked 8133 Write 10 New Done Done true
7135 Acked 8134 Read 7 New Done Done true
7136 Acked 8135 Write 10 New Done Done true
7137 Acked 8136 Read 4 New Done Done true
7138 Acked 8137 Write 10 New Done Done true
7139 Acked 8138 Write 8 New Done Done true
7140 Acked 8139 Write 2 New Done Done true
7141 Acked 8140 Flush 0 New Done Done true
7142 Acked 8141 Write 5 New Done Done true
7143 Acked 8142 Read 8 New Done Done true
7144 Acked 8143 Read 10 New Done Done true
7145 Acked 8144 Read 9 New Done Done true
7146 Acked 8145 Read 6 New Done Done true
7147 Acked 8146 Write 1 New Done Done true
7148 Acked 8147 Read 4 New Done Done true
7149 Acked 8148 Flush 0 New Done Done true
7150 Acked 8149 Read 1 New Done Done true
7151 Acked 8150 Write 6 New Done Done true
7152 Acked 8151 Read 4 New Done Done true
7153 Acked 8152 Read 5 New Done Done true
7154 Acked 8153 Read 10 New Done Done true
7155 Acked 8154 Write 3 New Done Done true
7156 Acked 8155 Write 10 New Done Done true
7157 Acked 8156 Read 3 New Done Done true
7158 Acked 8157 Flush 0 New Done Done true
7159 Acked 8158 Read 2 New Done Done true
7160 Acked 8159 Read 4 New Done Done true
7161 Acked 8160 Write 1 New Done Done true
7162 Acked 8161 Write 7 New Done Done true
7163 Acked 8162 Read 3 New Done Done true
7164 Acked 8163 Read 2 New Done Done true
7165 Acked 8164 Write 9 New Done Done true
7166 Acked 8165 Flush 0 New Done Done true
7167 Acked 8166 Read 9 New Done Done true
7168 Acked 8167 Write 2 New Done Done true
7169 Acked 8168 Flush 0 New Done Done true
7170 Acked 8169 Write 5 New Done Done true
7171 Acked 8170 Read 5 New Done Done true
7172 Acked 8171 Read 4 New Done Done true
7173 Acked 8172 Write 9 New Done Done true
7174 Acked 8173 Read 9 New Done Done true
7175 Acked 8174 Write 4 New Done Done true
7176 Acked 8175 Write 2 New Done Done true
7177 Acked 8176 Read 9 New Done Done true
7178 Acked 8177 Write 3 New Done Done true
7179 Acked 8178 Write 9 New Done Done true
7180 Acked 8179 Read 7 New Done Done true
7181 Acked 8180 Read 1 New Done Done true
7182 Acked 8181 Flush 0 New Done Done true
7183 Acked 8182 Read 10 New Done Done true
7184 Acked 8183 Read 5 New Done Done true
7185 Acked 8184 Read 9 New Done Done true
7186 Acked 8185 Flush 0 New Done Done true
7187 Acked 8186 Flush 0 New Done Done true
7188 Acked 8187 Read 1 New Done Done true
7189 Acked 8188 Write 4 New Done Done true
7190 Acked 8189 Write 4 New Done Done true
7191 Acked 8190 Write 5 New Done Done true
7192 Acked 8191 Write 10 New Done Done true
7193 Acked 8192 Write 5 New Done Done true
7194 Acked 8193 Write 2 New Done Done true
7195 Acked 8194 Write 1 New Done Done true
7196 Acked 8195 Read 7 New Done Done true
7197 Acked 8196 Write 9 New Done Done true
7198 Acked 8197 Write 8 New Done Done true
7199 Acked 8198 Write 1 New Done Done true
7200 Acked 8199 Write 2 New Done Done true
7201 Acked 8200 Write 8 New Done Done true
7202 Acked 8201 Flush 0 New Done Done true
7203 Acked 8202 Read 3 New Done Done true
7204 Acked 8203 Write 6 New Done Done true
7205 Acked 8204 Write 9 New Done Done true
7206 Acked 8205 Read 7 New Done Done true
7207 Acked 8206 Write 7 New Done Done true
7208 Acked 8207 Write 1 New Done Done true
7209 Acked 8208 Write 1 New Done Done true
7210 Acked 8209 Read 5 New Done Done true
7211 Acked 8210 Write 10 New Done Done true
7212 Acked 8211 Write 10 New Done Done true
7213 Acked 8212 Read 3 New Done Done true
7214 Acked 8213 Read 2 New Done Done true
7215 Acked 8214 Flush 0 New Done Done true
7216 Acked 8215 Write 10 New Done Done true
7217 Acked 8216 Write 6 New Done Done true
7218 Acked 8217 Read 1 New Done Done true
7219 Acked 8218 Read 4 New Done Done true
7220 Acked 8219 Read 9 New Done Done true
7221 Acked 8220 Flush 0 New Done Done true
7222 Acked 8221 Read 1 New Done Done true
7223 Acked 8222 Read 7 New Done Done true
7224 Acked 8223 Read 9 New Done Done true
7225 Acked 8224 Read 2 New Done Done true
7226 Acked 8225 Read 5 New Done Done true
7227 Acked 8226 Read 9 New Done Done true
7228 Acked 8227 Read 7 New Done Done true
7229 Acked 8228 Read 4 New Done Done true
7230 Acked 8229 Write 1 New Done Done true
7231 Acked 8230 Read 10 New Done Done true
7232 Acked 8231 Read 4 New Done Done true
7233 Acked 8232 Write 6 New Done Done true
7234 Acked 8233 Flush 0 New Done Done true
STATES DS:0 DS:1 DS:2 TOTAL
New 302 0 0 302
Sent 0 0 0 0
Done 0 302 302 604
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 7931 8233 8233
Downstairs last five completed: 7931 7930 7929 7928 7927
Upstairs last five completed: 7234 7233 7232 7231 7230
CLIENT: Up:0 ds:302 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:09.633997078Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:09.634076435Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:09.635457112Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:09.635492451Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 7931","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:09.63551237Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 7931","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:09.635691773Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:09.635719772Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:09.635736172Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:09.635750631Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:09.63576585Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 8233 8232 8231 8230 8229
Upstairs last five completed: 7234 7233 7232 7231 7230
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(1)
stop 1
Feb 13 21:44:13.399 INFO request completed, latency_us: 350[8820] Got stop action so:false kr:false
, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 2fe114e4-4c14-4ea3-9748-e42326ef9f16, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Feb 13 21:44:13.399 INFO request completed, latency_us: 144, response_code: 200, uri: /state/cid/1, method: GET, req_id: d6bab2a0-8df3-4872-a636-25eebd82b812, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:13.399756023Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:13.39982567Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:13.399857679Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:13.399882928Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 0 jobs since flush 8233","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:13.399911047Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:13.399938036Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:14.400582709Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:14.400643087Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:14.400662996Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:14.400679975Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:14.400693085Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 0 jobs since flush 8233","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:14.400707784Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:14.400726954Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:15.4018415Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:15.401896018Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:15.401914277Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:15.401931116Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:15.401943856Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 0 jobs since flush 8233","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:15.401957805Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:15.401976615Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:16.403051462Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:16.403145048Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:16.403176237Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:16.403197236Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:16.403213236Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 0 jobs since flush 8233","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:16.403230875Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:16.403248394Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:44:17.401 INFO request completed, latency_us: 203, response_code: 200, uri: /state/cid/1, method: GET, req_id: 4004886a-9b37-42c4-acb7-b8944c29a0c8, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Read block 1848 len 40960
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:17.404253025Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:17.404303433Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:17.404322282Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:17.404338931Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:17.404351631Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"002/300msg":" Write block 4786 len 16384 data: 3[1] client re-new 1 jobs since flush 8233 4" 5, 5"
v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:17.40437834Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:17.404394069Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
003/300 Flush
004/300 Write block 831 len 28672 data: 5 4 3 3 4 4 4
005/300 Read block 3391 len 36864
006/300 Read block 3102 len 16384
007/300 Write block 1065 len 8192 data: 7 4
008/300 Read block 3249 len 36864
009/300 Write block 605 len 4096 data: 7
010/300 Read block 2238 len 36864
011/300 Read block 1973 len 36864
012/300 Read block 3362 len 16384
013/300 Read block 2650 len 28672
014/300 Read block 3896 len 40960
015/300 Write block 2429 len 28672 data: 4 5 5 5 6 5 6
016/300 Read block 2378 len 20480
017/300 Read block 2339 len 28672
018/300 Flush
019/300 Read block 1233 len 36864
020/300 Write block 161 len 12288 data: 6 6 9
021/300 Flush
022/300 Write block 460 len 28672 data: 4 7 8 8 8 8 8
023/300 Read block 3396 len 4096
024/300 Read block 4648 len 40960
025/300 Read block 4308 len 20480
026/300 Read block 2792 len 4096
027/300 Read block 2916 len 20480
028/300 Read block 2038 len 20480
029/300 Read block 719 len 40960
030/300 Write block 464 len 32768 data: 9 9 9 6 6 5 6 5
031/300 Write block 3120 len 24576 data: 7 7 6 6 8 7
032/300 Write block 2903 len 12288 data: 3 4 4
033/300 Flush
034/300 Read block 4733 len 32768
035/300 Read block 3104 len 28672
036/300 Read block 4180 len 12288
037/300 Write block 3839 len 28672 data: 7 6 5 5 4 5 5
038/300 Read block 3348 len 28672
039/300 Read block 4463 len 28672
040/300 Write block 2555 len 28672 data: 7 7 6 6 7 8 8
041/300 Read block 3293 len 28672
042/300 Read block 4319 len 40960
043/300 Read block 4959 len 36864
044/300 Flush
045/300 Read block 2257 len 32768
046/300 Read block 3906 len 28672
047/300 Write block 4209 len 8192 data: 6 4
048/300 Write block 4402 len 36864 data: 6 4 4 4 6 7 6 5 5
049/300 Write block 2692 len 28672 data: 5 5 3 3 2 2 3
050/300 Read block 726 len 8192
051/300 Write block 671 len 8192 data: 5 5
052/300 Read block 3061 len 16384
053/300 Write block 3808 len 40960 data: 6 6 6 7 7 7 7 6 5 7
054/300 Read block 4499 len 16384
055/300 Write block 2122 len 8192 data: 4 4
056/300 Write block 2399 len 36864 data: 3 3 5 7 7 8 9 8 8
057/300 Write block 132 len 8192 data: 9 7
058/300 Read block 197 len 4096
059/300 Read block 4352 len 24576
060/300 Read block 1078 len 32768
061/300 Write block 2911 len 40960 data: 5 5 7 9 10 10 9 7 6 6
062/300 Read block 2077 len 32768
063/300 Write block 14 len 28672 data: 3 3 3 3 4 5 5
064/300 Flush
065/300 Write block 1275 len 8192 data: 6 6
066/300 Write block 2626 len 12288 data: 6 7 7
067/300 Write block 2442 len 20480 data: 11 8 6 7 7
068/300 Read block 157 len 24576
069/300 Write block 2075 len 20480 data: 7 7 7 6 5
070/300 Read block 2228 len 40960
071/300 Read block 1749 len 4096
072/300 Read block 2043 len 24576
073/300 Read block 2424 len 20480
074/300 Write block 334 len 24576 data: 6 6 6 3 3 3
075/300 Read block 4227 len 24576
076/300 Write block 3688 len 4096 data: 10
077/300 Write block 3617 len 8192 data: 6 4
078/300 Write block 4207 len 16384 data: 5 5 7 5
079/300 Write block 57 len 32768 data: 6 8 8 11 13 13 10 10
080/300 Flush
081/300 Read block 4023 len 4096
082/300 Write block 3604 len 20480 data: 6 5 5 5 4
083/300 Flush
084/300 Write block 162 len 8192 data: 7 10
085/300 Write block 647 len 20480 data: 5 5 4 5 5
086/300 Write block 4645 len 16384 data: 4 5 6 5
087/300 Write block 1236 len 20480 data: 4 4 4 3 4
088/300 Write block 4679 len 12288 data: 4 5 8
089/300 Write block 1319 len 24576 data: 4 3 3 2 4 4
090/300 Write block 4928 len 32768 data: 5 5 4 4 5 5 4 4
091/300 Read block 1474 len 32768
092/300 Read block 2437 len 24576
093/300 Read block 4390 len 36864
094/300 Flush
095/300 Write block 1948 len 12288 data: 7 6 5
096/300 Read block 2518 len 32768
097/300 Read block 3490 len 12288
098/300 Read block 2821 len 24576
099/300 Read block 4616 len 24576
100/300 Read block 3278 len 40960
101/300 Read block 1596 len 12288
102/300 Write block 268 len 4096 data: 5
103/300 Read block 1600 len 12288
104/300 Write block 2764 len 20480 data: 3 3 3 2 2
105/300 Write block 1274 len 16384 data: 6 7 7 4
106/300 Read block 2071 len 40960
107/300 Read block 3313 len 16384
108/300 Write block 118 len 40960 data: 4 3 4 7 7 9 9 9 8 9
109/300 Write block 1820 len 16384 data: 9 7 7 6
110/300 Write block 3800 len 12288 data: 5 5 5
111/300 Read block 854 len 24576
112/300 Read block 589 len 32768
113/300 Read block 3292 len 8192
114/300 Write block 4158 len 20480 data: 5 5 5 7 6
115/300 Flush
116/300 Write block 1131 len 40960 data: 6 6 5 4 5 6 4 4 6 5
117/300 Read block 1135 len 32768
118/300 Read block 1797 len 40960
119/300 Write block 3213 len 28672 data: 6 5 5 5 6 6 8
120/300 Write block 4209 len 24576 data: 8 6 4 4 5 5
121/300 Flush
122/300 Read block 1746 len 36864
123/300 Flush
124/300 Read block 2951 len 40960
125/300 Write block 3730 len 36864 data: 4 4 5 5 5 4 6 7 7
126/300 Read block 2296 len 36864
127/300 Write block 2593 len 36864 data: 6 6 8 8 7 7 8 6 5
128/300 Read block 1411 len 16384
129/300 Read block 601 len 4096
130/300 Write block 2533 len 4096 data: 8
131/300 Write block 4443 len 28672 data: 6 7 6 5 5 6 5
132/300 Write block 3725 len 28672 data: 4 4 4 5 4 5 5
133/300 Read block 1059 len 8192
134/300 Write block 1648 len 36864 data: 6 4 3 3 3 4 5 6 7
135/300 Write block 4451 len 32768 data: 7 3 3 3 4 4 4 4
136/300 Write block 3677 len 4096 data: 6
137/300 Read block 3188 len 4096
138/300 Write block 4514 len 16384 data: 5 6 5 5
139/300 Write block 4037 len 16384 data: 3 7 5 6
140/300 Write block 3826 len 20480 data: 8 7 7 6 6
141/300 Read block 3496 len 28672
142/300 Write block 1781 len 16384 data: 3 3 2 2
143/300 Write block 1682 len 36864 data: 3 3 3 3 4 4 5 5 5
144/300 Flush
145/300 Read block 4990 len 4096
146/300 Write block 3294 len 36864 data: 3 4 7 7 7 6 5 4 4
147/300 Read block 3357 len 12288
148/300 Read block 1379 len 16384
149/300 Write block 3394 len 32768 data: 4 3 2 2 2 3 5 5
150/300 Read block 936 len 32768
151/300 Write block 4032 len 4096 data: 2
152/300 Read block 1728 len 20480
153/300 Read block 1414 len 4096
154/300 Write block 3391 len 24576 data: 3 4 5 5 4 3
155/300 Write block 2446 len 12288 data: 8 6 6
156/300 Write block 3077 len 8192 data: 5 5
157/300 Write block 3274 len 8192 data: 4 5
158/300 Read block 3959 len 12288
159/300 Flush
160/300 Flush
161/300 Read block 830 len 20480
162/300 Read block 2215 len 32768
163/300 Write block 1775 len 32768 data: 7 7 7 7 5 5 4 4
164/300 Write block 343 len 4096 data: 3
165/300 Write block 2819 len 28672 data: 4 3 5 5 5 5 6
166/300 Read block 381 len 20480
167/300 Read block 4926 len 4096
168/300 Read block 1903 len 36864
169/300 Write block 667 len 28672 data: 6 6 5 6 6 6 4
170/300 Read block 1935 len 16384
171/300 Write block 2656 len 28672 data: 6 7 7 7 7 6 8
172/300 Write block 1285 len 16384 data: 5 5 4 4
173/300 Read block 3040 len 8192
174/300 Write block 90 len 12288 data: 8 8 8
175/300 Read block 1341 len 8192
176/300 Read block 2035 len 20480
177/300 Read block 2560 len 40960
178/300 Read block 3124 len 32768
179/300 Read block 2626 len 32768
180/300 Write block 1522 len 36864 data: 6 6 7 7 5 5 5 5 3
181/300 Write block 2002 len 32768 data: 7 7 6 6 4 5 4 8
182/300 Write block 2620 len 20480 data: 2 2 2 2 2
183/300 Read block 163 len 4096
184/300 Write block 4630 len 4096 data: 3
185/300 Write block 3073 len 36864 data: 3 3 3 5 6 6 5 5 5
186/300 Read block 368 len 4096
187/300 Write block 2329 len 32768 data: 6 6 6 6 5 6 7 7
188/300 Read block 747 len 16384
189/300 Write block 124 len 28672 data: 10 10 9 10 8 11 11
190/300 Read block 1510 len 32768
191/300 Write block 4094 len 4096 data: 4
192/300 Write block 3948 len 28672 data: 4 4 5 6 6 6 6
193/300 Read block 56 len 28672
194/300 Flush
195/300 Read block 1432 len 32768
196/300 Write block 4872 len 12288 data: 3 4 3
197/300 Write block 4001 len 8192 data: 5 7
198/300 Write block 1994 len 4096 data: 5
199/300 Write block 2042 len 36864 data: 4 6 4 4 5 6 8 8 8
200/300 Write block 2587 len 28672 data: 5 5 5 4 3 3 7
201/300 Read block 4952 len 12288
202/300 Write block 454 len 40960 data: 8 7 7 7 6 5 5 8 9 9
203/300 Flush
204/300 Write block 4509 len 28672 data: 4 5 5 6 8 6 7
205/300 Write block 894 len 8192 data: 7 7
206/300 Write block 199 len 24576 data: 4 4 4 5 4 4
207/300 Read block 894 len 4096
208/300 Write block 3698 len 36864 data: 3 3 3 4 3 3 2 2 2
209/300 Read block 3928 len 24576
210/300 Flush
211/300 Read block 620 len 24576
212/300 Read block 4737 len 24576
213/300 Flush
214/300 Read block 1471 len 16384
215/300 Flush
216/300 Read block 1666 len 4096
217/300 Read block 381 len 16384
218/300 Read block 2770 len 40960
219/300 Write block 3867 len 20480 data: 6 4 5 5 5
220/300 Flush
221/300 Write block 4211 len 4096 data: 5
222/300 Read block 3871 len 28672
223/300 Read block 1928 len 16384
224/300 Read block 3333 len 28672
225/300 Read block 3411 len 8192
226/300 Write block 1981 len 8192 data: 3 3
227/300 Read block 3238 len 8192
228/300 Read block 498 len 8192
229/300 Write block 1142 len 8192 data: 5 5
230/300 Write block 4285 len 36864 data: 7 8 9 8 9 8 8 9 8
231/300 Read block 1780 len 28672
232/300 Read block 2249 len 24576
233/300 Write block 850 len 36864 data: 8 6 5 5 7 8 9 8 9
234/300 Read block 886 len 28672
235/300 Read block 4060 len 12288
236/300 Read block 4826 len 36864
237/300 Write block 208 len 4096 data: 4
238/300 Write block 4165 len 12288 data: 5 6 7
239/300 Write block 2020 len 28672 data: 8 6 5 5 2 3 3
240/300 Write block 820 len 20480 data: 4 5 6 6 6
241/300 Read block 2447 len 40960
242/300 Flush
243/300 Flush
244/300 Read block 3435 len 36864
245/300 Read block 108 len 24576
246/300 Read block 2542 len 28672
247/300 Read block 1925 len 40960
248/300 Write block 4623 len 16384 data: 8 8 7 5
249/300 Write block 2343 len 20480 data: 4 4 3 2 2
250/300 Read block 1707 len 28672
251/300 Read block 113 len 24576
252/300 Read block 486 len 12288
253/300 Write block 3469 len 24576 data: 3 3 3 3 3 2
254/300 Read block 2875 len 24576
255/300 Write block 1954 len 28672 data: 3 3 2 2 4 3 5
256/300 Write block 1805 len 32768 data: 5 5 6 4 5 6 4 4
257/300 Write block 2403 len 12288 data: 8 9 10
258/300 Read block 1833 len 28672
259/300 Read block 1802 len 16384
260/300 Flush
261/300 Read block 2661 len 20480
262/300 Read block 638 len 8192
263/300 Read block 849 len 12288
264/300 Write block 4461 len 24576 data: 5 6 6 6 7 8
265/300 Read block 2212 len 32768
266/300 Read block 3751 len 36864
267/300 Read block 2807 len 20480
268/300 Write block 3903 len 40960 data: 6 6 5 5 5 5 7 6 6 5
269/300 Write block 2206 len 24576 data: 5 5 5 6 6 7
270/300 Write block 2258 len 16384 data: 8 6 6 7
271/300 Write block 141 len 28672 data: 6 5 4 4 3 4 5
272/300 Write block 3736 len 20480 data: 7 8 8 6 8
273/300 Read block 3600 len 12288
274/300 Write block 2608 len 40960 data: 4 4 4 4 4 3 3 3 3 3
275/300 Read block 3068 len 28672
276/300 Write block 1617 len 24576 data: 6 7 7 5 5 4
277/300 Flush
278/300 Write block 2039 len 4096 data: 5
279/300 Read block 4780 len 40960
280/300 Write block 4169 len 40960 data: 7 5 3 3 3 3 3 4 2 2
281/300 Write block 3366 len 28672 data: 5 7 9 11 11 9 11
282/300 Write block 3657 len 24576 data: 7 7 4 3 4 6
283/300 Write block 432 len 40960 data: 6 6 5 6 5 5 3 3 3 5
284/300 Read block 3545 len 36864
285/300 Read block 3349 len 12288
286/300 Read block 4043 len 36864
287/300 Write block 4607 len 32768 data: 5 5 5 5 4 4 5 4
288/300 Write block 1916 len 24576 data: 5 5 5 4 4 4
289/300 Write block 1919 len 24576 data: 5 5 5 5 5 5
290/300 Read block 2463 len 28672
291/300 Read block 4310 len 8192
292/300 Write block 4266 len 28672 data: 4 4 5 4 5 5 5
293/300 Write block 4354 len 40960 data: 4 4 4 5 6 7 7 6 7 7
294/300 Read block 2621 len 36864
295/300 Write block 4950 len 20480 data: 5 4 4 4 4
296/300 Read block 2634 len 24576
297/300 Write block 2617 len 32768 data: 4 3 4 3 3 3 3 3
298/300 Read block 3128 len 28672
299/300 Write block 1638 len 4096 data: 5
300/300 Write block 4167 len 36864 data: 8 6 8 6 4 4 4 4 4
Main task has work to do, go find it
got dsc Start(1)
start 1
Feb 13 21:44:18.140 INFO request completed, latency_us: 208, response_code: 204, uri: /start/cid/1, method: POST, req_id: e1460c6c-5986-48ef-9776-70acb6457346, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 1, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Offline Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
7235 Acked 8234 Read 10 Done New Done true
7236 Acked 8235 Write 4 Done New Done false
7237 Acked 8236 Flush 0 Done New Done false
7238 Acked 8237 Write 7 Done New Done false
7239 Acked 8238 Read 9 Done New Done false
7240 Acked 8239 Read 4 Done New Done false
7241 Acked 8240 Write 2 Done New Done false
7242 Acked 8241 Read 9 Done New Done false
7243 Acked 8242 Write 1 Done New Done false
7244 Acked 8243 Read 9 Done New Done false
7245 Acked 8244 Read 9 Done New Done false
7246 Acked 8245 Read 4 Done New Done false
7247 Acked 8246 Read 7 Done New Done false
7248 Acked 8247 Read 10 Done New Done false
7249 Acked 8248 Write 7 Done New Done false
7250 Acked 8249 Read 5 Done New Done false
7251 Acked 8250 Read 7 Done New Done false
7252 Acked 8251 Flush 0 Done New Done false
7253 Acked 8252 Read 9 Done New Done false
7254 Acked 8253 Write 3 Done New Done false
7255 Acked 8254 Flush 0 Done New Done false
7256 Acked 8255 Write 7 Done New Done false
7257 Acked 8256 Read 1 Done New Done false
7258 Acked 8257 Read 10 Done New Done false
7259 Acked 8258 Read 5 Done New Done false
7260 Acked 8259 Read 1 Done New Done false
7261 Acked 8260 Read 5 Done New Done false
7262 Acked 8261 Read 5 Done New Done false
7263 Acked 8262 Read 10 Done New Done false
7264 Acked 8263 Write 8 Done New Done false
7265 Acked 8264 Write 6 Done New Done false
7266 Acked 8265 Write 3 Done New Done false
7267 Acked 8266 Flush 0 Done New Done false
7268 Acked 8267 Read 8 Done New Done false
7269 Acked 8268 Read 7 Done New Done false
7270 Acked 8269 Read 3 Done New Done false
7271 Acked 8270 Write 7 Done New Done false
7272 Acked 8271 Read 7 Done New Done false
7273 Acked 8272 Read 7 Done New Done false
7274 Acked 8273 Write 7 Done New Done false
7275 Acked 8274 Read 7 Done New Done false
7276 Acked 8275 Read 10 Done New Done false
7277 Acked 8276 Read 9 Done New Done false
7278 Acked 8277 Flush 0 Done New Done false
7279 Acked 8278 Read 8 Done New Done false
7280 Acked 8279 Read 7 Done New Done false
7281 Acked 8280 Write 2 Done New Done false
7282 Acked 8281 Write 9 Done New Done false
7283 Acked 8282 Write 7 Done New Done false
7284 Acked 8283 Read 2 Done New Done false
7285 Acked 8284 Write 2 Done New Done false
7286 Acked 8285 Read 4 Done New Done false
7287 Acked 8286 Write 10 Done New Done false
7288 Acked 8287 Read 4 Done New Done false
7289 Acked 8288 Write 2 Done New Done false
7290 Acked 8289 Write 9 Done New Done false
7291 Acked 8290 Write 2 Done New Done false
7292 Acked 8291 Read 1 Done New Done false
7293 Acked 8292 Read 6 Done New Done false
7294 Acked 8293 Read 8 Done New Done false
7295 Acked 8294 Write 10 Done New Done false
7296 Acked 8295 Read 8 Done New Done false
7297 Acked 8296 Write 7 Done New Done false
7298 Acked 8297 Flush 0 Done New Done false
7299 Acked 8298 Write 2 Done New Done false
7300 Acked 8299 Write 3 Done New Done false
7301 Acked 8300 Write 5 Done New Done false
7302 Acked 8301 Read 6 Done New Done false
7303 Acked 8302 Write 5 Done New Done false
7304 Acked 8303 Read 10 Done New Done false
7305 Acked 8304 Read 1 Done New Done false
7306 Acked 8305 Read 6 Done New Done false
7307 Acked 8306 Read 5 Done New Done false
7308 Acked 8307 Write 6 Done New Done false
7309 Acked 8308 Read 6 Done New Done false
7310 Acked 8309 Write 1 Done New Done false
7311 Acked 8310 Write 2 Done New Done false
7312 Acked 8311 Write 4 Done New Done false
7313 Acked 8312 Write 8 Done New Done false
7314 Acked 8313 Flush 0 Done New Done false
7315 Acked 8314 Read 1 Done New Done false
7316 Acked 8315 Write 5 Done New Done false
7317 Acked 8316 Flush 0 Done New Done false
7318 Acked 8317 Write 2 Done New Done false
7319 Acked 8318 Write 5 Done New Done false
7320 Acked 8319 Write 4 Done New Done false
7321 Acked 8320 Write 5 Done New Done false
7322 Acked 8321 Write 3 Done New Done false
7323 Acked 8322 Write 6 Done New Done false
7324 Acked 8323 Write 8 Done New Done false
7325 Acked 8324 Read 8 Done New Done false
7326 Acked 8325 Read 6 Done New Done false
7327 Acked 8326 Read 9 Done New Done false
7328 Acked 8327 Flush 0 Done New Done false
7329 Acked 8328 Write 3 Done New Done false
7330 Acked 8329 Read 8 Done New Done false
7331 Acked 8330 Read 3 Done New Done false
7332 Acked 8331 Read 6 Done New Done false
7333 Acked 8332 Read 6 Done New Done false
7334 Acked 8333 Read 10 Done New Done false
7335 Acked 8334 Read 3 Done New Done false
7336 Acked 8335 Write 1 Done New Done false
7337 Acked 8336 Read 3 Done New Done false
7338 Acked 8337 Write 5 Done New Done false
7339 Acked 8338 Write 4 Done New Done false
7340 Acked 8339 Read 10 Done New Done false
7341 Acked 8340 Read 4 Done New Done false
7342 Acked 8341 Write 10 Done New Done false
7343 Acked 8342 Write 4 Done New Done false
7344 Acked 8343 Write 3 Done New Done false
7345 Acked 8344 Read 6 Done New Done false
7346 Acked 8345 Read 8 Done New Done false
7347 Acked 8346 Read 2 Done New Done false
7348 Acked 8347 Write 5 Done New Done false
7349 Acked 8348 Flush 0 Done New Done false
7350 Acked 8349 Write 10 Done New Done false
7351 Acked 8350 Read 8 Done New Done false
7352 Acked 8351 Read 10 Done New Done false
7353 Acked 8352 Write 7 Done New Done false
7354 Acked 8353 Write 6 Done New Done false
7355 Acked 8354 Flush 0 Done New Done false
7356 Acked 8355 Read 9 Done New Done false
7357 Acked 8356 Flush 0 Done New Done false
7358 Acked 8357 Read 10 Done New Done false
7359 Acked 8358 Write 9 Done New Done false
7360 Acked 8359 Read 9 Done New Done false
7361 Acked 8360 Write 9 Done New Done false
7362 Acked 8361 Read 4 Done New Done false
7363 Acked 8362 Read 1 Done New Done false
7364 Acked 8363 Write 1 Done New Done false
7365 Acked 8364 Write 7 Done New Done false
7366 Acked 8365 Write 7 Done New Done false
7367 Acked 8366 Read 2 Done New Done false
7368 Acked 8367 Write 9 Done New Done false
7369 Acked 8368 Write 8 Done New Done false
7370 Acked 8369 Write 1 Done New Done false
7371 Acked 8370 Read 1 Done New Done false
7372 Acked 8371 Write 4 Done New Done false
7373 Acked 8372 Write 4 Done New Done false
7374 Acked 8373 Write 5 Done New Done false
7375 Acked 8374 Read 7 Done New Done false
7376 Acked 8375 Write 4 Done New Done false
7377 Acked 8376 Write 9 Done New Done false
7378 Acked 8377 Flush 0 Done New Done false
7379 Acked 8378 Read 1 Done New Done false
7380 Acked 8379 Write 9 Done New Done false
7381 Acked 8380 Read 3 Done New Done false
7382 Acked 8381 Read 4 Done New Done false
7383 Acked 8382 Write 8 Done New Done false
7384 Acked 8383 Read 8 Done New Done false
7385 Acked 8384 Write 1 Done New Done false
7386 Acked 8385 Read 5 Done New Done false
7387 Acked 8386 Read 1 Done New Done false
7388 Acked 8387 Write 6 Done New Done false
7389 Acked 8388 Write 3 Done New Done false
7390 Acked 8389 Write 2 Done New Done false
7391 Acked 8390 Write 2 Done New Done false
7392 Acked 8391 Read 3 Done New Done false
7393 Acked 8392 Flush 0 Done New Done false
7394 Acked 8393 Flush 0 Done New Done false
7395 Acked 8394 Read 5 Done New Done false
7396 Acked 8395 Read 8 Done New Done false
7397 Acked 8396 Write 8 Done New Done false
7398 Acked 8397 Write 1 Done New Done false
7399 Acked 8398 Write 7 Done New Done false
7400 Acked 8399 Read 5 Done New Done false
7401 Acked 8400 Read 1 Done New Done false
7402 Acked 8401 Read 9 Done New Done false
7403 Acked 8402 Write 7 Done New Done false
7404 Acked 8403 Read 4 Done New Done false
7405 Acked 8404 Write 7 Done New Done false
7406 Acked 8405 Write 4 Done New Done false
7407 Acked 8406 Read 2 Done New Done false
7408 Acked 8407 Write 3 Done New Done false
7409 Acked 8408 Read 2 Done New Done false
7410 Acked 8409 Read 5 Done New Done false
7411 Acked 8410 Read 10 Done New Done false
7412 Acked 8411 Read 8 Done New Done false
7413 Acked 8412 Read 8 Done New Done false
7414 Acked 8413 Write 9 Done New Done false
7415 Acked 8414 Write 8 Done New Done false
7416 Acked 8415 Write 5 Done New Done false
7417 Acked 8416 Read 1 Done New Done false
7418 Acked 8417 Write 1 Done New Done false
7419 Acked 8418 Write 9 Done New Done false
7420 Acked 8419 Read 1 Done New Done false
7421 Acked 8420 Write 8 Done New Done false
7422 Acked 8421 Read 4 Done New Done false
7423 Acked 8422 Write 7 Done New Done false
7424 Acked 8423 Read 8 Done New Done false
7425 Acked 8424 Write 1 Done New Done false
7426 Acked 8425 Write 7 Done New Done false
7427 Acked 8426 Read 7 Done New Done false
7428 Acked 8427 Flush 0 Done New Done false
7429 Acked 8428 Read 8 Done New Done false
7430 Acked 8429 Write 3 Done New Done false
7431 Acked 8430 Write 2 Done New Done false
7432 Acked 8431 Write 1 Done New Done false
7433 Acked 8432 Write 9 Done New Done false
7434 Acked 8433 Write 7 Done New Done false
7435 Acked 8434 Read 3 Done New Done false
7436 Acked 8435 Write 10 Done New Done false
7437 Acked 8436 Flush 0 Done New Done false
7438 Acked 8437 Write 7 Done New Done false
7439 Acked 8438 Write 2 Done New Done false
7440 Acked 8439 Write 6 Done New Done false
7441 Acked 8440 Read 1 Done New Done false
7442 Acked 8441 Write 9 Done New Done false
7443 Acked 8442 Read 6 Done New Done false
7444 Acked 8443 Flush 0 Done New Done false
7445 Acked 8444 Read 6 Done New Done false
7446 Acked 8445 Read 6 Done New Done false
7447 Acked 8446 Flush 0 Done New Done false
7448 Acked 8447 Read 4 Done New Done false
7449 Acked 8448 Flush 0 Done New Done false
7450 Acked 8449 Read 1 Done New Done false
7451 Acked 8450 Read 4 Done New Done false
7452 Acked 8451 Read 10 Done New Done false
7453 Acked 8452 Write 5 Done New Done false
7454 Acked 8453 Flush 0 Done New Done false
7455 Acked 8454 Write 1 Done New Done false
7456 Acked 8455 Read 7 Done New Done false
7457 Acked 8456 Read 4 Done New Done false
7458 Acked 8457 Read 7 Done New Done false
7459 Acked 8458 Read 2 Done New Done false
7460 Acked 8459 Write 2 Done New Done false
7461 Acked 8460 Read 2 Done New Done false
7462 Acked 8461 Read 2 Done New Done false
7463 Acked 8462 Write 2 Done New Done false
7464 Acked 8463 Write 9 Done New Done false
7465 Acked 8464 Read 7 Done New Done false
7466 Acked 8465 Read 6 Done New Done false
7467 Acked 8466 Write 9 Done New Done false
7468 Acked 8467 Read 7 Done New Done false
7469 Acked 8468 Read 3 Done New Done false
7470 Acked 8469 Read 9 Done New Done false
7471 Acked 8470 Write 1 Done New Done false
7472 Acked 8471 Write 3 Done New Done false
7473 Acked 8472 Write 7 Done New Done false
7474 Acked 8473 Write 5 Done New Done false
7475 Acked 8474 Read 10 Done New Done false
7476 Acked 8475 Flush 0 Done New Done false
7477 Acked 8476 Flush 0 Done New Done false
7478 Acked 8477 Read 9 Done New Done false
7479 Acked 8478 Read 6 Done New Done false
7480 Acked 8479 Read 7 Done New Done false
7481 Acked 8480 Read 10 Done New Done false
7482 Acked 8481 Write 4 Done New Done false
7483 Acked 8482 Write 5 Done New Done false
7484 Acked 8483 Read 7 Done New Done false
7485 Acked 8484 Read 6 Done New Done false
7486 Acked 8485 Read 3 Done New Done false
7487 Acked 8486 Flush 0 Done New Done false
7488 Acked 8487 Write 6 Done New Done false
7489 Acked 8488 Read 6 Done New Done false
7490 Acked 8489 Write 7 Done New Done false
7491 Acked 8490 Write 8 Done New Done false
7492 Acked 8491 Write 3 Done New Done false
7493 Acked 8492 Read 7 Done New Done false
7494 Acked 8493 Read 4 Done New Done false
7495 Acked 8494 Flush 0 Done New Done false
7496 Acked 8495 Read 5 Done New Done false
7497 Acked 8496 Read 2 Done New Done false
7498 Acked 8497 Read 3 Done New Done false
7499 Acked 8498 Write 6 Done New Done false
7500 Acked 8499 Read 8 Done New Done false
7501 Acked 8500 Read 9 Done New Done false
7502 Acked 8501 Read 5 Done New Done false
7503 Acked 8502 Write 10 Done New Done false
7504 Acked 8503 Write 6 Done New Done false
7505 Acked 8504 Write 4 Done New Done false
7506 Acked 8505 Write 7 Done New Done false
7507 Acked 8506 Write 5 Done New Done false
7508 Acked 8507 Read 3 Done New Done false
7509 Acked 8508 Write 10 Done New Done false
7510 Acked 8509 Read 7 Done New Done false
7511 Acked 8510 Write 6 Done New Done false
7512 Acked 8511 Flush 0 Done New Done false
7513 Acked 8512 Write 1 Done New Done false
7514 Acked 8513 Read 10 Done New Done false
7515 Acked 8514 Write 10 Done New Done false
7516 Acked 8515 Write 7 Done New Done false
7517 Acked 8516 Write 6 Done New Done false
7518 Acked 8517 Write 10 Done New Done false
7519 Acked 8518 Read 9 Done New Done false
7520 Acked 8519 Read 3 Done New Done false
7521 Acked 8520 Read 9 Done New Done false
7522 Acked 8521 Write 8 Done New Done false
7523 Acked 8522 Write 6 Done New Done false
7524 Acked 8523 Write 6 Done New Done false
7525 Acked 8524 Read 7 Done New Done false
7526 Acked 8525 Read 2 Done New Done false
7527 Acked 8526 Write 7 Done New Done false
7528 Acked 8527 Write 10 Done New Done false
7529 Acked 8528 Read 9 Done New Done false
7530 Acked 8529 Write 5 Done New Done false
7531 Acked 8530 Read 6 Done New Done false
7532 Acked 8531 Write 8 Done New Done false
7533 Acked 8532 Read 7 Done New Done false
7534 Acked 8533 Write 1 Done New Done false
7535 Acked 8534 Write 9 Done New Done false
STATES DS:0 DS:1 DS:2 TOTAL
New 0 301 0 301
Sent 0 0 0 0
Done 301 0 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 8511 8233 8511
Downstairs last five completed: 8233 8232 8231 8230 8229
Upstairs last five completed: 7535 7534 7533 7532 7531
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:18.405879991Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:18.405931439Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:18.405950428Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:18.405968477Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:18.405981247Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 301 jobs since flush 8233","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:18.405997496Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:18.406141221Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:19.406708377Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:19.406769975Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:19.406801044Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:19.406822913Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:19.406839742Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] client re-new 302 jobs since flush 8233","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:19.406857851Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:19.406966497Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8820] got start action
[8820] I am going to restart
Starting downstairs at port 8820
Make output file at "/tmp/dsc/downstairs-8820.txt"
[8820][1] reports Starting
Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1189)
[8820][1] reports Running
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:20.408077194Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:20.408151891Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:20.409522928Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008820 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:20.409555297Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 8233","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:20.409574067Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 8233","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:20.409726121Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:20.409762519Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:20.409790548Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:20.409818507Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[1] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:20.409836266Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"1","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 8535 8534 8533 8532 8531
Upstairs last five completed: 7536 7535 7534 7533 7532
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(2)
stop 2
Feb 13 21:44:22.143 INFO request completed[8830] Got stop action so:false kr:false
, latency_us: 357, response_code: 204, uri: /stop/cid/2, method: POST, req_id: e6c25b4a-4887-4977-8f8c-db03c3809b12, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Feb 13 21:44:22.144 INFO {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:22.144409495Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
request completed, latency_us: 123, response_code: 200, uri: /state/cid/2, method: GET, req_id: 48cf2f1d-155d-4676-ad7f-6cba5020cfd6{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:22.144491382Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:22.144519491Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
1:9998
{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:22.144580488Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 8535","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:22.144616737Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:22.144635496Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:23.145426825Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:23.145486983Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:23.145505912Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:23.145529401Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:23.14554274Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 8535","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:23.14555744Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:23.145571319Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:24.147173936Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:24.147229894Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:24.147249733Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:24.147266512Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:24.147279352Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 8535","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:24.147308081Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:24.14733299Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:25.14937514Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:25.149428868Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:25.149448787Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:25.149472796Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:25.149486136Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 8535","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:25.149500825Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:25.149514934Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:44:26.146 INFO request completed, latency_us: 186, response_code: 200, uri: /state/cid/2, method: GET, req_id: 1d1b89da-7ec1-43e3-ac65-72b8bb7d80ed, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Write block 2035 len 36864 data: 3 5 4 4 6 5 4 5 7
002/300 Read block 1108 len 32768
003/300 Write block 4192 len 28672 data: 5 5 5 6 6 7 7
004/300 Read block 2573 len 12288
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:26.1506188Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:26.150686208Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:26.150737596Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:26.150782974Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:26.150803553Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 4 jobs since flush 8535","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:26.150818553Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:26.150832572Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
005/300 Read block 3654 len 28672
006/300 Read block 2926 len 8192
007/300 Write block 2015 len 40960 data: 7 8 10 8 7 9 7 6 6 3
008/300 Read block 567 len 40960
009/300 Read block 965 len 24576
010/300 Write block 4918 len 28672 data: 7 7 6 4 4 3 4
011/300 Write block 845 len 4096 data: 7
012/300 Read block 597 len 16384
013/300 Read block 1769 len 40960
014/300 Read block 3914 len 16384
015/300 Flush
016/300 Read block 4445 len 28672
017/300 Write block 1250 len 4096 data: 7
018/300 Read block 599 len 40960
019/300 Write block 3153 len 12288 data: 5 6 5
020/300 Flush
021/300 Write block 1395 len 12288 data: 4 4 6
022/300 Flush
023/300 Read block 4116 len 24576
024/300 Read block 1107 len 8192
025/300 Read block 1110 len 32768
026/300 Read block 2242 len 32768
027/300 Write block 1290 len 32768 data: 3 3 2 2 4 4 3 5
028/300 Write block 2489 len 12288 data: 7 7 5
029/300 Write block 2551 len 16384 data: 6 5 5 5
030/300 Read block 4809 len 8192
031/300 Read block 2726 len 36864
032/300 Read block 519 len 36864
033/300 Flush
034/300 Read block 4946 len 8192
035/300 Read block 4468 len 16384
036/300 Read block 2152 len 16384
037/300 Read block 4036 len 16384
038/300 Read block 1738 len 20480
039/300 Write block 4528 len 4096 data: 5
040/300 Read block 1496 len 24576
041/300 Write block 3666 len 12288 data: 8 5 6
042/300 Read block 443 len 16384
043/300 Write block 4720 len 36864 data: 5 7 12 9 10 10 10 9 7
044/300 Read block 1504 len 8192
045/300 Read block 1281 len 24576
046/300 Write block 2795 len 4096 data: 3
047/300 Write block 4908 len 40960 data: 4 4 4 2 3 3 4 5 5 6
048/300 Write block 2090 len 12288 data: 5 5 5
049/300 Flush
050/300 Read block 3962 len 20480
051/300 Read block 4490 len 24576
052/300 Write block 607 len 28672 data: 4 2 2 2 2 2 2
053/300 Write block 4629 len 12288 data: 3 4 4
054/300 Read block 57 len 32768
055/300 Write block 1907 len 8192 data: 6 4
056/300 Write block 3283 len 12288 data: 8 8 8
057/300 Read block 3485 len 32768
058/300 Write block 1880 len 28672 data: 5 4 4 4 4 5 6
059/300 Read block 3727 len 28672
060/300 Write block 726 len 4096 data: 5
061/300 Read block 4240 len 24576
062/300 Write block 2286 len 4096 data: 6
063/300 Read block 3295 len 40960
064/300 Write block 2538 len 28672 data: 7 5 5 4 4 3 2
065/300 Read block 4146 len 4096
066/300 Write block 2704 len 12288 data: 5 6 6
067/300 Read block 736 len 32768
068/300 Read block 2555 len 16384
069/300 Write block 2032 len 20480 data: 3 3 3 4 6
070/300 Read block 4601 len 28672
071/300 Read block 2409 len 24576
072/300 Write block 3738 len 16384 data: 9 7 9 7
073/300 Write block 2069 len 16384 data: 4 4 4 3
074/300 Write block 653 len 4096 data: 4
075/300 Write block 4521 len 20480 data: 3 3 3 4 4
076/300 Flush
077/300 Read block 672 len 36864
078/300 Read block 4269 len 12288
079/300 Read block 3697 len 36864
080/300 Read block 1051 len 4096
081/300 Read block 162 len 16384
082/300 Write block 3663 len 4096 data: 7
083/300 Read block 315 len 8192
084/300 Write block 509 len 24576 data: 4 3 3 4 4 4
085/300 Read block 4618 len 40960
086/300 Write block 865 len 4096 data: 8
087/300 Read block 3677 len 28672
088/300 Write block 3100 len 20480 data: 4 6 5 5 5
089/300 Read block 4262 len 20480
090/300 Read block 2402 len 20480
091/300 Write block 2684 len 36864 data: 3 4 4 5 5 5 4 5 6
092/300 Flush
093/300 Read block 164 len 20480
094/300 Read block 3665 len 36864
095/300 Write block 4218 len 20480 data: 7 5 5 6 7
096/300 Write block 4167 len 28672 data: 9 7 9 7 5 5 5
097/300 Write block 2997 len 40960 data: 4 6 5 5 5 5 5 6 7 7
098/300 Write block 4940 len 28672 data: 8 8 7 6 7 7 7
099/300 Write block 1011 len 28672 data: 3 4 4 4 5 4 4
100/300 Read block 1317 len 8192
101/300 Read block 4631 len 4096
102/300 Write block 3963 len 28672 data: 5 4 4 4 2 2 3
103/300 Read block 3932 len 32768
104/300 Read block 2021 len 24576
105/300 Read block 4477 len 20480
106/300 Read block 2872 len 36864
107/300 Read block 2224 len 4096
108/300 Read block 600 len 4096
109/300 Read block 1628 len 28672
110/300 Write block 1005 len 16384 data: 5 3 3 3
111/300 Flush
112/300 Write block 2468 len 16384 data: 4 4 3 5
113/300 Flush
114/300 Write block 116 len 40960 data: 3 4 5 4 5 8 8 10 11 11
115/300 Write block 783 len 32768 data: 4 4 3 4 5 3 3 4
116/300 Write block 3456 len 8192 data: 5 5
117/300 Read block 557 len 32768
118/300 Write block 2439 len 12288 data: 6 7 9
119/300 Read block 1938 len 20480
120/300 Write block 4466 len 32768 data: 9 7 7 7 8 7 6 5
121/300 Read block 4582 len 32768
122/300 Flush
123/300 Read block 1852 len 32768
124/300 Write block 50 len 36864 data: 7 7 6 6 6 6 6 7 9
125/300 Write block 2611 len 16384 data: 5 5 4 4
126/300 Write block 4770 len 4096 data: 6
127/300 Read block 4934 len 24576
128/300 Read block 651 len 20480
129/300 Write block 3312 len 4096 data: 5
130/300 Write block 2818 len 28672 data: 4 5 4 6 6 6 6
131/300 Write block 3851 len 4096 data: 8
132/300 Write block 4526 len 40960 data: 4 4 6 4 5 6 5 5 5 7
133/300 Read block 3146 len 4096
134/300 Write block 1075 len 16384 data: 6 6 6 5
135/300 Read block 381 len 40960
136/300 Write block 2042 len 8192 data: 6 8
137/300 Read block 2293 len 20480
138/300 Write block 4355 len 40960 data: 5 5 6 7 8 8 7 8 8 5
139/300 Flush
140/300 Write block 1505 len 20480 data: 2 2 2 4 4
141/300 Flush
142/300 Write block 2032 len 12288 data: 4 4 4
143/300 Read block 2803 len 8192
144/300 Write block 3621 len 4096 data: 5
145/300 Write block 2688 len 36864 data: 6 6 5 6 7 6 4 4 3
146/300 Read block 136 len 40960
147/300 Write block 4904 len 24576 data: 6 5 5 4 5 5
148/300 Write block 2066 len 28672 data: 3 4 5 5 5 5 4
149/300 Write block 2037 len 8192 data: 5 5
150/300 Read block 3017 len 24576
151/300 Write block 3348 len 12288 data: 5 6 8
152/300 Write block 304 len 32768 data: 8 8 8 8 8 6 6 5
153/300 Read block 4367 len 32768
154/300 Flush
155/300 Write block 2539 len 28672 data: 6 6 5 5 4 3 2
156/300 Write block 4912 len 36864 data: 4 4 5 6 6 7 8 8 7
157/300 Read block 2270 len 32768
158/300 Write block 2829 len 28672 data: 6 8 5 6 7 6 6
159/300 Flush
160/300 Read block 1201 len 8192
161/300 Write block 2441 len 4096 data: 10
162/300 Flush
163/300 Flush
164/300 Read block 1893 len 40960
165/300 Read block 3719 len 32768
166/300 Read block 4654 len 28672
167/300 Write block 2095 len 28672 data: 3 4 4 4 4 3 4
168/300 Read block 1545 len 28672
169/300 Read block 1850 len 8192
170/300 Read block 151 len 20480
171/300 Read block 888 len 28672
172/300 Flush
173/300 Read block 2237 len 40960
174/300 Read block 4117 len 36864
175/300 Write block 1094 len 12288 data: 3 3 3
176/300 Flush
177/300 Write block 1276 len 8192 data: 8 5
178/300 Flush
179/300 Write block 3138 len 40960 data: 6 7 7 7 6 6 7 8 5 6
180/300 Read block 2671 len 20480
181/300 Read block 3845 len 24576
182/300 Read block 2154 len 28672
183/300 Write block 4641 len 32768 data: 5 6 4 4 5 6 7 6
184/300 Write block 2247 len 28672 data: 6 4 6 6 4 4 5
185/300 Read block 4024 len 20480
186/300 Read block 1644 len 12288
187/300 Read block 3727 len 36864
188/300 Write block 449 len 20480 data: 7 6 9 8 9
189/300 Read block 4693 len 20480
190/300 Write block 4835 len 16384 data: 9 7 6 8
191/300 Read block 4264 len 4096
192/300 Read block 874 len 32768
193/300 Read block 3901 len 16384
194/300 Read block 3439 len 24576
195/300 Write block 4091 len 32768 data: 7 7 5 5 4 2 3 4
196/300 Read block 912 len 36864
197/300 Read block 4998 len 4096
198/300 Read block 2891 len 8192
199/300 Write block 413 len 12288 data: 4 5 5
200/300 Write block 3163 len 24576 data: 4 2 2 2 3 3
201/300 Write block 3021 len 40960 data: 5 6 9 8 6 6 8 8 6 5
202/300 Read block 3784 len 28672
203/300 Write block 4040 len 40960 data: 7 8 7 5 6 5 5 5 5 5
204/300 Read block 3680 len 20480
205/300 Write block 1008 len 12288 data: 4 3 3
206/300 Write block 4658 len 24576 data: 5 6 7 6 7 8
207/300 Read block 4742 len 12288
208/300 Flush
209/300 Read block 1227 len 24576
210/300 Write block 2351 len 12288 data: 6 6 7
211/300 Write block 2564 len 12288 data: 7 8 10
212/300 Write block 1680 len 36864 data: 2 3 4 4 4 4 5 5 6
213/300 Write block 379 len 28672 data: 3 4 5 7 8 10 10
214/300 Read block 3060 len 32768
215/300 Read block 4311 len 28672
216/300 Read block 2835 len 8192
217/300 Write block 620 len 8192 data: 5 8
218/300 Write block 689 len 32768 data: 7 6 5 5 4 3 3 3
219/300 Flush
220/300 Write block 3277 len 4096 data: 8
221/300 Write block 2203 len 16384 data: 7 5 6 6
222/300 Write block 1596 len 4096 data: 7
223/300 Read block 2915 len 4096
224/300 Write block 221 len 8192 data: 5 5
225/300 Read block 415 len 36864
226/300 Write block 2948 len 12288 data: 5 4 2
227/300 Write block 2297 len 24576 data: 4 5 6 7 7 9
228/300 Read block 2577 len 8192
229/300 Write block 3901 len 12288 data: 4 6 7
230/300 Read block 2528 len 40960
231/300 Read block 366 len 20480
232/300 Flush
233/300 Write block 1098 len 20480 data: 3 3 4 5 4
234/300 Write block 733 len 20480 data: 10 9 6 5 4
235/300 Read block 1920 len 16384
236/300 Write block 4284 len 24576 data: 6 8 9 10 9 10
237/300 Read block 4334 len 20480
238/300 Read block 1870 len 36864
239/300 Read block 3671 len 36864
240/300 Read block 860 len 28672
241/300 Write block 3886 len 8192 data: 9 6
242/300 Write block 4733 len 24576 data: 7 6 5 3 3 4
243/300 Write block 1065 len 24576 data: 8 5 6 5 6 8
244/300 Read block 2014 len 28672
245/300 Read block 3976 len 16384
246/300 Write block 2628 len 20480 data: 8 7 5 4 5
247/300 Flush
248/300 Write block 1721 len 8192 data: 7 7
249/300 Read block 2020 len 4096
250/300 Flush
251/300 Write block 4437 len 28672 data: 5 4 3 6 7 7 7
252/300 Write block 265 len 20480 data: 3 3 4 6 6
253/300 Write block 1766 len 36864 data: 8 7 7 6 6 5 5 8 7
254/300 Flush
255/300 Read block 42 len 28672
256/300 Read block 1172 len 4096
257/300 Read block 623 len 20480
258/300 Read block 3274 len 8192
259/300 Write block 2617 len 24576 data: 5 4 5 4 4 4
260/300 Read block 3860 len 36864
261/300 Write block 388 len 16384 data: 7 7 5 5
262/300 Flush
263/300 Write block 1709 len 40960 data: 4 3 3 3 3 3 3 5 6 6
264/300 Read block 4398 len 24576
265/300 Write block 87 len 32768 data: 6 6 6 9 9 9 7 6
266/300 Read block 2578 len 24576
267/300 Read block 191 len 20480
268/300 Read block 3152 len 4096
269/300 Write block 4200 len 24576 data: 6 5 4 6 5 6
270/300 Read block 3401 len 28672
271/300 Flush
272/300 Write block 4649 len 24576 data: 5 6 8 5 5 5
273/300 Write block 4901 len 16384 data: 5 6 6 7
274/300 Write block 3398 len 12288 data: 3 4 6
275/300 Read block 2787 len 20480
276/300 Read block 383 len 28672
277/300 Write block 901 len 24576 data: 6 5 4 5 5 3
278/300 Write block 4836 len 32768 data: 8 7 9 6 5 5 6 5
279/300 Read block 4924 len 8192
280/300 Write block 1550 len 32768 data: 5 5 4 4 4 5 5 4
281/300 Flush
282/300 Write block 4903 len 40960 data: 7 8 6 6 5 6 6 5 3 5
283/300 Read block 4556 len 28672
284/300 Write block 1113 len 20480 data: 4 3 3 3 3
285/300 Flush
286/300 Read block 3395 len 32768
287/300 Read block 620 len 20480
288/300 Write block 2267 len 36864 data: 4 3 4 4 4 5 4 3 4
289/300 Read block 2744 len 8192
290/300 Read block 3073 len 28672
291/300 Write block 2789 len 36864 data: 2 2 2 2 2 3 4 3 3
292/300 Write block 3074 len 16384 data: 4 4 6 7
293/300 Read block 4638 len 12288
294/300 Read block 1923 len 8192
295/300 Read block 4529 len 12288
296/300 Read block 3667 len 20480
297/300 Write block 4883 len 8192 data: 4 4
298/300 Write block 3272 len 24576 data: 3 4 5 6 7 9
299/300 Read block 3817 len 32768
300/300 Read block 4149 len 12288
Main task has work to do, go find it
got dsc Start(2)
start 2
Feb 13 21:44:26.482 INFO request completed, latency_us: 208, response_code: 204, uri: /start/cid/2, method: POST, req_id: 4ce51587-f593-4805-9a19-1d9ea9be7756, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
States: Active Active Offline
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
7537 Acked 8536 Write 9 Done Done New true
7538 Acked 8537 Read 8 Done Done New true
7539 Acked 8538 Write 7 Done Done New true
7540 Acked 8539 Read 3 Done Done New true
7541 Acked 8540 Read 7 Done Done New false
7542 Acked 8541 Read 2 Done Done New false
7543 Acked 8542 Write 10 Done Done New false
7544 Acked 8543 Read 10 Done Done New false
7545 Acked 8544 Read 6 Done Done New false
7546 Acked 8545 Write 7 Done Done New false
7547 Acked 8546 Write 1 Done Done New false
7548 Acked 8547 Read 4 Done Done New false
7549 Acked 8548 Read 10 Done Done New false
7550 Acked 8549 Read 4 Done Done New false
7551 Acked 8550 Flush 0 Done Done New false
7552 Acked 8551 Read 7 Done Done New false
7553 Acked 8552 Write 1 Done Done New false
7554 Acked 8553 Read 10 Done Done New false
7555 Acked 8554 Write 3 Done Done New false
7556 Acked 8555 Flush 0 Done Done New false
7557 Acked 8556 Write 3 Done Done New false
7558 Acked 8557 Flush 0 Done Done New false
7559 Acked 8558 Read 6 Done Done New false
7560 Acked 8559 Read 2 Done Done New false
7561 Acked 8560 Read 8 Done Done New false
7562 Acked 8561 Read 8 Done Done New false
7563 Acked 8562 Write 8 Done Done New false
7564 Acked 8563 Write 3 Done Done New false
7565 Acked 8564 Write 4 Done Done New false
7566 Acked 8565 Read 2 Done Done New false
7567 Acked 8566 Read 9 Done Done New false
7568 Acked 8567 Read 9 Done Done New false
7569 Acked 8568 Flush 0 Done Done New false
7570 Acked 8569 Read 2 Done Done New false
7571 Acked 8570 Read 4 Done Done New false
7572 Acked 8571 Read 4 Done Done New false
7573 Acked 8572 Read 4 Done Done New false
7574 Acked 8573 Read 5 Done Done New false
7575 Acked 8574 Write 1 Done Done New false
7576 Acked 8575 Read 6 Done Done New false
7577 Acked 8576 Write 3 Done Done New false
7578 Acked 8577 Read 4 Done Done New false
7579 Acked 8578 Write 9 Done Done New false
7580 Acked 8579 Read 2 Done Done New false
7581 Acked 8580 Read 6 Done Done New false
7582 Acked 8581 Write 1 Done Done New false
7583 Acked 8582 Write 10 Done Done New false
7584 Acked 8583 Write 3 Done Done New false
7585 Acked 8584 Flush 0 Done Done New false
7586 Acked 8585 Read 5 Done Done New false
7587 Acked 8586 Read 6 Done Done New false
7588 Acked 8587 Write 7 Done Done New false
7589 Acked 8588 Write 3 Done Done New false
7590 Acked 8589 Read 8 Done Done New false
7591 Acked 8590 Write 2 Done Done New false
7592 Acked 8591 Write 3 Done Done New false
7593 Acked 8592 Read 8 Done Done New false
7594 Acked 8593 Write 7 Done Done New false
7595 Acked 8594 Read 7 Done Done New false
7596 Acked 8595 Write 1 Done Done New false
7597 Acked 8596 Read 6 Done Done New false
7598 Acked 8597 Write 1 Done Done New false
7599 Acked 8598 Read 10 Done Done New false
7600 Acked 8599 Write 7 Done Done New false
7601 Acked 8600 Read 1 Done Done New false
7602 Acked 8601 Write 3 Done Done New false
7603 Acked 8602 Read 8 Done Done New false
7604 Acked 8603 Read 4 Done Done New false
7605 Acked 8604 Write 5 Done Done New false
7606 Acked 8605 Read 7 Done Done New false
7607 Acked 8606 Read 6 Done Done New false
7608 Acked 8607 Write 4 Done Done New false
7609 Acked 8608 Write 4 Done Done New false
7610 Acked 8609 Write 1 Done Done New false
7611 Acked 8610 Write 5 Done Done New false
7612 Acked 8611 Flush 0 Done Done New false
7613 Acked 8612 Read 9 Done Done New false
7614 Acked 8613 Read 3 Done Done New false
7615 Acked 8614 Read 9 Done Done New false
7616 Acked 8615 Read 1 Done Done New false
7617 Acked 8616 Read 4 Done Done New false
7618 Acked 8617 Write 1 Done Done New false
7619 Acked 8618 Read 2 Done Done New false
7620 Acked 8619 Write 6 Done Done New false
7621 Acked 8620 Read 10 Done Done New false
7622 Acked 8621 Write 1 Done Done New false
7623 Acked 8622 Read 7 Done Done New false
7624 Acked 8623 Write 5 Done Done New false
7625 Acked 8624 Read 5 Done Done New false
7626 Acked 8625 Read 5 Done Done New false
7627 Acked 8626 Write 9 Done Done New false
7628 Acked 8627 Flush 0 Done Done New false
7629 Acked 8628 Read 5 Done Done New false
7630 Acked 8629 Read 9 Done Done New false
7631 Acked 8630 Write 5 Done Done New false
7632 Acked 8631 Write 7 Done Done New false
7633 Acked 8632 Write 10 Done Done New false
7634 Acked 8633 Write 7 Done Done New false
7635 Acked 8634 Write 7 Done Done New false
7636 Acked 8635 Read 2 Done Done New false
7637 Acked 8636 Read 1 Done Done New false
7638 Acked 8637 Write 7 Done Done New false
7639 Acked 8638 Read 8 Done Done New false
7640 Acked 8639 Read 6 Done Done New false
7641 Acked 8640 Read 5 Done Done New false
7642 Acked 8641 Read 9 Done Done New false
7643 Acked 8642 Read 1 Done Done New false
7644 Acked 8643 Read 1 Done Done New false
7645 Acked 8644 Read 7 Done Done New false
7646 Acked 8645 Write 4 Done Done New false
7647 Acked 8646 Flush 0 Done Done New false
7648 Acked 8647 Write 4 Done Done New false
7649 Acked 8648 Flush 0 Done Done New false
7650 Acked 8649 Write 10 Done Done New false
7651 Acked 8650 Write 8 Done Done New false
7652 Acked 8651 Write 2 Done Done New false
7653 Acked 8652 Read 8 Done Done New false
7654 Acked 8653 Write 3 Done Done New false
7655 Acked 8654 Read 5 Done Done New false
7656 Acked 8655 Write 8 Done Done New false
7657 Acked 8656 Read 8 Done Done New false
7658 Acked 8657 Flush 0 Done Done New false
7659 Acked 8658 Read 8 Done Done New false
7660 Acked 8659 Write 9 Done Done New false
7661 Acked 8660 Write 4 Done Done New false
7662 Acked 8661 Write 1 Done Done New false
7663 Acked 8662 Read 6 Done Done New false
7664 Acked 8663 Read 5 Done Done New false
7665 Acked 8664 Write 1 Done Done New false
7666 Acked 8665 Write 7 Done Done New false
7667 Acked 8666 Write 1 Done Done New false
7668 Acked 8667 Write 10 Done Done New false
7669 Acked 8668 Read 1 Done Done New false
7670 Acked 8669 Write 4 Done Done New false
7671 Acked 8670 Read 10 Done Done New false
7672 Acked 8671 Write 2 Done Done New false
7673 Acked 8672 Read 5 Done Done New false
7674 Acked 8673 Write 10 Done Done New false
7675 Acked 8674 Flush 0 Done Done New false
7676 Acked 8675 Write 5 Done Done New false
7677 Acked 8676 Flush 0 Done Done New false
7678 Acked 8677 Write 3 Done Done New false
7679 Acked 8678 Read 2 Done Done New false
7680 Acked 8679 Write 1 Done Done New false
7681 Acked 8680 Write 9 Done Done New false
7682 Acked 8681 Read 10 Done Done New false
7683 Acked 8682 Write 6 Done Done New false
7684 Acked 8683 Write 7 Done Done New false
7685 Acked 8684 Write 2 Done Done New false
7686 Acked 8685 Read 6 Done Done New false
7687 Acked 8686 Write 3 Done Done New false
7688 Acked 8687 Write 8 Done Done New false
7689 Acked 8688 Read 8 Done Done New false
7690 Acked 8689 Flush 0 Done Done New false
7691 Acked 8690 Write 7 Done Done New false
7692 Acked 8691 Write 9 Done Done New false
7693 Acked 8692 Read 8 Done Done New false
7694 Acked 8693 Write 7 Done Done New false
7695 Acked 8694 Flush 0 Done Done New false
7696 Acked 8695 Read 2 Done Done New false
7697 Acked 8696 Write 1 Done Done New false
7698 Acked 8697 Flush 0 Done Done New false
7699 Acked 8698 Flush 0 Done Done New false
7700 Acked 8699 Read 10 Done Done New false
7701 Acked 8700 Read 8 Done Done New false
7702 Acked 8701 Read 7 Done Done New false
7703 Acked 8702 Write 7 Done Done New false
7704 Acked 8703 Read 7 Done Done New false
7705 Acked 8704 Read 2 Done Done New false
7706 Acked 8705 Read 5 Done Done New false
7707 Acked 8706 Read 7 Done Done New false
7708 Acked 8707 Flush 0 Done Done New false
7709 Acked 8708 Read 10 Done Done New false
7710 Acked 8709 Read 9 Done Done New false
7711 Acked 8710 Write 3 Done Done New false
7712 Acked 8711 Flush 0 Done Done New false
7713 Acked 8712 Write 2 Done Done New false
7714 Acked 8713 Flush 0 Done Done New false
7715 Acked 8714 Write 10 Done Done New false
7716 Acked 8715 Read 5 Done Done New false
7717 Acked 8716 Read 6 Done Done New false
7718 Acked 8717 Read 7 Done Done New false
7719 Acked 8718 Write 8 Done Done New false
7720 Acked 8719 Write 7 Done Done New false
7721 Acked 8720 Read 5 Done Done New false
7722 Acked 8721 Read 3 Done Done New false
7723 Acked 8722 Read 9 Done Done New false
7724 Acked 8723 Write 5 Done Done New false
7725 Acked 8724 Read 5 Done Done New false
7726 Acked 8725 Write 4 Done Done New false
7727 Acked 8726 Read 1 Done Done New false
7728 Acked 8727 Read 8 Done Done New false
7729 Acked 8728 Read 4 Done Done New false
7730 Acked 8729 Read 6 Done Done New false
7731 Acked 8730 Write 8 Done Done New false
7732 Acked 8731 Read 9 Done Done New false
7733 Acked 8732 Read 1 Done Done New false
7734 Acked 8733 Read 2 Done Done New false
7735 Acked 8734 Write 3 Done Done New false
7736 Acked 8735 Write 6 Done Done New false
7737 Acked 8736 Write 10 Done Done New false
7738 Acked 8737 Read 7 Done Done New false
7739 Acked 8738 Write 10 Done Done New false
7740 Acked 8739 Read 5 Done Done New false
7741 Acked 8740 Write 3 Done Done New false
7742 Acked 8741 Write 6 Done Done New false
7743 Acked 8742 Read 3 Done Done New false
7744 Acked 8743 Flush 0 Done Done New false
7745 Acked 8744 Read 6 Done Done New false
7746 Acked 8745 Write 3 Done Done New false
7747 Acked 8746 Write 3 Done Done New false
7748 Acked 8747 Write 9 Done Done New false
7749 Acked 8748 Write 7 Done Done New false
7750 Acked 8749 Read 8 Done Done New false
7751 Acked 8750 Read 7 Done Done New false
7752 Acked 8751 Read 2 Done Done New false
7753 Acked 8752 Write 2 Done Done New false
7754 Acked 8753 Write 8 Done Done New false
7755 Acked 8754 Flush 0 Done Done New false
7756 Acked 8755 Write 1 Done Done New false
7757 Acked 8756 Write 4 Done Done New false
7758 Acked 8757 Write 1 Done Done New false
7759 Acked 8758 Read 1 Done Done New false
7760 Acked 8759 Write 2 Done Done New false
7761 Acked 8760 Read 9 Done Done New false
7762 Acked 8761 Write 3 Done Done New false
7763 Acked 8762 Write 6 Done Done New false
7764 Acked 8763 Read 2 Done Done New false
7765 Acked 8764 Write 3 Done Done New false
7766 Acked 8765 Read 10 Done Done New false
7767 Acked 8766 Read 5 Done Done New false
7768 Acked 8767 Flush 0 Done Done New false
7769 Acked 8768 Write 5 Done Done New false
7770 Acked 8769 Write 5 Done Done New false
7771 Acked 8770 Read 4 Done Done New false
7772 Acked 8771 Write 6 Done Done New false
7773 Acked 8772 Read 5 Done Done New false
7774 Acked 8773 Read 9 Done Done New false
7775 Acked 8774 Read 9 Done Done New false
7776 Acked 8775 Read 7 Done Done New false
7777 Acked 8776 Write 2 Done Done New false
7778 Acked 8777 Write 6 Done Done New false
7779 Acked 8778 Write 6 Done Done New false
7780 Acked 8779 Read 7 Done Done New false
7781 Acked 8780 Read 4 Done Done New false
7782 Acked 8781 Write 5 Done Done New false
7783 Acked 8782 Flush 0 Done Done New false
7784 Acked 8783 Write 2 Done Done New false
7785 Acked 8784 Read 1 Done Done New false
7786 Acked 8785 Flush 0 Done Done New false
7787 Acked 8786 Write 7 Done Done New false
7788 Acked 8787 Write 5 Done Done New false
7789 Acked 8788 Write 9 Done Done New false
7790 Acked 8789 Flush 0 Done Done New false
7791 Acked 8790 Read 7 Done Done New false
7792 Acked 8791 Read 1 Done Done New false
7793 Acked 8792 Read 5 Done Done New false
7794 Acked 8793 Read 2 Done Done New false
7795 Acked 8794 Write 6 Done Done New false
7796 Acked 8795 Read 9 Done Done New false
7797 Acked 8796 Write 4 Done Done New false
7798 Acked 8797 Flush 0 Done Done New false
7799 Acked 8798 Write 10 Done Done New false
7800 Acked 8799 Read 6 Done Done New false
7801 Acked 8800 Write 8 Done Done New false
7802 Acked 8801 Read 6 Done Done New false
7803 Acked 8802 Read 5 Done Done New false
7804 Acked 8803 Read 1 Done Done New false
7805 Acked 8804 Write 6 Done Done New false
7806 Acked 8805 Read 7 Done Done New false
7807 Acked 8806 Flush 0 Done Done New false
7808 Acked 8807 Write 6 Done Done New false
7809 Acked 8808 Write 4 Done Done New false
7810 Acked 8809 Write 3 Done Done New false
7811 Acked 8810 Read 5 Done Done New false
7812 Acked 8811 Read 7 Done Done New false
7813 Acked 8812 Write 6 Done Done New false
7814 Acked 8813 Write 8 Done Done New false
7815 Acked 8814 Read 2 Done Done New false
7816 Acked 8815 Write 8 Done Done New false
7817 Acked 8816 Flush 0 Done Done New false
7818 Acked 8817 Write 10 Done Done New false
7819 Acked 8818 Read 7 Done Done New false
7820 Acked 8819 Write 5 Done Done New false
7821 Acked 8820 Flush 0 Done Done New false
7822 Acked 8821 Read 8 Done Done New false
7823 Acked 8822 Read 5 Done Done New false
7824 Acked 8823 Write 9 Done Done New false
7825 Acked 8824 Read 2 Done Done New false
7826 Acked 8825 Read 7 Done Done New false
7827 Acked 8826 Write 9 Done Done New false
7828 Acked 8827 Write 4 Done Done New false
7829 Acked 8828 Read 3 Done Done New false
7830 Acked 8829 Read 2 Done Done New false
7831 Acked 8830 Read 3 Done Done New false
7832 Acked 8831 Read 5 Done Done New false
7833 Acked 8832 Write 2 Done Done New false
7834 Acked 8833 Write 6 Done Done New false
7835 Acked 8834 Read 8 Done Done New false
7836 Acked 8835 Read 3 Done Done New false
STATES DS:0 DS:1 DS:2 TOTAL
New 0 0 300 300
Sent 0 0 0 0
Done 300 300 0 600
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 8820 8820 8535
Downstairs last five completed: 8535 8534 8533 8532 8531
Upstairs last five completed: 7836 7835 7834 7833 7832
CLIENT: Up:0 ds:300 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:27.152314324Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:27.152375781Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:27.152394971Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:27.15241225Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:27.15242545Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 301 jobs since flush 8535","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:27.152454809Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:27.152614732Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:28.154257889Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:28.154327966Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:28.154359065Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:28.154388184Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:28.154404793Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 301 jobs since flush 8535","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:28.154425262Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:28.154583706Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1190)
[8830][2] reports Running
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:29.156334898Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:29.156398176Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:29.157723445Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:29.157752004Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 8535","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:29.157767283Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 8535","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:29.157926947Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:29.157950756Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:29.157964926Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:29.157978805Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:29.157992775Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 8836 8835 8834 8833 8832
Upstairs last five completed: 7837 7836 7835 7834 7833
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Feb 13 21:44:30.486 INFO request completed, latency_us: 374, response_code: [8810] Got stop action so:false kr:false
204, uri: /stop/cid/0, method: POST, req_id: 1a81cce5-15d1-4258-8888-5b19d762b0a2, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:30.487121247Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:30.487197844Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:30.487231843Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:44:30.487 INFO request completed{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:30.487255291Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
, latency_us: 151, response_code: 200, uri: /state/cid/0, method: GET, req_id: e94f00e4-0540-4bb7-9654-307e27a49e7f, remote_addr: 127.0.0.1:52124, {"msg":"[0] client re-new 0 jobs since flush 8836","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:30.487275081Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
local_addr: 127.0.0.1{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:30.487332939Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
:9998
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:31.488390027Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:31.488452725Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:31.488476304Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:31.488498173Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:31.488514002Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 8836","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:31.488540551Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:31.48857437Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:32.490540204Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:32.490769895Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:32.490858992Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:32.490924229Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:32.490946448Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 8836","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:32.490961108Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:32.490975047Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:33.491585513Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:33.4916514Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:33.491670529Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:33.491687559Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:33.491700328Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 8836","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:33.491714428Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:33.491733937Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:44:34.489 INFO request completed, latency_us: 205, response_code: 200, uri: /state/cid/0, method: GET, req_id: 25c381d8-2a28-48f7-a33d-95a761efc860, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Write block 4029 len 4096 data: 3
002/300 Write block 886 len 4096 data: 12
003/300 Read block 4360 len 40960
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:34.492430008Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:34.492479466Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40004/300 Read block 334 len 40960
,"time":"2024-02-13T21:44:34.492498536Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:34.492528285Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:34.492553564Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 3 jobs since flush 8836","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:34.492597772Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:34.492630131Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
005/300 Write block 2002 len 28672 data: 8 8 7 7 5 6 5
006/300 Read block 1360 len 4096
007/300 Write block 47 len 24576 data: 4 6 6 8 8 7
008/300 Read block 63 len 8192
009/300 Read block 4864 len 16384
010/300 Write block 83 len 36864 data: 4 6 6 6 7 7 7 10 10
011/300 Flush
012/300 Read block 2509 len 40960
013/300 Read block 2878 len 16384
014/300 Read block 3456 len 16384
015/300 Read block 943 len 32768
016/300 Write block 1731 len 12288 data: 4 4 6
017/300 Write block 2405 len 8192 data: 11 9
018/300 Read block 3497 len 24576
019/300 Read block 4517 len 20480
020/300 Read block 1201 len 16384
021/300 Write block 2415 len 12288 data: 5 5 4
022/300 Read block 3197 len 12288
023/300 Write block 3918 len 16384 data: 6 5 6 5
024/300 Read block 1028 len 20480
025/300 Read block 2248 len 36864
026/300 Read block 3765 len 40960
027/300 Write block 1434 len 24576 data: 6 6 6 6 5 4
028/300 Read block 623 len 36864
029/300 Write block 2218 len 28672 data: 2 2 2 3 3 3 3
030/300 Read block 484 len 8192
031/300 Write block 4414 len 8192 data: 2 2
032/300 Flush
033/300 Read block 407 len 16384
034/300 Flush
035/300 Write block 1942 len 8192 data: 7 7
036/300 Write block 4203 len 4096 data: 7
037/300 Read block 2121 len 40960
038/300 Read block 3455 len 12288
039/300 Flush
040/300 Flush
041/300 Read block 1040 len 36864
042/300 Read block 2731 len 28672
043/300 Read block 1701 len 36864
044/300 Read block 3913 len 40960
045/300 Write block 4064 len 32768 data: 5 5 5 5 5 5 5 3
046/300 Write block 4453 len 24576 data: 4 4 5 5 5 5
047/300 Flush
048/300 Read block 1871 len 12288
049/300 Write block 24 len 8192 data: 4 5
050/300 Write block 2609 len 16384 data: 5 5 6 6
051/300 Read block 4827 len 4096
052/300 Write block 423 len 32768 data: 3 2 3 3 4 6 6 4
053/300 Read block 3391 len 32768
054/300 Flush
055/300 Write block 306 len 36864 data: 9 9 9 7 7 6 5 3 3
056/300 Write block 3310 len 12288 data: 8 8 6
057/300 Write block 2035 len 24576 data: 5 7 6 6 7 6
058/300 Write block 3857 len 32768 data: 7 8 7 7 6 6 5 3
059/300 Read block 2381 len 16384
060/300 Write block 3025 len 40960 data: 7 7 9 9 7 6 5 6 7 7
061/300 Write block 4919 len 40960 data: 9 8 5 5 4 5 2 2 5 6
062/300 Write block 104 len 8192 data: 6 8
063/300 Read block 3317 len 36864
064/300 Read block 1963 len 40960
065/300 Read block 3202 len 40960
066/300 Flush
067/300 Write block 3385 len 24576 data: 6 4 5 5 5 4
068/300 Write block 3210 len 24576 data: 5 6 7 7 6 6
069/300 Read block 2561 len 4096
070/300 Read block 899 len 20480
071/300 Flush
072/300 Flush
073/300 Write block 1810 len 4096 data: 7
074/300 Flush
075/300 Write block 3036 len 12288 data: 5 4 5
076/300 Write block 3694 len 8192 data: 4 3
077/300 Flush
078/300 Write block 3173 len 40960 data: 6 5 5 6 5 4 4 5 4 4
079/300 Flush
080/300 Write block 3589 len 36864 data: 8 8 7 6 7 5 4 4 4
081/300 Read block 4017 len 40960
082/300 Read block 2034 len 4096
083/300 Write block 3693 len 4096 data: 4
084/300 Read block 2091 len 12288
085/300 Read block 1627 len 24576
086/300 Read block 2147 len 24576
087/300 Read block 4194 len 12288
088/300 Read block 4611 len 24576
089/300 Read block 2832 len 4096
090/300 Read block 1583 len 16384
091/300 Write block 1951 len 32768 data: 4 5 4 4 4 3 3 5
092/300 Flush
093/300 Read block 1676 len 40960
094/300 Write block 2380 len 32768 data: 3 3 3 4 5 5 4 2
095/300 Flush
096/300 Write block 3191 len 32768 data: 6 8 8 10 9 10 12 14
097/300 Read block 212 len 16384
098/300 Flush
099/300 Write block 2901 len 40960 data: 2 3 4 5 5 4 4 5 5 4
100/300 Write block 4911 len 24576 data: 4 6 5 6 7 7
101/300 Read block 2712 len 24576
102/300 Read block 4607 len 16384
103/300 Read block 2785 len 32768
104/300 Read block 3124 len 12288
105/300 Flush
106/300 Read block 4214 len 8192
107/300 Read block 1341 len 8192
108/300 Read block 3104 len 4096
109/300 Read block 3064 len 24576
110/300 Flush
111/300 Read block 1299 len 16384
112/300 Write block 2736 len 40960 data: 6 5 5 4 3 3 4 6 6 7
113/300 Flush
114/300 Flush
115/300 Write block 79 len 4096 data: 3
116/300 Write block 1918 len 28672 data: 6 6 6 6 6 6 6
117/300 Read block 1645 len 40960
118/300 Read block 2134 len 40960
119/300 Flush
120/300 Read block 2849 len 32768
121/300 Read block 2736 len 20480
122/300 Read block 1921 len 32768
123/300 Read block 3037 len 4096
124/300 Read block 2936 len 4096
125/300 Read block 3291 len 8192
126/300 Write block 4716 len 8192 data: 6 5
127/300 Read block 971 len 40960
128/300 Read block 987 len 16384
129/300 Read block 3833 len 40960
130/300 Flush
131/300 Read block 233 len 36864
132/300 Read block 884 len 28672
133/300 Read block 1045 len 4096
134/300 Write block 1804 len 12288 data: 6 6 6
135/300 Read block 234 len 40960
136/300 Flush
137/300 Write block 3213 len 20480 data: 8 7 7 6 7
138/300 Flush
139/300 Write block 1086 len 4096 data: 4
140/300 Write block 2588 len 8192 data: 6 6
141/300 Flush
142/300 Read block 2870 len 20480
143/300 Read block 1914 len 16384
144/300 Read block 4198 len 8192
145/300 Read block 938 len 36864
146/300 Read block 2613 len 28672
147/300 Write block 4826 len 24576 data: 5 8 7 7 7 10
148/300 Write block 4545 len 32768 data: 5 3 5 4 4 4 4 6
149/300 Flush
150/300 Write block 1596 len 36864 data: 8 6 6 6 5 7 7 7 6
151/300 Write block 4084 len 12288 data: 6 6 3
152/300 Read block 3672 len 20480
153/300 Write block 3410 len 20480 data: 9 9 8 6 3
154/300 Read block 3213 len 8192
155/300 Read block 1450 len 20480
156/300 Read block 4946 len 4096
157/300 Write block 3757 len 4096 data: 6
158/300 Read block 96 len 36864
159/300 Read block 2365 len 24576
160/300 Write block 309 len 20480 data: 8 8 7 6 4
161/300 Read block 2174 len 40960
162/300 Flush
163/300 Write block 1584 len 16384 data: 4 4 4 5
164/300 Read block 1670 len 12288
165/300 Write block 2309 len 28672 data: 6 7 7 7 6 4 4
166/300 Read block 1112 len 4096
167/300 Write block 953 len 40960 data: 3 3 3 3 3 5 4 4 4 5
168/300 Flush
169/300 Read block 4389 len 4096
170/300 Write block 3900 len 36864 data: 4 5 7 8 7 6 6 6 6
171/300 Read block 2096 len 36864
172/300 Write block 3055 len 8192 data: 4 4
173/300 Write block 2925 len 4096 data: 4
174/300 Write block 2044 len 12288 data: 5 5 6
175/300 Read block 2317 len 4096
176/300 Read block 1692 len 24576
177/300 Write block 3981 len 12288 data: 5 6 6
178/300 Write block 4520 len 28672 data: 3 4 4 4 5 5 5
179/300 Read block 2299 len 8192
180/300 Write block 1453 len 32768 data: 6 5 5 5 5 5 5 6
181/300 Read block 2944 len 24576
182/300 Read block 4389 len 32768
183/300 Write block 2015 len 40960 data: 8 9 11 9 8 10 8 7 7 4
184/300 Write block 3154 len 12288 data: 7 6 4
185/300 Read block 4479 len 12288
186/300 Read block 4432 len 32768
187/300 Write block 1137 len 12288 data: 5 5 7
188/300 Read block 2850 len 8192
189/300 Write block 1622 len 32768 data: 5 4 4 6 7 7 6 5
190/300 Write block 2904 len 20480 data: 6 6 5 5 6
191/300 Write block 997 len 40960 data: 6 7 8 6 4 5 6 5 6 4
192/300 Read block 2084 len 28672
193/300 Write block 4889 len 24576 data: 4 3 3 3 3 4
194/300 Flush
195/300 Read block 3358 len 12288
196/300 Write block 3817 len 40960 data: 8 7 8 7 8 8 8 7 8 9
197/300 Write block 1459 len 28672 data: 6 7 5 5 4 7 8
198/300 Flush
199/300 Write block 1373 len 8192 data: 6 5
200/300 Read block 3363 len 24576
201/300 Write block 2204 len 32768 data: 6 7 7 6 6 7 7 8
202/300 Read block 4353 len 12288
203/300 Write block 3798 len 16384 data: 6 6 6 6
204/300 Read block 1686 len 8192
205/300 Read block 4055 len 12288
206/300 Write block 1746 len 28672 data: 4 3 3 4 4 5 5
207/300 Read block 183 len 28672
208/300 Read block 3754 len 16384
209/300 Write block 121 len 8192 data: 9 9
210/300 Read block 434 len 36864
211/300 Read block 1728 len 20480
212/300 Flush
213/300 Write block 3775 len 12288 data: 6 7 8
214/300 Read block 3120 len 16384
215/300 Read block 2807 len 4096
216/300 Read block 4701 len 16384
217/300 Read block 895 len 20480
218/300 Write block 4029 len 12288 data: 4 3 2
219/300 Write block 271 len 36864 data: 7 7 5 4 3 3 3 3 4
220/300 Write block 2595 len 24576 data: 9 9 8 8 9 7
221/300 Write block 4497 len 4096 data: 5
222/300 Write block 1595 len 36864 data: 8 9 7 7 7 6 8 8 8
223/300 Write block 527 len 16384 data: 6 6 5 5
224/300 Flush
225/300 Write block 1561 len 36864 data: 3 3 4 4 5 5 4 6 6
226/300 Read block 3983 len 24576
227/300 Read block 4178 len 4096
228/300 Read block 4799 len 12288
229/300 Read block 1909 len 8192
230/300 Flush
231/300 Write block 144 len 32768 data: 5 4 5 6 5 3 3 5
232/300 Read block 3774 len 4096
233/300 Write block 3818 len 16384 data: 8 9 8 9
234/300 Write block 4062 len 16384 data: 5 5 6 6
235/300 Write block 2740 len 32768 data: 4 4 5 7 7 8 7 7
236/300 Flush
237/300 Flush
238/300 Write block 1646 len 36864 data: 8 6 7 5 4 4 4 5 6
239/300 Read block 35 len 8192
240/300 Read block 3312 len 24576
241/300 Write block 1791 len 36864 data: 3 3 3 3 3 3 3 3 3
242/300 Write block 1430 len 24576 data: 4 5 6 7 7 7
243/300 Flush
244/300 Read block 2605 len 32768
245/300 Write block 4463 len 20480 data: 7 7 8 10 8
246/300 Read block 4277 len 16384
247/300 Write block 4314 len 28672 data: 2 3 5 5 5 7 9
248/300 Write block 861 len 12288 data: 7 7 8
249/300 Read block 3276 len 32768
250/300 Read block 4197 len 40960
251/300 Read block 4460 len 20480
252/300 Read block 3223 len 20480
253/300 Write block 2294 len 16384 data: 3 4 4 5
254/300 Flush
255/300 Write block 429 len 16384 data: 7 5 4 7
256/300 Read block 2138 len 8192
257/300 Write block 2131 len 16384 data: 3 6 6 6
258/300 Write block 228 len 36864 data: 4 4 4 6 7 7 6 6 6
259/300 Read block 4338 len 16384
260/300 Write block 1787 len 36864 data: 2 2 3 3 4 4 4 4 4
261/300 Write block 750 len 24576 data: 7 7 6 5 3 3
262/300 Read block 631 len 28672
263/300 Write block 3338 len 8192 data: 7 9
264/300 Read block 613 len 40960
265/300 Write block 1318 len 20480 data: 4 5 4 4 3
266/300 Write block 927 len 12288 data: 5 6 4
267/300 Write block 2603 len 20480 data: 2 2 3 4 4
268/300 Read block 1830 len 20480
269/300 Write block 1395 len 16384 data: 5 5 7 5
270/300 Write block 2814 len 12288 data: 3 4 5
271/300 Flush
272/300 Write block 1631 len 40960 data: 6 6 5 9 8 6 6 6 9 10
273/300 Flush
274/300 Write block 4195 len 12288 data: 7 7 8
275/300 Read block 1918 len 8192
276/300 Read block 2415 len 32768
277/300 Write block 737 len 12288 data: 5 5 6
278/300 Write block 3517 len 28672 data: 9 7 6 6 6 6 5
279/300 Read block 4204 len 36864
280/300 Read block 1858 len 36864
281/300 Write block 2722 len 24576 data: 5 5 5 5 4 2
282/300 Read block 3724 len 32768
283/300 Write block 2503 len 4096 data: 11
284/300 Read block 4354 len 32768
285/300 Write block 1011 len 32768 data: 4 5 5 5 6 5 5 4
286/300 Write block 3510 len 40960 data: 7 6 7 8 9 8 8 10 8 7
287/300 Read block 2248 len 24576
288/300 Read block 1054 len 36864
289/300 Flush
290/300 Write block 3094 len 8192 data: 7 5
291/300 Read block 272 len 12288
292/300 Read block 4322 len 16384
293/300 Read block 449 len 12288
294/300 Write block 804 len 24576 data: 6 7 8 6 6 5
295/300 Flush
296/300 Read block 4192 len 4096
297/300 Read block 2944 len 28672
298/300 Read block 2211 len 8192
299/300 Write block 3320 len 16384 data: 6 6 7 6
300/300 Flush
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 13 21:44:34.832 INFO request completed, latency_us: 217, response_code: 204, uri: /start/cid/0, method: POST, req_id: f87f89f7-ffd8-4490-aca7-3d578728a371, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
7838 Acked 8837 Write 1 New Done Done true
7839 Acked 8838 Write 1 New Done Done true
7840 Acked 8839 Read 10 New Done Done true
7841 Acked 8840 Read 10 New Done Done false
7842 Acked 8841 Write 7 New Done Done false
7843 Acked 8842 Read 1 New Done Done false
7844 Acked 8843 Write 6 New Done Done false
7845 Acked 8844 Read 2 New Done Done false
7846 Acked 8845 Read 4 New Done Done false
7847 Acked 8846 Write 9 New Done Done false
7848 Acked 8847 Flush 0 New Done Done false
7849 Acked 8848 Read 10 New Done Done false
7850 Acked 8849 Read 4 New Done Done false
7851 Acked 8850 Read 4 New Done Done false
7852 Acked 8851 Read 8 New Done Done false
7853 Acked 8852 Write 3 New Done Done false
7854 Acked 8853 Write 2 New Done Done false
7855 Acked 8854 Read 6 New Done Done false
7856 Acked 8855 Read 5 New Done Done false
7857 Acked 8856 Read 4 New Done Done false
7858 Acked 8857 Write 3 New Done Done false
7859 Acked 8858 Read 3 New Done Done false
7860 Acked 8859 Write 4 New Done Done false
7861 Acked 8860 Read 5 New Done Done false
7862 Acked 8861 Read 9 New Done Done false
7863 Acked 8862 Read 10 New Done Done false
7864 Acked 8863 Write 6 New Done Done false
7865 Acked 8864 Read 9 New Done Done false
7866 Acked 8865 Write 7 New Done Done false
7867 Acked 8866 Read 2 New Done Done false
7868 Acked 8867 Write 2 New Done Done false
7869 Acked 8868 Flush 0 New Done Done false
7870 Acked 8869 Read 4 New Done Done false
7871 Acked 8870 Flush 0 New Done Done false
7872 Acked 8871 Write 2 New Done Done false
7873 Acked 8872 Write 1 New Done Done false
7874 Acked 8873 Read 10 New Done Done false
7875 Acked 8874 Read 3 New Done Done false
7876 Acked 8875 Flush 0 New Done Done false
7877 Acked 8876 Flush 0 New Done Done false
7878 Acked 8877 Read 9 New Done Done false
7879 Acked 8878 Read 7 New Done Done false
7880 Acked 8879 Read 9 New Done Done false
7881 Acked 8880 Read 10 New Done Done false
7882 Acked 8881 Write 8 New Done Done false
7883 Acked 8882 Write 6 New Done Done false
7884 Acked 8883 Flush 0 New Done Done false
7885 Acked 8884 Read 3 New Done Done false
7886 Acked 8885 Write 2 New Done Done false
7887 Acked 8886 Write 4 New Done Done false
7888 Acked 8887 Read 1 New Done Done false
7889 Acked 8888 Flush 0 New Done Done false
7890 Acked 8889 Write 8 New Done Done false
7891 Acked 8890 Read 8 New Done Done false
7892 Acked 8891 Flush 0 New Done Done false
7893 Acked 8892 Write 9 New Done Done false
7894 Acked 8893 Write 3 New Done Done false
7895 Acked 8894 Write 6 New Done Done false
7896 Acked 8895 Write 8 New Done Done false
7897 Acked 8896 Read 4 New Done Done false
7898 Acked 8897 Write 10 New Done Done false
7899 Acked 8898 Write 10 New Done Done false
7900 Acked 8899 Write 2 New Done Done false
7901 Acked 8900 Read 9 New Done Done false
7902 Acked 8901 Read 10 New Done Done false
7903 Acked 8902 Read 10 New Done Done false
7904 Acked 8903 Flush 0 New Done Done false
7905 Acked 8904 Write 6 New Done Done false
7906 Acked 8905 Write 6 New Done Done false
7907 Acked 8906 Read 1 New Done Done false
7908 Acked 8907 Read 5 New Done Done false
7909 Acked 8908 Flush 0 New Done Done false
7910 Acked 8909 Flush 0 New Done Done false
7911 Acked 8910 Write 1 New Done Done false
7912 Acked 8911 Flush 0 New Done Done false
7913 Acked 8912 Write 3 New Done Done false
7914 Acked 8913 Write 2 New Done Done false
7915 Acked 8914 Flush 0 New Done Done false
7916 Acked 8915 Write 10 New Done Done false
7917 Acked 8916 Flush 0 New Done Done false
7918 Acked 8917 Write 9 New Done Done false
7919 Acked 8918 Read 10 New Done Done false
7920 Acked 8919 Read 1 New Done Done false
7921 Acked 8920 Write 1 New Done Done false
7922 Acked 8921 Read 3 New Done Done false
7923 Acked 8922 Read 6 New Done Done false
7924 Acked 8923 Read 6 New Done Done false
7925 Acked 8924 Read 3 New Done Done false
7926 Acked 8925 Read 6 New Done Done false
7927 Acked 8926 Read 1 New Done Done false
7928 Acked 8927 Read 4 New Done Done false
7929 Acked 8928 Write 8 New Done Done false
7930 Acked 8929 Flush 0 New Done Done false
7931 Acked 8930 Read 10 New Done Done false
7932 Acked 8931 Write 8 New Done Done false
7933 Acked 8932 Flush 0 New Done Done false
7934 Acked 8933 Write 8 New Done Done false
7935 Acked 8934 Read 4 New Done Done false
7936 Acked 8935 Flush 0 New Done Done false
7937 Acked 8936 Write 10 New Done Done false
7938 Acked 8937 Write 6 New Done Done false
7939 Acked 8938 Read 6 New Done Done false
7940 Acked 8939 Read 4 New Done Done false
7941 Acked 8940 Read 8 New Done Done false
7942 Acked 8941 Read 3 New Done Done false
7943 Acked 8942 Flush 0 New Done Done false
7944 Acked 8943 Read 2 New Done Done false
7945 Acked 8944 Read 2 New Done Done false
7946 Acked 8945 Read 1 New Done Done false
7947 Acked 8946 Read 6 New Done Done false
7948 Acked 8947 Flush 0 New Done Done false
7949 Acked 8948 Read 4 New Done Done false
7950 Acked 8949 Write 10 New Done Done false
7951 Acked 8950 Flush 0 New Done Done false
7952 Acked 8951 Flush 0 New Done Done false
7953 Acked 8952 Write 1 New Done Done false
7954 Acked 8953 Write 7 New Done Done false
7955 Acked 8954 Read 10 New Done Done false
7956 Acked 8955 Read 10 New Done Done false
7957 Acked 8956 Flush 0 New Done Done false
7958 Acked 8957 Read 8 New Done Done false
7959 Acked 8958 Read 5 New Done Done false
7960 Acked 8959 Read 8 New Done Done false
7961 Acked 8960 Read 1 New Done Done false
7962 Acked 8961 Read 1 New Done Done false
7963 Acked 8962 Read 2 New Done Done false
7964 Acked 8963 Write 2 New Done Done false
7965 Acked 8964 Read 10 New Done Done false
7966 Acked 8965 Read 4 New Done Done false
7967 Acked 8966 Read 10 New Done Done false
7968 Acked 8967 Flush 0 New Done Done false
7969 Acked 8968 Read 9 New Done Done false
7970 Acked 8969 Read 7 New Done Done false
7971 Acked 8970 Read 1 New Done Done false
7972 Acked 8971 Write 3 New Done Done false
7973 Acked 8972 Read 10 New Done Done false
7974 Acked 8973 Flush 0 New Done Done false
7975 Acked 8974 Write 5 New Done Done false
7976 Acked 8975 Flush 0 New Done Done false
7977 Acked 8976 Write 1 New Done Done false
7978 Acked 8977 Write 2 New Done Done false
7979 Acked 8978 Flush 0 New Done Done false
7980 Acked 8979 Read 5 New Done Done false
7981 Acked 8980 Read 4 New Done Done false
7982 Acked 8981 Read 2 New Done Done false
7983 Acked 8982 Read 9 New Done Done false
7984 Acked 8983 Read 7 New Done Done false
7985 Acked 8984 Write 6 New Done Done false
7986 Acked 8985 Write 8 New Done Done false
7987 Acked 8986 Flush 0 New Done Done false
7988 Acked 8987 Write 9 New Done Done false
7989 Acked 8988 Write 3 New Done Done false
7990 Acked 8989 Read 5 New Done Done false
7991 Acked 8990 Write 5 New Done Done false
7992 Acked 8991 Read 2 New Done Done false
7993 Acked 8992 Read 5 New Done Done false
7994 Acked 8993 Read 1 New Done Done false
7995 Acked 8994 Write 1 New Done Done false
7996 Acked 8995 Read 9 New Done Done false
7997 Acked 8996 Read 6 New Done Done false
7998 Acked 8997 Write 5 New Done Done false
7999 Acked 8998 Read 10 New Done Done false
8000 Acked 8999 Flush 0 New Done Done false
8001 Acked 9000 Write 4 New Done Done false
8002 Acked 9001 Read 3 New Done Done false
8003 Acked 9002 Write 7 New Done Done false
8004 Acked 9003 Read 1 New Done Done false
8005 Acked 9004 Write 10 New Done Done false
8006 Acked 9005 Flush 0 New Done Done false
8007 Acked 9006 Read 1 New Done Done false
8008 Acked 9007 Write 9 New Done Done false
8009 Acked 9008 Read 9 New Done Done false
8010 Acked 9009 Write 2 New Done Done false
8011 Acked 9010 Write 1 New Done Done false
8012 Acked 9011 Write 3 New Done Done false
8013 Acked 9012 Read 1 New Done Done false
8014 Acked 9013 Read 6 New Done Done false
8015 Acked 9014 Write 3 New Done Done false
8016 Acked 9015 Write 7 New Done Done false
8017 Acked 9016 Read 2 New Done Done false
8018 Acked 9017 Write 8 New Done Done false
8019 Acked 9018 Read 6 New Done Done false
8020 Acked 9019 Read 8 New Done Done false
8021 Acked 9020 Write 10 New Done Done false
8022 Acked 9021 Write 3 New Done Done false
8023 Acked 9022 Read 3 New Done Done false
8024 Acked 9023 Read 8 New Done Done false
8025 Acked 9024 Write 3 New Done Done false
8026 Acked 9025 Read 2 New Done Done false
8027 Acked 9026 Write 8 New Done Done false
8028 Acked 9027 Write 5 New Done Done false
8029 Acked 9028 Write 10 New Done Done false
8030 Acked 9029 Read 7 New Done Done false
8031 Acked 9030 Write 6 New Done Done false
8032 Acked 9031 Flush 0 New Done Done false
8033 Acked 9032 Read 3 New Done Done false
8034 Acked 9033 Write 10 New Done Done false
8035 Acked 9034 Write 7 New Done Done false
8036 Acked 9035 Flush 0 New Done Done false
8037 Acked 9036 Write 2 New Done Done false
8038 Acked 9037 Read 6 New Done Done false
8039 Acked 9038 Write 8 New Done Done false
8040 Acked 9039 Read 3 New Done Done false
8041 Acked 9040 Write 4 New Done Done false
8042 Acked 9041 Read 2 New Done Done false
8043 Acked 9042 Read 3 New Done Done false
8044 Acked 9043 Write 7 New Done Done false
8045 Acked 9044 Read 7 New Done Done false
8046 Acked 9045 Read 4 New Done Done false
8047 Acked 9046 Write 2 New Done Done false
8048 Acked 9047 Read 9 New Done Done false
8049 Acked 9048 Read 5 New Done Done false
8050 Acked 9049 Flush 0 New Done Done false
8051 Acked 9050 Write 3 New Done Done false
8052 Acked 9051 Read 4 New Done Done false
8053 Acked 9052 Read 1 New Done Done false
8054 Acked 9053 Read 4 New Done Done false
8055 Acked 9054 Read 5 New Done Done false
8056 Acked 9055 Write 3 New Done Done false
8057 Acked 9056 Write 9 New Done Done false
8058 Acked 9057 Write 6 New Done Done false
8059 Acked 9058 Write 1 New Done Done false
8060 Acked 9059 Write 9 New Done Done false
8061 Acked 9060 Write 4 New Done Done false
8062 Acked 9061 Flush 0 New Done Done false
8063 Acked 9062 Write 9 New Done Done false
8064 Acked 9063 Read 6 New Done Done false
8065 Acked 9064 Read 1 New Done Done false
8066 Acked 9065 Read 3 New Done Done false
8067 Acked 9066 Read 2 New Done Done false
8068 Acked 9067 Flush 0 New Done Done false
8069 Acked 9068 Write 8 New Done Done false
8070 Acked 9069 Read 1 New Done Done false
8071 Acked 9070 Write 4 New Done Done false
8072 Acked 9071 Write 4 New Done Done false
8073 Acked 9072 Write 8 New Done Done false
8074 Acked 9073 Flush 0 New Done Done false
8075 Acked 9074 Flush 0 New Done Done false
8076 Acked 9075 Write 9 New Done Done false
8077 Acked 9076 Read 2 New Done Done false
8078 Acked 9077 Read 6 New Done Done false
8079 Acked 9078 Write 9 New Done Done false
8080 Acked 9079 Write 6 New Done Done false
8081 Acked 9080 Flush 0 New Done Done false
8082 Acked 9081 Read 8 New Done Done false
8083 Acked 9082 Write 5 New Done Done false
8084 Acked 9083 Read 4 New Done Done false
8085 Acked 9084 Write 7 New Done Done false
8086 Acked 9085 Write 3 New Done Done false
8087 Acked 9086 Read 8 New Done Done false
8088 Acked 9087 Read 10 New Done Done false
8089 Acked 9088 Read 5 New Done Done false
8090 Acked 9089 Read 5 New Done Done false
8091 Acked 9090 Write 4 New Done Done false
8092 Acked 9091 Flush 0 New Done Done false
8093 Acked 9092 Write 4 New Done Done false
8094 Acked 9093 Read 2 New Done Done false
8095 Acked 9094 Write 4 New Done Done false
8096 Acked 9095 Write 9 New Done Done false
8097 Acked 9096 Read 4 New Done Done false
8098 Acked 9097 Write 9 New Done Done false
8099 Acked 9098 Write 6 New Done Done false
8100 Acked 9099 Read 7 New Done Done false
8101 Acked 9100 Write 2 New Done Done false
8102 Acked 9101 Read 10 New Done Done false
8103 Acked 9102 Write 5 New Done Done false
8104 Acked 9103 Write 3 New Done Done false
8105 Acked 9104 Write 5 New Done Done false
8106 Acked 9105 Read 5 New Done Done false
8107 Acked 9106 Write 4 New Done Done false
8108 Acked 9107 Write 3 New Done Done false
8109 Acked 9108 Flush 0 New Done Done false
8110 Acked 9109 Write 10 New Done Done false
8111 Acked 9110 Flush 0 New Done Done false
8112 Acked 9111 Write 3 New Done Done false
8113 Acked 9112 Read 2 New Done Done false
8114 Acked 9113 Read 8 New Done Done false
8115 Acked 9114 Write 3 New Done Done false
8116 Acked 9115 Write 7 New Done Done false
8117 Acked 9116 Read 9 New Done Done false
8118 Acked 9117 Read 9 New Done Done false
8119 Acked 9118 Write 6 New Done Done false
8120 Acked 9119 Read 8 New Done Done false
8121 Acked 9120 Write 1 New Done Done false
8122 Acked 9121 Read 8 New Done Done false
8123 Acked 9122 Write 8 New Done Done false
8124 Acked 9123 Write 10 New Done Done false
8125 Acked 9124 Read 6 New Done Done false
8126 Acked 9125 Read 9 New Done Done false
8127 Acked 9126 Flush 0 New Done Done false
8128 Acked 9127 Write 2 New Done Done false
8129 Acked 9128 Read 3 New Done Done false
8130 Acked 9129 Read 4 New Done Done false
8131 Acked 9130 Read 3 New Done Done false
8132 Acked 9131 Write 6 New Done Done false
8133 Acked 9132 Flush 0 New Done Done false
8134 Acked 9133 Read 1 New Done Done false
8135 Acked 9134 Read 7 New Done Done false
8136 Acked 9135 Read 2 New Done Done false
8137 Acked 9136 Write 4 New Done Done false
8138 Acked 9137 Flush 0 New Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
New 301 0 0 301
Sent 0 0 0 0
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 8836 9137 9137
Downstairs last five completed: 8836 8835 8834 8833 8832
Upstairs last five completed: 8138 8137 8136 8135 8134
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:35.493805214Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:35.493864282Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:35.493888121Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:35.49390935Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:35.493925659Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 8836","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:35.493948038Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:35.494084583Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:36.494618122Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:36.49467209Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:36.494693469Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:36.494713608Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:36.494729568Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 8836","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:36.494802485Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:36.494944959Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:37.4956576Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:37.495709318Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:37.495729607Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:37.495749517Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:37.495770636Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 8836","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:37.495798055Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:37.49593637Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:38.496515956Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:38.496588043Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:38.496610893Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:38.496632412Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:38.496648741Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 8836","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:38.49667026Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:38.496860723Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1191)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
7838 Acked 8837 Write 1 New Done Done true
7839 Acked 8838 Write 1 New Done Done true
7840 Acked 8839 Read 10 New Done Done true
7841 Acked 8840 Read 10 New Done Done true
7842 Acked 8841 Write 7 New Done Done true
7843 Acked 8842 Read 1 New Done Done true
7844 Acked 8843 Write 6 New Done Done true
7845 Acked 8844 Read 2 New Done Done true
7846 Acked 8845 Read 4 New Done Done true
7847 Acked 8846 Write 9 New Done Done true
7848 Acked 8847 Flush 0 New Done Done true
7849 Acked 8848 Read 10 New Done Done true
7850 Acked 8849 Read 4 New Done Done true
7851 Acked 8850 Read 4 New Done Done true
7852 Acked 8851 Read 8 New Done Done true
7853 Acked 8852 Write 3 New Done Done true
7854 Acked 8853 Write 2 New Done Done true
7855 Acked 8854 Read 6 New Done Done true
7856 Acked 8855 Read 5 New Done Done true
7857 Acked 8856 Read 4 New Done Done true
7858 Acked 8857 Write 3 New Done Done true
7859 Acked 8858 Read 3 New Done Done true
7860 Acked 8859 Write 4 New Done Done true
7861 Acked 8860 Read 5 New Done Done true
7862 Acked 8861 Read 9 New Done Done true
7863 Acked 8862 Read 10 New Done Done true
7864 Acked 8863 Write 6 New Done Done true
7865 Acked 8864 Read 9 New Done Done true
7866 Acked 8865 Write 7 New Done Done true
7867 Acked 8866 Read 2 New Done Done true
7868 Acked 8867 Write 2 New Done Done true
7869 Acked 8868 Flush 0 New Done Done true
7870 Acked 8869 Read 4 New Done Done true
7871 Acked 8870 Flush 0 New Done Done true
7872 Acked 8871 Write 2 New Done Done true
7873 Acked 8872 Write 1 New Done Done true
7874 Acked 8873 Read 10 New Done Done true
7875 Acked 8874 Read 3 New Done Done true
7876 Acked 8875 Flush 0 New Done Done true
7877 Acked 8876 Flush 0 New Done Done true
7878 Acked 8877 Read 9 New Done Done true
7879 Acked 8878 Read 7 New Done Done true
7880 Acked 8879 Read 9 New Done Done true
7881 Acked 8880 Read 10 New Done Done true
7882 Acked 8881 Write 8 New Done Done true
7883 Acked 8882 Write 6 New Done Done true
7884 Acked 8883 Flush 0 New Done Done true
7885 Acked 8884 Read 3 New Done Done true
7886 Acked 8885 Write 2 New Done Done true
7887 Acked 8886 Write 4 New Done Done true
7888 Acked 8887 Read 1 New Done Done true
7889 Acked 8888 Flush 0 New Done Done true
7890 Acked 8889 Write 8 New Done Done true
7891 Acked 8890 Read 8 New Done Done true
7892 Acked 8891 Flush 0 New Done Done true
7893 Acked 8892 Write 9 New Done Done true
7894 Acked 8893 Write 3 New Done Done true
7895 Acked 8894 Write 6 New Done Done true
7896 Acked 8895 Write 8 New Done Done true
7897 Acked 8896 Read 4 New Done Done true
7898 Acked 8897 Write 10 New Done Done true
7899 Acked 8898 Write 10 New Done Done true
7900 Acked 8899 Write 2 New Done Done true
7901 Acked 8900 Read 9 New Done Done true
7902 Acked 8901 Read 10 New Done Done true
7903 Acked 8902 Read 10 New Done Done true
7904 Acked 8903 Flush 0 New Done Done true
7905 Acked 8904 Write 6 New Done Done true
7906 Acked 8905 Write 6 New Done Done true
7907 Acked 8906 Read 1 New Done Done true
7908 Acked 8907 Read 5 New Done Done true
7909 Acked 8908 Flush 0 New Done Done true
7910 Acked 8909 Flush 0 New Done Done true
7911 Acked 8910 Write 1 New Done Done true
7912 Acked 8911 Flush 0 New Done Done true
7913 Acked 8912 Write 3 New Done Done true
7914 Acked 8913 Write 2 New Done Done true
7915 Acked 8914 Flush 0 New Done Done true
7916 Acked 8915 Write 10 New Done Done true
7917 Acked 8916 Flush 0 New Done Done true
7918 Acked 8917 Write 9 New Done Done true
7919 Acked 8918 Read 10 New Done Done true
7920 Acked 8919 Read 1 New Done Done true
7921 Acked 8920 Write 1 New Done Done true
7922 Acked 8921 Read 3 New Done Done true
7923 Acked 8922 Read 6 New Done Done true
7924 Acked 8923 Read 6 New Done Done true
7925 Acked 8924 Read 3 New Done Done true
7926 Acked 8925 Read 6 New Done Done true
7927 Acked 8926 Read 1 New Done Done true
7928 Acked 8927 Read 4 New Done Done true
7929 Acked 8928 Write 8 New Done Done true
7930 Acked 8929 Flush 0 New Done Done true
7931 Acked 8930 Read 10 New Done Done true
7932 Acked 8931 Write 8 New Done Done true
7933 Acked 8932 Flush 0 New Done Done true
7934 Acked 8933 Write 8 New Done Done true
7935 Acked 8934 Read 4 New Done Done true
7936 Acked 8935 Flush 0 New Done Done true
7937 Acked 8936 Write 10 New Done Done true
7938 Acked 8937 Write 6 New Done Done true
7939 Acked 8938 Read 6 New Done Done true
7940 Acked 8939 Read 4 New Done Done true
7941 Acked 8940 Read 8 New Done Done true
7942 Acked 8941 Read 3 New Done Done true
7943 Acked 8942 Flush 0 New Done Done true
7944 Acked 8943 Read 2 New Done Done true
7945 Acked 8944 Read 2 New Done Done true
7946 Acked 8945 Read 1 New Done Done true
7947 Acked 8946 Read 6 New Done Done true
7948 Acked 8947 Flush 0 New Done Done true
7949 Acked 8948 Read 4 New Done Done true
7950 Acked 8949 Write 10 New Done Done true
7951 Acked 8950 Flush 0 New Done Done true
7952 Acked 8951 Flush 0 New Done Done true
7953 Acked 8952 Write 1 New Done Done true
7954 Acked 8953 Write 7 New Done Done true
7955 Acked 8954 Read 10 New Done Done true
7956 Acked 8955 Read 10 New Done Done true
7957 Acked 8956 Flush 0 New Done Done true
7958 Acked 8957 Read 8 New Done Done true
7959 Acked 8958 Read 5 New Done Done true
7960 Acked 8959 Read 8 New Done Done true
7961 Acked 8960 Read 1 New Done Done true
7962 Acked 8961 Read 1 New Done Done true
7963 Acked 8962 Read 2 New Done Done true
7964 Acked 8963 Write 2 New Done Done true
7965 Acked 8964 Read 10 New Done Done true
7966 Acked 8965 Read 4 New Done Done true
7967 Acked 8966 Read 10 New Done Done true
7968 Acked 8967 Flush 0 New Done Done true
7969 Acked 8968 Read 9 New Done Done true
7970 Acked 8969 Read 7 New Done Done true
7971 Acked 8970 Read 1 New Done Done true
7972 Acked 8971 Write 3 New Done Done true
7973 Acked 8972 Read 10 New Done Done true
7974 Acked 8973 Flush 0 New Done Done true
7975 Acked 8974 Write 5 New Done Done true
7976 Acked 8975 Flush 0 New Done Done true
7977 Acked 8976 Write 1 New Done Done true
7978 Acked 8977 Write 2 New Done Done true
7979 Acked 8978 Flush 0 New Done Done true
7980 Acked 8979 Read 5 New Done Done true
7981 Acked 8980 Read 4 New Done Done true
7982 Acked 8981 Read 2 New Done Done true
7983 Acked 8982 Read 9 New Done Done true
7984 Acked 8983 Read 7 New Done Done true
7985 Acked 8984 Write 6 New Done Done true
7986 Acked 8985 Write 8 New Done Done true
7987 Acked 8986 Flush 0 New Done Done true
7988 Acked 8987 Write 9 New Done Done true
7989 Acked 8988 Write 3 New Done Done true
7990 Acked 8989 Read 5 New Done Done true
7991 Acked 8990 Write 5 New Done Done true
7992 Acked 8991 Read 2 New Done Done true
7993 Acked 8992 Read 5 New Done Done true
7994 Acked 8993 Read 1 New Done Done true
7995 Acked 8994 Write 1 New Done Done true
7996 Acked 8995 Read 9 New Done Done true
7997 Acked 8996 Read 6 New Done Done true
7998 Acked 8997 Write 5 New Done Done true
7999 Acked 8998 Read 10 New Done Done true
8000 Acked 8999 Flush 0 New Done Done true
8001 Acked 9000 Write 4 New Done Done true
8002 Acked 9001 Read 3 New Done Done true
8003 Acked 9002 Write 7 New Done Done true
8004 Acked 9003 Read 1 New Done Done true
8005 Acked 9004 Write 10 New Done Done true
8006 Acked 9005 Flush 0 New Done Done true
8007 Acked 9006 Read 1 New Done Done true
8008 Acked 9007 Write 9 New Done Done true
8009 Acked 9008 Read 9 New Done Done true
8010 Acked 9009 Write 2 New Done Done true
8011 Acked 9010 Write 1 New Done Done true
8012 Acked 9011 Write 3 New Done Done true
8013 Acked 9012 Read 1 New Done Done true
8014 Acked 9013 Read 6 New Done Done true
8015 Acked 9014 Write 3 New Done Done true
8016 Acked 9015 Write 7 New Done Done true
8017 Acked 9016 Read 2 New Done Done true
8018 Acked 9017 Write 8 New Done Done true
8019 Acked 9018 Read 6 New Done Done true
8020 Acked 9019 Read 8 New Done Done true
8021 Acked 9020 Write 10 New Done Done true
8022 Acked 9021 Write 3 New Done Done true
8023 Acked 9022 Read 3 New Done Done true
8024 Acked 9023 Read 8 New Done Done true
8025 Acked 9024 Write 3 New Done Done true
8026 Acked 9025 Read 2 New Done Done true
8027 Acked 9026 Write 8 New Done Done true
8028 Acked 9027 Write 5 New Done Done true
8029 Acked 9028 Write 10 New Done Done true
8030 Acked 9029 Read 7 New Done Done true
8031 Acked 9030 Write 6 New Done Done true
8032 Acked 9031 Flush 0 New Done Done true
8033 Acked 9032 Read 3 New Done Done true
8034 Acked 9033 Write 10 New Done Done true
8035 Acked 9034 Write 7 New Done Done true
8036 Acked 9035 Flush 0 New Done Done true
8037 Acked 9036 Write 2 New Done Done true
8038 Acked 9037 Read 6 New Done Done true
8039 Acked 9038 Write 8 New Done Done true
8040 Acked 9039 Read 3 New Done Done true
8041 Acked 9040 Write 4 New Done Done true
8042 Acked 9041 Read 2 New Done Done true
8043 Acked 9042 Read 3 New Done Done true
8044 Acked 9043 Write 7 New Done Done true
8045 Acked 9044 Read 7 New Done Done true
8046 Acked 9045 Read 4 New Done Done true
8047 Acked 9046 Write 2 New Done Done true
8048 Acked 9047 Read 9 New Done Done true
8049 Acked 9048 Read 5 New Done Done true
8050 Acked 9049 Flush 0 New Done Done true
8051 Acked 9050 Write 3 New Done Done true
8052 Acked 9051 Read 4 New Done Done true
8053 Acked 9052 Read 1 New Done Done true
8054 Acked 9053 Read 4 New Done Done true
8055 Acked 9054 Read 5 New Done Done true
8056 Acked 9055 Write 3 New Done Done true
8057 Acked 9056 Write 9 New Done Done true
8058 Acked 9057 Write 6 New Done Done true
8059 Acked 9058 Write 1 New Done Done true
8060 Acked 9059 Write 9 New Done Done true
8061 Acked 9060 Write 4 New Done Done true
8062 Acked 9061 Flush 0 New Done Done true
8063 Acked 9062 Write 9 New Done Done true
8064 Acked 9063 Read 6 New Done Done true
8065 Acked 9064 Read 1 New Done Done true
8066 Acked 9065 Read 3 New Done Done true
8067 Acked 9066 Read 2 New Done Done true
8068 Acked 9067 Flush 0 New Done Done true
8069 Acked 9068 Write 8 New Done Done true
8070 Acked 9069 Read 1 New Done Done true
8071 Acked 9070 Write 4 New Done Done true
8072 Acked 9071 Write 4 New Done Done true
8073 Acked 9072 Write 8 New Done Done true
8074 Acked 9073 Flush 0 New Done Done true
8075 Acked 9074 Flush 0 New Done Done true
8076 Acked 9075 Write 9 New Done Done true
8077 Acked 9076 Read 2 New Done Done true
8078 Acked 9077 Read 6 New Done Done true
8079 Acked 9078 Write 9 New Done Done true
8080 Acked 9079 Write 6 New Done Done true
8081 Acked 9080 Flush 0 New Done Done true
8082 Acked 9081 Read 8 New Done Done true
8083 Acked 9082 Write 5 New Done Done true
8084 Acked 9083 Read 4 New Done Done true
8085 Acked 9084 Write 7 New Done Done true
8086 Acked 9085 Write 3 New Done Done true
8087 Acked 9086 Read 8 New Done Done true
8088 Acked 9087 Read 10 New Done Done true
8089 Acked 9088 Read 5 New Done Done true
8090 Acked 9089 Read 5 New Done Done true
8091 Acked 9090 Write 4 New Done Done true
8092 Acked 9091 Flush 0 New Done Done true
8093 Acked 9092 Write 4 New Done Done true
8094 Acked 9093 Read 2 New Done Done true
8095 Acked 9094 Write 4 New Done Done true
8096 Acked 9095 Write 9 New Done Done true
8097 Acked 9096 Read 4 New Done Done true
8098 Acked 9097 Write 9 New Done Done true
8099 Acked 9098 Write 6 New Done Done true
8100 Acked 9099 Read 7 New Done Done true
8101 Acked 9100 Write 2 New Done Done true
8102 Acked 9101 Read 10 New Done Done true
8103 Acked 9102 Write 5 New Done Done true
8104 Acked 9103 Write 3 New Done Done true
8105 Acked 9104 Write 5 New Done Done true
8106 Acked 9105 Read 5 New Done Done true
8107 Acked 9106 Write 4 New Done Done true
8108 Acked 9107 Write 3 New Done Done true
8109 Acked 9108 Flush 0 New Done Done true
8110 Acked 9109 Write 10 New Done Done true
8111 Acked 9110 Flush 0 New Done Done true
8112 Acked 9111 Write 3 New Done Done true
8113 Acked 9112 Read 2 New Done Done true
8114 Acked 9113 Read 8 New Done Done true
8115 Acked 9114 Write 3 New Done Done true
8116 Acked 9115 Write 7 New Done Done true
8117 Acked 9116 Read 9 New Done Done true
8118 Acked 9117 Read 9 New Done Done true
8119 Acked 9118 Write 6 New Done Done true
8120 Acked 9119 Read 8 New Done Done true
8121 Acked 9120 Write 1 New Done Done true
8122 Acked 9121 Read 8 New Done Done true
8123 Acked 9122 Write 8 New Done Done true
8124 Acked 9123 Write 10 New Done Done true
8125 Acked 9124 Read 6 New Done Done true
8126 Acked 9125 Read 9 New Done Done true
8127 Acked 9126 Flush 0 New Done Done true
8128 Acked 9127 Write 2 New Done Done true
8129 Acked 9128 Read 3 New Done Done true
8130 Acked 9129 Read 4 New Done Done true
8131 Acked 9130 Read 3 New Done Done true
8132 Acked 9131 Write 6 New Done Done true
8133 Acked 9132 Flush 0 New Done Done true
8134 Acked 9133 Read 1 New Done Done true
8135 Acked 9134 Read 7 New Done Done true
8136 Acked 9135 Read 2 New Done Done true
8137 Acked 9136 Write 4 New Done Done true
8138 Acked 9137 Flush 0 New Done Done true
STATES DS:0 DS:1 DS:2 TOTAL
New 301 0 0 301
Sent 0 0 0 0
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 8836 9137 9137
Downstairs last five completed: 8836 8835 8834 8833 8832
Upstairs last five completed: 8138 8137 8136 8135 8134
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:39.497939201Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:39.498027288Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:39.4992774Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:39.499305779Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 8836","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:39.499321798Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 8836","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:39.499481552Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:39.499501281Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:39.499515671Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:39.49953381Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:39.499547959Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 9137 9136 9135 9134 9133
Upstairs last five completed: 8138 8137 8136 8135 8134
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(2)
stop 2
[8830] Got stop action so:false kr:false
Feb 13 21:44:42.838 INFO request completed, latency_us: 353, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 5b051da5-8d54-4d5f-84bf-609741e8fe16, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:42.838989964Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:42.839041202Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:42.839078011Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:42.83909157Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:44:42.839 INFO request completed, latency_us: 150{"msg":"[2] client re-new 0 jobs since flush 9137","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:42.839159848Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
, response_code: 200, uri: /state/cid/2, method: GET, {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:42.839192946Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
req_id: fcd77ebf-54c3-4833-b6c5-732ee1367870, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:43.840276194Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:43.840334522Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:43.840356921Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:43.84038247Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:43.840399529Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 9137","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:43.840417308Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:43.840434798Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:44.841567923Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:44.84162839Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:44.841651609Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:44.841672669Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:44.841688258Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 9137","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:44.841705647Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:44.841722797Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:45.842764256Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:45.842826984Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:45.842860122Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:45.842896841Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:45.84291396Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 0 jobs since flush 9137","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:45.842931889Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:45.842949339Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:44:46.841 INFO request completed, latency_us: 237, response_code: 200, uri: /state/cid/2, method: GET, req_id: 0d5dc869-6552-47c3-b90c-661cf1e7f1e2, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Write block 4678 len 28672 data: 4 5 6 9 7 6 4
002/300 Read block 966 len 40960
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:46.843507547Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:46.843560624Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:46.843581164Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:46.843598253Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:46.843611232Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 2 jobs since flush 9137","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:46.843625442Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:46.843639491Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
003/300 Read block 1071 len 32768
004/300 Write block 160 len 40960 data: 5 7 8 11 10 10 7 7 7 8
005/300 Read block 1903 len 20480
006/300 Read block 3054 len 24576
007/300 Flush
008/300 Read block 1440 len 40960
009/300 Read block 4656 len 20480
010/300 Write block 2828 len 32768 data: 7 7 9 6 7 8 7 7
011/300 Write block 4295 len 4096 data: 6
012/300 Read block 4588 len 20480
013/300 Write block 2438 len 28672 data: 6 7 8 11 12 9 7
014/300 Write block 1165 len 16384 data: 6 5 5 5
015/300 Read block 2993 len 12288
016/300 Write block 4721 len 36864 data: 8 13 10 11 11 11 10 8 8
017/300 Read block 2234 len 28672
018/300 Write block 3475 len 24576 data: 5 4 5 5 4 4
019/300 Write block 4205 len 40960 data: 7 6 6 6 9 7 6 5 6 6
020/300 Write block 2154 len 20480 data: 4 4 3 4 4
021/300 Read block 1561 len 8192
022/300 Read block 4266 len 24576
023/300 Flush
024/300 Read block 2523 len 20480
025/300 Read block 4791 len 16384
026/300 Read block 4153 len 36864
027/300 Write block 1365 len 16384 data: 4 4 5 6
028/300 Read block 2360 len 4096
029/300 Write block 2486 len 12288 data: 10 7 7
030/300 Read block 4778 len 28672
031/300 Write block 4342 len 12288 data: 4 4 5
032/300 Flush
033/300 Write block 2735 len 8192 data: 5 7
034/300 Read block 4577 len 4096
035/300 Write block 3688 len 8192 data: 11 10
036/300 Write block 2070 len 28672 data: 6 6 5 4 7 8 8
037/300 Read block 3842 len 4096
038/300 Write block 2890 len 4096 data: 6
039/300 Read block 2691 len 4096
040/300 Write block 819 len 32768 data: 4 5 6 7 7 7 4 7
041/300 Write block 1330 len 12288 data: 4 4 5
042/300 Read block 4281 len 20480
043/300 Read block 4448 len 4096
044/300 Read block 275 len 24576
045/300 Write block 852 len 16384 data: 6 6 8 9
046/300 Write block 4418 len 4096 data: 5
047/300 Read block 1001 len 24576
048/300 Write block 4137 len 12288 data: 3 3 3
049/300 Read block 4637 len 40960
050/300 Read block 1754 len 36864
051/300 Read block 3921 len 32768
052/300 Read block 1267 len 8192
053/300 Write block 2243 len 24576 data: 5 7 7 6 7 5
054/300 Read block 3308 len 32768
055/300 Flush
056/300 Read block 1417 len 24576
057/300 Write block 2926 len 40960 data: 6 6 6 6 6 7 7 7 7 5
058/300 Write block 451 len 28672 data: 10 9 10 9 8 8 8
059/300 Read block 4042 len 36864
060/300 Read block 4089 len 20480
061/300 Write block 182 len 28672 data: 4 4 4 4 6 7 7
062/300 Write block 2363 len 32768 data: 2 2 3 3 5 5 5 5
063/300 Write block 3503 len 24576 data: 5 4 5 5 5 4
064/300 Write block 4252 len 12288 data: 4 5 4
065/300 Flush
066/300 Write block 2967 len 32768 data: 5 7 7 7 7 7 8 8
067/300 Read block 3513 len 8192
068/300 Read block 4206 len 36864
069/300 Write block 4779 len 40960 data: 5 4 3 3 3 3 3 4 5 6
070/300 Read block 950 len 8192
071/300 Read block 615 len 40960
072/300 Read block 4180 len 36864
073/300 Read block 340 len 24576
074/300 Write block 3402 len 28672 data: 7 7 11 11 10 8 11
075/300 Read block 2237 len 24576
076/300 Write block 1115 len 40960 data: 4 4 4 4 4 4 5 4 4 3
077/300 Read block 4301 len 12288
078/300 Read block 4035 len 8192
079/300 Read block 2863 len 12288
080/300 Read block 1509 len 40960
081/300 Read block 3926 len 32768
082/300 Read block 3015 len 24576
083/300 Read block 46 len 24576
084/300 Write block 581 len 16384 data: 5 4 4 3
085/300 Read block 4143 len 20480
086/300 Write block 811 len 24576 data: 3 3 4 4 5 6
087/300 Read block 415 len 40960
088/300 Read block 1807 len 12288
089/300 Read block 1966 len 16384
090/300 Write block 3777 len 32768 data: 9 8 7 6 6 6 5 5
091/300 Read block 1742 len 40960
092/300 Write block 762 len 16384 data: 4 5 6 6
093/300 Read block 3459 len 12288
094/300 Read block 1210 len 16384
095/300 Read block 2657 len 40960
096/300 Read block 491 len 28672
097/300 Flush
098/300 Read block 2725 len 40960
099/300 Read block 1433 len 24576
100/300 Read block 19 len 4096
101/300 Read block 523 len 4096
102/300 Read block 580 len 32768
103/300 Write block 1434 len 24576 data: 8 8 7 7 6 5
104/300 Read block 18 len 16384
105/300 Read block 1988 len 8192
106/300 Read block 2955 len 40960
107/300 Read block 1562 len 28672
108/300 Write block 3328 len 24576 data: 5 4 2 2 2 2
109/300 Flush
110/300 Read block 1746 len 12288
111/300 Read block 2149 len 16384
112/300 Write block 1409 len 20480 data: 7 7 7 6 6
113/300 Flush
114/300 Write block 4861 len 20480 data: 7 7 7 8 6
115/300 Write block 4325 len 40960 data: 11 11 12 10 8 7 6 6 7 9
116/300 Read block 1696 len 24576
117/300 Read block 4656 len 28672
118/300 Read block 2384 len 40960
119/300 Read block 4197 len 12288
120/300 Write block 4131 len 12288 data: 5 4 2
121/300 Write block 1505 len 4096 data: 3
122/300 Write block 364 len 28672 data: 7 6 5 4 4 4 4
123/300 Read block 1714 len 4096
124/300 Read block 2204 len 8192
125/300 Read block 563 len 32768
126/300 Read block 4497 len 28672
127/300 Read block 1700 len 8192
128/300 Read block 2606 len 4096
129/300 Read block 800 len 8192
130/300 Write block 4549 len 20480 data: 5 5 5 7 6
131/300 Read block 3017 len 36864
132/300 Flush
133/300 Read block 781 len 40960
134/300 Write block 1769 len 32768 data: 7 7 6 6 9 8 8 8
135/300 Read block 2813 len 16384
136/300 Write block 329 len 32768 data: 5 7 11 10 7 7 7 7
137/300 Read block 3327 len 4096
138/300 Read block 2303 len 24576
139/300 Read block 362 len 24576
140/300 Read block 3925 len 28672
141/300 Read block 2785 len 40960
142/300 Write block 26 len 20480 data: 4 5 3 4 4
143/300 Read block 4855 len 36864
144/300 Read block 55 len 12288
145/300 Write block 3949 len 16384 data: 5 6 7 7
146/300 Read block 437 len 32768
147/300 Read block 2497 len 4096
148/300 Flush
149/300 Write block 2562 len 28672 data: 9 10 8 9 11 10 8
150/300 Read block 4222 len 20480
151/300 Write block 3173 len 20480 data: 7 6 6 7 6
152/300 Read block 1811 len 32768
153/300 Write block 2164 len 20480 data: 6 7 5 4 4
154/300 Write block 3460 len 20480 data: 4 4 4 3 3
155/300 Write block 2850 len 12288 data: 5 5 7
156/300 Read block 2749 len 32768
157/300 Read block 3624 len 16384
158/300 Read block 989 len 40960
159/300 Flush
160/300 Read block 3028 len 20480
161/300 Read block 3482 len 8192
162/300 Write block 945 len 32768 data: 5 4 3 3 3 2 4 3
163/300 Write block 2899 len 12288 data: 4 3 3
164/300 Write block 314 len 8192 data: 4 2
165/300 Read block 731 len 28672
166/300 Flush
167/300 Read block 1321 len 4096
168/300 Write block 4564 len 8192 data: 5 5
169/300 Read block 581 len 16384
170/300 Flush
171/300 Flush
172/300 Flush
173/300 Write block 1084 len 24576 data: 5 4 5 4 3 3
174/300 Write block 4375 len 28672 data: 3 3 5 6 9 8 7
175/300 Flush
176/300 Flush
177/300 Write block 2561 len 28672 data: 9 10 11 9 10 12 11
178/300 Read block 2153 len 40960
179/300 Write block 3563 len 20480 data: 7 5 3 3 4
180/300 Write block 3648 len 4096 data: 6
181/300 Read block 2179 len 20480
182/300 Read block 4274 len 32768
183/300 Write block 4429 len 16384 data: 4 5 5 5
184/300 Write block 897 len 4096 data: 7
185/300 Read block 1701 len 12288
186/300 Flush
187/300 Read block 1893 len 32768
188/300 Write block 4151 len 40960 data: 7 5 5 6 7 5 4 6 6 6
189/300 Read block 2961 len 20480
190/300 Read block 192 len 16384
191/300 Write block 2381 len 40960 data: 4 4 5 6 6 5 3 2 2 2
192/300 Read block 291 len 32768
193/300 Read block 4887 len 16384
194/300 Read block 3123 len 12288
195/300 Read block 3801 len 16384
196/300 Write block 3808 len 32768 data: 7 7 7 8 8 8 8 7
197/300 Read block 999 len 24576
198/300 Write block 2512 len 8192 data: 8 7
199/300 Read block 4674 len 24576
200/300 Read block 2497 len 12288
201/300 Flush
202/300 Flush
203/300 Read block 3281 len 28672
204/300 Flush
205/300 Read block 3410 len 4096
206/300 Read block 615 len 24576
207/300 Flush
208/300 Read block 4868 len 4096
209/300 Write block 4893 len 12288 data: 4 5 4
210/300 Flush
211/300 Write block 4864 len 12288 data: 9 7 6
212/300 Read block 2082 len 32768
213/300 Write block 2571 len 12288 data: 8 9 10
214/300 Write block 2447 len 4096 data: 7
215/300 Read block 1057 len 32768
216/300 Read block 4328 len 8192
217/300 Read block 686 len 12288
218/300 Write block 4888 len 16384 data: 4 5 4 4
219/300 Write block 1920 len 32768 data: 7 7 7 7 7 4 4 4
220/300 Write block 1497 len 16384 data: 7 6 6 6
221/300 Read block 3841 len 8192
222/300 Flush
223/300 Flush
224/300 Read block 2643 len 36864
225/300 Write block 988 len 4096 data: 6
226/300 Write block 3569 len 40960 data: 6 4 4 4 5 6 5 4 5 5
227/300 Read block 1322 len 20480
228/300 Read block 3962 len 40960
229/300 Flush
230/300 Write block 1966 len 4096 data: 7
231/300 Read block 325 len 16384
232/300 Flush
233/300 Write block 4411 len 40960 data: 4 2 2 3 3 3 4 6 6 6
234/300 Flush
235/300 Read block 3621 len 4096
236/300 Flush
237/300 Write block 2471 len 32768 data: 6 5 4 4 4 4 4 3
238/300 Write block 4743 len 28672 data: 5 6 6 5 4 4 4
239/300 Write block 4971 len 4096 data: 3
240/300 Read block 788 len 20480
241/300 Flush
242/300 Read block 2592 len 24576
243/300 Write block 3972 len 4096 data: 4
244/300 Write block 2459 len 36864 data: 6 6 6 6 7 8 7 6 6
245/300 Write block 40 len 36864 data: 7 6 6 6 6 7 6 5 7
246/300 Read block 4293 len 32768
247/300 Read block 1268 len 16384
248/300 Read block 122 len 16384
249/300 Read block 2799 len 4096
250/300 Write block 3632 len 36864 data: 3 2 2 2 2 2 2 2 2
251/300 Read block 1997 len 20480
252/300 Read block 2928 len 16384
253/300 Flush
254/300 Read block 1445 len 16384
255/300 Write block 3711 len 24576 data: 3 4 2 3 3 3
256/300 Write block 4306 len 16384 data: 5 5 7 7
257/300 Write block 2000 len 24576 data: 11 10 9 9 8 8
258/300 Read block 4516 len 4096
259/300 Flush
260/300 Write block 4761 len 4096 data: 5
261/300 Read block 3127 len 8192
262/300 Read block 1496 len 12288
263/300 Read block 2072 len 12288
264/300 Write block 2214 len 24576 data: 3 3 2 2 3 3
265/300 Read block 1415 len 24576
266/300 Flush
267/300 Write block 2249 len 36864 data: 7 7 5 5 6 7 7 8 9
268/300 Read block 3019 len 20480
269/300 Write block 4318 len 16384 data: 6 8 10 8
270/300 Write block 3755 len 28672 data: 7 8 7 7 6 6 5
271/300 Read block 2035 len 36864
272/300 Read block 3716 len 28672
273/300 Read block 1164 len 12288
274/300 Write block 2379 len 40960 data: 4 4 5 5 6 7 7 6 4 3
275/300 Write block 97 len 20480 data: 7 6 6 5 8
276/300 Write block 3508 len 24576 data: 5 5 8 7 8 9
277/300 Write block 4807 len 36864 data: 4 5 6 7 6 7 6 6 6
278/300 Read block 2642 len 40960
279/300 Write block 3794 len 16384 data: 3 3 4 6
280/300 Read block 635 len 24576
281/300 Write block 4569 len 36864 data: 5 4 5 7 8 7 6 6 6
282/300 Write block 2989 len 4096 data: 6
283/300 Flush
284/300 Write block 1974 len 16384 data: 3 4 4 4
285/300 Write block 1064 len 4096 data: 7
286/300 Write block 1565 len 12288 data: 6 6 5
287/300 Read block 3285 len 4096
288/300 Write block 1454 len 28672 data: 6 6 6 6 6 7 8
289/300 Read block 19 len 24576
290/300 Flush
291/300 Read block 3381 len 28672
292/300 Write block 1014 len 32768 data: 6 7 6 6 5 6 5 5
293/300 Write block 2125 len 32768 data: 5 3 4 4 3 3 4 7
294/300 Read block 2919 len 20480
295/300 Read block 3968 len 20480
296/300 Flush
297/300 Write block 3465 len 32768 data: 3 3 4 3 4 4 4 4
298/300 Read block 2211 len 36864
299/300 Flush
300/300 Read block 517 len 32768
Main task has work to do, go find it
got dsc Start(2)
start 2
Feb 13 21:44:47.183 INFO request completed, latency_us: 209, response_code: 204, uri: /start/cid/2, method: POST, req_id: cff2cdd7-f120-4b39-983f-d045122e28df, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 2, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
States: Active Active Offline
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
8139 Acked 9138 Write 7 Done Done New true
8140 Acked 9139 Read 10 Done Done New true
8141 Acked 9140 Read 8 Done Done New false
8142 Acked 9141 Write 10 Done Done New false
8143 Acked 9142 Read 5 Done Done New false
8144 Acked 9143 Read 6 Done Done New false
8145 Acked 9144 Flush 0 Done Done New false
8146 Acked 9145 Read 10 Done Done New false
8147 Acked 9146 Read 5 Done Done New false
8148 Acked 9147 Write 8 Done Done New false
8149 Acked 9148 Write 1 Done Done New false
8150 Acked 9149 Read 5 Done Done New false
8151 Acked 9150 Write 7 Done Done New false
8152 Acked 9151 Write 4 Done Done New false
8153 Acked 9152 Read 3 Done Done New false
8154 Acked 9153 Write 9 Done Done New false
8155 Acked 9154 Read 7 Done Done New false
8156 Acked 9155 Write 6 Done Done New false
8157 Acked 9156 Write 10 Done Done New false
8158 Acked 9157 Write 5 Done Done New false
8159 Acked 9158 Read 2 Done Done New false
8160 Acked 9159 Read 6 Done Done New false
8161 Acked 9160 Flush 0 Done Done New false
8162 Acked 9161 Read 5 Done Done New false
8163 Acked 9162 Read 4 Done Done New false
8164 Acked 9163 Read 9 Done Done New false
8165 Acked 9164 Write 4 Done Done New false
8166 Acked 9165 Read 1 Done Done New false
8167 Acked 9166 Write 3 Done Done New false
8168 Acked 9167 Read 7 Done Done New false
8169 Acked 9168 Write 3 Done Done New false
8170 Acked 9169 Flush 0 Done Done New false
8171 Acked 9170 Write 2 Done Done New false
8172 Acked 9171 Read 1 Done Done New false
8173 Acked 9172 Write 2 Done Done New false
8174 Acked 9173 Write 7 Done Done New false
8175 Acked 9174 Read 1 Done Done New false
8176 Acked 9175 Write 1 Done Done New false
8177 Acked 9176 Read 1 Done Done New false
8178 Acked 9177 Write 8 Done Done New false
8179 Acked 9178 Write 3 Done Done New false
8180 Acked 9179 Read 5 Done Done New false
8181 Acked 9180 Read 1 Done Done New false
8182 Acked 9181 Read 6 Done Done New false
8183 Acked 9182 Write 4 Done Done New false
8184 Acked 9183 Write 1 Done Done New false
8185 Acked 9184 Read 6 Done Done New false
8186 Acked 9185 Write 3 Done Done New false
8187 Acked 9186 Read 10 Done Done New false
8188 Acked 9187 Read 9 Done Done New false
8189 Acked 9188 Read 8 Done Done New false
8190 Acked 9189 Read 2 Done Done New false
8191 Acked 9190 Write 6 Done Done New false
8192 Acked 9191 Read 8 Done Done New false
8193 Acked 9192 Flush 0 Done Done New false
8194 Acked 9193 Read 6 Done Done New false
8195 Acked 9194 Write 10 Done Done New false
8196 Acked 9195 Write 7 Done Done New false
8197 Acked 9196 Read 9 Done Done New false
8198 Acked 9197 Read 5 Done Done New false
8199 Acked 9198 Write 7 Done Done New false
8200 Acked 9199 Write 8 Done Done New false
8201 Acked 9200 Write 6 Done Done New false
8202 Acked 9201 Write 3 Done Done New false
8203 Acked 9202 Flush 0 Done Done New false
8204 Acked 9203 Write 8 Done Done New false
8205 Acked 9204 Read 2 Done Done New false
8206 Acked 9205 Read 9 Done Done New false
8207 Acked 9206 Write 10 Done Done New false
8208 Acked 9207 Read 2 Done Done New false
8209 Acked 9208 Read 10 Done Done New false
8210 Acked 9209 Read 9 Done Done New false
8211 Acked 9210 Read 6 Done Done New false
8212 Acked 9211 Write 7 Done Done New false
8213 Acked 9212 Read 6 Done Done New false
8214 Acked 9213 Write 10 Done Done New false
8215 Acked 9214 Read 3 Done Done New false
8216 Acked 9215 Read 2 Done Done New false
8217 Acked 9216 Read 3 Done Done New false
8218 Acked 9217 Read 10 Done Done New false
8219 Acked 9218 Read 8 Done Done New false
8220 Acked 9219 Read 6 Done Done New false
8221 Acked 9220 Read 6 Done Done New false
8222 Acked 9221 Write 4 Done Done New false
8223 Acked 9222 Read 5 Done Done New false
8224 Acked 9223 Write 6 Done Done New false
8225 Acked 9224 Read 10 Done Done New false
8226 Acked 9225 Read 3 Done Done New false
8227 Acked 9226 Read 4 Done Done New false
8228 Acked 9227 Write 8 Done Done New false
8229 Acked 9228 Read 10 Done Done New false
8230 Acked 9229 Write 4 Done Done New false
8231 Acked 9230 Read 3 Done Done New false
8232 Acked 9231 Read 4 Done Done New false
8233 Acked 9232 Read 10 Done Done New false
8234 Acked 9233 Read 7 Done Done New false
8235 Acked 9234 Flush 0 Done Done New false
8236 Acked 9235 Read 10 Done Done New false
8237 Acked 9236 Read 6 Done Done New false
8238 Acked 9237 Read 1 Done Done New false
8239 Acked 9238 Read 1 Done Done New false
8240 Acked 9239 Read 8 Done Done New false
8241 Acked 9240 Write 6 Done Done New false
8242 Acked 9241 Read 4 Done Done New false
8243 Acked 9242 Read 2 Done Done New false
8244 Acked 9243 Read 10 Done Done New false
8245 Acked 9244 Read 7 Done Done New false
8246 Acked 9245 Write 6 Done Done New false
8247 Acked 9246 Flush 0 Done Done New false
8248 Acked 9247 Read 3 Done Done New false
8249 Acked 9248 Read 4 Done Done New false
8250 Acked 9249 Write 5 Done Done New false
8251 Acked 9250 Flush 0 Done Done New false
8252 Acked 9251 Write 5 Done Done New false
8253 Acked 9252 Write 10 Done Done New false
8254 Acked 9253 Read 6 Done Done New false
8255 Acked 9254 Read 7 Done Done New false
8256 Acked 9255 Read 10 Done Done New false
8257 Acked 9256 Read 3 Done Done New false
8258 Acked 9257 Write 3 Done Done New false
8259 Acked 9258 Write 1 Done Done New false
8260 Acked 9259 Write 7 Done Done New false
8261 Acked 9260 Read 1 Done Done New false
8262 Acked 9261 Read 2 Done Done New false
8263 Acked 9262 Read 8 Done Done New false
8264 Acked 9263 Read 7 Done Done New false
8265 Acked 9264 Read 2 Done Done New false
8266 Acked 9265 Read 1 Done Done New false
8267 Acked 9266 Read 2 Done Done New false
8268 Acked 9267 Write 5 Done Done New false
8269 Acked 9268 Read 9 Done Done New false
8270 Acked 9269 Flush 0 Done Done New false
8271 Acked 9270 Read 10 Done Done New false
8272 Acked 9271 Write 8 Done Done New false
8273 Acked 9272 Read 4 Done Done New false
8274 Acked 9273 Write 8 Done Done New false
8275 Acked 9274 Read 1 Done Done New false
8276 Acked 9275 Read 6 Done Done New false
8277 Acked 9276 Read 6 Done Done New false
8278 Acked 9277 Read 7 Done Done New false
8279 Acked 9278 Read 10 Done Done New false
8280 Acked 9279 Write 5 Done Done New false
8281 Acked 9280 Read 9 Done Done New false
8282 Acked 9281 Read 3 Done Done New false
8283 Acked 9282 Write 4 Done Done New false
8284 Acked 9283 Read 8 Done Done New false
8285 Acked 9284 Read 1 Done Done New false
8286 Acked 9285 Flush 0 Done Done New false
8287 Acked 9286 Write 7 Done Done New false
8288 Acked 9287 Read 5 Done Done New false
8289 Acked 9288 Write 5 Done Done New false
8290 Acked 9289 Read 8 Done Done New false
8291 Acked 9290 Write 5 Done Done New false
8292 Acked 9291 Write 5 Done Done New false
8293 Acked 9292 Write 3 Done Done New false
8294 Acked 9293 Read 8 Done Done New false
8295 Acked 9294 Read 4 Done Done New false
8296 Acked 9295 Read 10 Done Done New false
8297 Acked 9296 Flush 0 Done Done New false
8298 Acked 9297 Read 5 Done Done New false
8299 Acked 9298 Read 2 Done Done New false
8300 Acked 9299 Write 8 Done Done New false
8301 Acked 9300 Write 3 Done Done New false
8302 Acked 9301 Write 2 Done Done New false
8303 Acked 9302 Read 7 Done Done New false
8304 Acked 9303 Flush 0 Done Done New false
8305 Acked 9304 Read 1 Done Done New false
8306 Acked 9305 Write 2 Done Done New false
8307 Acked 9306 Read 4 Done Done New false
8308 Acked 9307 Flush 0 Done Done New false
8309 Acked 9308 Flush 0 Done Done New false
8310 Acked 9309 Flush 0 Done Done New false
8311 Acked 9310 Write 6 Done Done New false
8312 Acked 9311 Write 7 Done Done New false
8313 Acked 9312 Flush 0 Done Done New false
8314 Acked 9313 Flush 0 Done Done New false
8315 Acked 9314 Write 7 Done Done New false
8316 Acked 9315 Read 10 Done Done New false
8317 Acked 9316 Write 5 Done Done New false
8318 Acked 9317 Write 1 Done Done New false
8319 Acked 9318 Read 5 Done Done New false
8320 Acked 9319 Read 8 Done Done New false
8321 Acked 9320 Write 4 Done Done New false
8322 Acked 9321 Write 1 Done Done New false
8323 Acked 9322 Read 3 Done Done New false
8324 Acked 9323 Flush 0 Done Done New false
8325 Acked 9324 Read 8 Done Done New false
8326 Acked 9325 Write 10 Done Done New false
8327 Acked 9326 Read 5 Done Done New false
8328 Acked 9327 Read 4 Done Done New false
8329 Acked 9328 Write 10 Done Done New false
8330 Acked 9329 Read 8 Done Done New false
8331 Acked 9330 Read 4 Done Done New false
8332 Acked 9331 Read 3 Done Done New false
8333 Acked 9332 Read 4 Done Done New false
8334 Acked 9333 Write 8 Done Done New false
8335 Acked 9334 Read 6 Done Done New false
8336 Acked 9335 Write 2 Done Done New false
8337 Acked 9336 Read 6 Done Done New false
8338 Acked 9337 Read 3 Done Done New false
8339 Acked 9338 Flush 0 Done Done New false
8340 Acked 9339 Flush 0 Done Done New false
8341 Acked 9340 Read 7 Done Done New false
8342 Acked 9341 Flush 0 Done Done New false
8343 Acked 9342 Read 1 Done Done New false
8344 Acked 9343 Read 6 Done Done New false
8345 Acked 9344 Flush 0 Done Done New false
8346 Acked 9345 Read 1 Done Done New false
8347 Acked 9346 Write 3 Done Done New false
8348 Acked 9347 Flush 0 Done Done New false
8349 Acked 9348 Write 3 Done Done New false
8350 Acked 9349 Read 8 Done Done New false
8351 Acked 9350 Write 3 Done Done New false
8352 Acked 9351 Write 1 Done Done New false
8353 Acked 9352 Read 8 Done Done New false
8354 Acked 9353 Read 2 Done Done New false
8355 Acked 9354 Read 3 Done Done New false
8356 Acked 9355 Write 4 Done Done New false
8357 Acked 9356 Write 8 Done Done New false
8358 Acked 9357 Write 4 Done Done New false
8359 Acked 9358 Read 2 Done Done New false
8360 Acked 9359 Flush 0 Done Done New false
8361 Acked 9360 Flush 0 Done Done New false
8362 Acked 9361 Read 9 Done Done New false
8363 Acked 9362 Write 1 Done Done New false
8364 Acked 9363 Write 10 Done Done New false
8365 Acked 9364 Read 5 Done Done New false
8366 Acked 9365 Read 10 Done Done New false
8367 Acked 9366 Flush 0 Done Done New false
8368 Acked 9367 Write 1 Done Done New false
8369 Acked 9368 Read 4 Done Done New false
8370 Acked 9369 Flush 0 Done Done New false
8371 Acked 9370 Write 10 Done Done New false
8372 Acked 9371 Flush 0 Done Done New false
8373 Acked 9372 Read 1 Done Done New false
8374 Acked 9373 Flush 0 Done Done New false
8375 Acked 9374 Write 8 Done Done New false
8376 Acked 9375 Write 7 Done Done New false
8377 Acked 9376 Write 1 Done Done New false
8378 Acked 9377 Read 5 Done Done New false
8379 Acked 9378 Flush 0 Done Done New false
8380 Acked 9379 Read 6 Done Done New false
8381 Acked 9380 Write 1 Done Done New false
8382 Acked 9381 Write 9 Done Done New false
8383 Acked 9382 Write 9 Done Done New false
8384 Acked 9383 Read 8 Done Done New false
8385 Acked 9384 Read 4 Done Done New false
8386 Acked 9385 Read 4 Done Done New false
8387 Acked 9386 Read 1 Done Done New false
8388 Acked 9387 Write 9 Done Done New false
8389 Acked 9388 Read 5 Done Done New false
8390 Acked 9389 Read 4 Done Done New false
8391 Acked 9390 Flush 0 Done Done New false
8392 Acked 9391 Read 4 Done Done New false
8393 Acked 9392 Write 6 Done Done New false
8394 Acked 9393 Write 4 Done Done New false
8395 Acked 9394 Write 6 Done Done New false
8396 Acked 9395 Read 1 Done Done New false
8397 Acked 9396 Flush 0 Done Done New false
8398 Acked 9397 Write 1 Done Done New false
8399 Acked 9398 Read 2 Done Done New false
8400 Acked 9399 Read 3 Done Done New false
8401 Acked 9400 Read 3 Done Done New false
8402 Acked 9401 Write 6 Done Done New false
8403 Acked 9402 Read 6 Done Done New false
8404 Acked 9403 Flush 0 Done Done New false
8405 Acked 9404 Write 9 Done Done New false
8406 Acked 9405 Read 5 Done Done New false
8407 Acked 9406 Write 4 Done Done New false
8408 Acked 9407 Write 7 Done Done New false
8409 Acked 9408 Read 9 Done Done New false
8410 Acked 9409 Read 7 Done Done New false
8411 Acked 9410 Read 3 Done Done New false
8412 Acked 9411 Write 10 Done Done New false
8413 Acked 9412 Write 5 Done Done New false
8414 Acked 9413 Write 6 Done Done New false
8415 Acked 9414 Write 9 Done Done New false
8416 Acked 9415 Read 10 Done Done New false
8417 Acked 9416 Write 4 Done Done New false
8418 Acked 9417 Read 6 Done Done New false
8419 Acked 9418 Write 9 Done Done New false
8420 Acked 9419 Write 1 Done Done New false
8421 Acked 9420 Flush 0 Done Done New false
8422 Acked 9421 Write 4 Done Done New false
8423 Acked 9422 Write 1 Done Done New false
8424 Acked 9423 Write 3 Done Done New false
8425 Acked 9424 Read 1 Done Done New false
8426 Acked 9425 Write 7 Done Done New false
8427 Acked 9426 Read 6 Done Done New false
8428 Acked 9427 Flush 0 Done Done New false
8429 Acked 9428 Read 7 Done Done New false
8430 Acked 9429 Write 8 Done Done New false
8431 Acked 9430 Write 8 Done Done New false
8432 Acked 9431 Read 5 Done Done New false
8433 Acked 9432 Read 5 Done Done New false
8434 Acked 9433 Flush 0 Done Done New false
8435 Acked 9434 Write 8 Done Done New false
8436 Acked 9435 Read 9 Done Done New false
8437 Acked 9436 Flush 0 Done Done New false
8438 Acked 9437 Read 8 Done Done New false
STATES DS:0 DS:1 DS:2 TOTAL
New 0 0 300 300
Sent 0 0 0 0
Done 300 300 0 600
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 9436 9436 9137
Downstairs last five completed: 9137 9136 9135 9134 9133
Upstairs last five completed: 8438 8437 8436 8435 8434
CLIENT: Up:0 ds:300 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:47.845064435Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:47.845130453Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:47.845154922Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:47.845176561Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:47.84519295Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 301 jobs since flush 9137","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:47.84521111Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:47.845251138Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:48.845818745Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:48.845868433Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:48.845891352Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:48.845912312Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:48.845928021Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 301 jobs since flush 9137","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:48.84594602Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:48.846063936Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:49.847252778Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:49.847316926Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:49.847341625Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:49.847362834Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:49.847378874Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 301 jobs since flush 9137","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:49.847396253Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:49.847499129Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8830] got start action
[8830] I am going to restart
Starting downstairs at port 8830
Make output file at "/tmp/dsc/downstairs-8830.txt"
[8830][2] reports Starting
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:50.84848726Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:50.848562177Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:50.848600226Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:50.848636654Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:50.848664823Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] client re-new 301 jobs since flush 9137","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:50.848694732Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:50.848728841Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1192)
[8830][2] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Active Active Offline
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
8139 Acked 9138 Write 7 Done Done New true
8140 Acked 9139 Read 10 Done Done New true
8141 Acked 9140 Read 8 Done Done New true
8142 Acked 9141 Write 10 Done Done New true
8143 Acked 9142 Read 5 Done Done New true
8144 Acked 9143 Read 6 Done Done New true
8145 Acked 9144 Flush 0 Done Done New true
8146 Acked 9145 Read 10 Done Done New true
8147 Acked 9146 Read 5 Done Done New true
8148 Acked 9147 Write 8 Done Done New true
8149 Acked 9148 Write 1 Done Done New true
8150 Acked 9149 Read 5 Done Done New true
8151 Acked 9150 Write 7 Done Done New true
8152 Acked 9151 Write 4 Done Done New true
8153 Acked 9152 Read 3 Done Done New true
8154 Acked 9153 Write 9 Done Done New true
8155 Acked 9154 Read 7 Done Done New true
8156 Acked 9155 Write 6 Done Done New true
8157 Acked 9156 Write 10 Done Done New true
8158 Acked 9157 Write 5 Done Done New true
8159 Acked 9158 Read 2 Done Done New true
8160 Acked 9159 Read 6 Done Done New true
8161 Acked 9160 Flush 0 Done Done New true
8162 Acked 9161 Read 5 Done Done New true
8163 Acked 9162 Read 4 Done Done New true
8164 Acked 9163 Read 9 Done Done New true
8165 Acked 9164 Write 4 Done Done New true
8166 Acked 9165 Read 1 Done Done New true
8167 Acked 9166 Write 3 Done Done New true
8168 Acked 9167 Read 7 Done Done New true
8169 Acked 9168 Write 3 Done Done New true
8170 Acked 9169 Flush 0 Done Done New true
8171 Acked 9170 Write 2 Done Done New true
8172 Acked 9171 Read 1 Done Done New true
8173 Acked 9172 Write 2 Done Done New true
8174 Acked 9173 Write 7 Done Done New true
8175 Acked 9174 Read 1 Done Done New true
8176 Acked 9175 Write 1 Done Done New true
8177 Acked 9176 Read 1 Done Done New true
8178 Acked 9177 Write 8 Done Done New true
8179 Acked 9178 Write 3 Done Done New true
8180 Acked 9179 Read 5 Done Done New true
8181 Acked 9180 Read 1 Done Done New true
8182 Acked 9181 Read 6 Done Done New true
8183 Acked 9182 Write 4 Done Done New true
8184 Acked 9183 Write 1 Done Done New true
8185 Acked 9184 Read 6 Done Done New true
8186 Acked 9185 Write 3 Done Done New true
8187 Acked 9186 Read 10 Done Done New true
8188 Acked 9187 Read 9 Done Done New true
8189 Acked 9188 Read 8 Done Done New true
8190 Acked 9189 Read 2 Done Done New true
8191 Acked 9190 Write 6 Done Done New true
8192 Acked 9191 Read 8 Done Done New true
8193 Acked 9192 Flush 0 Done Done New true
8194 Acked 9193 Read 6 Done Done New true
8195 Acked 9194 Write 10 Done Done New true
8196 Acked 9195 Write 7 Done Done New true
8197 Acked 9196 Read 9 Done Done New true
8198 Acked 9197 Read 5 Done Done New true
8199 Acked 9198 Write 7 Done Done New true
8200 Acked 9199 Write 8 Done Done New true
8201 Acked 9200 Write 6 Done Done New true
8202 Acked 9201 Write 3 Done Done New true
8203 Acked 9202 Flush 0 Done Done New true
8204 Acked 9203 Write 8 Done Done New true
8205 Acked 9204 Read 2 Done Done New true
8206 Acked 9205 Read 9 Done Done New true
8207 Acked 9206 Write 10 Done Done New true
8208 Acked 9207 Read 2 Done Done New true
8209 Acked 9208 Read 10 Done Done New true
8210 Acked 9209 Read 9 Done Done New true
8211 Acked 9210 Read 6 Done Done New true
8212 Acked 9211 Write 7 Done Done New true
8213 Acked 9212 Read 6 Done Done New true
8214 Acked 9213 Write 10 Done Done New true
8215 Acked 9214 Read 3 Done Done New true
8216 Acked 9215 Read 2 Done Done New true
8217 Acked 9216 Read 3 Done Done New true
8218 Acked 9217 Read 10 Done Done New true
8219 Acked 9218 Read 8 Done Done New true
8220 Acked 9219 Read 6 Done Done New true
8221 Acked 9220 Read 6 Done Done New true
8222 Acked 9221 Write 4 Done Done New true
8223 Acked 9222 Read 5 Done Done New true
8224 Acked 9223 Write 6 Done Done New true
8225 Acked 9224 Read 10 Done Done New true
8226 Acked 9225 Read 3 Done Done New true
8227 Acked 9226 Read 4 Done Done New true
8228 Acked 9227 Write 8 Done Done New true
8229 Acked 9228 Read 10 Done Done New true
8230 Acked 9229 Write 4 Done Done New true
8231 Acked 9230 Read 3 Done Done New true
8232 Acked 9231 Read 4 Done Done New true
8233 Acked 9232 Read 10 Done Done New true
8234 Acked 9233 Read 7 Done Done New true
8235 Acked 9234 Flush 0 Done Done New true
8236 Acked 9235 Read 10 Done Done New true
8237 Acked 9236 Read 6 Done Done New true
8238 Acked 9237 Read 1 Done Done New true
8239 Acked 9238 Read 1 Done Done New true
8240 Acked 9239 Read 8 Done Done New true
8241 Acked 9240 Write 6 Done Done New true
8242 Acked 9241 Read 4 Done Done New true
8243 Acked 9242 Read 2 Done Done New true
8244 Acked 9243 Read 10 Done Done New true
8245 Acked 9244 Read 7 Done Done New true
8246 Acked 9245 Write 6 Done Done New true
8247 Acked 9246 Flush 0 Done Done New true
8248 Acked 9247 Read 3 Done Done New true
8249 Acked 9248 Read 4 Done Done New true
8250 Acked 9249 Write 5 Done Done New true
8251 Acked 9250 Flush 0 Done Done New true
8252 Acked 9251 Write 5 Done Done New true
8253 Acked 9252 Write 10 Done Done New true
8254 Acked 9253 Read 6 Done Done New true
8255 Acked 9254 Read 7 Done Done New true
8256 Acked 9255 Read 10 Done Done New true
8257 Acked 9256 Read 3 Done Done New true
8258 Acked 9257 Write 3 Done Done New true
8259 Acked 9258 Write 1 Done Done New true
8260 Acked 9259 Write 7 Done Done New true
8261 Acked 9260 Read 1 Done Done New true
8262 Acked 9261 Read 2 Done Done New true
8263 Acked 9262 Read 8 Done Done New true
8264 Acked 9263 Read 7 Done Done New true
8265 Acked 9264 Read 2 Done Done New true
8266 Acked 9265 Read 1 Done Done New true
8267 Acked 9266 Read 2 Done Done New true
8268 Acked 9267 Write 5 Done Done New true
8269 Acked 9268 Read 9 Done Done New true
8270 Acked 9269 Flush 0 Done Done New true
8271 Acked 9270 Read 10 Done Done New true
8272 Acked 9271 Write 8 Done Done New true
8273 Acked 9272 Read 4 Done Done New true
8274 Acked 9273 Write 8 Done Done New true
8275 Acked 9274 Read 1 Done Done New true
8276 Acked 9275 Read 6 Done Done New true
8277 Acked 9276 Read 6 Done Done New true
8278 Acked 9277 Read 7 Done Done New true
8279 Acked 9278 Read 10 Done Done New true
8280 Acked 9279 Write 5 Done Done New true
8281 Acked 9280 Read 9 Done Done New true
8282 Acked 9281 Read 3 Done Done New true
8283 Acked 9282 Write 4 Done Done New true
8284 Acked 9283 Read 8 Done Done New true
8285 Acked 9284 Read 1 Done Done New true
8286 Acked 9285 Flush 0 Done Done New true
8287 Acked 9286 Write 7 Done Done New true
8288 Acked 9287 Read 5 Done Done New true
8289 Acked 9288 Write 5 Done Done New true
8290 Acked 9289 Read 8 Done Done New true
8291 Acked 9290 Write 5 Done Done New true
8292 Acked 9291 Write 5 Done Done New true
8293 Acked 9292 Write 3 Done Done New true
8294 Acked 9293 Read 8 Done Done New true
8295 Acked 9294 Read 4 Done Done New true
8296 Acked 9295 Read 10 Done Done New true
8297 Acked 9296 Flush 0 Done Done New true
8298 Acked 9297 Read 5 Done Done New true
8299 Acked 9298 Read 2 Done Done New true
8300 Acked 9299 Write 8 Done Done New true
8301 Acked 9300 Write 3 Done Done New true
8302 Acked 9301 Write 2 Done Done New true
8303 Acked 9302 Read 7 Done Done New true
8304 Acked 9303 Flush 0 Done Done New true
8305 Acked 9304 Read 1 Done Done New true
8306 Acked 9305 Write 2 Done Done New true
8307 Acked 9306 Read 4 Done Done New true
8308 Acked 9307 Flush 0 Done Done New true
8309 Acked 9308 Flush 0 Done Done New true
8310 Acked 9309 Flush 0 Done Done New true
8311 Acked 9310 Write 6 Done Done New true
8312 Acked 9311 Write 7 Done Done New true
8313 Acked 9312 Flush 0 Done Done New true
8314 Acked 9313 Flush 0 Done Done New true
8315 Acked 9314 Write 7 Done Done New true
8316 Acked 9315 Read 10 Done Done New true
8317 Acked 9316 Write 5 Done Done New true
8318 Acked 9317 Write 1 Done Done New true
8319 Acked 9318 Read 5 Done Done New true
8320 Acked 9319 Read 8 Done Done New true
8321 Acked 9320 Write 4 Done Done New true
8322 Acked 9321 Write 1 Done Done New true
8323 Acked 9322 Read 3 Done Done New true
8324 Acked 9323 Flush 0 Done Done New true
8325 Acked 9324 Read 8 Done Done New true
8326 Acked 9325 Write 10 Done Done New true
8327 Acked 9326 Read 5 Done Done New true
8328 Acked 9327 Read 4 Done Done New true
8329 Acked 9328 Write 10 Done Done New true
8330 Acked 9329 Read 8 Done Done New true
8331 Acked 9330 Read 4 Done Done New true
8332 Acked 9331 Read 3 Done Done New true
8333 Acked 9332 Read 4 Done Done New true
8334 Acked 9333 Write 8 Done Done New true
8335 Acked 9334 Read 6 Done Done New true
8336 Acked 9335 Write 2 Done Done New true
8337 Acked 9336 Read 6 Done Done New true
8338 Acked 9337 Read 3 Done Done New true
8339 Acked 9338 Flush 0 Done Done New true
8340 Acked 9339 Flush 0 Done Done New true
8341 Acked 9340 Read 7 Done Done New true
8342 Acked 9341 Flush 0 Done Done New true
8343 Acked 9342 Read 1 Done Done New true
8344 Acked 9343 Read 6 Done Done New true
8345 Acked 9344 Flush 0 Done Done New true
8346 Acked 9345 Read 1 Done Done New true
8347 Acked 9346 Write 3 Done Done New true
8348 Acked 9347 Flush 0 Done Done New true
8349 Acked 9348 Write 3 Done Done New true
8350 Acked 9349 Read 8 Done Done New true
8351 Acked 9350 Write 3 Done Done New true
8352 Acked 9351 Write 1 Done Done New true
8353 Acked 9352 Read 8 Done Done New true
8354 Acked 9353 Read 2 Done Done New true
8355 Acked 9354 Read 3 Done Done New true
8356 Acked 9355 Write 4 Done Done New true
8357 Acked 9356 Write 8 Done Done New true
8358 Acked 9357 Write 4 Done Done New true
8359 Acked 9358 Read 2 Done Done New true
8360 Acked 9359 Flush 0 Done Done New true
8361 Acked 9360 Flush 0 Done Done New true
8362 Acked 9361 Read 9 Done Done New true
8363 Acked 9362 Write 1 Done Done New true
8364 Acked 9363 Write 10 Done Done New true
8365 Acked 9364 Read 5 Done Done New true
8366 Acked 9365 Read 10 Done Done New true
8367 Acked 9366 Flush 0 Done Done New true
8368 Acked 9367 Write 1 Done Done New true
8369 Acked 9368 Read 4 Done Done New true
8370 Acked 9369 Flush 0 Done Done New true
8371 Acked 9370 Write 10 Done Done New true
8372 Acked 9371 Flush 0 Done Done New true
8373 Acked 9372 Read 1 Done Done New true
8374 Acked 9373 Flush 0 Done Done New true
8375 Acked 9374 Write 8 Done Done New true
8376 Acked 9375 Write 7 Done Done New true
8377 Acked 9376 Write 1 Done Done New true
8378 Acked 9377 Read 5 Done Done New true
8379 Acked 9378 Flush 0 Done Done New true
8380 Acked 9379 Read 6 Done Done New true
8381 Acked 9380 Write 1 Done Done New true
8382 Acked 9381 Write 9 Done Done New true
8383 Acked 9382 Write 9 Done Done New true
8384 Acked 9383 Read 8 Done Done New true
8385 Acked 9384 Read 4 Done Done New true
8386 Acked 9385 Read 4 Done Done New true
8387 Acked 9386 Read 1 Done Done New true
8388 Acked 9387 Write 9 Done Done New true
8389 Acked 9388 Read 5 Done Done New true
8390 Acked 9389 Read 4 Done Done New true
8391 Acked 9390 Flush 0 Done Done New true
8392 Acked 9391 Read 4 Done Done New true
8393 Acked 9392 Write 6 Done Done New true
8394 Acked 9393 Write 4 Done Done New true
8395 Acked 9394 Write 6 Done Done New true
8396 Acked 9395 Read 1 Done Done New true
8397 Acked 9396 Flush 0 Done Done New true
8398 Acked 9397 Write 1 Done Done New true
8399 Acked 9398 Read 2 Done Done New true
8400 Acked 9399 Read 3 Done Done New true
8401 Acked 9400 Read 3 Done Done New true
8402 Acked 9401 Write 6 Done Done New true
8403 Acked 9402 Read 6 Done Done New true
8404 Acked 9403 Flush 0 Done Done New true
8405 Acked 9404 Write 9 Done Done New true
8406 Acked 9405 Read 5 Done Done New true
8407 Acked 9406 Write 4 Done Done New true
8408 Acked 9407 Write 7 Done Done New true
8409 Acked 9408 Read 9 Done Done New true
8410 Acked 9409 Read 7 Done Done New true
8411 Acked 9410 Read 3 Done Done New true
8412 Acked 9411 Write 10 Done Done New true
8413 Acked 9412 Write 5 Done Done New true
8414 Acked 9413 Write 6 Done Done New true
8415 Acked 9414 Write 9 Done Done New true
8416 Acked 9415 Read 10 Done Done New true
8417 Acked 9416 Write 4 Done Done New true
8418 Acked 9417 Read 6 Done Done New true
8419 Acked 9418 Write 9 Done Done New true
8420 Acked 9419 Write 1 Done Done New true
8421 Acked 9420 Flush 0 Done Done New true
8422 Acked 9421 Write 4 Done Done New true
8423 Acked 9422 Write 1 Done Done New true
8424 Acked 9423 Write 3 Done Done New true
8425 Acked 9424 Read 1 Done Done New true
8426 Acked 9425 Write 7 Done Done New true
8427 Acked 9426 Read 6 Done Done New true
8428 Acked 9427 Flush 0 Done Done New true
8429 Acked 9428 Read 7 Done Done New true
8430 Acked 9429 Write 8 Done Done New true
8431 Acked 9430 Write 8 Done Done New true
8432 Acked 9431 Read 5 Done Done New true
8433 Acked 9432 Read 5 Done Done New true
8434 Acked 9433 Flush 0 Done Done New true
8435 Acked 9434 Write 8 Done Done New true
8436 Acked 9435 Read 9 Done Done New true
8437 Acked 9436 Flush 0 Done Done New true
8438 Acked 9437 Read 8 Done Done New true
8439 Acked 9438 Flush 0 Done Done New true
STATES DS:0 DS:1 DS:2 TOTAL
New 0 0 301 301
Sent 0 0 0 0
Done 301 301 0 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 9438 9438 9137
Downstairs last five completed: 9137 9136 9135 9134 9133
Upstairs last five completed: 8439 8438 8437 8436 8435
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:51.849600136Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:51.849660243Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:51.850943534Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008830 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:51.850973733Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 9137","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:51.850989772Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 9137","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:51.851151166Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:51.851174965Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:51.851189465Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:51.851203444Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[2] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:51.851217294Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 9438 9437 9436 9435 9434
Upstairs last five completed: 8439 8438 8437 8436 8435
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(0)
stop 0
[8810] Got stop action so:false kr:false
Feb 13 21:44:55.188 INFO request completed, latency_us: 372, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 11d5a611-a70a-4dab-a692-1bb1f2be4c6e, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:55.189512131Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:55.189564599Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:44:55.189 INFO request completed, latency_us: 112, response_code: 200{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:55.189587278Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
, uri: /state/cid/0, method: {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:55.189602958Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
GET, req_id: 9f2baf68-886f-4a9f-bfd0-87ff67264aa5, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"[0] client re-new 0 jobs since flush 9438","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:55.189664655Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:55.189722133Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:56.19105574Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:56.191118168Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:56.191141857Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:56.191163026Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:56.191178916Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 9438","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:56.191196855Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:56.191214944Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:57.192279352Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:57.192343839Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:57.192365578Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:57.192386138Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:57.192401927Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 9438","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:57.192427986Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:57.192446245Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:58.194070711Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:58.194187787Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:58.194214186Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:58.194250144Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:58.194281293Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 9438","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:58.194314282Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:58.194334931Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:44:59.191 INFO request completed, latency_us: 211, response_code: 200, uri: /state/cid/0, method: GET, req_id: 1ffd5dc6-0d30-4a16-9a36-814ee8661e7f, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Write block 1844 len 8192 data: 4 4
002/300 Write block 3832 len 16384 data: 6 6 5 5
003/300 Read block 1516 len 20480
004/300 Read block 1777 len 36864
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:59.195400069Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:59.195458297Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:59.195477376Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:44:59.195517485Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:59.195532174Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 4 jobs since flush 9438","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:59.195558073Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:44:59.195685018Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
005/300 Write block 943 len 40960 data: 4 4 6 5 4 4 4 3 5 4
006/300 Read block 4354 len 4096
007/300 Read block 1523 len 12288
008/300 Read block 3313 len 12288
009/300 Read block 4664 len 24576
010/300 Write block 4073 len 4096 data: 3
011/300 Write block 369 len 12288 data: 5 5 4
012/300 Write block 2495 len 24576 data: 3 4 6 7 8 9
013/300 Read block 1349 len 40960
014/300 Flush
015/300 Read block 4843 len 32768
016/300 Write block 1258 len 20480 data: 5 6 8 8 8
017/300 Read block 2538 len 16384
018/300 Read block 1308 len 12288
019/300 Write block 3848 len 24576 data: 6 5 6 9 10 9
020/300 Write block 492 len 40960 data: 7 8 8 7 4 5 5 6 4 4
021/300 Read block 1388 len 8192
022/300 Write block 4244 len 36864 data: 4 5 4 4 5 4 4 5 5
023/300 Read block 12 len 16384
024/300 Read block 3566 len 24576
025/300 Read block 2698 len 36864
026/300 Flush
027/300 Read block 708 len 32768
028/300 Read block 1818 len 20480
029/300 Read block 81 len 36864
030/300 Flush
031/300 Read block 2181 len 16384
032/300 Write block 3464 len 20480 data: 4 4 4 5 4
033/300 Write block 882 len 32768 data: 4 4 8 10 13 10 9 7
034/300 Read block 4057 len 36864
035/300 Read block 397 len 4096
036/300 Read block 3291 len 28672
037/300 Read block 2633 len 40960
038/300 Flush
039/300 Read block 1471 len 8192
040/300 Read block 334 len 32768
041/300 Flush
042/300 Read block 4447 len 24576
043/300 Write block 3548 len 12288 data: 4 5 5
044/300 Read block 2669 len 28672
045/300 Read block 4501 len 40960
046/300 Read block 1936 len 36864
047/300 Read block 4225 len 24576
048/300 Write block 1263 len 24576 data: 8 7 6 6 7 6
049/300 Write block 2938 len 4096 data: 6
050/300 Write block 782 len 20480 data: 4 5 5 4 5
051/300 Read block 3516 len 40960
052/300 Read block 2482 len 20480
053/300 Write block 668 len 16384 data: 7 6 7 7
054/300 Write block 349 len 28672 data: 8 8 10 9 8 9 9
055/300 Read block 137 len 12288
056/300 Write block 2811 len 36864 data: 3 2 2 4 5 6 5 5 6
057/300 Flush
058/300 Write block 506 len 40960 data: 4 3 3 5 4 4 5 5 5 4
059/300 Read block 1849 len 24576
060/300 Write block 972 len 8192 data: 6 6
061/300 Read block 1555 len 20480
062/300 Write block 1712 len 4096 data: 4
063/300 Read block 1754 len 4096
064/300 Write block 564 len 20480 data: 6 8 8 8 6
065/300 Write block 3253 len 36864 data: 5 5 5 6 8 9 7 6 5
066/300 Read block 1064 len 12288
067/300 Read block 4941 len 12288
068/300 Read block 1095 len 40960
069/300 Read block 4048 len 12288
070/300 Read block 2446 len 4096
071/300 Write block 3046 len 16384 data: 5 2 3 3
072/300 Write block 3436 len 28672 data: 6 5 5 4 3 3 3
073/300 Write block 4534 len 12288 data: 6 8 7
074/300 Flush
075/300 Read block 2212 len 32768
076/300 Read block 3566 len 8192
077/300 Read block 4642 len 32768
078/300 Write block 4849 len 8192 data: 5 4
079/300 Read block 3683 len 8192
080/300 Read block 4505 len 20480
081/300 Write block 1390 len 16384 data: 5 7 6 5
082/300 Read block 3969 len 8192
083/300 Read block 1473 len 24576
084/300 Read block 2897 len 8192
085/300 Read block 3011 len 24576
086/300 Write block 335 len 36864 data: 8 8 4 4 4 4 4 3 4
087/300 Flush
088/300 Write block 4029 len 4096 data: 5
089/300 Read block 3823 len 32768
090/300 Write block 2178 len 32768 data: 5 7 6 6 6 6 6 7
091/300 Read block 4501 len 36864
092/300 Read block 2114 len 36864
093/300 Write block 4694 len 32768 data: 6 6 7 6 5 4 6 5
094/300 Read block 2695 len 28672
095/300 Read block 4107 len 8192
096/300 Flush
097/300 Write block 229 len 12288 data: 5 5 7
098/300 Write block 579 len 28672 data: 4 3 6 5 5 4 4
099/300 Flush
100/300 Read block 2214 len 8192
101/300 Read block 1877 len 40960
102/300 Read block 3962 len 8192
103/300 Read block 3146 len 12288
104/300 Write block 2168 len 24576 data: 5 4 6 5 4 3
105/300 Read block 2129 len 40960
106/300 Read block 1353 len 40960
107/300 Read block 3966 len 40960
108/300 Flush
109/300 Read block 2951 len 20480
110/300 Write block 1074 len 4096 data: 5
111/300 Flush
112/300 Read block 4783 len 24576
113/300 Write block 3060 len 32768 data: 4 4 5 4 4 2 2 2
114/300 Read block 1305 len 40960
115/300 Flush
116/300 Flush
117/300 Read block 799 len 20480
118/300 Write block 640 len 28672 data: 9 11 10 9 10 8 7
119/300 Read block 3346 len 24576
120/300 Read block 4209 len 24576
121/300 Read block 2897 len 32768
122/300 Write block 4323 len 8192 data: 12 11
123/300 Read block 1030 len 40960
124/300 Flush
125/300 Write block 3182 len 36864 data: 5 5 5 5 4 4 6 7 6
126/300 Write block 1477 len 28672 data: 6 5 5 3 4 5 4
127/300 Write block 4104 len 12288 data: 3 3 6
128/300 Read block 4745 len 12288
129/300 Write block 1712 len 36864 data: 5 4 4 4 6 7 7 8 8
130/300 Flush
131/300 Write block 2388 len 40960 data: 4 3 3 3 6 6 6 5 5 5
132/300 Read block 4744 len 12288
133/300 Read block 813 len 12288
134/300 Read block 2468 len 40960
135/300 Write block 4188 len 12288 data: 5 5 4
136/300 Write block 2716 len 24576 data: 3 4 5 5 5 5
137/300 Write block 1635 len 4096 data: 9
138/300 Read block 3548 len 16384
139/300 Write block 276 len 24576 data: 4 4 4 5 4 3
140/300 Read block 2610 len 24576
141/300 Write block 3123 len 24576 data: 7 9 8 6 3 5
142/300 Write block 4844 len 20480 data: 6 6 4 5 4
143/300 Write block 3506 len 20480 data: 6 6 6 6 9
144/300 Read block 2671 len 28672
145/300 Read block 3107 len 4096
146/300 Read block 2725 len 12288
147/300 Read block 1863 len 8192
148/300 Write block 57 len 24576 data: 8 10 9 12 14 14
149/300 Write block 4282 len 32768 data: 7 7 7 9 10 11 10 11
150/300 Read block 1709 len 24576
151/300 Flush
152/300 Read block 4438 len 40960
153/300 Read block 1908 len 12288
154/300 Read block 527 len 16384
155/300 Write block 2812 len 28672 data: 3 3 5 6 7 6 6
156/300 Read block 2546 len 4096
157/300 Read block 733 len 40960
158/300 Write block 1294 len 20480 data: 5 5 4 6 7
159/300 Flush
160/300 Write block 4010 len 24576 data: 6 5 5 5 6 6
161/300 Read block 1304 len 16384
162/300 Write block 864 len 40960 data: 7 9 8 7 5 6 6 4 6 5
163/300 Read block 2058 len 36864
164/300 Read block 1809 len 4096
165/300 Write block 4785 len 32768 data: 4 5 6 7 6 4 3 4
166/300 Write block 1366 len 20480 data: 5 6 7 7 7
167/300 Read block 1400 len 36864
168/300 Write block 3773 len 16384 data: 6 5 7 8
169/300 Write block 658 len 32768 data: 3 4 4 4 5 5 4 5
170/300 Read block 3237 len 40960
171/300 Read block 4527 len 8192
172/300 Write block 177 len 36864 data: 5 4 4 3 4 5 5 5 5
173/300 Read block 3557 len 36864
174/300 Read block 1287 len 40960
175/300 Flush
176/300 Read block 4453 len 8192
177/300 Read block 3976 len 20480
178/300 Read block 2353 len 28672
179/300 Write block 1412 len 16384 data: 7 7 6 6
180/300 Write block 60 len 8192 data: 13 15
181/300 Write block 4800 len 16384 data: 7 7 7 8
182/300 Write block 3742 len 16384 data: 7 4 3 3
183/300 Read block 808 len 16384
184/300 Write block 4717 len 24576 data: 6 5 4 6 9 14
185/300 Read block 3489 len 8192
186/300 Read block 326 len 16384
187/300 Flush
188/300 Read block 3245 len 36864
189/300 Write block 2047 len 20480 data: 7 9 9 9 7
190/300 Read block 774 len 28672
191/300 Write block 2522 len 12288 data: 7 7 5
192/300 Write block 4012 len 4096 data: 6
193/300 Read block 4005 len 28672
194/300 Read block 3254 len 36864
195/300 Write block 1689 len 40960 data: 6 6 5 6 6 5 3 3 3 5
196/300 Write block 69 len 12288 data: 9 8 8
197/300 Write block 3384 len 8192 data: 4 7
198/300 Write block 357 len 40960 data: 7 8 8 8 8 8 8 8 7 6
199/300 Read block 3262 len 32768
200/300 Read block 3355 len 32768
201/300 Write block 1050 len 36864 data: 4 4 4 6 6 4 4 5 5
202/300 Read block 4550 len 36864
203/300 Read block 4127 len 32768
204/300 Write block 2296 len 20480 data: 5 6 6 7 8
205/300 Read block 4198 len 28672
206/300 Read block 3849 len 24576
207/300 Write block 4734 len 4096 data: 7
208/300 Read block 2292 len 16384
209/300 Write block 3014 len 40960 data: 7 5 6 6 4 4 4 6 7 10
210/300 Read block 4263 len 36864
211/300 Flush
212/300 Read block 3171 len 12288
213/300 Write block 88 len 32768 data: 8 8 11 11 10 8 7 7
214/300 Write block 687 len 40960 data: 9 8 8 7 6 6 5 4 4 4
215/300 Write block 2724 len 40960 data: 6 6 5 3 2 2 3 5 7 6
216/300 Flush
217/300 Write block 3479 len 24576 data: 5 5 4 5 5 7
218/300 Read block 1086 len 32768
219/300 Read block 3052 len 36864
220/300 Read block 3655 len 32768
221/300 Read block 2064 len 4096
222/300 Flush
223/300 Read block 1661 len 24576
224/300 Write block 2589 len 32768 data: 7 5 4 4 8 7 10 10
225/300 Write block 1773 len 20480 data: 10 9 9 9 8
226/300 Flush
227/300 Write block 1874 len 24576 data: 5 5 5 5 5 5
228/300 Write block 4360 len 12288 data: 9 8 9
229/300 Flush
230/300 Read block 2840 len 36864
231/300 Flush
232/300 Read block 3676 len 28672
233/300 Write block 3527 len 24576 data: 4 4 5 7 7 7
234/300 Read block 1473 len 16384
235/300 Read block 3009 len 28672
236/300 Read block 734 len 24576
237/300 Write block 3239 len 32768 data: 4 4 5 5 5 4 3 4
238/300 Read block 1251 len 32768
239/300 Read block 4251 len 12288
240/300 Read block 3157 len 4096
241/300 Write block 2892 len 4096 data: 5
242/300 Write block 1776 len 4096 data: 10
243/300 Write block 2970 len 8192 data: 8 8
244/300 Read block 1046 len 36864
245/300 Write block 1617 len 20480 data: 7 8 8 6 6
246/300 Read block 3092 len 20480
247/300 Write block 4359 len 16384 data: 9 10 9 10
248/300 Write block 751 len 4096 data: 8
249/300 Write block 2744 len 12288 data: 8 9 8
250/300 Write block 2129 len 4096 data: 4
251/300 Read block 299 len 20480
252/300 Read block 2486 len 12288
253/300 Write block 2895 len 32768 data: 8 8 6 5 5 4 4 4
254/300 Write block 1450 len 36864 data: 5 4 5 7 7 7 7 7 7
255/300 Read block 2086 len 32768
256/300 Write block 2134 len 28672 data: 7 5 6 6 6 6 7
257/300 Read block 2900 len 40960
258/300 Read block 1125 len 32768
259/300 Write block 4705 len 40960 data: 7 5 6 7 7 6 7 10 7 9
260/300 Read block 2503 len 28672
261/300 Write block 2866 len 36864 data: 4 3 2 2 2 4 3 3 3
262/300 Write block 736 len 16384 data: 6 6 6 7
263/300 Read block 3880 len 16384
264/300 Read block 2873 len 40960
265/300 Flush
266/300 Read block 4965 len 36864
267/300 Read block 1211 len 28672
268/300 Read block 2960 len 40960
269/300 Write block 2542 len 36864 data: 6 5 4 3 3 4 5 6 5
270/300 Write block 2699 len 12288 data: 3 3 3
271/300 Flush
272/300 Write block 2043 len 32768 data: 9 6 6 7 8 10 10 10
273/300 Write block 3655 len 8192 data: 8 8
274/300 Write block 2630 len 20480 data: 6 5 6 5 3
275/300 Write block 1264 len 12288 data: 8 7 7
276/300 Flush
277/300 Read block 4952 len 4096
278/300 Read block 4778 len 32768
279/300 Flush
280/300 Write block 4845 len 32768 data: 7 5 6 5 6 5 3 4
281/300 Flush
282/300 Write block 3894 len 4096 data: 6
283/300 Write block 4326 len 32768 data: 12 13 11 9 8 7 7 8
284/300 Write block 2745 len 8192 data: 10 9
285/300 Write block 2471 len 4096 data: 7
286/300 Read block 247 len 28672
287/300 Write block 4428 len 20480 data: 4 5 6 6 6
288/300 Read block 3948 len 8192
289/300 Write block 4763 len 20480 data: 4 5 4 3 3
290/300 Write block 179 len 36864 data: 5 4 5 6 6 6 6 7 8
291/300 Write block 574 len 8192 data: 5 4
292/300 Read block 2097 len 8192
293/300 Flush
294/300 Write block 2619 len 16384 data: 6 5 5 5
295/300 Write block 3157 len 32768 data: 4 5 4 4 5 5 5 3
296/300 Write block 4782 len 28672 data: 4 4 4 5 6 7 8
297/300 Read block 4026 len 16384
298/300 Write block 1162 len 16384 data: 5 5 6 7
299/300 Write block 3617 len 20480 data: 7 5 4 7 6
300/300 Write block 3415 len 12288 data: 3 2 3
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 13 21:44:59.537 INFO request completed, latency_us: 306, response_code: 204, uri: /start/cid/0, method: POST, req_id: 03b07377-e8b0-41ce-b750-ecff3dcf22ec, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
8440 Acked 9439 Write 2 New Done Done true
8441 Acked 9440 Write 4 New Done Done true
8442 Acked 9441 Read 5 New Done Done true
8443 Acked 9442 Read 9 New Done Done true
8444 Acked 9443 Write 10 New Done Done false
8445 Acked 9444 Read 1 New Done Done false
8446 Acked 9445 Read 3 New Done Done false
8447 Acked 9446 Read 3 New Done Done false
8448 Acked 9447 Read 6 New Done Done false
8449 Acked 9448 Write 1 New Done Done false
8450 Acked 9449 Write 3 New Done Done false
8451 Acked 9450 Write 6 New Done Done false
8452 Acked 9451 Read 10 New Done Done false
8453 Acked 9452 Flush 0 New Done Done false
8454 Acked 9453 Read 8 New Done Done false
8455 Acked 9454 Write 5 New Done Done false
8456 Acked 9455 Read 4 New Done Done false
8457 Acked 9456 Read 3 New Done Done false
8458 Acked 9457 Write 6 New Done Done false
8459 Acked 9458 Write 10 New Done Done false
8460 Acked 9459 Read 2 New Done Done false
8461 Acked 9460 Write 9 New Done Done false
8462 Acked 9461 Read 4 New Done Done false
8463 Acked 9462 Read 6 New Done Done false
8464 Acked 9463 Read 9 New Done Done false
8465 Acked 9464 Flush 0 New Done Done false
8466 Acked 9465 Read 8 New Done Done false
8467 Acked 9466 Read 5 New Done Done false
8468 Acked 9467 Read 9 New Done Done false
8469 Acked 9468 Flush 0 New Done Done false
8470 Acked 9469 Read 4 New Done Done false
8471 Acked 9470 Write 5 New Done Done false
8472 Acked 9471 Write 8 New Done Done false
8473 Acked 9472 Read 9 New Done Done false
8474 Acked 9473 Read 1 New Done Done false
8475 Acked 9474 Read 7 New Done Done false
8476 Acked 9475 Read 10 New Done Done false
8477 Acked 9476 Flush 0 New Done Done false
8478 Acked 9477 Read 2 New Done Done false
8479 Acked 9478 Read 8 New Done Done false
8480 Acked 9479 Flush 0 New Done Done false
8481 Acked 9480 Read 6 New Done Done false
8482 Acked 9481 Write 3 New Done Done false
8483 Acked 9482 Read 7 New Done Done false
8484 Acked 9483 Read 10 New Done Done false
8485 Acked 9484 Read 9 New Done Done false
8486 Acked 9485 Read 6 New Done Done false
8487 Acked 9486 Write 6 New Done Done false
8488 Acked 9487 Write 1 New Done Done false
8489 Acked 9488 Write 5 New Done Done false
8490 Acked 9489 Read 10 New Done Done false
8491 Acked 9490 Read 5 New Done Done false
8492 Acked 9491 Write 4 New Done Done false
8493 Acked 9492 Write 7 New Done Done false
8494 Acked 9493 Read 3 New Done Done false
8495 Acked 9494 Write 9 New Done Done false
8496 Acked 9495 Flush 0 New Done Done false
8497 Acked 9496 Write 10 New Done Done false
8498 Acked 9497 Read 6 New Done Done false
8499 Acked 9498 Write 2 New Done Done false
8500 Acked 9499 Read 5 New Done Done false
8501 Acked 9500 Write 1 New Done Done false
8502 Acked 9501 Read 1 New Done Done false
8503 Acked 9502 Write 5 New Done Done false
8504 Acked 9503 Write 9 New Done Done false
8505 Acked 9504 Read 3 New Done Done false
8506 Acked 9505 Read 3 New Done Done false
8507 Acked 9506 Read 10 New Done Done false
8508 Acked 9507 Read 3 New Done Done false
8509 Acked 9508 Read 1 New Done Done false
8510 Acked 9509 Write 4 New Done Done false
8511 Acked 9510 Write 7 New Done Done false
8512 Acked 9511 Write 3 New Done Done false
8513 Acked 9512 Flush 0 New Done Done false
8514 Acked 9513 Read 8 New Done Done false
8515 Acked 9514 Read 2 New Done Done false
8516 Acked 9515 Read 8 New Done Done false
8517 Acked 9516 Write 2 New Done Done false
8518 Acked 9517 Read 2 New Done Done false
8519 Acked 9518 Read 5 New Done Done false
8520 Acked 9519 Write 4 New Done Done false
8521 Acked 9520 Read 2 New Done Done false
8522 Acked 9521 Read 6 New Done Done false
8523 Acked 9522 Read 2 New Done Done false
8524 Acked 9523 Read 6 New Done Done false
8525 Acked 9524 Write 9 New Done Done false
8526 Acked 9525 Flush 0 New Done Done false
8527 Acked 9526 Write 1 New Done Done false
8528 Acked 9527 Read 8 New Done Done false
8529 Acked 9528 Write 8 New Done Done false
8530 Acked 9529 Read 9 New Done Done false
8531 Acked 9530 Read 9 New Done Done false
8532 Acked 9531 Write 8 New Done Done false
8533 Acked 9532 Read 7 New Done Done false
8534 Acked 9533 Read 2 New Done Done false
8535 Acked 9534 Flush 0 New Done Done false
8536 Acked 9535 Write 3 New Done Done false
8537 Acked 9536 Write 7 New Done Done false
8538 Acked 9537 Flush 0 New Done Done false
8539 Acked 9538 Read 2 New Done Done false
8540 Acked 9539 Read 10 New Done Done false
8541 Acked 9540 Read 2 New Done Done false
8542 Acked 9541 Read 3 New Done Done false
8543 Acked 9542 Write 6 New Done Done false
8544 Acked 9543 Read 10 New Done Done false
8545 Acked 9544 Read 10 New Done Done false
8546 Acked 9545 Read 10 New Done Done false
8547 Acked 9546 Flush 0 New Done Done false
8548 Acked 9547 Read 5 New Done Done false
8549 Acked 9548 Write 1 New Done Done false
8550 Acked 9549 Flush 0 New Done Done false
8551 Acked 9550 Read 6 New Done Done false
8552 Acked 9551 Write 8 New Done Done false
8553 Acked 9552 Read 10 New Done Done false
8554 Acked 9553 Flush 0 New Done Done false
8555 Acked 9554 Flush 0 New Done Done false
8556 Acked 9555 Read 5 New Done Done false
8557 Acked 9556 Write 7 New Done Done false
8558 Acked 9557 Read 6 New Done Done false
8559 Acked 9558 Read 6 New Done Done false
8560 Acked 9559 Read 8 New Done Done false
8561 Acked 9560 Write 2 New Done Done false
8562 Acked 9561 Read 10 New Done Done false
8563 Acked 9562 Flush 0 New Done Done false
8564 Acked 9563 Write 9 New Done Done false
8565 Acked 9564 Write 7 New Done Done false
8566 Acked 9565 Write 3 New Done Done false
8567 Acked 9566 Read 3 New Done Done false
8568 Acked 9567 Write 9 New Done Done false
8569 Acked 9568 Flush 0 New Done Done false
8570 Acked 9569 Write 10 New Done Done false
8571 Acked 9570 Read 3 New Done Done false
8572 Acked 9571 Read 3 New Done Done false
8573 Acked 9572 Read 10 New Done Done false
8574 Acked 9573 Write 3 New Done Done false
8575 Acked 9574 Write 6 New Done Done false
8576 Acked 9575 Write 1 New Done Done false
8577 Acked 9576 Read 4 New Done Done false
8578 Acked 9577 Write 6 New Done Done false
8579 Acked 9578 Read 6 New Done Done false
8580 Acked 9579 Write 6 New Done Done false
8581 Acked 9580 Write 5 New Done Done false
8582 Acked 9581 Write 5 New Done Done false
8583 Acked 9582 Read 7 New Done Done false
8584 Acked 9583 Read 1 New Done Done false
8585 Acked 9584 Read 3 New Done Done false
8586 Acked 9585 Read 2 New Done Done false
8587 Acked 9586 Write 6 New Done Done false
8588 Acked 9587 Write 8 New Done Done false
8589 Acked 9588 Read 6 New Done Done false
8590 Acked 9589 Flush 0 New Done Done false
8591 Acked 9590 Read 10 New Done Done false
8592 Acked 9591 Read 3 New Done Done false
8593 Acked 9592 Read 4 New Done Done false
8594 Acked 9593 Write 7 New Done Done false
8595 Acked 9594 Read 1 New Done Done false
8596 Acked 9595 Read 10 New Done Done false
8597 Acked 9596 Write 5 New Done Done false
8598 Acked 9597 Flush 0 New Done Done false
8599 Acked 9598 Write 6 New Done Done false
8600 Acked 9599 Read 4 New Done Done false
8601 Acked 9600 Write 10 New Done Done false
8602 Acked 9601 Read 9 New Done Done false
8603 Acked 9602 Read 1 New Done Done false
8604 Acked 9603 Write 8 New Done Done false
8605 Acked 9604 Write 5 New Done Done false
8606 Acked 9605 Read 9 New Done Done false
8607 Acked 9606 Write 4 New Done Done false
8608 Acked 9607 Write 8 New Done Done false
8609 Acked 9608 Read 10 New Done Done false
8610 Acked 9609 Read 2 New Done Done false
8611 Acked 9610 Write 9 New Done Done false
8612 Acked 9611 Read 9 New Done Done false
8613 Acked 9612 Read 10 New Done Done false
8614 Acked 9613 Flush 0 New Done Done false
8615 Acked 9614 Read 2 New Done Done false
8616 Acked 9615 Read 5 New Done Done false
8617 Acked 9616 Read 7 New Done Done false
8618 Acked 9617 Write 4 New Done Done false
8619 Acked 9618 Write 2 New Done Done false
8620 Acked 9619 Write 4 New Done Done false
8621 Acked 9620 Write 4 New Done Done false
8622 Acked 9621 Read 4 New Done Done false
8623 Acked 9622 Write 6 New Done Done false
8624 Acked 9623 Read 2 New Done Done false
8625 Acked 9624 Read 4 New Done Done false
8626 Acked 9625 Flush 0 New Done Done false
8627 Acked 9626 Read 9 New Done Done false
8628 Acked 9627 Write 5 New Done Done false
8629 Acked 9628 Read 7 New Done Done false
8630 Acked 9629 Write 3 New Done Done false
8631 Acked 9630 Write 1 New Done Done false
8632 Acked 9631 Read 7 New Done Done false
8633 Acked 9632 Read 9 New Done Done false
8634 Acked 9633 Write 10 New Done Done false
8635 Acked 9634 Write 3 New Done Done false
8636 Acked 9635 Write 2 New Done Done false
8637 Acked 9636 Write 10 New Done Done false
8638 Acked 9637 Read 8 New Done Done false
8639 Acked 9638 Read 8 New Done Done false
8640 Acked 9639 Write 9 New Done Done false
8641 Acked 9640 Read 9 New Done Done false
8642 Acked 9641 Read 8 New Done Done false
8643 Acked 9642 Write 5 New Done Done false
8644 Acked 9643 Read 7 New Done Done false
8645 Acked 9644 Read 6 New Done Done false
8646 Acked 9645 Write 1 New Done Done false
8647 Acked 9646 Read 4 New Done Done false
8648 Acked 9647 Write 10 New Done Done false
8649 Acked 9648 Read 9 New Done Done false
8650 Acked 9649 Flush 0 New Done Done false
8651 Acked 9650 Read 3 New Done Done false
8652 Acked 9651 Write 8 New Done Done false
8653 Acked 9652 Write 10 New Done Done false
8654 Acked 9653 Write 10 New Done Done false
8655 Acked 9654 Flush 0 New Done Done false
8656 Acked 9655 Write 6 New Done Done false
8657 Acked 9656 Read 8 New Done Done false
8658 Acked 9657 Read 9 New Done Done false
8659 Acked 9658 Read 8 New Done Done false
8660 Acked 9659 Read 1 New Done Done false
8661 Acked 9660 Flush 0 New Done Done false
8662 Acked 9661 Read 6 New Done Done false
8663 Acked 9662 Write 8 New Done Done false
8664 Acked 9663 Write 5 New Done Done false
8665 Acked 9664 Flush 0 New Done Done false
8666 Acked 9665 Write 6 New Done Done false
8667 Acked 9666 Write 3 New Done Done false
8668 Acked 9667 Flush 0 New Done Done false
8669 Acked 9668 Read 9 New Done Done false
8670 Acked 9669 Flush 0 New Done Done false
8671 Acked 9670 Read 7 New Done Done false
8672 Acked 9671 Write 6 New Done Done false
8673 Acked 9672 Read 4 New Done Done false
8674 Acked 9673 Read 7 New Done Done false
8675 Acked 9674 Read 6 New Done Done false
8676 Acked 9675 Write 8 New Done Done false
8677 Acked 9676 Read 8 New Done Done false
8678 Acked 9677 Read 3 New Done Done false
8679 Acked 9678 Read 1 New Done Done false
8680 Acked 9679 Write 1 New Done Done false
8681 Acked 9680 Write 1 New Done Done false
8682 Acked 9681 Write 2 New Done Done false
8683 Acked 9682 Read 9 New Done Done false
8684 Acked 9683 Write 5 New Done Done false
8685 Acked 9684 Read 5 New Done Done false
8686 Acked 9685 Write 4 New Done Done false
8687 Acked 9686 Write 1 New Done Done false
8688 Acked 9687 Write 3 New Done Done false
8689 Acked 9688 Write 1 New Done Done false
8690 Acked 9689 Read 5 New Done Done false
8691 Acked 9690 Read 3 New Done Done false
8692 Acked 9691 Write 8 New Done Done false
8693 Acked 9692 Write 9 New Done Done false
8694 Acked 9693 Read 8 New Done Done false
8695 Acked 9694 Write 7 New Done Done false
8696 Acked 9695 Read 10 New Done Done false
8697 Acked 9696 Read 8 New Done Done false
8698 Acked 9697 Write 10 New Done Done false
8699 Acked 9698 Read 7 New Done Done false
8700 Acked 9699 Write 9 New Done Done false
8701 Acked 9700 Write 4 New Done Done false
8702 Acked 9701 Read 4 New Done Done false
8703 Acked 9702 Read 10 New Done Done false
8704 Acked 9703 Flush 0 New Done Done false
8705 Acked 9704 Read 9 New Done Done false
8706 Acked 9705 Read 7 New Done Done false
8707 Acked 9706 Read 10 New Done Done false
8708 Acked 9707 Write 9 New Done Done false
8709 Acked 9708 Write 3 New Done Done false
8710 Acked 9709 Flush 0 New Done Done false
8711 Acked 9710 Write 8 New Done Done false
8712 Acked 9711 Write 2 New Done Done false
8713 Acked 9712 Write 5 New Done Done false
8714 Acked 9713 Write 3 New Done Done false
8715 Acked 9714 Flush 0 New Done Done false
8716 Acked 9715 Read 1 New Done Done false
8717 Acked 9716 Read 8 New Done Done false
8718 Acked 9717 Flush 0 New Done Done false
8719 Acked 9718 Write 8 New Done Done false
8720 Acked 9719 Flush 0 New Done Done false
8721 Acked 9720 Write 1 New Done Done false
8722 Acked 9721 Write 8 New Done Done false
8723 Acked 9722 Write 2 New Done Done false
8724 Acked 9723 Write 1 New Done Done false
8725 Acked 9724 Read 7 New Done Done false
8726 Acked 9725 Write 5 New Done Done false
8727 Acked 9726 Read 2 New Done Done false
8728 Acked 9727 Write 5 New Done Done false
8729 Acked 9728 Write 9 New Done Done false
8730 Acked 9729 Write 2 New Done Done false
8731 Acked 9730 Read 2 New Done Done false
8732 Acked 9731 Flush 0 New Done Done false
8733 Acked 9732 Write 4 New Done Done false
8734 Acked 9733 Write 8 New Done Done false
8735 Acked 9734 Write 7 New Done Done false
8736 Acked 9735 Read 4 New Done Done false
8737 Acked 9736 Write 4 New Done Done false
8738 Acked 9737 Write 5 New Done Done false
8739 Acked 9738 Write 3 New Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
New 300 0 0 300
Sent 0 0 0 0
Done 0 300 300 600
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 9438 9731 9731
Downstairs last five completed: 9438 9437 9436 9435 9434
Upstairs last five completed: 8739 8738 8737 8736 8735
CLIENT: Up:0 ds:300 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:00.196774986Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:00.196841863Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:00.196867452Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:00.196888261Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:00.196903901Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 9438","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:00.19692116Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:00.197043565Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:01.198235178Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:01.198293476Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:01.198323565Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:01.198344974Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:01.198361263Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 9438","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:01.198379463Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:01.198480299Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:02.199511338Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:02.199560016Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:02.199581906Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:02.199602495Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:02.199618394Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 9438","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:02.199636133Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:02.199772828Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:03.200446071Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:03.200531168Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:03.200572137Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:03.200611265Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:03.200639524Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 9438","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:03.200668553Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:03.200771009Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1193)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
8440 Acked 9439 Write 2 New Done Done true
8441 Acked 9440 Write 4 New Done Done true
8442 Acked 9441 Read 5 New Done Done true
8443 Acked 9442 Read 9 New Done Done true
8444 Acked 9443 Write 10 New Done Done true
8445 Acked 9444 Read 1 New Done Done true
8446 Acked 9445 Read 3 New Done Done true
8447 Acked 9446 Read 3 New Done Done true
8448 Acked 9447 Read 6 New Done Done true
8449 Acked 9448 Write 1 New Done Done true
8450 Acked 9449 Write 3 New Done Done true
8451 Acked 9450 Write 6 New Done Done true
8452 Acked 9451 Read 10 New Done Done true
8453 Acked 9452 Flush 0 New Done Done true
8454 Acked 9453 Read 8 New Done Done true
8455 Acked 9454 Write 5 New Done Done true
8456 Acked 9455 Read 4 New Done Done true
8457 Acked 9456 Read 3 New Done Done true
8458 Acked 9457 Write 6 New Done Done true
8459 Acked 9458 Write 10 New Done Done true
8460 Acked 9459 Read 2 New Done Done true
8461 Acked 9460 Write 9 New Done Done true
8462 Acked 9461 Read 4 New Done Done true
8463 Acked 9462 Read 6 New Done Done true
8464 Acked 9463 Read 9 New Done Done true
8465 Acked 9464 Flush 0 New Done Done true
8466 Acked 9465 Read 8 New Done Done true
8467 Acked 9466 Read 5 New Done Done true
8468 Acked 9467 Read 9 New Done Done true
8469 Acked 9468 Flush 0 New Done Done true
8470 Acked 9469 Read 4 New Done Done true
8471 Acked 9470 Write 5 New Done Done true
8472 Acked 9471 Write 8 New Done Done true
8473 Acked 9472 Read 9 New Done Done true
8474 Acked 9473 Read 1 New Done Done true
8475 Acked 9474 Read 7 New Done Done true
8476 Acked 9475 Read 10 New Done Done true
8477 Acked 9476 Flush 0 New Done Done true
8478 Acked 9477 Read 2 New Done Done true
8479 Acked 9478 Read 8 New Done Done true
8480 Acked 9479 Flush 0 New Done Done true
8481 Acked 9480 Read 6 New Done Done true
8482 Acked 9481 Write 3 New Done Done true
8483 Acked 9482 Read 7 New Done Done true
8484 Acked 9483 Read 10 New Done Done true
8485 Acked 9484 Read 9 New Done Done true
8486 Acked 9485 Read 6 New Done Done true
8487 Acked 9486 Write 6 New Done Done true
8488 Acked 9487 Write 1 New Done Done true
8489 Acked 9488 Write 5 New Done Done true
8490 Acked 9489 Read 10 New Done Done true
8491 Acked 9490 Read 5 New Done Done true
8492 Acked 9491 Write 4 New Done Done true
8493 Acked 9492 Write 7 New Done Done true
8494 Acked 9493 Read 3 New Done Done true
8495 Acked 9494 Write 9 New Done Done true
8496 Acked 9495 Flush 0 New Done Done true
8497 Acked 9496 Write 10 New Done Done true
8498 Acked 9497 Read 6 New Done Done true
8499 Acked 9498 Write 2 New Done Done true
8500 Acked 9499 Read 5 New Done Done true
8501 Acked 9500 Write 1 New Done Done true
8502 Acked 9501 Read 1 New Done Done true
8503 Acked 9502 Write 5 New Done Done true
8504 Acked 9503 Write 9 New Done Done true
8505 Acked 9504 Read 3 New Done Done true
8506 Acked 9505 Read 3 New Done Done true
8507 Acked 9506 Read 10 New Done Done true
8508 Acked 9507 Read 3 New Done Done true
8509 Acked 9508 Read 1 New Done Done true
8510 Acked 9509 Write 4 New Done Done true
8511 Acked 9510 Write 7 New Done Done true
8512 Acked 9511 Write 3 New Done Done true
8513 Acked 9512 Flush 0 New Done Done true
8514 Acked 9513 Read 8 New Done Done true
8515 Acked 9514 Read 2 New Done Done true
8516 Acked 9515 Read 8 New Done Done true
8517 Acked 9516 Write 2 New Done Done true
8518 Acked 9517 Read 2 New Done Done true
8519 Acked 9518 Read 5 New Done Done true
8520 Acked 9519 Write 4 New Done Done true
8521 Acked 9520 Read 2 New Done Done true
8522 Acked 9521 Read 6 New Done Done true
8523 Acked 9522 Read 2 New Done Done true
8524 Acked 9523 Read 6 New Done Done true
8525 Acked 9524 Write 9 New Done Done true
8526 Acked 9525 Flush 0 New Done Done true
8527 Acked 9526 Write 1 New Done Done true
8528 Acked 9527 Read 8 New Done Done true
8529 Acked 9528 Write 8 New Done Done true
8530 Acked 9529 Read 9 New Done Done true
8531 Acked 9530 Read 9 New Done Done true
8532 Acked 9531 Write 8 New Done Done true
8533 Acked 9532 Read 7 New Done Done true
8534 Acked 9533 Read 2 New Done Done true
8535 Acked 9534 Flush 0 New Done Done true
8536 Acked 9535 Write 3 New Done Done true
8537 Acked 9536 Write 7 New Done Done true
8538 Acked 9537 Flush 0 New Done Done true
8539 Acked 9538 Read 2 New Done Done true
8540 Acked 9539 Read 10 New Done Done true
8541 Acked 9540 Read 2 New Done Done true
8542 Acked 9541 Read 3 New Done Done true
8543 Acked 9542 Write 6 New Done Done true
8544 Acked 9543 Read 10 New Done Done true
8545 Acked 9544 Read 10 New Done Done true
8546 Acked 9545 Read 10 New Done Done true
8547 Acked 9546 Flush 0 New Done Done true
8548 Acked 9547 Read 5 New Done Done true
8549 Acked 9548 Write 1 New Done Done true
8550 Acked 9549 Flush 0 New Done Done true
8551 Acked 9550 Read 6 New Done Done true
8552 Acked 9551 Write 8 New Done Done true
8553 Acked 9552 Read 10 New Done Done true
8554 Acked 9553 Flush 0 New Done Done true
8555 Acked 9554 Flush 0 New Done Done true
8556 Acked 9555 Read 5 New Done Done true
8557 Acked 9556 Write 7 New Done Done true
8558 Acked 9557 Read 6 New Done Done true
8559 Acked 9558 Read 6 New Done Done true
8560 Acked 9559 Read 8 New Done Done true
8561 Acked 9560 Write 2 New Done Done true
8562 Acked 9561 Read 10 New Done Done true
8563 Acked 9562 Flush 0 New Done Done true
8564 Acked 9563 Write 9 New Done Done true
8565 Acked 9564 Write 7 New Done Done true
8566 Acked 9565 Write 3 New Done Done true
8567 Acked 9566 Read 3 New Done Done true
8568 Acked 9567 Write 9 New Done Done true
8569 Acked 9568 Flush 0 New Done Done true
8570 Acked 9569 Write 10 New Done Done true
8571 Acked 9570 Read 3 New Done Done true
8572 Acked 9571 Read 3 New Done Done true
8573 Acked 9572 Read 10 New Done Done true
8574 Acked 9573 Write 3 New Done Done true
8575 Acked 9574 Write 6 New Done Done true
8576 Acked 9575 Write 1 New Done Done true
8577 Acked 9576 Read 4 New Done Done true
8578 Acked 9577 Write 6 New Done Done true
8579 Acked 9578 Read 6 New Done Done true
8580 Acked 9579 Write 6 New Done Done true
8581 Acked 9580 Write 5 New Done Done true
8582 Acked 9581 Write 5 New Done Done true
8583 Acked 9582 Read 7 New Done Done true
8584 Acked 9583 Read 1 New Done Done true
8585 Acked 9584 Read 3 New Done Done true
8586 Acked 9585 Read 2 New Done Done true
8587 Acked 9586 Write 6 New Done Done true
8588 Acked 9587 Write 8 New Done Done true
8589 Acked 9588 Read 6 New Done Done true
8590 Acked 9589 Flush 0 New Done Done true
8591 Acked 9590 Read 10 New Done Done true
8592 Acked 9591 Read 3 New Done Done true
8593 Acked 9592 Read 4 New Done Done true
8594 Acked 9593 Write 7 New Done Done true
8595 Acked 9594 Read 1 New Done Done true
8596 Acked 9595 Read 10 New Done Done true
8597 Acked 9596 Write 5 New Done Done true
8598 Acked 9597 Flush 0 New Done Done true
8599 Acked 9598 Write 6 New Done Done true
8600 Acked 9599 Read 4 New Done Done true
8601 Acked 9600 Write 10 New Done Done true
8602 Acked 9601 Read 9 New Done Done true
8603 Acked 9602 Read 1 New Done Done true
8604 Acked 9603 Write 8 New Done Done true
8605 Acked 9604 Write 5 New Done Done true
8606 Acked 9605 Read 9 New Done Done true
8607 Acked 9606 Write 4 New Done Done true
8608 Acked 9607 Write 8 New Done Done true
8609 Acked 9608 Read 10 New Done Done true
8610 Acked 9609 Read 2 New Done Done true
8611 Acked 9610 Write 9 New Done Done true
8612 Acked 9611 Read 9 New Done Done true
8613 Acked 9612 Read 10 New Done Done true
8614 Acked 9613 Flush 0 New Done Done true
8615 Acked 9614 Read 2 New Done Done true
8616 Acked 9615 Read 5 New Done Done true
8617 Acked 9616 Read 7 New Done Done true
8618 Acked 9617 Write 4 New Done Done true
8619 Acked 9618 Write 2 New Done Done true
8620 Acked 9619 Write 4 New Done Done true
8621 Acked 9620 Write 4 New Done Done true
8622 Acked 9621 Read 4 New Done Done true
8623 Acked 9622 Write 6 New Done Done true
8624 Acked 9623 Read 2 New Done Done true
8625 Acked 9624 Read 4 New Done Done true
8626 Acked 9625 Flush 0 New Done Done true
8627 Acked 9626 Read 9 New Done Done true
8628 Acked 9627 Write 5 New Done Done true
8629 Acked 9628 Read 7 New Done Done true
8630 Acked 9629 Write 3 New Done Done true
8631 Acked 9630 Write 1 New Done Done true
8632 Acked 9631 Read 7 New Done Done true
8633 Acked 9632 Read 9 New Done Done true
8634 Acked 9633 Write 10 New Done Done true
8635 Acked 9634 Write 3 New Done Done true
8636 Acked 9635 Write 2 New Done Done true
8637 Acked 9636 Write 10 New Done Done true
8638 Acked 9637 Read 8 New Done Done true
8639 Acked 9638 Read 8 New Done Done true
8640 Acked 9639 Write 9 New Done Done true
8641 Acked 9640 Read 9 New Done Done true
8642 Acked 9641 Read 8 New Done Done true
8643 Acked 9642 Write 5 New Done Done true
8644 Acked 9643 Read 7 New Done Done true
8645 Acked 9644 Read 6 New Done Done true
8646 Acked 9645 Write 1 New Done Done true
8647 Acked 9646 Read 4 New Done Done true
8648 Acked 9647 Write 10 New Done Done true
8649 Acked 9648 Read 9 New Done Done true
8650 Acked 9649 Flush 0 New Done Done true
8651 Acked 9650 Read 3 New Done Done true
8652 Acked 9651 Write 8 New Done Done true
8653 Acked 9652 Write 10 New Done Done true
8654 Acked 9653 Write 10 New Done Done true
8655 Acked 9654 Flush 0 New Done Done true
8656 Acked 9655 Write 6 New Done Done true
8657 Acked 9656 Read 8 New Done Done true
8658 Acked 9657 Read 9 New Done Done true
8659 Acked 9658 Read 8 New Done Done true
8660 Acked 9659 Read 1 New Done Done true
8661 Acked 9660 Flush 0 New Done Done true
8662 Acked 9661 Read 6 New Done Done true
8663 Acked 9662 Write 8 New Done Done true
8664 Acked 9663 Write 5 New Done Done true
8665 Acked 9664 Flush 0 New Done Done true
8666 Acked 9665 Write 6 New Done Done true
8667 Acked 9666 Write 3 New Done Done true
8668 Acked 9667 Flush 0 New Done Done true
8669 Acked 9668 Read 9 New Done Done true
8670 Acked 9669 Flush 0 New Done Done true
8671 Acked 9670 Read 7 New Done Done true
8672 Acked 9671 Write 6 New Done Done true
8673 Acked 9672 Read 4 New Done Done true
8674 Acked 9673 Read 7 New Done Done true
8675 Acked 9674 Read 6 New Done Done true
8676 Acked 9675 Write 8 New Done Done true
8677 Acked 9676 Read 8 New Done Done true
8678 Acked 9677 Read 3 New Done Done true
8679 Acked 9678 Read 1 New Done Done true
8680 Acked 9679 Write 1 New Done Done true
8681 Acked 9680 Write 1 New Done Done true
8682 Acked 9681 Write 2 New Done Done true
8683 Acked 9682 Read 9 New Done Done true
8684 Acked 9683 Write 5 New Done Done true
8685 Acked 9684 Read 5 New Done Done true
8686 Acked 9685 Write 4 New Done Done true
8687 Acked 9686 Write 1 New Done Done true
8688 Acked 9687 Write 3 New Done Done true
8689 Acked 9688 Write 1 New Done Done true
8690 Acked 9689 Read 5 New Done Done true
8691 Acked 9690 Read 3 New Done Done true
8692 Acked 9691 Write 8 New Done Done true
8693 Acked 9692 Write 9 New Done Done true
8694 Acked 9693 Read 8 New Done Done true
8695 Acked 9694 Write 7 New Done Done true
8696 Acked 9695 Read 10 New Done Done true
8697 Acked 9696 Read 8 New Done Done true
8698 Acked 9697 Write 10 New Done Done true
8699 Acked 9698 Read 7 New Done Done true
8700 Acked 9699 Write 9 New Done Done true
8701 Acked 9700 Write 4 New Done Done true
8702 Acked 9701 Read 4 New Done Done true
8703 Acked 9702 Read 10 New Done Done true
8704 Acked 9703 Flush 0 New Done Done true
8705 Acked 9704 Read 9 New Done Done true
8706 Acked 9705 Read 7 New Done Done true
8707 Acked 9706 Read 10 New Done Done true
8708 Acked 9707 Write 9 New Done Done true
8709 Acked 9708 Write 3 New Done Done true
8710 Acked 9709 Flush 0 New Done Done true
8711 Acked 9710 Write 8 New Done Done true
8712 Acked 9711 Write 2 New Done Done true
8713 Acked 9712 Write 5 New Done Done true
8714 Acked 9713 Write 3 New Done Done true
8715 Acked 9714 Flush 0 New Done Done true
8716 Acked 9715 Read 1 New Done Done true
8717 Acked 9716 Read 8 New Done Done true
8718 Acked 9717 Flush 0 New Done Done true
8719 Acked 9718 Write 8 New Done Done true
8720 Acked 9719 Flush 0 New Done Done true
8721 Acked 9720 Write 1 New Done Done true
8722 Acked 9721 Write 8 New Done Done true
8723 Acked 9722 Write 2 New Done Done true
8724 Acked 9723 Write 1 New Done Done true
8725 Acked 9724 Read 7 New Done Done true
8726 Acked 9725 Write 5 New Done Done true
8727 Acked 9726 Read 2 New Done Done true
8728 Acked 9727 Write 5 New Done Done true
8729 Acked 9728 Write 9 New Done Done true
8730 Acked 9729 Write 2 New Done Done true
8731 Acked 9730 Read 2 New Done Done true
8732 Acked 9731 Flush 0 New Done Done true
8733 Acked 9732 Write 4 New Done Done true
8734 Acked 9733 Write 8 New Done Done true
8735 Acked 9734 Write 7 New Done Done true
8736 Acked 9735 Read 4 New Done Done true
8737 Acked 9736 Write 4 New Done Done true
8738 Acked 9737 Write 5 New Done Done true
8739 Acked 9738 Write 3 New Done Done true
8740 Acked 9739 Flush 0 New Done Done true
STATES DS:0 DS:1 DS:2 TOTAL
New 301 0 0 301
Sent 0 0 0 0
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 9438 9739 9739
Downstairs last five completed: 9438 9437 9436 9435 9434
Upstairs last five completed: 8740 8739 8738 8737 8736
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:04.201454441Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:04.201535788Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:04.202953804Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:04.202978423Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 9438","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:04.202993322Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 9438","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:04.203160346Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:04.203195984Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:04.203224173Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:04.203246812Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:04.203265252Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 9739 9738 9737 9736 9735
Upstairs last five completed: 8740 8739 8738 8737 8736
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Main task has work to do, go find it
got dsc Stop(0)
stop 0
Feb 13 21:45:07.542 INFO request completed, latency_us:[8810] Got stop action so:false kr:false
352, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 7af383ef-a2b8-4caf-aa16-ea33533d3f95, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:07.542683538Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:07.542749506Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:07.542784014Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:07.542798584Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 9739","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:07.542877391Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:07.54290829Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:45:07.542 INFO request completed, latency_us: 157, response_code: 200, uri: /state/cid/0, method: GET, req_id: fee0d93a-8771-4358-8266-cd128af69a72, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:08.543862651Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:08.543927399Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:08.543951338Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:08.543972867Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:08.543989366Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 9739","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:08.544006746Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:08.544032035Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:09.544666148Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:09.544732196Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:09.544755085Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:09.544775814Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:09.544791864Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 9739","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:09.544809553Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:09.544826702Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:10.545778405Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:10.545837343Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:10.545859222Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:10.545881031Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:10.54589714Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 0 jobs since flush 9739","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:10.545920669Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:10.545938309Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
Feb 13 21:45:11.544 INFO request completed, latency_us: 214, response_code: 200, uri: /state/cid/0, method: GET, req_id: fa1788dd-0d50-4f80-990a-d119f65b48b5, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
001/300 Write block 4636 len 36864 data: 7 7 6 6 5 6 7 5 5
002/300 Read block 4948 len 24576
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:11.546377911Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:11.546450078Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:11.546484447Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:11.546513305Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:11.546527335Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 2 jobs since flush 9739","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:11.546541364Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:11.546572143Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
003/300 Read block 4607 len 32768
004/300 Read block 670 len 28672
005/300 Write block 2428 len 16384 data: 3 5 6 6
006/300 Write block 4385 len 4096 data: 9
007/300 Write block 1239 len 8192 data: 4 5
008/300 Flush
009/300 Read block 1177 len 28672
010/300 Write block 4967 len 40960 data: 4 3 3 3 4 5 5 5 5 5
011/300 Read block 1253 len 20480
012/300 Read block 3521 len 24576
013/300 Write block 113 len 24576 data: 3 4 4 4 5 6
014/300 Read block 4768 len 24576
015/300 Read block 338 len 16384
016/300 Write block 2437 len 20480 data: 6 7 8 9 12
017/300 Write block 76 len 36864 data: 7 6 6 4 3 3 3 5 7
018/300 Write block 758 len 16384 data: 4 4 3 5
019/300 Read block 3268 len 16384
020/300 Write block 4659 len 8192 data: 7 8
021/300 Read block 1756 len 36864
022/300 Read block 2151 len 40960
023/300 Flush
024/300 Flush
025/300 Read block 4448 len 24576
026/300 Write block 2113 len 8192 data: 6 8
027/300 Write block 4885 len 24576 data: 3 3 3 5 6 5
028/300 Read block 240 len 32768
029/300 Read block 2972 len 12288
030/300 Read block 4897 len 12288
031/300 Flush
032/300 Read block 1265 len 36864
033/300 Write block 421 len 32768 data: 3 3 4 3 4 4 5 7
034/300 Write block 2249 len 40960 data: 8 8 6 6 7 8 8 9 10 9
035/300 Read block 4979 len 28672
036/300 Flush
037/300 Write block 2528 len 20480 data: 8 10 9 6 6
038/300 Read block 128 len 4096
039/300 Write block 3872 len 28672 data: 4 2 2 2 2 2 3
040/300 Read block 439 len 40960
041/300 Write block 1237 len 24576 data: 5 5 5 6 4 3
042/300 Read block 334 len 40960
043/300 Read block 2613 len 40960
044/300 Read block 3942 len 24576
045/300 Write block 609 len 16384 data: 3 3 3 3
046/300 Write block 32 len 12288 data: 6 4 5
047/300 Read block 4154 len 8192
048/300 Read block 2890 len 32768
049/300 Read block 2013 len 20480
050/300 Write block 2851 len 4096 data: 6
051/300 Read block 267 len 8192
052/300 Write block 3522 len 36864 data: 7 6 4 4 4 5 5 6 8
053/300 Read block 4574 len 4096
054/300 Read block 1656 len 4096
055/300 Write block 2623 len 40960 data: 4 4 6 7 8 9 8 7 6 7
056/300 Flush
057/300 Read block 4904 len 12288
058/300 Read block 3613 len 8192
059/300 Read block 2104 len 24576
060/300 Write block 3917 len 8192 data: 7 7
061/300 Read block 617 len 4096
062/300 Read block 2928 len 20480
063/300 Read block 4152 len 4096
064/300 Read block 1297 len 40960
065/300 Write block 2603 len 36864 data: 3 3 4 5 5 5 6 6 7
066/300 Flush
067/300 Flush
068/300 Flush
069/300 Read block 4334 len 4096
070/300 Read block 3834 len 40960
071/300 Read block 3400 len 16384
072/300 Write block 1663 len 28672 data: 6 5 6 5 3 5 5
073/300 Write block 1914 len 32768 data: 6 5 6 6 7 7 8 8
074/300 Write block 4853 len 36864 data: 4 4 5 6 6 5 5 7 8
075/300 Read block 2871 len 24576
076/300 Write block 483 len 32768 data: 2 2 3 2 2 3 4 4
077/300 Read block 2740 len 12288
078/300 Write block 2821 len 32768 data: 7 7 7 7 7 6 7 8
079/300 Write block 2910 len 24576 data: 5 6 6 8 10 11
080/300 Read block 1462 len 12288
081/300 Read block 1236 len 28672
082/300 Write block 2505 len 28672 data: 10 11 10 10 10 8 7
083/300 Flush
084/300 Read block 1706 len 4096
085/300 Write block 619 len 32768 data: 4 6 9 6 6 5 4 4
086/300 Read block 985 len 32768
087/300 Read block 2673 len 24576
088/300 Flush
089/300 Read block 4349 len 32768
090/300 Read block 4452 len 28672
091/300 Write block 3378 len 40960 data: 3 6 6 5 4 4 5 8 5 6
092/300 Read block 3434 len 28672
093/300 Read block 4385 len 16384
094/300 Write block 4726 len 32768 data: 12 11 9 9 9 8 7 8
095/300 Write block 4335 len 36864 data: 10 8 7 6 5 5 5 5 5
096/300 Read block 2537 len 32768
097/300 Flush
098/300 Write block 1920 len 36864 data: 9 9 8 8 8 5 5 5 4
099/300 Read block 2292 len 16384
100/300 Write block 4715 len 24576 data: 8 7 7 6 5 7
101/300 Write block 3183 len 20480 data: 6 6 6 5 5
102/300 Write block 4140 len 8192 data: 3 3
103/300 Write block 47 len 16384 data: 6 8 7 9
104/300 Read block 589 len 8192
105/300 Read block 684 len 4096
106/300 Write block 2152 len 12288 data: 4 5 5
107/300 Read block 208 len 32768
108/300 Read block 4229 len 8192
109/300 Write block 2809 len 4096 data: 6
110/300 Read block 4718 len 28672
111/300 Write block 4067 len 8192 data: 6 6
112/300 Flush
113/300 Read block 73 len 4096
114/300 Read block 4739 len 8192
115/300 Read block 741 len 36864
116/300 Flush
117/300 Read block 4526 len 32768
118/300 Write block 2746 len 16384 data: 10 8 6 5
119/300 Read block 3589 len 24576
120/300 Flush
121/300 Read block 3827 len 36864
122/300 Read block 2395 len 12288
123/300 Read block 3347 len 32768
124/300 Read block 4127 len 20480
125/300 Write block 4799 len 24576 data: 7 8 8 8 9 4
126/300 Read block 1809 len 24576
127/300 Read block 4014 len 4096
128/300 Write block 4306 len 16384 data: 6 6 8 8
129/300 Read block 174 len 28672
130/300 Read block 2406 len 32768
131/300 Read block 2029 len 16384
132/300 Write block 495 len 40960 data: 8 5 6 6 7 5 5 4 5 4
133/300 Read block 2198 len 24576
134/300 Flush
135/300 Read block 1417 len 32768
136/300 Read block 1931 len 16384
137/300 Read block 836 len 16384
138/300 Read block 818 len 40960
139/300 Read block 4890 len 32768
140/300 Read block 73 len 36864
141/300 Write block 3706 len 12288 data: 3 3 3
142/300 Write block 3029 len 12288 data: 8 7 6
143/300 Write block 3226 len 32768 data: 11 11 11 7 6 6 5 5
144/300 Read block 4011 len 8192
145/300 Write block 1377 len 8192 data: 4 3
146/300 Read block 3673 len 12288
147/300 Write block 2438 len 24576 data: 8 9 10 13 13 10
148/300 Write block 4749 len 16384 data: 5 4 4 4
149/300 Write block 392 len 20480 data: 4 4 4 5 6
150/300 Read block 2641 len 8192
151/300 Write block 4183 len 12288 data: 4 5 5
152/300 Write block 3969 len 8192 data: 4 4
153/300 Read block 4689 len 28672
154/300 Read block 569 len 12288
155/300 Read block 3732 len 28672
156/300 Flush
157/300 Read block 665 len 8192
158/300 Write block 1980 len 8192 data: 3 4
159/300 Read block 3368 len 20480
160/300 Read block 683 len 36864
161/300 Read block 2078 len 32768
162/300 Write block 1516 len 20480 data: 4 4 2 3 3
163/300 Flush
164/300 Write block 1430 len 32768 data: 5 6 7 8 9 9 8 8
165/300 Read block 848 len 8192
166/300 Read block 2427 len 12288
167/300 Flush
168/300 Read block 520 len 4096
169/300 Write block 2257 len 36864 data: 11 10 7 7 8 7 6 4 4
170/300 Write block 2156 len 12288 data: 4 5 5
171/300 Read block 4942 len 28672
172/300 Flush
173/300 Read block 4473 len 32768
174/300 Read block 843 len 32768
175/300 Read block 4858 len 40960
176/300 Write block 256 len 28672 data: 5 5 5 4 3 3 3
177/300 Write block 3024 len 36864 data: 9 8 8 10 10 9 8 7 7
178/300 Flush
179/300 Read block 771 len 12288
180/300 Read block 3614 len 36864
181/300 Write block 2802 len 4096 data: 5
182/300 Write block 2120 len 4096 data: 4
183/300 Write block 3100 len 12288 data: 5 7 6
184/300 Read block 860 len 16384
185/300 Read block 2009 len 36864
186/300 Write block 4159 len 20480 data: 7 7 8 7 5
187/300 Write block 4669 len 24576 data: 10 9 10 8 6 6
188/300 Read block 1078 len 16384
189/300 Write block 1148 len 16384 data: 7 7 7 6
190/300 Write block 3731 len 16384 data: 6 6 6 6
191/300 Read block 4670 len 36864
192/300 Write block 3137 len 32768 data: 7 7 8 8 8 7 7 8
193/300 Read block 3922 len 16384
194/300 Read block 3569 len 40960
195/300 Write block 3316 len 20480 data: 5 6 6 5 7
196/300 Write block 4989 len 20480 data: 5 5 5 6 6
197/300 Write block 924 len 16384 data: 8 6 6 6
198/300 Write block 2924 len 24576 data: 3 5 7 7 7 7
199/300 Read block 2226 len 4096
200/300 Read block 4711 len 32768
201/300 Read block 3261 len 20480
202/300 Write block 3523 len 4096 data: 7
203/300 Write block 1673 len 12288 data: 5 5 5
204/300 Read block 4091 len 4096
205/300 Read block 2631 len 20480
206/300 Write block 1049 len 24576 data: 3 5 5 5 7 7
207/300 Write block 1404 len 36864 data: 6 7 7 6 7 8 8 8 8
208/300 Read block 4099 len 16384
209/300 Write block 1786 len 28672 data: 2 3 3 4 4 5 5
210/300 Write block 2876 len 8192 data: 5 4
211/300 Read block 894 len 28672
212/300 Read block 4551 len 32768
213/300 Read block 3473 len 32768
214/300 Write block 2812 len 4096 data: 4
215/300 Read block 2015 len 8192
216/300 Write block 2271 len 20480 data: 5 6 5 4 5
217/300 Read block 3145 len 24576
218/300 Read block 3319 len 8192
219/300 Read block 1286 len 8192
220/300 Flush
221/300 Read block 2388 len 8192
222/300 Write block 1755 len 36864 data: 3 3 4 3 3 5 7 6 8
223/300 Write block 1510 len 4096 data: 3
224/300 Read block 3788 len 12288
225/300 Write block 4261 len 32768 data: 4 4 7 7 5 5 5 6
226/300 Write block 4538 len 20480 data: 8 6 5 5 6
227/300 Read block 4027 len 12288
228/300 Read block 4126 len 12288
229/300 Write block 895 len 8192 data: 8 7
230/300 Read block 4943 len 20480
231/300 Write block 4657 len 8192 data: 5 6
232/300 Read block 4506 len 28672
233/300 Write block 3309 len 32768 data: 9 9 9 7 7 7 5 6
234/300 Write block 1335 len 16384 data: 4 3 3 2
235/300 Write block 3992 len 28672 data: 6 5 5 5 5 4 5
236/300 Write block 3096 len 32768 data: 5 7 5 5 6 8 7 6
237/300 Flush
238/300 Read block 1619 len 20480
239/300 Read block 2673 len 20480
240/300 Write block 2836 len 20480 data: 8 9 8 7 7
241/300 Write block 2545 len 36864 data: 4 4 5 6 7 6 7 6 6
242/300 Read block 2438 len 4096
243/300 Read block 3387 len 8192
244/300 Read block 1034 len 12288
245/300 Read block 2617 len 36864
246/300 Write block 2094 len 12288 data: 3 4 5
247/300 Write block 689 len 32768 data: 9 8 7 7 6 5 5 5
248/300 Write block 4363 len 8192 data: 9 6
249/300 Flush
250/300 Read block 37 len 8192
251/300 Flush
252/300 Read block 4153 len 16384
253/300 Flush
254/300 Read block 1081 len 8192
255/300 Write block 1556 len 36864 data: 6 5 4 4 4 4 4 5 5
256/300 Read block 2061 len 16384
257/300 Read block 3610 len 36864
258/300 Read block 3908 len 4096
259/300 Write block 56 len 24576 data: 7 9 11 10 14 16
260/300 Read block 2977 len 4096
261/300 Write block 131 len 20480 data: 11 10 8 6 5
262/300 Flush
263/300 Write block 929 len 40960 data: 5 3 4 5 5 5 6 6 6 5
264/300 Write block 3626 len 28672 data: 7 6 6 5 4 4 4
265/300 Read block 1132 len 40960
266/300 Read block 4401 len 16384
267/300 Write block 4930 len 40960 data: 5 5 6 6 5 5 4 5 5 7
268/300 Read block 7 len 20480
269/300 Write block 2068 len 40960 data: 6 6 7 7 6 5 8 9 9 8
270/300 Read block 4504 len 28672
271/300 Write block 2735 len 8192 data: 6 8
272/300 Flush
273/300 Write block 3279 len 40960 data: 7 7 7 6 9 9 9 9 8 7
274/300 Read block 2829 len 28672
275/300 Write block 4115 len 24576 data: 9 9 9 8 8 4
276/300 Write block 1671 len 28672 data: 6 5 6 6 6 5 5
277/300 Read block 4761 len 32768
278/300 Read block 38 len 8192
279/300 Flush
280/300 Read block 1546 len 40960
281/300 Read block 1850 len 28672
282/300 Read block 3875 len 40960
283/300 Read block 890 len 8192
284/300 Read block 3421 len 16384
285/300 Read block 3210 len 20480
286/300 Flush
287/300 Read block 2247 len 32768
288/300 Read block 4179 len 8192
289/300 Read block 2371 len 4096
290/300 Flush
291/300 Read block 2763 len 32768
292/300 Read block 3264 len 36864
293/300 Write block 4212 len 28672 data: 6 7 7 6 6 6 8
294/300 Write block 312 len 20480 data: 7 5 5 3 2
295/300 Write block 3724 len 24576 data: 4 5 5 5 6 5
296/300 Read block 1232 len 36864
297/300 Read block 3726 len 36864
298/300 Flush
299/300 Read block 2842 len 28672
300/300 Read block 4652 len 12288
Main task has work to do, go find it
got dsc Start(0)
start 0
Feb 13 21:45:11.881 INFO request completed, latency_us: 208, response_code: 204, uri: /start/cid/0, method: POST, req_id: ae883754-e7d8-4c47-9447-cbcce75215f6, remote_addr: 127.0.0.1:52124, local_addr: 127.0.0.1:9998
Replay: started 0, returned:Ok(())
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
8741 Acked 9740 Write 9 New Done Done true
8742 Acked 9741 Read 6 New Done Done true
8743 Acked 9742 Read 8 New Done Done false
8744 Acked 9743 Read 7 New Done Done false
8745 Acked 9744 Write 4 New Done Done false
8746 Acked 9745 Write 1 New Done Done false
8747 Acked 9746 Write 2 New Done Done false
8748 Acked 9747 Flush 0 New Done Done false
8749 Acked 9748 Read 7 New Done Done false
8750 Acked 9749 Write 10 New Done Done false
8751 Acked 9750 Read 5 New Done Done false
8752 Acked 9751 Read 6 New Done Done false
8753 Acked 9752 Write 6 New Done Done false
8754 Acked 9753 Read 6 New Done Done false
8755 Acked 9754 Read 4 New Done Done false
8756 Acked 9755 Write 5 New Done Done false
8757 Acked 9756 Write 9 New Done Done false
8758 Acked 9757 Write 4 New Done Done false
8759 Acked 9758 Read 4 New Done Done false
8760 Acked 9759 Write 2 New Done Done false
8761 Acked 9760 Read 9 New Done Done false
8762 Acked 9761 Read 10 New Done Done false
8763 Acked 9762 Flush 0 New Done Done false
8764 Acked 9763 Flush 0 New Done Done false
8765 Acked 9764 Read 6 New Done Done false
8766 Acked 9765 Write 2 New Done Done false
8767 Acked 9766 Write 6 New Done Done false
8768 Acked 9767 Read 8 New Done Done false
8769 Acked 9768 Read 3 New Done Done false
8770 Acked 9769 Read 3 New Done Done false
8771 Acked 9770 Flush 0 New Done Done false
8772 Acked 9771 Read 9 New Done Done false
8773 Acked 9772 Write 8 New Done Done false
8774 Acked 9773 Write 10 New Done Done false
8775 Acked 9774 Read 7 New Done Done false
8776 Acked 9775 Flush 0 New Done Done false
8777 Acked 9776 Write 5 New Done Done false
8778 Acked 9777 Read 1 New Done Done false
8779 Acked 9778 Write 7 New Done Done false
8780 Acked 9779 Read 10 New Done Done false
8781 Acked 9780 Write 6 New Done Done false
8782 Acked 9781 Read 10 New Done Done false
8783 Acked 9782 Read 10 New Done Done false
8784 Acked 9783 Read 6 New Done Done false
8785 Acked 9784 Write 4 New Done Done false
8786 Acked 9785 Write 3 New Done Done false
8787 Acked 9786 Read 2 New Done Done false
8788 Acked 9787 Read 8 New Done Done false
8789 Acked 9788 Read 5 New Done Done false
8790 Acked 9789 Write 1 New Done Done false
8791 Acked 9790 Read 2 New Done Done false
8792 Acked 9791 Write 9 New Done Done false
8793 Acked 9792 Read 1 New Done Done false
8794 Acked 9793 Read 1 New Done Done false
8795 Acked 9794 Write 10 New Done Done false
8796 Acked 9795 Flush 0 New Done Done false
8797 Acked 9796 Read 3 New Done Done false
8798 Acked 9797 Read 2 New Done Done false
8799 Acked 9798 Read 6 New Done Done false
8800 Acked 9799 Write 2 New Done Done false
8801 Acked 9800 Read 1 New Done Done false
8802 Acked 9801 Read 5 New Done Done false
8803 Acked 9802 Read 1 New Done Done false
8804 Acked 9803 Read 10 New Done Done false
8805 Acked 9804 Write 9 New Done Done false
8806 Acked 9805 Flush 0 New Done Done false
8807 Acked 9806 Flush 0 New Done Done false
8808 Acked 9807 Flush 0 New Done Done false
8809 Acked 9808 Read 1 New Done Done false
8810 Acked 9809 Read 10 New Done Done false
8811 Acked 9810 Read 4 New Done Done false
8812 Acked 9811 Write 7 New Done Done false
8813 Acked 9812 Write 8 New Done Done false
8814 Acked 9813 Write 9 New Done Done false
8815 Acked 9814 Read 6 New Done Done false
8816 Acked 9815 Write 8 New Done Done false
8817 Acked 9816 Read 3 New Done Done false
8818 Acked 9817 Write 8 New Done Done false
8819 Acked 9818 Write 6 New Done Done false
8820 Acked 9819 Read 3 New Done Done false
8821 Acked 9820 Read 7 New Done Done false
8822 Acked 9821 Write 7 New Done Done false
8823 Acked 9822 Flush 0 New Done Done false
8824 Acked 9823 Read 1 New Done Done false
8825 Acked 9824 Write 8 New Done Done false
8826 Acked 9825 Read 8 New Done Done false
8827 Acked 9826 Read 6 New Done Done false
8828 Acked 9827 Flush 0 New Done Done false
8829 Acked 9828 Read 8 New Done Done false
8830 Acked 9829 Read 7 New Done Done false
8831 Acked 9830 Write 10 New Done Done false
8832 Acked 9831 Read 7 New Done Done false
8833 Acked 9832 Read 4 New Done Done false
8834 Acked 9833 Write 8 New Done Done false
8835 Acked 9834 Write 9 New Done Done false
8836 Acked 9835 Read 8 New Done Done false
8837 Acked 9836 Flush 0 New Done Done false
8838 Acked 9837 Write 9 New Done Done false
8839 Acked 9838 Read 4 New Done Done false
8840 Acked 9839 Write 6 New Done Done false
8841 Acked 9840 Write 5 New Done Done false
8842 Acked 9841 Write 2 New Done Done false
8843 Acked 9842 Write 4 New Done Done false
8844 Acked 9843 Read 2 New Done Done false
8845 Acked 9844 Read 1 New Done Done false
8846 Acked 9845 Write 3 New Done Done false
8847 Acked 9846 Read 8 New Done Done false
8848 Acked 9847 Read 2 New Done Done false
8849 Acked 9848 Write 1 New Done Done false
8850 Acked 9849 Read 7 New Done Done false
8851 Acked 9850 Write 2 New Done Done false
8852 Acked 9851 Flush 0 New Done Done false
8853 Acked 9852 Read 1 New Done Done false
8854 Acked 9853 Read 2 New Done Done false
8855 Acked 9854 Read 9 New Done Done false
8856 Acked 9855 Flush 0 New Done Done false
8857 Acked 9856 Read 8 New Done Done false
8858 Acked 9857 Write 4 New Done Done false
8859 Acked 9858 Read 6 New Done Done false
8860 Acked 9859 Flush 0 New Done Done false
8861 Acked 9860 Read 9 New Done Done false
8862 Acked 9861 Read 3 New Done Done false
8863 Acked 9862 Read 8 New Done Done false
8864 Acked 9863 Read 5 New Done Done false
8865 Acked 9864 Write 6 New Done Done false
8866 Acked 9865 Read 6 New Done Done false
8867 Acked 9866 Read 1 New Done Done false
8868 Acked 9867 Write 4 New Done Done false
8869 Acked 9868 Read 7 New Done Done false
8870 Acked 9869 Read 8 New Done Done false
8871 Acked 9870 Read 4 New Done Done false
8872 Acked 9871 Write 10 New Done Done false
8873 Acked 9872 Read 6 New Done Done false
8874 Acked 9873 Flush 0 New Done Done false
8875 Acked 9874 Read 8 New Done Done false
8876 Acked 9875 Read 4 New Done Done false
8877 Acked 9876 Read 4 New Done Done false
8878 Acked 9877 Read 10 New Done Done false
8879 Acked 9878 Read 8 New Done Done false
8880 Acked 9879 Read 9 New Done Done false
8881 Acked 9880 Write 3 New Done Done false
8882 Acked 9881 Write 3 New Done Done false
8883 Acked 9882 Write 8 New Done Done false
8884 Acked 9883 Read 2 New Done Done false
8885 Acked 9884 Write 2 New Done Done false
8886 Acked 9885 Read 3 New Done Done false
8887 Acked 9886 Write 6 New Done Done false
8888 Acked 9887 Write 4 New Done Done false
8889 Acked 9888 Write 5 New Done Done false
8890 Acked 9889 Read 2 New Done Done false
8891 Acked 9890 Write 3 New Done Done false
8892 Acked 9891 Write 2 New Done Done false
8893 Acked 9892 Read 7 New Done Done false
8894 Acked 9893 Read 3 New Done Done false
8895 Acked 9894 Read 7 New Done Done false
8896 Acked 9895 Flush 0 New Done Done false
8897 Acked 9896 Read 2 New Done Done false
8898 Acked 9897 Write 2 New Done Done false
8899 Acked 9898 Read 5 New Done Done false
8900 Acked 9899 Read 9 New Done Done false
8901 Acked 9900 Read 8 New Done Done false
8902 Acked 9901 Write 5 New Done Done false
8903 Acked 9902 Flush 0 New Done Done false
8904 Acked 9903 Write 8 New Done Done false
8905 Acked 9904 Read 2 New Done Done false
8906 Acked 9905 Read 3 New Done Done false
8907 Acked 9906 Flush 0 New Done Done false
8908 Acked 9907 Read 1 New Done Done false
8909 Acked 9908 Write 9 New Done Done false
8910 Acked 9909 Write 3 New Done Done false
8911 Acked 9910 Read 7 New Done Done false
8912 Acked 9911 Flush 0 New Done Done false
8913 Acked 9912 Read 8 New Done Done false
8914 Acked 9913 Read 8 New Done Done false
8915 Acked 9914 Read 10 New Done Done false
8916 Acked 9915 Write 7 New Done Done false
8917 Acked 9916 Write 9 New Done Done false
8918 Acked 9917 Flush 0 New Done Done false
8919 Acked 9918 Read 3 New Done Done false
8920 Acked 9919 Read 9 New Done Done false
8921 Acked 9920 Write 1 New Done Done false
8922 Acked 9921 Write 1 New Done Done false
8923 Acked 9922 Write 3 New Done Done false
8924 Acked 9923 Read 4 New Done Done false
8925 Acked 9924 Read 9 New Done Done false
8926 Acked 9925 Write 5 New Done Done false
8927 Acked 9926 Write 6 New Done Done false
8928 Acked 9927 Read 4 New Done Done false
8929 Acked 9928 Write 4 New Done Done false
8930 Acked 9929 Write 4 New Done Done false
8931 Acked 9930 Read 9 New Done Done false
8932 Acked 9931 Write 8 New Done Done false
8933 Acked 9932 Read 4 New Done Done false
8934 Acked 9933 Read 10 New Done Done false
8935 Acked 9934 Write 5 New Done Done false
8936 Acked 9935 Write 5 New Done Done false
8937 Acked 9936 Write 4 New Done Done false
8938 Acked 9937 Write 6 New Done Done false
8939 Acked 9938 Read 1 New Done Done false
8940 Acked 9939 Read 8 New Done Done false
8941 Acked 9940 Read 5 New Done Done false
8942 Acked 9941 Write 1 New Done Done false
8943 Acked 9942 Write 3 New Done Done false
8944 Acked 9943 Read 1 New Done Done false
8945 Acked 9944 Read 5 New Done Done false
8946 Acked 9945 Write 6 New Done Done false
8947 Acked 9946 Write 9 New Done Done false
8948 Acked 9947 Read 4 New Done Done false
8949 Acked 9948 Write 7 New Done Done false
8950 Acked 9949 Write 2 New Done Done false
8951 Acked 9950 Read 7 New Done Done false
8952 Acked 9951 Read 8 New Done Done false
8953 Acked 9952 Read 8 New Done Done false
8954 Acked 9953 Write 1 New Done Done false
8955 Acked 9954 Read 2 New Done Done false
8956 Acked 9955 Write 5 New Done Done false
8957 Acked 9956 Read 6 New Done Done false
8958 Acked 9957 Read 2 New Done Done false
8959 Acked 9958 Read 2 New Done Done false
8960 Acked 9959 Flush 0 New Done Done false
8961 Acked 9960 Read 2 New Done Done false
8962 Acked 9961 Write 9 New Done Done false
8963 Acked 9962 Write 1 New Done Done false
8964 Acked 9963 Read 3 New Done Done false
8965 Acked 9964 Write 8 New Done Done false
8966 Acked 9965 Write 5 New Done Done false
8967 Acked 9966 Read 3 New Done Done false
8968 Acked 9967 Read 3 New Done Done false
8969 Acked 9968 Write 2 New Done Done false
8970 Acked 9969 Read 5 New Done Done false
8971 Acked 9970 Write 2 New Done Done false
8972 Acked 9971 Read 7 New Done Done false
8973 Acked 9972 Write 8 New Done Done false
8974 Acked 9973 Write 4 New Done Done false
8975 Acked 9974 Write 7 New Done Done false
8976 Acked 9975 Write 8 New Done Done false
8977 Acked 9976 Flush 0 New Done Done false
8978 Acked 9977 Read 5 New Done Done false
8979 Acked 9978 Read 5 New Done Done false
8980 Acked 9979 Write 5 New Done Done false
8981 Acked 9980 Write 9 New Done Done false
8982 Acked 9981 Read 1 New Done Done false
8983 Acked 9982 Read 2 New Done Done false
8984 Acked 9983 Read 3 New Done Done false
8985 Acked 9984 Read 9 New Done Done false
8986 Acked 9985 Write 3 New Done Done false
8987 Acked 9986 Write 8 New Done Done false
8988 Acked 9987 Write 2 New Done Done false
8989 Acked 9988 Flush 0 New Done Done false
8990 Acked 9989 Read 2 New Done Done false
8991 Acked 9990 Flush 0 New Done Done false
8992 Acked 9991 Read 4 New Done Done false
8993 Acked 9992 Flush 0 New Done Done false
8994 Acked 9993 Read 2 New Done Done false
8995 Acked 9994 Write 9 New Done Done false
8996 Acked 9995 Read 4 New Done Done false
8997 Acked 9996 Read 9 New Done Done false
8998 Acked 9997 Read 1 New Done Done false
8999 Acked 9998 Write 6 New Done Done false
9000 Acked 9999 Read 1 New Done Done false
9001 Acked 10000 Write 5 New Done Done false
9002 Acked 10001 Flush 0 New Done Done false
9003 Acked 10002 Write 10 New Done Done false
9004 Acked 10003 Write 7 New Done Done false
9005 Acked 10004 Read 10 New Done Done false
9006 Acked 10005 Read 4 New Done Done false
9007 Acked 10006 Write 10 New Done Done false
9008 Acked 10007 Read 5 New Done Done false
9009 Acked 10008 Write 10 New Done Done false
9010 Acked 10009 Read 7 New Done Done false
9011 Acked 10010 Write 2 New Done Done false
9012 Acked 10011 Flush 0 New Done Done false
9013 Acked 10012 Write 10 New Done Done false
9014 Acked 10013 Read 7 New Done Done false
9015 Acked 10014 Write 6 New Done Done false
9016 Acked 10015 Write 7 New Done Done false
9017 Acked 10016 Read 8 New Done Done false
9018 Acked 10017 Read 2 New Done Done false
9019 Acked 10018 Flush 0 New Done Done false
9020 Acked 10019 Read 10 New Done Done false
9021 Acked 10020 Read 7 New Done Done false
9022 Acked 10021 Read 10 New Done Done false
9023 Acked 10022 Read 2 New Done Done false
9024 Acked 10023 Read 4 New Done Done false
9025 Acked 10024 Read 5 New Done Done false
9026 Acked 10025 Flush 0 New Done Done false
9027 Acked 10026 Read 8 New Done Done false
9028 Acked 10027 Read 2 New Done Done false
9029 Acked 10028 Read 1 New Done Done false
9030 Acked 10029 Flush 0 New Done Done false
9031 Acked 10030 Read 8 New Done Done false
9032 Acked 10031 Read 9 New Done Done false
9033 Acked 10032 Write 7 New Done Done false
9034 Acked 10033 Write 5 New Done Done false
9035 Acked 10034 Write 6 New Done Done false
9036 Acked 10035 Read 9 New Done Done false
9037 Acked 10036 Read 9 New Done Done false
9038 Acked 10037 Flush 0 New Done Done false
9039 Acked 10038 Read 7 New Done Done false
9040 Acked 10039 Read 3 New Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
New 300 0 0 300
Sent 0 0 0 0
Done 0 300 300 600
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 9739 10037 10037
Downstairs last five completed: 9739 9738 9737 9736 9735
Upstairs last five completed: 9040 9039 9038 9037 9036
CLIENT: Up:0 ds:300 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:12.547893621Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:12.547957819Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:12.547981658Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:12.548012717Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:12.548029526Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 9739","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:12.548047196Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:12.54818587Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:13.548868523Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:13.548959809Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:13.549001918Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:13.549040756Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:13.549070105Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 9739","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:13.549104774Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:13.549355404Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:14.550108644Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:14.550170081Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:14.55019423Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:14.550221659Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:14.550238129Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 9739","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:14.550255378Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:14.550400923Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:15.55227805Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:15.552336187Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:15.552359817Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" })","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:15.552381166Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Still missing, state is Offline","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:15.552397105Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] client re-new 301 jobs since flush 9739","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:15.552414475Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:15.55253135Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
[8810] got start action
[8810] I am going to restart
Starting downstairs at port 8810
Make output file at "/tmp/dsc/downstairs-8810.txt"
[8810][0] reports Starting
Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1194)
[8810][0] reports Running
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
States: Offline Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
8741 Acked 9740 Write 9 New Done Done true
8742 Acked 9741 Read 6 New Done Done true
8743 Acked 9742 Read 8 New Done Done true
8744 Acked 9743 Read 7 New Done Done true
8745 Acked 9744 Write 4 New Done Done true
8746 Acked 9745 Write 1 New Done Done true
8747 Acked 9746 Write 2 New Done Done true
8748 Acked 9747 Flush 0 New Done Done true
8749 Acked 9748 Read 7 New Done Done true
8750 Acked 9749 Write 10 New Done Done true
8751 Acked 9750 Read 5 New Done Done true
8752 Acked 9751 Read 6 New Done Done true
8753 Acked 9752 Write 6 New Done Done true
8754 Acked 9753 Read 6 New Done Done true
8755 Acked 9754 Read 4 New Done Done true
8756 Acked 9755 Write 5 New Done Done true
8757 Acked 9756 Write 9 New Done Done true
8758 Acked 9757 Write 4 New Done Done true
8759 Acked 9758 Read 4 New Done Done true
8760 Acked 9759 Write 2 New Done Done true
8761 Acked 9760 Read 9 New Done Done true
8762 Acked 9761 Read 10 New Done Done true
8763 Acked 9762 Flush 0 New Done Done true
8764 Acked 9763 Flush 0 New Done Done true
8765 Acked 9764 Read 6 New Done Done true
8766 Acked 9765 Write 2 New Done Done true
8767 Acked 9766 Write 6 New Done Done true
8768 Acked 9767 Read 8 New Done Done true
8769 Acked 9768 Read 3 New Done Done true
8770 Acked 9769 Read 3 New Done Done true
8771 Acked 9770 Flush 0 New Done Done true
8772 Acked 9771 Read 9 New Done Done true
8773 Acked 9772 Write 8 New Done Done true
8774 Acked 9773 Write 10 New Done Done true
8775 Acked 9774 Read 7 New Done Done true
8776 Acked 9775 Flush 0 New Done Done true
8777 Acked 9776 Write 5 New Done Done true
8778 Acked 9777 Read 1 New Done Done true
8779 Acked 9778 Write 7 New Done Done true
8780 Acked 9779 Read 10 New Done Done true
8781 Acked 9780 Write 6 New Done Done true
8782 Acked 9781 Read 10 New Done Done true
8783 Acked 9782 Read 10 New Done Done true
8784 Acked 9783 Read 6 New Done Done true
8785 Acked 9784 Write 4 New Done Done true
8786 Acked 9785 Write 3 New Done Done true
8787 Acked 9786 Read 2 New Done Done true
8788 Acked 9787 Read 8 New Done Done true
8789 Acked 9788 Read 5 New Done Done true
8790 Acked 9789 Write 1 New Done Done true
8791 Acked 9790 Read 2 New Done Done true
8792 Acked 9791 Write 9 New Done Done true
8793 Acked 9792 Read 1 New Done Done true
8794 Acked 9793 Read 1 New Done Done true
8795 Acked 9794 Write 10 New Done Done true
8796 Acked 9795 Flush 0 New Done Done true
8797 Acked 9796 Read 3 New Done Done true
8798 Acked 9797 Read 2 New Done Done true
8799 Acked 9798 Read 6 New Done Done true
8800 Acked 9799 Write 2 New Done Done true
8801 Acked 9800 Read 1 New Done Done true
8802 Acked 9801 Read 5 New Done Done true
8803 Acked 9802 Read 1 New Done Done true
8804 Acked 9803 Read 10 New Done Done true
8805 Acked 9804 Write 9 New Done Done true
8806 Acked 9805 Flush 0 New Done Done true
8807 Acked 9806 Flush 0 New Done Done true
8808 Acked 9807 Flush 0 New Done Done true
8809 Acked 9808 Read 1 New Done Done true
8810 Acked 9809 Read 10 New Done Done true
8811 Acked 9810 Read 4 New Done Done true
8812 Acked 9811 Write 7 New Done Done true
8813 Acked 9812 Write 8 New Done Done true
8814 Acked 9813 Write 9 New Done Done true
8815 Acked 9814 Read 6 New Done Done true
8816 Acked 9815 Write 8 New Done Done true
8817 Acked 9816 Read 3 New Done Done true
8818 Acked 9817 Write 8 New Done Done true
8819 Acked 9818 Write 6 New Done Done true
8820 Acked 9819 Read 3 New Done Done true
8821 Acked 9820 Read 7 New Done Done true
8822 Acked 9821 Write 7 New Done Done true
8823 Acked 9822 Flush 0 New Done Done true
8824 Acked 9823 Read 1 New Done Done true
8825 Acked 9824 Write 8 New Done Done true
8826 Acked 9825 Read 8 New Done Done true
8827 Acked 9826 Read 6 New Done Done true
8828 Acked 9827 Flush 0 New Done Done true
8829 Acked 9828 Read 8 New Done Done true
8830 Acked 9829 Read 7 New Done Done true
8831 Acked 9830 Write 10 New Done Done true
8832 Acked 9831 Read 7 New Done Done true
8833 Acked 9832 Read 4 New Done Done true
8834 Acked 9833 Write 8 New Done Done true
8835 Acked 9834 Write 9 New Done Done true
8836 Acked 9835 Read 8 New Done Done true
8837 Acked 9836 Flush 0 New Done Done true
8838 Acked 9837 Write 9 New Done Done true
8839 Acked 9838 Read 4 New Done Done true
8840 Acked 9839 Write 6 New Done Done true
8841 Acked 9840 Write 5 New Done Done true
8842 Acked 9841 Write 2 New Done Done true
8843 Acked 9842 Write 4 New Done Done true
8844 Acked 9843 Read 2 New Done Done true
8845 Acked 9844 Read 1 New Done Done true
8846 Acked 9845 Write 3 New Done Done true
8847 Acked 9846 Read 8 New Done Done true
8848 Acked 9847 Read 2 New Done Done true
8849 Acked 9848 Write 1 New Done Done true
8850 Acked 9849 Read 7 New Done Done true
8851 Acked 9850 Write 2 New Done Done true
8852 Acked 9851 Flush 0 New Done Done true
8853 Acked 9852 Read 1 New Done Done true
8854 Acked 9853 Read 2 New Done Done true
8855 Acked 9854 Read 9 New Done Done true
8856 Acked 9855 Flush 0 New Done Done true
8857 Acked 9856 Read 8 New Done Done true
8858 Acked 9857 Write 4 New Done Done true
8859 Acked 9858 Read 6 New Done Done true
8860 Acked 9859 Flush 0 New Done Done true
8861 Acked 9860 Read 9 New Done Done true
8862 Acked 9861 Read 3 New Done Done true
8863 Acked 9862 Read 8 New Done Done true
8864 Acked 9863 Read 5 New Done Done true
8865 Acked 9864 Write 6 New Done Done true
8866 Acked 9865 Read 6 New Done Done true
8867 Acked 9866 Read 1 New Done Done true
8868 Acked 9867 Write 4 New Done Done true
8869 Acked 9868 Read 7 New Done Done true
8870 Acked 9869 Read 8 New Done Done true
8871 Acked 9870 Read 4 New Done Done true
8872 Acked 9871 Write 10 New Done Done true
8873 Acked 9872 Read 6 New Done Done true
8874 Acked 9873 Flush 0 New Done Done true
8875 Acked 9874 Read 8 New Done Done true
8876 Acked 9875 Read 4 New Done Done true
8877 Acked 9876 Read 4 New Done Done true
8878 Acked 9877 Read 10 New Done Done true
8879 Acked 9878 Read 8 New Done Done true
8880 Acked 9879 Read 9 New Done Done true
8881 Acked 9880 Write 3 New Done Done true
8882 Acked 9881 Write 3 New Done Done true
8883 Acked 9882 Write 8 New Done Done true
8884 Acked 9883 Read 2 New Done Done true
8885 Acked 9884 Write 2 New Done Done true
8886 Acked 9885 Read 3 New Done Done true
8887 Acked 9886 Write 6 New Done Done true
8888 Acked 9887 Write 4 New Done Done true
8889 Acked 9888 Write 5 New Done Done true
8890 Acked 9889 Read 2 New Done Done true
8891 Acked 9890 Write 3 New Done Done true
8892 Acked 9891 Write 2 New Done Done true
8893 Acked 9892 Read 7 New Done Done true
8894 Acked 9893 Read 3 New Done Done true
8895 Acked 9894 Read 7 New Done Done true
8896 Acked 9895 Flush 0 New Done Done true
8897 Acked 9896 Read 2 New Done Done true
8898 Acked 9897 Write 2 New Done Done true
8899 Acked 9898 Read 5 New Done Done true
8900 Acked 9899 Read 9 New Done Done true
8901 Acked 9900 Read 8 New Done Done true
8902 Acked 9901 Write 5 New Done Done true
8903 Acked 9902 Flush 0 New Done Done true
8904 Acked 9903 Write 8 New Done Done true
8905 Acked 9904 Read 2 New Done Done true
8906 Acked 9905 Read 3 New Done Done true
8907 Acked 9906 Flush 0 New Done Done true
8908 Acked 9907 Read 1 New Done Done true
8909 Acked 9908 Write 9 New Done Done true
8910 Acked 9909 Write 3 New Done Done true
8911 Acked 9910 Read 7 New Done Done true
8912 Acked 9911 Flush 0 New Done Done true
8913 Acked 9912 Read 8 New Done Done true
8914 Acked 9913 Read 8 New Done Done true
8915 Acked 9914 Read 10 New Done Done true
8916 Acked 9915 Write 7 New Done Done true
8917 Acked 9916 Write 9 New Done Done true
8918 Acked 9917 Flush 0 New Done Done true
8919 Acked 9918 Read 3 New Done Done true
8920 Acked 9919 Read 9 New Done Done true
8921 Acked 9920 Write 1 New Done Done true
8922 Acked 9921 Write 1 New Done Done true
8923 Acked 9922 Write 3 New Done Done true
8924 Acked 9923 Read 4 New Done Done true
8925 Acked 9924 Read 9 New Done Done true
8926 Acked 9925 Write 5 New Done Done true
8927 Acked 9926 Write 6 New Done Done true
8928 Acked 9927 Read 4 New Done Done true
8929 Acked 9928 Write 4 New Done Done true
8930 Acked 9929 Write 4 New Done Done true
8931 Acked 9930 Read 9 New Done Done true
8932 Acked 9931 Write 8 New Done Done true
8933 Acked 9932 Read 4 New Done Done true
8934 Acked 9933 Read 10 New Done Done true
8935 Acked 9934 Write 5 New Done Done true
8936 Acked 9935 Write 5 New Done Done true
8937 Acked 9936 Write 4 New Done Done true
8938 Acked 9937 Write 6 New Done Done true
8939 Acked 9938 Read 1 New Done Done true
8940 Acked 9939 Read 8 New Done Done true
8941 Acked 9940 Read 5 New Done Done true
8942 Acked 9941 Write 1 New Done Done true
8943 Acked 9942 Write 3 New Done Done true
8944 Acked 9943 Read 1 New Done Done true
8945 Acked 9944 Read 5 New Done Done true
8946 Acked 9945 Write 6 New Done Done true
8947 Acked 9946 Write 9 New Done Done true
8948 Acked 9947 Read 4 New Done Done true
8949 Acked 9948 Write 7 New Done Done true
8950 Acked 9949 Write 2 New Done Done true
8951 Acked 9950 Read 7 New Done Done true
8952 Acked 9951 Read 8 New Done Done true
8953 Acked 9952 Read 8 New Done Done true
8954 Acked 9953 Write 1 New Done Done true
8955 Acked 9954 Read 2 New Done Done true
8956 Acked 9955 Write 5 New Done Done true
8957 Acked 9956 Read 6 New Done Done true
8958 Acked 9957 Read 2 New Done Done true
8959 Acked 9958 Read 2 New Done Done true
8960 Acked 9959 Flush 0 New Done Done true
8961 Acked 9960 Read 2 New Done Done true
8962 Acked 9961 Write 9 New Done Done true
8963 Acked 9962 Write 1 New Done Done true
8964 Acked 9963 Read 3 New Done Done true
8965 Acked 9964 Write 8 New Done Done true
8966 Acked 9965 Write 5 New Done Done true
8967 Acked 9966 Read 3 New Done Done true
8968 Acked 9967 Read 3 New Done Done true
8969 Acked 9968 Write 2 New Done Done true
8970 Acked 9969 Read 5 New Done Done true
8971 Acked 9970 Write 2 New Done Done true
8972 Acked 9971 Read 7 New Done Done true
8973 Acked 9972 Write 8 New Done Done true
8974 Acked 9973 Write 4 New Done Done true
8975 Acked 9974 Write 7 New Done Done true
8976 Acked 9975 Write 8 New Done Done true
8977 Acked 9976 Flush 0 New Done Done true
8978 Acked 9977 Read 5 New Done Done true
8979 Acked 9978 Read 5 New Done Done true
8980 Acked 9979 Write 5 New Done Done true
8981 Acked 9980 Write 9 New Done Done true
8982 Acked 9981 Read 1 New Done Done true
8983 Acked 9982 Read 2 New Done Done true
8984 Acked 9983 Read 3 New Done Done true
8985 Acked 9984 Read 9 New Done Done true
8986 Acked 9985 Write 3 New Done Done true
8987 Acked 9986 Write 8 New Done Done true
8988 Acked 9987 Write 2 New Done Done true
8989 Acked 9988 Flush 0 New Done Done true
8990 Acked 9989 Read 2 New Done Done true
8991 Acked 9990 Flush 0 New Done Done true
8992 Acked 9991 Read 4 New Done Done true
8993 Acked 9992 Flush 0 New Done Done true
8994 Acked 9993 Read 2 New Done Done true
8995 Acked 9994 Write 9 New Done Done true
8996 Acked 9995 Read 4 New Done Done true
8997 Acked 9996 Read 9 New Done Done true
8998 Acked 9997 Read 1 New Done Done true
8999 Acked 9998 Write 6 New Done Done true
9000 Acked 9999 Read 1 New Done Done true
9001 Acked 10000 Write 5 New Done Done true
9002 Acked 10001 Flush 0 New Done Done true
9003 Acked 10002 Write 10 New Done Done true
9004 Acked 10003 Write 7 New Done Done true
9005 Acked 10004 Read 10 New Done Done true
9006 Acked 10005 Read 4 New Done Done true
9007 Acked 10006 Write 10 New Done Done true
9008 Acked 10007 Read 5 New Done Done true
9009 Acked 10008 Write 10 New Done Done true
9010 Acked 10009 Read 7 New Done Done true
9011 Acked 10010 Write 2 New Done Done true
9012 Acked 10011 Flush 0 New Done Done true
9013 Acked 10012 Write 10 New Done Done true
9014 Acked 10013 Read 7 New Done Done true
9015 Acked 10014 Write 6 New Done Done true
9016 Acked 10015 Write 7 New Done Done true
9017 Acked 10016 Read 8 New Done Done true
9018 Acked 10017 Read 2 New Done Done true
9019 Acked 10018 Flush 0 New Done Done true
9020 Acked 10019 Read 10 New Done Done true
9021 Acked 10020 Read 7 New Done Done true
9022 Acked 10021 Read 10 New Done Done true
9023 Acked 10022 Read 2 New Done Done true
9024 Acked 10023 Read 4 New Done Done true
9025 Acked 10024 Read 5 New Done Done true
9026 Acked 10025 Flush 0 New Done Done true
9027 Acked 10026 Read 8 New Done Done true
9028 Acked 10027 Read 2 New Done Done true
9029 Acked 10028 Read 1 New Done Done true
9030 Acked 10029 Flush 0 New Done Done true
9031 Acked 10030 Read 8 New Done Done true
9032 Acked 10031 Read 9 New Done Done true
9033 Acked 10032 Write 7 New Done Done true
9034 Acked 10033 Write 5 New Done Done true
9035 Acked 10034 Write 6 New Done Done true
9036 Acked 10035 Read 9 New Done Done true
9037 Acked 10036 Read 9 New Done Done true
9038 Acked 10037 Flush 0 New Done Done true
9039 Acked 10038 Read 7 New Done Done true
9040 Acked 10039 Read 3 New Done Done true
9041 Acked 10040 Flush 0 New Done Done true
STATES DS:0 DS:1 DS:2 TOTAL
New 301 0 0 301
Sent 0 0 0 0
Done 0 301 301 602
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 9739 10040 10040
Downstairs last five completed: 9739 9738 9737 9736 9735
Upstairs last five completed: 9041 9040 9039 9038 9037
CLIENT: Up:0 ds:301 act:2
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:16.554373418Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:16.554431606Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:16.55561626Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Returning UUID:12345678-0000-0000-0000-000000008810 matches","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:16.555643529Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"send last flush ID to this DS: 9739","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:16.555658108Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"Replied this last flush ID: 9739","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:16.555814743Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:16.555842781Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Offline to Replay","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:16.555858091Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"ds_transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:16.55587202Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
{"msg":"[0] Transition from Replay to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:16.55588603Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"7daa0b95-273f-4ce8-935e-89c0961e151d"}
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 10040 10039 10038 10037 10036
Upstairs last five completed: 9041 9040 9039 9038 9037
CLIENT: Up:0 ds:0 act:3
Replay: All jobs finished, all DS active.
Test replay has completed
Wrote out file "alan"
CLIENT: Tests done. All submitted work has been ACK'd
----------------------------------------------------------------
Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 10040 10039 10038 10037 10036
Upstairs last five completed: 9041 9040 9039 9038 9037
CLIENT: Up:0 ds:0 act:3
CLIENT: All jobs finished, all DS active.
Replays:30 time: 5:45
Do final verify
{"msg":"Upstairs starts","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.918181211Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199}
{"msg":"Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"29185f5538632a13039785e911d1aaa0146bfa14\",\n git_commit_timestamp: \"2024-02-13T21:13:12.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.75.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"82e1608dfa6e0b5569232559e3d385fea5a93112\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.918433451Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199}
{"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.91845025Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199}
{"msg":"Crucible f28e57d3-e1e6-4393-a18a-ed478fe90694 has session id: 779cb5e2-f58d-4951-84fc-7237a9faee7c","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.91846407Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"Upstairs opts: Upstairs UUID: f28e57d3-e1e6-4393-a18a-ed478fe90694, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.918490699Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"Crucible stats registered with UUID: f28e57d3-e1e6-4393-a18a-ed478fe90694","v":0,"name":"crucible","level":30Crucible runtime is spawned
,"time":"2024-02-13T21:45:19.918512968Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"The guest has requested activation with gen:3","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.918600924Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199}
{"msg":"f28e57d3-e1e6-4393-a18a-ed478fe90694 active request set","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.918624014Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.918638253Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"0","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.918653033Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"0","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.918676262Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"1","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.918703791Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"1","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"sending connect oneshot to client","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.91872881Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"2","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"client set_active_request while in New; waiting...","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.918753949Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"2","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.918779468Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"":"io task","client":"0","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"connecting to 127.0.0.1:8820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.918808037Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"":"io task","client":"1","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"connecting to 127.0.0.1:8830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.918834616Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"":"io task","client":"2","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.919045327Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"":"io task","client":"0","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.919084686Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"":"io task","client":"1","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"ds_connection connected","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.919112055Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"":"io task","client":"2","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.919711742Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"1","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"[1] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.91975234Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"1","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.919788409Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"2","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"[2] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.919819458Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"2","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"ds_transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.919848337Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"0","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"[0] Transition from New to WaitActive","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.919876796Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"0","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.920340808Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"2","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.920480793Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"1","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.920518921Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"0","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.920992903Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"2","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"[2] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921032012Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"2","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921053131Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921244834Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"1","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"[1] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921272293Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"1","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"Waiting for true more clients to be ready","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921287462Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"ds_transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921300091Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"0","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"[0] Transition from WaitActive to WaitQuorum","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921317301Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"0","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"[0]R flush_numbers[0..12]: [920, 920, 914, 925, 909, 909, 917, 895, 916, 921, 915, 915]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.92133262Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"[0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.92134723Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"[0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921362069Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"[1]R flush_numbers[0..12]: [920, 920, 914, 925, 909, 909, 917, 895, 916, 921, 915, 915]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921376378Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"[1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921390258Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921404847Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"[2]R flush_numbers[0..12]: [920, 920, 914, 925, 909, 909, 917, 895, 916, 921, 915, 915]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921419187Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"[2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921433406Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","Wait for a query_work_queue command to finish before sending IO
pid":1199,"":"downstairs","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921455866Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"Max found gen is 3","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921480414Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"Generation requested: 3 >= found:3","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921497634Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"Next flush: 926","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921511163Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"All extents match","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921524593Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"No downstairs reconciliation required","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921540592Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921555732Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"0","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921579951Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"1","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.92159694Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"client":"2","":"downstairs","session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"All required reconciliation work is completed","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921611249Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"Set Downstairs and Upstairs active after reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921624119Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"f28e57d3-e1e6-4393-a18a-ed478fe90694 is now active with session: 779cb5e2-f58d-4951-84fc-7237a9faee7c","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921642668Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"Set Active after no reconciliation","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921664787Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199,"session_id":"779cb5e2-f58d-4951-84fc-7237a9faee7c"}
{"msg":"The guest has finished waiting for activation with:3","v":0,"name":"crucible","level":30,"time":"2024-02-13T21:45:19.921678607Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199}
Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
Loading write count information from file "alan"
Read and Verify all blocks (0..5000 range:false)
Wrote out file "alan"
Verify test completed
Wrote out file "alan"
CLIENT: Tests done. All submitted work has been ACK'd
----------------------------------------------------------------
Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:5
States: Active Active Active
GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
47 Acked 1046 Read 100 Done Done Done false
48 Acked 1047 Read 100 Done Done Done false
49 Acked 1048 Read 100 Done Done Done false
50 Acked 1049 Read 100 Done Done Done false
51 Acked 1050 Read 100 Done Done Done false
STATES DS:0 DS:1 DS:2 TOTAL
New 0 0 0 0
Sent 0 0 0 0
Done 5 5 5 15
Skipped 0 0 0 0
Error 0 0 0 0
Last Flush: 1045 1045 1045
Downstairs last five completed: 1045 1044 1043 1042 1041
Upstairs last five completed: 51 50 49 48 47
CLIENT: Up:0 ds:5 act:3
----------------------------------------------------------------
Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
Downstairs last five completed: 1051 1050 1049 1048 1047
Upstairs last five completed: 52 51 50 49 48
CLIENT: Up:0 ds:0 act:3
CLIENT: All crucible jobs finished, exiting program
{"msg":"Guest handle has been dropped","v":0,"name":"crucible","level":40,"time":"2024-02-13T21:45:24.48434826Z","hostname":"ip-10-150-1-87.us-west-2.compute.internal","pid":1199}
Feb 13 21:45:24.544 INFO accepted connection, remote_addr: 127.0.0.1:53299, local_addr: 127.0.0.1:9998
Main task has work to do, go find it
got dsc Shutdown
Shutdown
Shut it down
Feb 13 21:45:24.545 INFO request completed, latency_us: 225, response_code: 204, uri: /shutdown, [8810] Disable keep_running
method:[8810] Got stop action so:false kr:false
POST, req_id: 89092254-67dc-4e71-a2e9-bba4c6b34286, remote_addr:[8820] Disable keep_running
127.0.0.1:53299, local_addr:[8820] Got stop action so:false kr:false
[8830] Disable keep_running
127[8830] Got stop action so:false kr:false
.0.0.1:9998
[8830] Exited with: ExitStatus(unix_wait_status(9))
[8830][2] reports Exit
[8810] Exited with: ExitStatus(unix_wait_status(9))
[8810][0] reports Exit
[8820] Exited with: ExitStatus(unix_wait_status(9))
[8820][1] reports Exit
February 13, 2024 at 09:45:29 PM UTC Test ends with 0