Max level shown:
1 starting February 28, 2024 at 10:13:00 PM UTC
2 Creating downstairs regions
3 Creating region directory at: /var/tmp/dsc/region
4 Creating dsc directory at: /tmp/dsc
5 Downstairs region 0 created at /var/tmp/dsc/region/8810 in 1.8262451
6 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 1.12761
7 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.46473092
8 Region set with 3 regions was created
9 Starting downstairs
10 Using existing output directory "/tmp/dsc"
11 start ds: 8810
12 start access at:127.0.0.1:9998
13 start ds: 8820
14 start ds: 8830
15 Starting downstairs at port 8810
16 Make output file at "/tmp/dsc/downstairs-8810.txt"
17 [8810][0] reports Starting
18 Starting downstairs at port 8820
19 Make output file at "/tmp/dsc/downstairs-8820.txt"
20 [8820][1] reports Starting
21 Starting downstairs at port 8830
22 Make output file at "/tmp/dsc/downstairs-8830.txt"
23 [8830][2] reports Starting
24 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1154)
25 [8810][0] reports Running
26 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1155)
27 [8820][1] reports Running
28 Control access at:127.0.0.1:9998
29 Feb 28 22:13:03.664 INFO listening, local_addr: 127.0.0.1:9998
30 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1156)
31 [8830][2] reports Running
32 Running initial fill
332024-02-28T22:13:08.701ZINFOcrucible: Upstairs starts
342024-02-28T22:13:08.703ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "80b6f734175c2bc8e222c8d6f7875e53cbd5c9ca", git_commit_timestamp: "2024-02-28T21:51:33.000000000Z", git_branch: "main", rustc_semver: "1.75.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "82e1608dfa6e0b5569232559e3d385fea5a93112", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
352024-02-28T22:13:08.703ZINFOcrucible: Upstairs <-> Downstairs Message Version: 5
362024-02-28T22:13:08.703ZINFOcrucible: Crucible 82d63dd5-d25e-4720-8ba3-6260c1df6dfc has session id: d5ad99c8-e34b-44f6-b802-4267befe06ff session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
372024-02-28T22:13:08.703ZINFOcrucible: Upstairs opts: Upstairs UUID: 82d63dd5-d25e-4720-8ba3-6260c1df6dfc, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
382024-02-28T22:13:08.703ZINFOcrucible: Crucible stats registered with UUID: 82d63dd5-d25e-4720-8ba3-6260c1df6dfc session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
39 Crucible runtime is spawned
402024-02-28T22:13:08.704ZINFOcrucible: The guest has requested activation with gen:1
412024-02-28T22:13:08.704ZINFOcrucible: 82d63dd5-d25e-4720-8ba3-6260c1df6dfc active request set session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
422024-02-28T22:13:08.704ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
432024-02-28T22:13:08.704ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 0 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
442024-02-28T22:13:08.704ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
452024-02-28T22:13:08.704ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 1 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
462024-02-28T22:13:08.704ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
472024-02-28T22:13:08.704ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 2 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
482024-02-28T22:13:08.704ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
492024-02-28T22:13:08.704ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
502024-02-28T22:13:08.705ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
512024-02-28T22:13:08.705ZINFOcrucible: ds_connection connected = io task client = 0 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
522024-02-28T22:13:08.705ZINFOcrucible: ds_connection connected = io task client = 2 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
532024-02-28T22:13:08.705ZINFOcrucible: ds_connection connected = io task client = 1 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
542024-02-28T22:13:08.707ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 2 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
552024-02-28T22:13:08.707ZINFOcrucible: [2] Transition from New to WaitActive = downstairs client = 2 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
562024-02-28T22:13:08.707ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 0 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
572024-02-28T22:13:08.707ZINFOcrucible: [0] Transition from New to WaitActive = downstairs client = 0 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
582024-02-28T22:13:08.707ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 1 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
592024-02-28T22:13:08.707ZINFOcrucible: [1] Transition from New to WaitActive = downstairs client = 1 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
602024-02-28T22:13:08.709ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
612024-02-28T22:13:08.709ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
622024-02-28T22:13:08.709ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
632024-02-28T22:13:08.710ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
642024-02-28T22:13:08.710ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
652024-02-28T22:13:08.710ZINFOcrucible: Waiting for true more clients to be ready session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
662024-02-28T22:13:08.710ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
672024-02-28T22:13:08.710ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
682024-02-28T22:13:08.710ZINFOcrucible: Waiting for true more clients to be ready session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
692024-02-28T22:13:08.710ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
702024-02-28T22:13:08.710ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
712024-02-28T22:13:08.711ZINFOcrucible: [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
722024-02-28T22:13:08.711ZINFOcrucible: [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
732024-02-28T22:13:08.711ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
742024-02-28T22:13:08.711ZINFOcrucible: [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
752024-02-28T22:13:08.711ZINFOcrucible: [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
762024-02-28T22:13:08.711ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
772024-02-28T22:13:08.711ZINFOcrucible: [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
782024-02-28T22:13:08.711ZINFOcrucible: [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
792024-02-28T22:13:08.711ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
802024-02-28T22:13:08.711ZINFOcrucible: Max found gen is 1 = downstairs session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
812024-02-28T22:13:08.711ZINFOcrucible: Generation requested: 1 >= found:1 = downstairs session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
822024-02-28T22:13:08.711ZINFOcrucible: Next flush: 1 = downstairs session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
832024-02-28T22:13:08.712ZINFOcrucible: All extents match = downstairs session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
842024-02-28T22:13:08.712ZINFOcrucible: No downstairs reconciliation required session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
852024-02-28T22:13:08.712ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
86 {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Activate successful
87 ,Wait for a query_work_queue command to finish before sending IO
88 "time":"2024-02-28T22:13:08.712195399Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1160,"client":"1","":"downstairs","session_id":"d5ad99c8-e34b-44f6-b802-4267befe06ff"}
892024-02-28T22:13:08.712ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
902024-02-28T22:13:08.712ZINFOcrucible: All required reconciliation work is completed session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
912024-02-28T22:13:08.712ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
922024-02-28T22:13:08.712ZINFOcrucible: 82d63dd5-d25e-4720-8ba3-6260c1df6dfc is now active with session: d5ad99c8-e34b-44f6-b802-4267befe06ff session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
932024-02-28T22:13:08.712ZINFOcrucible: The guest has finished waiting for activation with:1
942024-02-28T22:13:08.712ZINFOcrucible: Set Active after no reconciliation session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
95 Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
96 Fill test
97 Read and Verify all blocks (0..5000 range:false)
98 Wrote out file "alan"
99 CLIENT: Tests done. All submitted work has been ACK'd
100 ----------------------------------------------------------------
101 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:4
102 States: Active Active Active
103 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
104 99 Acked 1098 Read 100 Done Done Done false
105 100 Acked 1099 Read 100 Done Done Done false
106 101 Acked 1100 Read 100 Done Done Done false
107 102 Acked 1101 Read 100 Done Done Done false
108 STATES DS:0 DS:1 DS:2 TOTAL
109 New 0 0 0 0
110 Sent 0 0 0 0
111 Done 4 4 4 12
112 Skipped 0 0 0 0
113 Error 0 0 0 0
114 Last Flush: 1097 1097 1097
115 Downstairs last five completed: 1097 1096 1095 1094 1093
116 Upstairs last five completed: 102 101 100 99 98
117 CLIENT: Up:0 ds:4 act:3
118 ----------------------------------------------------------------
119 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
120 Downstairs last five completed: 1102 1101 1100 1099 1098
121 Upstairs last five completed: 103 102 101 100 99
122 CLIENT: Up:0 ds:0 act:3
123 CLIENT: All crucible jobs finished, exiting program
1242024-02-28T22:13:13.760ZWARNcrucible: Guest handle has been dropped
1252024-02-28T22:13:13.760ZWARNcrucible: client request queue closed unexpectedly; is the program exiting? = io task client = 0 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
1262024-02-28T22:13:13.760ZWARNcrucible: client task is sending Done(QueueClosed) = io task client = 0 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
1272024-02-28T22:13:13.760ZWARNcrucible: client request queue closed unexpectedly; is the program exiting? = io task client = 1 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
1282024-02-28T22:13:13.760ZWARNcrucible: client task could not reply to main task; shutting down? = io task client = 0 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
1292024-02-28T22:13:13.760ZINFOcrucible: client task is exiting = io task client = 0 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
1302024-02-28T22:13:13.760ZWARNcrucible: client task is sending Done(QueueClosed) = io task client = 1 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
1312024-02-28T22:13:13.760ZWARNcrucible: client task could not reply to main task; shutting down? = io task client = 1 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
1322024-02-28T22:13:13.760ZINFOcrucible: client task is exiting = io task client = 1 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
1332024-02-28T22:13:13.760ZWARNcrucible: client_stop_rx closed unexpectedly: RecvError(()) = io task client = 2 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
1342024-02-28T22:13:13.760ZWARNcrucible: client task is sending Done(QueueClosed) = io task client = 2 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
1352024-02-28T22:13:13.760ZWARNcrucible: client task could not reply to main task; shutting down? = io task client = 2 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
1362024-02-28T22:13:13.760ZINFOcrucible: client task is exiting = io task client = 2 session_id = d5ad99c8-e34b-44f6-b802-4267befe06ff
137 Replay loop starts now February 28, 2024 at 10:13:13 PM UTC
1382024-02-28T22:13:13.785ZINFOcrucible: Upstairs starts
1392024-02-28T22:13:13.786ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "80b6f734175c2bc8e222c8d6f7875e53cbd5c9ca", git_commit_timestamp: "2024-02-28T21:51:33.000000000Z", git_branch: "main", rustc_semver: "1.75.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "82e1608dfa6e0b5569232559e3d385fea5a93112", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
1402024-02-28T22:13:13.786ZINFOcrucible: Upstairs <-> Downstairs Message Version: 5
1412024-02-28T22:13:13.786ZINFOcrucible: Crucible 7d2be74c-72cc-486f-817a-a33f5afe4f1c has session id: 69c7fbb2-d5b2-4965-9fa4-441c4d60207e session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1422024-02-28T22:13:13.786ZINFOcrucible: Upstairs opts: Upstairs UUID: 7d2be74c-72cc-486f-817a-a33f5afe4f1c, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143 {"msg":"Crucible stats registered with UUID: 7d2be74c-72cc-486f-817a-a33f5afe4f1c","v":0,"name":"crucible","level":Crucible runtime is spawned
144 30,"time":"2024-02-28T22:13:13.786277686Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
1452024-02-28T22:13:13.786ZINFOcrucible: The guest has requested activation with gen:2
1462024-02-28T22:13:13.786ZINFOcrucible: 7d2be74c-72cc-486f-817a-a33f5afe4f1c active request set session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1472024-02-28T22:13:13.786ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1482024-02-28T22:13:13.786ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1492024-02-28T22:13:13.786ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1502024-02-28T22:13:13.786ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1512024-02-28T22:13:13.786ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1522024-02-28T22:13:13.786ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1532024-02-28T22:13:13.786ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1542024-02-28T22:13:13.786ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1552024-02-28T22:13:13.786ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1562024-02-28T22:13:13.786ZINFOcrucible: ds_connection connected = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1572024-02-28T22:13:13.787ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1582024-02-28T22:13:13.787ZINFOcrucible: ds_connection connected = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1592024-02-28T22:13:13.787ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1602024-02-28T22:13:13.787ZINFOcrucible: [1] Transition from New to WaitActive = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1612024-02-28T22:13:13.787ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1622024-02-28T22:13:13.787ZINFOcrucible: [2] Transition from New to WaitActive = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1632024-02-28T22:13:13.787ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1642024-02-28T22:13:13.787ZINFOcrucible: [0] Transition from New to WaitActive = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1652024-02-28T22:13:13.788ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1662024-02-28T22:13:13.788ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1672024-02-28T22:13:13.788ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1682024-02-28T22:13:13.789ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1692024-02-28T22:13:13.789ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1702024-02-28T22:13:13.789ZINFOcrucible: Waiting for true more clients to be ready session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1712024-02-28T22:13:13.789ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1722024-02-28T22:13:13.789ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1732024-02-28T22:13:13.789ZINFOcrucible: Waiting for true more clients to be ready session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1742024-02-28T22:13:13.789ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1752024-02-28T22:13:13.789ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1762024-02-28T22:13:13.789ZINFOcrucible: [0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1772024-02-28T22:13:13.789ZINFOcrucible: [0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1782024-02-28T22:13:13.789ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1792024-02-28T22:13:13.789ZINFOcrucible: [1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1802024-02-28T22:13:13.789ZINFOcrucible: [1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1812024-02-28T22:13:13.789ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1822024-02-28T22:13:13.789ZINFOcrucible: [2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
183 {"msg":"[2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:13.789326838Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"downstairs","":"Activate successful
184 downstairs"Wait for a query_work_queue command to finish before sending IO
185 ,"session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
1862024-02-28T22:13:13.789ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1872024-02-28T22:13:13.789ZINFOcrucible: Max found gen is 2 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1882024-02-28T22:13:13.789ZINFOcrucible: Generation requested: 2 >= found:2 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1892024-02-28T22:13:13.789ZINFOcrucible: Next flush: 2 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1902024-02-28T22:13:13.789ZINFOcrucible: All extents match = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1912024-02-28T22:13:13.789ZINFOcrucible: No downstairs reconciliation required session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1922024-02-28T22:13:13.789ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1932024-02-28T22:13:13.789ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1942024-02-28T22:13:13.789ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1952024-02-28T22:13:13.789ZINFOcrucible: All required reconciliation work is completed session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1962024-02-28T22:13:13.789ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1972024-02-28T22:13:13.789ZINFOcrucible: 7d2be74c-72cc-486f-817a-a33f5afe4f1c is now active with session: 69c7fbb2-d5b2-4965-9fa4-441c4d60207e session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1982024-02-28T22:13:13.789ZINFOcrucible: Set Active after no reconciliation session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1992024-02-28T22:13:13.789ZINFOcrucible: The guest has finished waiting for activation with:2
200 Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
201 Loading write count information from file "alan"
202 Feb 28 22:13:13.840 INFO accepted connection, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
203 Main task has work to do, go find it
204 got dsc Stop(2)
205 stop 2
206 [8830] Got stop action so:false kr:false
207 Feb 28 22:13:13.843 INFO request completed, latency_us: 1013, response_code: 204, uri: /stop/cid/2, method: POST, req_id: a94b10ad-fa65-4866-8c1e-594923d39bbc, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
2082024-02-28T22:13:13.844ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2092024-02-28T22:13:13.844ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2102024-02-28T22:13:13.844ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2112024-02-28T22:13:13.844ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2122024-02-28T22:13:13.844ZINFOcrucible: [2] client re-new 0 jobs since flush 0 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2132024-02-28T22:13:13.844ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
214 Feb 28 22:13:13.845 INFO request completed, latency_us: 192, response_code: 200, uri: /state/cid/2, method: GET, req_id: 8031a778-181d-424d-a91a-139d48679f8f, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
215 [8830] Exited with: ExitStatus(unix_wait_status(9))
216 [8830][2] reports Exit
2172024-02-28T22:13:14.845ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2182024-02-28T22:13:14.845ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2192024-02-28T22:13:14.845ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2202024-02-28T22:13:14.845ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2212024-02-28T22:13:14.845ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2222024-02-28T22:13:14.845ZINFOcrucible: [2] client re-new 0 jobs since flush 0 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2232024-02-28T22:13:14.845ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2242024-02-28T22:13:15.846ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2252024-02-28T22:13:15.846ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2262024-02-28T22:13:15.846ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2272024-02-28T22:13:15.846ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2282024-02-28T22:13:15.846ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2292024-02-28T22:13:15.846ZINFOcrucible: [2] client re-new 0 jobs since flush 0 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2302024-02-28T22:13:15.846ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2312024-02-28T22:13:16.847ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2322024-02-28T22:13:16.847ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2332024-02-28T22:13:16.847ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2342024-02-28T22:13:16.847ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2352024-02-28T22:13:16.847ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2362024-02-28T22:13:16.847ZINFOcrucible: [2] client re-new 0 jobs since flush 0 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2372024-02-28T22:13:16.847ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
238 Feb 28 22:13:17.846 INFO request completed, latency_us: 191, response_code: 200, uri: /state/cid/2, method: GET, req_id: 0bd99fb2-6e0d-4bd5-a449-410b37f1eab6, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
239 001/300 Write block 2820 len 4096 data: 2
2402024-02-28T22:13:17.848ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2412024-02-28T22:13:17.848ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2422024-02-28T22:13:17.848ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2432024-02-28T22:13:17.848ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2442024-02-28T22:13:17.848ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2452024-02-28T22:13:17.848ZINFOcrucible: [2] client re-new 0 jobs since flush 0 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2462024-02-28T22:13:17.849ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
247 002/300 Read block 3800 len 32768
248 003/300 Read block 4865 len 28672
249 004/300 Read block 653 len 16384
250 005/300 Read block 4991 len 4096
251 006/300 Read block 1125 len 28672
252 007/300 Read block 4446 len 36864
253 008/300 Read block 3712 len 28672
254 009/300 Write block 2706 len 8192 data: 2 2
255 010/300 Write block 2646 len 40960 data: 2 2 2 2 2 2 2 2 2 2
256 011/300 Read block 2382 len 24576
257 012/300 Write block 1684 len 40960 data: 2 2 2 2 2 2 2 2 2 2
258 013/300 Write block 4510 len 32768 data: 2 2 2 2 2 2 2 2
259 014/300 Read block 3075 len 32768
260 015/300 Flush
261 016/300 Flush
262 017/300 Write block 4998 len 8192 data: 2 2
263 018/300 Read block 2641 len 32768
264 019/300 Flush
265 020/300 Read block 2582 len 36864
266 021/300 Write block 493 len 16384 data: 2 2 2 2
267 022/300 Read block 2844 len 40960
268 023/300 Write block 179 len 8192 data: 2 2
269 024/300 Write block 1528 len 36864 data: 2 2 2 2 2 2 2 2 2
270 025/300 Write block 2199 len 4096 data: 2
271 026/300 Write block 1678 len 16384 data: 2 2 2 2
272 027/300 Write block 1580 len 28672 data: 2 2 2 2 2 2 2
273 028/300 Read block 4239 len 36864
274 029/300 Read block 1217 len 36864
275 030/300 Read block 3539 len 32768
276 031/300 Write block 4138 len 28672 data: 2 2 2 2 2 2 2
277 032/300 Write block 1490 len 4096 data: 2
278 033/300 Write block 2551 len 20480 data: 2 2 2 2 2
279 034/300 Write block 3144 len 8192 data: 2 2
280 035/300 Read block 2335 len 12288
281 036/300 Write block 1615 len 8192 data: 2 2
282 037/300 Write block 1762 len 24576 data: 2 2 2 2 2 2
283 038/300 Write block 3083 len 4096 data: 2
284 039/300 Write block 3339 len 4096 data: 2
285 040/300 Flush
286 041/300 Read block 4038 len 20480
287 042/300 Read block 4522 len 8192
288 043/300 Read block 1811 len 40960
289 044/300 Read block 4654 len 16384
290 045/300 Write block 3220 len 28672 data: 2 2 2 2 2 2 2
291 046/300 Write block 4801 len 36864 data: 2 2 2 2 2 2 2 2 2
292 047/300 Read block 4354 len 24576
293 048/300 Write block 397 len 16384 data: 2 2 2 2
294 049/300 Read block 2925 len 36864
295 050/300 Write block 202 len 36864 data: 2 2 2 2 2 2 2 2 2
296 051/300 Read block 2705 len 12288
297 052/300 Write block 663 len 32768 data: 2 2 2 2 2 2 2 2
298 053/300 Read block 1196 len 4096
299 054/300 Write block 3899 len 4096 data: 2
300 055/300 Write block 3146 len 24576 data: 2 2 2 2 2 2
301 056/300 Write block 1824 len 32768 data: 2 2 2 2 2 2 2 2
302 057/300 Read block 264 len 12288
303 058/300 Read block 192 len 8192
304 059/300 Read block 4877 len 16384
305 060/300 Write block 948 len 40960 data: 2 2 2 2 2 2 2 2 2 2
306 061/300 Read block 1959 len 24576
307 062/300 Read block 327 len 8192
308 063/300 Write block 454 len 36864 data: 2 2 2 2 2 2 2 2 2
309 064/300 Read block 1478 len 24576
310 065/300 Read block 4955 len 4096
311 066/300 Write block 568 len 32768 data: 2 2 2 2 2 2 2 2
312 067/300 Write block 3729 len 28672 data: 2 2 2 2 2 2 2
313 068/300 Write block 2128 len 24576 data: 2 2 2 2 2 2
314 069/300 Write block 599 len 4096 data: 2
315 070/300 Flush
316 071/300 Write block 372 len 32768 data: 2 2 2 2 2 2 2 2
317 072/300 Write block 577 len 4096 data: 2
318 073/300 Write block 206 len 16384 data: 3 3 3 3
319 074/300 Write block 3060 len 28672 data: 2 2 2 2 2 2 2
320 075/300 Read block 902 len 40960
321 076/300 Write block 2415 len 24576 data: 2 2 2 2 2 2
322 077/300 Write block 202 len 16384 data: 3 3 3 3
323 078/300 Write block 1204 len 8192 data: 2 2
324 079/300 Read block 4370 len 20480
325 080/300 Write block 958 len 16384 data: 2 2 2 2
326 081/300 Read block 1244 len 20480
327 082/300 Write block 2244 len 20480 data: 2 2 2 2 2
328 083/300 Write block 3627 len 4096 data: 2
329 084/300 Write block 4238 len 16384 data: 2 2 2 2
330 085/300 Read block 4158 len 28672
331 086/300 Read block 877 len 16384
332 087/300 Write block 2835 len 40960 data: 2 2 2 2 2 2 2 2 2 2
333 088/300 Read block 108 len 28672
334 089/300 Read block 2341 len 40960
335 090/300 Write block 4504 len 4096 data: 2
336 091/300 Read block 3333 len 36864
337 092/300 Read block 4982 len 8192
338 093/300 Write block 4796 len 40960 data: 2 2 2 2 2 3 3 3 3 3
339 094/300 Write block 1329 len 28672 data: 2 2 2 2 2 2 2
340 095/300 Read block 1437 len 16384
341 096/300 Flush
342 097/300 Read block 872 len 40960
343 098/300 Read block 3476 len 12288
344 099/300 Read block 4566 len 16384
345 100/300 Read block 376 len 24576
346 101/300 Write block 3338 len 8192 data: 2 3
347 102/300 Write block 1125 len 16384 data: 2 2 2 2
348 103/300 Write block 2648 len 32768 data: 3 3 3 3 3 3 3 3
349 104/300 Write block 4355 len 8192 data: 2 2
350 105/300 Flush
351 106/300 Read block 4203 len 36864
352 107/300 Write block 3895 len 4096 data: 2
353 108/300 Read block 3972 len 28672
354 109/300 Read block 3101 len 20480
355 110/300 Read block 1066 len 8192
356 111/300 Write block 810 len 8192 data: 2 2
357 112/300 Read block 3931 len 28672
358 113/300 Read block 1792 len 16384
359 114/300 Read block 4835 len 24576
360 115/300 Read block 2610 len 28672
361 116/300 Read block 1218 len 32768
362 117/300 Read block 2162 len 40960
363 118/300 Write block 1414 len 16384 data: 2 2 2 2
364 119/300 Write block 2916 len 40960 data: 2 2 2 2 2 2 2 2 2 2
365 120/300 Read block 3122 len 40960
366 121/300 Read block 3430 len 36864
367 122/300 Read block 2084 len 28672
368 123/300 Read block 2942 len 20480
369 124/300 Read block 4897 len 12288
370 125/300 Write block 3942 len 24576 data: 2 2 2 2 2 2
371 126/300 Read block 2250 len 4096
372 127/300 Read block 574 len 16384
373 128/300 Write block 762 len 12288 data: 2 2 2
374 129/300 Write block 4582 len 8192 data: 2 2
375 130/300 Read block 713 len 12288
376 131/300 Read block 3494 len 32768
377 132/300 Read block 2567 len 24576
378 133/300 Read block 3643 len 20480
379 134/300 Read block 2847 len 12288
380 135/300 Write block 2033 len 20480 data: 2 2 2 2 2
381 136/300 Read block 893 len 40960
382 137/300 Write block 3289 len 36864 data: 2 2 2 2 2 2 2 2 2
383 138/300 Read block 1049 len 32768
384 139/300 Read block 3692 len 12288
385 140/300 Read block 4739 len 40960
386 141/300 Write block 4494 len 28672 data: 2 2 2 2 2 2 2
387 142/300 Write block 4431 len 24576 data: 2 2 2 2 2 2
388 143/300 Write block 2960 len 32768 data: 2 2 2 2 2 2 2 2
389 144/300 Write block 4615 len 24576 data: 2 2 2 2 2 2
390 145/300 Write block 4159 len 40960 data: 2 2 2 2 2 2 2 2 2 2
391 146/300 Read block 4973 len 4096
392 147/300 Read block 1635 len 4096
393 148/300 Write block 457 len 20480 data: 3 3 3 3 3
394 149/300 Read block 3335 len 24576
395 150/300 Write block 3955 len 28672 data: 2 2 2 2 2 2 2
396 151/300 Read block 3920 len 24576
397 152/300 Flush
398 153/300 Write block 959 len 32768 data: 3 3 3 2 2 2 2 2
399 154/300 Write block 4127 len 4096 data: 2
400 155/300 Read block 3447 len 8192
401 156/300 Write block 3394 len 24576 data: 2 2 2 2 2 2
402 157/300 Write block 3305 len 32768 data: 2 2 2 2 2 2 2 2
403 158/300 Read block 321 len 16384
404 159/300 Read block 2807 len 12288
405 160/300 Read block 4122 len 20480
406 161/300 Write block 2520 len 8192 data: 2 2
407 162/300 Read block 1965 len 24576
408 163/300 Write block 4758 len 24576 data: 2 2 2 2 2 2
409 164/300 Read block 3839 len 12288
410 165/300 Read block 912 len 8192
411 166/300 Read block 3191 len 24576
412 167/300 Read block 583 len 20480
413 168/300 Read block 4327 len 16384
414 169/300 Read block 85 len 40960
415 170/300 Flush
416 171/300 Write block 3894 len 4096 data: 2
417 172/300 Read block 3660 len 4096
418 173/300 Read block 379 len 36864
419 174/300 Read block 3211 len 28672
420 175/300 Write block 4068 len 24576 data: 2 2 2 2 2 2
421 176/300 Read block 3105 len 40960
422 177/300 Flush
423 178/300 Read block 1353 len 12288
424 179/300 Read block 4524 len 12288
425 180/300 Flush
426 181/300 Read block 1505 len 36864
427 182/300 Write block 2843 len 28672 data: 3 3 2 2 2 2 2
428 183/300 Flush
429 184/300 Read block 513 len 36864
430 185/300 Read block 1507 len 20480
431 186/300 Read block 3343 len 40960
432 187/300 Flush
433 188/300 Flush
434 189/300 Write block 2033 len 16384 data: 3 3 3 3
435 190/300 Write block 2441 len 40960 data: 2 2 2 2 2 2 2 2 2 2
436 191/300 Read block 2108 len 16384
437 192/300 Read block 3740 len 24576
438 193/300 Write block 1251 len 40960 data: 2 2 2 2 2 2 2 2 2 2
439 194/300 Write block 3521 len 32768 data: 2 2 2 2 2 2 2 2
440 195/300 Read block 1787 len 40960
441 196/300 Read block 4772 len 4096
442 197/300 Write block 4529 len 36864 data: 2 2 2 2 2 2 2 2 2
443 198/300 Flush
444 199/300 Write block 3550 len 24576 data: 2 2 2 2 2 2
445 200/300 Read block 2748 len 20480
446 201/300 Write block 2526 len 16384 data: 2 2 2 2
447 202/300 Read block 2242 len 4096
448 203/300 Flush
449 204/300 Read block 453 len 4096
450 205/300 Write block 4056 len 32768 data: 2 2 2 2 2 2 2 2
451 206/300 Flush
452 207/300 Write block 2949 len 36864 data: 2 2 2 2 2 2 2 2 2
453 208/300 Write block 997 len 32768 data: 2 2 2 2 2 2 2 2
454 209/300 Write block 3572 len 8192 data: 2 2
455 210/300 Write block 4672 len 4096 data: 2
456 211/300 Read block 4678 len 32768
457 212/300 Flush
458 213/300 Read block 416 len 4096
459 214/300 Read block 222 len 8192
460 215/300 Read block 2034 len 32768
461 216/300 Read block 2558 len 36864
462 217/300 Flush
463 218/300 Read block 2702 len 28672
464 219/300 Read block 4041 len 32768
465 220/300 Write block 3152 len 24576 data: 2 2 2 2 2 2
466 221/300 Flush
467 222/300 Read block 599 len 40960
468 223/300 Read block 3245 len 40960
469 224/300 Write block 2466 len 36864 data: 2 2 2 2 2 2 2 2 2
470 225/300 Read block 2766 len 28672
471 226/300 Read block 4333 len 16384
472 227/300 Flush
473 228/300 Read block 4621 len 24576
474 229/300 Read block 1923 len 16384
475 230/300 Write block 876 len 36864 data: 2 2 2 2 2 2 2 2 2
476 231/300 Write block 1715 len 36864 data: 2 2 2 2 2 2 2 2 2
477 232/300 Read block 267 len 40960
478 233/300 Read block 2140 len 32768
479 234/300 Read block 2231 len 32768
480 235/300 Write block 3740 len 32768 data: 2 2 2 2 2 2 2 2
481 236/300 Flush
482 237/300 Read block 1730 len 8192
483 238/300 Write block 3462 len 16384 data: 2 2 2 2
484 239/300 Read block 1258 len 40960
485 240/300 Read block 730 len 8192
486 241/300 Read block 4945 len 28672
487 242/300 Write block 3013 len 32768 data: 2 2 2 2 2 2 2 2
488 243/300 Flush
489 244/300 Read block 1102 len 36864
490 245/300 Read block 415 len 16384
491 246/300 Read block 4178 len 8192
492 247/300 Read block 1720 len 4096
493 248/300 Flush
494 249/300 Write block 276 len 20480 data: 2 2 2 2 2
495 250/300 Flush
496 251/300 Read block 3622 len 12288
497 252/300 Read block 2778 len 32768
498 253/300 Read block 926 len 20480
499 254/300 Flush
500 255/300 Read block 1685 len 16384
501 256/300 Write block 3699 len 28672 data: 2 2 2 2 2 2 2
502 257/300 Read block 3031 len 20480
503 258/300 Write block 1240 len 24576 data: 2 2 2 2 2 2
504 259/300 Write block 2088 len 24576 data: 2 2 2 2 2 2
505 260/300 Read block 4783 len 4096
506 261/300 Write block 1670 len 4096 data: 2
507 262/300 Write block 3613 len 12288 data: 2 2 2
508 263/300 Read block 930 len 24576
509 264/300 Write block 2671 len 16384 data: 2 2 2 2
510 265/300 Read block 1221 len 16384
511 266/300 Write block 623 len 28672 data: 2 2 2 2 2 2 2
512 267/300 Write block 3111 len 8192 data: 2 2
513 268/300 Flush
514 269/300 Write block 4548 len 20480 data: 2 2 2 2 2
515 270/300 Write block 3529 len 20480 data: 2 2 2 2 2
516 271/300 Flush
517 272/300 Write block 4188 len 28672 data: 2 2 2 2 2 2 2
518 273/300 Write block 1266 len 24576 data: 2 2 2 2 2 2
519 274/300 Read block 2741 len 40960
520 275/300 Read block 4066 len 4096
521 276/300 Write block 580 len 20480 data: 2 2 2 2 2
522 277/300 Write block 3953 len 40960 data: 2 2 3 3 3 3 3 3 3 2
523 278/300 Flush
524 279/300 Read block 4978 len 12288
525 280/300 Read block 1248 len 20480
526 281/300 Read block 1467 len 4096
527 282/300 Write block 3355 len 28672 data: 2 2 2 2 2 2 2
528 283/300 Read block 457 len 24576
529 284/300 Read block 1113 len 40960
530 285/300 Write block 3732 len 20480 data: 3 3 3 3 2
531 286/300 Read block 1833 len 24576
532 287/300 Write block 2850 len 8192 data: 2 2
533 288/300 Read block 1378 len 28672
534 289/300 Write block 4506 len 20480 data: 2 2 2 2 3
535 290/300 Write block 41 len 4096 data: 2
536 291/300 Read block 4418 len 4096
537 292/300 Write block 897 len 8192 data: 2 2
538 293/300 Read block 101 len 32768
539 294/300 Read block 2249 len 32768
540 295/300 Flush
541 296/300 Read block 3488 len 16384
542 297/300 Write block 4596 len 32768 data: 2 2 2 2 2 2 2 2
543 298/300 Write block 1637 len 32768 data: 2 2 2 2 2 2 2 2
544 299/300 Read block 1951 len 4096
545 300/300 Write block 3912 len 28672 data: 2 2 2 2 2 2 2
546 Main task has work to do, go find it
547 got dsc Start(2)
548 start 2
549 Feb 28 22:13:18.522 INFO request completed, latency_us: 248, response_code: 204, uri: /start/cid/2, method: POST, req_id: b3e8b091-aebf-4da1-a445-985ab645c73c, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
550 Replay: started 2, returned:Ok(())
551 ----------------------------------------------------------------
552 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
553 States: Active Active Offline
554 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
555 1 Acked 1000 Write 1 Done Done New false
556 2 Acked 1001 Read 8 Done Done New false
557 3 Acked 1002 Read 7 Done Done New false
558 4 Acked 1003 Read 4 Done Done New false
559 5 Acked 1004 Read 1 Done Done New false
560 6 Acked 1005 Read 7 Done Done New false
561 7 Acked 1006 Read 9 Done Done New false
562 8 Acked 1007 Read 7 Done Done New false
563 9 Acked 1008 Write 2 Done Done New false
564 10 Acked 1009 Write 10 Done Done New false
565 11 Acked 1010 Read 6 Done Done New false
566 12 Acked 1011 Write 10 Done Done New false
567 13 Acked 1012 Write 8 Done Done New false
568 14 Acked 1013 Read 8 Done Done New false
569 15 Acked 1014 Flush 0 Done Done New false
570 16 Acked 1015 Flush 0 Done Done New false
571 17 Acked 1016 Write 2 Done Done New false
572 18 Acked 1017 Read 8 Done Done New false
573 19 Acked 1018 Flush 0 Done Done New false
574 20 Acked 1019 Read 9 Done Done New false
575 21 Acked 1020 Write 4 Done Done New false
576 22 Acked 1021 Read 10 Done Done New false
577 23 Acked 1022 Write 2 Done Done New false
578 24 Acked 1023 Write 9 Done Done New false
579 25 Acked 1024 Write 1 Done Done New false
580 26 Acked 1025 Write 4 Done Done New false
581 27 Acked 1026 Write 7 Done Done New false
582 28 Acked 1027 Read 9 Done Done New false
583 29 Acked 1028 Read 9 Done Done New false
584 30 Acked 1029 Read 8 Done Done New false
585 31 Acked 1030 Write 7 Done Done New false
586 32 Acked 1031 Write 1 Done Done New false
587 33 Acked 1032 Write 5 Done Done New false
588 34 Acked 1033 Write 2 Done Done New false
589 35 Acked 1034 Read 3 Done Done New false
590 36 Acked 1035 Write 2 Done Done New false
591 37 Acked 1036 Write 6 Done Done New false
592 38 Acked 1037 Write 1 Done Done New false
593 39 Acked 1038 Write 1 Done Done New false
594 40 Acked 1039 Flush 0 Done Done New false
595 41 Acked 1040 Read 5 Done Done New false
596 42 Acked 1041 Read 2 Done Done New false
597 43 Acked 1042 Read 10 Done Done New false
598 44 Acked 1043 Read 4 Done Done New false
599 45 Acked 1044 Write 7 Done Done New false
600 46 Acked 1045 Write 9 Done Done New false
601 47 Acked 1046 Read 6 Done Done New false
602 48 Acked 1047 Write 4 Done Done New false
603 49 Acked 1048 Read 9 Done Done New false
604 50 Acked 1049 Write 9 Done Done New false
605 51 Acked 1050 Read 3 Done Done New false
606 52 Acked 1051 Write 8 Done Done New false
607 53 Acked 1052 Read 1 Done Done New false
608 54 Acked 1053 Write 1 Done Done New false
609 55 Acked 1054 Write 6 Done Done New false
610 56 Acked 1055 Write 8 Done Done New false
611 57 Acked 1056 Read 3 Done Done New false
612 58 Acked 1057 Read 2 Done Done New false
613 59 Acked 1058 Read 4 Done Done New false
614 60 Acked 1059 Write 10 Done Done New false
615 61 Acked 1060 Read 6 Done Done New false
616 62 Acked 1061 Read 2 Done Done New false
617 63 Acked 1062 Write 9 Done Done New false
618 64 Acked 1063 Read 6 Done Done New false
619 65 Acked 1064 Read 1 Done Done New false
620 66 Acked 1065 Write 8 Done Done New false
621 67 Acked 1066 Write 7 Done Done New false
622 68 Acked 1067 Write 6 Done Done New false
623 69 Acked 1068 Write 1 Done Done New false
624 70 Acked 1069 Flush 0 Done Done New false
625 71 Acked 1070 Write 8 Done Done New false
626 72 Acked 1071 Write 1 Done Done New false
627 73 Acked 1072 Write 4 Done Done New false
628 74 Acked 1073 Write 7 Done Done New false
629 75 Acked 1074 Read 10 Done Done New false
630 76 Acked 1075 Write 6 Done Done New false
631 77 Acked 1076 Write 4 Done Done New false
632 78 Acked 1077 Write 2 Done Done New false
633 79 Acked 1078 Read 5 Done Done New false
634 80 Acked 1079 Write 4 Done Done New false
635 81 Acked 1080 Read 5 Done Done New false
636 82 Acked 1081 Write 5 Done Done New false
637 83 Acked 1082 Write 1 Done Done New false
638 84 Acked 1083 Write 4 Done Done New false
639 85 Acked 1084 Read 7 Done Done New false
640 86 Acked 1085 Read 4 Done Done New false
641 87 Acked 1086 Write 10 Done Done New false
642 88 Acked 1087 Read 7 Done Done New false
643 89 Acked 1088 Read 10 Done Done New false
644 90 Acked 1089 Write 1 Done Done New false
645 91 Acked 1090 Read 9 Done Done New false
646 92 Acked 1091 Read 2 Done Done New false
647 93 Acked 1092 Write 10 Done Done New false
648 94 Acked 1093 Write 7 Done Done New false
649 95 Acked 1094 Read 4 Done Done New false
650 96 Acked 1095 Flush 0 Done Done New false
651 97 Acked 1096 Read 10 Done Done New false
652 98 Acked 1097 Read 3 Done Done New false
653 99 Acked 1098 Read 4 Done Done New false
654 100 Acked 1099 Read 6 Done Done New false
655 101 Acked 1100 Write 2 Done Done New false
656 102 Acked 1101 Write 4 Done Done New false
657 103 Acked 1102 Write 8 Done Done New false
658 104 Acked 1103 Write 2 Done Done New false
659 105 Acked 1104 Flush 0 Done Done New false
660 106 Acked 1105 Read 9 Done Done New false
661 107 Acked 1106 Write 1 Done Done New false
662 108 Acked 1107 Read 7 Done Done New false
663 109 Acked 1108 Read 5 Done Done New false
664 110 Acked 1109 Read 2 Done Done New false
665 111 Acked 1110 Write 2 Done Done New false
666 112 Acked 1111 Read 7 Done Done New false
667 113 Acked 1112 Read 4 Done Done New false
668 114 Acked 1113 Read 6 Done Done New false
669 115 Acked 1114 Read 7 Done Done New false
670 116 Acked 1115 Read 8 Done Done New false
671 117 Acked 1116 Read 10 Done Done New false
672 118 Acked 1117 Write 4 Done Done New false
673 119 Acked 1118 Write 10 Done Done New false
674 120 Acked 1119 Read 10 Done Done New false
675 121 Acked 1120 Read 9 Done Done New false
676 122 Acked 1121 Read 7 Done Done New false
677 123 Acked 1122 Read 5 Done Done New false
678 124 Acked 1123 Read 3 Done Done New false
679 125 Acked 1124 Write 6 Done Done New false
680 126 Acked 1125 Read 1 Done Done New false
681 127 Acked 1126 Read 4 Done Done New false
682 128 Acked 1127 Write 3 Done Done New false
683 129 Acked 1128 Write 2 Done Done New false
684 130 Acked 1129 Read 3 Done Done New false
685 131 Acked 1130 Read 8 Done Done New false
686 132 Acked 1131 Read 6 Done Done New false
687 133 Acked 1132 Read 5 Done Done New false
688 134 Acked 1133 Read 3 Done Done New false
689 135 Acked 1134 Write 5 Done Done New false
690 136 Acked 1135 Read 10 Done Done New false
691 137 Acked 1136 Write 9 Done Done New false
692 138 Acked 1137 Read 8 Done Done New false
693 139 Acked 1138 Read 3 Done Done New false
694 140 Acked 1139 Read 10 Done Done New false
695 141 Acked 1140 Write 7 Done Done New false
696 142 Acked 1141 Write 6 Done Done New false
697 143 Acked 1142 Write 8 Done Done New false
698 144 Acked 1143 Write 6 Done Done New false
699 145 Acked 1144 Write 10 Done Done New false
700 146 Acked 1145 Read 1 Done Done New false
701 147 Acked 1146 Read 1 Done Done New false
702 148 Acked 1147 Write 5 Done Done New false
703 149 Acked 1148 Read 6 Done Done New false
704 150 Acked 1149 Write 7 Done Done New false
705 151 Acked 1150 Read 6 Done Done New false
706 152 Acked 1151 Flush 0 Done Done New false
707 153 Acked 1152 Write 8 Done Done New false
708 154 Acked 1153 Write 1 Done Done New false
709 155 Acked 1154 Read 2 Done Done New false
710 156 Acked 1155 Write 6 Done Done New false
711 157 Acked 1156 Write 8 Done Done New false
712 158 Acked 1157 Read 4 Done Done New false
713 159 Acked 1158 Read 3 Done Done New false
714 160 Acked 1159 Read 5 Done Done New false
715 161 Acked 1160 Write 2 Done Done New false
716 162 Acked 1161 Read 6 Done Done New false
717 163 Acked 1162 Write 6 Done Done New false
718 164 Acked 1163 Read 3 Done Done New false
719 165 Acked 1164 Read 2 Done Done New false
720 166 Acked 1165 Read 6 Done Done New false
721 167 Acked 1166 Read 5 Done Done New false
722 168 Acked 1167 Read 4 Done Done New false
723 169 Acked 1168 Read 10 Done Done New false
724 170 Acked 1169 Flush 0 Done Done New false
725 171 Acked 1170 Write 1 Done Done New false
726 172 Acked 1171 Read 1 Done Done New false
727 173 Acked 1172 Read 9 Done Done New false
728 174 Acked 1173 Read 7 Done Done New false
729 175 Acked 1174 Write 6 Done Done New false
730 176 Acked 1175 Read 10 Done Done New false
731 177 Acked 1176 Flush 0 Done Done New false
732 178 Acked 1177 Read 3 Done Done New false
733 179 Acked 1178 Read 3 Done Done New false
734 180 Acked 1179 Flush 0 Done Done New false
735 181 Acked 1180 Read 9 Done Done New false
736 182 Acked 1181 Write 7 Done Done New false
737 183 Acked 1182 Flush 0 Done Done New false
738 184 Acked 1183 Read 9 Done Done New false
739 185 Acked 1184 Read 5 Done Done New false
740 186 Acked 1185 Read 10 Done Done New false
741 187 Acked 1186 Flush 0 Done Done New false
742 188 Acked 1187 Flush 0 Done Done New false
743 189 Acked 1188 Write 4 Done Done New false
744 190 Acked 1189 Write 10 Done Done New false
745 191 Acked 1190 Read 4 Done Done New false
746 192 Acked 1191 Read 6 Done Done New false
747 193 Acked 1192 Write 10 Done Done New false
748 194 Acked 1193 Write 8 Done Done New false
749 195 Acked 1194 Read 10 Done Done New false
750 196 Acked 1195 Read 1 Done Done New false
751 197 Acked 1196 Write 9 Done Done New false
752 198 Acked 1197 Flush 0 Done Done New false
753 199 Acked 1198 Write 6 Done Done New false
754 200 Acked 1199 Read 5 Done Done New false
755 201 Acked 1200 Write 4 Done Done New false
756 202 Acked 1201 Read 1 Done Done New false
757 203 Acked 1202 Flush 0 Done Done New false
758 204 Acked 1203 Read 1 Done Done New false
759 205 Acked 1204 Write 8 Done Done New false
760 206 Acked 1205 Flush 0 Done Done New false
761 207 Acked 1206 Write 9 Done Done New false
762 208 Acked 1207 Write 8 Done Done New false
763 209 Acked 1208 Write 2 Done Done New false
764 210 Acked 1209 Write 1 Done Done New false
765 211 Acked 1210 Read 8 Done Done New false
766 212 Acked 1211 Flush 0 Done Done New false
767 213 Acked 1212 Read 1 Done Done New false
768 214 Acked 1213 Read 2 Done Done New false
769 215 Acked 1214 Read 8 Done Done New false
770 216 Acked 1215 Read 9 Done Done New false
771 217 Acked 1216 Flush 0 Done Done New false
772 218 Acked 1217 Read 7 Done Done New false
773 219 Acked 1218 Read 8 Done Done New false
774 220 Acked 1219 Flush 0 Done Done New false
775 221 Acked 1220 Write 6 Done Done New false
776 222 Acked 1221 Flush 0 Done Done New false
777 223 Acked 1222 Read 10 Done Done New false
778 224 Acked 1223 Read 10 Done Done New false
779 225 Acked 1224 Write 9 Done Done New false
780 226 Acked 1225 Read 7 Done Done New false
781 227 Acked 1226 Read 4 Done Done New false
782 228 Acked 1227 Flush 0 Done Done New false
783 229 Acked 1228 Read 6 Done Done New false
784 230 Acked 1229 Read 4 Done Done New false
785 231 Acked 1230 Write 9 Done Done New false
786 232 Acked 1231 Write 9 Done Done New false
787 233 Acked 1232 Read 10 Done Done New false
788 234 Acked 1233 Read 8 Done Done New false
789 235 Acked 1234 Read 8 Done Done New false
790 236 Acked 1235 Write 8 Done Done New false
791 237 Acked 1236 Flush 0 Done Done New false
792 238 Acked 1237 Read 2 Done Done New false
793 239 Acked 1238 Write 4 Done Done New false
794 240 Acked 1239 Read 10 Done Done New false
795 241 Acked 1240 Read 2 Done Done New false
796 242 Acked 1241 Read 7 Done Done New false
797 243 Acked 1242 Write 8 Done Done New false
798 244 Acked 1243 Flush 0 Done Done New false
799 245 Acked 1244 Read 9 Done Done New false
800 246 Acked 1245 Read 4 Done Done New false
801 247 Acked 1246 Read 2 Done Done New false
802 248 Acked 1247 Read 1 Done Done New false
803 249 Acked 1248 Flush 0 Done Done New false
804 250 Acked 1249 Write 5 Done Done New false
805 251 Acked 1250 Flush 0 Done Done New false
806 252 Acked 1251 Read 3 Done Done New false
807 253 Acked 1252 Read 8 Done Done New false
808 254 Acked 1253 Read 5 Done Done New false
809 255 Acked 1254 Flush 0 Done Done New false
810 256 Acked 1255 Read 4 Done Done New false
811 257 Acked 1256 Write 7 Done Done New false
812 258 Acked 1257 Read 5 Done Done New false
813 259 Acked 1258 Write 6 Done Done New false
814 260 Acked 1259 Write 6 Done Done New false
815 261 Acked 1260 Read 1 Done Done New false
816 262 Acked 1261 Write 1 Done Done New false
817 263 Acked 1262 Write 3 Done Done New false
818 264 Acked 1263 Read 6 Done Done New false
819 265 Acked 1264 Write 4 Done Done New false
820 266 Acked 1265 Read 4 Done Done New false
821 267 Acked 1266 Write 7 Done Done New false
822 268 Acked 1267 Write 2 Done Done New false
823 269 Acked 1268 Flush 0 Done Done New false
824 270 Acked 1269 Write 5 Done Done New false
825 271 Acked 1270 Write 5 Done Done New false
826 272 Acked 1271 Flush 0 Done Done New false
827 273 Acked 1272 Write 7 Done Done New false
828 274 Acked 1273 Write 6 Done Done New false
829 275 Acked 1274 Read 10 Done Done New false
830 276 Acked 1275 Read 1 Done Done New false
831 277 Acked 1276 Write 5 Done Done New false
832 278 Acked 1277 Write 10 Done Done New false
833 279 Acked 1278 Flush 0 Done Done New false
834 280 Acked 1279 Read 3 Done Done New false
835 281 Acked 1280 Read 5 Done Done New false
836 282 Acked 1281 Read 1 Done Done New false
837 283 Acked 1282 Write 7 Done Done New false
838 284 Acked 1283 Read 6 Done Done New false
839 285 Acked 1284 Read 10 Done Done New false
840 286 Acked 1285 Write 5 Done Done New false
841 287 Acked 1286 Read 6 Done Done New false
842 288 Acked 1287 Write 2 Done Done New false
843 289 Acked 1288 Read 7 Done Done New false
844 290 Acked 1289 Write 5 Done Done New false
845 291 Acked 1290 Write 1 Done Done New false
846 292 Acked 1291 Read 1 Done Done New false
847 293 Acked 1292 Write 2 Done Done New false
848 294 Acked 1293 Read 8 Done Done New false
849 295 Acked 1294 Read 8 Done Done New false
850 296 Acked 1295 Flush 0 Done Done New false
851 297 Acked 1296 Read 4 Done Done New false
852 298 Acked 1297 Write 8 Done Done New false
853 299 Acked 1298 Write 8 Done Done New false
854 300 Acked 1299 Read 1 Done Done New false
855 301 Acked 1300 Write 7 Done Done New false
856 STATES DS:0 DS:1 DS:2 TOTAL
857 New 0 0 301 301
858 Sent 0 0 0 0
859 Done 301 301 0 602
860 Skipped 0 0 0 0
861 Error 0 0 0 0
862 Last Flush: 1295 1295 0
863 Downstairs last five completed:
864 Upstairs last five completed: 301 300 299 298 297
865 CLIENT: Up:0 ds:301 act:2
8662024-02-28T22:13:18.849ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8672024-02-28T22:13:18.849ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8682024-02-28T22:13:18.849ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8692024-02-28T22:13:18.850ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8702024-02-28T22:13:18.850ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8712024-02-28T22:13:18.850ZINFOcrucible: [2] client re-new 302 jobs since flush 0 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8722024-02-28T22:13:18.850ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8732024-02-28T22:13:19.851ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8742024-02-28T22:13:19.851ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8752024-02-28T22:13:19.851ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8762024-02-28T22:13:19.851ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8772024-02-28T22:13:19.851ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8782024-02-28T22:13:19.851ZINFOcrucible: [2] client re-new 302 jobs since flush 0 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8792024-02-28T22:13:19.851ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
880 [8830] got start action
881 [8830] I am going to restart
882 Starting downstairs at port 8830
883 Make output file at "/tmp/dsc/downstairs-8830.txt"
884 [8830][2] reports Starting
885 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1165)
886 [8830][2] reports Running
8872024-02-28T22:13:20.852ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8882024-02-28T22:13:20.852ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8892024-02-28T22:13:20.854ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8902024-02-28T22:13:20.854ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8912024-02-28T22:13:20.854ZINFOcrucible: send last flush ID to this DS: 0 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8922024-02-28T22:13:20.854ZINFOcrucible: Replied this last flush ID: 0 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8932024-02-28T22:13:20.854ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8942024-02-28T22:13:20.854ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8952024-02-28T22:13:20.854ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8962024-02-28T22:13:20.854ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
897 ----------------------------------------------------------------
898 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
899 Downstairs last five completed: 1301 1300 1299 1298 1297
900 Upstairs last five completed: 302 301 300 299 298
901 CLIENT: Up:0 ds:0 act:3
902 Replay: All jobs finished, all DS active.
903 Main task has work to do, go find it
904 got dsc Stop(2)
905 stop 2
906 Feb 28 22:13:22.526 INFO request completed, latency_us: 336, response_code: 204, uri:[8830] Got stop action so:false kr:false
907 /stop/cid/2, method: POST, req_id: 07fed78e-da1d-41a8-8925-590d18031e51, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
9082024-02-28T22:13:22.526ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
909 Feb 28 22:13:22.526 INFO request completed, latency_us: 144, response_code: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-28T22:13:22.526794065Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
910 200, uri:{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-28T22:13:22.526836244Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
911 /state/cid/2, method: GET, req_id:{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:22.526877692Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
912 f62f0477-a8f0-4a95-a981-73da69c4be00, remote_addr{"msg":"[2] client re-new 0 jobs since flush 1301","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:22.526904791Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
913 : 127.0.0.1:36617, local_addr: 127.0.0.{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:22.52692579Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
914 1:9998
915 [8830] Exited with: ExitStatus(unix_wait_status(9))
916 [8830][2] reports Exit
9172024-02-28T22:13:23.528ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9182024-02-28T22:13:23.528ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9192024-02-28T22:13:23.528ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9202024-02-28T22:13:23.528ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9212024-02-28T22:13:23.528ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9222024-02-28T22:13:23.528ZINFOcrucible: [2] client re-new 0 jobs since flush 1301 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9232024-02-28T22:13:23.528ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9242024-02-28T22:13:24.529ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9252024-02-28T22:13:24.530ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9262024-02-28T22:13:24.530ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9272024-02-28T22:13:24.530ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9282024-02-28T22:13:24.530ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9292024-02-28T22:13:24.530ZINFOcrucible: [2] client re-new 0 jobs since flush 1301 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9302024-02-28T22:13:24.530ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9312024-02-28T22:13:25.531ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9322024-02-28T22:13:25.531ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9332024-02-28T22:13:25.531ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9342024-02-28T22:13:25.531ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9352024-02-28T22:13:25.531ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9362024-02-28T22:13:25.531ZINFOcrucible: [2] client re-new 0 jobs since flush 1301 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9372024-02-28T22:13:25.531ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
938 Feb 28 22:13:26.529 INFO request completed, latency_us: 292, response_code: 200, uri: /state/cid/2, method: GET, req_id: 5cd4b44e-74bb-4f8f-8070-72a8c7e9e48f, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
939 001/300 Read block 3392 len 36864
940 002/300 Write block 3632 len 16384 data: 2 2 2 2
941 003/300 Read block 2713 len 8192
9422024-02-28T22:13:26.532ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9432024-02-28T22:13:26.532ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9442024-02-28T22:13:26.532ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9452024-02-28T22:13:26.532ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9462024-02-28T22:13:26.532ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9472024-02-28T22:13:26.532ZINFOcrucible: [2] client re-new 3 jobs since flush 1301 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9482024-02-28T22:13:26.532ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
949 004/300 Read block 4311 len 12288
950 005/300 Read block 1658 len 4096
951 006/300 Read block 1266 len 12288
952 007/300 Write block 3884 len 36864 data: 2 2 2 2 2 2 2 2 2
953 008/300 Write block 1892 len 12288 data: 2 2 2
954 009/300 Write block 4165 len 32768 data: 3 3 3 3 2 2 2 2
955 010/300 Write block 4211 len 12288 data: 2 2 2
956 011/300 Read block 4771 len 20480
957 012/300 Write block 1953 len 40960 data: 2 2 2 2 2 2 2 2 2 2
958 013/300 Write block 2470 len 12288 data: 3 3 3
959 014/300 Write block 1111 len 36864 data: 2 2 2 2 2 2 2 2 2
960 015/300 Read block 3113 len 28672
961 016/300 Read block 514 len 20480
962 017/300 Flush
963 018/300 Write block 1094 len 4096 data: 2
964 019/300 Write block 4629 len 4096 data: 2
965 020/300 Write block 842 len 12288 data: 2 2 2
966 021/300 Flush
967 022/300 Write block 779 len 36864 data: 2 2 2 2 2 2 2 2 2
968 023/300 Read block 2586 len 28672
969 024/300 Write block 2881 len 40960 data: 2 2 2 2 2 2 2 2 2 2
970 025/300 Read block 1458 len 20480
971 026/300 Write block 1056 len 36864 data: 2 2 2 2 2 2 2 2 2
972 027/300 Write block 1227 len 20480 data: 2 2 2 2 2
973 028/300 Write block 2677 len 36864 data: 2 2 2 2 2 2 2 2 2
974 029/300 Read block 1815 len 24576
975 030/300 Flush
976 031/300 Write block 423 len 12288 data: 2 2 2
977 032/300 Write block 3772 len 4096 data: 2
978 033/300 Read block 74 len 4096
979 034/300 Flush
980 035/300 Read block 3522 len 24576
981 036/300 Write block 674 len 20480 data: 2 2 2 2 2
982 037/300 Write block 3533 len 24576 data: 3 2 2 2 2 2
983 038/300 Read block 756 len 8192
984 039/300 Read block 1153 len 16384
985 040/300 Read block 2642 len 24576
986 041/300 Read block 3298 len 20480
987 042/300 Read block 4731 len 20480
988 043/300 Flush
989 044/300 Read block 2881 len 8192
990 045/300 Read block 3174 len 40960
991 046/300 Write block 3628 len 36864 data: 2 2 2 2 3 3 3 3 2
992 047/300 Read block 4306 len 24576
993 048/300 Read block 4723 len 8192
994 049/300 Write block 1798 len 24576 data: 2 2 2 2 2 2
995 050/300 Write block 3869 len 20480 data: 2 2 2 2 2
996 051/300 Flush
997 052/300 Write block 1819 len 8192 data: 2 2
998 053/300 Write block 1516 len 12288 data: 2 2 2
999 054/300 Write block 160 len 24576 data: 2 2 2 2 2 2
1000 055/300 Read block 829 len 24576
1001 056/300 Read block 279 len 40960
1002 057/300 Write block 1559 len 20480 data: 2 2 2 2 2
1003 058/300 Write block 1673 len 24576 data: 2 2 2 2 2 3
1004 059/300 Flush
1005 060/300 Write block 4030 len 36864 data: 2 2 2 2 2 2 2 2 2
1006 061/300 Read block 1093 len 20480
1007 062/300 Read block 3125 len 36864
1008 063/300 Write block 4563 len 12288 data: 2 2 2
1009 064/300 Write block 4498 len 32768 data: 3 3 3 2 2 2 3 2
1010 065/300 Read block 28 len 36864
1011 066/300 Read block 1396 len 28672
1012 067/300 Read block 2127 len 28672
1013 068/300 Read block 4096 len 36864
1014 069/300 Write block 484 len 20480 data: 2 2 2 2 2
1015 070/300 Read block 3308 len 28672
1016 071/300 Write block 3715 len 4096 data: 2
1017 072/300 Write block 1457 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1018 073/300 Read block 3938 len 24576
1019 074/300 Read block 1375 len 4096
1020 075/300 Read block 3284 len 4096
1021 076/300 Write block 2822 len 20480 data: 2 2 2 2 2
1022 077/300 Write block 3513 len 36864 data: 2 2 2 2 2 2 2 2 3
1023 078/300 Read block 660 len 28672
1024 079/300 Write block 1137 len 32768 data: 2 2 2 2 2 2 2 2
1025 080/300 Read block 1781 len 32768
1026 081/300 Write block 1022 len 36864 data: 2 2 2 2 2 2 2 2 2
1027 082/300 Read block 1705 len 40960
1028 083/300 Read block 4410 len 28672
1029 084/300 Write block 3568 len 4096 data: 2
1030 085/300 Write block 655 len 28672 data: 2 2 2 2 2 2 2
1031 086/300 Write block 4819 len 24576 data: 2 2 2 2 2 2
1032 087/300 Read block 4299 len 20480
1033 088/300 Read block 2698 len 28672
1034 089/300 Write block 2359 len 20480 data: 2 2 2 2 2
1035 090/300 Write block 2882 len 12288 data: 3 3 3
1036 091/300 Read block 4216 len 24576
1037 092/300 Read block 2713 len 28672
1038 093/300 Read block 1218 len 20480
1039 094/300 Read block 998 len 4096
1040 095/300 Flush
1041 096/300 Read block 392 len 32768
1042 097/300 Read block 2035 len 20480
1043 098/300 Read block 4382 len 36864
1044 099/300 Read block 2552 len 28672
1045 100/300 Write block 453 len 28672 data: 2 3 3 3 4 4 4
1046 101/300 Flush
1047 102/300 Write block 1048 len 40960 data: 2 2 2 2 2 2 2 2 3 3
1048 103/300 Read block 1052 len 36864
1049 104/300 Flush
1050 105/300 Read block 2749 len 16384
1051 106/300 Write block 886 len 4096 data: 2
1052 107/300 Read block 3423 len 8192
1053 108/300 Write block 3721 len 36864 data: 2 2 2 2 2 2 2 2 3
1054 109/300 Write block 651 len 16384 data: 2 2 2 2
1055 110/300 Write block 1636 len 36864 data: 2 3 3 3 3 3 3 3 3
1056 111/300 Read block 1608 len 12288
1057 112/300 Write block 3749 len 32768 data: 2 2 2 2 2 2 2 2
1058 113/300 Read block 2092 len 20480
1059 114/300 Write block 1112 len 4096 data: 3
1060 115/300 Read block 2273 len 40960
1061 116/300 Read block 522 len 36864
1062 117/300 Read block 3151 len 32768
1063 118/300 Read block 4544 len 4096
1064 119/300 Write block 3566 len 36864 data: 2 2 3 2 2 2 3 3 2
1065 120/300 Write block 3395 len 20480 data: 3 3 3 3 3
1066 121/300 Read block 2139 len 32768
1067 122/300 Read block 660 len 32768
1068 123/300 Read block 2744 len 4096
1069 124/300 Read block 4703 len 8192
1070 125/300 Read block 3334 len 32768
1071 126/300 Write block 4347 len 4096 data: 2
1072 127/300 Read block 884 len 16384
1073 128/300 Read block 2267 len 8192
1074 129/300 Write block 2600 len 12288 data: 2 2 2
1075 130/300 Read block 3544 len 24576
1076 131/300 Flush
1077 132/300 Read block 978 len 28672
1078 133/300 Read block 819 len 20480
1079 134/300 Write block 2990 len 4096 data: 2
1080 135/300 Read block 3596 len 16384
1081 136/300 Read block 4064 len 12288
1082 137/300 Read block 902 len 40960
1083 138/300 Read block 3043 len 28672
1084 139/300 Write block 2660 len 8192 data: 2 2
1085 140/300 Flush
1086 141/300 Read block 798 len 40960
1087 142/300 Read block 2159 len 32768
1088 143/300 Read block 4765 len 36864
1089 144/300 Flush
1090 145/300 Write block 3745 len 24576 data: 3 3 3 2 3 3
1091 146/300 Write block 977 len 24576 data: 2 2 2 2 2 2
1092 147/300 Write block 2368 len 20480 data: 2 2 2 2 2
1093 148/300 Write block 1629 len 24576 data: 2 2 2 2 2 2
1094 149/300 Write block 2437 len 12288 data: 2 2 2
1095 150/300 Read block 4322 len 4096
1096 151/300 Write block 1053 len 20480 data: 3 3 3 4 4
1097 152/300 Write block 329 len 8192 data: 2 2
1098 153/300 Write block 2114 len 24576 data: 2 2 2 2 2 2
1099 154/300 Read block 3645 len 24576
1100 155/300 Write block 389 len 24576 data: 2 2 2 2 2 2
1101 156/300 Read block 1509 len 12288
1102 157/300 Read block 4249 len 32768
1103 158/300 Write block 3371 len 28672 data: 2 2 2 2 2 2 2
1104 159/300 Flush
1105 160/300 Read block 4122 len 16384
1106 161/300 Write block 1238 len 28672 data: 2 2 3 3 3 3 3
1107 162/300 Read block 2313 len 16384
1108 163/300 Read block 2790 len 4096
1109 164/300 Flush
1110 165/300 Flush
1111 166/300 Write block 1569 len 20480 data: 2 2 2 2 2
1112 167/300 Read block 4412 len 24576
1113 168/300 Read block 4452 len 40960
1114 169/300 Read block 2284 len 36864
1115 170/300 Write block 4140 len 32768 data: 3 3 3 3 3 2 2 2
1116 171/300 Read block 2612 len 20480
1117 172/300 Write block 4070 len 8192 data: 3 3
1118 173/300 Read block 4288 len 32768
1119 174/300 Read block 1098 len 40960
1120 175/300 Write block 1536 len 24576 data: 3 2 2 2 2 2
1121 176/300 Write block 3499 len 36864 data: 2 2 2 2 2 2 2 2 2
1122 177/300 Read block 3147 len 4096
1123 178/300 Read block 2623 len 20480
1124 179/300 Write block 2816 len 36864 data: 2 2 2 2 3 2 3 3 3
1125 180/300 Write block 1327 len 12288 data: 2 2 3
1126 181/300 Read block 1915 len 8192
1127 182/300 Write block 3548 len 16384 data: 2 2 3 3
1128 183/300 Read block 2239 len 12288
1129 184/300 Write block 1081 len 20480 data: 2 2 2 2 2
1130 185/300 Write block 3543 len 40960 data: 2 2 2 2 2 3 3 4 4 3
1131 186/300 Flush
1132 187/300 Read block 43 len 24576
1133 188/300 Read block 4379 len 20480
1134 189/300 Write block 2416 len 8192 data: 3 3
1135 190/300 Write block 1851 len 8192 data: 2 2
1136 191/300 Read block 2681 len 24576
1137 192/300 Flush
1138 193/300 Write block 2886 len 16384 data: 3 3 3 3
1139 194/300 Read block 629 len 8192
1140 195/300 Write block 1623 len 32768 data: 2 2 2 2 2 2 3 3
1141 196/300 Read block 3477 len 16384
1142 197/300 Read block 46 len 20480
1143 198/300 Read block 3681 len 16384
1144 199/300 Read block 3675 len 36864
1145 200/300 Read block 572 len 40960
1146 201/300 Flush
1147 202/300 Read block 1241 len 12288
1148 203/300 Write block 498 len 20480 data: 2 2 2 2 2
1149 204/300 Flush
1150 205/300 Read block 2545 len 32768
1151 206/300 Read block 1989 len 40960
1152 207/300 Read block 12 len 32768
1153 208/300 Flush
1154 209/300 Write block 3970 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1155 210/300 Write block 2051 len 12288 data: 2 2 2
1156 211/300 Write block 3062 len 4096 data: 3
1157 212/300 Read block 1706 len 40960
1158 213/300 Read block 4490 len 28672
1159 214/300 Write block 1771 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1160 215/300 Write block 3695 len 32768 data: 2 2 2 2 3 3 3 3
1161 216/300 Write block 3953 len 24576 data: 3 3 4 4 4 4
1162 217/300 Write block 2762 len 24576 data: 2 2 2 2 2 2
1163 218/300 Read block 154 len 16384
1164 219/300 Flush
1165 220/300 Read block 2402 len 36864
1166 221/300 Write block 4748 len 4096 data: 2
1167 222/300 Read block 4247 len 4096
1168 223/300 Read block 942 len 12288
1169 224/300 Write block 1230 len 40960 data: 3 3 2 2 2 2 2 2 3 3
1170 225/300 Read block 2666 len 28672
1171 226/300 Flush
1172 227/300 Write block 4520 len 4096 data: 2
1173 228/300 Read block 2412 len 32768
1174 229/300 Read block 2801 len 36864
1175 230/300 Read block 4173 len 24576
1176 231/300 Write block 2726 len 28672 data: 2 2 2 2 2 2 2
1177 232/300 Write block 2395 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1178 233/300 Read block 143 len 36864
1179 234/300 Write block 3876 len 36864 data: 2 2 2 2 2 2 2 2 3
1180 235/300 Read block 1413 len 8192
1181 236/300 Write block 1085 len 20480 data: 3 2 2 2 2
1182 237/300 Read block 4818 len 28672
1183 238/300 Read block 2463 len 12288
1184 239/300 Write block 4418 len 16384 data: 2 2 2 2
1185 240/300 Read block 4485 len 40960
1186 241/300 Read block 108 len 40960
1187 242/300 Read block 4317 len 36864
1188 243/300 Write block 4606 len 28672 data: 2 2 2 2 2 2 2
1189 244/300 Read block 1230 len 24576
1190 245/300 Write block 3901 len 20480 data: 2 2 2 2 2
1191 246/300 Read block 364 len 16384
1192 247/300 Write block 1761 len 40960 data: 2 3 3 3 3 3 3 2 2 2
1193 248/300 Write block 3389 len 32768 data: 2 2 2 2 2 3 4 4
1194 249/300 Flush
1195 250/300 Flush
1196 251/300 Read block 1893 len 28672
1197 252/300 Write block 492 len 28672 data: 2 3 3 3 3 2 3
1198 253/300 Read block 2888 len 24576
1199 254/300 Write block 1191 len 8192 data: 2 2
1200 255/300 Read block 962 len 12288
1201 256/300 Flush
1202 257/300 Read block 4178 len 20480
1203 258/300 Write block 1659 len 20480 data: 2 2 2 2 2
1204 259/300 Read block 564 len 8192
1205 260/300 Flush
1206 261/300 Read block 922 len 16384
1207 262/300 Read block 3777 len 12288
1208 263/300 Read block 2481 len 36864
1209 264/300 Write block 1103 len 20480 data: 2 2 2 2 2
1210 265/300 Read block 4232 len 32768
1211 266/300 Read block 4341 len 24576
1212 267/300 Read block 1327 len 28672
1213 268/300 Write block 2871 len 12288 data: 2 2 2
1214 269/300 Read block 2545 len 12288
1215 270/300 Write block 3178 len 12288 data: 2 2 2
1216 271/300 Read block 2082 len 12288
1217 272/300 Read block 3406 len 36864
1218 273/300 Write block 1673 len 24576 data: 3 3 3 3 3 4
1219 274/300 Read block 720 len 36864
1220 275/300 Write block 1332 len 12288 data: 3 3 3
1221 276/300 Read block 3463 len 16384
1222 277/300 Write block 1860 len 28672 data: 2 2 2 2 2 2 2
1223 278/300 Write block 1064 len 20480 data: 3 2 2 2 2
1224 279/300 Write block 1762 len 16384 data: 4 4 4 4
1225 280/300 Read block 685 len 28672
1226 281/300 Write block 4769 len 24576 data: 2 2 2 2 2 2
1227 282/300 Read block 547 len 20480
1228 283/300 Read block 395 len 24576
1229 284/300 Read block 2340 len 24576
1230 285/300 Read block 3841 len 8192
1231 286/300 Write block 987 len 24576 data: 2 2 2 2 2 2
1232 287/300 Write block 4422 len 8192 data: 2 2
1233 288/300 Read block 1074 len 28672
1234 289/300 Read block 4103 len 32768
1235 290/300 Flush
1236 291/300 Write block 3366 len 24576 data: 2 2 2 2 2 3
1237 292/300 Read block 4210 len 8192
1238 293/300 Read block 4526 len 24576
1239 294/300 Write block 3981 len 36864 data: 2 2 2 2 2 2 2 2 2
1240 295/300 Write block 1663 len 4096 data: 3
1241 296/300 Write block 4938 len 32768 data: 2 2 2 2 2 2 2 2
1242 297/300 Read block 142 len 4096
1243 298/300 Write block 474 len 12288 data: 2 2 2
1244 299/300 Write block 4084 len 32768 data: 2 2 2 2 2 2 2 2
1245 300/300 Write block 3903 len 32768 data: 3 3 3 2 2 2 2 2
1246 Main task has work to do, go find it
1247 got dsc Start(2)
1248 start 2
1249 Feb 28 22:13:27.272 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/2, method: POST, req_id: 7f7a8813-8fd3-4a84-a897-51fbc8c8b1db, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
1250 Replay: started 2, returned:Ok(())
1251 ----------------------------------------------------------------
1252 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1253 States: Active Active Offline
1254 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
1255 303 Acked 1302 Read 9 Done Done New true
1256 304 Acked 1303 Write 4 Done Done New true
1257 305 Acked 1304 Read 2 Done Done New true
1258 306 Acked 1305 Read 3 Done Done New false
1259 307 Acked 1306 Read 1 Done Done New false
1260 308 Acked 1307 Read 3 Done Done New false
1261 309 Acked 1308 Write 9 Done Done New false
1262 310 Acked 1309 Write 3 Done Done New false
1263 311 Acked 1310 Write 8 Done Done New false
1264 312 Acked 1311 Write 3 Done Done New false
1265 313 Acked 1312 Read 5 Done Done New false
1266 314 Acked 1313 Write 10 Done Done New false
1267 315 Acked 1314 Write 3 Done Done New false
1268 316 Acked 1315 Write 9 Done Done New false
1269 317 Acked 1316 Read 7 Done Done New false
1270 318 Acked 1317 Read 5 Done Done New false
1271 319 Acked 1318 Flush 0 Done Done New false
1272 320 Acked 1319 Write 1 Done Done New false
1273 321 Acked 1320 Write 1 Done Done New false
1274 322 Acked 1321 Write 3 Done Done New false
1275 323 Acked 1322 Flush 0 Done Done New false
1276 324 Acked 1323 Write 9 Done Done New false
1277 325 Acked 1324 Read 7 Done Done New false
1278 326 Acked 1325 Write 10 Done Done New false
1279 327 Acked 1326 Read 5 Done Done New false
1280 328 Acked 1327 Write 9 Done Done New false
1281 329 Acked 1328 Write 5 Done Done New false
1282 330 Acked 1329 Write 9 Done Done New false
1283 331 Acked 1330 Read 6 Done Done New false
1284 332 Acked 1331 Flush 0 Done Done New false
1285 333 Acked 1332 Write 3 Done Done New false
1286 334 Acked 1333 Write 1 Done Done New false
1287 335 Acked 1334 Read 1 Done Done New false
1288 336 Acked 1335 Flush 0 Done Done New false
1289 337 Acked 1336 Read 6 Done Done New false
1290 338 Acked 1337 Write 5 Done Done New false
1291 339 Acked 1338 Write 6 Done Done New false
1292 340 Acked 1339 Read 2 Done Done New false
1293 341 Acked 1340 Read 4 Done Done New false
1294 342 Acked 1341 Read 6 Done Done New false
1295 343 Acked 1342 Read 5 Done Done New false
1296 344 Acked 1343 Read 5 Done Done New false
1297 345 Acked 1344 Flush 0 Done Done New false
1298 346 Acked 1345 Read 2 Done Done New false
1299 347 Acked 1346 Read 10 Done Done New false
1300 348 Acked 1347 Write 9 Done Done New false
1301 349 Acked 1348 Read 6 Done Done New false
1302 350 Acked 1349 Read 2 Done Done New false
1303 351 Acked 1350 Write 6 Done Done New false
1304 352 Acked 1351 Write 5 Done Done New false
1305 353 Acked 1352 Flush 0 Done Done New false
1306 354 Acked 1353 Write 2 Done Done New false
1307 355 Acked 1354 Write 3 Done Done New false
1308 356 Acked 1355 Write 6 Done Done New false
1309 357 Acked 1356 Read 6 Done Done New false
1310 358 Acked 1357 Read 10 Done Done New false
1311 359 Acked 1358 Write 5 Done Done New false
1312 360 Acked 1359 Write 6 Done Done New false
1313 361 Acked 1360 Flush 0 Done Done New false
1314 362 Acked 1361 Write 9 Done Done New false
1315 363 Acked 1362 Read 5 Done Done New false
1316 364 Acked 1363 Read 9 Done Done New false
1317 365 Acked 1364 Write 3 Done Done New false
1318 366 Acked 1365 Write 8 Done Done New false
1319 367 Acked 1366 Read 9 Done Done New false
1320 368 Acked 1367 Read 7 Done Done New false
1321 369 Acked 1368 Read 7 Done Done New false
1322 370 Acked 1369 Read 9 Done Done New false
1323 371 Acked 1370 Write 5 Done Done New false
1324 372 Acked 1371 Read 7 Done Done New false
1325 373 Acked 1372 Write 1 Done Done New false
1326 374 Acked 1373 Write 10 Done Done New false
1327 375 Acked 1374 Read 6 Done Done New false
1328 376 Acked 1375 Read 1 Done Done New false
1329 377 Acked 1376 Read 1 Done Done New false
1330 378 Acked 1377 Write 5 Done Done New false
1331 379 Acked 1378 Write 9 Done Done New false
1332 380 Acked 1379 Read 7 Done Done New false
1333 381 Acked 1380 Write 8 Done Done New false
1334 382 Acked 1381 Read 8 Done Done New false
1335 383 Acked 1382 Write 9 Done Done New false
1336 384 Acked 1383 Read 10 Done Done New false
1337 385 Acked 1384 Read 7 Done Done New false
1338 386 Acked 1385 Write 1 Done Done New false
1339 387 Acked 1386 Write 7 Done Done New false
1340 388 Acked 1387 Write 6 Done Done New false
1341 389 Acked 1388 Read 5 Done Done New false
1342 390 Acked 1389 Read 7 Done Done New false
1343 391 Acked 1390 Write 5 Done Done New false
1344 392 Acked 1391 Write 3 Done Done New false
1345 393 Acked 1392 Read 6 Done Done New false
1346 394 Acked 1393 Read 7 Done Done New false
1347 395 Acked 1394 Read 5 Done Done New false
1348 396 Acked 1395 Read 1 Done Done New false
1349 397 Acked 1396 Flush 0 Done Done New false
1350 398 Acked 1397 Read 8 Done Done New false
1351 399 Acked 1398 Flush 0 Done Done New false
1352 400 Acked 1399 Read 5 Done Done New false
1353 401 Acked 1400 Read 9 Done Done New false
1354 402 Acked 1401 Read 7 Done Done New false
1355 403 Acked 1402 Write 7 Done Done New false
1356 404 Acked 1403 Flush 0 Done Done New false
1357 405 Acked 1404 Write 10 Done Done New false
1358 406 Acked 1405 Read 9 Done Done New false
1359 407 Acked 1406 Flush 0 Done Done New false
1360 408 Acked 1407 Read 4 Done Done New false
1361 409 Acked 1408 Write 1 Done Done New false
1362 410 Acked 1409 Read 2 Done Done New false
1363 411 Acked 1410 Write 9 Done Done New false
1364 412 Acked 1411 Write 4 Done Done New false
1365 413 Acked 1412 Write 9 Done Done New false
1366 414 Acked 1413 Read 3 Done Done New false
1367 415 Acked 1414 Write 8 Done Done New false
1368 416 Acked 1415 Read 5 Done Done New false
1369 417 Acked 1416 Write 1 Done Done New false
1370 418 Acked 1417 Read 10 Done Done New false
1371 419 Acked 1418 Read 9 Done Done New false
1372 420 Acked 1419 Read 8 Done Done New false
1373 421 Acked 1420 Read 1 Done Done New false
1374 422 Acked 1421 Write 9 Done Done New false
1375 423 Acked 1422 Write 5 Done Done New false
1376 424 Acked 1423 Read 8 Done Done New false
1377 425 Acked 1424 Read 8 Done Done New false
1378 426 Acked 1425 Read 1 Done Done New false
1379 427 Acked 1426 Read 2 Done Done New false
1380 428 Acked 1427 Read 8 Done Done New false
1381 429 Acked 1428 Write 1 Done Done New false
1382 430 Acked 1429 Read 4 Done Done New false
1383 431 Acked 1430 Read 2 Done Done New false
1384 432 Acked 1431 Write 3 Done Done New false
1385 433 Acked 1432 Read 6 Done Done New false
1386 434 Acked 1433 Flush 0 Done Done New false
1387 435 Acked 1434 Read 7 Done Done New false
1388 436 Acked 1435 Read 5 Done Done New false
1389 437 Acked 1436 Write 1 Done Done New false
1390 438 Acked 1437 Read 4 Done Done New false
1391 439 Acked 1438 Read 3 Done Done New false
1392 440 Acked 1439 Read 10 Done Done New false
1393 441 Acked 1440 Read 7 Done Done New false
1394 442 Acked 1441 Write 2 Done Done New false
1395 443 Acked 1442 Flush 0 Done Done New false
1396 444 Acked 1443 Read 10 Done Done New false
1397 445 Acked 1444 Read 8 Done Done New false
1398 446 Acked 1445 Read 9 Done Done New false
1399 447 Acked 1446 Flush 0 Done Done New false
1400 448 Acked 1447 Write 6 Done Done New false
1401 449 Acked 1448 Write 6 Done Done New false
1402 450 Acked 1449 Write 5 Done Done New false
1403 451 Acked 1450 Write 6 Done Done New false
1404 452 Acked 1451 Write 3 Done Done New false
1405 453 Acked 1452 Read 1 Done Done New false
1406 454 Acked 1453 Write 5 Done Done New false
1407 455 Acked 1454 Write 2 Done Done New false
1408 456 Acked 1455 Write 6 Done Done New false
1409 457 Acked 1456 Read 6 Done Done New false
1410 458 Acked 1457 Write 6 Done Done New false
1411 459 Acked 1458 Read 3 Done Done New false
1412 460 Acked 1459 Read 8 Done Done New false
1413 461 Acked 1460 Write 7 Done Done New false
1414 462 Acked 1461 Flush 0 Done Done New false
1415 463 Acked 1462 Read 4 Done Done New false
1416 464 Acked 1463 Write 7 Done Done New false
1417 465 Acked 1464 Read 4 Done Done New false
1418 466 Acked 1465 Read 1 Done Done New false
1419 467 Acked 1466 Flush 0 Done Done New false
1420 468 Acked 1467 Flush 0 Done Done New false
1421 469 Acked 1468 Write 5 Done Done New false
1422 470 Acked 1469 Read 6 Done Done New false
1423 471 Acked 1470 Read 10 Done Done New false
1424 472 Acked 1471 Read 9 Done Done New false
1425 473 Acked 1472 Write 8 Done Done New false
1426 474 Acked 1473 Read 5 Done Done New false
1427 475 Acked 1474 Write 2 Done Done New false
1428 476 Acked 1475 Read 8 Done Done New false
1429 477 Acked 1476 Read 10 Done Done New false
1430 478 Acked 1477 Write 6 Done Done New false
1431 479 Acked 1478 Write 9 Done Done New false
1432 480 Acked 1479 Read 1 Done Done New false
1433 481 Acked 1480 Read 5 Done Done New false
1434 482 Acked 1481 Write 9 Done Done New false
1435 483 Acked 1482 Write 3 Done Done New false
1436 484 Acked 1483 Read 2 Done Done New false
1437 485 Acked 1484 Write 4 Done Done New false
1438 486 Acked 1485 Read 3 Done Done New false
1439 487 Acked 1486 Write 5 Done Done New false
1440 488 Acked 1487 Write 10 Done Done New false
1441 489 Acked 1488 Flush 0 Done Done New false
1442 490 Acked 1489 Read 6 Done Done New false
1443 491 Acked 1490 Read 5 Done Done New false
1444 492 Acked 1491 Write 2 Done Done New false
1445 493 Acked 1492 Write 2 Done Done New false
1446 494 Acked 1493 Read 6 Done Done New false
1447 495 Acked 1494 Flush 0 Done Done New false
1448 496 Acked 1495 Write 4 Done Done New false
1449 497 Acked 1496 Read 2 Done Done New false
1450 498 Acked 1497 Write 8 Done Done New false
1451 499 Acked 1498 Read 4 Done Done New false
1452 500 Acked 1499 Read 5 Done Done New false
1453 501 Acked 1500 Read 4 Done Done New false
1454 502 Acked 1501 Read 9 Done Done New false
1455 503 Acked 1502 Read 10 Done Done New false
1456 504 Acked 1503 Flush 0 Done Done New false
1457 505 Acked 1504 Read 3 Done Done New false
1458 506 Acked 1505 Write 5 Done Done New false
1459 507 Acked 1506 Flush 0 Done Done New false
1460 508 Acked 1507 Read 8 Done Done New false
1461 509 Acked 1508 Read 10 Done Done New false
1462 510 Acked 1509 Read 8 Done Done New false
1463 511 Acked 1510 Flush 0 Done Done New false
1464 512 Acked 1511 Write 10 Done Done New false
1465 513 Acked 1512 Write 3 Done Done New false
1466 514 Acked 1513 Write 1 Done Done New false
1467 515 Acked 1514 Read 10 Done Done New false
1468 516 Acked 1515 Read 7 Done Done New false
1469 517 Acked 1516 Write 10 Done Done New false
1470 518 Acked 1517 Write 8 Done Done New false
1471 519 Acked 1518 Write 6 Done Done New false
1472 520 Acked 1519 Write 6 Done Done New false
1473 521 Acked 1520 Read 4 Done Done New false
1474 522 Acked 1521 Flush 0 Done Done New false
1475 523 Acked 1522 Read 9 Done Done New false
1476 524 Acked 1523 Write 1 Done Done New false
1477 525 Acked 1524 Read 1 Done Done New false
1478 526 Acked 1525 Read 3 Done Done New false
1479 527 Acked 1526 Write 10 Done Done New false
1480 528 Acked 1527 Read 7 Done Done New false
1481 529 Acked 1528 Flush 0 Done Done New false
1482 530 Acked 1529 Write 1 Done Done New false
1483 531 Acked 1530 Read 8 Done Done New false
1484 532 Acked 1531 Read 9 Done Done New false
1485 533 Acked 1532 Read 6 Done Done New false
1486 534 Acked 1533 Write 7 Done Done New false
1487 535 Acked 1534 Write 10 Done Done New false
1488 536 Acked 1535 Read 9 Done Done New false
1489 537 Acked 1536 Write 9 Done Done New false
1490 538 Acked 1537 Read 2 Done Done New false
1491 539 Acked 1538 Write 5 Done Done New false
1492 540 Acked 1539 Read 7 Done Done New false
1493 541 Acked 1540 Read 3 Done Done New false
1494 542 Acked 1541 Write 4 Done Done New false
1495 543 Acked 1542 Read 10 Done Done New false
1496 544 Acked 1543 Read 10 Done Done New false
1497 545 Acked 1544 Read 9 Done Done New false
1498 546 Acked 1545 Write 7 Done Done New false
1499 547 Acked 1546 Read 6 Done Done New false
1500 548 Acked 1547 Write 5 Done Done New false
1501 549 Acked 1548 Read 4 Done Done New false
1502 550 Acked 1549 Write 10 Done Done New false
1503 551 Acked 1550 Write 8 Done Done New false
1504 552 Acked 1551 Flush 0 Done Done New false
1505 553 Acked 1552 Flush 0 Done Done New false
1506 554 Acked 1553 Read 7 Done Done New false
1507 555 Acked 1554 Write 7 Done Done New false
1508 556 Acked 1555 Read 6 Done Done New false
1509 557 Acked 1556 Write 2 Done Done New false
1510 558 Acked 1557 Read 3 Done Done New false
1511 559 Acked 1558 Flush 0 Done Done New false
1512 560 Acked 1559 Read 5 Done Done New false
1513 561 Acked 1560 Write 5 Done Done New false
1514 562 Acked 1561 Read 2 Done Done New false
1515 563 Acked 1562 Flush 0 Done Done New false
1516 564 Acked 1563 Read 4 Done Done New false
1517 565 Acked 1564 Read 3 Done Done New false
1518 566 Acked 1565 Read 9 Done Done New false
1519 567 Acked 1566 Write 5 Done Done New false
1520 568 Acked 1567 Read 8 Done Done New false
1521 569 Acked 1568 Read 6 Done Done New false
1522 570 Acked 1569 Read 7 Done Done New false
1523 571 Acked 1570 Write 3 Done Done New false
1524 572 Acked 1571 Read 3 Done Done New false
1525 573 Acked 1572 Write 3 Done Done New false
1526 574 Acked 1573 Read 3 Done Done New false
1527 575 Acked 1574 Read 9 Done Done New false
1528 576 Acked 1575 Write 6 Done Done New false
1529 577 Acked 1576 Read 9 Done Done New false
1530 578 Acked 1577 Write 3 Done Done New false
1531 579 Acked 1578 Read 4 Done Done New false
1532 580 Acked 1579 Write 7 Done Done New false
1533 581 Acked 1580 Write 5 Done Done New false
1534 582 Acked 1581 Write 4 Done Done New false
1535 583 Acked 1582 Read 7 Done Done New false
1536 584 Acked 1583 Write 6 Done Done New false
1537 585 Acked 1584 Read 5 Done Done New false
1538 586 Acked 1585 Read 6 Done Done New false
1539 587 Acked 1586 Read 6 Done Done New false
1540 588 Acked 1587 Read 2 Done Done New false
1541 589 Acked 1588 Write 6 Done Done New false
1542 590 Acked 1589 Write 2 Done Done New false
1543 591 Acked 1590 Read 7 Done Done New false
1544 592 Acked 1591 Read 8 Done Done New false
1545 593 Acked 1592 Flush 0 Done Done New false
1546 594 Acked 1593 Write 6 Done Done New false
1547 595 Acked 1594 Read 2 Done Done New false
1548 596 Acked 1595 Read 6 Done Done New false
1549 597 Acked 1596 Write 9 Done Done New false
1550 598 Acked 1597 Write 1 Done Done New false
1551 599 Acked 1598 Write 8 Done Done New false
1552 600 Acked 1599 Read 1 Done Done New false
1553 601 Acked 1600 Write 3 Done Done New false
1554 602 Acked 1601 Write 8 Done Done New false
1555 603 Acked 1602 Write 8 Done Done New false
1556 STATES DS:0 DS:1 DS:2 TOTAL
1557 New 0 0 301 301
1558 Sent 0 0 0 0
1559 Done 301 301 0 602
1560 Skipped 0 0 0 0
1561 Error 0 0 0 0
1562 Last Flush: 1592 1592 1301
1563 Downstairs last five completed: 1301 1300 1299 1298 1297
1564 Upstairs last five completed: 603 602 601 600 599
1565 CLIENT: Up:0 ds:301 act:2
15662024-02-28T22:13:27.533ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15672024-02-28T22:13:27.533ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15682024-02-28T22:13:27.533ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15692024-02-28T22:13:27.533ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15702024-02-28T22:13:27.533ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15712024-02-28T22:13:27.533ZINFOcrucible: [2] client re-new 302 jobs since flush 1301 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15722024-02-28T22:13:27.534ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1573 [8830] got start action
1574 [8830] I am going to restart
1575 Starting downstairs at port 8830
1576 Make output file at "/tmp/dsc/downstairs-8830.txt"
1577 [8830][2] reports Starting
15782024-02-28T22:13:28.534ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15792024-02-28T22:13:28.534ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15802024-02-28T22:13:28.535ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15812024-02-28T22:13:28.535ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15822024-02-28T22:13:28.535ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15832024-02-28T22:13:28.535ZINFOcrucible: [2] client re-new 302 jobs since flush 1301 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15842024-02-28T22:13:28.535ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1585 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1166)
1586 [8830][2] reports Running
15872024-02-28T22:13:29.536ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15882024-02-28T22:13:29.536ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15892024-02-28T22:13:29.538ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15902024-02-28T22:13:29.538ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15912024-02-28T22:13:29.538ZINFOcrucible: send last flush ID to this DS: 1301 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15922024-02-28T22:13:29.538ZINFOcrucible: Replied this last flush ID: 1301 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15932024-02-28T22:13:29.538ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15942024-02-28T22:13:29.538ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15952024-02-28T22:13:29.538ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15962024-02-28T22:13:29.538ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1597 ----------------------------------------------------------------
1598 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
1599 Downstairs last five completed: 1603 1602 1601 1600 1599
1600 Upstairs last five completed: 604 603 602 601 600
1601 CLIENT: Up:0 ds:0 act:3
1602 Replay: All jobs finished, all DS active.
1603 Main task has work to do, go find it
1604 got dsc Stop(1)
1605 stop 1
1606 Feb 28 22:13:31.275 INFO request completed, latency_us: 359, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 49bcd8d5-40d7-4e1a-a418-47f42dc7251b, remote_addr[8820] Got stop action so:false kr:false
1607 : 127.0.0.1:36617, local_addr: 127.0.0.1:9998
16082024-02-28T22:13:31.276ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16092024-02-28T22:13:31.276ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16102024-02-28T22:13:31.276ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16112024-02-28T22:13:31.276ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1612 Feb 28 22:13:31.276 INFO request completed, latency_us: 119, response_code: 200, uri: /state/cid/1, method: GET, req_id: bf8dfd9e-0220-4aa3-bd7e-4eb9530fccca, remote_addr: 127.0.0.1:36617, local_addr:{"msg":"[1] client re-new 0 jobs since flush 1603","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:31.276710291Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
1613 127.0.0.1:9998
16142024-02-28T22:13:31.276ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1615 [8820] Exited with: ExitStatus(unix_wait_status(9))
1616 [8820][1] reports Exit
16172024-02-28T22:13:32.277ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16182024-02-28T22:13:32.277ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16192024-02-28T22:13:32.277ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16202024-02-28T22:13:32.277ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16212024-02-28T22:13:32.277ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16222024-02-28T22:13:32.278ZINFOcrucible: [1] client re-new 0 jobs since flush 1603 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16232024-02-28T22:13:32.278ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16242024-02-28T22:13:33.279ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16252024-02-28T22:13:33.279ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16262024-02-28T22:13:33.279ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16272024-02-28T22:13:33.279ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16282024-02-28T22:13:33.280ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16292024-02-28T22:13:33.280ZINFOcrucible: [1] client re-new 0 jobs since flush 1603 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16302024-02-28T22:13:33.280ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16312024-02-28T22:13:34.281ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16322024-02-28T22:13:34.281ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16332024-02-28T22:13:34.281ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16342024-02-28T22:13:34.281ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16352024-02-28T22:13:34.281ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16362024-02-28T22:13:34.281ZINFOcrucible: [1] client re-new 0 jobs since flush 1603 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16372024-02-28T22:13:34.281ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1638 Feb 28 22:13:35.278 INFO request completed, latency_us: 339, response_code: 200, uri: /state/cid/1, method: GET, req_id: c11d21b3-b3da-4c89-8e38-28400486f033, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
1639 001/300 Write block 3511 len 8192 data: 2 2
1640 002/300 Read block 3897 len 36864
1641 003/300 Read block 2793 len 8192
16422024-02-28T22:13:35.282ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16432024-02-28T22:13:35.282ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16442024-02-28T22:13:35.282ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16452024-02-28T22:13:35.282ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16462024-02-28T22:13:35.282ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16472024-02-28T22:13:35.282ZINFOcrucible: [1] client re-new 3 jobs since flush 1603 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16482024-02-28T22:13:35.282ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
1649 004/300 Read block 2037 len 32768
1650 005/300 Read block 3538 len 24576
1651 006/300 Read block 16 len 28672
1652 007/300 Read block 4888 len 28672
1653 008/300 Read block 3165 len 28672
1654 009/300 Write block 3488 len 36864 data: 2 2 2 2 2 2 2 2 2
1655 010/300 Read block 1055 len 40960
1656 011/300 Read block 960 len 12288
1657 012/300 Read block 4420 len 12288
1658 013/300 Write block 3229 len 12288 data: 2 2 2
1659 014/300 Flush
1660 015/300 Read block 1658 len 16384
1661 016/300 Write block 601 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1662 017/300 Write block 2402 len 8192 data: 3 3
1663 018/300 Write block 2360 len 4096 data: 3
1664 019/300 Write block 926 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1665 020/300 Read block 4670 len 8192
1666 021/300 Read block 4592 len 24576
1667 022/300 Read block 306 len 4096
1668 023/300 Read block 3001 len 28672
1669 024/300 Write block 4261 len 24576 data: 2 2 2 2 2 2
1670 025/300 Read block 564 len 40960
1671 026/300 Read block 1018 len 40960
1672 027/300 Write block 4127 len 12288 data: 3 2 2
1673 028/300 Read block 2378 len 4096
1674 029/300 Read block 2992 len 32768
1675 030/300 Flush
1676 031/300 Read block 3271 len 32768
1677 032/300 Read block 2428 len 40960
1678 033/300 Write block 4975 len 28672 data: 2 2 2 2 2 2 2
1679 034/300 Read block 2176 len 12288
1680 035/300 Read block 1623 len 16384
1681 036/300 Write block 1623 len 16384 data: 3 3 3 3
1682 037/300 Read block 4749 len 36864
1683 038/300 Write block 3544 len 36864 data: 3 3 3 3 4 4 5 5 4
1684 039/300 Write block 4149 len 32768 data: 2 2 2 2 2 2 2 2
1685 040/300 Read block 2993 len 20480
1686 041/300 Read block 3411 len 40960
1687 042/300 Write block 1431 len 24576 data: 2 2 2 2 2 2
1688 043/300 Read block 1816 len 32768
1689 044/300 Flush
1690 045/300 Read block 3403 len 4096
1691 046/300 Write block 4490 len 40960 data: 2 2 2 2 3 3 3 3 4 4
1692 047/300 Read block 3458 len 24576
1693 048/300 Flush
1694 049/300 Read block 2020 len 40960
1695 050/300 Write block 4304 len 4096 data: 2
1696 051/300 Read block 1066 len 8192
1697 052/300 Flush
1698 053/300 Flush
1699 054/300 Read block 4838 len 4096
1700 055/300 Write block 1133 len 16384 data: 2 2 2 2
1701 056/300 Read block 1361 len 24576
1702 057/300 Write block 2673 len 28672 data: 3 3 2 2 3 3 3
1703 058/300 Flush
1704 059/300 Read block 3020 len 28672
1705 060/300 Write block 1124 len 20480 data: 2 3 3 3 3
1706 061/300 Read block 2069 len 20480
1707 062/300 Flush
1708 063/300 Write block 3240 len 8192 data: 2 2
1709 064/300 Read block 3441 len 4096
1710 065/300 Flush
1711 066/300 Read block 2282 len 36864
1712 067/300 Write block 4681 len 32768 data: 2 2 2 2 2 2 2 2
1713 068/300 Write block 294 len 24576 data: 2 2 2 2 2 2
1714 069/300 Write block 3786 len 24576 data: 2 2 2 2 2 2
1715 070/300 Write block 216 len 28672 data: 2 2 2 2 2 2 2
1716 071/300 Read block 4756 len 16384
1717 072/300 Write block 148 len 28672 data: 2 2 2 2 2 2 2
1718 073/300 Read block 1397 len 12288
1719 074/300 Read block 619 len 20480
1720 075/300 Read block 1953 len 36864
1721 076/300 Write block 606 len 40960 data: 3 3 3 3 3 2 2 2 2 2
1722 077/300 Read block 3307 len 28672
1723 078/300 Read block 1788 len 24576
1724 079/300 Write block 3709 len 12288 data: 2 2 2
1725 080/300 Read block 3469 len 4096
1726 081/300 Write block 2718 len 28672 data: 2 2 2 2 2 2 2
1727 082/300 Read block 510 len 28672
1728 083/300 Read block 1061 len 4096
1729 084/300 Write block 461 len 8192 data: 4 3
1730 085/300 Read block 4656 len 36864
1731 086/300 Write block 467 len 20480 data: 2 2 2 2 2
1732 087/300 Read block 1470 len 8192
1733 088/300 Write block 1517 len 16384 data: 3 3 2 2
1734 089/300 Read block 3662 len 20480
1735 090/300 Flush
1736 091/300 Read block 802 len 32768
1737 092/300 Read block 1362 len 24576
1738 093/300 Flush
1739 094/300 Read block 3077 len 32768
1740 095/300 Write block 2514 len 20480 data: 2 2 2 2 2
1741 096/300 Read block 4164 len 12288
1742 097/300 Write block 3727 len 24576 data: 3 3 4 3 3 4
1743 098/300 Write block 4456 len 32768 data: 2 2 2 2 2 2 2 2
1744 099/300 Flush
1745 100/300 Read block 2209 len 12288
1746 101/300 Read block 4963 len 8192
1747 102/300 Flush
1748 103/300 Read block 460 len 8192
1749 104/300 Read block 4353 len 24576
1750 105/300 Flush
1751 106/300 Write block 2321 len 36864 data: 2 2 2 2 2 2 2 2 2
1752 107/300 Write block 3436 len 4096 data: 2
1753 108/300 Write block 3862 len 20480 data: 2 2 2 2 2
1754 109/300 Write block 588 len 20480 data: 2 2 2 2 2
1755 110/300 Flush
1756 111/300 Write block 1030 len 12288 data: 3 2 2
1757 112/300 Write block 593 len 28672 data: 2 2 2 2 2 2 3
1758 113/300 Flush
1759 114/300 Read block 1320 len 12288
1760 115/300 Flush
1761 116/300 Read block 4134 len 8192
1762 117/300 Read block 1973 len 32768
1763 118/300 Read block 2540 len 24576
1764 119/300 Read block 1583 len 24576
1765 120/300 Write block 903 len 4096 data: 2
1766 121/300 Write block 2384 len 36864 data: 2 2 2 2 2 2 2 2 2
1767 122/300 Write block 3692 len 12288 data: 2 2 2
1768 123/300 Write block 9 len 36864 data: 2 2 2 2 2 2 2 2 2
1769 124/300 Write block 3370 len 40960 data: 3 4 3 3 3 3 3 3 2 2
1770 125/300 Write block 1858 len 8192 data: 2 2
1771 126/300 Read block 864 len 16384
1772 127/300 Write block 1418 len 32768 data: 2 2 2 2 2 2 2 2
1773 128/300 Write block 4304 len 28672 data: 3 2 2 2 2 2 2
1774 129/300 Write block 1711 len 24576 data: 2 2 2 2 3 3
1775 130/300 Write block 2814 len 40960 data: 2 2 3 3 3 3 4 3 4 4
1776 131/300 Write block 1247 len 4096 data: 2
1777 132/300 Read block 1826 len 12288
1778 133/300 Write block 2660 len 12288 data: 3 3 2
1779 134/300 Read block 1925 len 40960
1780 135/300 Write block 4912 len 20480 data: 2 2 2 2 2
1781 136/300 Write block 3734 len 36864 data: 4 4 3 2 2 2 3 3 3
1782 137/300 Read block 1939 len 8192
1783 138/300 Write block 1306 len 28672 data: 2 2 2 2 2 2 2
1784 139/300 Write block 670 len 20480 data: 3 2 2 2 3
1785 140/300 Read block 4676 len 8192
1786 141/300 Flush
1787 142/300 Read block 3947 len 24576
1788 143/300 Write block 2094 len 32768 data: 2 2 2 2 2 2 2 2
1789 144/300 Write block 3069 len 28672 data: 2 2 2 2 2 2 2
1790 145/300 Read block 4036 len 20480
1791 146/300 Read block 1577 len 8192
1792 147/300 Write block 4318 len 16384 data: 2 2 2 2
1793 148/300 Read block 1388 len 40960
1794 149/300 Read block 84 len 28672
1795 150/300 Write block 1994 len 16384 data: 2 2 2 2
1796 151/300 Write block 784 len 20480 data: 3 3 3 3 2
1797 152/300 Read block 2994 len 32768
1798 153/300 Read block 3852 len 8192
1799 154/300 Read block 4237 len 4096
1800 155/300 Read block 1348 len 24576
1801 156/300 Write block 3203 len 28672 data: 2 2 2 2 2 2 2
1802 157/300 Read block 1926 len 32768
1803 158/300 Read block 1873 len 28672
1804 159/300 Write block 3816 len 4096 data: 2
1805 160/300 Read block 2992 len 32768
1806 161/300 Write block 2822 len 4096 data: 5
1807 162/300 Read block 1252 len 16384
1808 163/300 Flush
1809 164/300 Write block 4014 len 28672 data: 2 2 2 2 2 2 2
1810 165/300 Flush
1811 166/300 Write block 1973 len 28672 data: 2 2 2 2 2 2 2
1812 167/300 Write block 117 len 12288 data: 2 2 2
1813 168/300 Read block 2432 len 20480
1814 169/300 Write block 2095 len 24576 data: 3 3 3 3 3 3
1815 170/300 Write block 3871 len 28672 data: 3 3 3 2 2 3 3
1816 171/300 Write block 3061 len 12288 data: 3 4 3
1817 172/300 Read block 1370 len 16384
1818 173/300 Write block 4225 len 12288 data: 2 2 2
1819 174/300 Flush
1820 175/300 Flush
1821 176/300 Read block 3508 len 36864
1822 177/300 Write block 652 len 36864 data: 3 3 3 3 3 3 3 3 3
1823 178/300 Write block 3249 len 24576 data: 2 2 2 2 2 2
1824 179/300 Write block 1981 len 16384 data: 2 2 2 2
1825 180/300 Read block 3307 len 28672
1826 181/300 Read block 4184 len 4096
1827 182/300 Write block 1177 len 12288 data: 2 2 2
1828 183/300 Flush
1829 184/300 Write block 4307 len 36864 data: 3 3 3 3 2 2 2 2 2
1830 185/300 Read block 3474 len 16384
1831 186/300 Write block 4013 len 16384 data: 2 3 3 3
1832 187/300 Flush
1833 188/300 Write block 4788 len 8192 data: 2 2
1834 189/300 Write block 2878 len 12288 data: 2 2 2
1835 190/300 Read block 2576 len 24576
1836 191/300 Write block 328 len 32768 data: 2 3 3 2 2 2 2 2
1837 192/300 Read block 680 len 16384
1838 193/300 Read block 2112 len 8192
1839 194/300 Write block 4587 len 24576 data: 2 2 2 2 2 2
1840 195/300 Write block 1513 len 4096 data: 2
1841 196/300 Write block 3921 len 36864 data: 2 2 2 2 2 2 2 2 2
1842 197/300 Write block 2335 len 20480 data: 2 2 2 2 2
1843 198/300 Read block 3949 len 36864
1844 199/300 Flush
1845 200/300 Write block 4137 len 4096 data: 2
1846 201/300 Write block 2428 len 24576 data: 2 2 2 2 2 2
1847 202/300 Read block 4639 len 12288
1848 203/300 Read block 3646 len 40960
1849 204/300 Write block 4827 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1850 205/300 Write block 4858 len 28672 data: 2 2 2 2 2 2 2
1851 206/300 Read block 3304 len 24576
1852 207/300 Write block 2109 len 8192 data: 2 2
1853 208/300 Flush
1854 209/300 Write block 516 len 4096 data: 2
1855 210/300 Read block 223 len 4096
1856 211/300 Read block 728 len 28672
1857 212/300 Write block 2099 len 4096 data: 4
1858 213/300 Read block 385 len 40960
1859 214/300 Read block 4580 len 20480
1860 215/300 Read block 2875 len 8192
1861 216/300 Write block 1791 len 32768 data: 2 2 2 2 2 2 2 3
1862 217/300 Flush
1863 218/300 Write block 1150 len 36864 data: 2 2 2 2 2 2 2 2 2
1864 219/300 Read block 2657 len 12288
1865 220/300 Write block 963 len 16384 data: 3 3 3 3
1866 221/300 Read block 1678 len 8192
1867 222/300 Read block 88 len 36864
1868 223/300 Write block 1099 len 32768 data: 2 2 2 2 3 3 3 3
1869 224/300 Read block 118 len 12288
1870 225/300 Write block 1907 len 32768 data: 2 2 2 2 2 2 2 2
1871 226/300 Write block 3882 len 32768 data: 3 3 4 3 3 3 3 3
1872 227/300 Read block 2514 len 4096
1873 228/300 Flush
1874 229/300 Write block 1349 len 4096 data: 2
1875 230/300 Read block 3062 len 12288
1876 231/300 Read block 553 len 8192
1877 232/300 Write block 1578 len 20480 data: 2 2 3 3 3
1878 233/300 Read block 333 len 12288
1879 234/300 Read block 2724 len 28672
1880 235/300 Write block 2984 len 32768 data: 2 2 2 2 2 2 3 2
1881 236/300 Read block 483 len 16384
1882 237/300 Write block 3766 len 4096 data: 2
1883 238/300 Write block 3742 len 20480 data: 4 3 3 4 4
1884 239/300 Read block 2870 len 4096
1885 240/300 Write block 258 len 16384 data: 2 2 2 2
1886 241/300 Write block 4291 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1887 242/300 Read block 2618 len 4096
1888 243/300 Read block 4014 len 8192
1889 244/300 Read block 712 len 16384
1890 245/300 Read block 861 len 24576
1891 246/300 Read block 850 len 16384
1892 247/300 Write block 1722 len 12288 data: 3 3 2
1893 248/300 Read block 1792 len 8192
1894 249/300 Write block 2592 len 32768 data: 2 2 2 2 2 2 2 2
1895 250/300 Read block 3965 len 24576
1896 251/300 Read block 1786 len 28672
1897 252/300 Read block 1824 len 20480
1898 253/300 Read block 4701 len 16384
1899 254/300 Read block 283 len 4096
1900 255/300 Write block 4945 len 8192 data: 3 2
1901 256/300 Write block 2126 len 36864 data: 2 2 3 3 3 3 3 3 2
1902 257/300 Write block 1720 len 36864 data: 3 3 4 4 3 2 2 2 2
1903 258/300 Write block 1991 len 16384 data: 2 2 2 3
1904 259/300 Read block 1535 len 4096
1905 260/300 Read block 913 len 36864
1906 261/300 Flush
1907 262/300 Write block 2522 len 12288 data: 2 2 2
1908 263/300 Read block 3792 len 20480
1909 264/300 Write block 3322 len 32768 data: 2 2 2 2 2 2 2 2
1910 265/300 Write block 171 len 8192 data: 2 2
1911 266/300 Read block 3348 len 12288
1912 267/300 Read block 4785 len 4096
1913 268/300 Read block 1326 len 24576
1914 269/300 Read block 2934 len 24576
1915 270/300 Flush
1916 271/300 Read block 848 len 40960
1917 272/300 Read block 1698 len 8192
1918 273/300 Read block 4199 len 24576
1919 274/300 Read block 1189 len 32768
1920 275/300 Write block 4529 len 12288 data: 3 3 3
1921 276/300 Read block 2777 len 16384
1922 277/300 Read block 2855 len 24576
1923 278/300 Read block 474 len 4096
1924 279/300 Write block 1072 len 8192 data: 2 2
1925 280/300 Read block 1019 len 16384
1926 281/300 Read block 1237 len 12288
1927 282/300 Write block 3786 len 4096 data: 3
1928 283/300 Write block 4907 len 28672 data: 2 2 2 2 2 3 3
1929 284/300 Write block 628 len 24576 data: 3 3 2 2 2 2
1930 285/300 Write block 3815 len 4096 data: 2
1931 286/300 Read block 92 len 16384
1932 287/300 Read block 4529 len 16384
1933 288/300 Write block 2066 len 36864 data: 2 2 2 2 2 2 2 2 2
1934 289/300 Read block 3530 len 36864
1935 290/300 Write block 4494 len 24576 data: 4 4 4 4 5 5
1936 291/300 Read block 2105 len 32768
1937 292/300 Read block 3649 len 24576
1938 293/300 Write block 4987 len 4096 data: 2
1939 294/300 Read block 3476 len 24576
1940 295/300 Read block 2948 len 28672
1941 296/300 Write block 1119 len 12288 data: 3 2 2
1942 297/300 Read block 25 len 8192
1943 298/300 Read block 4528 len 4096
1944 299/300 Read block 3937 len 4096
1945 300/300 Write block 2926 len 28672 data: 2 2 2 2 2 2 2
1946 Main task has work to do, go find it
1947 got dsc Start(1)
1948 start 1
1949 Feb 28 22:13:35.980 INFO request completed, latency_us: 236, response_code: 204, uri: /start/cid/1, method: POST, req_id: 54cfad2f-43fb-4f36-8ef0-45bec710503d, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
1950 Replay: started 1, returned:Ok(())
1951 ----------------------------------------------------------------
1952 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
1953 States: Active Offline Active
1954 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
1955 605 Acked 1604 Write 2 Done New Done true
1956 606 Acked 1605 Read 9 Done New Done true
1957 607 Acked 1606 Read 2 Done New Done true
1958 608 Acked 1607 Read 8 Done New Done false
1959 609 Acked 1608 Read 6 Done New Done false
1960 610 Acked 1609 Read 7 Done New Done false
1961 611 Acked 1610 Read 7 Done New Done false
1962 612 Acked 1611 Read 7 Done New Done false
1963 613 Acked 1612 Write 9 Done New Done false
1964 614 Acked 1613 Read 10 Done New Done false
1965 615 Acked 1614 Read 3 Done New Done false
1966 616 Acked 1615 Read 3 Done New Done false
1967 617 Acked 1616 Write 3 Done New Done false
1968 618 Acked 1617 Flush 0 Done New Done false
1969 619 Acked 1618 Read 4 Done New Done false
1970 620 Acked 1619 Write 10 Done New Done false
1971 621 Acked 1620 Write 2 Done New Done false
1972 622 Acked 1621 Write 1 Done New Done false
1973 623 Acked 1622 Write 10 Done New Done false
1974 624 Acked 1623 Read 2 Done New Done false
1975 625 Acked 1624 Read 6 Done New Done false
1976 626 Acked 1625 Read 1 Done New Done false
1977 627 Acked 1626 Read 7 Done New Done false
1978 628 Acked 1627 Write 6 Done New Done false
1979 629 Acked 1628 Read 10 Done New Done false
1980 630 Acked 1629 Read 10 Done New Done false
1981 631 Acked 1630 Write 3 Done New Done false
1982 632 Acked 1631 Read 1 Done New Done false
1983 633 Acked 1632 Read 8 Done New Done false
1984 634 Acked 1633 Flush 0 Done New Done false
1985 635 Acked 1634 Read 8 Done New Done false
1986 636 Acked 1635 Read 10 Done New Done false
1987 637 Acked 1636 Write 7 Done New Done false
1988 638 Acked 1637 Read 3 Done New Done false
1989 639 Acked 1638 Read 4 Done New Done false
1990 640 Acked 1639 Write 4 Done New Done false
1991 641 Acked 1640 Read 9 Done New Done false
1992 642 Acked 1641 Write 9 Done New Done false
1993 643 Acked 1642 Write 8 Done New Done false
1994 644 Acked 1643 Read 5 Done New Done false
1995 645 Acked 1644 Read 10 Done New Done false
1996 646 Acked 1645 Write 6 Done New Done false
1997 647 Acked 1646 Read 8 Done New Done false
1998 648 Acked 1647 Flush 0 Done New Done false
1999 649 Acked 1648 Read 1 Done New Done false
2000 650 Acked 1649 Write 10 Done New Done false
2001 651 Acked 1650 Read 6 Done New Done false
2002 652 Acked 1651 Flush 0 Done New Done false
2003 653 Acked 1652 Read 10 Done New Done false
2004 654 Acked 1653 Write 1 Done New Done false
2005 655 Acked 1654 Read 2 Done New Done false
2006 656 Acked 1655 Flush 0 Done New Done false
2007 657 Acked 1656 Flush 0 Done New Done false
2008 658 Acked 1657 Read 1 Done New Done false
2009 659 Acked 1658 Write 4 Done New Done false
2010 660 Acked 1659 Read 6 Done New Done false
2011 661 Acked 1660 Write 7 Done New Done false
2012 662 Acked 1661 Flush 0 Done New Done false
2013 663 Acked 1662 Read 7 Done New Done false
2014 664 Acked 1663 Write 5 Done New Done false
2015 665 Acked 1664 Read 5 Done New Done false
2016 666 Acked 1665 Flush 0 Done New Done false
2017 667 Acked 1666 Write 2 Done New Done false
2018 668 Acked 1667 Read 1 Done New Done false
2019 669 Acked 1668 Flush 0 Done New Done false
2020 670 Acked 1669 Read 9 Done New Done false
2021 671 Acked 1670 Write 8 Done New Done false
2022 672 Acked 1671 Write 6 Done New Done false
2023 673 Acked 1672 Write 6 Done New Done false
2024 674 Acked 1673 Write 7 Done New Done false
2025 675 Acked 1674 Read 4 Done New Done false
2026 676 Acked 1675 Write 7 Done New Done false
2027 677 Acked 1676 Read 3 Done New Done false
2028 678 Acked 1677 Read 5 Done New Done false
2029 679 Acked 1678 Read 9 Done New Done false
2030 680 Acked 1679 Write 10 Done New Done false
2031 681 Acked 1680 Read 7 Done New Done false
2032 682 Acked 1681 Read 6 Done New Done false
2033 683 Acked 1682 Write 3 Done New Done false
2034 684 Acked 1683 Read 1 Done New Done false
2035 685 Acked 1684 Write 7 Done New Done false
2036 686 Acked 1685 Read 7 Done New Done false
2037 687 Acked 1686 Read 1 Done New Done false
2038 688 Acked 1687 Write 2 Done New Done false
2039 689 Acked 1688 Read 9 Done New Done false
2040 690 Acked 1689 Write 5 Done New Done false
2041 691 Acked 1690 Read 2 Done New Done false
2042 692 Acked 1691 Write 4 Done New Done false
2043 693 Acked 1692 Read 5 Done New Done false
2044 694 Acked 1693 Flush 0 Done New Done false
2045 695 Acked 1694 Read 8 Done New Done false
2046 696 Acked 1695 Read 6 Done New Done false
2047 697 Acked 1696 Flush 0 Done New Done false
2048 698 Acked 1697 Read 8 Done New Done false
2049 699 Acked 1698 Write 5 Done New Done false
2050 700 Acked 1699 Read 3 Done New Done false
2051 701 Acked 1700 Write 6 Done New Done false
2052 702 Acked 1701 Write 8 Done New Done false
2053 703 Acked 1702 Flush 0 Done New Done false
2054 704 Acked 1703 Read 3 Done New Done false
2055 705 Acked 1704 Read 2 Done New Done false
2056 706 Acked 1705 Flush 0 Done New Done false
2057 707 Acked 1706 Read 2 Done New Done false
2058 708 Acked 1707 Read 6 Done New Done false
2059 709 Acked 1708 Flush 0 Done New Done false
2060 710 Acked 1709 Write 9 Done New Done false
2061 711 Acked 1710 Write 1 Done New Done false
2062 712 Acked 1711 Write 5 Done New Done false
2063 713 Acked 1712 Write 5 Done New Done false
2064 714 Acked 1713 Flush 0 Done New Done false
2065 715 Acked 1714 Write 3 Done New Done false
2066 716 Acked 1715 Write 7 Done New Done false
2067 717 Acked 1716 Flush 0 Done New Done false
2068 718 Acked 1717 Read 3 Done New Done false
2069 719 Acked 1718 Flush 0 Done New Done false
2070 720 Acked 1719 Read 2 Done New Done false
2071 721 Acked 1720 Read 8 Done New Done false
2072 722 Acked 1721 Read 6 Done New Done false
2073 723 Acked 1722 Read 6 Done New Done false
2074 724 Acked 1723 Write 1 Done New Done false
2075 725 Acked 1724 Write 9 Done New Done false
2076 726 Acked 1725 Write 3 Done New Done false
2077 727 Acked 1726 Write 9 Done New Done false
2078 728 Acked 1727 Write 10 Done New Done false
2079 729 Acked 1728 Write 2 Done New Done false
2080 730 Acked 1729 Read 4 Done New Done false
2081 731 Acked 1730 Write 8 Done New Done false
2082 732 Acked 1731 Write 7 Done New Done false
2083 733 Acked 1732 Write 6 Done New Done false
2084 734 Acked 1733 Write 10 Done New Done false
2085 735 Acked 1734 Write 1 Done New Done false
2086 736 Acked 1735 Read 3 Done New Done false
2087 737 Acked 1736 Write 3 Done New Done false
2088 738 Acked 1737 Read 10 Done New Done false
2089 739 Acked 1738 Write 5 Done New Done false
2090 740 Acked 1739 Write 9 Done New Done false
2091 741 Acked 1740 Read 2 Done New Done false
2092 742 Acked 1741 Write 7 Done New Done false
2093 743 Acked 1742 Write 5 Done New Done false
2094 744 Acked 1743 Read 2 Done New Done false
2095 745 Acked 1744 Flush 0 Done New Done false
2096 746 Acked 1745 Read 6 Done New Done false
2097 747 Acked 1746 Write 8 Done New Done false
2098 748 Acked 1747 Write 7 Done New Done false
2099 749 Acked 1748 Read 5 Done New Done false
2100 750 Acked 1749 Read 2 Done New Done false
2101 751 Acked 1750 Write 4 Done New Done false
2102 752 Acked 1751 Read 10 Done New Done false
2103 753 Acked 1752 Read 7 Done New Done false
2104 754 Acked 1753 Write 4 Done New Done false
2105 755 Acked 1754 Write 5 Done New Done false
2106 756 Acked 1755 Read 8 Done New Done false
2107 757 Acked 1756 Read 2 Done New Done false
2108 758 Acked 1757 Read 1 Done New Done false
2109 759 Acked 1758 Read 6 Done New Done false
2110 760 Acked 1759 Write 7 Done New Done false
2111 761 Acked 1760 Read 8 Done New Done false
2112 762 Acked 1761 Read 7 Done New Done false
2113 763 Acked 1762 Write 1 Done New Done false
2114 764 Acked 1763 Read 8 Done New Done false
2115 765 Acked 1764 Write 1 Done New Done false
2116 766 Acked 1765 Read 4 Done New Done false
2117 767 Acked 1766 Flush 0 Done New Done false
2118 768 Acked 1767 Write 7 Done New Done false
2119 769 Acked 1768 Flush 0 Done New Done false
2120 770 Acked 1769 Write 7 Done New Done false
2121 771 Acked 1770 Write 3 Done New Done false
2122 772 Acked 1771 Read 5 Done New Done false
2123 773 Acked 1772 Write 6 Done New Done false
2124 774 Acked 1773 Write 7 Done New Done false
2125 775 Acked 1774 Write 3 Done New Done false
2126 776 Acked 1775 Read 4 Done New Done false
2127 777 Acked 1776 Write 3 Done New Done false
2128 778 Acked 1777 Flush 0 Done New Done false
2129 779 Acked 1778 Flush 0 Done New Done false
2130 780 Acked 1779 Read 9 Done New Done false
2131 781 Acked 1780 Write 9 Done New Done false
2132 782 Acked 1781 Write 6 Done New Done false
2133 783 Acked 1782 Write 4 Done New Done false
2134 784 Acked 1783 Read 7 Done New Done false
2135 785 Acked 1784 Read 1 Done New Done false
2136 786 Acked 1785 Write 3 Done New Done false
2137 787 Acked 1786 Flush 0 Done New Done false
2138 788 Acked 1787 Write 9 Done New Done false
2139 789 Acked 1788 Read 4 Done New Done false
2140 790 Acked 1789 Write 4 Done New Done false
2141 791 Acked 1790 Flush 0 Done New Done false
2142 792 Acked 1791 Write 2 Done New Done false
2143 793 Acked 1792 Write 3 Done New Done false
2144 794 Acked 1793 Read 6 Done New Done false
2145 795 Acked 1794 Write 8 Done New Done false
2146 796 Acked 1795 Read 4 Done New Done false
2147 797 Acked 1796 Read 2 Done New Done false
2148 798 Acked 1797 Write 6 Done New Done false
2149 799 Acked 1798 Write 1 Done New Done false
2150 800 Acked 1799 Write 9 Done New Done false
2151 801 Acked 1800 Write 5 Done New Done false
2152 802 Acked 1801 Read 9 Done New Done false
2153 803 Acked 1802 Flush 0 Done New Done false
2154 804 Acked 1803 Write 1 Done New Done false
2155 805 Acked 1804 Write 6 Done New Done false
2156 806 Acked 1805 Read 3 Done New Done false
2157 807 Acked 1806 Read 10 Done New Done false
2158 808 Acked 1807 Write 10 Done New Done false
2159 809 Acked 1808 Write 7 Done New Done false
2160 810 Acked 1809 Read 6 Done New Done false
2161 811 Acked 1810 Write 2 Done New Done false
2162 812 Acked 1811 Flush 0 Done New Done false
2163 813 Acked 1812 Write 1 Done New Done false
2164 814 Acked 1813 Read 1 Done New Done false
2165 815 Acked 1814 Read 7 Done New Done false
2166 816 Acked 1815 Write 1 Done New Done false
2167 817 Acked 1816 Read 10 Done New Done false
2168 818 Acked 1817 Read 5 Done New Done false
2169 819 Acked 1818 Read 2 Done New Done false
2170 820 Acked 1819 Write 8 Done New Done false
2171 821 Acked 1820 Flush 0 Done New Done false
2172 822 Acked 1821 Write 9 Done New Done false
2173 823 Acked 1822 Read 3 Done New Done false
2174 824 Acked 1823 Write 4 Done New Done false
2175 825 Acked 1824 Read 2 Done New Done false
2176 826 Acked 1825 Read 9 Done New Done false
2177 827 Acked 1826 Write 8 Done New Done false
2178 828 Acked 1827 Read 3 Done New Done false
2179 829 Acked 1828 Write 8 Done New Done false
2180 830 Acked 1829 Write 8 Done New Done false
2181 831 Acked 1830 Read 1 Done New Done false
2182 832 Acked 1831 Flush 0 Done New Done false
2183 833 Acked 1832 Write 1 Done New Done false
2184 834 Acked 1833 Read 3 Done New Done false
2185 835 Acked 1834 Read 2 Done New Done false
2186 836 Acked 1835 Write 5 Done New Done false
2187 837 Acked 1836 Read 3 Done New Done false
2188 838 Acked 1837 Read 7 Done New Done false
2189 839 Acked 1838 Write 8 Done New Done false
2190 840 Acked 1839 Read 4 Done New Done false
2191 841 Acked 1840 Write 1 Done New Done false
2192 842 Acked 1841 Write 5 Done New Done false
2193 843 Acked 1842 Read 1 Done New Done false
2194 844 Acked 1843 Write 4 Done New Done false
2195 845 Acked 1844 Write 10 Done New Done false
2196 846 Acked 1845 Read 1 Done New Done false
2197 847 Acked 1846 Read 2 Done New Done false
2198 848 Acked 1847 Read 4 Done New Done false
2199 849 Acked 1848 Read 6 Done New Done false
2200 850 Acked 1849 Read 4 Done New Done false
2201 851 Acked 1850 Write 3 Done New Done false
2202 852 Acked 1851 Read 2 Done New Done false
2203 853 Acked 1852 Write 8 Done New Done false
2204 854 Acked 1853 Read 6 Done New Done false
2205 855 Acked 1854 Read 7 Done New Done false
2206 856 Acked 1855 Read 5 Done New Done false
2207 857 Acked 1856 Read 4 Done New Done false
2208 858 Acked 1857 Read 1 Done New Done false
2209 859 Acked 1858 Write 2 Done New Done false
2210 860 Acked 1859 Write 9 Done New Done false
2211 861 Acked 1860 Write 9 Done New Done false
2212 862 Acked 1861 Write 4 Done New Done false
2213 863 Acked 1862 Read 1 Done New Done false
2214 864 Acked 1863 Read 9 Done New Done false
2215 865 Acked 1864 Flush 0 Done New Done false
2216 866 Acked 1865 Write 3 Done New Done false
2217 867 Acked 1866 Read 5 Done New Done false
2218 868 Acked 1867 Write 8 Done New Done false
2219 869 Acked 1868 Write 2 Done New Done false
2220 870 Acked 1869 Read 3 Done New Done false
2221 871 Acked 1870 Read 1 Done New Done false
2222 872 Acked 1871 Read 6 Done New Done false
2223 873 Acked 1872 Read 6 Done New Done false
2224 874 Acked 1873 Flush 0 Done New Done false
2225 875 Acked 1874 Read 10 Done New Done false
2226 876 Acked 1875 Read 2 Done New Done false
2227 877 Acked 1876 Read 6 Done New Done false
2228 878 Acked 1877 Read 8 Done New Done false
2229 879 Acked 1878 Write 3 Done New Done false
2230 880 Acked 1879 Read 4 Done New Done false
2231 881 Acked 1880 Read 6 Done New Done false
2232 882 Acked 1881 Read 1 Done New Done false
2233 883 Acked 1882 Write 2 Done New Done false
2234 884 Acked 1883 Read 4 Done New Done false
2235 885 Acked 1884 Read 3 Done New Done false
2236 886 Acked 1885 Write 1 Done New Done false
2237 887 Acked 1886 Write 7 Done New Done false
2238 888 Acked 1887 Write 6 Done New Done false
2239 889 Acked 1888 Write 1 Done New Done false
2240 890 Acked 1889 Read 4 Done New Done false
2241 891 Acked 1890 Read 4 Done New Done false
2242 892 Acked 1891 Write 9 Done New Done false
2243 893 Acked 1892 Read 9 Done New Done false
2244 894 Acked 1893 Write 6 Done New Done false
2245 895 Acked 1894 Read 8 Done New Done false
2246 896 Acked 1895 Read 6 Done New Done false
2247 897 Acked 1896 Write 1 Done New Done false
2248 898 Acked 1897 Read 6 Done New Done false
2249 899 Acked 1898 Read 7 Done New Done false
2250 900 Acked 1899 Write 3 Done New Done false
2251 901 Acked 1900 Read 2 Done New Done false
2252 902 Acked 1901 Read 1 Done New Done false
2253 903 Acked 1902 Read 1 Done New Done false
2254 904 Acked 1903 Write 7 Done New Done false
2255 STATES DS:0 DS:1 DS:2 TOTAL
2256 New 0 300 0 300
2257 Sent 0 0 0 0
2258 Done 300 0 300 600
2259 Skipped 0 0 0 0
2260 Error 0 0 0 0
2261 Last Flush: 1873 1603 1873
2262 Downstairs last five completed: 1603 1602 1601 1600 1599
2263 Upstairs last five completed: 904 903 902 901 900
2264 CLIENT: Up:0 ds:300 act:2
22652024-02-28T22:13:36.283ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22662024-02-28T22:13:36.283ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22672024-02-28T22:13:36.283ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22682024-02-28T22:13:36.283ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22692024-02-28T22:13:36.283ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22702024-02-28T22:13:36.283ZINFOcrucible: [1] client re-new 300 jobs since flush 1603 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22712024-02-28T22:13:36.284ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2272 [8820] got start action
2273 [8820] I am going to restart
2274 Starting downstairs at port 8820
2275 Make output file at "/tmp/dsc/downstairs-8820.txt"
2276 [8820][1] reports Starting
22772024-02-28T22:13:37.285ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22782024-02-28T22:13:37.285ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22792024-02-28T22:13:37.285ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22802024-02-28T22:13:37.285ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22812024-02-28T22:13:37.285ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22822024-02-28T22:13:37.285ZINFOcrucible: [1] client re-new 301 jobs since flush 1603 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22832024-02-28T22:13:37.285ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2284 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1167)
2285 [8820][1] reports Running
22862024-02-28T22:13:38.286ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22872024-02-28T22:13:38.287ZINFOcrucible: ds_connection connected = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22882024-02-28T22:13:38.288ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22892024-02-28T22:13:38.288ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22902024-02-28T22:13:38.288ZINFOcrucible: send last flush ID to this DS: 1603 = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22912024-02-28T22:13:38.288ZINFOcrucible: Replied this last flush ID: 1603 = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22922024-02-28T22:13:38.288ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22932024-02-28T22:13:38.288ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22942024-02-28T22:13:38.288ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22952024-02-28T22:13:38.289ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2296 ----------------------------------------------------------------
2297 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
2298 Downstairs last five completed: 1904 1903 1902 1901 1900
2299 Upstairs last five completed: 905 904 903 902 901
2300 CLIENT: Up:0 ds:0 act:3
2301 Replay: All jobs finished, all DS active.
2302 Main task has work to do, go find it
2303 got dsc Stop(2)
2304 stop 2
2305 Feb 28 22:13:39.984 INFO request completed, latency_us: 350, response_code:[8830] Got stop action so:false kr:false
2306 204, uri: /stop/cid/2, method: POST, req_id: 0e187e71-345f-4166-80de-9e08454877a0, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
23072024-02-28T22:13:39.984ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23082024-02-28T22:13:39.984ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23092024-02-28T22:13:39.984ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23102024-02-28T22:13:39.984ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2311 Feb 28 22:13:39.984 INFO request completed, latency_us: 117, response_code: 200, uri: /state/cid/2, method: GET, req_id: d58add60-ce08-4fb3-97a2-bba80f2be979, remote_addr: 127.0.0.1:36617, local_addr: 127.{"msg":"[2] client re-new 0 jobs since flush 1904","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:13:39.984977569Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
2312 0.0.1:9998
23132024-02-28T22:13:39.985ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2314 [8830] Exited with: ExitStatus(unix_wait_status(9))
2315 [8830][2] reports Exit
23162024-02-28T22:13:40.987ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23172024-02-28T22:13:40.987ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23182024-02-28T22:13:40.987ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23192024-02-28T22:13:40.987ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23202024-02-28T22:13:40.987ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23212024-02-28T22:13:40.987ZINFOcrucible: [2] client re-new 0 jobs since flush 1904 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23222024-02-28T22:13:40.987ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23232024-02-28T22:13:41.988ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23242024-02-28T22:13:41.988ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23252024-02-28T22:13:41.988ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23262024-02-28T22:13:41.988ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23272024-02-28T22:13:41.988ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23282024-02-28T22:13:41.988ZINFOcrucible: [2] client re-new 0 jobs since flush 1904 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23292024-02-28T22:13:41.988ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23302024-02-28T22:13:42.989ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23312024-02-28T22:13:42.989ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23322024-02-28T22:13:42.989ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23332024-02-28T22:13:42.989ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23342024-02-28T22:13:42.989ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23352024-02-28T22:13:42.989ZINFOcrucible: [2] client re-new 0 jobs since flush 1904 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23362024-02-28T22:13:42.989ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2337 Feb 28 22:13:43.987 INFO request completed, latency_us: 307, response_code: 200, uri: /state/cid/2, method: GET, req_id: a191d763-9a9d-4745-be93-51d37e683eb6, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
2338 001/300 Write block 3656 len 32768 data: 2 2 2 2 2 2 2 2
2339 002/300 Write block 978 len 24576 data: 3 3 3 3 3 2
2340 003/300 Write block 4892 len 20480 data: 2 2 2 2 2
2341 004/300 Read block 218 len 16384
23422024-02-28T22:13:43.990ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23432024-02-28T22:13:43.990ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23442024-02-28T22:13:43.990ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23452024-02-28T22:13:43.990ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23462024-02-28T22:13:43.990ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23472024-02-28T22:13:43.990ZINFOcrucible: [2] client re-new 4 jobs since flush 1904 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23482024-02-28T22:13:43.990ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2349 005/300 Read block 2746 len 16384
2350 006/300 Write block 3637 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2351 007/300 Read block 3840 len 8192
2352 008/300 Read block 234 len 24576
2353 009/300 Read block 1067 len 12288
2354 010/300 Read block 388 len 28672
2355 011/300 Read block 3987 len 36864
2356 012/300 Read block 4460 len 32768
2357 013/300 Read block 1328 len 16384
2358 014/300 Read block 2088 len 4096
2359 015/300 Flush
2360 016/300 Write block 4463 len 16384 data: 3 2 2 2
2361 017/300 Flush
2362 018/300 Read block 2232 len 12288
2363 019/300 Read block 1137 len 16384
2364 020/300 Write block 3148 len 32768 data: 3 3 3 3 3 3 3 3
2365 021/300 Write block 1204 len 32768 data: 3 3 2 2 2 2 2 2
2366 022/300 Read block 4804 len 4096
2367 023/300 Write block 2409 len 40960 data: 2 2 2 2 2 2 3 4 4 3
2368 024/300 Read block 1133 len 36864
2369 025/300 Read block 788 len 28672
2370 026/300 Write block 4599 len 12288 data: 3 3 3
2371 027/300 Write block 1885 len 16384 data: 2 2 2 2
2372 028/300 Write block 4251 len 36864 data: 2 2 2 2 2 2 2 2 2
2373 029/300 Write block 1360 len 16384 data: 2 2 2 2
2374 030/300 Write block 680 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2375 031/300 Write block 827 len 4096 data: 2
2376 032/300 Write block 768 len 28672 data: 2 2 2 2 2 2 2
2377 033/300 Read block 3682 len 24576
2378 034/300 Write block 1491 len 8192 data: 2 2
2379 035/300 Read block 597 len 8192
2380 036/300 Read block 4293 len 40960
2381 037/300 Write block 2172 len 28672 data: 2 2 2 2 2 2 2
2382 038/300 Write block 1711 len 28672 data: 3 3 3 3 4 4 3
2383 039/300 Read block 2275 len 28672
2384 040/300 Read block 2974 len 4096
2385 041/300 Write block 2577 len 24576 data: 2 2 2 2 2 2
2386 042/300 Write block 4009 len 24576 data: 2 2 2 2 3 4
2387 043/300 Read block 209 len 8192
2388 044/300 Write block 3430 len 28672 data: 2 2 2 2 2 2 3
2389 045/300 Read block 323 len 20480
2390 046/300 Read block 1677 len 40960
2391 047/300 Write block 2617 len 20480 data: 2 2 2 2 2
2392 048/300 Write block 1125 len 28672 data: 4 4 4 4 2 2 2
2393 049/300 Write block 3669 len 4096 data: 2
2394 050/300 Write block 3532 len 4096 data: 3
2395 051/300 Write block 2034 len 16384 data: 4 4 4 3
2396 052/300 Read block 2751 len 8192
2397 053/300 Read block 3156 len 4096
2398 054/300 Read block 2437 len 24576
2399 055/300 Read block 3042 len 8192
2400 056/300 Write block 4228 len 12288 data: 2 2 2
2401 057/300 Read block 4982 len 28672
2402 058/300 Read block 1807 len 32768
2403 059/300 Read block 1853 len 32768
2404 060/300 Write block 4490 len 20480 data: 3 3 3 3 5
2405 061/300 Read block 1469 len 4096
2406 062/300 Read block 2360 len 16384
2407 063/300 Write block 519 len 12288 data: 2 2 2
2408 064/300 Write block 4782 len 28672 data: 2 2 2 2 2 2 3
2409 065/300 Write block 3422 len 36864 data: 2 2 2 2 2 2 2 2 3
2410 066/300 Read block 2565 len 12288
2411 067/300 Flush
2412 068/300 Write block 4361 len 28672 data: 2 2 2 2 2 2 2
2413 069/300 Write block 759 len 32768 data: 2 2 2 3 3 3 2 2
2414 070/300 Read block 4336 len 28672
2415 071/300 Read block 2800 len 16384
2416 072/300 Read block 2323 len 36864
2417 073/300 Read block 920 len 32768
2418 074/300 Read block 973 len 12288
2419 075/300 Read block 523 len 4096
2420 076/300 Write block 1320 len 28672 data: 2 2 2 2 2 2 2
2421 077/300 Read block 4777 len 16384
2422 078/300 Read block 1924 len 8192
2423 079/300 Write block 1884 len 32768 data: 2 3 3 3 3 2 2 2
2424 080/300 Read block 1665 len 12288
2425 081/300 Read block 533 len 36864
2426 082/300 Read block 821 len 12288
2427 083/300 Read block 875 len 4096
2428 084/300 Flush
2429 085/300 Read block 2611 len 4096
2430 086/300 Write block 3151 len 8192 data: 4 4
2431 087/300 Write block 3855 len 4096 data: 2
2432 088/300 Write block 2259 len 24576 data: 2 2 2 2 2 2
2433 089/300 Read block 1433 len 24576
2434 090/300 Write block 4248 len 28672 data: 2 2 2 3 3 3 3
2435 091/300 Read block 4574 len 40960
2436 092/300 Read block 2881 len 8192
2437 093/300 Write block 1259 len 16384 data: 3 3 2 2
2438 094/300 Write block 4746 len 28672 data: 2 2 3 2 2 2 2
2439 095/300 Write block 417 len 40960 data: 2 2 2 2 2 2 3 3 3 2
2440 096/300 Read block 3270 len 36864
2441 097/300 Read block 1801 len 24576
2442 098/300 Write block 675 len 4096 data: 3
2443 099/300 Read block 2774 len 12288
2444 100/300 Write block 1580 len 36864 data: 4 4 4 3 3 3 3 2 2
2445 101/300 Flush
2446 102/300 Write block 1403 len 28672 data: 2 2 2 2 2 2 2
2447 103/300 Read block 1044 len 16384
2448 104/300 Read block 3819 len 32768
2449 105/300 Write block 3842 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2450 106/300 Write block 1267 len 36864 data: 3 3 3 3 3 2 2 2 2
2451 107/300 Read block 2617 len 12288
2452 108/300 Write block 4318 len 4096 data: 3
2453 109/300 Read block 2136 len 16384
2454 110/300 Read block 1881 len 28672
2455 111/300 Write block 4360 len 28672 data: 2 3 3 3 3 3 3
2456 112/300 Write block 3777 len 4096 data: 2
2457 113/300 Read block 2885 len 16384
2458 114/300 Write block 3842 len 8192 data: 3 3
2459 115/300 Read block 4127 len 16384
2460 116/300 Flush
2461 117/300 Read block 4804 len 4096
2462 118/300 Write block 2890 len 20480 data: 3 2 2 2 2
2463 119/300 Read block 1995 len 16384
2464 120/300 Read block 2164 len 36864
2465 121/300 Write block 353 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2466 122/300 Write block 464 len 36864 data: 2 2 2 3 3 3 3 3 2
2467 123/300 Write block 3122 len 4096 data: 2
2468 124/300 Read block 2935 len 40960
2469 125/300 Read block 1125 len 20480
2470 126/300 Read block 4942 len 8192
2471 127/300 Read block 2625 len 20480
2472 128/300 Write block 4941 len 12288 data: 3 3 3
2473 129/300 Read block 2688 len 12288
2474 130/300 Read block 4334 len 20480
2475 131/300 Read block 4555 len 4096
2476 132/300 Write block 1316 len 20480 data: 2 2 2 2 3
2477 133/300 Read block 3184 len 32768
2478 134/300 Write block 3122 len 12288 data: 3 2 2
2479 135/300 Write block 956 len 32768 data: 3 3 3 4 4 4 3 4
2480 136/300 Read block 4482 len 16384
2481 137/300 Flush
2482 138/300 Write block 131 len 16384 data: 2 2 2 2
2483 139/300 Read block 4144 len 24576
2484 140/300 Write block 5 len 16384 data: 2 2 2 2
2485 141/300 Read block 3062 len 12288
2486 142/300 Read block 1215 len 16384
2487 143/300 Write block 254 len 8192 data: 2 2
2488 144/300 Read block 1630 len 20480
2489 145/300 Write block 552 len 4096 data: 2
2490 146/300 Flush
2491 147/300 Write block 1967 len 28672 data: 2 2 2 2 2 2 3
2492 148/300 Flush
2493 149/300 Read block 993 len 8192
2494 150/300 Read block 4303 len 28672
2495 151/300 Read block 518 len 12288
2496 152/300 Read block 1871 len 4096
2497 153/300 Read block 920 len 32768
2498 154/300 Write block 3971 len 4096 data: 3
2499 155/300 Write block 3283 len 32768 data: 2 2 2 2 2 2 3 3
2500 156/300 Write block 916 len 28672 data: 2 2 2 2 2 2 2
2501 157/300 Write block 2380 len 32768 data: 2 2 2 2 3 3 3 3
2502 158/300 Write block 2428 len 20480 data: 3 3 3 3 3
2503 159/300 Write block 2926 len 24576 data: 3 3 3 3 3 3
2504 160/300 Write block 2298 len 16384 data: 2 2 2 2
2505 161/300 Read block 1507 len 24576
2506 162/300 Read block 2761 len 28672
2507 163/300 Write block 508 len 8192 data: 2 2
2508 164/300 Flush
2509 165/300 Read block 3192 len 16384
2510 166/300 Read block 1331 len 20480
2511 167/300 Flush
2512 168/300 Read block 1314 len 40960
2513 169/300 Read block 720 len 36864
2514 170/300 Read block 4792 len 32768
2515 171/300 Read block 777 len 32768
2516 172/300 Write block 2240 len 28672 data: 2 2 2 2 3 3 3
2517 173/300 Flush
2518 174/300 Read block 1274 len 32768
2519 175/300 Write block 4941 len 8192 data: 4 4
2520 176/300 Write block 1059 len 36864 data: 3 3 3 3 3 4 3 3 3
2521 177/300 Read block 4278 len 40960
2522 178/300 Write block 2293 len 40960 data: 2 2 2 2 2 3 3 3 3 2
2523 179/300 Flush
2524 180/300 Write block 3096 len 16384 data: 2 2 2 2
2525 181/300 Write block 3141 len 4096 data: 2
2526 182/300 Read block 888 len 4096
2527 183/300 Read block 3299 len 16384
2528 184/300 Write block 4547 len 12288 data: 2 3 3
2529 185/300 Flush
2530 186/300 Read block 2504 len 28672
2531 187/300 Read block 4754 len 40960
2532 188/300 Flush
2533 189/300 Read block 2481 len 40960
2534 190/300 Read block 499 len 20480
2535 191/300 Write block 4763 len 12288 data: 3 2 2
2536 192/300 Write block 2531 len 32768 data: 2 2 2 2 2 2 2 2
2537 193/300 Write block 648 len 28672 data: 2 2 2 3 4 4 4
2538 194/300 Read block 4179 len 24576
2539 195/300 Read block 3849 len 28672
2540 196/300 Read block 126 len 4096
2541 197/300 Write block 3145 len 16384 data: 3 3 3 4
2542 198/300 Flush
2543 199/300 Write block 1112 len 40960 data: 4 3 3 3 3 3 3 4 3 3
2544 200/300 Flush
2545 201/300 Write block 2536 len 12288 data: 3 3 3
2546 202/300 Write block 2419 len 28672 data: 3 3 2 2 2 2 2
2547 203/300 Write block 3725 len 24576 data: 3 3 4 4 5 4
2548 204/300 Write block 1490 len 16384 data: 3 3 3 2
2549 205/300 Read block 554 len 12288
2550 206/300 Read block 597 len 36864
2551 207/300 Write block 2348 len 4096 data: 2
2552 208/300 Write block 2877 len 8192 data: 2 3
2553 209/300 Write block 488 len 32768 data: 3 2 2 2 3 4 4 4
2554 210/300 Read block 3891 len 4096
2555 211/300 Read block 1006 len 40960
2556 212/300 Write block 3740 len 16384 data: 4 4 5 4
2557 213/300 Read block 227 len 16384
2558 214/300 Read block 4112 len 20480
2559 215/300 Read block 4763 len 16384
2560 216/300 Read block 2936 len 40960
2561 217/300 Read block 3983 len 20480
2562 218/300 Read block 2660 len 20480
2563 219/300 Read block 1334 len 8192
2564 220/300 Write block 4992 len 8192 data: 2 2
2565 221/300 Write block 532 len 36864 data: 2 2 2 2 2 2 2 2 2
2566 222/300 Read block 4297 len 12288
2567 223/300 Read block 2613 len 36864
2568 224/300 Write block 2267 len 4096 data: 2
2569 225/300 Write block 2051 len 20480 data: 3 3 3 2 2
2570 226/300 Read block 1124 len 36864
2571 227/300 Write block 749 len 16384 data: 2 2 2 2
2572 228/300 Read block 4360 len 16384
2573 229/300 Write block 130 len 12288 data: 2 3 3
2574 230/300 Read block 1364 len 32768
2575 231/300 Write block 186 len 8192 data: 2 2
2576 232/300 Write block 4304 len 20480 data: 4 3 3 4 4
2577 233/300 Write block 1638 len 36864 data: 4 4 4 4 4 4 4 2 2
2578 234/300 Read block 2369 len 32768
2579 235/300 Read block 1195 len 40960
2580 236/300 Write block 2617 len 28672 data: 3 3 3 3 3 2 2
2581 237/300 Read block 1154 len 16384
2582 238/300 Write block 3380 len 16384 data: 2 2 2 2
2583 239/300 Read block 4625 len 8192
2584 240/300 Write block 2474 len 36864 data: 3 2 2 2 2 2 2 2 2
2585 241/300 Write block 3005 len 32768 data: 2 2 2 2 2 2 2 2
2586 242/300 Write block 317 len 36864 data: 2 2 2 2 2 2 2 2 2
2587 243/300 Read block 2516 len 36864
2588 244/300 Read block 4333 len 16384
2589 245/300 Write block 2783 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2590 246/300 Write block 4244 len 12288 data: 2 2 2
2591 247/300 Write block 2684 len 24576 data: 3 3 2 2 2 2
2592 248/300 Write block 2813 len 12288 data: 2 3 3
2593 249/300 Read block 2601 len 24576
2594 250/300 Write block 1699 len 12288 data: 2 2 2
2595 251/300 Write block 3254 len 4096 data: 3
2596 252/300 Write block 1469 len 28672 data: 2 2 2 2 2 2 2
2597 253/300 Read block 2555 len 40960
2598 254/300 Write block 4091 len 28672 data: 3 2 2 2 2 2 2
2599 255/300 Read block 2379 len 12288
2600 256/300 Flush
2601 257/300 Flush
2602 258/300 Write block 2375 len 4096 data: 2
2603 259/300 Flush
2604 260/300 Write block 1169 len 20480 data: 2 2 2 2 2
2605 261/300 Read block 1917 len 36864
2606 262/300 Read block 2753 len 24576
2607 263/300 Write block 406 len 28672 data: 2 2 2 2 2 2 2
2608 264/300 Write block 81 len 8192 data: 2 2
2609 265/300 Write block 2667 len 4096 data: 2
2610 266/300 Write block 4063 len 28672 data: 3 2 2 2 2 3 3
2611 267/300 Read block 2433 len 8192
2612 268/300 Read block 563 len 28672
2613 269/300 Write block 821 len 28672 data: 2 2 2 2 2 2 3
2614 270/300 Read block 2328 len 24576
2615 271/300 Flush
2616 272/300 Read block 3707 len 12288
2617 273/300 Read block 4021 len 28672
2618 274/300 Write block 3791 len 24576 data: 3 2 2 2 2 2
2619 275/300 Read block 2752 len 20480
2620 276/300 Write block 3163 len 12288 data: 2 2 2
2621 277/300 Write block 1474 len 12288 data: 3 3 2
2622 278/300 Write block 122 len 24576 data: 2 2 2 2 2 2
2623 279/300 Read block 3588 len 32768
2624 280/300 Write block 2021 len 16384 data: 2 2 2 2
2625 281/300 Read block 3800 len 40960
2626 282/300 Flush
2627 283/300 Write block 4612 len 36864 data: 3 2 2 3 3 3 3 3 3
2628 284/300 Read block 2810 len 20480
2629 285/300 Write block 3473 len 32768 data: 2 2 2 2 2 2 2 2
2630 286/300 Write block 3281 len 16384 data: 2 2 3 3
2631 287/300 Write block 378 len 4096 data: 3
2632 288/300 Write block 1031 len 32768 data: 3 3 2 2 2 2 2 2
2633 289/300 Write block 2025 len 24576 data: 2 2 2 2 2 2
2634 290/300 Read block 3146 len 24576
2635 291/300 Read block 1820 len 20480
2636 292/300 Read block 2871 len 4096
2637 293/300 Write block 1140 len 20480 data: 3 3 3 3 3
2638 294/300 Read block 2929 len 16384
2639 295/300 Read block 238 len 32768
2640 296/300 Read block 4522 len 8192
2641 297/300 Read block 1795 len 24576
2642 298/300 Read block 4895 len 32768
2643 299/300 Read block 653 len 20480
2644 300/300 Write block 3060 len 4096 data: 3
2645 Main task has work to do, go find it
2646 got dsc Start(2)
2647 start 2
2648 Feb 28 22:13:44.654 INFO request completed, latency_us: 218, response_code: 204, uri: /start/cid/2, method: POST, req_id: 2334ce16-742f-43c6-9774-6ade6ea3da17, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
2649 Replay: started 2, returned:Ok(())
2650 ----------------------------------------------------------------
2651 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
2652 States: Active Active Offline
2653 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
2654 906 Acked 1905 Write 8 Done Done New true
2655 907 Acked 1906 Write 6 Done Done New true
2656 908 Acked 1907 Write 5 Done Done New true
2657 909 Acked 1908 Read 4 Done Done New true
2658 910 Acked 1909 Read 4 Done Done New false
2659 911 Acked 1910 Write 10 Done Done New false
2660 912 Acked 1911 Read 2 Done Done New false
2661 913 Acked 1912 Read 6 Done Done New false
2662 914 Acked 1913 Read 3 Done Done New false
2663 915 Acked 1914 Read 7 Done Done New false
2664 916 Acked 1915 Read 9 Done Done New false
2665 917 Acked 1916 Read 8 Done Done New false
2666 918 Acked 1917 Read 4 Done Done New false
2667 919 Acked 1918 Read 1 Done Done New false
2668 920 Acked 1919 Flush 0 Done Done New false
2669 921 Acked 1920 Write 4 Done Done New false
2670 922 Acked 1921 Flush 0 Done Done New false
2671 923 Acked 1922 Read 3 Done Done New false
2672 924 Acked 1923 Read 4 Done Done New false
2673 925 Acked 1924 Write 8 Done Done New false
2674 926 Acked 1925 Write 8 Done Done New false
2675 927 Acked 1926 Read 1 Done Done New false
2676 928 Acked 1927 Write 10 Done Done New false
2677 929 Acked 1928 Read 9 Done Done New false
2678 930 Acked 1929 Read 7 Done Done New false
2679 931 Acked 1930 Write 3 Done Done New false
2680 932 Acked 1931 Write 4 Done Done New false
2681 933 Acked 1932 Write 9 Done Done New false
2682 934 Acked 1933 Write 4 Done Done New false
2683 935 Acked 1934 Write 10 Done Done New false
2684 936 Acked 1935 Write 1 Done Done New false
2685 937 Acked 1936 Write 7 Done Done New false
2686 938 Acked 1937 Read 6 Done Done New false
2687 939 Acked 1938 Write 2 Done Done New false
2688 940 Acked 1939 Read 2 Done Done New false
2689 941 Acked 1940 Read 10 Done Done New false
2690 942 Acked 1941 Write 7 Done Done New false
2691 943 Acked 1942 Write 7 Done Done New false
2692 944 Acked 1943 Read 7 Done Done New false
2693 945 Acked 1944 Read 1 Done Done New false
2694 946 Acked 1945 Write 6 Done Done New false
2695 947 Acked 1946 Write 6 Done Done New false
2696 948 Acked 1947 Read 2 Done Done New false
2697 949 Acked 1948 Write 7 Done Done New false
2698 950 Acked 1949 Read 5 Done Done New false
2699 951 Acked 1950 Read 10 Done Done New false
2700 952 Acked 1951 Write 5 Done Done New false
2701 953 Acked 1952 Write 7 Done Done New false
2702 954 Acked 1953 Write 1 Done Done New false
2703 955 Acked 1954 Write 1 Done Done New false
2704 956 Acked 1955 Write 4 Done Done New false
2705 957 Acked 1956 Read 2 Done Done New false
2706 958 Acked 1957 Read 1 Done Done New false
2707 959 Acked 1958 Read 6 Done Done New false
2708 960 Acked 1959 Read 2 Done Done New false
2709 961 Acked 1960 Write 3 Done Done New false
2710 962 Acked 1961 Read 7 Done Done New false
2711 963 Acked 1962 Read 8 Done Done New false
2712 964 Acked 1963 Read 8 Done Done New false
2713 965 Acked 1964 Write 5 Done Done New false
2714 966 Acked 1965 Read 1 Done Done New false
2715 967 Acked 1966 Read 4 Done Done New false
2716 968 Acked 1967 Write 3 Done Done New false
2717 969 Acked 1968 Write 7 Done Done New false
2718 970 Acked 1969 Write 9 Done Done New false
2719 971 Acked 1970 Read 3 Done Done New false
2720 972 Acked 1971 Flush 0 Done Done New false
2721 973 Acked 1972 Write 7 Done Done New false
2722 974 Acked 1973 Write 8 Done Done New false
2723 975 Acked 1974 Read 7 Done Done New false
2724 976 Acked 1975 Read 4 Done Done New false
2725 977 Acked 1976 Read 9 Done Done New false
2726 978 Acked 1977 Read 8 Done Done New false
2727 979 Acked 1978 Read 3 Done Done New false
2728 980 Acked 1979 Read 1 Done Done New false
2729 981 Acked 1980 Write 7 Done Done New false
2730 982 Acked 1981 Read 4 Done Done New false
2731 983 Acked 1982 Read 2 Done Done New false
2732 984 Acked 1983 Write 8 Done Done New false
2733 985 Acked 1984 Read 3 Done Done New false
2734 986 Acked 1985 Read 9 Done Done New false
2735 987 Acked 1986 Read 3 Done Done New false
2736 988 Acked 1987 Read 1 Done Done New false
2737 989 Acked 1988 Flush 0 Done Done New false
2738 990 Acked 1989 Read 1 Done Done New false
2739 991 Acked 1990 Write 2 Done Done New false
2740 992 Acked 1991 Write 1 Done Done New false
2741 993 Acked 1992 Write 6 Done Done New false
2742 994 Acked 1993 Read 6 Done Done New false
2743 995 Acked 1994 Write 7 Done Done New false
2744 996 Acked 1995 Read 10 Done Done New false
2745 997 Acked 1996 Read 2 Done Done New false
2746 998 Acked 1997 Write 4 Done Done New false
2747 999 Acked 1998 Write 7 Done Done New false
2748 1000 Acked 1999 Write 10 Done Done New false
2749 1001 Acked 2000 Read 9 Done Done New false
2750 1002 Acked 2001 Read 6 Done Done New false
2751 1003 Acked 2002 Write 1 Done Done New false
2752 1004 Acked 2003 Read 3 Done Done New false
2753 1005 Acked 2004 Write 9 Done Done New false
2754 1006 Acked 2005 Flush 0 Done Done New false
2755 1007 Acked 2006 Write 7 Done Done New false
2756 1008 Acked 2007 Read 4 Done Done New false
2757 1009 Acked 2008 Read 8 Done Done New false
2758 1010 Acked 2009 Write 10 Done Done New false
2759 1011 Acked 2010 Write 9 Done Done New false
2760 1012 Acked 2011 Read 3 Done Done New false
2761 1013 Acked 2012 Write 1 Done Done New false
2762 1014 Acked 2013 Read 4 Done Done New false
2763 1015 Acked 2014 Read 7 Done Done New false
2764 1016 Acked 2015 Write 7 Done Done New false
2765 1017 Acked 2016 Write 1 Done Done New false
2766 1018 Acked 2017 Read 4 Done Done New false
2767 1019 Acked 2018 Write 2 Done Done New false
2768 1020 Acked 2019 Read 4 Done Done New false
2769 1021 Acked 2020 Flush 0 Done Done New false
2770 1022 Acked 2021 Read 1 Done Done New false
2771 1023 Acked 2022 Write 5 Done Done New false
2772 1024 Acked 2023 Read 4 Done Done New false
2773 1025 Acked 2024 Read 9 Done Done New false
2774 1026 Acked 2025 Write 10 Done Done New false
2775 1027 Acked 2026 Write 9 Done Done New false
2776 1028 Acked 2027 Write 1 Done Done New false
2777 1029 Acked 2028 Read 10 Done Done New false
2778 1030 Acked 2029 Read 5 Done Done New false
2779 1031 Acked 2030 Read 2 Done Done New false
2780 1032 Acked 2031 Read 5 Done Done New false
2781 1033 Acked 2032 Write 3 Done Done New false
2782 1034 Acked 2033 Read 3 Done Done New false
2783 1035 Acked 2034 Read 5 Done Done New false
2784 1036 Acked 2035 Read 1 Done Done New false
2785 1037 Acked 2036 Write 5 Done Done New false
2786 1038 Acked 2037 Read 8 Done Done New false
2787 1039 Acked 2038 Write 3 Done Done New false
2788 1040 Acked 2039 Write 8 Done Done New false
2789 1041 Acked 2040 Read 4 Done Done New false
2790 1042 Acked 2041 Flush 0 Done Done New false
2791 1043 Acked 2042 Write 4 Done Done New false
2792 1044 Acked 2043 Read 6 Done Done New false
2793 1045 Acked 2044 Write 4 Done Done New false
2794 1046 Acked 2045 Read 3 Done Done New false
2795 1047 Acked 2046 Read 4 Done Done New false
2796 1048 Acked 2047 Write 2 Done Done New false
2797 1049 Acked 2048 Read 5 Done Done New false
2798 1050 Acked 2049 Write 1 Done Done New false
2799 1051 Acked 2050 Flush 0 Done Done New false
2800 1052 Acked 2051 Write 7 Done Done New false
2801 1053 Acked 2052 Flush 0 Done Done New false
2802 1054 Acked 2053 Read 2 Done Done New false
2803 1055 Acked 2054 Read 7 Done Done New false
2804 1056 Acked 2055 Read 3 Done Done New false
2805 1057 Acked 2056 Read 1 Done Done New false
2806 1058 Acked 2057 Read 8 Done Done New false
2807 1059 Acked 2058 Write 1 Done Done New false
2808 1060 Acked 2059 Write 8 Done Done New false
2809 1061 Acked 2060 Write 7 Done Done New false
2810 1062 Acked 2061 Write 8 Done Done New false
2811 1063 Acked 2062 Write 5 Done Done New false
2812 1064 Acked 2063 Write 6 Done Done New false
2813 1065 Acked 2064 Write 4 Done Done New false
2814 1066 Acked 2065 Read 6 Done Done New false
2815 1067 Acked 2066 Read 7 Done Done New false
2816 1068 Acked 2067 Write 2 Done Done New false
2817 1069 Acked 2068 Flush 0 Done Done New false
2818 1070 Acked 2069 Read 4 Done Done New false
2819 1071 Acked 2070 Read 5 Done Done New false
2820 1072 Acked 2071 Flush 0 Done Done New false
2821 1073 Acked 2072 Read 10 Done Done New false
2822 1074 Acked 2073 Read 9 Done Done New false
2823 1075 Acked 2074 Read 8 Done Done New false
2824 1076 Acked 2075 Read 8 Done Done New false
2825 1077 Acked 2076 Write 7 Done Done New false
2826 1078 Acked 2077 Flush 0 Done Done New false
2827 1079 Acked 2078 Read 8 Done Done New false
2828 1080 Acked 2079 Write 2 Done Done New false
2829 1081 Acked 2080 Write 9 Done Done New false
2830 1082 Acked 2081 Read 10 Done Done New false
2831 1083 Acked 2082 Write 10 Done Done New false
2832 1084 Acked 2083 Flush 0 Done Done New false
2833 1085 Acked 2084 Write 4 Done Done New false
2834 1086 Acked 2085 Write 1 Done Done New false
2835 1087 Acked 2086 Read 1 Done Done New false
2836 1088 Acked 2087 Read 4 Done Done New false
2837 1089 Acked 2088 Write 3 Done Done New false
2838 1090 Acked 2089 Flush 0 Done Done New false
2839 1091 Acked 2090 Read 7 Done Done New false
2840 1092 Acked 2091 Read 10 Done Done New false
2841 1093 Acked 2092 Flush 0 Done Done New false
2842 1094 Acked 2093 Read 10 Done Done New false
2843 1095 Acked 2094 Read 5 Done Done New false
2844 1096 Acked 2095 Write 3 Done Done New false
2845 1097 Acked 2096 Write 8 Done Done New false
2846 1098 Acked 2097 Write 7 Done Done New false
2847 1099 Acked 2098 Read 6 Done Done New false
2848 1100 Acked 2099 Read 7 Done Done New false
2849 1101 Acked 2100 Read 1 Done Done New false
2850 1102 Acked 2101 Write 4 Done Done New false
2851 1103 Acked 2102 Flush 0 Done Done New false
2852 1104 Acked 2103 Write 10 Done Done New false
2853 1105 Acked 2104 Flush 0 Done Done New false
2854 1106 Acked 2105 Write 3 Done Done New false
2855 1107 Acked 2106 Write 7 Done Done New false
2856 1108 Acked 2107 Write 6 Done Done New false
2857 1109 Acked 2108 Write 4 Done Done New false
2858 1110 Acked 2109 Read 3 Done Done New false
2859 1111 Acked 2110 Read 9 Done Done New false
2860 1112 Acked 2111 Write 1 Done Done New false
2861 1113 Acked 2112 Write 2 Done Done New false
2862 1114 Acked 2113 Write 8 Done Done New false
2863 1115 Acked 2114 Read 1 Done Done New false
2864 1116 Acked 2115 Read 10 Done Done New false
2865 1117 Acked 2116 Write 4 Done Done New false
2866 1118 Acked 2117 Read 4 Done Done New false
2867 1119 Acked 2118 Read 5 Done Done New false
2868 1120 Acked 2119 Read 4 Done Done New false
2869 1121 Acked 2120 Read 10 Done Done New false
2870 1122 Acked 2121 Read 5 Done Done New false
2871 1123 Acked 2122 Read 5 Done Done New false
2872 1124 Acked 2123 Read 2 Done Done New false
2873 1125 Acked 2124 Write 2 Done Done New false
2874 1126 Acked 2125 Write 9 Done Done New false
2875 1127 Acked 2126 Read 3 Done Done New false
2876 1128 Acked 2127 Read 9 Done Done New false
2877 1129 Acked 2128 Write 1 Done Done New false
2878 1130 Acked 2129 Write 5 Done Done New false
2879 1131 Acked 2130 Read 9 Done Done New false
2880 1132 Acked 2131 Write 4 Done Done New false
2881 1133 Acked 2132 Read 4 Done Done New false
2882 1134 Acked 2133 Write 3 Done Done New false
2883 1135 Acked 2134 Read 8 Done Done New false
2884 1136 Acked 2135 Write 2 Done Done New false
2885 1137 Acked 2136 Write 5 Done Done New false
2886 1138 Acked 2137 Write 9 Done Done New false
2887 1139 Acked 2138 Read 8 Done Done New false
2888 1140 Acked 2139 Read 10 Done Done New false
2889 1141 Acked 2140 Write 7 Done Done New false
2890 1142 Acked 2141 Read 4 Done Done New false
2891 1143 Acked 2142 Write 4 Done Done New false
2892 1144 Acked 2143 Read 2 Done Done New false
2893 1145 Acked 2144 Write 9 Done Done New false
2894 1146 Acked 2145 Write 8 Done Done New false
2895 1147 Acked 2146 Write 9 Done Done New false
2896 1148 Acked 2147 Read 9 Done Done New false
2897 1149 Acked 2148 Read 4 Done Done New false
2898 1150 Acked 2149 Write 10 Done Done New false
2899 1151 Acked 2150 Write 3 Done Done New false
2900 1152 Acked 2151 Write 6 Done Done New false
2901 1153 Acked 2152 Write 3 Done Done New false
2902 1154 Acked 2153 Read 6 Done Done New false
2903 1155 Acked 2154 Write 3 Done Done New false
2904 1156 Acked 2155 Write 1 Done Done New false
2905 1157 Acked 2156 Write 7 Done Done New false
2906 1158 Acked 2157 Read 10 Done Done New false
2907 1159 Acked 2158 Write 7 Done Done New false
2908 1160 Acked 2159 Read 3 Done Done New false
2909 1161 Acked 2160 Flush 0 Done Done New false
2910 1162 Acked 2161 Flush 0 Done Done New false
2911 1163 Acked 2162 Write 1 Done Done New false
2912 1164 Acked 2163 Flush 0 Done Done New false
2913 1165 Acked 2164 Write 5 Done Done New false
2914 1166 Acked 2165 Read 9 Done Done New false
2915 1167 Acked 2166 Read 6 Done Done New false
2916 1168 Acked 2167 Write 7 Done Done New false
2917 1169 Acked 2168 Write 2 Done Done New false
2918 1170 Acked 2169 Write 1 Done Done New false
2919 1171 Acked 2170 Write 7 Done Done New false
2920 1172 Acked 2171 Read 2 Done Done New false
2921 1173 Acked 2172 Read 7 Done Done New false
2922 1174 Acked 2173 Write 7 Done Done New false
2923 1175 Acked 2174 Read 6 Done Done New false
2924 1176 Acked 2175 Flush 0 Done Done New false
2925 1177 Acked 2176 Read 3 Done Done New false
2926 1178 Acked 2177 Read 7 Done Done New false
2927 1179 Acked 2178 Write 6 Done Done New false
2928 1180 Acked 2179 Read 5 Done Done New false
2929 1181 Acked 2180 Write 3 Done Done New false
2930 1182 Acked 2181 Write 3 Done Done New false
2931 1183 Acked 2182 Write 6 Done Done New false
2932 1184 Acked 2183 Read 8 Done Done New false
2933 1185 Acked 2184 Write 4 Done Done New false
2934 1186 Acked 2185 Read 10 Done Done New false
2935 1187 Acked 2186 Flush 0 Done Done New false
2936 1188 Acked 2187 Write 9 Done Done New false
2937 1189 Acked 2188 Read 5 Done Done New false
2938 1190 Acked 2189 Write 8 Done Done New false
2939 1191 Acked 2190 Write 4 Done Done New false
2940 1192 Acked 2191 Write 1 Done Done New false
2941 1193 Acked 2192 Write 8 Done Done New false
2942 1194 Acked 2193 Write 6 Done Done New false
2943 1195 Acked 2194 Read 6 Done Done New false
2944 1196 Acked 2195 Read 5 Done Done New false
2945 1197 Acked 2196 Read 1 Done Done New false
2946 1198 Acked 2197 Write 5 Done Done New false
2947 1199 Acked 2198 Read 4 Done Done New false
2948 1200 Acked 2199 Read 8 Done Done New false
2949 1201 Acked 2200 Read 2 Done Done New false
2950 1202 Acked 2201 Read 6 Done Done New false
2951 1203 Acked 2202 Read 8 Done Done New false
2952 1204 Acked 2203 Read 5 Done Done New false
2953 1205 Acked 2204 Write 1 Done Done New false
2954 STATES DS:0 DS:1 DS:2 TOTAL
2955 New 0 0 300 300
2956 Sent 0 0 0 0
2957 Done 300 300 0 600
2958 Skipped 0 0 0 0
2959 Error 0 0 0 0
2960 Last Flush: 2186 2186 1904
2961 Downstairs last five completed: 1904 1903 1902 1901 1900
2962 Upstairs last five completed: 1205 1204 1203 1202 1201
2963 CLIENT: Up:0 ds:300 act:2
29642024-02-28T22:13:44.991ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29652024-02-28T22:13:44.991ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29662024-02-28T22:13:44.991ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29672024-02-28T22:13:44.991ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29682024-02-28T22:13:44.991ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29692024-02-28T22:13:44.991ZINFOcrucible: [2] client re-new 301 jobs since flush 1904 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29702024-02-28T22:13:44.992ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29712024-02-28T22:13:45.994ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29722024-02-28T22:13:45.994ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29732024-02-28T22:13:45.994ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29742024-02-28T22:13:45.994ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29752024-02-28T22:13:45.994ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29762024-02-28T22:13:45.994ZINFOcrucible: [2] client re-new 301 jobs since flush 1904 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29772024-02-28T22:13:45.994ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29782024-02-28T22:13:46.996ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29792024-02-28T22:13:46.996ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29802024-02-28T22:13:46.996ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29812024-02-28T22:13:46.996ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29822024-02-28T22:13:46.996ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29832024-02-28T22:13:46.996ZINFOcrucible: [2] client re-new 301 jobs since flush 1904 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29842024-02-28T22:13:46.996ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2985 [8830] got start action
2986 [8830] I am going to restart
2987 Starting downstairs at port 8830
2988 Make output file at "/tmp/dsc/downstairs-8830.txt"
2989 [8830][2] reports Starting
2990 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1168)
2991 [8830][2] reports Running
29922024-02-28T22:13:47.997ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29932024-02-28T22:13:47.997ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29942024-02-28T22:13:47.997ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29952024-02-28T22:13:47.997ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29962024-02-28T22:13:47.997ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29972024-02-28T22:13:47.997ZINFOcrucible: [2] client re-new 301 jobs since flush 1904 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
29982024-02-28T22:13:47.997ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
2999 ----------------------------------------------------------------
3000 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3001 States: Active Active Offline
3002 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
3003 906 Acked 1905 Write 8 Done Done New true
3004 907 Acked 1906 Write 6 Done Done New true
3005 908 Acked 1907 Write 5 Done Done New true
3006 909 Acked 1908 Read 4 Done Done New true
3007 910 Acked 1909 Read 4 Done Done New true
3008 911 Acked 1910 Write 10 Done Done New true
3009 912 Acked 1911 Read 2 Done Done New true
3010 913 Acked 1912 Read 6 Done Done New true
3011 914 Acked 1913 Read 3 Done Done New true
3012 915 Acked 1914 Read 7 Done Done New true
3013 916 Acked 1915 Read 9 Done Done New true
3014 917 Acked 1916 Read 8 Done Done New true
3015 918 Acked 1917 Read 4 Done Done New true
3016 919 Acked 1918 Read 1 Done Done New true
3017 920 Acked 1919 Flush 0 Done Done New true
3018 921 Acked 1920 Write 4 Done Done New true
3019 922 Acked 1921 Flush 0 Done Done New true
3020 923 Acked 1922 Read 3 Done Done New true
3021 924 Acked 1923 Read 4 Done Done New true
3022 925 Acked 1924 Write 8 Done Done New true
3023 926 Acked 1925 Write 8 Done Done New true
3024 927 Acked 1926 Read 1 Done Done New true
3025 928 Acked 1927 Write 10 Done Done New true
3026 929 Acked 1928 Read 9 Done Done New true
3027 930 Acked 1929 Read 7 Done Done New true
3028 931 Acked 1930 Write 3 Done Done New true
3029 932 Acked 1931 Write 4 Done Done New true
3030 933 Acked 1932 Write 9 Done Done New true
3031 934 Acked 1933 Write 4 Done Done New true
3032 935 Acked 1934 Write 10 Done Done New true
3033 936 Acked 1935 Write 1 Done Done New true
3034 937 Acked 1936 Write 7 Done Done New true
3035 938 Acked 1937 Read 6 Done Done New true
3036 939 Acked 1938 Write 2 Done Done New true
3037 940 Acked 1939 Read 2 Done Done New true
3038 941 Acked 1940 Read 10 Done Done New true
3039 942 Acked 1941 Write 7 Done Done New true
3040 943 Acked 1942 Write 7 Done Done New true
3041 944 Acked 1943 Read 7 Done Done New true
3042 945 Acked 1944 Read 1 Done Done New true
3043 946 Acked 1945 Write 6 Done Done New true
3044 947 Acked 1946 Write 6 Done Done New true
3045 948 Acked 1947 Read 2 Done Done New true
3046 949 Acked 1948 Write 7 Done Done New true
3047 950 Acked 1949 Read 5 Done Done New true
3048 951 Acked 1950 Read 10 Done Done New true
3049 952 Acked 1951 Write 5 Done Done New true
3050 953 Acked 1952 Write 7 Done Done New true
3051 954 Acked 1953 Write 1 Done Done New true
3052 955 Acked 1954 Write 1 Done Done New true
3053 956 Acked 1955 Write 4 Done Done New true
3054 957 Acked 1956 Read 2 Done Done New true
3055 958 Acked 1957 Read 1 Done Done New true
3056 959 Acked 1958 Read 6 Done Done New true
3057 960 Acked 1959 Read 2 Done Done New true
3058 961 Acked 1960 Write 3 Done Done New true
3059 962 Acked 1961 Read 7 Done Done New true
3060 963 Acked 1962 Read 8 Done Done New true
3061 964 Acked 1963 Read 8 Done Done New true
3062 965 Acked 1964 Write 5 Done Done New true
3063 966 Acked 1965 Read 1 Done Done New true
3064 967 Acked 1966 Read 4 Done Done New true
3065 968 Acked 1967 Write 3 Done Done New true
3066 969 Acked 1968 Write 7 Done Done New true
3067 970 Acked 1969 Write 9 Done Done New true
3068 971 Acked 1970 Read 3 Done Done New true
3069 972 Acked 1971 Flush 0 Done Done New true
3070 973 Acked 1972 Write 7 Done Done New true
3071 974 Acked 1973 Write 8 Done Done New true
3072 975 Acked 1974 Read 7 Done Done New true
3073 976 Acked 1975 Read 4 Done Done New true
3074 977 Acked 1976 Read 9 Done Done New true
3075 978 Acked 1977 Read 8 Done Done New true
3076 979 Acked 1978 Read 3 Done Done New true
3077 980 Acked 1979 Read 1 Done Done New true
3078 981 Acked 1980 Write 7 Done Done New true
3079 982 Acked 1981 Read 4 Done Done New true
3080 983 Acked 1982 Read 2 Done Done New true
3081 984 Acked 1983 Write 8 Done Done New true
3082 985 Acked 1984 Read 3 Done Done New true
3083 986 Acked 1985 Read 9 Done Done New true
3084 987 Acked 1986 Read 3 Done Done New true
3085 988 Acked 1987 Read 1 Done Done New true
3086 989 Acked 1988 Flush 0 Done Done New true
3087 990 Acked 1989 Read 1 Done Done New true
3088 991 Acked 1990 Write 2 Done Done New true
3089 992 Acked 1991 Write 1 Done Done New true
3090 993 Acked 1992 Write 6 Done Done New true
3091 994 Acked 1993 Read 6 Done Done New true
3092 995 Acked 1994 Write 7 Done Done New true
3093 996 Acked 1995 Read 10 Done Done New true
3094 997 Acked 1996 Read 2 Done Done New true
3095 998 Acked 1997 Write 4 Done Done New true
3096 999 Acked 1998 Write 7 Done Done New true
3097 1000 Acked 1999 Write 10 Done Done New true
3098 1001 Acked 2000 Read 9 Done Done New true
3099 1002 Acked 2001 Read 6 Done Done New true
3100 1003 Acked 2002 Write 1 Done Done New true
3101 1004 Acked 2003 Read 3 Done Done New true
3102 1005 Acked 2004 Write 9 Done Done New true
3103 1006 Acked 2005 Flush 0 Done Done New true
3104 1007 Acked 2006 Write 7 Done Done New true
3105 1008 Acked 2007 Read 4 Done Done New true
3106 1009 Acked 2008 Read 8 Done Done New true
3107 1010 Acked 2009 Write 10 Done Done New true
3108 1011 Acked 2010 Write 9 Done Done New true
3109 1012 Acked 2011 Read 3 Done Done New true
3110 1013 Acked 2012 Write 1 Done Done New true
3111 1014 Acked 2013 Read 4 Done Done New true
3112 1015 Acked 2014 Read 7 Done Done New true
3113 1016 Acked 2015 Write 7 Done Done New true
3114 1017 Acked 2016 Write 1 Done Done New true
3115 1018 Acked 2017 Read 4 Done Done New true
3116 1019 Acked 2018 Write 2 Done Done New true
3117 1020 Acked 2019 Read 4 Done Done New true
3118 1021 Acked 2020 Flush 0 Done Done New true
3119 1022 Acked 2021 Read 1 Done Done New true
3120 1023 Acked 2022 Write 5 Done Done New true
3121 1024 Acked 2023 Read 4 Done Done New true
3122 1025 Acked 2024 Read 9 Done Done New true
3123 1026 Acked 2025 Write 10 Done Done New true
3124 1027 Acked 2026 Write 9 Done Done New true
3125 1028 Acked 2027 Write 1 Done Done New true
3126 1029 Acked 2028 Read 10 Done Done New true
3127 1030 Acked 2029 Read 5 Done Done New true
3128 1031 Acked 2030 Read 2 Done Done New true
3129 1032 Acked 2031 Read 5 Done Done New true
3130 1033 Acked 2032 Write 3 Done Done New true
3131 1034 Acked 2033 Read 3 Done Done New true
3132 1035 Acked 2034 Read 5 Done Done New true
3133 1036 Acked 2035 Read 1 Done Done New true
3134 1037 Acked 2036 Write 5 Done Done New true
3135 1038 Acked 2037 Read 8 Done Done New true
3136 1039 Acked 2038 Write 3 Done Done New true
3137 1040 Acked 2039 Write 8 Done Done New true
3138 1041 Acked 2040 Read 4 Done Done New true
3139 1042 Acked 2041 Flush 0 Done Done New true
3140 1043 Acked 2042 Write 4 Done Done New true
3141 1044 Acked 2043 Read 6 Done Done New true
3142 1045 Acked 2044 Write 4 Done Done New true
3143 1046 Acked 2045 Read 3 Done Done New true
3144 1047 Acked 2046 Read 4 Done Done New true
3145 1048 Acked 2047 Write 2 Done Done New true
3146 1049 Acked 2048 Read 5 Done Done New true
3147 1050 Acked 2049 Write 1 Done Done New true
3148 1051 Acked 2050 Flush 0 Done Done New true
3149 1052 Acked 2051 Write 7 Done Done New true
3150 1053 Acked 2052 Flush 0 Done Done New true
3151 1054 Acked 2053 Read 2 Done Done New true
3152 1055 Acked 2054 Read 7 Done Done New true
3153 1056 Acked 2055 Read 3 Done Done New true
3154 1057 Acked 2056 Read 1 Done Done New true
3155 1058 Acked 2057 Read 8 Done Done New true
3156 1059 Acked 2058 Write 1 Done Done New true
3157 1060 Acked 2059 Write 8 Done Done New true
3158 1061 Acked 2060 Write 7 Done Done New true
3159 1062 Acked 2061 Write 8 Done Done New true
3160 1063 Acked 2062 Write 5 Done Done New true
3161 1064 Acked 2063 Write 6 Done Done New true
3162 1065 Acked 2064 Write 4 Done Done New true
3163 1066 Acked 2065 Read 6 Done Done New true
3164 1067 Acked 2066 Read 7 Done Done New true
3165 1068 Acked 2067 Write 2 Done Done New true
3166 1069 Acked 2068 Flush 0 Done Done New true
3167 1070 Acked 2069 Read 4 Done Done New true
3168 1071 Acked 2070 Read 5 Done Done New true
3169 1072 Acked 2071 Flush 0 Done Done New true
3170 1073 Acked 2072 Read 10 Done Done New true
3171 1074 Acked 2073 Read 9 Done Done New true
3172 1075 Acked 2074 Read 8 Done Done New true
3173 1076 Acked 2075 Read 8 Done Done New true
3174 1077 Acked 2076 Write 7 Done Done New true
3175 1078 Acked 2077 Flush 0 Done Done New true
3176 1079 Acked 2078 Read 8 Done Done New true
3177 1080 Acked 2079 Write 2 Done Done New true
3178 1081 Acked 2080 Write 9 Done Done New true
3179 1082 Acked 2081 Read 10 Done Done New true
3180 1083 Acked 2082 Write 10 Done Done New true
3181 1084 Acked 2083 Flush 0 Done Done New true
3182 1085 Acked 2084 Write 4 Done Done New true
3183 1086 Acked 2085 Write 1 Done Done New true
3184 1087 Acked 2086 Read 1 Done Done New true
3185 1088 Acked 2087 Read 4 Done Done New true
3186 1089 Acked 2088 Write 3 Done Done New true
3187 1090 Acked 2089 Flush 0 Done Done New true
3188 1091 Acked 2090 Read 7 Done Done New true
3189 1092 Acked 2091 Read 10 Done Done New true
3190 1093 Acked 2092 Flush 0 Done Done New true
3191 1094 Acked 2093 Read 10 Done Done New true
3192 1095 Acked 2094 Read 5 Done Done New true
3193 1096 Acked 2095 Write 3 Done Done New true
3194 1097 Acked 2096 Write 8 Done Done New true
3195 1098 Acked 2097 Write 7 Done Done New true
3196 1099 Acked 2098 Read 6 Done Done New true
3197 1100 Acked 2099 Read 7 Done Done New true
3198 1101 Acked 2100 Read 1 Done Done New true
3199 1102 Acked 2101 Write 4 Done Done New true
3200 1103 Acked 2102 Flush 0 Done Done New true
3201 1104 Acked 2103 Write 10 Done Done New true
3202 1105 Acked 2104 Flush 0 Done Done New true
3203 1106 Acked 2105 Write 3 Done Done New true
3204 1107 Acked 2106 Write 7 Done Done New true
3205 1108 Acked 2107 Write 6 Done Done New true
3206 1109 Acked 2108 Write 4 Done Done New true
3207 1110 Acked 2109 Read 3 Done Done New true
3208 1111 Acked 2110 Read 9 Done Done New true
3209 1112 Acked 2111 Write 1 Done Done New true
3210 1113 Acked 2112 Write 2 Done Done New true
3211 1114 Acked 2113 Write 8 Done Done New true
3212 1115 Acked 2114 Read 1 Done Done New true
3213 1116 Acked 2115 Read 10 Done Done New true
3214 1117 Acked 2116 Write 4 Done Done New true
3215 1118 Acked 2117 Read 4 Done Done New true
3216 1119 Acked 2118 Read 5 Done Done New true
3217 1120 Acked 2119 Read 4 Done Done New true
3218 1121 Acked 2120 Read 10 Done Done New true
3219 1122 Acked 2121 Read 5 Done Done New true
3220 1123 Acked 2122 Read 5 Done Done New true
3221 1124 Acked 2123 Read 2 Done Done New true
3222 1125 Acked 2124 Write 2 Done Done New true
3223 1126 Acked 2125 Write 9 Done Done New true
3224 1127 Acked 2126 Read 3 Done Done New true
3225 1128 Acked 2127 Read 9 Done Done New true
3226 1129 Acked 2128 Write 1 Done Done New true
3227 1130 Acked 2129 Write 5 Done Done New true
3228 1131 Acked 2130 Read 9 Done Done New true
3229 1132 Acked 2131 Write 4 Done Done New true
3230 1133 Acked 2132 Read 4 Done Done New true
3231 1134 Acked 2133 Write 3 Done Done New true
3232 1135 Acked 2134 Read 8 Done Done New true
3233 1136 Acked 2135 Write 2 Done Done New true
3234 1137 Acked 2136 Write 5 Done Done New true
3235 1138 Acked 2137 Write 9 Done Done New true
3236 1139 Acked 2138 Read 8 Done Done New true
3237 1140 Acked 2139 Read 10 Done Done New true
3238 1141 Acked 2140 Write 7 Done Done New true
3239 1142 Acked 2141 Read 4 Done Done New true
3240 1143 Acked 2142 Write 4 Done Done New true
3241 1144 Acked 2143 Read 2 Done Done New true
3242 1145 Acked 2144 Write 9 Done Done New true
3243 1146 Acked 2145 Write 8 Done Done New true
3244 1147 Acked 2146 Write 9 Done Done New true
3245 1148 Acked 2147 Read 9 Done Done New true
3246 1149 Acked 2148 Read 4 Done Done New true
3247 1150 Acked 2149 Write 10 Done Done New true
3248 1151 Acked 2150 Write 3 Done Done New true
3249 1152 Acked 2151 Write 6 Done Done New true
3250 1153 Acked 2152 Write 3 Done Done New true
3251 1154 Acked 2153 Read 6 Done Done New true
3252 1155 Acked 2154 Write 3 Done Done New true
3253 1156 Acked 2155 Write 1 Done Done New true
3254 1157 Acked 2156 Write 7 Done Done New true
3255 1158 Acked 2157 Read 10 Done Done New true
3256 1159 Acked 2158 Write 7 Done Done New true
3257 1160 Acked 2159 Read 3 Done Done New true
3258 1161 Acked 2160 Flush 0 Done Done New true
3259 1162 Acked 2161 Flush 0 Done Done New true
3260 1163 Acked 2162 Write 1 Done Done New true
3261 1164 Acked 2163 Flush 0 Done Done New true
3262 1165 Acked 2164 Write 5 Done Done New true
3263 1166 Acked 2165 Read 9 Done Done New true
3264 1167 Acked 2166 Read 6 Done Done New true
3265 1168 Acked 2167 Write 7 Done Done New true
3266 1169 Acked 2168 Write 2 Done Done New true
3267 1170 Acked 2169 Write 1 Done Done New true
3268 1171 Acked 2170 Write 7 Done Done New true
3269 1172 Acked 2171 Read 2 Done Done New true
3270 1173 Acked 2172 Read 7 Done Done New true
3271 1174 Acked 2173 Write 7 Done Done New true
3272 1175 Acked 2174 Read 6 Done Done New true
3273 1176 Acked 2175 Flush 0 Done Done New true
3274 1177 Acked 2176 Read 3 Done Done New true
3275 1178 Acked 2177 Read 7 Done Done New true
3276 1179 Acked 2178 Write 6 Done Done New true
3277 1180 Acked 2179 Read 5 Done Done New true
3278 1181 Acked 2180 Write 3 Done Done New true
3279 1182 Acked 2181 Write 3 Done Done New true
3280 1183 Acked 2182 Write 6 Done Done New true
3281 1184 Acked 2183 Read 8 Done Done New true
3282 1185 Acked 2184 Write 4 Done Done New true
3283 1186 Acked 2185 Read 10 Done Done New true
3284 1187 Acked 2186 Flush 0 Done Done New true
3285 1188 Acked 2187 Write 9 Done Done New true
3286 1189 Acked 2188 Read 5 Done Done New true
3287 1190 Acked 2189 Write 8 Done Done New true
3288 1191 Acked 2190 Write 4 Done Done New true
3289 1192 Acked 2191 Write 1 Done Done New true
3290 1193 Acked 2192 Write 8 Done Done New true
3291 1194 Acked 2193 Write 6 Done Done New true
3292 1195 Acked 2194 Read 6 Done Done New true
3293 1196 Acked 2195 Read 5 Done Done New true
3294 1197 Acked 2196 Read 1 Done Done New true
3295 1198 Acked 2197 Write 5 Done Done New true
3296 1199 Acked 2198 Read 4 Done Done New true
3297 1200 Acked 2199 Read 8 Done Done New true
3298 1201 Acked 2200 Read 2 Done Done New true
3299 1202 Acked 2201 Read 6 Done Done New true
3300 1203 Acked 2202 Read 8 Done Done New true
3301 1204 Acked 2203 Read 5 Done Done New true
3302 1205 Acked 2204 Write 1 Done Done New true
3303 1206 Acked 2205 Flush 0 Done Done New true
3304 STATES DS:0 DS:1 DS:2 TOTAL
3305 New 0 0 301 301
3306 Sent 0 0 0 0
3307 Done 301 301 0 602
3308 Skipped 0 0 0 0
3309 Error 0 0 0 0
3310 Last Flush: 2205 2205 1904
3311 Downstairs last five completed: 1904 1903 1902 1901 1900
3312 Upstairs last five completed: 1206 1205 1204 1203 1202
3313 CLIENT: Up:0 ds:301 act:2
33142024-02-28T22:13:48.998ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33152024-02-28T22:13:48.998ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33162024-02-28T22:13:49.000ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33172024-02-28T22:13:49.000ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33182024-02-28T22:13:49.000ZINFOcrucible: send last flush ID to this DS: 1904 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33192024-02-28T22:13:49.000ZINFOcrucible: Replied this last flush ID: 1904 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33202024-02-28T22:13:49.000ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33212024-02-28T22:13:49.000ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33222024-02-28T22:13:49.000ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33232024-02-28T22:13:49.000ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
3324 ----------------------------------------------------------------
3325 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
3326 Downstairs last five completed: 2205 2204 2203 2202 2201
3327 Upstairs last five completed: 1206 1205 1204 1203 1202
3328 CLIENT: Up:0 ds:0 act:3
3329 Replay: All jobs finished, all DS active.
3330 Main task has work to do, go find it
3331 got dsc Stop(2)
3332 stop 2
3333 Feb 28 22:13:52.660 INFO [8830] Got stop action so:false kr:false
3334 request completed, latency_us: 352, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 8c43d269-95b7-4ce4-8ae1-e4c25f4156c2, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
33352024-02-28T22:13:52.661ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33362024-02-28T22:13:52.661ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33372024-02-28T22:13:52.661ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33382024-02-28T22:13:52.661ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33392024-02-28T22:13:52.661ZINFOcrucible: [2] client re-new 0 jobs since flush 2205 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33402024-02-28T22:13:52.661ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
3341 Feb 28 22:13:52.661 INFO request completed, latency_us: 177, response_code: 200, uri: /state/cid/2, method: GET, req_id: 4825ab57-9e8a-4b15-b476-1cccd767e5b3, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
3342 [8830] Exited with: ExitStatus(unix_wait_status(9))
3343 [8830][2] reports Exit
33442024-02-28T22:13:53.662ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33452024-02-28T22:13:53.662ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33462024-02-28T22:13:53.662ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33472024-02-28T22:13:53.662ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33482024-02-28T22:13:53.662ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33492024-02-28T22:13:53.662ZINFOcrucible: [2] client re-new 0 jobs since flush 2205 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33502024-02-28T22:13:53.662ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33512024-02-28T22:13:54.664ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33522024-02-28T22:13:54.664ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33532024-02-28T22:13:54.664ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33542024-02-28T22:13:54.664ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33552024-02-28T22:13:54.664ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33562024-02-28T22:13:54.664ZINFOcrucible: [2] client re-new 0 jobs since flush 2205 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33572024-02-28T22:13:54.664ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33582024-02-28T22:13:55.665ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33592024-02-28T22:13:55.665ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33602024-02-28T22:13:55.665ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33612024-02-28T22:13:55.665ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33622024-02-28T22:13:55.665ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33632024-02-28T22:13:55.665ZINFOcrucible: [2] client re-new 0 jobs since flush 2205 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33642024-02-28T22:13:55.665ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
3365 Feb 28 22:13:56.664 INFO request completed, latency_us: 314, response_code: 200, uri: /state/cid/2, method: GET, req_id: 5b39f23a-6527-462c-97b5-89868fd7eade, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
3366 001/300 Read block 4727 len 8192
33672024-02-28T22:13:56.666ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33682024-02-28T22:13:56.666ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33692024-02-28T22:13:56.666ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33702024-02-28T22:13:56.666ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33712024-02-28T22:13:56.666ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33722024-02-28T22:13:56.666ZINFOcrucible: [2] client re-new 1 jobs since flush 2205 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
33732024-02-28T22:13:56.666ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
3374 002/300 Read block 2871 len 8192
3375 003/300 Write block 1878 len 36864 data: 2 2 2 2 2 2 3 4 4
3376 004/300 Write block 2947 len 8192 data: 2 2
3377 005/300 Flush
3378 006/300 Read block 4608 len 16384
3379 007/300 Read block 2383 len 8192
3380 008/300 Write block 1621 len 20480 data: 2 2 4 4 4
3381 009/300 Read block 667 len 4096
3382 010/300 Read block 2154 len 40960
3383 011/300 Write block 643 len 4096 data: 2
3384 012/300 Read block 4719 len 40960
3385 013/300 Read block 4332 len 40960
3386 014/300 Read block 2580 len 8192
3387 015/300 Read block 2062 len 24576
3388 016/300 Write block 1349 len 12288 data: 3 2 2
3389 017/300 Read block 2355 len 8192
3390 018/300 Read block 932 len 24576
3391 019/300 Read block 1406 len 12288
3392 020/300 Read block 1056 len 24576
3393 021/300 Write block 2149 len 36864 data: 2 2 2 2 2 2 2 2 2
3394 022/300 Read block 4792 len 36864
3395 023/300 Write block 4983 len 24576 data: 2 2 2 2 3 2
3396 024/300 Flush
3397 025/300 Read block 2071 len 8192
3398 026/300 Write block 4887 len 12288 data: 2 2 2
3399 027/300 Read block 3756 len 12288
3400 028/300 Read block 1008 len 36864
3401 029/300 Read block 2940 len 8192
3402 030/300 Write block 3462 len 20480 data: 3 3 3 3 2
3403 031/300 Write block 2516 len 16384 data: 3 3 3 2
3404 032/300 Read block 3129 len 32768
3405 033/300 Write block 1287 len 24576 data: 2 2 2 2 2 2
3406 034/300 Read block 3368 len 8192
3407 035/300 Flush
3408 036/300 Read block 4123 len 16384
3409 037/300 Flush
3410 038/300 Read block 1452 len 8192
3411 039/300 Write block 711 len 20480 data: 2 2 2 2 2
3412 040/300 Read block 2646 len 4096
3413 041/300 Write block 324 len 12288 data: 3 3 2
3414 042/300 Read block 2466 len 4096
3415 043/300 Read block 123 len 12288
3416 044/300 Flush
3417 045/300 Read block 999 len 36864
3418 046/300 Flush
3419 047/300 Read block 1368 len 36864
3420 048/300 Flush
3421 049/300 Read block 4481 len 36864
3422 050/300 Read block 645 len 28672
3423 051/300 Write block 888 len 24576 data: 2 2 2 2 2 2
3424 052/300 Read block 4834 len 36864
3425 053/300 Read block 925 len 36864
3426 054/300 Flush
3427 055/300 Write block 745 len 24576 data: 2 2 2 2 3 3
3428 056/300 Write block 2924 len 12288 data: 3 3 4
3429 057/300 Read block 2942 len 16384
3430 058/300 Write block 2498 len 4096 data: 2
3431 059/300 Write block 2212 len 36864 data: 2 2 2 2 2 2 2 2 2
3432 060/300 Read block 2565 len 8192
3433 061/300 Read block 4158 len 24576
3434 062/300 Read block 734 len 40960
3435 063/300 Read block 1179 len 16384
3436 064/300 Read block 4031 len 4096
3437 065/300 Read block 4341 len 20480
3438 066/300 Flush
3439 067/300 Write block 1331 len 36864 data: 3 4 4 4 3 2 2 2 2
3440 068/300 Write block 4111 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3441 069/300 Write block 1712 len 28672 data: 4 4 4 5 5 4 3
3442 070/300 Write block 4135 len 4096 data: 2
3443 071/300 Read block 3061 len 32768
3444 072/300 Read block 2934 len 36864
3445 073/300 Read block 1348 len 40960
3446 074/300 Read block 1371 len 28672
3447 075/300 Write block 681 len 16384 data: 3 3 3 3
3448 076/300 Read block 2842 len 12288
3449 077/300 Read block 633 len 8192
3450 078/300 Read block 1704 len 24576
3451 079/300 Read block 4921 len 28672
3452 080/300 Write block 1479 len 8192 data: 2 2
3453 081/300 Read block 2854 len 8192
3454 082/300 Read block 534 len 16384
3455 083/300 Read block 4785 len 20480
3456 084/300 Read block 4544 len 32768
3457 085/300 Read block 1282 len 24576
3458 086/300 Read block 3348 len 40960
3459 087/300 Write block 4237 len 20480 data: 2 3 3 3 3
3460 088/300 Write block 3891 len 20480 data: 3 3 2 3 3
3461 089/300 Write block 3357 len 28672 data: 3 3 3 3 3 2 2
3462 090/300 Read block 2181 len 28672
3463 091/300 Write block 855 len 4096 data: 2
3464 092/300 Write block 1008 len 36864 data: 2 2 2 2 2 2 2 2 2
3465 093/300 Write block 4867 len 16384 data: 2 2 2 2
3466 094/300 Write block 4654 len 12288 data: 2 2 2
3467 095/300 Read block 4137 len 4096
3468 096/300 Read block 2456 len 4096
3469 097/300 Write block 4499 len 8192 data: 6 4
3470 098/300 Read block 4157 len 16384
3471 099/300 Read block 1395 len 20480
3472 100/300 Flush
3473 101/300 Read block 1695 len 36864
3474 102/300 Read block 3087 len 8192
3475 103/300 Read block 2764 len 12288
3476 104/300 Write block 1473 len 24576 data: 3 4 4 3 2 2
3477 105/300 Read block 2674 len 4096
3478 106/300 Write block 655 len 40960 data: 4 4 4 4 4 4 3 2 3 3
3479 107/300 Read block 2963 len 28672
3480 108/300 Write block 484 len 16384 data: 3 3 3 3
3481 109/300 Write block 2933 len 12288 data: 2 2 2
3482 110/300 Flush
3483 111/300 Read block 1009 len 4096
3484 112/300 Write block 4897 len 36864 data: 2 2 2 2 2 2 2 2 2
3485 113/300 Write block 2189 len 28672 data: 2 2 2 2 2 2 2
3486 114/300 Read block 857 len 8192
3487 115/300 Read block 3244 len 8192
3488 116/300 Flush
3489 117/300 Write block 3057 len 4096 data: 2
3490 118/300 Flush
3491 119/300 Read block 661 len 20480
3492 120/300 Read block 1230 len 16384
3493 121/300 Write block 2901 len 24576 data: 2 2 2 2 2 2
3494 122/300 Read block 4291 len 40960
3495 123/300 Write block 188 len 20480 data: 2 2 2 2 2
3496 124/300 Read block 2179 len 20480
3497 125/300 Write block 1890 len 32768 data: 3 3 3 3 3 2 2 2
3498 126/300 Read block 1202 len 36864
3499 127/300 Write block 676 len 32768 data: 3 3 3 2 3 4 4 4
3500 128/300 Read block 4279 len 36864
3501 129/300 Write block 2006 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3502 130/300 Write block 1878 len 36864 data: 3 3 3 3 3 3 4 5 5
3503 131/300 Write block 979 len 4096 data: 4
3504 132/300 Write block 1738 len 24576 data: 2 2 2 2 2 2
3505 133/300 Write block 1795 len 40960 data: 3 3 3 4 3 3 3 3 3 2
3506 134/300 Write block 2183 len 40960 data: 2 2 2 2 2 2 3 3 3 3
3507 135/300 Flush
3508 136/300 Read block 2307 len 32768
3509 137/300 Write block 2826 len 12288 data: 3 2 2
3510 138/300 Read block 4201 len 32768
3511 139/300 Write block 2364 len 32768 data: 2 2 2 2 3 3 3 3
3512 140/300 Write block 2200 len 32768 data: 2 2 2 2 2 2 2 2
3513 141/300 Read block 4483 len 40960
3514 142/300 Read block 198 len 36864
3515 143/300 Write block 3432 len 36864 data: 3 3 3 3 4 2 2 2 2
3516 144/300 Write block 48 len 36864 data: 2 2 2 2 2 2 2 2 2
3517 145/300 Write block 1906 len 8192 data: 2 3
3518 146/300 Flush
3519 147/300 Write block 4811 len 8192 data: 2 2
3520 148/300 Write block 446 len 32768 data: 2 2 2 2 2 2 2 3
3521 149/300 Write block 3520 len 8192 data: 3 4
3522 150/300 Read block 3854 len 4096
3523 151/300 Read block 2553 len 20480
3524 152/300 Write block 1180 len 12288 data: 2 2 2
3525 153/300 Write block 180 len 32768 data: 3 2 2 2 2 2 3 3
3526 154/300 Write block 2029 len 36864 data: 3 3 2 2 4 5 5 5 4
3527 155/300 Read block 2888 len 28672
3528 156/300 Read block 1853 len 24576
3529 157/300 Write block 2621 len 16384 data: 4 3 3 2
3530 158/300 Flush
3531 159/300 Write block 4254 len 20480 data: 4 3 3 3 3
3532 160/300 Read block 3460 len 12288
3533 161/300 Write block 2595 len 36864 data: 3 3 3 3 3 3 3 3 2
3534 162/300 Write block 4219 len 40960 data: 2 2 2 2 2 2 3 3 3 3
3535 163/300 Write block 3432 len 36864 data: 4 4 4 4 5 3 3 3 3
3536 164/300 Write block 870 len 8192 data: 2 2
3537 165/300 Write block 3259 len 8192 data: 2 2
3538 166/300 Read block 2574 len 20480
3539 167/300 Write block 874 len 28672 data: 2 2 3 3 3 3 3
3540 168/300 Write block 2968 len 32768 data: 2 2 2 2 2 2 2 2
3541 169/300 Read block 3664 len 8192
3542 170/300 Read block 3001 len 20480
3543 171/300 Read block 1232 len 32768
3544 172/300 Write block 3903 len 24576 data: 4 4 4 3 3 3
3545 173/300 Read block 2367 len 4096
3546 174/300 Read block 1181 len 32768
3547 175/300 Write block 4870 len 8192 data: 3 2
3548 176/300 Flush
3549 177/300 Read block 2352 len 28672
3550 178/300 Read block 668 len 12288
3551 179/300 Read block 3100 len 8192
3552 180/300 Read block 1168 len 16384
3553 181/300 Read block 3232 len 16384
3554 182/300 Read block 2155 len 40960
3555 183/300 Write block 3231 len 36864 data: 3 2 2 2 2 2 2 2 2
3556 184/300 Read block 3594 len 28672
3557 185/300 Write block 168 len 28672 data: 2 2 2 3 3 2 2
3558 186/300 Read block 2569 len 32768
3559 187/300 Read block 2905 len 32768
3560 188/300 Write block 2900 len 36864 data: 2 3 3 3 3 3 3 2 2
3561 189/300 Read block 4281 len 8192
3562 190/300 Write block 4300 len 20480 data: 3 2 2 2 5
3563 191/300 Read block 4013 len 32768
3564 192/300 Write block 2179 len 8192 data: 2 2
3565 193/300 Read block 2970 len 28672
3566 194/300 Read block 1741 len 36864
3567 195/300 Write block 696 len 4096 data: 2
3568 196/300 Write block 1272 len 24576 data: 3 3 3 3 2 2
3569 197/300 Read block 318 len 16384
3570 198/300 Read block 2146 len 32768
3571 199/300 Read block 4425 len 24576
3572 200/300 Read block 2389 len 4096
3573 201/300 Write block 4180 len 28672 data: 2 2 2 2 2 2 2
3574 202/300 Read block 1364 len 8192
3575 203/300 Write block 3621 len 40960 data: 2 2 2 2 2 2 3 3 3 3
3576 204/300 Read block 716 len 28672
3577 205/300 Flush
3578 206/300 Write block 3541 len 12288 data: 2 2 3
3579 207/300 Read block 725 len 24576
3580 208/300 Write block 1128 len 12288 data: 5 3 3
3581 209/300 Read block 2237 len 20480
3582 210/300 Write block 769 len 40960 data: 3 3 3 3 3 3 2 2 2 2
3583 211/300 Read block 4326 len 8192
3584 212/300 Write block 3662 len 4096 data: 3
3585 213/300 Read block 4278 len 16384
3586 214/300 Read block 1088 len 20480
3587 215/300 Write block 1104 len 40960 data: 4 4 4 3 2 2 2 3 5 4
3588 216/300 Write block 3274 len 12288 data: 2 2 2
3589 217/300 Read block 1688 len 36864
3590 218/300 Flush
3591 219/300 Read block 3629 len 32768
3592 220/300 Write block 3719 len 16384 data: 2 2 3 3
3593 221/300 Read block 3105 len 28672
3594 222/300 Write block 4712 len 32768 data: 2 2 2 2 2 2 2 2
3595 223/300 Write block 4119 len 20480 data: 3 3 2 2 2
3596 224/300 Write block 754 len 40960 data: 2 2 2 2 2 3 3 3 4 4
3597 225/300 Flush
3598 226/300 Read block 2903 len 24576
3599 227/300 Read block 4772 len 36864
3600 228/300 Read block 2137 len 20480
3601 229/300 Read block 944 len 12288
3602 230/300 Read block 1936 len 24576
3603 231/300 Read block 3790 len 36864
3604 232/300 Read block 4928 len 32768
3605 233/300 Flush
3606 234/300 Read block 233 len 16384
3607 235/300 Write block 1465 len 36864 data: 3 3 2 2 3 3 3 3 4
3608 236/300 Read block 4736 len 40960
3609 237/300 Read block 4528 len 16384
3610 238/300 Write block 3407 len 20480 data: 2 2 2 2 2
3611 239/300 Write block 1188 len 40960 data: 2 2 2 3 3 2 2 2 2 2
3612 240/300 Read block 4737 len 12288
3613 241/300 Read block 3206 len 40960
3614 242/300 Write block 3879 len 32768 data: 3 3 3 4 4 5 4 4
3615 243/300 Write block 2707 len 8192 data: 3 2
3616 244/300 Flush
3617 245/300 Write block 2512 len 28672 data: 2 2 3 3 4 4 4
3618 246/300 Read block 1014 len 40960
3619 247/300 Write block 792 len 32768 data: 2 2 2 2 2 2 2 2
3620 248/300 Read block 183 len 12288
3621 249/300 Write block 1 len 16384 data: 2 2 2 2
3622 250/300 Read block 3781 len 32768
3623 251/300 Read block 3476 len 24576
3624 252/300 Write block 3016 len 24576 data: 3 3 3 3 3 2
3625 253/300 Read block 4702 len 12288
3626 254/300 Write block 4589 len 12288 data: 3 3 3
3627 255/300 Read block 3474 len 36864
3628 256/300 Flush
3629 257/300 Read block 3890 len 8192
3630 258/300 Read block 4861 len 40960
3631 259/300 Flush
3632 260/300 Write block 2050 len 28672 data: 2 4 4 4 3 3 2
3633 261/300 Write block 4626 len 8192 data: 2 2
3634 262/300 Flush
3635 263/300 Flush
3636 264/300 Write block 4328 len 36864 data: 2 2 2 2 2 2 2 2 2
3637 265/300 Flush
3638 266/300 Write block 2680 len 40960 data: 3 3 3 3 4 4 3 3 3 3
3639 267/300 Flush
3640 268/300 Write block 3135 len 32768 data: 2 2 2 2 2 2 3 2
3641 269/300 Write block 1576 len 4096 data: 2
3642 270/300 Flush
3643 271/300 Read block 3157 len 36864
3644 272/300 Write block 4670 len 4096 data: 2
3645 273/300 Read block 4345 len 16384
3646 274/300 Read block 3097 len 20480
3647 275/300 Read block 539 len 8192
3648 276/300 Read block 4545 len 24576
3649 277/300 Read block 619 len 40960
3650 278/300 Flush
3651 279/300 Write block 3837 len 12288 data: 2 2 2
3652 280/300 Read block 846 len 8192
3653 281/300 Read block 916 len 40960
3654 282/300 Read block 2364 len 20480
3655 283/300 Write block 3525 len 36864 data: 3 3 3 3 3 3 3 4 4
3656 284/300 Write block 2680 len 16384 data: 4 4 4 4
3657 285/300 Write block 1385 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3658 286/300 Write block 2438 len 16384 data: 3 3 2 3
3659 287/300 Read block 3423 len 20480
3660 288/300 Read block 4638 len 28672
3661 289/300 Write block 1518 len 36864 data: 4 3 3 2 2 2 2 2 2
3662 290/300 Flush
3663 291/300 Read block 3089 len 40960
3664 292/300 Read block 4834 len 8192
3665 293/300 Write block 2537 len 40960 data: 4 4 2 2 2 2 2 2 2 2
3666 294/300 Read block 3556 len 36864
3667 295/300 Write block 2615 len 20480 data: 2 2 4 4 4
3668 296/300 Read block 3295 len 8192
3669 297/300 Write block 2971 len 32768 data: 3 3 3 3 3 2 2 2
3670 298/300 Write block 4146 len 16384 data: 3 3 2 3
3671 299/300 Write block 2254 len 28672 data: 2 2 2 2 2 3 3
3672 300/300 Read block 96 len 8192
3673 Main task has work to do, go find it
3674 got dsc Start(2)
3675 start 2
3676 Feb 28 22:13:57.317 INFO request completed, latency_us: 206, response_code: 204, uri: /start/cid/2, method: POST, req_id: 1176b44e-04b7-491a-9e9c-45561e4b7a4b, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
3677 Replay: started 2, returned:Ok(())
3678 ----------------------------------------------------------------
3679 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
3680 States: Active Active Offline
3681 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
3682 1207 Acked 2206 Read 2 Done Done New true
3683 1208 Acked 2207 Read 2 Done Done New false
3684 1209 Acked 2208 Write 9 Done Done New false
3685 1210 Acked 2209 Write 2 Done Done New false
3686 1211 Acked 2210 Flush 0 Done Done New false
3687 1212 Acked 2211 Read 4 Done Done New false
3688 1213 Acked 2212 Read 2 Done Done New false
3689 1214 Acked 2213 Write 5 Done Done New false
3690 1215 Acked 2214 Read 1 Done Done New false
3691 1216 Acked 2215 Read 10 Done Done New false
3692 1217 Acked 2216 Write 1 Done Done New false
3693 1218 Acked 2217 Read 10 Done Done New false
3694 1219 Acked 2218 Read 10 Done Done New false
3695 1220 Acked 2219 Read 2 Done Done New false
3696 1221 Acked 2220 Read 6 Done Done New false
3697 1222 Acked 2221 Write 3 Done Done New false
3698 1223 Acked 2222 Read 2 Done Done New false
3699 1224 Acked 2223 Read 6 Done Done New false
3700 1225 Acked 2224 Read 3 Done Done New false
3701 1226 Acked 2225 Read 6 Done Done New false
3702 1227 Acked 2226 Write 9 Done Done New false
3703 1228 Acked 2227 Read 9 Done Done New false
3704 1229 Acked 2228 Write 6 Done Done New false
3705 1230 Acked 2229 Flush 0 Done Done New false
3706 1231 Acked 2230 Read 2 Done Done New false
3707 1232 Acked 2231 Write 3 Done Done New false
3708 1233 Acked 2232 Read 3 Done Done New false
3709 1234 Acked 2233 Read 9 Done Done New false
3710 1235 Acked 2234 Read 2 Done Done New false
3711 1236 Acked 2235 Write 5 Done Done New false
3712 1237 Acked 2236 Write 4 Done Done New false
3713 1238 Acked 2237 Read 8 Done Done New false
3714 1239 Acked 2238 Write 6 Done Done New false
3715 1240 Acked 2239 Read 2 Done Done New false
3716 1241 Acked 2240 Flush 0 Done Done New false
3717 1242 Acked 2241 Read 4 Done Done New false
3718 1243 Acked 2242 Flush 0 Done Done New false
3719 1244 Acked 2243 Read 2 Done Done New false
3720 1245 Acked 2244 Write 5 Done Done New false
3721 1246 Acked 2245 Read 1 Done Done New false
3722 1247 Acked 2246 Write 3 Done Done New false
3723 1248 Acked 2247 Read 1 Done Done New false
3724 1249 Acked 2248 Read 3 Done Done New false
3725 1250 Acked 2249 Flush 0 Done Done New false
3726 1251 Acked 2250 Read 9 Done Done New false
3727 1252 Acked 2251 Flush 0 Done Done New false
3728 1253 Acked 2252 Read 9 Done Done New false
3729 1254 Acked 2253 Flush 0 Done Done New false
3730 1255 Acked 2254 Read 9 Done Done New false
3731 1256 Acked 2255 Read 7 Done Done New false
3732 1257 Acked 2256 Write 6 Done Done New false
3733 1258 Acked 2257 Read 9 Done Done New false
3734 1259 Acked 2258 Read 9 Done Done New false
3735 1260 Acked 2259 Flush 0 Done Done New false
3736 1261 Acked 2260 Write 6 Done Done New false
3737 1262 Acked 2261 Write 3 Done Done New false
3738 1263 Acked 2262 Read 4 Done Done New false
3739 1264 Acked 2263 Write 1 Done Done New false
3740 1265 Acked 2264 Write 9 Done Done New false
3741 1266 Acked 2265 Read 2 Done Done New false
3742 1267 Acked 2266 Read 6 Done Done New false
3743 1268 Acked 2267 Read 10 Done Done New false
3744 1269 Acked 2268 Read 4 Done Done New false
3745 1270 Acked 2269 Read 1 Done Done New false
3746 1271 Acked 2270 Read 5 Done Done New false
3747 1272 Acked 2271 Flush 0 Done Done New false
3748 1273 Acked 2272 Write 9 Done Done New false
3749 1274 Acked 2273 Write 10 Done Done New false
3750 1275 Acked 2274 Write 7 Done Done New false
3751 1276 Acked 2275 Write 1 Done Done New false
3752 1277 Acked 2276 Read 8 Done Done New false
3753 1278 Acked 2277 Read 9 Done Done New false
3754 1279 Acked 2278 Read 10 Done Done New false
3755 1280 Acked 2279 Read 7 Done Done New false
3756 1281 Acked 2280 Write 4 Done Done New false
3757 1282 Acked 2281 Read 3 Done Done New false
3758 1283 Acked 2282 Read 2 Done Done New false
3759 1284 Acked 2283 Read 6 Done Done New false
3760 1285 Acked 2284 Read 7 Done Done New false
3761 1286 Acked 2285 Write 2 Done Done New false
3762 1287 Acked 2286 Read 2 Done Done New false
3763 1288 Acked 2287 Read 4 Done Done New false
3764 1289 Acked 2288 Read 5 Done Done New false
3765 1290 Acked 2289 Read 8 Done Done New false
3766 1291 Acked 2290 Read 6 Done Done New false
3767 1292 Acked 2291 Read 10 Done Done New false
3768 1293 Acked 2292 Write 5 Done Done New false
3769 1294 Acked 2293 Write 5 Done Done New false
3770 1295 Acked 2294 Write 7 Done Done New false
3771 1296 Acked 2295 Read 7 Done Done New false
3772 1297 Acked 2296 Write 1 Done Done New false
3773 1298 Acked 2297 Write 9 Done Done New false
3774 1299 Acked 2298 Write 4 Done Done New false
3775 1300 Acked 2299 Write 3 Done Done New false
3776 1301 Acked 2300 Read 1 Done Done New false
3777 1302 Acked 2301 Read 1 Done Done New false
3778 1303 Acked 2302 Write 2 Done Done New false
3779 1304 Acked 2303 Read 4 Done Done New false
3780 1305 Acked 2304 Read 5 Done Done New false
3781 1306 Acked 2305 Flush 0 Done Done New false
3782 1307 Acked 2306 Read 9 Done Done New false
3783 1308 Acked 2307 Read 2 Done Done New false
3784 1309 Acked 2308 Read 3 Done Done New false
3785 1310 Acked 2309 Write 6 Done Done New false
3786 1311 Acked 2310 Read 1 Done Done New false
3787 1312 Acked 2311 Write 10 Done Done New false
3788 1313 Acked 2312 Read 7 Done Done New false
3789 1314 Acked 2313 Write 4 Done Done New false
3790 1315 Acked 2314 Write 3 Done Done New false
3791 1316 Acked 2315 Flush 0 Done Done New false
3792 1317 Acked 2316 Read 1 Done Done New false
3793 1318 Acked 2317 Write 9 Done Done New false
3794 1319 Acked 2318 Write 7 Done Done New false
3795 1320 Acked 2319 Read 2 Done Done New false
3796 1321 Acked 2320 Read 2 Done Done New false
3797 1322 Acked 2321 Flush 0 Done Done New false
3798 1323 Acked 2322 Write 1 Done Done New false
3799 1324 Acked 2323 Flush 0 Done Done New false
3800 1325 Acked 2324 Read 5 Done Done New false
3801 1326 Acked 2325 Read 4 Done Done New false
3802 1327 Acked 2326 Write 6 Done Done New false
3803 1328 Acked 2327 Read 10 Done Done New false
3804 1329 Acked 2328 Write 5 Done Done New false
3805 1330 Acked 2329 Read 5 Done Done New false
3806 1331 Acked 2330 Write 8 Done Done New false
3807 1332 Acked 2331 Read 9 Done Done New false
3808 1333 Acked 2332 Write 8 Done Done New false
3809 1334 Acked 2333 Read 9 Done Done New false
3810 1335 Acked 2334 Write 10 Done Done New false
3811 1336 Acked 2335 Write 9 Done Done New false
3812 1337 Acked 2336 Write 1 Done Done New false
3813 1338 Acked 2337 Write 6 Done Done New false
3814 1339 Acked 2338 Write 10 Done Done New false
3815 1340 Acked 2339 Write 10 Done Done New false
3816 1341 Acked 2340 Flush 0 Done Done New false
3817 1342 Acked 2341 Read 8 Done Done New false
3818 1343 Acked 2342 Write 3 Done Done New false
3819 1344 Acked 2343 Read 8 Done Done New false
3820 1345 Acked 2344 Write 8 Done Done New false
3821 1346 Acked 2345 Write 8 Done Done New false
3822 1347 Acked 2346 Read 10 Done Done New false
3823 1348 Acked 2347 Read 9 Done Done New false
3824 1349 Acked 2348 Write 9 Done Done New false
3825 1350 Acked 2349 Write 9 Done Done New false
3826 1351 Acked 2350 Write 2 Done Done New false
3827 1352 Acked 2351 Flush 0 Done Done New false
3828 1353 Acked 2352 Write 2 Done Done New false
3829 1354 Acked 2353 Write 8 Done Done New false
3830 1355 Acked 2354 Write 2 Done Done New false
3831 1356 Acked 2355 Read 1 Done Done New false
3832 1357 Acked 2356 Read 5 Done Done New false
3833 1358 Acked 2357 Write 3 Done Done New false
3834 1359 Acked 2358 Write 8 Done Done New false
3835 1360 Acked 2359 Write 9 Done Done New false
3836 1361 Acked 2360 Read 7 Done Done New false
3837 1362 Acked 2361 Read 6 Done Done New false
3838 1363 Acked 2362 Write 4 Done Done New false
3839 1364 Acked 2363 Flush 0 Done Done New false
3840 1365 Acked 2364 Write 5 Done Done New false
3841 1366 Acked 2365 Read 3 Done Done New false
3842 1367 Acked 2366 Write 9 Done Done New false
3843 1368 Acked 2367 Write 10 Done Done New false
3844 1369 Acked 2368 Write 9 Done Done New false
3845 1370 Acked 2369 Write 2 Done Done New false
3846 1371 Acked 2370 Write 2 Done Done New false
3847 1372 Acked 2371 Read 5 Done Done New false
3848 1373 Acked 2372 Write 7 Done Done New false
3849 1374 Acked 2373 Write 8 Done Done New false
3850 1375 Acked 2374 Read 2 Done Done New false
3851 1376 Acked 2375 Read 5 Done Done New false
3852 1377 Acked 2376 Read 8 Done Done New false
3853 1378 Acked 2377 Write 6 Done Done New false
3854 1379 Acked 2378 Read 1 Done Done New false
3855 1380 Acked 2379 Read 8 Done Done New false
3856 1381 Acked 2380 Write 2 Done Done New false
3857 1382 Acked 2381 Flush 0 Done Done New false
3858 1383 Acked 2382 Read 7 Done Done New false
3859 1384 Acked 2383 Read 3 Done Done New false
3860 1385 Acked 2384 Read 2 Done Done New false
3861 1386 Acked 2385 Read 4 Done Done New false
3862 1387 Acked 2386 Read 4 Done Done New false
3863 1388 Acked 2387 Read 10 Done Done New false
3864 1389 Acked 2388 Write 9 Done Done New false
3865 1390 Acked 2389 Read 7 Done Done New false
3866 1391 Acked 2390 Write 7 Done Done New false
3867 1392 Acked 2391 Read 8 Done Done New false
3868 1393 Acked 2392 Read 8 Done Done New false
3869 1394 Acked 2393 Write 9 Done Done New false
3870 1395 Acked 2394 Read 2 Done Done New false
3871 1396 Acked 2395 Write 5 Done Done New false
3872 1397 Acked 2396 Read 8 Done Done New false
3873 1398 Acked 2397 Write 2 Done Done New false
3874 1399 Acked 2398 Read 7 Done Done New false
3875 1400 Acked 2399 Read 9 Done Done New false
3876 1401 Acked 2400 Write 1 Done Done New false
3877 1402 Acked 2401 Write 6 Done Done New false
3878 1403 Acked 2402 Read 4 Done Done New false
3879 1404 Acked 2403 Read 8 Done Done New false
3880 1405 Acked 2404 Read 6 Done Done New false
3881 1406 Acked 2405 Read 1 Done Done New false
3882 1407 Acked 2406 Write 7 Done Done New false
3883 1408 Acked 2407 Read 2 Done Done New false
3884 1409 Acked 2408 Write 10 Done Done New false
3885 1410 Acked 2409 Read 7 Done Done New false
3886 1411 Acked 2410 Flush 0 Done Done New false
3887 1412 Acked 2411 Write 3 Done Done New false
3888 1413 Acked 2412 Read 6 Done Done New false
3889 1414 Acked 2413 Write 3 Done Done New false
3890 1415 Acked 2414 Read 5 Done Done New false
3891 1416 Acked 2415 Write 10 Done Done New false
3892 1417 Acked 2416 Read 2 Done Done New false
3893 1418 Acked 2417 Write 1 Done Done New false
3894 1419 Acked 2418 Read 4 Done Done New false
3895 1420 Acked 2419 Read 5 Done Done New false
3896 1421 Acked 2420 Write 10 Done Done New false
3897 1422 Acked 2421 Write 3 Done Done New false
3898 1423 Acked 2422 Read 9 Done Done New false
3899 1424 Acked 2423 Flush 0 Done Done New false
3900 1425 Acked 2424 Read 8 Done Done New false
3901 1426 Acked 2425 Write 4 Done Done New false
3902 1427 Acked 2426 Read 7 Done Done New false
3903 1428 Acked 2427 Write 8 Done Done New false
3904 1429 Acked 2428 Write 5 Done Done New false
3905 1430 Acked 2429 Write 10 Done Done New false
3906 1431 Acked 2430 Flush 0 Done Done New false
3907 1432 Acked 2431 Read 6 Done Done New false
3908 1433 Acked 2432 Read 9 Done Done New false
3909 1434 Acked 2433 Read 5 Done Done New false
3910 1435 Acked 2434 Read 3 Done Done New false
3911 1436 Acked 2435 Read 6 Done Done New false
3912 1437 Acked 2436 Read 9 Done Done New false
3913 1438 Acked 2437 Read 8 Done Done New false
3914 1439 Acked 2438 Flush 0 Done Done New false
3915 1440 Acked 2439 Read 4 Done Done New false
3916 1441 Acked 2440 Write 9 Done Done New false
3917 1442 Acked 2441 Read 10 Done Done New false
3918 1443 Acked 2442 Read 4 Done Done New false
3919 1444 Acked 2443 Write 5 Done Done New false
3920 1445 Acked 2444 Write 10 Done Done New false
3921 1446 Acked 2445 Read 3 Done Done New false
3922 1447 Acked 2446 Read 10 Done Done New false
3923 1448 Acked 2447 Write 8 Done Done New false
3924 1449 Acked 2448 Write 2 Done Done New false
3925 1450 Acked 2449 Flush 0 Done Done New false
3926 1451 Acked 2450 Write 7 Done Done New false
3927 1452 Acked 2451 Read 10 Done Done New false
3928 1453 Acked 2452 Write 8 Done Done New false
3929 1454 Acked 2453 Read 3 Done Done New false
3930 1455 Acked 2454 Write 4 Done Done New false
3931 1456 Acked 2455 Read 8 Done Done New false
3932 1457 Acked 2456 Read 6 Done Done New false
3933 1458 Acked 2457 Write 6 Done Done New false
3934 1459 Acked 2458 Read 3 Done Done New false
3935 1460 Acked 2459 Write 3 Done Done New false
3936 1461 Acked 2460 Read 9 Done Done New false
3937 1462 Acked 2461 Flush 0 Done Done New false
3938 1463 Acked 2462 Read 2 Done Done New false
3939 1464 Acked 2463 Read 10 Done Done New false
3940 1465 Acked 2464 Flush 0 Done Done New false
3941 1466 Acked 2465 Write 7 Done Done New false
3942 1467 Acked 2466 Write 2 Done Done New false
3943 1468 Acked 2467 Flush 0 Done Done New false
3944 1469 Acked 2468 Flush 0 Done Done New false
3945 1470 Acked 2469 Write 9 Done Done New false
3946 1471 Acked 2470 Flush 0 Done Done New false
3947 1472 Acked 2471 Write 10 Done Done New false
3948 1473 Acked 2472 Flush 0 Done Done New false
3949 1474 Acked 2473 Write 8 Done Done New false
3950 1475 Acked 2474 Write 1 Done Done New false
3951 1476 Acked 2475 Flush 0 Done Done New false
3952 1477 Acked 2476 Read 9 Done Done New false
3953 1478 Acked 2477 Write 1 Done Done New false
3954 1479 Acked 2478 Read 4 Done Done New false
3955 1480 Acked 2479 Read 5 Done Done New false
3956 1481 Acked 2480 Read 2 Done Done New false
3957 1482 Acked 2481 Read 6 Done Done New false
3958 1483 Acked 2482 Read 10 Done Done New false
3959 1484 Acked 2483 Flush 0 Done Done New false
3960 1485 Acked 2484 Write 3 Done Done New false
3961 1486 Acked 2485 Read 2 Done Done New false
3962 1487 Acked 2486 Read 10 Done Done New false
3963 1488 Acked 2487 Read 5 Done Done New false
3964 1489 Acked 2488 Write 9 Done Done New false
3965 1490 Acked 2489 Write 4 Done Done New false
3966 1491 Acked 2490 Write 10 Done Done New false
3967 1492 Acked 2491 Write 4 Done Done New false
3968 1493 Acked 2492 Read 5 Done Done New false
3969 1494 Acked 2493 Read 7 Done Done New false
3970 1495 Acked 2494 Write 9 Done Done New false
3971 1496 Acked 2495 Flush 0 Done Done New false
3972 1497 Acked 2496 Read 10 Done Done New false
3973 1498 Acked 2497 Read 2 Done Done New false
3974 1499 Acked 2498 Write 10 Done Done New false
3975 1500 Acked 2499 Read 9 Done Done New false
3976 1501 Acked 2500 Write 5 Done Done New false
3977 1502 Acked 2501 Read 2 Done Done New false
3978 1503 Acked 2502 Write 8 Done Done New false
3979 1504 Acked 2503 Write 4 Done Done New false
3980 1505 Acked 2504 Write 7 Done Done New false
3981 1506 Acked 2505 Read 2 Done Done New false
3982 STATES DS:0 DS:1 DS:2 TOTAL
3983 New 0 0 300 300
3984 Sent 0 0 0 0
3985 Done 300 300 0 600
3986 Skipped 0 0 0 0
3987 Error 0 0 0 0
3988 Last Flush: 2495 2495 2205
3989 Downstairs last five completed: 2205 2204 2203 2202 2201
3990 Upstairs last five completed: 1506 1505 1504 1503 1502
3991 CLIENT: Up:0 ds:300 act:2
39922024-02-28T22:13:57.667ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
39932024-02-28T22:13:57.667ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
39942024-02-28T22:13:57.667ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
39952024-02-28T22:13:57.667ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
39962024-02-28T22:13:57.667ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
39972024-02-28T22:13:57.667ZINFOcrucible: [2] client re-new 301 jobs since flush 2205 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
39982024-02-28T22:13:57.667ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
39992024-02-28T22:13:58.668ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40002024-02-28T22:13:58.668ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40012024-02-28T22:13:58.668ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40022024-02-28T22:13:58.668ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40032024-02-28T22:13:58.668ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40042024-02-28T22:13:58.668ZINFOcrucible: [2] client re-new 301 jobs since flush 2205 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40052024-02-28T22:13:58.668ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
4006 [8830] got start action
4007 [8830] I am going to restart
4008 Starting downstairs at port 8830
4009 Make output file at "/tmp/dsc/downstairs-8830.txt"
4010 [8830][2] reports Starting
4011 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1169)
4012 [8830][2] reports Running
40132024-02-28T22:13:59.669ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40142024-02-28T22:13:59.670ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40152024-02-28T22:13:59.671ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40162024-02-28T22:13:59.671ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40172024-02-28T22:13:59.671ZINFOcrucible: send last flush ID to this DS: 2205 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40182024-02-28T22:13:59.671ZINFOcrucible: Replied this last flush ID: 2205 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40192024-02-28T22:13:59.671ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40202024-02-28T22:13:59.671ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40212024-02-28T22:13:59.671ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40222024-02-28T22:13:59.671ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
4023 ----------------------------------------------------------------
4024 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
4025 Downstairs last five completed: 2506 2505 2504 2503 2502
4026 Upstairs last five completed: 1507 1506 1505 1504 1503
4027 CLIENT: Up:0 ds:0 act:3
4028 Replay: All jobs finished, all DS active.
4029 Main task has work to do, go find it
4030 got dsc Stop(2)
4031 stop 2
4032 Feb 28 22:14:01.320 INFO request completed[8830] Got stop action so:false kr:false
4033 , latency_us: 341, response_code: 204, uri: /stop/cid/2, method: POST, req_id: faa00cbe-4389-496f-94b3-61bf467893c9, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
4034 Feb 28 22:14:01.321 INFO accepted connection, remote_addr: 127.0.0.1:38750, local_addr: 127.0.0.1:9998
40352024-02-28T22:14:01.321ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40362024-02-28T22:14:01.321ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40372024-02-28T22:14:01.321ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40382024-02-28T22:14:01.321ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40392024-02-28T22:14:01.321ZINFOcrucible: [2] client re-new 0 jobs since flush 2506 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40402024-02-28T22:14:01.321ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
4041 Feb 28 22:14:01.321 INFO request completed, latency_us: 165, response_code: 200, uri: /state/cid/2, method: GET, req_id: 98cf8404-1442-4b1c-8de9-2511f933ab4f, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
4042 [8830] Exited with: ExitStatus(unix_wait_status(9))
4043 [8830][2] reports Exit
40442024-02-28T22:14:02.322ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40452024-02-28T22:14:02.323ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40462024-02-28T22:14:02.323ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40472024-02-28T22:14:02.323ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40482024-02-28T22:14:02.323ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40492024-02-28T22:14:02.323ZINFOcrucible: [2] client re-new 0 jobs since flush 2506 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40502024-02-28T22:14:02.323ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40512024-02-28T22:14:03.325ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40522024-02-28T22:14:03.325ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40532024-02-28T22:14:03.325ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40542024-02-28T22:14:03.325ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40552024-02-28T22:14:03.325ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40562024-02-28T22:14:03.325ZINFOcrucible: [2] client re-new 0 jobs since flush 2506 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40572024-02-28T22:14:03.325ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40582024-02-28T22:14:04.326ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40592024-02-28T22:14:04.326ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40602024-02-28T22:14:04.326ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40612024-02-28T22:14:04.326ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40622024-02-28T22:14:04.326ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40632024-02-28T22:14:04.326ZINFOcrucible: [2] client re-new 0 jobs since flush 2506 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40642024-02-28T22:14:04.326ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
4065 Feb 28 22:14:05.323 INFO request completed, latency_us: 335, response_code: 200, uri: /state/cid/2, method: GET, req_id: 3ee8f2b1-c528-4b90-b82d-24bf0396841d, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
4066 001/300 Flush
4067 002/300 Read block 3097 len 20480
4068 003/300 Write block 2890 len 8192 data: 4 3
4069 004/300 Read block 1986 len 12288
40702024-02-28T22:14:05.327ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40712024-02-28T22:14:05.327ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40722024-02-28T22:14:05.327ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40732024-02-28T22:14:05.327ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40742024-02-28T22:14:05.327ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40752024-02-28T22:14:05.327ZINFOcrucible: [2] client re-new 4 jobs since flush 2506 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
40762024-02-28T22:14:05.327ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
4077 005/300 Write block 4091 len 24576 data: 4 3 3 3 3 3
4078 006/300 Read block 4869 len 32768
4079 007/300 Flush
4080 008/300 Read block 1670 len 32768
4081 009/300 Read block 1622 len 12288
4082 010/300 Read block 4036 len 32768
4083 011/300 Write block 1855 len 28672 data: 2 2 2 3 3 3 3
4084 012/300 Write block 506 len 16384 data: 2 2 3 3
4085 013/300 Read block 1209 len 8192
4086 014/300 Flush
4087 015/300 Write block 4496 len 4096 data: 5
4088 016/300 Write block 4432 len 32768 data: 3 3 3 3 3 2 2 2
4089 017/300 Flush
4090 018/300 Write block 294 len 20480 data: 3 3 3 3 3
4091 019/300 Write block 2 len 24576 data: 3 3 3 3 3 3
4092 020/300 Read block 262 len 28672
4093 021/300 Read block 3509 len 36864
4094 022/300 Read block 2508 len 8192
4095 023/300 Write block 656 len 4096 data: 5
4096 024/300 Write block 4012 len 36864 data: 3 4 5 4 4 3 3 3 3
4097 025/300 Read block 1141 len 20480
4098 026/300 Write block 2438 len 8192 data: 4 4
4099 027/300 Write block 154 len 28672 data: 3 2 2 2 2 2 3
4100 028/300 Flush
4101 029/300 Write block 1836 len 4096 data: 2
4102 030/300 Write block 3595 len 4096 data: 2
4103 031/300 Read block 1392 len 28672
4104 032/300 Write block 1740 len 40960 data: 3 3 3 3 2 2 2 2 2 2
4105 033/300 Write block 3555 len 40960 data: 3 2 2 2 2 2 2 2 2 2
4106 034/300 Read block 4440 len 40960
4107 035/300 Read block 1366 len 12288
4108 036/300 Write block 1591 len 20480 data: 2 2 2 2 2
4109 037/300 Read block 2268 len 36864
4110 038/300 Flush
4111 039/300 Write block 4257 len 32768 data: 4 4 3 2 3 3 3 3
4112 040/300 Write block 1211 len 40960 data: 3 2 2 2 2 2 2 2 2 2
4113 041/300 Write block 3648 len 36864 data: 2 2 2 2 2 2 2 2 3
4114 042/300 Read block 1461 len 32768
4115 043/300 Read block 321 len 16384
4116 044/300 Write block 510 len 28672 data: 2 2 2 2 2 2 3
4117 045/300 Read block 4257 len 20480
4118 046/300 Flush
4119 047/300 Write block 268 len 16384 data: 2 2 2 2
4120 048/300 Read block 2659 len 16384
4121 049/300 Write block 472 len 36864 data: 3 2 3 3 3 2 2 2 2
4122 050/300 Write block 114 len 36864 data: 2 2 2 3 3 3 2 2 3
4123 051/300 Write block 2883 len 40960 data: 4 4 3 4 4 4 4 5 4 3
4124 052/300 Write block 681 len 4096 data: 5
4125 053/300 Write block 2310 len 28672 data: 2 2 2 2 2 2 2
4126 054/300 Read block 3624 len 16384
4127 055/300 Read block 226 len 12288
4128 056/300 Read block 201 len 28672
4129 057/300 Read block 2767 len 20480
4130 058/300 Write block 999 len 40960 data: 3 3 3 3 3 3 2 2 2 3
4131 059/300 Write block 575 len 4096 data: 3
4132 060/300 Write block 2895 len 40960 data: 2 2 2 2 2 3 4 4 4 4
4133 061/300 Read block 3362 len 12288
4134 062/300 Write block 1295 len 12288 data: 2 2 2
4135 063/300 Read block 2066 len 28672
4136 064/300 Read block 1041 len 32768
4137 065/300 Read block 4352 len 24576
4138 066/300 Read block 2715 len 28672
4139 067/300 Read block 4952 len 8192
4140 068/300 Write block 2453 len 28672 data: 2 2 2 2 2 2 2
4141 069/300 Flush
4142 070/300 Read block 3709 len 32768
4143 071/300 Read block 4163 len 40960
4144 072/300 Write block 2972 len 28672 data: 4 4 4 4 3 3 3
4145 073/300 Flush
4146 074/300 Read block 3434 len 4096
4147 075/300 Read block 3582 len 12288
4148 076/300 Read block 2280 len 24576
4149 077/300 Flush
4150 078/300 Write block 1599 len 12288 data: 2 2 2
4151 079/300 Read block 1478 len 12288
4152 080/300 Flush
4153 081/300 Read block 1012 len 36864
4154 082/300 Write block 1140 len 16384 data: 4 4 4 4
4155 083/300 Flush
4156 084/300 Read block 4312 len 8192
4157 085/300 Write block 2357 len 4096 data: 2
4158 086/300 Flush
4159 087/300 Flush
4160 088/300 Write block 4122 len 36864 data: 3 3 2 2 2 4 3 3 2
4161 089/300 Read block 1338 len 32768
4162 090/300 Read block 2267 len 32768
4163 091/300 Read block 4007 len 36864
4164 092/300 Write block 2421 len 4096 data: 3
4165 093/300 Read block 4226 len 12288
4166 094/300 Read block 81 len 28672
4167 095/300 Write block 134 len 16384 data: 3 2 2 2
4168 096/300 Read block 2678 len 40960
4169 097/300 Write block 408 len 36864 data: 3 3 3 3 3 2 2 2 2
4170 098/300 Read block 4097 len 12288
4171 099/300 Flush
4172 100/300 Read block 4896 len 32768
4173 101/300 Write block 2920 len 12288 data: 3 3 3
4174 102/300 Write block 2139 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4175 103/300 Read block 755 len 40960
4176 104/300 Write block 1944 len 16384 data: 2 2 2 2
4177 105/300 Read block 3743 len 40960
4178 106/300 Read block 4449 len 40960
4179 107/300 Read block 1823 len 8192
4180 108/300 Write block 2059 len 20480 data: 2 2 2 2 2
4181 109/300 Read block 4835 len 4096
4182 110/300 Read block 1712 len 36864
4183 111/300 Read block 4451 len 12288
4184 112/300 Write block 1710 len 4096 data: 2
4185 113/300 Write block 3504 len 24576 data: 3 3 3 3 2 2
4186 114/300 Write block 4975 len 12288 data: 3 3 3
4187 115/300 Read block 1124 len 4096
4188 116/300 Read block 660 len 36864
4189 117/300 Read block 2 len 20480
4190 118/300 Read block 1793 len 20480
4191 119/300 Write block 2939 len 20480 data: 2 2 2 2 2
4192 120/300 Flush
4193 121/300 Read block 1851 len 12288
4194 122/300 Read block 518 len 16384
4195 123/300 Write block 4515 len 4096 data: 3
4196 124/300 Write block 4555 len 32768 data: 2 2 2 2 2 2 2 2
4197 125/300 Write block 1919 len 8192 data: 2 2
4198 126/300 Write block 2975 len 32768 data: 5 4 4 4 2 2 2 2
4199 127/300 Write block 134 len 4096 data: 4
4200 128/300 Read block 4057 len 4096
4201 129/300 Write block 3012 len 12288 data: 3 3 3
4202 130/300 Read block 577 len 16384
4203 131/300 Write block 2405 len 8192 data: 2 2
4204 132/300 Read block 2566 len 40960
4205 133/300 Read block 4365 len 20480
4206 134/300 Read block 4581 len 16384
4207 135/300 Flush
4208 136/300 Read block 2108 len 16384
4209 137/300 Write block 3612 len 12288 data: 2 3 3
4210 138/300 Read block 3744 len 4096
4211 139/300 Read block 110 len 8192
4212 140/300 Write block 2429 len 40960 data: 4 4 4 4 3 2 2 2 3 5
4213 141/300 Read block 2594 len 16384
4214 142/300 Write block 1748 len 40960 data: 3 3 2 2 2 2 2 2 2 2
4215 143/300 Read block 53 len 24576
4216 144/300 Read block 4078 len 8192
4217 145/300 Write block 2579 len 28672 data: 3 3 3 3 2 2 2
4218 146/300 Write block 1133 len 36864 data: 3 3 3 3 3 3 3 5 5
4219 147/300 Read block 1588 len 24576
4220 148/300 Write block 1716 len 20480 data: 6 5 4 3 4
4221 149/300 Read block 103 len 24576
4222 150/300 Read block 51 len 36864
4223 151/300 Read block 2794 len 40960
4224 152/300 Read block 4692 len 24576
4225 153/300 Write block 24 len 20480 data: 2 2 2 2 2
4226 154/300 Read block 3428 len 8192
4227 155/300 Write block 1542 len 32768 data: 2 2 2 2 2 2 2 2
4228 156/300 Write block 400 len 24576 data: 3 2 2 2 2 2
4229 157/300 Flush
4230 158/300 Read block 4910 len 4096
4231 159/300 Read block 1026 len 20480
4232 160/300 Read block 4351 len 4096
4233 161/300 Write block 2167 len 12288 data: 2 2 2
4234 162/300 Read block 3478 len 16384
4235 163/300 Read block 4641 len 36864
4236 164/300 Write block 3204 len 32768 data: 3 3 3 3 3 3 2 2
4237 165/300 Read block 2195 len 20480
4238 166/300 Read block 1977 len 40960
4239 167/300 Read block 117 len 8192
4240 168/300 Read block 3857 len 8192
4241 169/300 Read block 468 len 40960
4242 170/300 Write block 559 len 28672 data: 2 2 2 2 2 2 2
4243 171/300 Read block 1600 len 12288
4244 172/300 Read block 3504 len 16384
4245 173/300 Read block 2135 len 28672
4246 174/300 Write block 4732 len 36864 data: 2 2 2 2 2 2 2 2 2
4247 175/300 Read block 1452 len 28672
4248 176/300 Read block 4476 len 4096
4249 177/300 Read block 725 len 36864
4250 178/300 Write block 909 len 24576 data: 2 2 2 2 2 2
4251 179/300 Write block 4277 len 20480 data: 2 2 2 2 2
4252 180/300 Write block 98 len 4096 data: 2
4253 181/300 Read block 3312 len 32768
4254 182/300 Write block 939 len 20480 data: 2 2 2 2 2
4255 183/300 Write block 4088 len 16384 data: 3 3 3 5
4256 184/300 Write block 958 len 36864 data: 4 5 5 5 4 5 4 4 4
4257 185/300 Write block 1915 len 16384 data: 2 2 2 2
4258 186/300 Read block 3700 len 8192
4259 187/300 Write block 714 len 24576 data: 3 3 2 2 2 2
4260 188/300 Write block 951 len 8192 data: 3 3
4261 189/300 Write block 2462 len 8192 data: 2 2
4262 190/300 Read block 3146 len 8192
4263 191/300 Read block 1855 len 40960
4264 192/300 Write block 3387 len 36864 data: 2 2 3 3 3 3 3 4 5
4265 193/300 Write block 3884 len 8192 data: 6 5
4266 194/300 Read block 2100 len 12288
4267 195/300 Write block 1555 len 28672 data: 2 2 2 2 3 3 3
4268 196/300 Write block 1210 len 4096 data: 3
4269 197/300 Write block 2794 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4270 198/300 Read block 2410 len 24576
4271 199/300 Read block 3047 len 12288
4272 200/300 Read block 3234 len 24576
4273 201/300 Read block 405 len 20480
4274 202/300 Write block 3815 len 12288 data: 3 3 2
4275 203/300 Write block 119 len 40960 data: 4 3 3 4 3 3 3 3 3 2
4276 204/300 Flush
4277 205/300 Read block 1485 len 8192
4278 206/300 Read block 180 len 32768
4279 207/300 Write block 3079 len 24576 data: 2 2 2 2 3 2
4280 208/300 Write block 2310 len 28672 data: 3 3 3 3 3 3 3
4281 209/300 Read block 3664 len 32768
4282 210/300 Read block 2045 len 16384
4283 211/300 Write block 4709 len 28672 data: 2 2 2 3 3 3 3
4284 212/300 Read block 754 len 24576
4285 213/300 Write block 1114 len 16384 data: 4 4 4 4
4286 214/300 Write block 2692 len 4096 data: 2
4287 215/300 Read block 4087 len 20480
4288 216/300 Read block 3591 len 16384
4289 217/300 Flush
4290 218/300 Flush
4291 219/300 Write block 2336 len 32768 data: 3 3 3 3 2 2 2 2
4292 220/300 Read block 1513 len 36864
4293 221/300 Write block 2357 len 16384 data: 3 2 3 4
4294 222/300 Read block 2018 len 4096
4295 223/300 Write block 1762 len 16384 data: 5 5 5 5
4296 224/300 Read block 1459 len 32768
4297 225/300 Flush
4298 226/300 Write block 3400 len 40960 data: 2 2 2 2 2 2 2 3 3 3
4299 227/300 Write block 4499 len 36864 data: 7 5 3 3 3 4 3 3 3
4300 228/300 Write block 2875 len 12288 data: 2 2 3
4301 229/300 Write block 3040 len 36864 data: 2 2 2 2 2 2 2 2 2
4302 230/300 Write block 4865 len 4096 data: 2
4303 231/300 Write block 4089 len 28672 data: 4 4 6 4 4 4 4
4304 232/300 Read block 4924 len 40960
4305 233/300 Read block 890 len 16384
4306 234/300 Read block 984 len 8192
4307 235/300 Read block 3427 len 12288
4308 236/300 Flush
4309 237/300 Read block 1465 len 12288
4310 238/300 Write block 3443 len 8192 data: 2 2
4311 239/300 Read block 291 len 28672
4312 240/300 Read block 253 len 32768
4313 241/300 Write block 803 len 40960 data: 2 2 2 2 2 2 2 3 3 2
4314 242/300 Write block 4902 len 12288 data: 3 3 3
4315 243/300 Read block 4169 len 36864
4316 244/300 Write block 154 len 32768 data: 4 3 3 3 3 3 4 3
4317 245/300 Write block 3550 len 12288 data: 6 6 5
4318 246/300 Write block 1283 len 12288 data: 2 2 2
4319 247/300 Flush
4320 248/300 Write block 3087 len 32768 data: 2 2 2 2 2 2 2 2
4321 249/300 Write block 3582 len 32768 data: 2 2 2 2 2 2 2 2
4322 250/300 Read block 3617 len 32768
4323 251/300 Read block 333 len 16384
4324 252/300 Write block 3958 len 8192 data: 5 4
4325 253/300 Flush
4326 254/300 Write block 2036 len 32768 data: 6 5 2 2 2 2 2 2
4327 255/300 Read block 1146 len 4096
4328 256/300 Write block 3887 len 8192 data: 4 4
4329 257/300 Read block 4758 len 20480
4330 258/300 Write block 4759 len 24576 data: 3 3 3 3 4 3
4331 259/300 Write block 1220 len 40960 data: 3 2 2 2 2 2 2 3 3 3
4332 260/300 Write block 212 len 12288 data: 2 2 2
4333 261/300 Flush
4334 262/300 Write block 2084 len 8192 data: 2 2
4335 263/300 Write block 3534 len 8192 data: 3 3
4336 264/300 Write block 2507 len 4096 data: 2
4337 265/300 Write block 1989 len 16384 data: 2 2 3 3
4338 266/300 Read block 4679 len 20480
4339 267/300 Read block 712 len 8192
4340 268/300 Read block 4986 len 24576
4341 269/300 Read block 1805 len 8192
4342 270/300 Read block 1465 len 16384
4343 271/300 Read block 81 len 40960
4344 272/300 Read block 579 len 36864
4345 273/300 Read block 1432 len 20480
4346 274/300 Read block 1514 len 28672
4347 275/300 Write block 3512 len 36864 data: 3 3 3 3 3 3 3 3 4
4348 276/300 Read block 2236 len 4096
4349 277/300 Read block 665 len 8192
4350 278/300 Read block 1080 len 24576
4351 279/300 Read block 3350 len 16384
4352 280/300 Read block 1566 len 36864
4353 281/300 Flush
4354 282/300 Write block 1630 len 32768 data: 4 3 3 3 3 2 3 4
4355 283/300 Write block 4883 len 16384 data: 2 2 2 2
4356 284/300 Write block 3120 len 20480 data: 2 2 4 3 3
4357 285/300 Read block 2790 len 32768
4358 286/300 Read block 233 len 16384
4359 287/300 Write block 2372 len 4096 data: 3
4360 288/300 Write block 481 len 16384 data: 2 2 2 4
4361 289/300 Read block 4934 len 40960
4362 290/300 Read block 3736 len 24576
4363 291/300 Read block 4398 len 20480
4364 292/300 Read block 810 len 20480
4365 293/300 Write block 2725 len 16384 data: 2 3 3 3
4366 294/300 Write block 1733 len 16384 data: 2 2 2 2
4367 295/300 Read block 220 len 36864
4368 296/300 Read block 3503 len 8192
4369 297/300 Read block 3142 len 40960
4370 298/300 Read block 814 len 40960
4371 299/300 Read block 1943 len 40960
4372 300/300 Read block 2373 len 40960
4373 Main task has work to do, go find it
4374 got dsc Start(2)
4375 start 2
4376 Feb 28 22:14:05.864 INFO request completed, latency_us: 228, response_code: 204, uri: /start/cid/2, method: POST, req_id: 5464065e-f232-449d-88c0-f95872724604, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
4377 Replay: started 2, returned:Ok(())
4378 ----------------------------------------------------------------
4379 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
4380 States: Active Active Offline
4381 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
4382 1508 Acked 2507 Flush 0 Done Done New true
4383 1509 Acked 2508 Read 5 Done Done New true
4384 1510 Acked 2509 Write 2 Done Done New true
4385 1511 Acked 2510 Read 3 Done Done New true
4386 1512 Acked 2511 Write 6 Done Done New false
4387 1513 Acked 2512 Read 8 Done Done New false
4388 1514 Acked 2513 Flush 0 Done Done New false
4389 1515 Acked 2514 Read 8 Done Done New false
4390 1516 Acked 2515 Read 3 Done Done New false
4391 1517 Acked 2516 Read 8 Done Done New false
4392 1518 Acked 2517 Write 7 Done Done New false
4393 1519 Acked 2518 Write 4 Done Done New false
4394 1520 Acked 2519 Read 2 Done Done New false
4395 1521 Acked 2520 Flush 0 Done Done New false
4396 1522 Acked 2521 Write 1 Done Done New false
4397 1523 Acked 2522 Write 8 Done Done New false
4398 1524 Acked 2523 Flush 0 Done Done New false
4399 1525 Acked 2524 Write 5 Done Done New false
4400 1526 Acked 2525 Write 6 Done Done New false
4401 1527 Acked 2526 Read 7 Done Done New false
4402 1528 Acked 2527 Read 9 Done Done New false
4403 1529 Acked 2528 Read 2 Done Done New false
4404 1530 Acked 2529 Write 1 Done Done New false
4405 1531 Acked 2530 Write 9 Done Done New false
4406 1532 Acked 2531 Read 5 Done Done New false
4407 1533 Acked 2532 Write 2 Done Done New false
4408 1534 Acked 2533 Write 7 Done Done New false
4409 1535 Acked 2534 Flush 0 Done Done New false
4410 1536 Acked 2535 Write 1 Done Done New false
4411 1537 Acked 2536 Write 1 Done Done New false
4412 1538 Acked 2537 Read 7 Done Done New false
4413 1539 Acked 2538 Write 10 Done Done New false
4414 1540 Acked 2539 Write 10 Done Done New false
4415 1541 Acked 2540 Read 10 Done Done New false
4416 1542 Acked 2541 Read 3 Done Done New false
4417 1543 Acked 2542 Write 5 Done Done New false
4418 1544 Acked 2543 Read 9 Done Done New false
4419 1545 Acked 2544 Flush 0 Done Done New false
4420 1546 Acked 2545 Write 8 Done Done New false
4421 1547 Acked 2546 Write 10 Done Done New false
4422 1548 Acked 2547 Write 9 Done Done New false
4423 1549 Acked 2548 Read 8 Done Done New false
4424 1550 Acked 2549 Read 4 Done Done New false
4425 1551 Acked 2550 Write 7 Done Done New false
4426 1552 Acked 2551 Read 5 Done Done New false
4427 1553 Acked 2552 Flush 0 Done Done New false
4428 1554 Acked 2553 Write 4 Done Done New false
4429 1555 Acked 2554 Read 4 Done Done New false
4430 1556 Acked 2555 Write 9 Done Done New false
4431 1557 Acked 2556 Write 9 Done Done New false
4432 1558 Acked 2557 Write 10 Done Done New false
4433 1559 Acked 2558 Write 1 Done Done New false
4434 1560 Acked 2559 Write 7 Done Done New false
4435 1561 Acked 2560 Read 4 Done Done New false
4436 1562 Acked 2561 Read 3 Done Done New false
4437 1563 Acked 2562 Read 7 Done Done New false
4438 1564 Acked 2563 Read 5 Done Done New false
4439 1565 Acked 2564 Write 10 Done Done New false
4440 1566 Acked 2565 Write 1 Done Done New false
4441 1567 Acked 2566 Write 10 Done Done New false
4442 1568 Acked 2567 Read 3 Done Done New false
4443 1569 Acked 2568 Write 3 Done Done New false
4444 1570 Acked 2569 Read 7 Done Done New false
4445 1571 Acked 2570 Read 8 Done Done New false
4446 1572 Acked 2571 Read 6 Done Done New false
4447 1573 Acked 2572 Read 7 Done Done New false
4448 1574 Acked 2573 Read 2 Done Done New false
4449 1575 Acked 2574 Write 7 Done Done New false
4450 1576 Acked 2575 Flush 0 Done Done New false
4451 1577 Acked 2576 Read 8 Done Done New false
4452 1578 Acked 2577 Read 10 Done Done New false
4453 1579 Acked 2578 Write 7 Done Done New false
4454 1580 Acked 2579 Flush 0 Done Done New false
4455 1581 Acked 2580 Read 1 Done Done New false
4456 1582 Acked 2581 Read 3 Done Done New false
4457 1583 Acked 2582 Read 6 Done Done New false
4458 1584 Acked 2583 Flush 0 Done Done New false
4459 1585 Acked 2584 Write 3 Done Done New false
4460 1586 Acked 2585 Read 3 Done Done New false
4461 1587 Acked 2586 Flush 0 Done Done New false
4462 1588 Acked 2587 Read 9 Done Done New false
4463 1589 Acked 2588 Write 4 Done Done New false
4464 1590 Acked 2589 Flush 0 Done Done New false
4465 1591 Acked 2590 Read 2 Done Done New false
4466 1592 Acked 2591 Write 1 Done Done New false
4467 1593 Acked 2592 Flush 0 Done Done New false
4468 1594 Acked 2593 Flush 0 Done Done New false
4469 1595 Acked 2594 Write 9 Done Done New false
4470 1596 Acked 2595 Read 8 Done Done New false
4471 1597 Acked 2596 Read 8 Done Done New false
4472 1598 Acked 2597 Read 9 Done Done New false
4473 1599 Acked 2598 Write 1 Done Done New false
4474 1600 Acked 2599 Read 3 Done Done New false
4475 1601 Acked 2600 Read 7 Done Done New false
4476 1602 Acked 2601 Write 4 Done Done New false
4477 1603 Acked 2602 Read 10 Done Done New false
4478 1604 Acked 2603 Write 9 Done Done New false
4479 1605 Acked 2604 Read 3 Done Done New false
4480 1606 Acked 2605 Flush 0 Done Done New false
4481 1607 Acked 2606 Read 8 Done Done New false
4482 1608 Acked 2607 Write 3 Done Done New false
4483 1609 Acked 2608 Write 10 Done Done New false
4484 1610 Acked 2609 Read 10 Done Done New false
4485 1611 Acked 2610 Write 4 Done Done New false
4486 1612 Acked 2611 Read 10 Done Done New false
4487 1613 Acked 2612 Read 10 Done Done New false
4488 1614 Acked 2613 Read 2 Done Done New false
4489 1615 Acked 2614 Write 5 Done Done New false
4490 1616 Acked 2615 Read 1 Done Done New false
4491 1617 Acked 2616 Read 9 Done Done New false
4492 1618 Acked 2617 Read 3 Done Done New false
4493 1619 Acked 2618 Write 1 Done Done New false
4494 1620 Acked 2619 Write 6 Done Done New false
4495 1621 Acked 2620 Write 3 Done Done New false
4496 1622 Acked 2621 Read 1 Done Done New false
4497 1623 Acked 2622 Read 9 Done Done New false
4498 1624 Acked 2623 Read 5 Done Done New false
4499 1625 Acked 2624 Read 5 Done Done New false
4500 1626 Acked 2625 Write 5 Done Done New false
4501 1627 Acked 2626 Flush 0 Done Done New false
4502 1628 Acked 2627 Read 3 Done Done New false
4503 1629 Acked 2628 Read 4 Done Done New false
4504 1630 Acked 2629 Write 1 Done Done New false
4505 1631 Acked 2630 Write 8 Done Done New false
4506 1632 Acked 2631 Write 2 Done Done New false
4507 1633 Acked 2632 Write 8 Done Done New false
4508 1634 Acked 2633 Write 1 Done Done New false
4509 1635 Acked 2634 Read 1 Done Done New false
4510 1636 Acked 2635 Write 3 Done Done New false
4511 1637 Acked 2636 Read 4 Done Done New false
4512 1638 Acked 2637 Write 2 Done Done New false
4513 1639 Acked 2638 Read 10 Done Done New false
4514 1640 Acked 2639 Read 5 Done Done New false
4515 1641 Acked 2640 Read 4 Done Done New false
4516 1642 Acked 2641 Flush 0 Done Done New false
4517 1643 Acked 2642 Read 4 Done Done New false
4518 1644 Acked 2643 Write 3 Done Done New false
4519 1645 Acked 2644 Read 1 Done Done New false
4520 1646 Acked 2645 Read 2 Done Done New false
4521 1647 Acked 2646 Write 10 Done Done New false
4522 1648 Acked 2647 Read 4 Done Done New false
4523 1649 Acked 2648 Write 10 Done Done New false
4524 1650 Acked 2649 Read 6 Done Done New false
4525 1651 Acked 2650 Read 2 Done Done New false
4526 1652 Acked 2651 Write 7 Done Done New false
4527 1653 Acked 2652 Write 9 Done Done New false
4528 1654 Acked 2653 Read 6 Done Done New false
4529 1655 Acked 2654 Write 5 Done Done New false
4530 1656 Acked 2655 Read 6 Done Done New false
4531 1657 Acked 2656 Read 9 Done Done New false
4532 1658 Acked 2657 Read 10 Done Done New false
4533 1659 Acked 2658 Read 6 Done Done New false
4534 1660 Acked 2659 Write 5 Done Done New false
4535 1661 Acked 2660 Read 2 Done Done New false
4536 1662 Acked 2661 Write 8 Done Done New false
4537 1663 Acked 2662 Write 6 Done Done New false
4538 1664 Acked 2663 Flush 0 Done Done New false
4539 1665 Acked 2664 Read 1 Done Done New false
4540 1666 Acked 2665 Read 5 Done Done New false
4541 1667 Acked 2666 Read 1 Done Done New false
4542 1668 Acked 2667 Write 3 Done Done New false
4543 1669 Acked 2668 Read 4 Done Done New false
4544 1670 Acked 2669 Read 9 Done Done New false
4545 1671 Acked 2670 Write 8 Done Done New false
4546 1672 Acked 2671 Read 5 Done Done New false
4547 1673 Acked 2672 Read 10 Done Done New false
4548 1674 Acked 2673 Read 2 Done Done New false
4549 1675 Acked 2674 Read 2 Done Done New false
4550 1676 Acked 2675 Read 10 Done Done New false
4551 1677 Acked 2676 Write 7 Done Done New false
4552 1678 Acked 2677 Read 3 Done Done New false
4553 1679 Acked 2678 Read 4 Done Done New false
4554 1680 Acked 2679 Read 7 Done Done New false
4555 1681 Acked 2680 Write 9 Done Done New false
4556 1682 Acked 2681 Read 7 Done Done New false
4557 1683 Acked 2682 Read 1 Done Done New false
4558 1684 Acked 2683 Read 9 Done Done New false
4559 1685 Acked 2684 Write 6 Done Done New false
4560 1686 Acked 2685 Write 5 Done Done New false
4561 1687 Acked 2686 Write 1 Done Done New false
4562 1688 Acked 2687 Read 8 Done Done New false
4563 1689 Acked 2688 Write 5 Done Done New false
4564 1690 Acked 2689 Write 4 Done Done New false
4565 1691 Acked 2690 Write 9 Done Done New false
4566 1692 Acked 2691 Write 4 Done Done New false
4567 1693 Acked 2692 Read 2 Done Done New false
4568 1694 Acked 2693 Write 6 Done Done New false
4569 1695 Acked 2694 Write 2 Done Done New false
4570 1696 Acked 2695 Write 2 Done Done New false
4571 1697 Acked 2696 Read 2 Done Done New false
4572 1698 Acked 2697 Read 10 Done Done New false
4573 1699 Acked 2698 Write 9 Done Done New false
4574 1700 Acked 2699 Write 2 Done Done New false
4575 1701 Acked 2700 Read 3 Done Done New false
4576 1702 Acked 2701 Write 7 Done Done New false
4577 1703 Acked 2702 Write 1 Done Done New false
4578 1704 Acked 2703 Write 10 Done Done New false
4579 1705 Acked 2704 Read 6 Done Done New false
4580 1706 Acked 2705 Read 3 Done Done New false
4581 1707 Acked 2706 Read 6 Done Done New false
4582 1708 Acked 2707 Read 5 Done Done New false
4583 1709 Acked 2708 Write 3 Done Done New false
4584 1710 Acked 2709 Write 10 Done Done New false
4585 1711 Acked 2710 Flush 0 Done Done New false
4586 1712 Acked 2711 Read 2 Done Done New false
4587 1713 Acked 2712 Read 8 Done Done New false
4588 1714 Acked 2713 Write 6 Done Done New false
4589 1715 Acked 2714 Write 7 Done Done New false
4590 1716 Acked 2715 Read 8 Done Done New false
4591 1717 Acked 2716 Read 4 Done Done New false
4592 1718 Acked 2717 Write 7 Done Done New false
4593 1719 Acked 2718 Read 6 Done Done New false
4594 1720 Acked 2719 Write 4 Done Done New false
4595 1721 Acked 2720 Write 1 Done Done New false
4596 1722 Acked 2721 Read 5 Done Done New false
4597 1723 Acked 2722 Read 4 Done Done New false
4598 1724 Acked 2723 Flush 0 Done Done New false
4599 1725 Acked 2724 Flush 0 Done Done New false
4600 1726 Acked 2725 Write 8 Done Done New false
4601 1727 Acked 2726 Read 9 Done Done New false
4602 1728 Acked 2727 Write 4 Done Done New false
4603 1729 Acked 2728 Read 1 Done Done New false
4604 1730 Acked 2729 Write 4 Done Done New false
4605 1731 Acked 2730 Read 8 Done Done New false
4606 1732 Acked 2731 Flush 0 Done Done New false
4607 1733 Acked 2732 Write 10 Done Done New false
4608 1734 Acked 2733 Write 9 Done Done New false
4609 1735 Acked 2734 Write 3 Done Done New false
4610 1736 Acked 2735 Write 9 Done Done New false
4611 1737 Acked 2736 Write 1 Done Done New false
4612 1738 Acked 2737 Write 7 Done Done New false
4613 1739 Acked 2738 Read 10 Done Done New false
4614 1740 Acked 2739 Read 4 Done Done New false
4615 1741 Acked 2740 Read 2 Done Done New false
4616 1742 Acked 2741 Read 3 Done Done New false
4617 1743 Acked 2742 Flush 0 Done Done New false
4618 1744 Acked 2743 Read 3 Done Done New false
4619 1745 Acked 2744 Write 2 Done Done New false
4620 1746 Acked 2745 Read 7 Done Done New false
4621 1747 Acked 2746 Read 8 Done Done New false
4622 1748 Acked 2747 Write 10 Done Done New false
4623 1749 Acked 2748 Write 3 Done Done New false
4624 1750 Acked 2749 Read 9 Done Done New false
4625 1751 Acked 2750 Write 8 Done Done New false
4626 1752 Acked 2751 Write 3 Done Done New false
4627 1753 Acked 2752 Write 3 Done Done New false
4628 1754 Acked 2753 Flush 0 Done Done New false
4629 1755 Acked 2754 Write 8 Done Done New false
4630 1756 Acked 2755 Write 8 Done Done New false
4631 1757 Acked 2756 Read 8 Done Done New false
4632 1758 Acked 2757 Read 4 Done Done New false
4633 1759 Acked 2758 Write 2 Done Done New false
4634 1760 Acked 2759 Flush 0 Done Done New false
4635 1761 Acked 2760 Write 8 Done Done New false
4636 1762 Acked 2761 Read 1 Done Done New false
4637 1763 Acked 2762 Write 2 Done Done New false
4638 1764 Acked 2763 Read 5 Done Done New false
4639 1765 Acked 2764 Write 6 Done Done New false
4640 1766 Acked 2765 Write 10 Done Done New false
4641 1767 Acked 2766 Write 3 Done Done New false
4642 1768 Acked 2767 Flush 0 Done Done New false
4643 1769 Acked 2768 Write 2 Done Done New false
4644 1770 Acked 2769 Write 2 Done Done New false
4645 1771 Acked 2770 Write 1 Done Done New false
4646 1772 Acked 2771 Write 4 Done Done New false
4647 1773 Acked 2772 Read 5 Done Done New false
4648 1774 Acked 2773 Read 2 Done Done New false
4649 1775 Acked 2774 Read 6 Done Done New false
4650 1776 Acked 2775 Read 2 Done Done New false
4651 1777 Acked 2776 Read 4 Done Done New false
4652 1778 Acked 2777 Read 10 Done Done New false
4653 1779 Acked 2778 Read 9 Done Done New false
4654 1780 Acked 2779 Read 5 Done Done New false
4655 1781 Acked 2780 Read 7 Done Done New false
4656 1782 Acked 2781 Write 9 Done Done New false
4657 1783 Acked 2782 Read 1 Done Done New false
4658 1784 Acked 2783 Read 2 Done Done New false
4659 1785 Acked 2784 Read 6 Done Done New false
4660 1786 Acked 2785 Read 4 Done Done New false
4661 1787 Acked 2786 Read 9 Done Done New false
4662 1788 Acked 2787 Flush 0 Done Done New false
4663 1789 Acked 2788 Write 8 Done Done New false
4664 1790 Acked 2789 Write 4 Done Done New false
4665 1791 Acked 2790 Write 5 Done Done New false
4666 1792 Acked 2791 Read 8 Done Done New false
4667 1793 Acked 2792 Read 4 Done Done New false
4668 1794 Acked 2793 Write 1 Done Done New false
4669 1795 Acked 2794 Write 4 Done Done New false
4670 1796 Acked 2795 Read 10 Done Done New false
4671 1797 Acked 2796 Read 6 Done Done New false
4672 1798 Acked 2797 Read 5 Done Done New false
4673 1799 Acked 2798 Read 5 Done Done New false
4674 1800 Acked 2799 Write 4 Done Done New false
4675 1801 Acked 2800 Write 4 Done Done New false
4676 1802 Acked 2801 Read 9 Done Done New false
4677 1803 Acked 2802 Read 2 Done Done New false
4678 1804 Acked 2803 Read 10 Done Done New false
4679 1805 Acked 2804 Read 10 Done Done New false
4680 1806 Acked 2805 Read 10 Done Done New false
4681 1807 Acked 2806 Read 10 Done Done New false
4682 STATES DS:0 DS:1 DS:2 TOTAL
4683 New 0 0 300 300
4684 Sent 0 0 0 0
4685 Done 300 300 0 600
4686 Skipped 0 0 0 0
4687 Error 0 0 0 0
4688 Last Flush: 2787 2787 2506
4689 Downstairs last five completed: 2506 2505 2504 2503 2502
4690 Upstairs last five completed: 1807 1806 1805 1804 1803
4691 CLIENT: Up:0 ds:300 act:2
46922024-02-28T22:14:06.328ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
46932024-02-28T22:14:06.328ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
46942024-02-28T22:14:06.329ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
46952024-02-28T22:14:06.329ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
46962024-02-28T22:14:06.329ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
46972024-02-28T22:14:06.329ZINFOcrucible: [2] client re-new 301 jobs since flush 2506 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
46982024-02-28T22:14:06.329ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
4699 [8830] got start action
4700 [8830] I am going to restart
4701 Starting downstairs at port 8830
4702 Make output file at "/tmp/dsc/downstairs-8830.txt"
4703 [8830][2] reports Starting
47042024-02-28T22:14:07.330ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47052024-02-28T22:14:07.330ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47062024-02-28T22:14:07.330ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47072024-02-28T22:14:07.330ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47082024-02-28T22:14:07.330ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47092024-02-28T22:14:07.330ZINFOcrucible: [2] client re-new 301 jobs since flush 2506 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47102024-02-28T22:14:07.330ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
4711 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1170)
4712 [8830][2] reports Running
47132024-02-28T22:14:08.331ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47142024-02-28T22:14:08.331ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47152024-02-28T22:14:08.332ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47162024-02-28T22:14:08.332ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47172024-02-28T22:14:08.332ZINFOcrucible: send last flush ID to this DS: 2506 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47182024-02-28T22:14:08.333ZINFOcrucible: Replied this last flush ID: 2506 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47192024-02-28T22:14:08.333ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47202024-02-28T22:14:08.333ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47212024-02-28T22:14:08.333ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47222024-02-28T22:14:08.333ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
4723 ----------------------------------------------------------------
4724 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
4725 Downstairs last five completed: 2807 2806 2805 2804 2803
4726 Upstairs last five completed: 1808 1807 1806 1805 1804
4727 CLIENT: Up:0 ds:0 act:3
4728 Replay: All jobs finished, all DS active.
4729 Main task has work to do, go find it
4730 got dsc Stop(1)
4731 stop 1
4732 Feb 28 22:14:09.868 INFO request completed, latency_us: 325, response_code:[8820] Got stop action so:false kr:false
4733 204, uri: /stop/cid/1, method: POST, req_id: c46cf4a4-0758-4cda-97c0-b1cf260424a8, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
47342024-02-28T22:14:09.868ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47352024-02-28T22:14:09.868ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47362024-02-28T22:14:09.869ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47372024-02-28T22:14:09.869ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
4738 Feb 28 22:14:09.869 INFO request completed, latency_us{"msg":"[1] client re-new 0 jobs since flush 2807","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:09.86910135Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
4739 : 128, response_code: 200, uri: /state/cid/1, method: GET, req_id: 04ce02eb-2ba9-45d3-9197-8e7ac0616c5e, remote_addr:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:14:09.869139319Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
4740 127.0.0.1:36617, local_addr: 127.0.0.1:9998
4741 [8820] Exited with: ExitStatus(unix_wait_status(9))
4742 [8820][1] reports Exit
47432024-02-28T22:14:10.870ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47442024-02-28T22:14:10.870ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47452024-02-28T22:14:10.870ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47462024-02-28T22:14:10.870ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47472024-02-28T22:14:10.870ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47482024-02-28T22:14:10.870ZINFOcrucible: [1] client re-new 0 jobs since flush 2807 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47492024-02-28T22:14:10.870ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47502024-02-28T22:14:11.871ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47512024-02-28T22:14:11.871ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47522024-02-28T22:14:11.871ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47532024-02-28T22:14:11.871ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47542024-02-28T22:14:11.871ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47552024-02-28T22:14:11.871ZINFOcrucible: [1] client re-new 0 jobs since flush 2807 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47562024-02-28T22:14:11.871ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47572024-02-28T22:14:12.872ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47582024-02-28T22:14:12.872ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47592024-02-28T22:14:12.872ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47602024-02-28T22:14:12.872ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47612024-02-28T22:14:12.872ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47622024-02-28T22:14:12.872ZINFOcrucible: [1] client re-new 0 jobs since flush 2807 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47632024-02-28T22:14:12.872ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
4764 Feb 28 22:14:13.870 INFO request completed, latency_us: 223, response_code: 200, uri: /state/cid/1, method: GET, req_id: b7885371-cabd-48e7-9652-1d088d31abfe, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
4765 001/300 Write block 4910 len 28672 data: 3 3 4 4 3 3 3
4766 002/300 Write block 1103 len 32768 data: 4 5 5 5 4 3 3 3
4767 003/300 Write block 1830 len 28672 data: 3 3 2 2 2 2 3
47682024-02-28T22:14:13.873ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47692024-02-28T22:14:13.873ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47702024-02-28T22:14:13.873ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47712024-02-28T22:14:13.873ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47722024-02-28T22:14:13.873ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
47732024-02-28T22:14:13.873ZINFOcrucible: [1] client re-new 2 jobs since flush 2807 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
4774 {"msg":"client task is exiting","v":0,"name":"crucible","level":30004/300 Flush
4775 ,"time":"2024-02-28T22:14:13.873557341Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
4776 005/300 Read block 243 len 32768
4777 006/300 Read block 688 len 24576
4778 007/300 Read block 511 len 16384
4779 008/300 Write block 3382 len 8192 data: 3 3
4780 009/300 Flush
4781 010/300 Flush
4782 011/300 Read block 2508 len 24576
4783 012/300 Read block 3667 len 32768
4784 013/300 Read block 2198 len 24576
4785 014/300 Read block 3618 len 16384
4786 015/300 Flush
4787 016/300 Write block 4301 len 32768 data: 3 3 3 6 4 4 5 5
4788 017/300 Read block 1787 len 8192
4789 018/300 Read block 3167 len 28672
4790 019/300 Flush
4791 020/300 Flush
4792 021/300 Read block 1094 len 36864
4793 022/300 Write block 2692 len 28672 data: 3 2 2 2 2 2 2
4794 023/300 Read block 1745 len 40960
4795 024/300 Read block 536 len 8192
4796 025/300 Read block 4200 len 16384
4797 026/300 Write block 4459 len 16384 data: 3 3 3 3
4798 027/300 Write block 4882 len 32768 data: 2 3 3 3 3 3 3 3
4799 028/300 Write block 3857 len 20480 data: 2 2 2 2 2
4800 029/300 Read block 4550 len 16384
4801 030/300 Read block 4444 len 4096
4802 031/300 Flush
4803 032/300 Write block 4565 len 36864 data: 3 2 2 2 2 2 2 2 2
4804 033/300 Read block 4310 len 32768
4805 034/300 Read block 321 len 32768
4806 035/300 Read block 51 len 32768
4807 036/300 Read block 3102 len 8192
4808 037/300 Flush
4809 038/300 Read block 4873 len 4096
4810 039/300 Read block 4631 len 40960
4811 040/300 Write block 3758 len 24576 data: 2 2 2 2 2 2
4812 041/300 Read block 295 len 4096
4813 042/300 Flush
4814 043/300 Read block 575 len 8192
4815 044/300 Read block 4386 len 4096
4816 045/300 Read block 1305 len 36864
4817 046/300 Read block 2298 len 28672
4818 047/300 Write block 2889 len 20480 data: 5 6 5 4 3
4819 048/300 Read block 4350 len 28672
4820 049/300 Read block 4670 len 40960
4821 050/300 Write block 4401 len 28672 data: 2 2 2 2 2 2 2
4822 051/300 Write block 3820 len 36864 data: 2 2 2 2 2 2 2 2 2
4823 052/300 Write block 4798 len 28672 data: 3 3 3 4 4 4 4
4824 053/300 Read block 4030 len 16384
4825 054/300 Read block 824 len 8192
4826 055/300 Write block 3867 len 24576 data: 2 2 3 3 4 4
4827 056/300 Write block 1228 len 40960 data: 4 4 4 4 3 3 3 3 3 3
4828 057/300 Write block 2029 len 4096 data: 4
4829 058/300 Write block 2318 len 12288 data: 2 2 2
4830 059/300 Read block 2846 len 40960
4831 060/300 Write block 1686 len 32768 data: 3 3 3 3 3 3 3 3
4832 061/300 Read block 529 len 16384
4833 062/300 Read block 2585 len 24576
4834 063/300 Read block 901 len 20480
4835 064/300 Read block 583 len 32768
4836 065/300 Read block 4790 len 36864
4837 066/300 Flush
4838 067/300 Write block 2242 len 36864 data: 3 3 4 4 4 3 3 2 2
4839 068/300 Write block 1081 len 12288 data: 3 3 3
4840 069/300 Read block 2443 len 12288
4841 070/300 Write block 1350 len 16384 data: 3 3 2 2
4842 071/300 Write block 1164 len 36864 data: 2 2 2 2 2 3 3 3 3
4843 072/300 Read block 3914 len 36864
4844 073/300 Flush
4845 074/300 Write block 512 len 36864 data: 3 3 3 3 4 2 2 3 3
4846 075/300 Read block 2710 len 16384
4847 076/300 Flush
4848 077/300 Flush
4849 078/300 Write block 2283 len 32768 data: 2 2 2 2 2 2 2 2
4850 079/300 Read block 4352 len 36864
4851 080/300 Write block 4070 len 20480 data: 4 4 3 3 2
4852 081/300 Write block 1392 len 36864 data: 3 3 3 2 2 2 2 2 2
4853 082/300 Write block 2727 len 16384 data: 4 4 3 3
4854 083/300 Read block 4800 len 28672
4855 084/300 Read block 307 len 4096
4856 085/300 Write block 271 len 36864 data: 3 2 2 2 2 3 3 3 3
4857 086/300 Write block 827 len 40960 data: 4 2 2 2 2 2 2 2 2 2
4858 087/300 Read block 138 len 16384
4859 088/300 Write block 670 len 4096 data: 4
4860 089/300 Write block 3592 len 36864 data: 2 2 2 3 2 2 2 2 2
4861 090/300 Write block 2337 len 16384 data: 4 4 4 3
4862 091/300 Read block 4929 len 8192
4863 092/300 Read block 2483 len 12288
4864 093/300 Write block 2011 len 24576 data: 3 3 3 3 3 2
4865 094/300 Read block 1985 len 36864
4866 095/300 Write block 1128 len 28672 data: 6 4 4 3 2 4 4
4867 096/300 Read block 2416 len 8192
4868 097/300 Read block 1325 len 12288
4869 098/300 Write block 241 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4870 099/300 Read block 2235 len 32768
4871 100/300 Read block 2531 len 32768
4872 101/300 Write block 2782 len 20480 data: 2 3 3 3 3
4873 102/300 Read block 126 len 20480
4874 103/300 Read block 1903 len 28672
4875 104/300 Read block 3432 len 12288
4876 105/300 Flush
4877 106/300 Write block 1124 len 32768 data: 3 5 5 5 7 5 5 4
4878 107/300 Read block 389 len 8192
4879 108/300 Flush
4880 109/300 Read block 3144 len 32768
4881 110/300 Write block 4335 len 8192 data: 3 3
4882 111/300 Flush
4883 112/300 Read block 1981 len 4096
4884 113/300 Write block 3408 len 24576 data: 4 4 3 3 2 2
4885 114/300 Read block 1349 len 24576
4886 115/300 Flush
4887 116/300 Write block 2128 len 4096 data: 4
4888 117/300 Read block 664 len 28672
4889 118/300 Write block 3648 len 12288 data: 3 3 3
4890 119/300 Read block 210 len 12288
4891 120/300 Read block 3855 len 16384
4892 121/300 Flush
4893 122/300 Read block 670 len 36864
4894 123/300 Read block 163 len 36864
4895 124/300 Read block 3285 len 24576
4896 125/300 Write block 1684 len 28672 data: 3 3 4 4 4 4 4
4897 126/300 Read block 1828 len 28672
4898 127/300 Read block 3614 len 32768
4899 128/300 Read block 1026 len 40960
4900 129/300 Read block 3444 len 16384
4901 130/300 Flush
4902 131/300 Read block 2375 len 8192
4903 132/300 Read block 4313 len 8192
4904 133/300 Read block 2684 len 36864
4905 134/300 Read block 4711 len 20480
4906 135/300 Write block 4664 len 24576 data: 2 2 2 2 2 2
4907 136/300 Read block 2872 len 24576
4908 137/300 Read block 304 len 4096
4909 138/300 Write block 3752 len 24576 data: 3 3 3 3 3 2
4910 139/300 Read block 1807 len 12288
4911 140/300 Read block 1275 len 8192
4912 141/300 Read block 3300 len 36864
4913 142/300 Read block 3820 len 20480
4914 143/300 Write block 521 len 16384 data: 3 2 2 2
4915 144/300 Write block 800 len 28672 data: 2 2 2 3 3 3 3
4916 145/300 Read block 3951 len 20480
4917 146/300 Write block 2872 len 4096 data: 3
4918 147/300 Read block 4149 len 32768
4919 148/300 Read block 2279 len 16384
4920 149/300 Write block 342 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4921 150/300 Read block 1706 len 32768
4922 151/300 Write block 4943 len 36864 data: 4 3 4 3 2 2 2 2 2
4923 152/300 Read block 3550 len 40960
4924 153/300 Write block 3175 len 32768 data: 2 2 2 3 3 3 2 2
4925 154/300 Write block 2933 len 32768 data: 3 3 3 2 2 2 3 3
4926 155/300 Read block 363 len 36864
4927 156/300 Read block 3608 len 36864
4928 157/300 Write block 4710 len 28672 data: 3 3 4 4 4 4 3
4929 158/300 Read block 690 len 36864
4930 159/300 Write block 425 len 12288 data: 4 3 2
4931 160/300 Read block 3602 len 40960
4932 161/300 Write block 3751 len 32768 data: 3 4 4 4 4 4 3 3
4933 162/300 Write block 2093 len 28672 data: 3 3 4 4 4 4 5
4934 163/300 Read block 2486 len 20480
4935 164/300 Flush
4936 165/300 Read block 1902 len 32768
4937 166/300 Read block 1186 len 40960
4938 167/300 Write block 338 len 8192 data: 2 2
4939 168/300 Write block 3201 len 12288 data: 2 2 3
4940 169/300 Read block 2469 len 36864
4941 170/300 Read block 195 len 12288
4942 171/300 Write block 442 len 40960 data: 2 2 2 2 3 3 3 3 3 3
4943 172/300 Read block 886 len 24576
4944 173/300 Read block 2963 len 4096
4945 174/300 Write block 1423 len 28672 data: 3 3 3 2 2 2 2
4946 175/300 Read block 1350 len 12288
4947 176/300 Read block 2732 len 40960
4948 177/300 Read block 2553 len 20480
4949 178/300 Read block 1357 len 8192
4950 179/300 Write block 3291 len 16384 data: 3 3 3 3
4951 180/300 Write block 4829 len 36864 data: 3 3 3 3 3 3 3 3 2
4952 181/300 Read block 1913 len 36864
4953 182/300 Flush
4954 183/300 Read block 2261 len 16384
4955 184/300 Write block 1960 len 24576 data: 3 3 3 2 2 2
4956 185/300 Flush
4957 186/300 Read block 4018 len 20480
4958 187/300 Read block 3664 len 20480
4959 188/300 Read block 324 len 16384
4960 189/300 Write block 23 len 12288 data: 2 3 3
4961 190/300 Read block 407 len 36864
4962 191/300 Write block 1202 len 4096 data: 2
4963 192/300 Write block 3035 len 40960 data: 2 2 2 2 2 3 3 3 3 3
4964 193/300 Read block 33 len 20480
4965 194/300 Read block 51 len 20480
4966 195/300 Flush
4967 196/300 Read block 3753 len 40960
4968 197/300 Write block 692 len 8192 data: 2 2
4969 198/300 Write block 1127 len 36864 data: 6 8 6 6 5 3 5 5 4
4970 199/300 Read block 4380 len 16384
4971 200/300 Write block 2815 len 36864 data: 4 4 4 4 4 5 4 6 5
4972 201/300 Read block 4472 len 36864
4973 202/300 Read block 2703 len 4096
4974 203/300 Write block 4691 len 24576 data: 2 2 2 2 2 2
4975 204/300 Read block 1801 len 4096
4976 205/300 Read block 2559 len 12288
4977 206/300 Read block 1622 len 8192
4978 207/300 Write block 1973 len 28672 data: 4 3 3 3 3 3 3
4979 208/300 Write block 1663 len 20480 data: 4 2 2 2 2
4980 209/300 Write block 1433 len 20480 data: 3 3 3 3 2
4981 210/300 Read block 3752 len 16384
4982 211/300 Read block 2137 len 16384
4983 212/300 Read block 960 len 16384
4984 213/300 Write block 3110 len 16384 data: 2 3 3 2
4985 214/300 Read block 4948 len 20480
4986 215/300 Write block 312 len 24576 data: 2 2 2 2 2 3
4987 216/300 Read block 210 len 36864
4988 217/300 Read block 4560 len 20480
4989 218/300 Read block 774 len 36864
4990 219/300 Write block 4550 len 4096 data: 3
4991 220/300 Read block 756 len 40960
4992 221/300 Read block 3387 len 32768
4993 222/300 Write block 1184 len 16384 data: 2 2 2 2
4994 223/300 Read block 761 len 36864
4995 224/300 Read block 837 len 24576
4996 225/300 Write block 3542 len 24576 data: 3 4 4 4 4 4
4997 226/300 Write block 4656 len 32768 data: 3 2 2 2 2 2 2 2
4998 227/300 Write block 945 len 12288 data: 2 2 2
4999 228/300 Write block 930 len 40960 data: 3 3 3 3 3 3 2 2 2 3
5000 229/300 Read block 2130 len 40960
5001 230/300 Read block 4885 len 20480
5002 231/300 Write block 4224 len 36864 data: 3 4 4 4 4 3 3 2 2
5003 232/300 Read block 3580 len 40960
5004 233/300 Write block 4794 len 4096 data: 2
5005 234/300 Read block 943 len 36864
5006 235/300 Write block 4008 len 28672 data: 2 3 3 3 4 5 6
5007 236/300 Read block 3518 len 36864
5008 237/300 Write block 1345 len 40960 data: 2 2 2 2 4 4 4 3 3 2
5009 238/300 Read block 4843 len 36864
5010 239/300 Write block 2772 len 36864 data: 2 2 2 2 2 2 2 2 2
5011 240/300 Read block 873 len 28672
5012 241/300 Read block 1219 len 12288
5013 242/300 Flush
5014 243/300 Read block 2233 len 12288
5015 244/300 Flush
5016 245/300 Read block 1966 len 20480
5017 246/300 Read block 2028 len 40960
5018 247/300 Write block 3108 len 12288 data: 2 2 3
5019 248/300 Read block 1519 len 4096
5020 249/300 Read block 2674 len 4096
5021 250/300 Read block 1890 len 16384
5022 251/300 Read block 4776 len 32768
5023 252/300 Read block 826 len 36864
5024 253/300 Write block 1610 len 24576 data: 2 2 2 2 2 3
5025 254/300 Flush
5026 255/300 Write block 1750 len 28672 data: 3 3 3 3 3 3 3
5027 256/300 Read block 634 len 20480
5028 257/300 Read block 3586 len 16384
5029 258/300 Write block 167 len 12288 data: 2 3 3
5030 259/300 Read block 4589 len 8192
5031 260/300 Read block 1338 len 4096
5032 261/300 Read block 109 len 24576
5033 262/300 Read block 1918 len 28672
5034 263/300 Read block 242 len 28672
5035 264/300 Write block 3674 len 8192 data: 2 2
5036 265/300 Read block 3928 len 20480
5037 266/300 Read block 3109 len 4096
5038 267/300 Read block 667 len 20480
5039 268/300 Read block 78 len 28672
5040 269/300 Write block 1876 len 16384 data: 2 2 4 4
5041 270/300 Read block 2407 len 40960
5042 271/300 Read block 3650 len 28672
5043 272/300 Write block 3247 len 40960 data: 2 2 3 3 3 3 3 4 2 2
5044 273/300 Flush
5045 274/300 Read block 4761 len 8192
5046 275/300 Read block 4133 len 32768
5047 276/300 Read block 4556 len 12288
5048 277/300 Write block 1576 len 32768 data: 3 2 3 3 5 5 5 4
5049 278/300 Write block 976 len 28672 data: 2 3 4 5 4 4 4
5050 279/300 Flush
5051 280/300 Read block 354 len 24576
5052 281/300 Write block 2458 len 36864 data: 3 3 2 2 3 3 2 2 3
5053 282/300 Read block 1612 len 12288
5054 283/300 Read block 832 len 20480
5055 284/300 Write block 1688 len 12288 data: 5 5 5
5056 285/300 Read block 730 len 32768
5057 286/300 Write block 4424 len 16384 data: 2 2 2 2
5058 287/300 Read block 47 len 40960
5059 288/300 Write block 293 len 16384 data: 2 4 4 4
5060 289/300 Read block 1675 len 12288
5061 290/300 Flush
5062 291/300 Read block 2312 len 16384
5063 292/300 Read block 284 len 40960
5064 293/300 Read block 258 len 20480
5065 294/300 Flush
5066 295/300 Write block 531 len 8192 data: 2 3
5067 296/300 Write block 4399 len 28672 data: 2 2 3 3 3 3 3
5068 297/300 Write block 4335 len 4096 data: 4
5069 298/300 Flush
5070 299/300 Write block 1467 len 28672 data: 3 3 4 4 4 4 5
5071 300/300 Write block 4299 len 40960 data: 3 4 4 4 4 7 5 5 6 6
5072 Main task has work to do, go find it
5073 got dsc Start(1)
5074 start 1
5075 Feb 28 22:14:14.511 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/1, method: POST, req_id: 46e6d449-4019-4aec-a6c3-0c053b485093, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
5076 Replay: started 1, returned:Ok(())
5077 ----------------------------------------------------------------
5078 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
5079 States: Active Offline Active
5080 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
5081 1809 Acked 2808 Write 7 Done New Done true
5082 1810 Acked 2809 Write 8 Done New Done true
5083 1811 Acked 2810 Write 7 Done New Done false
5084 1812 Acked 2811 Flush 0 Done New Done false
5085 1813 Acked 2812 Read 8 Done New Done false
5086 1814 Acked 2813 Read 6 Done New Done false
5087 1815 Acked 2814 Read 4 Done New Done false
5088 1816 Acked 2815 Write 2 Done New Done false
5089 1817 Acked 2816 Flush 0 Done New Done false
5090 1818 Acked 2817 Flush 0 Done New Done false
5091 1819 Acked 2818 Read 6 Done New Done false
5092 1820 Acked 2819 Read 8 Done New Done false
5093 1821 Acked 2820 Read 6 Done New Done false
5094 1822 Acked 2821 Read 4 Done New Done false
5095 1823 Acked 2822 Flush 0 Done New Done false
5096 1824 Acked 2823 Write 8 Done New Done false
5097 1825 Acked 2824 Read 2 Done New Done false
5098 1826 Acked 2825 Read 7 Done New Done false
5099 1827 Acked 2826 Flush 0 Done New Done false
5100 1828 Acked 2827 Flush 0 Done New Done false
5101 1829 Acked 2828 Read 9 Done New Done false
5102 1830 Acked 2829 Write 7 Done New Done false
5103 1831 Acked 2830 Read 10 Done New Done false
5104 1832 Acked 2831 Read 2 Done New Done false
5105 1833 Acked 2832 Read 4 Done New Done false
5106 1834 Acked 2833 Write 4 Done New Done false
5107 1835 Acked 2834 Write 8 Done New Done false
5108 1836 Acked 2835 Write 5 Done New Done false
5109 1837 Acked 2836 Read 4 Done New Done false
5110 1838 Acked 2837 Read 1 Done New Done false
5111 1839 Acked 2838 Flush 0 Done New Done false
5112 1840 Acked 2839 Write 9 Done New Done false
5113 1841 Acked 2840 Read 8 Done New Done false
5114 1842 Acked 2841 Read 8 Done New Done false
5115 1843 Acked 2842 Read 8 Done New Done false
5116 1844 Acked 2843 Read 2 Done New Done false
5117 1845 Acked 2844 Flush 0 Done New Done false
5118 1846 Acked 2845 Read 1 Done New Done false
5119 1847 Acked 2846 Read 10 Done New Done false
5120 1848 Acked 2847 Write 6 Done New Done false
5121 1849 Acked 2848 Read 1 Done New Done false
5122 1850 Acked 2849 Flush 0 Done New Done false
5123 1851 Acked 2850 Read 2 Done New Done false
5124 1852 Acked 2851 Read 1 Done New Done false
5125 1853 Acked 2852 Read 9 Done New Done false
5126 1854 Acked 2853 Read 7 Done New Done false
5127 1855 Acked 2854 Write 5 Done New Done false
5128 1856 Acked 2855 Read 7 Done New Done false
5129 1857 Acked 2856 Read 10 Done New Done false
5130 1858 Acked 2857 Write 7 Done New Done false
5131 1859 Acked 2858 Write 9 Done New Done false
5132 1860 Acked 2859 Write 7 Done New Done false
5133 1861 Acked 2860 Read 4 Done New Done false
5134 1862 Acked 2861 Read 2 Done New Done false
5135 1863 Acked 2862 Write 6 Done New Done false
5136 1864 Acked 2863 Write 10 Done New Done false
5137 1865 Acked 2864 Write 1 Done New Done false
5138 1866 Acked 2865 Write 3 Done New Done false
5139 1867 Acked 2866 Read 10 Done New Done false
5140 1868 Acked 2867 Write 8 Done New Done false
5141 1869 Acked 2868 Read 4 Done New Done false
5142 1870 Acked 2869 Read 6 Done New Done false
5143 1871 Acked 2870 Read 5 Done New Done false
5144 1872 Acked 2871 Read 8 Done New Done false
5145 1873 Acked 2872 Read 9 Done New Done false
5146 1874 Acked 2873 Flush 0 Done New Done false
5147 1875 Acked 2874 Write 9 Done New Done false
5148 1876 Acked 2875 Write 3 Done New Done false
5149 1877 Acked 2876 Read 3 Done New Done false
5150 1878 Acked 2877 Write 4 Done New Done false
5151 1879 Acked 2878 Write 9 Done New Done false
5152 1880 Acked 2879 Read 9 Done New Done false
5153 1881 Acked 2880 Flush 0 Done New Done false
5154 1882 Acked 2881 Write 9 Done New Done false
5155 1883 Acked 2882 Read 4 Done New Done false
5156 1884 Acked 2883 Flush 0 Done New Done false
5157 1885 Acked 2884 Flush 0 Done New Done false
5158 1886 Acked 2885 Write 8 Done New Done false
5159 1887 Acked 2886 Read 9 Done New Done false
5160 1888 Acked 2887 Write 5 Done New Done false
5161 1889 Acked 2888 Write 9 Done New Done false
5162 1890 Acked 2889 Write 4 Done New Done false
5163 1891 Acked 2890 Read 7 Done New Done false
5164 1892 Acked 2891 Read 1 Done New Done false
5165 1893 Acked 2892 Write 9 Done New Done false
5166 1894 Acked 2893 Write 10 Done New Done false
5167 1895 Acked 2894 Read 4 Done New Done false
5168 1896 Acked 2895 Write 1 Done New Done false
5169 1897 Acked 2896 Write 9 Done New Done false
5170 1898 Acked 2897 Write 4 Done New Done false
5171 1899 Acked 2898 Read 2 Done New Done false
5172 1900 Acked 2899 Read 3 Done New Done false
5173 1901 Acked 2900 Write 6 Done New Done false
5174 1902 Acked 2901 Read 9 Done New Done false
5175 1903 Acked 2902 Write 7 Done New Done false
5176 1904 Acked 2903 Read 2 Done New Done false
5177 1905 Acked 2904 Read 3 Done New Done false
5178 1906 Acked 2905 Write 10 Done New Done false
5179 1907 Acked 2906 Read 8 Done New Done false
5180 1908 Acked 2907 Read 8 Done New Done false
5181 1909 Acked 2908 Write 5 Done New Done false
5182 1910 Acked 2909 Read 5 Done New Done false
5183 1911 Acked 2910 Read 7 Done New Done false
5184 1912 Acked 2911 Read 3 Done New Done false
5185 1913 Acked 2912 Flush 0 Done New Done false
5186 1914 Acked 2913 Write 8 Done New Done false
5187 1915 Acked 2914 Read 2 Done New Done false
5188 1916 Acked 2915 Flush 0 Done New Done false
5189 1917 Acked 2916 Read 8 Done New Done false
5190 1918 Acked 2917 Write 2 Done New Done false
5191 1919 Acked 2918 Flush 0 Done New Done false
5192 1920 Acked 2919 Read 1 Done New Done false
5193 1921 Acked 2920 Write 6 Done New Done false
5194 1922 Acked 2921 Read 6 Done New Done false
5195 1923 Acked 2922 Flush 0 Done New Done false
5196 1924 Acked 2923 Write 1 Done New Done false
5197 1925 Acked 2924 Read 7 Done New Done false
5198 1926 Acked 2925 Write 3 Done New Done false
5199 1927 Acked 2926 Read 3 Done New Done false
5200 1928 Acked 2927 Read 4 Done New Done false
5201 1929 Acked 2928 Flush 0 Done New Done false
5202 1930 Acked 2929 Read 9 Done New Done false
5203 1931 Acked 2930 Read 9 Done New Done false
5204 1932 Acked 2931 Read 6 Done New Done false
5205 1933 Acked 2932 Write 7 Done New Done false
5206 1934 Acked 2933 Read 7 Done New Done false
5207 1935 Acked 2934 Read 8 Done New Done false
5208 1936 Acked 2935 Read 10 Done New Done false
5209 1937 Acked 2936 Read 4 Done New Done false
5210 1938 Acked 2937 Flush 0 Done New Done false
5211 1939 Acked 2938 Read 2 Done New Done false
5212 1940 Acked 2939 Read 2 Done New Done false
5213 1941 Acked 2940 Read 9 Done New Done false
5214 1942 Acked 2941 Read 5 Done New Done false
5215 1943 Acked 2942 Write 6 Done New Done false
5216 1944 Acked 2943 Read 6 Done New Done false
5217 1945 Acked 2944 Read 1 Done New Done false
5218 1946 Acked 2945 Write 6 Done New Done false
5219 1947 Acked 2946 Read 3 Done New Done false
5220 1948 Acked 2947 Read 2 Done New Done false
5221 1949 Acked 2948 Read 9 Done New Done false
5222 1950 Acked 2949 Read 5 Done New Done false
5223 1951 Acked 2950 Write 4 Done New Done false
5224 1952 Acked 2951 Write 7 Done New Done false
5225 1953 Acked 2952 Read 5 Done New Done false
5226 1954 Acked 2953 Write 1 Done New Done false
5227 1955 Acked 2954 Read 8 Done New Done false
5228 1956 Acked 2955 Read 4 Done New Done false
5229 1957 Acked 2956 Write 10 Done New Done false
5230 1958 Acked 2957 Read 8 Done New Done false
5231 1959 Acked 2958 Write 9 Done New Done false
5232 1960 Acked 2959 Read 10 Done New Done false
5233 1961 Acked 2960 Write 8 Done New Done false
5234 1962 Acked 2961 Write 8 Done New Done false
5235 1963 Acked 2962 Read 9 Done New Done false
5236 1964 Acked 2963 Read 9 Done New Done false
5237 1965 Acked 2964 Write 7 Done New Done false
5238 1966 Acked 2965 Read 9 Done New Done false
5239 1967 Acked 2966 Write 3 Done New Done false
5240 1968 Acked 2967 Read 10 Done New Done false
5241 1969 Acked 2968 Write 8 Done New Done false
5242 1970 Acked 2969 Write 7 Done New Done false
5243 1971 Acked 2970 Read 5 Done New Done false
5244 1972 Acked 2971 Flush 0 Done New Done false
5245 1973 Acked 2972 Read 8 Done New Done false
5246 1974 Acked 2973 Read 10 Done New Done false
5247 1975 Acked 2974 Write 2 Done New Done false
5248 1976 Acked 2975 Write 3 Done New Done false
5249 1977 Acked 2976 Read 9 Done New Done false
5250 1978 Acked 2977 Read 3 Done New Done false
5251 1979 Acked 2978 Write 10 Done New Done false
5252 1980 Acked 2979 Read 6 Done New Done false
5253 1981 Acked 2980 Read 1 Done New Done false
5254 1982 Acked 2981 Write 7 Done New Done false
5255 1983 Acked 2982 Read 3 Done New Done false
5256 1984 Acked 2983 Read 10 Done New Done false
5257 1985 Acked 2984 Read 5 Done New Done false
5258 1986 Acked 2985 Read 2 Done New Done false
5259 1987 Acked 2986 Write 4 Done New Done false
5260 1988 Acked 2987 Write 9 Done New Done false
5261 1989 Acked 2988 Read 9 Done New Done false
5262 1990 Acked 2989 Flush 0 Done New Done false
5263 1991 Acked 2990 Read 4 Done New Done false
5264 1992 Acked 2991 Write 6 Done New Done false
5265 1993 Acked 2992 Flush 0 Done New Done false
5266 1994 Acked 2993 Read 5 Done New Done false
5267 1995 Acked 2994 Read 5 Done New Done false
5268 1996 Acked 2995 Read 4 Done New Done false
5269 1997 Acked 2996 Write 3 Done New Done false
5270 1998 Acked 2997 Read 9 Done New Done false
5271 1999 Acked 2998 Write 1 Done New Done false
5272 2000 Acked 2999 Write 10 Done New Done false
5273 2001 Acked 3000 Read 5 Done New Done false
5274 2002 Acked 3001 Read 5 Done New Done false
5275 2003 Acked 3002 Flush 0 Done New Done false
5276 2004 Acked 3003 Read 10 Done New Done false
5277 2005 Acked 3004 Write 2 Done New Done false
5278 2006 Acked 3005 Write 9 Done New Done false
5279 2007 Acked 3006 Read 4 Done New Done false
5280 2008 Acked 3007 Write 9 Done New Done false
5281 2009 Acked 3008 Read 9 Done New Done false
5282 2010 Acked 3009 Read 1 Done New Done false
5283 2011 Acked 3010 Write 6 Done New Done false
5284 2012 Acked 3011 Read 1 Done New Done false
5285 2013 Acked 3012 Read 3 Done New Done false
5286 2014 Acked 3013 Read 2 Done New Done false
5287 2015 Acked 3014 Write 7 Done New Done false
5288 2016 Acked 3015 Write 5 Done New Done false
5289 2017 Acked 3016 Write 5 Done New Done false
5290 2018 Acked 3017 Read 4 Done New Done false
5291 2019 Acked 3018 Read 4 Done New Done false
5292 2020 Acked 3019 Read 4 Done New Done false
5293 2021 Acked 3020 Write 4 Done New Done false
5294 2022 Acked 3021 Read 5 Done New Done false
5295 2023 Acked 3022 Write 6 Done New Done false
5296 2024 Acked 3023 Read 9 Done New Done false
5297 2025 Acked 3024 Read 5 Done New Done false
5298 2026 Acked 3025 Read 9 Done New Done false
5299 2027 Acked 3026 Write 1 Done New Done false
5300 2028 Acked 3027 Read 10 Done New Done false
5301 2029 Acked 3028 Read 8 Done New Done false
5302 2030 Acked 3029 Write 4 Done New Done false
5303 2031 Acked 3030 Read 9 Done New Done false
5304 2032 Acked 3031 Read 6 Done New Done false
5305 2033 Acked 3032 Write 6 Done New Done false
5306 2034 Acked 3033 Write 8 Done New Done false
5307 2035 Acked 3034 Write 3 Done New Done false
5308 2036 Acked 3035 Write 10 Done New Done false
5309 2037 Acked 3036 Read 10 Done New Done false
5310 2038 Acked 3037 Read 5 Done New Done false
5311 2039 Acked 3038 Write 9 Done New Done false
5312 2040 Acked 3039 Read 10 Done New Done false
5313 2041 Acked 3040 Write 1 Done New Done false
5314 2042 Acked 3041 Read 9 Done New Done false
5315 2043 Acked 3042 Write 7 Done New Done false
5316 2044 Acked 3043 Read 9 Done New Done false
5317 2045 Acked 3044 Write 10 Done New Done false
5318 2046 Acked 3045 Read 9 Done New Done false
5319 2047 Acked 3046 Write 9 Done New Done false
5320 2048 Acked 3047 Read 7 Done New Done false
5321 2049 Acked 3048 Read 3 Done New Done false
5322 2050 Acked 3049 Flush 0 Done New Done false
5323 2051 Acked 3050 Read 3 Done New Done false
5324 2052 Acked 3051 Flush 0 Done New Done false
5325 2053 Acked 3052 Read 5 Done New Done false
5326 2054 Acked 3053 Read 10 Done New Done false
5327 2055 Acked 3054 Write 3 Done New Done false
5328 2056 Acked 3055 Read 1 Done New Done false
5329 2057 Acked 3056 Read 1 Done New Done false
5330 2058 Acked 3057 Read 4 Done New Done false
5331 2059 Acked 3058 Read 8 Done New Done false
5332 2060 Acked 3059 Read 9 Done New Done false
5333 2061 Acked 3060 Write 6 Done New Done false
5334 2062 Acked 3061 Flush 0 Done New Done false
5335 2063 Acked 3062 Write 7 Done New Done false
5336 2064 Acked 3063 Read 5 Done New Done false
5337 2065 Acked 3064 Read 4 Done New Done false
5338 2066 Acked 3065 Write 3 Done New Done false
5339 2067 Acked 3066 Read 2 Done New Done false
5340 2068 Acked 3067 Read 1 Done New Done false
5341 2069 Acked 3068 Read 6 Done New Done false
5342 2070 Acked 3069 Read 7 Done New Done false
5343 2071 Acked 3070 Read 7 Done New Done false
5344 2072 Acked 3071 Write 2 Done New Done false
5345 2073 Acked 3072 Read 5 Done New Done false
5346 2074 Acked 3073 Read 1 Done New Done false
5347 2075 Acked 3074 Read 5 Done New Done false
5348 2076 Acked 3075 Read 7 Done New Done false
5349 2077 Acked 3076 Write 4 Done New Done false
5350 2078 Acked 3077 Read 10 Done New Done false
5351 2079 Acked 3078 Read 7 Done New Done false
5352 2080 Acked 3079 Write 10 Done New Done false
5353 2081 Acked 3080 Flush 0 Done New Done false
5354 2082 Acked 3081 Read 2 Done New Done false
5355 2083 Acked 3082 Read 8 Done New Done false
5356 2084 Acked 3083 Read 3 Done New Done false
5357 2085 Acked 3084 Write 8 Done New Done false
5358 2086 Acked 3085 Write 7 Done New Done false
5359 2087 Acked 3086 Flush 0 Done New Done false
5360 2088 Acked 3087 Read 6 Done New Done false
5361 2089 Acked 3088 Write 9 Done New Done false
5362 2090 Acked 3089 Read 3 Done New Done false
5363 2091 Acked 3090 Read 5 Done New Done false
5364 2092 Acked 3091 Write 3 Done New Done false
5365 2093 Acked 3092 Read 8 Done New Done false
5366 2094 Acked 3093 Write 4 Done New Done false
5367 2095 Acked 3094 Read 10 Done New Done false
5368 2096 Acked 3095 Write 4 Done New Done false
5369 2097 Acked 3096 Read 3 Done New Done false
5370 2098 Acked 3097 Flush 0 Done New Done false
5371 2099 Acked 3098 Read 4 Done New Done false
5372 2100 Acked 3099 Read 10 Done New Done false
5373 2101 Acked 3100 Read 5 Done New Done false
5374 2102 Acked 3101 Flush 0 Done New Done false
5375 2103 Acked 3102 Write 2 Done New Done false
5376 2104 Acked 3103 Write 7 Done New Done false
5377 2105 Acked 3104 Write 1 Done New Done false
5378 2106 Acked 3105 Flush 0 Done New Done false
5379 2107 Acked 3106 Write 7 Done New Done false
5380 2108 Acked 3107 Write 10 Done New Done false
5381 STATES DS:0 DS:1 DS:2 TOTAL
5382 New 0 300 0 300
5383 Sent 0 0 0 0
5384 Done 300 0 300 600
5385 Skipped 0 0 0 0
5386 Error 0 0 0 0
5387 Last Flush: 3105 2807 3105
5388 Downstairs last five completed: 2807 2806 2805 2804 2803
5389 Upstairs last five completed: 2108 2107 2106 2105 2104
5390 CLIENT: Up:0 ds:300 act:2
53912024-02-28T22:14:14.874ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
53922024-02-28T22:14:14.874ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
53932024-02-28T22:14:14.874ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
53942024-02-28T22:14:14.874ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
53952024-02-28T22:14:14.874ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
53962024-02-28T22:14:14.874ZINFOcrucible: [1] client re-new 300 jobs since flush 2807 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
53972024-02-28T22:14:14.875ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
53982024-02-28T22:14:15.876ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
53992024-02-28T22:14:15.877ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
54002024-02-28T22:14:15.877ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
54012024-02-28T22:14:15.877ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
54022024-02-28T22:14:15.877ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
54032024-02-28T22:14:15.877ZINFOcrucible: [1] client re-new 301 jobs since flush 2807 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
54042024-02-28T22:14:15.877ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
54052024-02-28T22:14:16.878ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
54062024-02-28T22:14:16.878ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
54072024-02-28T22:14:16.878ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
54082024-02-28T22:14:16.878ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
54092024-02-28T22:14:16.878ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
54102024-02-28T22:14:16.878ZINFOcrucible: [1] client re-new 301 jobs since flush 2807 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
54112024-02-28T22:14:16.878ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
5412 [8820] got start action
5413 [8820] I am going to restart
5414 Starting downstairs at port 8820
5415 Make output file at "/tmp/dsc/downstairs-8820.txt"
5416 [8820][1] reports Starting
54172024-02-28T22:14:17.879ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
54182024-02-28T22:14:17.879ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
54192024-02-28T22:14:17.879ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
54202024-02-28T22:14:17.879ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
54212024-02-28T22:14:17.879ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
54222024-02-28T22:14:17.879ZINFOcrucible: [1] client re-new 301 jobs since flush 2807 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
54232024-02-28T22:14:17.879ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
5424 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1171)
5425 [8820][1] reports Running
5426 ----------------------------------------------------------------
5427 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5428 States: Active Offline Active
5429 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
5430 1809 Acked 2808 Write 7 Done New Done true
5431 1810 Acked 2809 Write 8 Done New Done true
5432 1811 Acked 2810 Write 7 Done New Done true
5433 1812 Acked 2811 Flush 0 Done New Done true
5434 1813 Acked 2812 Read 8 Done New Done true
5435 1814 Acked 2813 Read 6 Done New Done true
5436 1815 Acked 2814 Read 4 Done New Done true
5437 1816 Acked 2815 Write 2 Done New Done true
5438 1817 Acked 2816 Flush 0 Done New Done true
5439 1818 Acked 2817 Flush 0 Done New Done true
5440 1819 Acked 2818 Read 6 Done New Done true
5441 1820 Acked 2819 Read 8 Done New Done true
5442 1821 Acked 2820 Read 6 Done New Done true
5443 1822 Acked 2821 Read 4 Done New Done true
5444 1823 Acked 2822 Flush 0 Done New Done true
5445 1824 Acked 2823 Write 8 Done New Done true
5446 1825 Acked 2824 Read 2 Done New Done true
5447 1826 Acked 2825 Read 7 Done New Done true
5448 1827 Acked 2826 Flush 0 Done New Done true
5449 1828 Acked 2827 Flush 0 Done New Done true
5450 1829 Acked 2828 Read 9 Done New Done true
5451 1830 Acked 2829 Write 7 Done New Done true
5452 1831 Acked 2830 Read 10 Done New Done true
5453 1832 Acked 2831 Read 2 Done New Done true
5454 1833 Acked 2832 Read 4 Done New Done true
5455 1834 Acked 2833 Write 4 Done New Done true
5456 1835 Acked 2834 Write 8 Done New Done true
5457 1836 Acked 2835 Write 5 Done New Done true
5458 1837 Acked 2836 Read 4 Done New Done true
5459 1838 Acked 2837 Read 1 Done New Done true
5460 1839 Acked 2838 Flush 0 Done New Done true
5461 1840 Acked 2839 Write 9 Done New Done true
5462 1841 Acked 2840 Read 8 Done New Done true
5463 1842 Acked 2841 Read 8 Done New Done true
5464 1843 Acked 2842 Read 8 Done New Done true
5465 1844 Acked 2843 Read 2 Done New Done true
5466 1845 Acked 2844 Flush 0 Done New Done true
5467 1846 Acked 2845 Read 1 Done New Done true
5468 1847 Acked 2846 Read 10 Done New Done true
5469 1848 Acked 2847 Write 6 Done New Done true
5470 1849 Acked 2848 Read 1 Done New Done true
5471 1850 Acked 2849 Flush 0 Done New Done true
5472 1851 Acked 2850 Read 2 Done New Done true
5473 1852 Acked 2851 Read 1 Done New Done true
5474 1853 Acked 2852 Read 9 Done New Done true
5475 1854 Acked 2853 Read 7 Done New Done true
5476 1855 Acked 2854 Write 5 Done New Done true
5477 1856 Acked 2855 Read 7 Done New Done true
5478 1857 Acked 2856 Read 10 Done New Done true
5479 1858 Acked 2857 Write 7 Done New Done true
5480 1859 Acked 2858 Write 9 Done New Done true
5481 1860 Acked 2859 Write 7 Done New Done true
5482 1861 Acked 2860 Read 4 Done New Done true
5483 1862 Acked 2861 Read 2 Done New Done true
5484 1863 Acked 2862 Write 6 Done New Done true
5485 1864 Acked 2863 Write 10 Done New Done true
5486 1865 Acked 2864 Write 1 Done New Done true
5487 1866 Acked 2865 Write 3 Done New Done true
5488 1867 Acked 2866 Read 10 Done New Done true
5489 1868 Acked 2867 Write 8 Done New Done true
5490 1869 Acked 2868 Read 4 Done New Done true
5491 1870 Acked 2869 Read 6 Done New Done true
5492 1871 Acked 2870 Read 5 Done New Done true
5493 1872 Acked 2871 Read 8 Done New Done true
5494 1873 Acked 2872 Read 9 Done New Done true
5495 1874 Acked 2873 Flush 0 Done New Done true
5496 1875 Acked 2874 Write 9 Done New Done true
5497 1876 Acked 2875 Write 3 Done New Done true
5498 1877 Acked 2876 Read 3 Done New Done true
5499 1878 Acked 2877 Write 4 Done New Done true
5500 1879 Acked 2878 Write 9 Done New Done true
5501 1880 Acked 2879 Read 9 Done New Done true
5502 1881 Acked 2880 Flush 0 Done New Done true
5503 1882 Acked 2881 Write 9 Done New Done true
5504 1883 Acked 2882 Read 4 Done New Done true
5505 1884 Acked 2883 Flush 0 Done New Done true
5506 1885 Acked 2884 Flush 0 Done New Done true
5507 1886 Acked 2885 Write 8 Done New Done true
5508 1887 Acked 2886 Read 9 Done New Done true
5509 1888 Acked 2887 Write 5 Done New Done true
5510 1889 Acked 2888 Write 9 Done New Done true
5511 1890 Acked 2889 Write 4 Done New Done true
5512 1891 Acked 2890 Read 7 Done New Done true
5513 1892 Acked 2891 Read 1 Done New Done true
5514 1893 Acked 2892 Write 9 Done New Done true
5515 1894 Acked 2893 Write 10 Done New Done true
5516 1895 Acked 2894 Read 4 Done New Done true
5517 1896 Acked 2895 Write 1 Done New Done true
5518 1897 Acked 2896 Write 9 Done New Done true
5519 1898 Acked 2897 Write 4 Done New Done true
5520 1899 Acked 2898 Read 2 Done New Done true
5521 1900 Acked 2899 Read 3 Done New Done true
5522 1901 Acked 2900 Write 6 Done New Done true
5523 1902 Acked 2901 Read 9 Done New Done true
5524 1903 Acked 2902 Write 7 Done New Done true
5525 1904 Acked 2903 Read 2 Done New Done true
5526 1905 Acked 2904 Read 3 Done New Done true
5527 1906 Acked 2905 Write 10 Done New Done true
5528 1907 Acked 2906 Read 8 Done New Done true
5529 1908 Acked 2907 Read 8 Done New Done true
5530 1909 Acked 2908 Write 5 Done New Done true
5531 1910 Acked 2909 Read 5 Done New Done true
5532 1911 Acked 2910 Read 7 Done New Done true
5533 1912 Acked 2911 Read 3 Done New Done true
5534 1913 Acked 2912 Flush 0 Done New Done true
5535 1914 Acked 2913 Write 8 Done New Done true
5536 1915 Acked 2914 Read 2 Done New Done true
5537 1916 Acked 2915 Flush 0 Done New Done true
5538 1917 Acked 2916 Read 8 Done New Done true
5539 1918 Acked 2917 Write 2 Done New Done true
5540 1919 Acked 2918 Flush 0 Done New Done true
5541 1920 Acked 2919 Read 1 Done New Done true
5542 1921 Acked 2920 Write 6 Done New Done true
5543 1922 Acked 2921 Read 6 Done New Done true
5544 1923 Acked 2922 Flush 0 Done New Done true
5545 1924 Acked 2923 Write 1 Done New Done true
5546 1925 Acked 2924 Read 7 Done New Done true
5547 1926 Acked 2925 Write 3 Done New Done true
5548 1927 Acked 2926 Read 3 Done New Done true
5549 1928 Acked 2927 Read 4 Done New Done true
5550 1929 Acked 2928 Flush 0 Done New Done true
5551 1930 Acked 2929 Read 9 Done New Done true
5552 1931 Acked 2930 Read 9 Done New Done true
5553 1932 Acked 2931 Read 6 Done New Done true
5554 1933 Acked 2932 Write 7 Done New Done true
5555 1934 Acked 2933 Read 7 Done New Done true
5556 1935 Acked 2934 Read 8 Done New Done true
5557 1936 Acked 2935 Read 10 Done New Done true
5558 1937 Acked 2936 Read 4 Done New Done true
5559 1938 Acked 2937 Flush 0 Done New Done true
5560 1939 Acked 2938 Read 2 Done New Done true
5561 1940 Acked 2939 Read 2 Done New Done true
5562 1941 Acked 2940 Read 9 Done New Done true
5563 1942 Acked 2941 Read 5 Done New Done true
5564 1943 Acked 2942 Write 6 Done New Done true
5565 1944 Acked 2943 Read 6 Done New Done true
5566 1945 Acked 2944 Read 1 Done New Done true
5567 1946 Acked 2945 Write 6 Done New Done true
5568 1947 Acked 2946 Read 3 Done New Done true
5569 1948 Acked 2947 Read 2 Done New Done true
5570 1949 Acked 2948 Read 9 Done New Done true
5571 1950 Acked 2949 Read 5 Done New Done true
5572 1951 Acked 2950 Write 4 Done New Done true
5573 1952 Acked 2951 Write 7 Done New Done true
5574 1953 Acked 2952 Read 5 Done New Done true
5575 1954 Acked 2953 Write 1 Done New Done true
5576 1955 Acked 2954 Read 8 Done New Done true
5577 1956 Acked 2955 Read 4 Done New Done true
5578 1957 Acked 2956 Write 10 Done New Done true
5579 1958 Acked 2957 Read 8 Done New Done true
5580 1959 Acked 2958 Write 9 Done New Done true
5581 1960 Acked 2959 Read 10 Done New Done true
5582 1961 Acked 2960 Write 8 Done New Done true
5583 1962 Acked 2961 Write 8 Done New Done true
5584 1963 Acked 2962 Read 9 Done New Done true
5585 1964 Acked 2963 Read 9 Done New Done true
5586 1965 Acked 2964 Write 7 Done New Done true
5587 1966 Acked 2965 Read 9 Done New Done true
5588 1967 Acked 2966 Write 3 Done New Done true
5589 1968 Acked 2967 Read 10 Done New Done true
5590 1969 Acked 2968 Write 8 Done New Done true
5591 1970 Acked 2969 Write 7 Done New Done true
5592 1971 Acked 2970 Read 5 Done New Done true
5593 1972 Acked 2971 Flush 0 Done New Done true
5594 1973 Acked 2972 Read 8 Done New Done true
5595 1974 Acked 2973 Read 10 Done New Done true
5596 1975 Acked 2974 Write 2 Done New Done true
5597 1976 Acked 2975 Write 3 Done New Done true
5598 1977 Acked 2976 Read 9 Done New Done true
5599 1978 Acked 2977 Read 3 Done New Done true
5600 1979 Acked 2978 Write 10 Done New Done true
5601 1980 Acked 2979 Read 6 Done New Done true
5602 1981 Acked 2980 Read 1 Done New Done true
5603 1982 Acked 2981 Write 7 Done New Done true
5604 1983 Acked 2982 Read 3 Done New Done true
5605 1984 Acked 2983 Read 10 Done New Done true
5606 1985 Acked 2984 Read 5 Done New Done true
5607 1986 Acked 2985 Read 2 Done New Done true
5608 1987 Acked 2986 Write 4 Done New Done true
5609 1988 Acked 2987 Write 9 Done New Done true
5610 1989 Acked 2988 Read 9 Done New Done true
5611 1990 Acked 2989 Flush 0 Done New Done true
5612 1991 Acked 2990 Read 4 Done New Done true
5613 1992 Acked 2991 Write 6 Done New Done true
5614 1993 Acked 2992 Flush 0 Done New Done true
5615 1994 Acked 2993 Read 5 Done New Done true
5616 1995 Acked 2994 Read 5 Done New Done true
5617 1996 Acked 2995 Read 4 Done New Done true
5618 1997 Acked 2996 Write 3 Done New Done true
5619 1998 Acked 2997 Read 9 Done New Done true
5620 1999 Acked 2998 Write 1 Done New Done true
5621 2000 Acked 2999 Write 10 Done New Done true
5622 2001 Acked 3000 Read 5 Done New Done true
5623 2002 Acked 3001 Read 5 Done New Done true
5624 2003 Acked 3002 Flush 0 Done New Done true
5625 2004 Acked 3003 Read 10 Done New Done true
5626 2005 Acked 3004 Write 2 Done New Done true
5627 2006 Acked 3005 Write 9 Done New Done true
5628 2007 Acked 3006 Read 4 Done New Done true
5629 2008 Acked 3007 Write 9 Done New Done true
5630 2009 Acked 3008 Read 9 Done New Done true
5631 2010 Acked 3009 Read 1 Done New Done true
5632 2011 Acked 3010 Write 6 Done New Done true
5633 2012 Acked 3011 Read 1 Done New Done true
5634 2013 Acked 3012 Read 3 Done New Done true
5635 2014 Acked 3013 Read 2 Done New Done true
5636 2015 Acked 3014 Write 7 Done New Done true
5637 2016 Acked 3015 Write 5 Done New Done true
5638 2017 Acked 3016 Write 5 Done New Done true
5639 2018 Acked 3017 Read 4 Done New Done true
5640 2019 Acked 3018 Read 4 Done New Done true
5641 2020 Acked 3019 Read 4 Done New Done true
5642 2021 Acked 3020 Write 4 Done New Done true
5643 2022 Acked 3021 Read 5 Done New Done true
5644 2023 Acked 3022 Write 6 Done New Done true
5645 2024 Acked 3023 Read 9 Done New Done true
5646 2025 Acked 3024 Read 5 Done New Done true
5647 2026 Acked 3025 Read 9 Done New Done true
5648 2027 Acked 3026 Write 1 Done New Done true
5649 2028 Acked 3027 Read 10 Done New Done true
5650 2029 Acked 3028 Read 8 Done New Done true
5651 2030 Acked 3029 Write 4 Done New Done true
5652 2031 Acked 3030 Read 9 Done New Done true
5653 2032 Acked 3031 Read 6 Done New Done true
5654 2033 Acked 3032 Write 6 Done New Done true
5655 2034 Acked 3033 Write 8 Done New Done true
5656 2035 Acked 3034 Write 3 Done New Done true
5657 2036 Acked 3035 Write 10 Done New Done true
5658 2037 Acked 3036 Read 10 Done New Done true
5659 2038 Acked 3037 Read 5 Done New Done true
5660 2039 Acked 3038 Write 9 Done New Done true
5661 2040 Acked 3039 Read 10 Done New Done true
5662 2041 Acked 3040 Write 1 Done New Done true
5663 2042 Acked 3041 Read 9 Done New Done true
5664 2043 Acked 3042 Write 7 Done New Done true
5665 2044 Acked 3043 Read 9 Done New Done true
5666 2045 Acked 3044 Write 10 Done New Done true
5667 2046 Acked 3045 Read 9 Done New Done true
5668 2047 Acked 3046 Write 9 Done New Done true
5669 2048 Acked 3047 Read 7 Done New Done true
5670 2049 Acked 3048 Read 3 Done New Done true
5671 2050 Acked 3049 Flush 0 Done New Done true
5672 2051 Acked 3050 Read 3 Done New Done true
5673 2052 Acked 3051 Flush 0 Done New Done true
5674 2053 Acked 3052 Read 5 Done New Done true
5675 2054 Acked 3053 Read 10 Done New Done true
5676 2055 Acked 3054 Write 3 Done New Done true
5677 2056 Acked 3055 Read 1 Done New Done true
5678 2057 Acked 3056 Read 1 Done New Done true
5679 2058 Acked 3057 Read 4 Done New Done true
5680 2059 Acked 3058 Read 8 Done New Done true
5681 2060 Acked 3059 Read 9 Done New Done true
5682 2061 Acked 3060 Write 6 Done New Done true
5683 2062 Acked 3061 Flush 0 Done New Done true
5684 2063 Acked 3062 Write 7 Done New Done true
5685 2064 Acked 3063 Read 5 Done New Done true
5686 2065 Acked 3064 Read 4 Done New Done true
5687 2066 Acked 3065 Write 3 Done New Done true
5688 2067 Acked 3066 Read 2 Done New Done true
5689 2068 Acked 3067 Read 1 Done New Done true
5690 2069 Acked 3068 Read 6 Done New Done true
5691 2070 Acked 3069 Read 7 Done New Done true
5692 2071 Acked 3070 Read 7 Done New Done true
5693 2072 Acked 3071 Write 2 Done New Done true
5694 2073 Acked 3072 Read 5 Done New Done true
5695 2074 Acked 3073 Read 1 Done New Done true
5696 2075 Acked 3074 Read 5 Done New Done true
5697 2076 Acked 3075 Read 7 Done New Done true
5698 2077 Acked 3076 Write 4 Done New Done true
5699 2078 Acked 3077 Read 10 Done New Done true
5700 2079 Acked 3078 Read 7 Done New Done true
5701 2080 Acked 3079 Write 10 Done New Done true
5702 2081 Acked 3080 Flush 0 Done New Done true
5703 2082 Acked 3081 Read 2 Done New Done true
5704 2083 Acked 3082 Read 8 Done New Done true
5705 2084 Acked 3083 Read 3 Done New Done true
5706 2085 Acked 3084 Write 8 Done New Done true
5707 2086 Acked 3085 Write 7 Done New Done true
5708 2087 Acked 3086 Flush 0 Done New Done true
5709 2088 Acked 3087 Read 6 Done New Done true
5710 2089 Acked 3088 Write 9 Done New Done true
5711 2090 Acked 3089 Read 3 Done New Done true
5712 2091 Acked 3090 Read 5 Done New Done true
5713 2092 Acked 3091 Write 3 Done New Done true
5714 2093 Acked 3092 Read 8 Done New Done true
5715 2094 Acked 3093 Write 4 Done New Done true
5716 2095 Acked 3094 Read 10 Done New Done true
5717 2096 Acked 3095 Write 4 Done New Done true
5718 2097 Acked 3096 Read 3 Done New Done true
5719 2098 Acked 3097 Flush 0 Done New Done true
5720 2099 Acked 3098 Read 4 Done New Done true
5721 2100 Acked 3099 Read 10 Done New Done true
5722 2101 Acked 3100 Read 5 Done New Done true
5723 2102 Acked 3101 Flush 0 Done New Done true
5724 2103 Acked 3102 Write 2 Done New Done true
5725 2104 Acked 3103 Write 7 Done New Done true
5726 2105 Acked 3104 Write 1 Done New Done true
5727 2106 Acked 3105 Flush 0 Done New Done true
5728 2107 Acked 3106 Write 7 Done New Done true
5729 2108 Acked 3107 Write 10 Done New Done true
5730 2109 Acked 3108 Flush 0 Done New Done true
5731 STATES DS:0 DS:1 DS:2 TOTAL
5732 New 0 301 0 301
5733 Sent 0 0 0 0
5734 Done 301 0 301 602
5735 Skipped 0 0 0 0
5736 Error 0 0 0 0
5737 Last Flush: 3108 2807 3108
5738 Downstairs last five completed: 2807 2806 2805 2804 2803
5739 Upstairs last five completed: 2109 2108 2107 2106 2105
5740 CLIENT: Up:0 ds:301 act:2
57412024-02-28T22:14:18.880ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57422024-02-28T22:14:18.880ZINFOcrucible: ds_connection connected = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57432024-02-28T22:14:18.881ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57442024-02-28T22:14:18.881ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57452024-02-28T22:14:18.881ZINFOcrucible: send last flush ID to this DS: 2807 = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57462024-02-28T22:14:18.882ZINFOcrucible: Replied this last flush ID: 2807 = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57472024-02-28T22:14:18.882ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57482024-02-28T22:14:18.882ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57492024-02-28T22:14:18.882ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57502024-02-28T22:14:18.882ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
5751 ----------------------------------------------------------------
5752 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
5753 Downstairs last five completed: 3108 3107 3106 3105 3104
5754 Upstairs last five completed: 2109 2108 2107 2106 2105
5755 CLIENT: Up:0 ds:0 act:3
5756 Replay: All jobs finished, all DS active.
5757 Main task has work to do, go find it
5758 got dsc Stop(0)
5759 stop 0
5760 Feb 28 22:14:22.517 INFO request completed, latency_us:[8810] Got stop action so:false kr:false
5761 311, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 1e2422f6-a386-442b-8f3b-0c67d09a63c3, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
5762 Feb 28 22:14:22.517 INFO request completed, latency_us: 116, response_code: 200, uri: /state/cid/0, method: GET, req_id: 6e8d7768-cce3-403e-b812-5e5f13bde6af, remote_addr: 127.0.0.1:36617, {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-28T22:14:22.517814293Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
5763 local_addr: 127.0.0.1:9998
57642024-02-28T22:14:22.517ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57652024-02-28T22:14:22.517ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57662024-02-28T22:14:22.517ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57672024-02-28T22:14:22.517ZINFOcrucible: [0] client re-new 0 jobs since flush 3108 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57682024-02-28T22:14:22.518ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
5769 [8810] Exited with: ExitStatus(unix_wait_status(9))
5770 [8810][0] reports Exit
57712024-02-28T22:14:23.519ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57722024-02-28T22:14:23.519ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57732024-02-28T22:14:23.519ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57742024-02-28T22:14:23.519ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57752024-02-28T22:14:23.519ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57762024-02-28T22:14:23.519ZINFOcrucible: [0] client re-new 0 jobs since flush 3108 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57772024-02-28T22:14:23.519ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57782024-02-28T22:14:24.521ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57792024-02-28T22:14:24.521ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57802024-02-28T22:14:24.521ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57812024-02-28T22:14:24.521ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57822024-02-28T22:14:24.521ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57832024-02-28T22:14:24.521ZINFOcrucible: [0] client re-new 0 jobs since flush 3108 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57842024-02-28T22:14:24.521ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57852024-02-28T22:14:25.522ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57862024-02-28T22:14:25.522ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57872024-02-28T22:14:25.522ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57882024-02-28T22:14:25.523ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57892024-02-28T22:14:25.523ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57902024-02-28T22:14:25.523ZINFOcrucible: [0] client re-new 0 jobs since flush 3108 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57912024-02-28T22:14:25.523ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
5792 Feb 28 22:14:26.520 INFO request completed, latency_us: 182, response_code: 200, uri: /state/cid/0, method: GET, req_id: 14e7ac8e-86a9-49ec-9c90-9ea3bd6e948f, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
5793 001/300 Write block 2917 len 36864 data: 3 3 3 4 4 4 3 4 4
5794 002/300 Write block 1450 len 16384 data: 2 2 2 2
5795 003/300 Flush
57962024-02-28T22:14:26.525ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57972024-02-28T22:14:26.525ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57982024-02-28T22:14:26.525ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
57992024-02-28T22:14:26.525ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
58002024-02-28T22:14:26.525ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
58012024-02-28T22:14:26.525ZINFOcrucible: [0] client re-new 3 jobs since flush 3108 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
58022024-02-28T22:14:26.525ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
5803 004/300 Read block 2083 len 32768
5804 005/300 Write block 3212 len 32768 data: 2 2 2 2 2 2 2 2
5805 006/300 Read block 218 len 8192
5806 007/300 Write block 4459 len 36864 data: 4 4 4 4 4 3 3 3 2
5807 008/300 Read block 1789 len 32768
5808 009/300 Flush
5809 010/300 Read block 2827 len 36864
5810 011/300 Read block 4908 len 24576
5811 012/300 Write block 2602 len 40960 data: 4 3 2 2 2 2 2 2 2 2
5812 013/300 Write block 1510 len 12288 data: 2 2 2
5813 014/300 Read block 4109 len 40960
5814 015/300 Read block 2732 len 20480
5815 016/300 Write block 3362 len 20480 data: 3 3 2 2 3
5816 017/300 Flush
5817 018/300 Read block 936 len 12288
5818 019/300 Write block 423 len 8192 data: 4 4
5819 020/300 Read block 4133 len 20480
5820 021/300 Write block 169 len 32768 data: 4 3 4 4 3 3 2 2
5821 022/300 Read block 4050 len 28672
5822 023/300 Flush
5823 024/300 Read block 2666 len 8192
5824 025/300 Flush
5825 026/300 Read block 1666 len 20480
5826 027/300 Write block 2679 len 8192 data: 4 5
5827 028/300 Flush
5828 029/300 Read block 2754 len 8192
5829 030/300 Read block 349 len 24576
5830 031/300 Read block 3924 len 8192
5831 032/300 Write block 3724 len 12288 data: 3 4 4
5832 033/300 Read block 131 len 24576
5833 034/300 Flush
5834 035/300 Write block 2973 len 20480 data: 5 5 6 5 5
5835 036/300 Write block 467 len 16384 data: 4 4 4 4
5836 037/300 Read block 3188 len 12288
5837 038/300 Flush
5838 039/300 Read block 3402 len 16384
5839 040/300 Read block 4061 len 24576
5840 041/300 Read block 844 len 20480
5841 042/300 Read block 2440 len 36864
5842 043/300 Write block 1611 len 4096 data: 3
5843 044/300 Read block 3496 len 36864
5844 045/300 Read block 4066 len 36864
5845 046/300 Read block 91 len 28672
5846 047/300 Read block 2114 len 16384
5847 048/300 Write block 4461 len 12288 data: 5 5 5
5848 049/300 Read block 1090 len 16384
5849 050/300 Read block 3413 len 28672
5850 051/300 Read block 2341 len 16384
5851 052/300 Read block 3135 len 36864
5852 053/300 Write block 2699 len 32768 data: 2 2 2 2 2 2 2 3
5853 054/300 Read block 665 len 24576
5854 055/300 Read block 1499 len 36864
5855 056/300 Read block 3997 len 24576
5856 057/300 Read block 2197 len 28672
5857 058/300 Write block 3772 len 32768 data: 3 2 2 2 2 3 2 2
5858 059/300 Flush
5859 060/300 Write block 2023 len 40960 data: 3 3 3 3 3 3 5 4 3 3
5860 061/300 Write block 1960 len 24576 data: 4 4 4 3 3 3
5861 062/300 Read block 3043 len 28672
5862 063/300 Read block 3590 len 12288
5863 064/300 Read block 109 len 12288
5864 065/300 Write block 3237 len 40960 data: 3 3 3 3 3 2 2 2 2 2
5865 066/300 Write block 3224 len 36864 data: 3 3 3 2 2 3 3 4 3
5866 067/300 Read block 148 len 16384
5867 068/300 Read block 4074 len 32768
5868 069/300 Write block 2230 len 12288 data: 2 2 2
5869 070/300 Write block 4716 len 16384 data: 4 3 3 3
5870 071/300 Read block 1924 len 24576
5871 072/300 Write block 1201 len 8192 data: 2 3
5872 073/300 Write block 2033 len 32768 data: 5 6 6 7 6 3 3 3
5873 074/300 Read block 3381 len 32768
5874 075/300 Read block 718 len 16384
5875 076/300 Write block 3470 len 4096 data: 2
5876 077/300 Write block 3129 len 36864 data: 2 2 2 2 2 2 3 3 3
5877 078/300 Read block 386 len 36864
5878 079/300 Read block 154 len 32768
5879 080/300 Flush
5880 081/300 Write block 2604 len 12288 data: 3 3 3
5881 082/300 Read block 40 len 12288
5882 083/300 Write block 2718 len 32768 data: 3 3 3 3 3 3 3 3
5883 084/300 Read block 839 len 20480
5884 085/300 Write block 1470 len 36864 data: 5 5 5 6 5 5 4 3 3
5885 086/300 Read block 4835 len 28672
5886 087/300 Write block 1040 len 12288 data: 2 2 2
5887 088/300 Write block 4697 len 32768 data: 2 2 2 2 2 2 2 2
5888 089/300 Read block 1922 len 36864
5889 090/300 Read block 4485 len 40960
5890 091/300 Write block 3677 len 20480 data: 2 2 2 2 2
5891 092/300 Read block 3404 len 16384
5892 093/300 Read block 1949 len 16384
5893 094/300 Read block 1150 len 40960
5894 095/300 Read block 4545 len 28672
5895 096/300 Flush
5896 097/300 Read block 1956 len 36864
5897 098/300 Read block 2316 len 16384
5898 099/300 Write block 3011 len 8192 data: 3 4
5899 100/300 Write block 4920 len 16384 data: 2 2 2 2
5900 101/300 Write block 1225 len 20480 data: 3 3 4 5 5
5901 102/300 Write block 4361 len 12288 data: 4 4 4
5902 103/300 Write block 4115 len 8192 data: 3 3
5903 104/300 Read block 138 len 20480
5904 105/300 Flush
5905 106/300 Write block 3854 len 20480 data: 2 3 2 3 3
5906 107/300 Flush
5907 108/300 Write block 4062 len 4096 data: 3
5908 109/300 Read block 4116 len 36864
5909 110/300 Write block 1278 len 28672 data: 2 2 2 2 2 3 3
5910 111/300 Read block 57 len 40960
5911 112/300 Write block 751 len 36864 data: 3 3 2 3 3 3 3 3 4
5912 113/300 Write block 4282 len 16384 data: 2 2 2 2
5913 114/300 Write block 1816 len 4096 data: 2
5914 115/300 Read block 4662 len 40960
5915 116/300 Write block 2916 len 32768 data: 3 4 4 4 5 5 5 4
5916 117/300 Write block 3138 len 16384 data: 3 3 3 4
5917 118/300 Read block 1517 len 8192
5918 119/300 Write block 3778 len 16384 data: 3 3 2 2
5919 120/300 Write block 4609 len 16384 data: 3 3 3 4
5920 121/300 Read block 1774 len 32768
5921 122/300 Write block 50 len 16384 data: 3 3 3 3
5922 123/300 Write block 196 len 12288 data: 2 2 2
5923 124/300 Read block 2611 len 40960
5924 125/300 Write block 1966 len 8192 data: 2 3
5925 126/300 Write block 3822 len 32768 data: 3 3 3 3 3 3 3 2
5926 127/300 Read block 4680 len 8192
5927 128/300 Write block 855 len 12288 data: 3 2 2
5928 129/300 Write block 949 len 28672 data: 3 3 4 4 3 3 3
5929 130/300 Flush
5930 131/300 Read block 1013 len 16384
5931 132/300 Write block 3601 len 8192 data: 2 2
5932 133/300 Write block 2238 len 24576 data: 2 2 3 3 4 4
5933 134/300 Write block 3543 len 20480 data: 5 5 5 5 5
5934 135/300 Write block 2971 len 40960 data: 4 5 6 6 7 6 6 5 3 3
5935 136/300 Read block 3051 len 24576
5936 137/300 Write block 2211 len 4096 data: 2
5937 138/300 Write block 3173 len 28672 data: 2 2 3 3 3 4 4
5938 139/300 Write block 3500 len 28672 data: 3 3 3 3 4 4 4
5939 140/300 Write block 3898 len 20480 data: 2 3 2 3 3
5940 141/300 Read block 2078 len 40960
5941 142/300 Write block 3387 len 12288 data: 3 3 4
5942 143/300 Read block 130 len 24576
5943 144/300 Flush
5944 145/300 Write block 1549 len 40960 data: 3 2 2 2 2 2 3 3 3 3
5945 146/300 Read block 996 len 40960
5946 147/300 Read block 447 len 4096
5947 148/300 Read block 4742 len 16384
5948 149/300 Read block 2114 len 12288
5949 150/300 Flush
5950 151/300 Write block 1129 len 24576 data: 7 7 6 4 6 6
5951 152/300 Read block 4367 len 4096
5952 153/300 Write block 2620 len 12288 data: 4 5 4
5953 154/300 Read block 4982 len 24576
5954 155/300 Read block 1385 len 20480
5955 156/300 Read block 1032 len 36864
5956 157/300 Write block 3446 len 16384 data: 2 2 2 2
5957 158/300 Read block 4712 len 16384
5958 159/300 Read block 3712 len 40960
5959 160/300 Write block 3502 len 4096 data: 4
5960 161/300 Write block 1751 len 24576 data: 4 4 4 4 4 4
5961 162/300 Read block 4686 len 12288
5962 163/300 Write block 790 len 8192 data: 2 2
5963 164/300 Read block 2318 len 4096
5964 165/300 Write block 2049 len 16384 data: 2 3 5 5
5965 166/300 Read block 2229 len 16384
5966 167/300 Read block 53 len 40960
5967 168/300 Write block 4362 len 40960 data: 5 5 4 4 4 3 2 2 2 2
5968 169/300 Read block 1879 len 28672
5969 170/300 Write block 719 len 20480 data: 3 2 2 2 2
5970 171/300 Read block 1792 len 32768
5971 172/300 Read block 4952 len 8192
5972 173/300 Write block 1280 len 12288 data: 3 3 3
5973 174/300 Write block 4059 len 32768 data: 3 3 3 4 4 3 3 3
5974 175/300 Write block 3645 len 20480 data: 3 3 2 4 4
5975 176/300 Read block 2379 len 32768
5976 177/300 Read block 2440 len 24576
5977 178/300 Write block 4327 len 24576 data: 2 3 3 3 3 3
5978 179/300 Read block 867 len 8192
5979 180/300 Read block 2751 len 12288
5980 181/300 Read block 3075 len 12288
5981 182/300 Write block 908 len 12288 data: 2 3 3
5982 183/300 Write block 1195 len 40960 data: 3 3 3 2 2 2 3 4 2 4
5983 184/300 Read block 4973 len 40960
5984 185/300 Read block 660 len 16384
5985 186/300 Write block 3940 len 20480 data: 2 2 3 3 3
5986 187/300 Read block 2560 len 24576
5987 188/300 Read block 4808 len 20480
5988 189/300 Write block 2265 len 16384 data: 2 2 3 2
5989 190/300 Write block 3798 len 20480 data: 2 2 2 2 2
5990 191/300 Read block 2279 len 32768
5991 192/300 Flush
5992 193/300 Write block 1228 len 32768 data: 6 6 5 5 4 4 4 4
5993 194/300 Read block 2180 len 20480
5994 195/300 Write block 2274 len 40960 data: 2 2 2 2 2 2 2 2 2 3
5995 196/300 Read block 4573 len 20480
5996 197/300 Write block 2116 len 40960 data: 3 3 3 3 2 2 2 2 2 2
5997 198/300 Flush
5998 199/300 Flush
5999 200/300 Write block 3063 len 4096 data: 4
6000 201/300 Read block 4299 len 8192
6001 202/300 Write block 4959 len 12288 data: 2 2 2
6002 203/300 Write block 1845 len 8192 data: 2 2
6003 204/300 Read block 286 len 16384
6004 205/300 Read block 256 len 36864
6005 206/300 Write block 2815 len 28672 data: 5 5 5 5 5 6 5
6006 207/300 Flush
6007 208/300 Read block 4006 len 28672
6008 209/300 Read block 1444 len 20480
6009 210/300 Write block 2413 len 28672 data: 3 3 4 5 5 4 4
6010 211/300 Write block 4639 len 20480 data: 2 2 2 2 2
6011 212/300 Write block 2300 len 36864 data: 4 4 3 2 2 2 2 2 2
6012 213/300 Flush
6013 214/300 Read block 1080 len 36864
6014 215/300 Write block 4626 len 40960 data: 3 3 2 3 2 2 2 2 2 2
6015 216/300 Read block 4528 len 16384
6016 217/300 Write block 1515 len 36864 data: 2 3 4 5 4 4 3 3 3
6017 218/300 Write block 4795 len 16384 data: 2 3 3 4
6018 219/300 Read block 94 len 20480
6019 220/300 Write block 4073 len 16384 data: 4 3 2 2
6020 221/300 Write block 4075 len 20480 data: 3 3 2 2 2
6021 222/300 Write block 3417 len 8192 data: 2 2
6022 223/300 Read block 3594 len 28672
6023 224/300 Write block 1167 len 40960 data: 3 3 4 4 4 4 3 2 2 2
6024 225/300 Write block 2890 len 20480 data: 7 6 5 4 3
6025 226/300 Write block 3188 len 28672 data: 2 2 2 2 2 2 2
6026 227/300 Read block 2662 len 28672
6027 228/300 Read block 4796 len 8192
6028 229/300 Read block 4123 len 16384
6029 230/300 Read block 463 len 4096
6030 231/300 Read block 653 len 40960
6031 232/300 Read block 4280 len 16384
6032 233/300 Read block 738 len 24576
6033 234/300 Flush
6034 235/300 Flush
6035 236/300 Write block 2052 len 24576 data: 6 5 4 4 3 2
6036 237/300 Read block 4838 len 8192
6037 238/300 Read block 2329 len 32768
6038 239/300 Flush
6039 240/300 Write block 3997 len 36864 data: 2 2 2 2 2 2 2 2 2
6040 241/300 Read block 3813 len 32768
6041 242/300 Write block 2915 len 32768 data: 2 4 5 5 5 6 6 6
6042 243/300 Write block 3740 len 4096 data: 5
6043 244/300 Flush
6044 245/300 Write block 3041 len 8192 data: 4 4
6045 246/300 Write block 2939 len 16384 data: 4 4 3 3
6046 247/300 Write block 885 len 16384 data: 2 3 2 3
6047 248/300 Write block 2536 len 28672 data: 4 5 5 3 3 3 3
6048 249/300 Write block 4723 len 12288 data: 2 2 2
6049 250/300 Read block 446 len 36864
6050 251/300 Write block 2178 len 20480 data: 3 3 3 2 2
6051 252/300 Read block 4978 len 40960
6052 253/300 Read block 2020 len 8192
6053 254/300 Flush
6054 255/300 Write block 2791 len 40960 data: 3 3 2 3 3 3 3 3 3 3
6055 256/300 Read block 2138 len 32768
6056 257/300 Read block 1087 len 24576
6057 258/300 Read block 3402 len 8192
6058 259/300 Flush
6059 260/300 Read block 1714 len 28672
6060 261/300 Write block 918 len 12288 data: 3 3 3
6061 262/300 Flush
6062 263/300 Flush
6063 264/300 Read block 4247 len 16384
6064 265/300 Write block 4435 len 20480 data: 4 4 3 3 3
6065 266/300 Write block 2979 len 12288 data: 4 4 3
6066 267/300 Write block 4681 len 4096 data: 3
6067 268/300 Read block 2738 len 32768
6068 269/300 Write block 2440 len 8192 data: 3 4
6069 270/300 Write block 420 len 4096 data: 3
6070 271/300 Read block 4459 len 36864
6071 272/300 Write block 2779 len 24576 data: 3 3 2 3 4 4
6072 273/300 Write block 846 len 40960 data: 2 2 2 2 2 2 2 2 2 4
6073 274/300 Read block 1650 len 12288
6074 275/300 Read block 1499 len 40960
6075 276/300 Read block 3208 len 8192
6076 277/300 Read block 4740 len 16384
6077 278/300 Read block 1740 len 32768
6078 279/300 Read block 4927 len 32768
6079 280/300 Write block 3186 len 28672 data: 2 2 3 3 3 3 3
6080 281/300 Read block 184 len 12288
6081 282/300 Read block 3716 len 32768
6082 283/300 Write block 1058 len 4096 data: 3
6083 284/300 Read block 2603 len 24576
6084 285/300 Read block 1445 len 20480
6085 286/300 Write block 662 len 12288 data: 3 4 4
6086 287/300 Read block 246 len 32768
6087 288/300 Write block 2963 len 8192 data: 3 3
6088 289/300 Read block 1546 len 8192
6089 290/300 Read block 2168 len 16384
6090 291/300 Read block 2346 len 32768
6091 292/300 Write block 3639 len 20480 data: 3 3 3 3 3
6092 293/300 Write block 4450 len 40960 data: 2 2 2 2 2 2 3 3 3 5
6093 294/300 Write block 1938 len 36864 data: 2 2 2 2 2 2 3 3 3
6094 295/300 Flush
6095 296/300 Read block 1020 len 40960
6096 297/300 Read block 2886 len 24576
6097 298/300 Write block 2989 len 20480 data: 3 4 3 2 2
6098 299/300 Write block 4038 len 20480 data: 3 2 2 2 2
6099 300/300 Read block 1590 len 8192
6100 Main task has work to do, go find it
6101 got dsc Start(0)
6102 start 0
6103 Feb 28 22:14:27.269 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/0, method: POST, req_id: 2d73bc9a-5698-4efe-8dfa-973ee48220df, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
6104 Replay: started 0, returned:Ok(())
6105 ----------------------------------------------------------------
6106 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6107 States: Offline Active Active
6108 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
6109 2110 Acked 3109 Write 9 New Done Done true
6110 2111 Acked 3110 Write 4 New Done Done true
6111 2112 Acked 3111 Flush 0 New Done Done true
6112 2113 Acked 3112 Read 8 New Done Done false
6113 2114 Acked 3113 Write 8 New Done Done false
6114 2115 Acked 3114 Read 2 New Done Done false
6115 2116 Acked 3115 Write 9 New Done Done false
6116 2117 Acked 3116 Read 8 New Done Done false
6117 2118 Acked 3117 Flush 0 New Done Done false
6118 2119 Acked 3118 Read 9 New Done Done false
6119 2120 Acked 3119 Read 6 New Done Done false
6120 2121 Acked 3120 Write 10 New Done Done false
6121 2122 Acked 3121 Write 3 New Done Done false
6122 2123 Acked 3122 Read 10 New Done Done false
6123 2124 Acked 3123 Read 5 New Done Done false
6124 2125 Acked 3124 Write 5 New Done Done false
6125 2126 Acked 3125 Flush 0 New Done Done false
6126 2127 Acked 3126 Read 3 New Done Done false
6127 2128 Acked 3127 Write 2 New Done Done false
6128 2129 Acked 3128 Read 5 New Done Done false
6129 2130 Acked 3129 Write 8 New Done Done false
6130 2131 Acked 3130 Read 7 New Done Done false
6131 2132 Acked 3131 Flush 0 New Done Done false
6132 2133 Acked 3132 Read 2 New Done Done false
6133 2134 Acked 3133 Flush 0 New Done Done false
6134 2135 Acked 3134 Read 5 New Done Done false
6135 2136 Acked 3135 Write 2 New Done Done false
6136 2137 Acked 3136 Flush 0 New Done Done false
6137 2138 Acked 3137 Read 2 New Done Done false
6138 2139 Acked 3138 Read 6 New Done Done false
6139 2140 Acked 3139 Read 2 New Done Done false
6140 2141 Acked 3140 Write 3 New Done Done false
6141 2142 Acked 3141 Read 6 New Done Done false
6142 2143 Acked 3142 Flush 0 New Done Done false
6143 2144 Acked 3143 Write 5 New Done Done false
6144 2145 Acked 3144 Write 4 New Done Done false
6145 2146 Acked 3145 Read 3 New Done Done false
6146 2147 Acked 3146 Flush 0 New Done Done false
6147 2148 Acked 3147 Read 4 New Done Done false
6148 2149 Acked 3148 Read 6 New Done Done false
6149 2150 Acked 3149 Read 5 New Done Done false
6150 2151 Acked 3150 Read 9 New Done Done false
6151 2152 Acked 3151 Write 1 New Done Done false
6152 2153 Acked 3152 Read 9 New Done Done false
6153 2154 Acked 3153 Read 9 New Done Done false
6154 2155 Acked 3154 Read 7 New Done Done false
6155 2156 Acked 3155 Read 4 New Done Done false
6156 2157 Acked 3156 Write 3 New Done Done false
6157 2158 Acked 3157 Read 4 New Done Done false
6158 2159 Acked 3158 Read 7 New Done Done false
6159 2160 Acked 3159 Read 4 New Done Done false
6160 2161 Acked 3160 Read 9 New Done Done false
6161 2162 Acked 3161 Write 8 New Done Done false
6162 2163 Acked 3162 Read 6 New Done Done false
6163 2164 Acked 3163 Read 9 New Done Done false
6164 2165 Acked 3164 Read 6 New Done Done false
6165 2166 Acked 3165 Read 7 New Done Done false
6166 2167 Acked 3166 Write 8 New Done Done false
6167 2168 Acked 3167 Flush 0 New Done Done false
6168 2169 Acked 3168 Write 10 New Done Done false
6169 2170 Acked 3169 Write 6 New Done Done false
6170 2171 Acked 3170 Read 7 New Done Done false
6171 2172 Acked 3171 Read 3 New Done Done false
6172 2173 Acked 3172 Read 3 New Done Done false
6173 2174 Acked 3173 Write 10 New Done Done false
6174 2175 Acked 3174 Write 9 New Done Done false
6175 2176 Acked 3175 Read 4 New Done Done false
6176 2177 Acked 3176 Read 8 New Done Done false
6177 2178 Acked 3177 Write 3 New Done Done false
6178 2179 Acked 3178 Write 4 New Done Done false
6179 2180 Acked 3179 Read 6 New Done Done false
6180 2181 Acked 3180 Write 2 New Done Done false
6181 2182 Acked 3181 Write 8 New Done Done false
6182 2183 Acked 3182 Read 8 New Done Done false
6183 2184 Acked 3183 Read 4 New Done Done false
6184 2185 Acked 3184 Write 1 New Done Done false
6185 2186 Acked 3185 Write 9 New Done Done false
6186 2187 Acked 3186 Read 9 New Done Done false
6187 2188 Acked 3187 Read 8 New Done Done false
6188 2189 Acked 3188 Flush 0 New Done Done false
6189 2190 Acked 3189 Write 3 New Done Done false
6190 2191 Acked 3190 Read 3 New Done Done false
6191 2192 Acked 3191 Write 8 New Done Done false
6192 2193 Acked 3192 Read 5 New Done Done false
6193 2194 Acked 3193 Write 9 New Done Done false
6194 2195 Acked 3194 Read 7 New Done Done false
6195 2196 Acked 3195 Write 3 New Done Done false
6196 2197 Acked 3196 Write 8 New Done Done false
6197 2198 Acked 3197 Read 9 New Done Done false
6198 2199 Acked 3198 Read 10 New Done Done false
6199 2200 Acked 3199 Write 5 New Done Done false
6200 2201 Acked 3200 Read 4 New Done Done false
6201 2202 Acked 3201 Read 4 New Done Done false
6202 2203 Acked 3202 Read 10 New Done Done false
6203 2204 Acked 3203 Read 7 New Done Done false
6204 2205 Acked 3204 Flush 0 New Done Done false
6205 2206 Acked 3205 Read 9 New Done Done false
6206 2207 Acked 3206 Read 4 New Done Done false
6207 2208 Acked 3207 Write 2 New Done Done false
6208 2209 Acked 3208 Write 4 New Done Done false
6209 2210 Acked 3209 Write 5 New Done Done false
6210 2211 Acked 3210 Write 3 New Done Done false
6211 2212 Acked 3211 Write 2 New Done Done false
6212 2213 Acked 3212 Read 5 New Done Done false
6213 2214 Acked 3213 Flush 0 New Done Done false
6214 2215 Acked 3214 Write 5 New Done Done false
6215 2216 Acked 3215 Flush 0 New Done Done false
6216 2217 Acked 3216 Write 1 New Done Done false
6217 2218 Acked 3217 Read 9 New Done Done false
6218 2219 Acked 3218 Write 7 New Done Done false
6219 2220 Acked 3219 Read 10 New Done Done false
6220 2221 Acked 3220 Write 9 New Done Done false
6221 2222 Acked 3221 Write 4 New Done Done false
6222 2223 Acked 3222 Write 1 New Done Done false
6223 2224 Acked 3223 Read 10 New Done Done false
6224 2225 Acked 3224 Write 8 New Done Done false
6225 2226 Acked 3225 Write 4 New Done Done false
6226 2227 Acked 3226 Read 2 New Done Done false
6227 2228 Acked 3227 Write 4 New Done Done false
6228 2229 Acked 3228 Write 4 New Done Done false
6229 2230 Acked 3229 Read 8 New Done Done false
6230 2231 Acked 3230 Write 4 New Done Done false
6231 2232 Acked 3231 Write 3 New Done Done false
6232 2233 Acked 3232 Read 10 New Done Done false
6233 2234 Acked 3233 Write 2 New Done Done false
6234 2235 Acked 3234 Write 8 New Done Done false
6235 2236 Acked 3235 Read 2 New Done Done false
6236 2237 Acked 3236 Write 3 New Done Done false
6237 2238 Acked 3237 Write 7 New Done Done false
6238 2239 Acked 3238 Flush 0 New Done Done false
6239 2240 Acked 3239 Read 4 New Done Done false
6240 2241 Acked 3240 Write 2 New Done Done false
6241 2242 Acked 3241 Write 6 New Done Done false
6242 2243 Acked 3242 Write 5 New Done Done false
6243 2244 Acked 3243 Write 10 New Done Done false
6244 2245 Acked 3244 Read 6 New Done Done false
6245 2246 Acked 3245 Write 1 New Done Done false
6246 2247 Acked 3246 Write 7 New Done Done false
6247 2248 Acked 3247 Write 7 New Done Done false
6248 2249 Acked 3248 Write 5 New Done Done false
6249 2250 Acked 3249 Read 10 New Done Done false
6250 2251 Acked 3250 Write 3 New Done Done false
6251 2252 Acked 3251 Read 6 New Done Done false
6252 2253 Acked 3252 Flush 0 New Done Done false
6253 2254 Acked 3253 Write 10 New Done Done false
6254 2255 Acked 3254 Read 10 New Done Done false
6255 2256 Acked 3255 Read 1 New Done Done false
6256 2257 Acked 3256 Read 4 New Done Done false
6257 2258 Acked 3257 Read 3 New Done Done false
6258 2259 Acked 3258 Flush 0 New Done Done false
6259 2260 Acked 3259 Write 6 New Done Done false
6260 2261 Acked 3260 Read 1 New Done Done false
6261 2262 Acked 3261 Write 3 New Done Done false
6262 2263 Acked 3262 Read 6 New Done Done false
6263 2264 Acked 3263 Read 5 New Done Done false
6264 2265 Acked 3264 Read 9 New Done Done false
6265 2266 Acked 3265 Write 4 New Done Done false
6266 2267 Acked 3266 Read 4 New Done Done false
6267 2268 Acked 3267 Read 10 New Done Done false
6268 2269 Acked 3268 Write 1 New Done Done false
6269 2270 Acked 3269 Write 6 New Done Done false
6270 2271 Acked 3270 Read 3 New Done Done false
6271 2272 Acked 3271 Write 2 New Done Done false
6272 2273 Acked 3272 Read 1 New Done Done false
6273 2274 Acked 3273 Write 4 New Done Done false
6274 2275 Acked 3274 Read 4 New Done Done false
6275 2276 Acked 3275 Read 10 New Done Done false
6276 2277 Acked 3276 Write 10 New Done Done false
6277 2278 Acked 3277 Read 7 New Done Done false
6278 2279 Acked 3278 Write 5 New Done Done false
6279 2280 Acked 3279 Read 8 New Done Done false
6280 2281 Acked 3280 Read 2 New Done Done false
6281 2282 Acked 3281 Write 3 New Done Done false
6282 2283 Acked 3282 Write 8 New Done Done false
6283 2284 Acked 3283 Write 5 New Done Done false
6284 2285 Acked 3284 Read 8 New Done Done false
6285 2286 Acked 3285 Read 6 New Done Done false
6286 2287 Acked 3286 Write 6 New Done Done false
6287 2288 Acked 3287 Read 2 New Done Done false
6288 2289 Acked 3288 Read 3 New Done Done false
6289 2290 Acked 3289 Read 3 New Done Done false
6290 2291 Acked 3290 Write 3 New Done Done false
6291 2292 Acked 3291 Write 10 New Done Done false
6292 2293 Acked 3292 Read 10 New Done Done false
6293 2294 Acked 3293 Read 4 New Done Done false
6294 2295 Acked 3294 Write 5 New Done Done false
6295 2296 Acked 3295 Read 6 New Done Done false
6296 2297 Acked 3296 Flush 0 New Done Done false
6297 2298 Acked 3297 Read 5 New Done Done false
6298 2299 Acked 3298 Write 4 New Done Done false
6299 2300 Acked 3299 Write 5 New Done Done false
6300 2301 Acked 3300 Read 8 New Done Done false
6301 2302 Acked 3301 Flush 0 New Done Done false
6302 2303 Acked 3302 Write 8 New Done Done false
6303 2304 Acked 3303 Read 5 New Done Done false
6304 2305 Acked 3304 Write 10 New Done Done false
6305 2306 Acked 3305 Read 5 New Done Done false
6306 2307 Acked 3306 Write 10 New Done Done false
6307 2308 Acked 3307 Flush 0 New Done Done false
6308 2309 Acked 3308 Flush 0 New Done Done false
6309 2310 Acked 3309 Write 1 New Done Done false
6310 2311 Acked 3310 Read 2 New Done Done false
6311 2312 Acked 3311 Write 3 New Done Done false
6312 2313 Acked 3312 Write 2 New Done Done false
6313 2314 Acked 3313 Read 4 New Done Done false
6314 2315 Acked 3314 Read 9 New Done Done false
6315 2316 Acked 3315 Write 7 New Done Done false
6316 2317 Acked 3316 Flush 0 New Done Done false
6317 2318 Acked 3317 Read 7 New Done Done false
6318 2319 Acked 3318 Read 5 New Done Done false
6319 2320 Acked 3319 Write 7 New Done Done false
6320 2321 Acked 3320 Write 5 New Done Done false
6321 2322 Acked 3321 Write 9 New Done Done false
6322 2323 Acked 3322 Flush 0 New Done Done false
6323 2324 Acked 3323 Read 9 New Done Done false
6324 2325 Acked 3324 Write 10 New Done Done false
6325 2326 Acked 3325 Read 4 New Done Done false
6326 2327 Acked 3326 Write 9 New Done Done false
6327 2328 Acked 3327 Write 4 New Done Done false
6328 2329 Acked 3328 Read 5 New Done Done false
6329 2330 Acked 3329 Write 4 New Done Done false
6330 2331 Acked 3330 Write 5 New Done Done false
6331 2332 Acked 3331 Write 2 New Done Done false
6332 2333 Acked 3332 Read 7 New Done Done false
6333 2334 Acked 3333 Write 10 New Done Done false
6334 2335 Acked 3334 Write 5 New Done Done false
6335 2336 Acked 3335 Write 7 New Done Done false
6336 2337 Acked 3336 Read 7 New Done Done false
6337 2338 Acked 3337 Read 2 New Done Done false
6338 2339 Acked 3338 Read 4 New Done Done false
6339 2340 Acked 3339 Read 1 New Done Done false
6340 2341 Acked 3340 Read 10 New Done Done false
6341 2342 Acked 3341 Read 4 New Done Done false
6342 2343 Acked 3342 Read 6 New Done Done false
6343 2344 Acked 3343 Flush 0 New Done Done false
6344 2345 Acked 3344 Flush 0 New Done Done false
6345 2346 Acked 3345 Write 6 New Done Done false
6346 2347 Acked 3346 Read 2 New Done Done false
6347 2348 Acked 3347 Read 8 New Done Done false
6348 2349 Acked 3348 Flush 0 New Done Done false
6349 2350 Acked 3349 Write 9 New Done Done false
6350 2351 Acked 3350 Read 8 New Done Done false
6351 2352 Acked 3351 Write 8 New Done Done false
6352 2353 Acked 3352 Write 1 New Done Done false
6353 2354 Acked 3353 Flush 0 New Done Done false
6354 2355 Acked 3354 Write 2 New Done Done false
6355 2356 Acked 3355 Write 4 New Done Done false
6356 2357 Acked 3356 Write 4 New Done Done false
6357 2358 Acked 3357 Write 7 New Done Done false
6358 2359 Acked 3358 Write 3 New Done Done false
6359 2360 Acked 3359 Read 9 New Done Done false
6360 2361 Acked 3360 Write 5 New Done Done false
6361 2362 Acked 3361 Read 10 New Done Done false
6362 2363 Acked 3362 Read 2 New Done Done false
6363 2364 Acked 3363 Flush 0 New Done Done false
6364 2365 Acked 3364 Write 10 New Done Done false
6365 2366 Acked 3365 Read 8 New Done Done false
6366 2367 Acked 3366 Read 6 New Done Done false
6367 2368 Acked 3367 Read 2 New Done Done false
6368 2369 Acked 3368 Flush 0 New Done Done false
6369 2370 Acked 3369 Read 7 New Done Done false
6370 2371 Acked 3370 Write 3 New Done Done false
6371 2372 Acked 3371 Flush 0 New Done Done false
6372 2373 Acked 3372 Flush 0 New Done Done false
6373 2374 Acked 3373 Read 4 New Done Done false
6374 2375 Acked 3374 Write 5 New Done Done false
6375 2376 Acked 3375 Write 3 New Done Done false
6376 2377 Acked 3376 Write 1 New Done Done false
6377 2378 Acked 3377 Read 8 New Done Done false
6378 2379 Acked 3378 Write 2 New Done Done false
6379 2380 Acked 3379 Write 1 New Done Done false
6380 2381 Acked 3380 Read 9 New Done Done false
6381 2382 Acked 3381 Write 6 New Done Done false
6382 2383 Acked 3382 Write 10 New Done Done false
6383 2384 Acked 3383 Read 3 New Done Done false
6384 2385 Acked 3384 Read 10 New Done Done false
6385 2386 Acked 3385 Read 2 New Done Done false
6386 2387 Acked 3386 Read 4 New Done Done false
6387 2388 Acked 3387 Read 8 New Done Done false
6388 2389 Acked 3388 Read 8 New Done Done false
6389 2390 Acked 3389 Write 7 New Done Done false
6390 2391 Acked 3390 Read 3 New Done Done false
6391 2392 Acked 3391 Read 8 New Done Done false
6392 2393 Acked 3392 Write 1 New Done Done false
6393 2394 Acked 3393 Read 6 New Done Done false
6394 2395 Acked 3394 Read 5 New Done Done false
6395 2396 Acked 3395 Write 3 New Done Done false
6396 2397 Acked 3396 Read 8 New Done Done false
6397 2398 Acked 3397 Write 2 New Done Done false
6398 2399 Acked 3398 Read 2 New Done Done false
6399 2400 Acked 3399 Read 4 New Done Done false
6400 2401 Acked 3400 Read 8 New Done Done false
6401 2402 Acked 3401 Write 5 New Done Done false
6402 2403 Acked 3402 Write 10 New Done Done false
6403 2404 Acked 3403 Write 9 New Done Done false
6404 2405 Acked 3404 Flush 0 New Done Done false
6405 2406 Acked 3405 Read 10 New Done Done false
6406 2407 Acked 3406 Read 6 New Done Done false
6407 2408 Acked 3407 Write 5 New Done Done false
6408 2409 Acked 3408 Write 5 New Done Done false
6409 2410 Acked 3409 Read 2 New Done Done false
6410 STATES DS:0 DS:1 DS:2 TOTAL
6411 New 301 0 0 301
6412 Sent 0 0 0 0
6413 Done 0 301 301 602
6414 Skipped 0 0 0 0
6415 Error 0 0 0 0
6416 Last Flush: 3108 3404 3404
6417 Downstairs last five completed: 3108 3107 3106 3105 3104
6418 Upstairs last five completed: 2410 2409 2408 2407 2406
6419 CLIENT: Up:0 ds:301 act:2
64202024-02-28T22:14:27.526ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64212024-02-28T22:14:27.526ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64222024-02-28T22:14:27.526ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64232024-02-28T22:14:27.526ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64242024-02-28T22:14:27.526ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64252024-02-28T22:14:27.526ZINFOcrucible: [0] client re-new 302 jobs since flush 3108 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64262024-02-28T22:14:27.526ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64272024-02-28T22:14:28.528ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64282024-02-28T22:14:28.528ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64292024-02-28T22:14:28.528ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64302024-02-28T22:14:28.528ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64312024-02-28T22:14:28.528ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64322024-02-28T22:14:28.528ZINFOcrucible: [0] client re-new 302 jobs since flush 3108 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64332024-02-28T22:14:28.528ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64342024-02-28T22:14:29.529ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64352024-02-28T22:14:29.530ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64362024-02-28T22:14:29.530ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64372024-02-28T22:14:29.530ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64382024-02-28T22:14:29.530ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64392024-02-28T22:14:29.530ZINFOcrucible: [0] client re-new 302 jobs since flush 3108 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64402024-02-28T22:14:29.530ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
6441 [8810] got start action
6442 [8810] I am going to restart
6443 Starting downstairs at port 8810
6444 Make output file at "/tmp/dsc/downstairs-8810.txt"
6445 [8810][0] reports Starting
6446 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1172)
6447 [8810][0] reports Running
64482024-02-28T22:14:30.531ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64492024-02-28T22:14:30.531ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64502024-02-28T22:14:30.531ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64512024-02-28T22:14:30.531ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64522024-02-28T22:14:30.531ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64532024-02-28T22:14:30.531ZINFOcrucible: [0] client re-new 302 jobs since flush 3108 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
64542024-02-28T22:14:30.531ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
6455 ----------------------------------------------------------------
6456 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
6457 States: Offline Active Active
6458 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
6459 2110 Acked 3109 Write 9 New Done Done true
6460 2111 Acked 3110 Write 4 New Done Done true
6461 2112 Acked 3111 Flush 0 New Done Done true
6462 2113 Acked 3112 Read 8 New Done Done true
6463 2114 Acked 3113 Write 8 New Done Done true
6464 2115 Acked 3114 Read 2 New Done Done true
6465 2116 Acked 3115 Write 9 New Done Done true
6466 2117 Acked 3116 Read 8 New Done Done true
6467 2118 Acked 3117 Flush 0 New Done Done true
6468 2119 Acked 3118 Read 9 New Done Done true
6469 2120 Acked 3119 Read 6 New Done Done true
6470 2121 Acked 3120 Write 10 New Done Done true
6471 2122 Acked 3121 Write 3 New Done Done true
6472 2123 Acked 3122 Read 10 New Done Done true
6473 2124 Acked 3123 Read 5 New Done Done true
6474 2125 Acked 3124 Write 5 New Done Done true
6475 2126 Acked 3125 Flush 0 New Done Done true
6476 2127 Acked 3126 Read 3 New Done Done true
6477 2128 Acked 3127 Write 2 New Done Done true
6478 2129 Acked 3128 Read 5 New Done Done true
6479 2130 Acked 3129 Write 8 New Done Done true
6480 2131 Acked 3130 Read 7 New Done Done true
6481 2132 Acked 3131 Flush 0 New Done Done true
6482 2133 Acked 3132 Read 2 New Done Done true
6483 2134 Acked 3133 Flush 0 New Done Done true
6484 2135 Acked 3134 Read 5 New Done Done true
6485 2136 Acked 3135 Write 2 New Done Done true
6486 2137 Acked 3136 Flush 0 New Done Done true
6487 2138 Acked 3137 Read 2 New Done Done true
6488 2139 Acked 3138 Read 6 New Done Done true
6489 2140 Acked 3139 Read 2 New Done Done true
6490 2141 Acked 3140 Write 3 New Done Done true
6491 2142 Acked 3141 Read 6 New Done Done true
6492 2143 Acked 3142 Flush 0 New Done Done true
6493 2144 Acked 3143 Write 5 New Done Done true
6494 2145 Acked 3144 Write 4 New Done Done true
6495 2146 Acked 3145 Read 3 New Done Done true
6496 2147 Acked 3146 Flush 0 New Done Done true
6497 2148 Acked 3147 Read 4 New Done Done true
6498 2149 Acked 3148 Read 6 New Done Done true
6499 2150 Acked 3149 Read 5 New Done Done true
6500 2151 Acked 3150 Read 9 New Done Done true
6501 2152 Acked 3151 Write 1 New Done Done true
6502 2153 Acked 3152 Read 9 New Done Done true
6503 2154 Acked 3153 Read 9 New Done Done true
6504 2155 Acked 3154 Read 7 New Done Done true
6505 2156 Acked 3155 Read 4 New Done Done true
6506 2157 Acked 3156 Write 3 New Done Done true
6507 2158 Acked 3157 Read 4 New Done Done true
6508 2159 Acked 3158 Read 7 New Done Done true
6509 2160 Acked 3159 Read 4 New Done Done true
6510 2161 Acked 3160 Read 9 New Done Done true
6511 2162 Acked 3161 Write 8 New Done Done true
6512 2163 Acked 3162 Read 6 New Done Done true
6513 2164 Acked 3163 Read 9 New Done Done true
6514 2165 Acked 3164 Read 6 New Done Done true
6515 2166 Acked 3165 Read 7 New Done Done true
6516 2167 Acked 3166 Write 8 New Done Done true
6517 2168 Acked 3167 Flush 0 New Done Done true
6518 2169 Acked 3168 Write 10 New Done Done true
6519 2170 Acked 3169 Write 6 New Done Done true
6520 2171 Acked 3170 Read 7 New Done Done true
6521 2172 Acked 3171 Read 3 New Done Done true
6522 2173 Acked 3172 Read 3 New Done Done true
6523 2174 Acked 3173 Write 10 New Done Done true
6524 2175 Acked 3174 Write 9 New Done Done true
6525 2176 Acked 3175 Read 4 New Done Done true
6526 2177 Acked 3176 Read 8 New Done Done true
6527 2178 Acked 3177 Write 3 New Done Done true
6528 2179 Acked 3178 Write 4 New Done Done true
6529 2180 Acked 3179 Read 6 New Done Done true
6530 2181 Acked 3180 Write 2 New Done Done true
6531 2182 Acked 3181 Write 8 New Done Done true
6532 2183 Acked 3182 Read 8 New Done Done true
6533 2184 Acked 3183 Read 4 New Done Done true
6534 2185 Acked 3184 Write 1 New Done Done true
6535 2186 Acked 3185 Write 9 New Done Done true
6536 2187 Acked 3186 Read 9 New Done Done true
6537 2188 Acked 3187 Read 8 New Done Done true
6538 2189 Acked 3188 Flush 0 New Done Done true
6539 2190 Acked 3189 Write 3 New Done Done true
6540 2191 Acked 3190 Read 3 New Done Done true
6541 2192 Acked 3191 Write 8 New Done Done true
6542 2193 Acked 3192 Read 5 New Done Done true
6543 2194 Acked 3193 Write 9 New Done Done true
6544 2195 Acked 3194 Read 7 New Done Done true
6545 2196 Acked 3195 Write 3 New Done Done true
6546 2197 Acked 3196 Write 8 New Done Done true
6547 2198 Acked 3197 Read 9 New Done Done true
6548 2199 Acked 3198 Read 10 New Done Done true
6549 2200 Acked 3199 Write 5 New Done Done true
6550 2201 Acked 3200 Read 4 New Done Done true
6551 2202 Acked 3201 Read 4 New Done Done true
6552 2203 Acked 3202 Read 10 New Done Done true
6553 2204 Acked 3203 Read 7 New Done Done true
6554 2205 Acked 3204 Flush 0 New Done Done true
6555 2206 Acked 3205 Read 9 New Done Done true
6556 2207 Acked 3206 Read 4 New Done Done true
6557 2208 Acked 3207 Write 2 New Done Done true
6558 2209 Acked 3208 Write 4 New Done Done true
6559 2210 Acked 3209 Write 5 New Done Done true
6560 2211 Acked 3210 Write 3 New Done Done true
6561 2212 Acked 3211 Write 2 New Done Done true
6562 2213 Acked 3212 Read 5 New Done Done true
6563 2214 Acked 3213 Flush 0 New Done Done true
6564 2215 Acked 3214 Write 5 New Done Done true
6565 2216 Acked 3215 Flush 0 New Done Done true
6566 2217 Acked 3216 Write 1 New Done Done true
6567 2218 Acked 3217 Read 9 New Done Done true
6568 2219 Acked 3218 Write 7 New Done Done true
6569 2220 Acked 3219 Read 10 New Done Done true
6570 2221 Acked 3220 Write 9 New Done Done true
6571 2222 Acked 3221 Write 4 New Done Done true
6572 2223 Acked 3222 Write 1 New Done Done true
6573 2224 Acked 3223 Read 10 New Done Done true
6574 2225 Acked 3224 Write 8 New Done Done true
6575 2226 Acked 3225 Write 4 New Done Done true
6576 2227 Acked 3226 Read 2 New Done Done true
6577 2228 Acked 3227 Write 4 New Done Done true
6578 2229 Acked 3228 Write 4 New Done Done true
6579 2230 Acked 3229 Read 8 New Done Done true
6580 2231 Acked 3230 Write 4 New Done Done true
6581 2232 Acked 3231 Write 3 New Done Done true
6582 2233 Acked 3232 Read 10 New Done Done true
6583 2234 Acked 3233 Write 2 New Done Done true
6584 2235 Acked 3234 Write 8 New Done Done true
6585 2236 Acked 3235 Read 2 New Done Done true
6586 2237 Acked 3236 Write 3 New Done Done true
6587 2238 Acked 3237 Write 7 New Done Done true
6588 2239 Acked 3238 Flush 0 New Done Done true
6589 2240 Acked 3239 Read 4 New Done Done true
6590 2241 Acked 3240 Write 2 New Done Done true
6591 2242 Acked 3241 Write 6 New Done Done true
6592 2243 Acked 3242 Write 5 New Done Done true
6593 2244 Acked 3243 Write 10 New Done Done true
6594 2245 Acked 3244 Read 6 New Done Done true
6595 2246 Acked 3245 Write 1 New Done Done true
6596 2247 Acked 3246 Write 7 New Done Done true
6597 2248 Acked 3247 Write 7 New Done Done true
6598 2249 Acked 3248 Write 5 New Done Done true
6599 2250 Acked 3249 Read 10 New Done Done true
6600 2251 Acked 3250 Write 3 New Done Done true
6601 2252 Acked 3251 Read 6 New Done Done true
6602 2253 Acked 3252 Flush 0 New Done Done true
6603 2254 Acked 3253 Write 10 New Done Done true
6604 2255 Acked 3254 Read 10 New Done Done true
6605 2256 Acked 3255 Read 1 New Done Done true
6606 2257 Acked 3256 Read 4 New Done Done true
6607 2258 Acked 3257 Read 3 New Done Done true
6608 2259 Acked 3258 Flush 0 New Done Done true
6609 2260 Acked 3259 Write 6 New Done Done true
6610 2261 Acked 3260 Read 1 New Done Done true
6611 2262 Acked 3261 Write 3 New Done Done true
6612 2263 Acked 3262 Read 6 New Done Done true
6613 2264 Acked 3263 Read 5 New Done Done true
6614 2265 Acked 3264 Read 9 New Done Done true
6615 2266 Acked 3265 Write 4 New Done Done true
6616 2267 Acked 3266 Read 4 New Done Done true
6617 2268 Acked 3267 Read 10 New Done Done true
6618 2269 Acked 3268 Write 1 New Done Done true
6619 2270 Acked 3269 Write 6 New Done Done true
6620 2271 Acked 3270 Read 3 New Done Done true
6621 2272 Acked 3271 Write 2 New Done Done true
6622 2273 Acked 3272 Read 1 New Done Done true
6623 2274 Acked 3273 Write 4 New Done Done true
6624 2275 Acked 3274 Read 4 New Done Done true
6625 2276 Acked 3275 Read 10 New Done Done true
6626 2277 Acked 3276 Write 10 New Done Done true
6627 2278 Acked 3277 Read 7 New Done Done true
6628 2279 Acked 3278 Write 5 New Done Done true
6629 2280 Acked 3279 Read 8 New Done Done true
6630 2281 Acked 3280 Read 2 New Done Done true
6631 2282 Acked 3281 Write 3 New Done Done true
6632 2283 Acked 3282 Write 8 New Done Done true
6633 2284 Acked 3283 Write 5 New Done Done true
6634 2285 Acked 3284 Read 8 New Done Done true
6635 2286 Acked 3285 Read 6 New Done Done true
6636 2287 Acked 3286 Write 6 New Done Done true
6637 2288 Acked 3287 Read 2 New Done Done true
6638 2289 Acked 3288 Read 3 New Done Done true
6639 2290 Acked 3289 Read 3 New Done Done true
6640 2291 Acked 3290 Write 3 New Done Done true
6641 2292 Acked 3291 Write 10 New Done Done true
6642 2293 Acked 3292 Read 10 New Done Done true
6643 2294 Acked 3293 Read 4 New Done Done true
6644 2295 Acked 3294 Write 5 New Done Done true
6645 2296 Acked 3295 Read 6 New Done Done true
6646 2297 Acked 3296 Flush 0 New Done Done true
6647 2298 Acked 3297 Read 5 New Done Done true
6648 2299 Acked 3298 Write 4 New Done Done true
6649 2300 Acked 3299 Write 5 New Done Done true
6650 2301 Acked 3300 Read 8 New Done Done true
6651 2302 Acked 3301 Flush 0 New Done Done true
6652 2303 Acked 3302 Write 8 New Done Done true
6653 2304 Acked 3303 Read 5 New Done Done true
6654 2305 Acked 3304 Write 10 New Done Done true
6655 2306 Acked 3305 Read 5 New Done Done true
6656 2307 Acked 3306 Write 10 New Done Done true
6657 2308 Acked 3307 Flush 0 New Done Done true
6658 2309 Acked 3308 Flush 0 New Done Done true
6659 2310 Acked 3309 Write 1 New Done Done true
6660 2311 Acked 3310 Read 2 New Done Done true
6661 2312 Acked 3311 Write 3 New Done Done true
6662 2313 Acked 3312 Write 2 New Done Done true
6663 2314 Acked 3313 Read 4 New Done Done true
6664 2315 Acked 3314 Read 9 New Done Done true
6665 2316 Acked 3315 Write 7 New Done Done true
6666 2317 Acked 3316 Flush 0 New Done Done true
6667 2318 Acked 3317 Read 7 New Done Done true
6668 2319 Acked 3318 Read 5 New Done Done true
6669 2320 Acked 3319 Write 7 New Done Done true
6670 2321 Acked 3320 Write 5 New Done Done true
6671 2322 Acked 3321 Write 9 New Done Done true
6672 2323 Acked 3322 Flush 0 New Done Done true
6673 2324 Acked 3323 Read 9 New Done Done true
6674 2325 Acked 3324 Write 10 New Done Done true
6675 2326 Acked 3325 Read 4 New Done Done true
6676 2327 Acked 3326 Write 9 New Done Done true
6677 2328 Acked 3327 Write 4 New Done Done true
6678 2329 Acked 3328 Read 5 New Done Done true
6679 2330 Acked 3329 Write 4 New Done Done true
6680 2331 Acked 3330 Write 5 New Done Done true
6681 2332 Acked 3331 Write 2 New Done Done true
6682 2333 Acked 3332 Read 7 New Done Done true
6683 2334 Acked 3333 Write 10 New Done Done true
6684 2335 Acked 3334 Write 5 New Done Done true
6685 2336 Acked 3335 Write 7 New Done Done true
6686 2337 Acked 3336 Read 7 New Done Done true
6687 2338 Acked 3337 Read 2 New Done Done true
6688 2339 Acked 3338 Read 4 New Done Done true
6689 2340 Acked 3339 Read 1 New Done Done true
6690 2341 Acked 3340 Read 10 New Done Done true
6691 2342 Acked 3341 Read 4 New Done Done true
6692 2343 Acked 3342 Read 6 New Done Done true
6693 2344 Acked 3343 Flush 0 New Done Done true
6694 2345 Acked 3344 Flush 0 New Done Done true
6695 2346 Acked 3345 Write 6 New Done Done true
6696 2347 Acked 3346 Read 2 New Done Done true
6697 2348 Acked 3347 Read 8 New Done Done true
6698 2349 Acked 3348 Flush 0 New Done Done true
6699 2350 Acked 3349 Write 9 New Done Done true
6700 2351 Acked 3350 Read 8 New Done Done true
6701 2352 Acked 3351 Write 8 New Done Done true
6702 2353 Acked 3352 Write 1 New Done Done true
6703 2354 Acked 3353 Flush 0 New Done Done true
6704 2355 Acked 3354 Write 2 New Done Done true
6705 2356 Acked 3355 Write 4 New Done Done true
6706 2357 Acked 3356 Write 4 New Done Done true
6707 2358 Acked 3357 Write 7 New Done Done true
6708 2359 Acked 3358 Write 3 New Done Done true
6709 2360 Acked 3359 Read 9 New Done Done true
6710 2361 Acked 3360 Write 5 New Done Done true
6711 2362 Acked 3361 Read 10 New Done Done true
6712 2363 Acked 3362 Read 2 New Done Done true
6713 2364 Acked 3363 Flush 0 New Done Done true
6714 2365 Acked 3364 Write 10 New Done Done true
6715 2366 Acked 3365 Read 8 New Done Done true
6716 2367 Acked 3366 Read 6 New Done Done true
6717 2368 Acked 3367 Read 2 New Done Done true
6718 2369 Acked 3368 Flush 0 New Done Done true
6719 2370 Acked 3369 Read 7 New Done Done true
6720 2371 Acked 3370 Write 3 New Done Done true
6721 2372 Acked 3371 Flush 0 New Done Done true
6722 2373 Acked 3372 Flush 0 New Done Done true
6723 2374 Acked 3373 Read 4 New Done Done true
6724 2375 Acked 3374 Write 5 New Done Done true
6725 2376 Acked 3375 Write 3 New Done Done true
6726 2377 Acked 3376 Write 1 New Done Done true
6727 2378 Acked 3377 Read 8 New Done Done true
6728 2379 Acked 3378 Write 2 New Done Done true
6729 2380 Acked 3379 Write 1 New Done Done true
6730 2381 Acked 3380 Read 9 New Done Done true
6731 2382 Acked 3381 Write 6 New Done Done true
6732 2383 Acked 3382 Write 10 New Done Done true
6733 2384 Acked 3383 Read 3 New Done Done true
6734 2385 Acked 3384 Read 10 New Done Done true
6735 2386 Acked 3385 Read 2 New Done Done true
6736 2387 Acked 3386 Read 4 New Done Done true
6737 2388 Acked 3387 Read 8 New Done Done true
6738 2389 Acked 3388 Read 8 New Done Done true
6739 2390 Acked 3389 Write 7 New Done Done true
6740 2391 Acked 3390 Read 3 New Done Done true
6741 2392 Acked 3391 Read 8 New Done Done true
6742 2393 Acked 3392 Write 1 New Done Done true
6743 2394 Acked 3393 Read 6 New Done Done true
6744 2395 Acked 3394 Read 5 New Done Done true
6745 2396 Acked 3395 Write 3 New Done Done true
6746 2397 Acked 3396 Read 8 New Done Done true
6747 2398 Acked 3397 Write 2 New Done Done true
6748 2399 Acked 3398 Read 2 New Done Done true
6749 2400 Acked 3399 Read 4 New Done Done true
6750 2401 Acked 3400 Read 8 New Done Done true
6751 2402 Acked 3401 Write 5 New Done Done true
6752 2403 Acked 3402 Write 10 New Done Done true
6753 2404 Acked 3403 Write 9 New Done Done true
6754 2405 Acked 3404 Flush 0 New Done Done true
6755 2406 Acked 3405 Read 10 New Done Done true
6756 2407 Acked 3406 Read 6 New Done Done true
6757 2408 Acked 3407 Write 5 New Done Done true
6758 2409 Acked 3408 Write 5 New Done Done true
6759 2410 Acked 3409 Read 2 New Done Done true
6760 2411 Acked 3410 Flush 0 New Done Done true
6761 STATES DS:0 DS:1 DS:2 TOTAL
6762 New 302 0 0 302
6763 Sent 0 0 0 0
6764 Done 0 302 302 604
6765 Skipped 0 0 0 0
6766 Error 0 0 0 0
6767 Last Flush: 3108 3410 3410
6768 Downstairs last five completed: 3108 3107 3106 3105 3104
6769 Upstairs last five completed: 2411 2410 2409 2408 2407
6770 CLIENT: Up:0 ds:302 act:2
67712024-02-28T22:14:31.532ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
67722024-02-28T22:14:31.532ZINFOcrucible: ds_connection connected = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
67732024-02-28T22:14:31.534ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
67742024-02-28T22:14:31.534ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
67752024-02-28T22:14:31.534ZINFOcrucible: send last flush ID to this DS: 3108 = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
67762024-02-28T22:14:31.534ZINFOcrucible: Replied this last flush ID: 3108 = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
67772024-02-28T22:14:31.534ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
67782024-02-28T22:14:31.534ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
67792024-02-28T22:14:31.534ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
67802024-02-28T22:14:31.534ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
6781 ----------------------------------------------------------------
6782 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
6783 Downstairs last five completed: 3410 3409 3408 3407 3406
6784 Upstairs last five completed: 2411 2410 2409 2408 2407
6785 CLIENT: Up:0 ds:0 act:3
6786 Replay: All jobs finished, all DS active.
6787 Main task has work to do, go find it
6788 got dsc Stop(2)
6789 stop 2
6790 Feb 28 22:14:35.275 INFO request completed, latency_us: 295, [8830] Got stop action so:false kr:false
6791 response_code: 204, uri: /stop/cid/2, method: POST, req_id: c1a9063e-854b-4194-9542-9b87e6094392, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
67922024-02-28T22:14:35.276ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
6793 Feb 28 22:14:35.276 INFO request completed, latency_us: 122, response_code: 200{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-28T22:14:35.276335799Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
6794 , uri: /state/cid/2, method: GET, req_id: 88a1c9e9-8947-4628-811c-73e681e44d0f, remote_addr: 127.0.0.1:36617{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-28T22:14:35.276367968Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
6795 , local_addr: 127.0.0.1:9998
67962024-02-28T22:14:35.276ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
67972024-02-28T22:14:35.276ZINFOcrucible: [2] client re-new 0 jobs since flush 3410 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
67982024-02-28T22:14:35.276ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
6799 [8830] Exited with: ExitStatus(unix_wait_status(9))
6800 [8830][2] reports Exit
68012024-02-28T22:14:36.277ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68022024-02-28T22:14:36.277ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68032024-02-28T22:14:36.277ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68042024-02-28T22:14:36.277ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68052024-02-28T22:14:36.277ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68062024-02-28T22:14:36.277ZINFOcrucible: [2] client re-new 0 jobs since flush 3410 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68072024-02-28T22:14:36.277ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68082024-02-28T22:14:37.279ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68092024-02-28T22:14:37.280ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68102024-02-28T22:14:37.280ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68112024-02-28T22:14:37.280ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68122024-02-28T22:14:37.280ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68132024-02-28T22:14:37.280ZINFOcrucible: [2] client re-new 0 jobs since flush 3410 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68142024-02-28T22:14:37.280ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68152024-02-28T22:14:38.281ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68162024-02-28T22:14:38.281ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68172024-02-28T22:14:38.281ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68182024-02-28T22:14:38.281ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68192024-02-28T22:14:38.281ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68202024-02-28T22:14:38.281ZINFOcrucible: [2] client re-new 0 jobs since flush 3410 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68212024-02-28T22:14:38.281ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
6822 Feb 28 22:14:39.278 INFO request completed, latency_us: 206, response_code: 200, uri: /state/cid/2, method: GET, req_id: 1fe99328-2787-4b4e-800e-6c7a1b216012, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
6823 001/300 Write block 4977 len 40960 data: 4 3 3 3 3 2 3 3 3 3
6824 002/300 Read block 301 len 24576
6825 003/300 Read block 1886 len 12288
6826 004/300 Write block 3212 len 12288 data: 3 3 3
6827 005/300 Read block 496 len 32768
68282024-02-28T22:14:39.283ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68292024-02-28T22:14:39.283ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68302024-02-28T22:14:39.283ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68312024-02-28T22:14:39.283ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68322024-02-28T22:14:39.283ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68332024-02-28T22:14:39.283ZINFOcrucible: [2] client re-new 5 jobs since flush 3410 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
68342024-02-28T22:14:39.283ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
6835 006/300 Read block 2556 len 20480
6836 007/300 Write block 1494 len 28672 data: 2 2 2 2 2 2 2
6837 008/300 Write block 4877 len 8192 data: 2 2
6838 009/300 Write block 2270 len 12288 data: 2 2 2
6839 010/300 Flush
6840 011/300 Write block 1235 len 24576 data: 5 4 4 4 4 4
6841 012/300 Read block 4530 len 36864
6842 013/300 Write block 3178 len 32768 data: 5 5 4 3 3 2 2 2
6843 014/300 Write block 3952 len 16384 data: 2 4 4 5
6844 015/300 Write block 503 len 40960 data: 2 2 2 3 3 4 4 3 3 4
6845 016/300 Read block 191 len 4096
6846 017/300 Flush
6847 018/300 Read block 2679 len 12288
6848 019/300 Write block 4128 len 40960 data: 4 4 3 2 2 2 2 3 2 3
6849 020/300 Flush
6850 021/300 Read block 4453 len 28672
6851 022/300 Read block 2108 len 32768
6852 023/300 Read block 514 len 20480
6853 024/300 Read block 1649 len 24576
6854 025/300 Write block 3101 len 8192 data: 2 2
6855 026/300 Read block 3404 len 16384
6856 027/300 Write block 1151 len 12288 data: 3 3 3
6857 028/300 Write block 212 len 8192 data: 3 3
6858 029/300 Read block 2926 len 20480
6859 030/300 Read block 822 len 32768
6860 031/300 Write block 539 len 36864 data: 3 3 2 2 2 2 2 2 2
6861 032/300 Write block 2442 len 12288 data: 3 3 3
6862 033/300 Write block 3335 len 4096 data: 2
6863 034/300 Read block 1930 len 8192
6864 035/300 Write block 433 len 16384 data: 2 2 2 2
6865 036/300 Flush
6866 037/300 Write block 2393 len 20480 data: 2 2 3 3 3
6867 038/300 Write block 2648 len 24576 data: 4 4 4 4 4 4
6868 039/300 Read block 165 len 20480
6869 040/300 Write block 94 len 12288 data: 2 2 2
6870 041/300 Write block 94 len 32768 data: 3 3 3 2 3 2 2 2
6871 042/300 Write block 2317 len 8192 data: 2 3
6872 043/300 Write block 3110 len 20480 data: 4 4 4 3 2
6873 044/300 Write block 1222 len 12288 data: 3 3 3
6874 045/300 Read block 3766 len 32768
6875 046/300 Read block 2770 len 36864
6876 047/300 Write block 2131 len 28672 data: 4 4 4 3 2 2 2
6877 048/300 Read block 1397 len 36864
6878 049/300 Read block 2985 len 16384
6879 050/300 Write block 1435 len 20480 data: 4 4 3 2 2
6880 051/300 Read block 2165 len 32768
6881 052/300 Write block 2387 len 28672 data: 4 3 3 3 3 3 3
6882 053/300 Read block 3598 len 20480
6883 054/300 Read block 1721 len 40960
6884 055/300 Read block 2657 len 16384
6885 056/300 Write block 1484 len 4096 data: 2
6886 057/300 Write block 1532 len 32768 data: 3 3 3 3 4 3 3 3
6887 058/300 Read block 1574 len 32768
6888 059/300 Write block 911 len 16384 data: 3 3 3 3
6889 060/300 Read block 838 len 12288
6890 061/300 Write block 2563 len 4096 data: 2
6891 062/300 Write block 4349 len 4096 data: 2
6892 063/300 Read block 2558 len 16384
6893 064/300 Write block 1826 len 40960 data: 3 3 3 3 4 4 3 3 3 3
6894 065/300 Write block 3516 len 4096 data: 4
6895 066/300 Write block 3029 len 12288 data: 2 2 2
6896 067/300 Flush
6897 068/300 Write block 1864 len 20480 data: 3 3 3 2 2
6898 069/300 Write block 1432 len 12288 data: 3 4 4
6899 070/300 Write block 3907 len 12288 data: 4 4 3
6900 071/300 Write block 4154 len 28672 data: 3 3 3 2 2 3 3
6901 072/300 Write block 1453 len 36864 data: 3 2 2 2 3 3 3 3 3
6902 073/300 Write block 1559 len 32768 data: 4 4 4 3 3 2 2 2
6903 074/300 Write block 4989 len 24576 data: 2 2 2 3 3 2
6904 075/300 Read block 2166 len 28672
6905 076/300 Write block 246 len 24576 data: 3 3 3 3 3 2
6906 077/300 Read block 1337 len 12288
6907 078/300 Write block 4320 len 16384 data: 3 3 2 2
6908 079/300 Read block 276 len 36864
6909 080/300 Read block 4116 len 4096
6910 081/300 Read block 743 len 16384
6911 082/300 Read block 4305 len 8192
6912 083/300 Read block 2937 len 4096
6913 084/300 Read block 2605 len 4096
6914 085/300 Write block 1164 len 40960 data: 3 3 3 4 4 5 5 5 5 4
6915 086/300 Flush
6916 087/300 Write block 4416 len 16384 data: 2 2 3 3
6917 088/300 Read block 2034 len 4096
6918 089/300 Write block 1943 len 36864 data: 3 4 4 4 3 2 2 2 2
6919 090/300 Write block 757 len 4096 data: 4
6920 091/300 Write block 4893 len 36864 data: 3 3 3 3 3 3 3 3 3
6921 092/300 Read block 677 len 12288
6922 093/300 Flush
6923 094/300 Read block 4658 len 12288
6924 095/300 Read block 4803 len 20480
6925 096/300 Read block 2999 len 16384
6926 097/300 Read block 1357 len 16384
6927 098/300 Read block 4070 len 20480
6928 099/300 Write block 3436 len 24576 data: 6 4 4 4 4 2
6929 100/300 Read block 1240 len 8192
6930 101/300 Write block 2306 len 4096 data: 3
6931 102/300 Read block 3431 len 32768
6932 103/300 Read block 4126 len 28672
6933 104/300 Read block 3471 len 12288
6934 105/300 Write block 3461 len 32768 data: 2 4 4 4 4 3 2 2
6935 106/300 Write block 1481 len 28672 data: 2 2 2 3 2 2 2
6936 107/300 Flush
6937 108/300 Read block 2968 len 36864
6938 109/300 Read block 1432 len 40960
6939 110/300 Read block 4698 len 36864
6940 111/300 Read block 1873 len 32768
6941 112/300 Read block 132 len 12288
6942 113/300 Read block 1321 len 12288
6943 114/300 Write block 1753 len 4096 data: 5
6944 115/300 Read block 814 len 36864
6945 116/300 Flush
6946 117/300 Read block 3300 len 36864
6947 118/300 Write block 2031 len 16384 data: 4 4 6 7
6948 119/300 Write block 2672 len 32768 data: 3 4 4 3 3 4 4 5
6949 120/300 Write block 3219 len 12288 data: 3 3 3
6950 121/300 Read block 3418 len 20480
6951 122/300 Write block 4458 len 16384 data: 4 6 5 6
6952 123/300 Read block 4738 len 32768
6953 124/300 Read block 4352 len 8192
6954 125/300 Read block 2456 len 8192
6955 126/300 Flush
6956 127/300 Read block 1080 len 16384
6957 128/300 Read block 613 len 8192
6958 129/300 Read block 4234 len 28672
6959 130/300 Flush
6960 131/300 Read block 746 len 40960
6961 132/300 Read block 2693 len 12288
6962 133/300 Flush
6963 134/300 Write block 125 len 8192 data: 4 4
6964 135/300 Write block 4137 len 36864 data: 4 3 3 4 4 4 4 4 3
6965 136/300 Read block 1202 len 20480
6966 137/300 Write block 4578 len 32768 data: 2 2 2 2 3 3 2 2
6967 138/300 Read block 1298 len 8192
6968 139/300 Flush
6969 140/300 Write block 3284 len 32768 data: 4 3 3 3 3 4 4 4
6970 141/300 Read block 2426 len 8192
6971 142/300 Read block 2828 len 8192
6972 143/300 Read block 4862 len 16384
6973 144/300 Read block 2005 len 36864
6974 145/300 Read block 878 len 28672
6975 146/300 Read block 3003 len 24576
6976 147/300 Read block 1813 len 40960
6977 148/300 Read block 2711 len 16384
6978 149/300 Write block 1215 len 40960 data: 3 3 3 3 3 4 3 4 4 4
6979 150/300 Write block 2412 len 16384 data: 3 4 4 5
6980 151/300 Read block 17 len 24576
6981 152/300 Write block 2659 len 4096 data: 2
6982 153/300 Write block 3302 len 20480 data: 2 2 2 3 3
6983 154/300 Write block 2412 len 24576 data: 4 5 5 6 6 6
6984 155/300 Read block 517 len 36864
6985 156/300 Read block 1774 len 28672
6986 157/300 Read block 1882 len 24576
6987 158/300 Read block 2969 len 36864
6988 159/300 Read block 1384 len 32768
6989 160/300 Write block 4352 len 12288 data: 2 2 2
6990 161/300 Read block 992 len 20480
6991 162/300 Flush
6992 163/300 Write block 1800 len 16384 data: 4 4 4 4
6993 164/300 Write block 2859 len 20480 data: 2 2 2 2 2
6994 165/300 Write block 4104 len 16384 data: 2 2 2 2
6995 166/300 Read block 2099 len 40960
6996 167/300 Read block 2161 len 20480
6997 168/300 Write block 1934 len 32768 data: 2 2 2 2 3 3 3 3
6998 169/300 Read block 4853 len 40960
6999 170/300 Read block 698 len 16384
7000 171/300 Read block 4412 len 40960
7001 172/300 Read block 1642 len 36864
7002 173/300 Write block 2563 len 4096 data: 3
7003 174/300 Write block 440 len 8192 data: 2 2
7004 175/300 Read block 2290 len 20480
7005 176/300 Flush
7006 177/300 Flush
7007 178/300 Write block 4047 len 28672 data: 2 2 2 2 2 2 2
7008 179/300 Write block 4362 len 20480 data: 6 6 5 5 5
7009 180/300 Write block 4801 len 32768 data: 5 5 5 5 4 3 3 3
7010 181/300 Write block 139 len 24576 data: 2 2 2 2 2 2
7011 182/300 Write block 2985 len 16384 data: 3 3 3 3
7012 183/300 Read block 262 len 4096
7013 184/300 Write block 3163 len 20480 data: 3 3 3 2 2
7014 185/300 Read block 81 len 16384
7015 186/300 Read block 1063 len 20480
7016 187/300 Read block 690 len 32768
7017 188/300 Flush
7018 189/300 Write block 870 len 32768 data: 3 3 2 2 3 3 4 4
7019 190/300 Flush
7020 191/300 Read block 2437 len 20480
7021 192/300 Write block 1599 len 40960 data: 3 3 3 2 2 2 2 2 2 2
7022 193/300 Read block 1245 len 36864
7023 194/300 Write block 4267 len 28672 data: 2 2 2 2 2 2 2
7024 195/300 Write block 54 len 40960 data: 3 3 3 2 2 2 2 2 2 2
7025 196/300 Flush
7026 197/300 Read block 2235 len 40960
7027 198/300 Flush
7028 199/300 Read block 3603 len 8192
7029 200/300 Write block 4878 len 24576 data: 3 2 2 2 3 4
7030 201/300 Write block 222 len 20480 data: 3 2 2 2 2
7031 202/300 Read block 888 len 20480
7032 203/300 Write block 813 len 4096 data: 2
7033 204/300 Write block 275 len 40960 data: 3 4 4 4 4 3 2 2 2 2
7034 205/300 Write block 3907 len 4096 data: 5
7035 206/300 Write block 957 len 24576 data: 4 5 6 6 6 5
7036 207/300 Read block 2331 len 4096
7037 208/300 Write block 304 len 28672 data: 2 2 2 2 2 2 2
7038 209/300 Flush
7039 210/300 Read block 2866 len 12288
7040 211/300 Write block 3288 len 36864 data: 4 5 5 5 4 4 4 3 3
7041 212/300 Read block 2066 len 12288
7042 213/300 Write block 430 len 16384 data: 2 2 2 3
7043 214/300 Write block 2418 len 24576 data: 5 5 4 4 3 3
7044 215/300 Read block 1100 len 4096
7045 216/300 Write block 3745 len 4096 data: 5
7046 217/300 Read block 1962 len 8192
7047 218/300 Write block 3573 len 24576 data: 4 3 2 2 2 2
7048 219/300 Write block 338 len 40960 data: 3 3 2 2 3 3 3 3 3 3
7049 220/300 Read block 953 len 12288
7050 221/300 Read block 3492 len 40960
7051 222/300 Read block 4468 len 36864
7052 223/300 Read block 3688 len 20480
7053 224/300 Read block 2327 len 20480
7054 225/300 Write block 757 len 28672 data: 5 4 5 4 4 5 5
7055 226/300 Flush
7056 227/300 Read block 2043 len 24576
7057 228/300 Write block 4748 len 20480 data: 4 3 3 3 3
7058 229/300 Read block 1183 len 4096
7059 230/300 Read block 104 len 16384
7060 231/300 Write block 3211 len 40960 data: 3 4 4 4 3 3 3 3 4 4
7061 232/300 Write block 3554 len 8192 data: 3 4
7062 233/300 Read block 4301 len 36864
7063 234/300 Write block 3447 len 24576 data: 3 3 3 2 2 2
7064 235/300 Write block 717 len 20480 data: 3 3 4 3 3
7065 236/300 Write block 1483 len 28672 data: 3 4 3 3 3 2 2
7066 237/300 Write block 1385 len 20480 data: 3 3 3 3 3
7067 238/300 Read block 4182 len 8192
7068 239/300 Write block 2316 len 40960 data: 4 3 4 3 3 3 3 3 3 3
7069 240/300 Read block 2697 len 28672
7070 241/300 Write block 3161 len 28672 data: 2 2 4 4 4 3 3
7071 242/300 Read block 4874 len 20480
7072 243/300 Write block 37 len 20480 data: 2 2 2 2 3
7073 244/300 Write block 2544 len 8192 data: 3 3
7074 245/300 Read block 545 len 4096
7075 246/300 Read block 2698 len 40960
7076 247/300 Read block 2424 len 4096
7077 248/300 Write block 1838 len 8192 data: 2 2
7078 249/300 Read block 4184 len 32768
7079 250/300 Read block 2800 len 32768
7080 251/300 Flush
7081 252/300 Read block 1019 len 32768
7082 253/300 Write block 4806 len 8192 data: 4 4
7083 254/300 Write block 3729 len 8192 data: 6 5
7084 255/300 Write block 140 len 32768 data: 3 3 3 3 3 2 2 2
7085 256/300 Write block 3639 len 12288 data: 4 4 4
7086 257/300 Write block 4927 len 20480 data: 2 2 2 2 2
7087 258/300 Read block 4292 len 32768
7088 259/300 Read block 549 len 28672
7089 260/300 Read block 1801 len 8192
7090 261/300 Flush
7091 262/300 Write block 2682 len 28672 data: 5 5 5 5 4 4 4
7092 263/300 Read block 2354 len 36864
7093 264/300 Write block 83 len 24576 data: 2 2 2 2 2 2
7094 265/300 Write block 4092 len 8192 data: 5 5
7095 266/300 Read block 2517 len 32768
7096 267/300 Read block 44 len 36864
7097 268/300 Write block 1124 len 16384 data: 4 6 6 7
7098 269/300 Read block 2469 len 16384
7099 270/300 Read block 3425 len 32768
7100 271/300 Write block 2442 len 32768 data: 4 4 4 3 3 3 3 3
7101 272/300 Read block 401 len 20480
7102 273/300 Flush
7103 274/300 Write block 353 len 12288 data: 3 3 3
7104 275/300 Read block 1507 len 4096
7105 276/300 Write block 678 len 16384 data: 4 3 4 6
7106 277/300 Write block 168 len 12288 data: 4 5 4
7107 278/300 Read block 728 len 20480
7108 279/300 Flush
7109 280/300 Read block 1541 len 24576
7110 281/300 Write block 4566 len 36864 data: 3 3 3 3 3 3 3 3 2
7111 282/300 Write block 4265 len 32768 data: 3 3 3 3 3 3 3 3
7112 283/300 Read block 4861 len 32768
7113 284/300 Read block 2978 len 16384
7114 285/300 Read block 2799 len 24576
7115 286/300 Write block 4071 len 28672 data: 5 4 5 4 4 4 3
7116 287/300 Read block 1873 len 32768
7117 288/300 Write block 4870 len 16384 data: 4 3 2 2
7118 289/300 Write block 3358 len 20480 data: 4 4 4 4 4
7119 290/300 Write block 4520 len 20480 data: 3 2 2 2 2
7120 291/300 Read block 4205 len 4096
7121 292/300 Write block 3446 len 28672 data: 3 4 4 4 3 3 3
7122 293/300 Write block 3357 len 20480 data: 4 5 5 5 5
7123 294/300 Read block 4232 len 32768
7124 295/300 Read block 299 len 36864
7125 296/300 Write block 4369 len 4096 data: 3
7126 297/300 Write block 4396 len 40960 data: 2 2 2 3 3 4 4 4 4 4
7127 298/300 Read block 1707 len 40960
7128 299/300 Read block 1765 len 28672
7129 300/300 Flush
7130 Main task has work to do, go find it
7131 got dsc Start(2)
7132 start 2
7133 Feb 28 22:14:40.006 INFO request completed, latency_us: 219, response_code: 204, uri: /start/cid/2, method: POST, req_id: d1686c6c-2ee6-4a9a-89e3-68b9684cf80c, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
7134 Replay: started 2, returned:Ok(())
7135 ----------------------------------------------------------------
7136 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7137 States: Active Active Offline
7138 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
7139 2412 Acked 3411 Write 10 Done Done New true
7140 2413 Acked 3412 Read 6 Done Done New true
7141 2414 Acked 3413 Read 3 Done Done New true
7142 2415 Acked 3414 Write 3 Done Done New true
7143 2416 Acked 3415 Read 8 Done Done New true
7144 2417 Acked 3416 Read 5 Done Done New false
7145 2418 Acked 3417 Write 7 Done Done New false
7146 2419 Acked 3418 Write 2 Done Done New false
7147 2420 Acked 3419 Write 3 Done Done New false
7148 2421 Acked 3420 Flush 0 Done Done New false
7149 2422 Acked 3421 Write 6 Done Done New false
7150 2423 Acked 3422 Read 9 Done Done New false
7151 2424 Acked 3423 Write 8 Done Done New false
7152 2425 Acked 3424 Write 4 Done Done New false
7153 2426 Acked 3425 Write 10 Done Done New false
7154 2427 Acked 3426 Read 1 Done Done New false
7155 2428 Acked 3427 Flush 0 Done Done New false
7156 2429 Acked 3428 Read 3 Done Done New false
7157 2430 Acked 3429 Write 10 Done Done New false
7158 2431 Acked 3430 Flush 0 Done Done New false
7159 2432 Acked 3431 Read 7 Done Done New false
7160 2433 Acked 3432 Read 8 Done Done New false
7161 2434 Acked 3433 Read 5 Done Done New false
7162 2435 Acked 3434 Read 6 Done Done New false
7163 2436 Acked 3435 Write 2 Done Done New false
7164 2437 Acked 3436 Read 4 Done Done New false
7165 2438 Acked 3437 Write 3 Done Done New false
7166 2439 Acked 3438 Write 2 Done Done New false
7167 2440 Acked 3439 Read 5 Done Done New false
7168 2441 Acked 3440 Read 8 Done Done New false
7169 2442 Acked 3441 Write 9 Done Done New false
7170 2443 Acked 3442 Write 3 Done Done New false
7171 2444 Acked 3443 Write 1 Done Done New false
7172 2445 Acked 3444 Read 2 Done Done New false
7173 2446 Acked 3445 Write 4 Done Done New false
7174 2447 Acked 3446 Flush 0 Done Done New false
7175 2448 Acked 3447 Write 5 Done Done New false
7176 2449 Acked 3448 Write 6 Done Done New false
7177 2450 Acked 3449 Read 5 Done Done New false
7178 2451 Acked 3450 Write 3 Done Done New false
7179 2452 Acked 3451 Write 8 Done Done New false
7180 2453 Acked 3452 Write 2 Done Done New false
7181 2454 Acked 3453 Write 5 Done Done New false
7182 2455 Acked 3454 Write 3 Done Done New false
7183 2456 Acked 3455 Read 8 Done Done New false
7184 2457 Acked 3456 Read 9 Done Done New false
7185 2458 Acked 3457 Write 7 Done Done New false
7186 2459 Acked 3458 Read 9 Done Done New false
7187 2460 Acked 3459 Read 4 Done Done New false
7188 2461 Acked 3460 Write 5 Done Done New false
7189 2462 Acked 3461 Read 8 Done Done New false
7190 2463 Acked 3462 Write 7 Done Done New false
7191 2464 Acked 3463 Read 5 Done Done New false
7192 2465 Acked 3464 Read 10 Done Done New false
7193 2466 Acked 3465 Read 4 Done Done New false
7194 2467 Acked 3466 Write 1 Done Done New false
7195 2468 Acked 3467 Write 8 Done Done New false
7196 2469 Acked 3468 Read 8 Done Done New false
7197 2470 Acked 3469 Write 4 Done Done New false
7198 2471 Acked 3470 Read 3 Done Done New false
7199 2472 Acked 3471 Write 1 Done Done New false
7200 2473 Acked 3472 Write 1 Done Done New false
7201 2474 Acked 3473 Read 4 Done Done New false
7202 2475 Acked 3474 Write 10 Done Done New false
7203 2476 Acked 3475 Write 1 Done Done New false
7204 2477 Acked 3476 Write 3 Done Done New false
7205 2478 Acked 3477 Flush 0 Done Done New false
7206 2479 Acked 3478 Write 5 Done Done New false
7207 2480 Acked 3479 Write 3 Done Done New false
7208 2481 Acked 3480 Write 3 Done Done New false
7209 2482 Acked 3481 Write 7 Done Done New false
7210 2483 Acked 3482 Write 9 Done Done New false
7211 2484 Acked 3483 Write 8 Done Done New false
7212 2485 Acked 3484 Write 6 Done Done New false
7213 2486 Acked 3485 Read 7 Done Done New false
7214 2487 Acked 3486 Write 6 Done Done New false
7215 2488 Acked 3487 Read 3 Done Done New false
7216 2489 Acked 3488 Write 4 Done Done New false
7217 2490 Acked 3489 Read 9 Done Done New false
7218 2491 Acked 3490 Read 1 Done Done New false
7219 2492 Acked 3491 Read 4 Done Done New false
7220 2493 Acked 3492 Read 2 Done Done New false
7221 2494 Acked 3493 Read 1 Done Done New false
7222 2495 Acked 3494 Read 1 Done Done New false
7223 2496 Acked 3495 Write 10 Done Done New false
7224 2497 Acked 3496 Flush 0 Done Done New false
7225 2498 Acked 3497 Write 4 Done Done New false
7226 2499 Acked 3498 Read 1 Done Done New false
7227 2500 Acked 3499 Write 9 Done Done New false
7228 2501 Acked 3500 Write 1 Done Done New false
7229 2502 Acked 3501 Write 9 Done Done New false
7230 2503 Acked 3502 Read 3 Done Done New false
7231 2504 Acked 3503 Flush 0 Done Done New false
7232 2505 Acked 3504 Read 3 Done Done New false
7233 2506 Acked 3505 Read 5 Done Done New false
7234 2507 Acked 3506 Read 4 Done Done New false
7235 2508 Acked 3507 Read 4 Done Done New false
7236 2509 Acked 3508 Read 5 Done Done New false
7237 2510 Acked 3509 Write 6 Done Done New false
7238 2511 Acked 3510 Read 2 Done Done New false
7239 2512 Acked 3511 Write 1 Done Done New false
7240 2513 Acked 3512 Read 8 Done Done New false
7241 2514 Acked 3513 Read 7 Done Done New false
7242 2515 Acked 3514 Read 3 Done Done New false
7243 2516 Acked 3515 Write 8 Done Done New false
7244 2517 Acked 3516 Write 7 Done Done New false
7245 2518 Acked 3517 Flush 0 Done Done New false
7246 2519 Acked 3518 Read 9 Done Done New false
7247 2520 Acked 3519 Read 10 Done Done New false
7248 2521 Acked 3520 Read 9 Done Done New false
7249 2522 Acked 3521 Read 8 Done Done New false
7250 2523 Acked 3522 Read 3 Done Done New false
7251 2524 Acked 3523 Read 3 Done Done New false
7252 2525 Acked 3524 Write 1 Done Done New false
7253 2526 Acked 3525 Read 9 Done Done New false
7254 2527 Acked 3526 Flush 0 Done Done New false
7255 2528 Acked 3527 Read 9 Done Done New false
7256 2529 Acked 3528 Write 4 Done Done New false
7257 2530 Acked 3529 Write 8 Done Done New false
7258 2531 Acked 3530 Write 3 Done Done New false
7259 2532 Acked 3531 Read 5 Done Done New false
7260 2533 Acked 3532 Write 4 Done Done New false
7261 2534 Acked 3533 Read 8 Done Done New false
7262 2535 Acked 3534 Read 2 Done Done New false
7263 2536 Acked 3535 Read 2 Done Done New false
7264 2537 Acked 3536 Flush 0 Done Done New false
7265 2538 Acked 3537 Read 4 Done Done New false
7266 2539 Acked 3538 Read 2 Done Done New false
7267 2540 Acked 3539 Read 7 Done Done New false
7268 2541 Acked 3540 Flush 0 Done Done New false
7269 2542 Acked 3541 Read 10 Done Done New false
7270 2543 Acked 3542 Read 3 Done Done New false
7271 2544 Acked 3543 Flush 0 Done Done New false
7272 2545 Acked 3544 Write 2 Done Done New false
7273 2546 Acked 3545 Write 9 Done Done New false
7274 2547 Acked 3546 Read 5 Done Done New false
7275 2548 Acked 3547 Write 8 Done Done New false
7276 2549 Acked 3548 Read 2 Done Done New false
7277 2550 Acked 3549 Flush 0 Done Done New false
7278 2551 Acked 3550 Write 8 Done Done New false
7279 2552 Acked 3551 Read 2 Done Done New false
7280 2553 Acked 3552 Read 2 Done Done New false
7281 2554 Acked 3553 Read 4 Done Done New false
7282 2555 Acked 3554 Read 9 Done Done New false
7283 2556 Acked 3555 Read 7 Done Done New false
7284 2557 Acked 3556 Read 6 Done Done New false
7285 2558 Acked 3557 Read 10 Done Done New false
7286 2559 Acked 3558 Read 4 Done Done New false
7287 2560 Acked 3559 Write 10 Done Done New false
7288 2561 Acked 3560 Write 4 Done Done New false
7289 2562 Acked 3561 Read 6 Done Done New false
7290 2563 Acked 3562 Write 1 Done Done New false
7291 2564 Acked 3563 Write 5 Done Done New false
7292 2565 Acked 3564 Write 6 Done Done New false
7293 2566 Acked 3565 Read 9 Done Done New false
7294 2567 Acked 3566 Read 7 Done Done New false
7295 2568 Acked 3567 Read 6 Done Done New false
7296 2569 Acked 3568 Read 9 Done Done New false
7297 2570 Acked 3569 Read 8 Done Done New false
7298 2571 Acked 3570 Write 3 Done Done New false
7299 2572 Acked 3571 Read 5 Done Done New false
7300 2573 Acked 3572 Flush 0 Done Done New false
7301 2574 Acked 3573 Write 4 Done Done New false
7302 2575 Acked 3574 Write 5 Done Done New false
7303 2576 Acked 3575 Write 4 Done Done New false
7304 2577 Acked 3576 Read 10 Done Done New false
7305 2578 Acked 3577 Read 5 Done Done New false
7306 2579 Acked 3578 Write 8 Done Done New false
7307 2580 Acked 3579 Read 10 Done Done New false
7308 2581 Acked 3580 Read 4 Done Done New false
7309 2582 Acked 3581 Read 10 Done Done New false
7310 2583 Acked 3582 Read 9 Done Done New false
7311 2584 Acked 3583 Write 1 Done Done New false
7312 2585 Acked 3584 Write 2 Done Done New false
7313 2586 Acked 3585 Read 5 Done Done New false
7314 2587 Acked 3586 Flush 0 Done Done New false
7315 2588 Acked 3587 Flush 0 Done Done New false
7316 2589 Acked 3588 Write 7 Done Done New false
7317 2590 Acked 3589 Write 5 Done Done New false
7318 2591 Acked 3590 Write 8 Done Done New false
7319 2592 Acked 3591 Write 6 Done Done New false
7320 2593 Acked 3592 Write 4 Done Done New false
7321 2594 Acked 3593 Read 1 Done Done New false
7322 2595 Acked 3594 Write 5 Done Done New false
7323 2596 Acked 3595 Read 4 Done Done New false
7324 2597 Acked 3596 Read 5 Done Done New false
7325 2598 Acked 3597 Read 8 Done Done New false
7326 2599 Acked 3598 Flush 0 Done Done New false
7327 2600 Acked 3599 Write 8 Done Done New false
7328 2601 Acked 3600 Flush 0 Done Done New false
7329 2602 Acked 3601 Read 5 Done Done New false
7330 2603 Acked 3602 Write 10 Done Done New false
7331 2604 Acked 3603 Read 9 Done Done New false
7332 2605 Acked 3604 Write 7 Done Done New false
7333 2606 Acked 3605 Write 10 Done Done New false
7334 2607 Acked 3606 Flush 0 Done Done New false
7335 2608 Acked 3607 Read 10 Done Done New false
7336 2609 Acked 3608 Flush 0 Done Done New false
7337 2610 Acked 3609 Read 2 Done Done New false
7338 2611 Acked 3610 Write 6 Done Done New false
7339 2612 Acked 3611 Write 5 Done Done New false
7340 2613 Acked 3612 Read 5 Done Done New false
7341 2614 Acked 3613 Write 1 Done Done New false
7342 2615 Acked 3614 Write 10 Done Done New false
7343 2616 Acked 3615 Write 1 Done Done New false
7344 2617 Acked 3616 Write 6 Done Done New false
7345 2618 Acked 3617 Read 1 Done Done New false
7346 2619 Acked 3618 Write 7 Done Done New false
7347 2620 Acked 3619 Flush 0 Done Done New false
7348 2621 Acked 3620 Read 3 Done Done New false
7349 2622 Acked 3621 Write 9 Done Done New false
7350 2623 Acked 3622 Read 3 Done Done New false
7351 2624 Acked 3623 Write 4 Done Done New false
7352 2625 Acked 3624 Write 6 Done Done New false
7353 2626 Acked 3625 Read 1 Done Done New false
7354 2627 Acked 3626 Write 1 Done Done New false
7355 2628 Acked 3627 Read 2 Done Done New false
7356 2629 Acked 3628 Write 6 Done Done New false
7357 2630 Acked 3629 Write 10 Done Done New false
7358 2631 Acked 3630 Read 3 Done Done New false
7359 2632 Acked 3631 Read 10 Done Done New false
7360 2633 Acked 3632 Read 9 Done Done New false
7361 2634 Acked 3633 Read 5 Done Done New false
7362 2635 Acked 3634 Read 5 Done Done New false
7363 2636 Acked 3635 Write 7 Done Done New false
7364 2637 Acked 3636 Flush 0 Done Done New false
7365 2638 Acked 3637 Read 6 Done Done New false
7366 2639 Acked 3638 Write 5 Done Done New false
7367 2640 Acked 3639 Read 1 Done Done New false
7368 2641 Acked 3640 Read 4 Done Done New false
7369 2642 Acked 3641 Write 10 Done Done New false
7370 2643 Acked 3642 Write 2 Done Done New false
7371 2644 Acked 3643 Read 9 Done Done New false
7372 2645 Acked 3644 Write 6 Done Done New false
7373 2646 Acked 3645 Write 5 Done Done New false
7374 2647 Acked 3646 Write 7 Done Done New false
7375 2648 Acked 3647 Write 5 Done Done New false
7376 2649 Acked 3648 Read 2 Done Done New false
7377 2650 Acked 3649 Write 10 Done Done New false
7378 2651 Acked 3650 Read 7 Done Done New false
7379 2652 Acked 3651 Write 7 Done Done New false
7380 2653 Acked 3652 Read 5 Done Done New false
7381 2654 Acked 3653 Write 5 Done Done New false
7382 2655 Acked 3654 Write 2 Done Done New false
7383 2656 Acked 3655 Read 1 Done Done New false
7384 2657 Acked 3656 Read 10 Done Done New false
7385 2658 Acked 3657 Read 1 Done Done New false
7386 2659 Acked 3658 Write 2 Done Done New false
7387 2660 Acked 3659 Read 8 Done Done New false
7388 2661 Acked 3660 Read 8 Done Done New false
7389 2662 Acked 3661 Flush 0 Done Done New false
7390 2663 Acked 3662 Read 8 Done Done New false
7391 2664 Acked 3663 Write 2 Done Done New false
7392 2665 Acked 3664 Write 2 Done Done New false
7393 2666 Acked 3665 Write 8 Done Done New false
7394 2667 Acked 3666 Write 3 Done Done New false
7395 2668 Acked 3667 Write 5 Done Done New false
7396 2669 Acked 3668 Read 8 Done Done New false
7397 2670 Acked 3669 Read 7 Done Done New false
7398 2671 Acked 3670 Read 2 Done Done New false
7399 2672 Acked 3671 Flush 0 Done Done New false
7400 2673 Acked 3672 Write 7 Done Done New false
7401 2674 Acked 3673 Read 9 Done Done New false
7402 2675 Acked 3674 Write 6 Done Done New false
7403 2676 Acked 3675 Write 2 Done Done New false
7404 2677 Acked 3676 Read 8 Done Done New false
7405 2678 Acked 3677 Read 9 Done Done New false
7406 2679 Acked 3678 Write 4 Done Done New false
7407 2680 Acked 3679 Read 4 Done Done New false
7408 2681 Acked 3680 Read 8 Done Done New false
7409 2682 Acked 3681 Write 8 Done Done New false
7410 2683 Acked 3682 Read 5 Done Done New false
7411 2684 Acked 3683 Flush 0 Done Done New false
7412 2685 Acked 3684 Write 3 Done Done New false
7413 2686 Acked 3685 Read 1 Done Done New false
7414 2687 Acked 3686 Write 4 Done Done New false
7415 2688 Acked 3687 Write 3 Done Done New false
7416 2689 Acked 3688 Read 5 Done Done New false
7417 2690 Acked 3689 Flush 0 Done Done New false
7418 2691 Acked 3690 Read 6 Done Done New false
7419 2692 Acked 3691 Write 9 Done Done New false
7420 2693 Acked 3692 Write 8 Done Done New false
7421 2694 Acked 3693 Read 8 Done Done New false
7422 2695 Acked 3694 Read 4 Done Done New false
7423 2696 Acked 3695 Read 6 Done Done New false
7424 2697 Acked 3696 Write 7 Done Done New false
7425 2698 Acked 3697 Read 8 Done Done New false
7426 2699 Acked 3698 Write 4 Done Done New false
7427 2700 Acked 3699 Write 5 Done Done New false
7428 2701 Acked 3700 Write 5 Done Done New false
7429 2702 Acked 3701 Read 1 Done Done New false
7430 2703 Acked 3702 Flush 0 Done Done New false
7431 2704 Acked 3703 Write 7 Done Done New false
7432 2705 Acked 3704 Write 5 Done Done New false
7433 2706 Acked 3705 Read 8 Done Done New false
7434 2707 Acked 3706 Read 9 Done Done New false
7435 2708 Acked 3707 Write 1 Done Done New false
7436 2709 Acked 3708 Write 10 Done Done New false
7437 2710 Acked 3709 Read 10 Done Done New false
7438 2711 Acked 3710 Read 7 Done Done New false
7439 2712 Acked 3711 Flush 0 Done Done New false
7440 STATES DS:0 DS:1 DS:2 TOTAL
7441 New 0 0 301 301
7442 Sent 0 0 0 0
7443 Done 301 301 0 602
7444 Skipped 0 0 0 0
7445 Error 0 0 0 0
7446 Last Flush: 3711 3711 3410
7447 Downstairs last five completed: 3410 3409 3408 3407 3406
7448 Upstairs last five completed: 2712 2711 2710 2709 2708
7449 CLIENT: Up:0 ds:301 act:2
74502024-02-28T22:14:40.284ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74512024-02-28T22:14:40.284ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74522024-02-28T22:14:40.284ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74532024-02-28T22:14:40.284ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74542024-02-28T22:14:40.284ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74552024-02-28T22:14:40.284ZINFOcrucible: [2] client re-new 301 jobs since flush 3410 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74562024-02-28T22:14:40.284ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74572024-02-28T22:14:41.285ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74582024-02-28T22:14:41.285ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74592024-02-28T22:14:41.285ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74602024-02-28T22:14:41.285ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74612024-02-28T22:14:41.285ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74622024-02-28T22:14:41.285ZINFOcrucible: [2] client re-new 301 jobs since flush 3410 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74632024-02-28T22:14:41.285ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74642024-02-28T22:14:42.286ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74652024-02-28T22:14:42.286ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74662024-02-28T22:14:42.286ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74672024-02-28T22:14:42.286ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74682024-02-28T22:14:42.286ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74692024-02-28T22:14:42.286ZINFOcrucible: [2] client re-new 301 jobs since flush 3410 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74702024-02-28T22:14:42.287ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
7471 [8830] got start action
7472 [8830] I am going to restart
7473 Starting downstairs at port 8830
7474 Make output file at "/tmp/dsc/downstairs-8830.txt"
7475 [8830][2] reports Starting
74762024-02-28T22:14:43.287ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74772024-02-28T22:14:43.288ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74782024-02-28T22:14:43.288ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74792024-02-28T22:14:43.288ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74802024-02-28T22:14:43.288ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74812024-02-28T22:14:43.288ZINFOcrucible: [2] client re-new 301 jobs since flush 3410 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
74822024-02-28T22:14:43.288ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
7483 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1173)
7484 [8830][2] reports Running
7485 ----------------------------------------------------------------
7486 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7487 States: Active Active Offline
7488 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
7489 2412 Acked 3411 Write 10 Done Done New true
7490 2413 Acked 3412 Read 6 Done Done New true
7491 2414 Acked 3413 Read 3 Done Done New true
7492 2415 Acked 3414 Write 3 Done Done New true
7493 2416 Acked 3415 Read 8 Done Done New true
7494 2417 Acked 3416 Read 5 Done Done New true
7495 2418 Acked 3417 Write 7 Done Done New true
7496 2419 Acked 3418 Write 2 Done Done New true
7497 2420 Acked 3419 Write 3 Done Done New true
7498 2421 Acked 3420 Flush 0 Done Done New true
7499 2422 Acked 3421 Write 6 Done Done New true
7500 2423 Acked 3422 Read 9 Done Done New true
7501 2424 Acked 3423 Write 8 Done Done New true
7502 2425 Acked 3424 Write 4 Done Done New true
7503 2426 Acked 3425 Write 10 Done Done New true
7504 2427 Acked 3426 Read 1 Done Done New true
7505 2428 Acked 3427 Flush 0 Done Done New true
7506 2429 Acked 3428 Read 3 Done Done New true
7507 2430 Acked 3429 Write 10 Done Done New true
7508 2431 Acked 3430 Flush 0 Done Done New true
7509 2432 Acked 3431 Read 7 Done Done New true
7510 2433 Acked 3432 Read 8 Done Done New true
7511 2434 Acked 3433 Read 5 Done Done New true
7512 2435 Acked 3434 Read 6 Done Done New true
7513 2436 Acked 3435 Write 2 Done Done New true
7514 2437 Acked 3436 Read 4 Done Done New true
7515 2438 Acked 3437 Write 3 Done Done New true
7516 2439 Acked 3438 Write 2 Done Done New true
7517 2440 Acked 3439 Read 5 Done Done New true
7518 2441 Acked 3440 Read 8 Done Done New true
7519 2442 Acked 3441 Write 9 Done Done New true
7520 2443 Acked 3442 Write 3 Done Done New true
7521 2444 Acked 3443 Write 1 Done Done New true
7522 2445 Acked 3444 Read 2 Done Done New true
7523 2446 Acked 3445 Write 4 Done Done New true
7524 2447 Acked 3446 Flush 0 Done Done New true
7525 2448 Acked 3447 Write 5 Done Done New true
7526 2449 Acked 3448 Write 6 Done Done New true
7527 2450 Acked 3449 Read 5 Done Done New true
7528 2451 Acked 3450 Write 3 Done Done New true
7529 2452 Acked 3451 Write 8 Done Done New true
7530 2453 Acked 3452 Write 2 Done Done New true
7531 2454 Acked 3453 Write 5 Done Done New true
7532 2455 Acked 3454 Write 3 Done Done New true
7533 2456 Acked 3455 Read 8 Done Done New true
7534 2457 Acked 3456 Read 9 Done Done New true
7535 2458 Acked 3457 Write 7 Done Done New true
7536 2459 Acked 3458 Read 9 Done Done New true
7537 2460 Acked 3459 Read 4 Done Done New true
7538 2461 Acked 3460 Write 5 Done Done New true
7539 2462 Acked 3461 Read 8 Done Done New true
7540 2463 Acked 3462 Write 7 Done Done New true
7541 2464 Acked 3463 Read 5 Done Done New true
7542 2465 Acked 3464 Read 10 Done Done New true
7543 2466 Acked 3465 Read 4 Done Done New true
7544 2467 Acked 3466 Write 1 Done Done New true
7545 2468 Acked 3467 Write 8 Done Done New true
7546 2469 Acked 3468 Read 8 Done Done New true
7547 2470 Acked 3469 Write 4 Done Done New true
7548 2471 Acked 3470 Read 3 Done Done New true
7549 2472 Acked 3471 Write 1 Done Done New true
7550 2473 Acked 3472 Write 1 Done Done New true
7551 2474 Acked 3473 Read 4 Done Done New true
7552 2475 Acked 3474 Write 10 Done Done New true
7553 2476 Acked 3475 Write 1 Done Done New true
7554 2477 Acked 3476 Write 3 Done Done New true
7555 2478 Acked 3477 Flush 0 Done Done New true
7556 2479 Acked 3478 Write 5 Done Done New true
7557 2480 Acked 3479 Write 3 Done Done New true
7558 2481 Acked 3480 Write 3 Done Done New true
7559 2482 Acked 3481 Write 7 Done Done New true
7560 2483 Acked 3482 Write 9 Done Done New true
7561 2484 Acked 3483 Write 8 Done Done New true
7562 2485 Acked 3484 Write 6 Done Done New true
7563 2486 Acked 3485 Read 7 Done Done New true
7564 2487 Acked 3486 Write 6 Done Done New true
7565 2488 Acked 3487 Read 3 Done Done New true
7566 2489 Acked 3488 Write 4 Done Done New true
7567 2490 Acked 3489 Read 9 Done Done New true
7568 2491 Acked 3490 Read 1 Done Done New true
7569 2492 Acked 3491 Read 4 Done Done New true
7570 2493 Acked 3492 Read 2 Done Done New true
7571 2494 Acked 3493 Read 1 Done Done New true
7572 2495 Acked 3494 Read 1 Done Done New true
7573 2496 Acked 3495 Write 10 Done Done New true
7574 2497 Acked 3496 Flush 0 Done Done New true
7575 2498 Acked 3497 Write 4 Done Done New true
7576 2499 Acked 3498 Read 1 Done Done New true
7577 2500 Acked 3499 Write 9 Done Done New true
7578 2501 Acked 3500 Write 1 Done Done New true
7579 2502 Acked 3501 Write 9 Done Done New true
7580 2503 Acked 3502 Read 3 Done Done New true
7581 2504 Acked 3503 Flush 0 Done Done New true
7582 2505 Acked 3504 Read 3 Done Done New true
7583 2506 Acked 3505 Read 5 Done Done New true
7584 2507 Acked 3506 Read 4 Done Done New true
7585 2508 Acked 3507 Read 4 Done Done New true
7586 2509 Acked 3508 Read 5 Done Done New true
7587 2510 Acked 3509 Write 6 Done Done New true
7588 2511 Acked 3510 Read 2 Done Done New true
7589 2512 Acked 3511 Write 1 Done Done New true
7590 2513 Acked 3512 Read 8 Done Done New true
7591 2514 Acked 3513 Read 7 Done Done New true
7592 2515 Acked 3514 Read 3 Done Done New true
7593 2516 Acked 3515 Write 8 Done Done New true
7594 2517 Acked 3516 Write 7 Done Done New true
7595 2518 Acked 3517 Flush 0 Done Done New true
7596 2519 Acked 3518 Read 9 Done Done New true
7597 2520 Acked 3519 Read 10 Done Done New true
7598 2521 Acked 3520 Read 9 Done Done New true
7599 2522 Acked 3521 Read 8 Done Done New true
7600 2523 Acked 3522 Read 3 Done Done New true
7601 2524 Acked 3523 Read 3 Done Done New true
7602 2525 Acked 3524 Write 1 Done Done New true
7603 2526 Acked 3525 Read 9 Done Done New true
7604 2527 Acked 3526 Flush 0 Done Done New true
7605 2528 Acked 3527 Read 9 Done Done New true
7606 2529 Acked 3528 Write 4 Done Done New true
7607 2530 Acked 3529 Write 8 Done Done New true
7608 2531 Acked 3530 Write 3 Done Done New true
7609 2532 Acked 3531 Read 5 Done Done New true
7610 2533 Acked 3532 Write 4 Done Done New true
7611 2534 Acked 3533 Read 8 Done Done New true
7612 2535 Acked 3534 Read 2 Done Done New true
7613 2536 Acked 3535 Read 2 Done Done New true
7614 2537 Acked 3536 Flush 0 Done Done New true
7615 2538 Acked 3537 Read 4 Done Done New true
7616 2539 Acked 3538 Read 2 Done Done New true
7617 2540 Acked 3539 Read 7 Done Done New true
7618 2541 Acked 3540 Flush 0 Done Done New true
7619 2542 Acked 3541 Read 10 Done Done New true
7620 2543 Acked 3542 Read 3 Done Done New true
7621 2544 Acked 3543 Flush 0 Done Done New true
7622 2545 Acked 3544 Write 2 Done Done New true
7623 2546 Acked 3545 Write 9 Done Done New true
7624 2547 Acked 3546 Read 5 Done Done New true
7625 2548 Acked 3547 Write 8 Done Done New true
7626 2549 Acked 3548 Read 2 Done Done New true
7627 2550 Acked 3549 Flush 0 Done Done New true
7628 2551 Acked 3550 Write 8 Done Done New true
7629 2552 Acked 3551 Read 2 Done Done New true
7630 2553 Acked 3552 Read 2 Done Done New true
7631 2554 Acked 3553 Read 4 Done Done New true
7632 2555 Acked 3554 Read 9 Done Done New true
7633 2556 Acked 3555 Read 7 Done Done New true
7634 2557 Acked 3556 Read 6 Done Done New true
7635 2558 Acked 3557 Read 10 Done Done New true
7636 2559 Acked 3558 Read 4 Done Done New true
7637 2560 Acked 3559 Write 10 Done Done New true
7638 2561 Acked 3560 Write 4 Done Done New true
7639 2562 Acked 3561 Read 6 Done Done New true
7640 2563 Acked 3562 Write 1 Done Done New true
7641 2564 Acked 3563 Write 5 Done Done New true
7642 2565 Acked 3564 Write 6 Done Done New true
7643 2566 Acked 3565 Read 9 Done Done New true
7644 2567 Acked 3566 Read 7 Done Done New true
7645 2568 Acked 3567 Read 6 Done Done New true
7646 2569 Acked 3568 Read 9 Done Done New true
7647 2570 Acked 3569 Read 8 Done Done New true
7648 2571 Acked 3570 Write 3 Done Done New true
7649 2572 Acked 3571 Read 5 Done Done New true
7650 2573 Acked 3572 Flush 0 Done Done New true
7651 2574 Acked 3573 Write 4 Done Done New true
7652 2575 Acked 3574 Write 5 Done Done New true
7653 2576 Acked 3575 Write 4 Done Done New true
7654 2577 Acked 3576 Read 10 Done Done New true
7655 2578 Acked 3577 Read 5 Done Done New true
7656 2579 Acked 3578 Write 8 Done Done New true
7657 2580 Acked 3579 Read 10 Done Done New true
7658 2581 Acked 3580 Read 4 Done Done New true
7659 2582 Acked 3581 Read 10 Done Done New true
7660 2583 Acked 3582 Read 9 Done Done New true
7661 2584 Acked 3583 Write 1 Done Done New true
7662 2585 Acked 3584 Write 2 Done Done New true
7663 2586 Acked 3585 Read 5 Done Done New true
7664 2587 Acked 3586 Flush 0 Done Done New true
7665 2588 Acked 3587 Flush 0 Done Done New true
7666 2589 Acked 3588 Write 7 Done Done New true
7667 2590 Acked 3589 Write 5 Done Done New true
7668 2591 Acked 3590 Write 8 Done Done New true
7669 2592 Acked 3591 Write 6 Done Done New true
7670 2593 Acked 3592 Write 4 Done Done New true
7671 2594 Acked 3593 Read 1 Done Done New true
7672 2595 Acked 3594 Write 5 Done Done New true
7673 2596 Acked 3595 Read 4 Done Done New true
7674 2597 Acked 3596 Read 5 Done Done New true
7675 2598 Acked 3597 Read 8 Done Done New true
7676 2599 Acked 3598 Flush 0 Done Done New true
7677 2600 Acked 3599 Write 8 Done Done New true
7678 2601 Acked 3600 Flush 0 Done Done New true
7679 2602 Acked 3601 Read 5 Done Done New true
7680 2603 Acked 3602 Write 10 Done Done New true
7681 2604 Acked 3603 Read 9 Done Done New true
7682 2605 Acked 3604 Write 7 Done Done New true
7683 2606 Acked 3605 Write 10 Done Done New true
7684 2607 Acked 3606 Flush 0 Done Done New true
7685 2608 Acked 3607 Read 10 Done Done New true
7686 2609 Acked 3608 Flush 0 Done Done New true
7687 2610 Acked 3609 Read 2 Done Done New true
7688 2611 Acked 3610 Write 6 Done Done New true
7689 2612 Acked 3611 Write 5 Done Done New true
7690 2613 Acked 3612 Read 5 Done Done New true
7691 2614 Acked 3613 Write 1 Done Done New true
7692 2615 Acked 3614 Write 10 Done Done New true
7693 2616 Acked 3615 Write 1 Done Done New true
7694 2617 Acked 3616 Write 6 Done Done New true
7695 2618 Acked 3617 Read 1 Done Done New true
7696 2619 Acked 3618 Write 7 Done Done New true
7697 2620 Acked 3619 Flush 0 Done Done New true
7698 2621 Acked 3620 Read 3 Done Done New true
7699 2622 Acked 3621 Write 9 Done Done New true
7700 2623 Acked 3622 Read 3 Done Done New true
7701 2624 Acked 3623 Write 4 Done Done New true
7702 2625 Acked 3624 Write 6 Done Done New true
7703 2626 Acked 3625 Read 1 Done Done New true
7704 2627 Acked 3626 Write 1 Done Done New true
7705 2628 Acked 3627 Read 2 Done Done New true
7706 2629 Acked 3628 Write 6 Done Done New true
7707 2630 Acked 3629 Write 10 Done Done New true
7708 2631 Acked 3630 Read 3 Done Done New true
7709 2632 Acked 3631 Read 10 Done Done New true
7710 2633 Acked 3632 Read 9 Done Done New true
7711 2634 Acked 3633 Read 5 Done Done New true
7712 2635 Acked 3634 Read 5 Done Done New true
7713 2636 Acked 3635 Write 7 Done Done New true
7714 2637 Acked 3636 Flush 0 Done Done New true
7715 2638 Acked 3637 Read 6 Done Done New true
7716 2639 Acked 3638 Write 5 Done Done New true
7717 2640 Acked 3639 Read 1 Done Done New true
7718 2641 Acked 3640 Read 4 Done Done New true
7719 2642 Acked 3641 Write 10 Done Done New true
7720 2643 Acked 3642 Write 2 Done Done New true
7721 2644 Acked 3643 Read 9 Done Done New true
7722 2645 Acked 3644 Write 6 Done Done New true
7723 2646 Acked 3645 Write 5 Done Done New true
7724 2647 Acked 3646 Write 7 Done Done New true
7725 2648 Acked 3647 Write 5 Done Done New true
7726 2649 Acked 3648 Read 2 Done Done New true
7727 2650 Acked 3649 Write 10 Done Done New true
7728 2651 Acked 3650 Read 7 Done Done New true
7729 2652 Acked 3651 Write 7 Done Done New true
7730 2653 Acked 3652 Read 5 Done Done New true
7731 2654 Acked 3653 Write 5 Done Done New true
7732 2655 Acked 3654 Write 2 Done Done New true
7733 2656 Acked 3655 Read 1 Done Done New true
7734 2657 Acked 3656 Read 10 Done Done New true
7735 2658 Acked 3657 Read 1 Done Done New true
7736 2659 Acked 3658 Write 2 Done Done New true
7737 2660 Acked 3659 Read 8 Done Done New true
7738 2661 Acked 3660 Read 8 Done Done New true
7739 2662 Acked 3661 Flush 0 Done Done New true
7740 2663 Acked 3662 Read 8 Done Done New true
7741 2664 Acked 3663 Write 2 Done Done New true
7742 2665 Acked 3664 Write 2 Done Done New true
7743 2666 Acked 3665 Write 8 Done Done New true
7744 2667 Acked 3666 Write 3 Done Done New true
7745 2668 Acked 3667 Write 5 Done Done New true
7746 2669 Acked 3668 Read 8 Done Done New true
7747 2670 Acked 3669 Read 7 Done Done New true
7748 2671 Acked 3670 Read 2 Done Done New true
7749 2672 Acked 3671 Flush 0 Done Done New true
7750 2673 Acked 3672 Write 7 Done Done New true
7751 2674 Acked 3673 Read 9 Done Done New true
7752 2675 Acked 3674 Write 6 Done Done New true
7753 2676 Acked 3675 Write 2 Done Done New true
7754 2677 Acked 3676 Read 8 Done Done New true
7755 2678 Acked 3677 Read 9 Done Done New true
7756 2679 Acked 3678 Write 4 Done Done New true
7757 2680 Acked 3679 Read 4 Done Done New true
7758 2681 Acked 3680 Read 8 Done Done New true
7759 2682 Acked 3681 Write 8 Done Done New true
7760 2683 Acked 3682 Read 5 Done Done New true
7761 2684 Acked 3683 Flush 0 Done Done New true
7762 2685 Acked 3684 Write 3 Done Done New true
7763 2686 Acked 3685 Read 1 Done Done New true
7764 2687 Acked 3686 Write 4 Done Done New true
7765 2688 Acked 3687 Write 3 Done Done New true
7766 2689 Acked 3688 Read 5 Done Done New true
7767 2690 Acked 3689 Flush 0 Done Done New true
7768 2691 Acked 3690 Read 6 Done Done New true
7769 2692 Acked 3691 Write 9 Done Done New true
7770 2693 Acked 3692 Write 8 Done Done New true
7771 2694 Acked 3693 Read 8 Done Done New true
7772 2695 Acked 3694 Read 4 Done Done New true
7773 2696 Acked 3695 Read 6 Done Done New true
7774 2697 Acked 3696 Write 7 Done Done New true
7775 2698 Acked 3697 Read 8 Done Done New true
7776 2699 Acked 3698 Write 4 Done Done New true
7777 2700 Acked 3699 Write 5 Done Done New true
7778 2701 Acked 3700 Write 5 Done Done New true
7779 2702 Acked 3701 Read 1 Done Done New true
7780 2703 Acked 3702 Flush 0 Done Done New true
7781 2704 Acked 3703 Write 7 Done Done New true
7782 2705 Acked 3704 Write 5 Done Done New true
7783 2706 Acked 3705 Read 8 Done Done New true
7784 2707 Acked 3706 Read 9 Done Done New true
7785 2708 Acked 3707 Write 1 Done Done New true
7786 2709 Acked 3708 Write 10 Done Done New true
7787 2710 Acked 3709 Read 10 Done Done New true
7788 2711 Acked 3710 Read 7 Done Done New true
7789 2712 Acked 3711 Flush 0 Done Done New true
7790 STATES DS:0 DS:1 DS:2 TOTAL
7791 New 0 0 301 301
7792 Sent 0 0 0 0
7793 Done 301 301 0 602
7794 Skipped 0 0 0 0
7795 Error 0 0 0 0
7796 Last Flush: 3711 3711 3410
7797 Downstairs last five completed: 3410 3409 3408 3407 3406
7798 Upstairs last five completed: 2712 2711 2710 2709 2708
7799 CLIENT: Up:0 ds:301 act:2
78002024-02-28T22:14:44.289ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78012024-02-28T22:14:44.289ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78022024-02-28T22:14:44.290ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78032024-02-28T22:14:44.290ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78042024-02-28T22:14:44.290ZINFOcrucible: send last flush ID to this DS: 3410 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78052024-02-28T22:14:44.291ZINFOcrucible: Replied this last flush ID: 3410 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78062024-02-28T22:14:44.291ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78072024-02-28T22:14:44.291ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78082024-02-28T22:14:44.291ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78092024-02-28T22:14:44.291ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
7810 ----------------------------------------------------------------
7811 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
7812 Downstairs last five completed: 3711 3710 3709 3708 3707
7813 Upstairs last five completed: 2712 2711 2710 2709 2708
7814 CLIENT: Up:0 ds:0 act:3
7815 Replay: All jobs finished, all DS active.
7816 Main task has work to do, go find it
7817 got dsc Stop(2)
7818 stop 2
7819 Feb 28 22:14:48.013 INFO request completed, latency_us: 321, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 0367f966-8376-4a94-a5b5-3af7403d78ee, remote_addr: 127.0.0.1:36617, local_addr: 127.0[8830] Got stop action so:false kr:false
7820 .0.1:9998
78212024-02-28T22:14:48.014ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
7822 Feb 28 22:14:48.014 INFO request completed, latency_us: 113, response_code: 200, uri: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-28T22:14:48.01430955Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
7823 /state/cid/2, method: GET, req_id: 0f7a7cca-0a96-4a84-a238-6e513cd86363, remote_addr: 127.0.0.1:36617, local_addr: 127.0.{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-28T22:14:48.014344759Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
7824 0.1:9998
78252024-02-28T22:14:48.014ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78262024-02-28T22:14:48.014ZINFOcrucible: [2] client re-new 0 jobs since flush 3711 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78272024-02-28T22:14:48.014ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
7828 [8830] Exited with: ExitStatus(unix_wait_status(9))
7829 [8830][2] reports Exit
78302024-02-28T22:14:49.015ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78312024-02-28T22:14:49.015ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78322024-02-28T22:14:49.015ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78332024-02-28T22:14:49.015ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78342024-02-28T22:14:49.015ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78352024-02-28T22:14:49.015ZINFOcrucible: [2] client re-new 0 jobs since flush 3711 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78362024-02-28T22:14:49.015ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78372024-02-28T22:14:50.016ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78382024-02-28T22:14:50.016ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78392024-02-28T22:14:50.016ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78402024-02-28T22:14:50.016ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78412024-02-28T22:14:50.016ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78422024-02-28T22:14:50.016ZINFOcrucible: [2] client re-new 0 jobs since flush 3711 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78432024-02-28T22:14:50.016ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78442024-02-28T22:14:51.017ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78452024-02-28T22:14:51.017ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78462024-02-28T22:14:51.017ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78472024-02-28T22:14:51.017ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78482024-02-28T22:14:51.017ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78492024-02-28T22:14:51.017ZINFOcrucible: [2] client re-new 0 jobs since flush 3711 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78502024-02-28T22:14:51.017ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
7851 Feb 28 22:14:52.015 INFO request completed, latency_us: 199, response_code: 200, uri: /state/cid/2, method: GET, req_id: b9718991-f2f7-47b1-8ea4-72eb16e0671d, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
7852 001/300 Write block 2316 len 12288 data: 5 4 5
7853 002/300 Write block 3424 len 8192 data: 3 3
7854 003/300 Flush
78552024-02-28T22:14:52.018ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78562024-02-28T22:14:52.018ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78572024-02-28T22:14:52.018ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78582024-02-28T22:14:52.018ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78592024-02-28T22:14:52.018ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78602024-02-28T22:14:52.018ZINFOcrucible: [2] client re-new 3 jobs since flush 3711 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
78612024-02-28T22:14:52.018ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
7862 004/300 Flush
7863 005/300 Write block 1944 len 8192 data: 5 5
7864 006/300 Write block 992 len 40960 data: 3 2 2 2 2 3 3 4 4 4
7865 007/300 Flush
7866 008/300 Read block 4724 len 28672
7867 009/300 Flush
7868 010/300 Read block 3289 len 40960
7869 011/300 Write block 293 len 40960 data: 3 5 5 5 4 4 3 2 2 2
7870 012/300 Write block 1012 len 16384 data: 3 3 3 3
7871 013/300 Read block 893 len 12288
7872 014/300 Read block 4446 len 28672
7873 015/300 Flush
7874 016/300 Read block 4748 len 40960
7875 017/300 Flush
7876 018/300 Read block 998 len 36864
7877 019/300 Write block 3444 len 8192 data: 3 2
7878 020/300 Write block 4826 len 24576 data: 2 3 3 4 4 4
7879 021/300 Flush
7880 022/300 Write block 3043 len 16384 data: 4 4 3 3
7881 023/300 Read block 3039 len 12288
7882 024/300 Read block 1039 len 32768
7883 025/300 Read block 2583 len 32768
7884 026/300 Read block 1224 len 20480
7885 027/300 Write block 4102 len 12288 data: 2 2 3
7886 028/300 Read block 3663 len 24576
7887 029/300 Flush
7888 030/300 Write block 4209 len 8192 data: 2 2
7889 031/300 Flush
7890 032/300 Read block 4834 len 8192
7891 033/300 Write block 202 len 20480 data: 4 4 4 4 4
7892 034/300 Flush
7893 035/300 Read block 506 len 8192
7894 036/300 Flush
7895 037/300 Read block 1432 len 12288
7896 038/300 Read block 3561 len 4096
7897 039/300 Write block 4167 len 32768 data: 4 4 3 3 3 3 2 2
7898 040/300 Read block 554 len 4096
7899 041/300 Read block 3171 len 12288
7900 042/300 Write block 4366 len 8192 data: 6 4
7901 043/300 Read block 2759 len 4096
7902 044/300 Write block 4254 len 40960 data: 5 4 4 5 5 4 3 4 4 4
7903 045/300 Write block 498 len 12288 data: 4 3 3
7904 046/300 Read block 1671 len 28672
7905 047/300 Read block 3681 len 8192
7906 048/300 Read block 4160 len 40960
7907 049/300 Flush
7908 050/300 Read block 411 len 24576
7909 051/300 Read block 3136 len 8192
7910 052/300 Read block 1229 len 20480
7911 053/300 Write block 1603 len 28672 data: 3 3 3 3 3 3 2
7912 054/300 Read block 3727 len 20480
7913 055/300 Read block 1560 len 16384
7914 056/300 Read block 4036 len 28672
7915 057/300 Write block 4272 len 12288 data: 4 3 2
7916 058/300 Read block 1018 len 4096
7917 059/300 Write block 4217 len 4096 data: 2
7918 060/300 Write block 1708 len 40960 data: 2 2 3 4 5 5 5 6 7 6
7919 061/300 Write block 3096 len 4096 data: 3
7920 062/300 Write block 1985 len 28672 data: 2 2 2 2 3 3 4
7921 063/300 Write block 952 len 32768 data: 5 4 4 4 4 5 6 7
7922 064/300 Write block 132 len 40960 data: 4 3 5 3 3 3 2 3 4 4
7923 065/300 Write block 1751 len 4096 data: 5
7924 066/300 Write block 4917 len 40960 data: 2 2 2 3 3 3 3 2 2 2
7925 067/300 Read block 4782 len 4096
7926 068/300 Write block 4426 len 16384 data: 3 3 2 2
7927 069/300 Write block 440 len 12288 data: 3 3 3
7928 070/300 Read block 697 len 32768
7929 071/300 Read block 683 len 20480
7930 072/300 Write block 422 len 24576 data: 3 5 5 5 4 3
7931 073/300 Flush
7932 074/300 Read block 4219 len 28672
7933 075/300 Read block 1972 len 20480
7934 076/300 Flush
7935 077/300 Write block 3934 len 36864 data: 2 2 2 2 2 2 3 3 4
7936 078/300 Read block 805 len 36864
7937 079/300 Write block 3129 len 20480 data: 3 3 3 3 3
7938 080/300 Write block 593 len 28672 data: 3 3 3 3 3 3 4
7939 081/300 Write block 1714 len 4096 data: 6
7940 082/300 Write block 1891 len 28672 data: 4 4 4 4 3 3 3
7941 083/300 Write block 3343 len 12288 data: 2 2 2
7942 084/300 Read block 3252 len 28672
7943 085/300 Write block 4505 len 16384 data: 4 4 4 3
7944 086/300 Write block 3973 len 4096 data: 3
7945 087/300 Read block 1261 len 40960
7946 088/300 Write block 1302 len 4096 data: 2
7947 089/300 Write block 819 len 28672 data: 2 2 3 3 3 3 3
7948 090/300 Read block 4133 len 8192
7949 091/300 Read block 2233 len 40960
7950 092/300 Read block 289 len 36864
7951 093/300 Write block 4651 len 40960 data: 2 2 2 3 3 4 3 3 3 3
7952 094/300 Read block 2969 len 36864
7953 095/300 Write block 1580 len 12288 data: 6 6 6
7954 096/300 Read block 3320 len 24576
7955 097/300 Read block 2263 len 32768
7956 098/300 Read block 2602 len 20480
7957 099/300 Read block 3535 len 4096
7958 100/300 Write block 2201 len 8192 data: 3 3
7959 101/300 Write block 117 len 8192 data: 4 4
7960 102/300 Write block 4858 len 12288 data: 3 3 3
7961 103/300 Read block 4186 len 24576
7962 104/300 Write block 386 len 16384 data: 2 2 2 3
7963 105/300 Write block 1456 len 8192 data: 3 4
7964 106/300 Read block 1613 len 40960
7965 107/300 Read block 4226 len 40960
7966 108/300 Read block 4315 len 40960
7967 109/300 Flush
7968 110/300 Read block 2807 len 12288
7969 111/300 Write block 2969 len 12288 data: 3 3 5
7970 112/300 Write block 1325 len 8192 data: 3 3
7971 113/300 Read block 868 len 20480
7972 114/300 Write block 4381 len 32768 data: 2 2 2 2 2 2 2 2
7973 115/300 Read block 4296 len 16384
7974 116/300 Write block 2144 len 24576 data: 3 3 3 3 3 3
7975 117/300 Read block 4102 len 28672
7976 118/300 Write block 1546 len 24576 data: 3 3 3 4 3 3
7977 119/300 Write block 3155 len 40960 data: 4 3 3 2 2 2 3 3 5 5
7978 120/300 Write block 4275 len 4096 data: 2
7979 121/300 Write block 3849 len 36864 data: 3 3 3 2 2 3 4 3 4
7980 122/300 Read block 1178 len 4096
7981 123/300 Read block 794 len 4096
7982 124/300 Read block 2452 len 16384
7983 125/300 Write block 1936 len 4096 data: 3
7984 126/300 Write block 1836 len 40960 data: 4 2 3 3 2 2 2 2 2 3
7985 127/300 Read block 200 len 12288
7986 128/300 Read block 4052 len 8192
7987 129/300 Read block 1746 len 4096
7988 130/300 Write block 2748 len 16384 data: 2 2 2 2
7989 131/300 Write block 3035 len 12288 data: 3 3 3
7990 132/300 Write block 3890 len 40960 data: 3 4 4 3 4 4 2 2 3 4
7991 133/300 Write block 57 len 40960 data: 3 3 3 3 3 3 3 2 2 2
7992 134/300 Read block 1606 len 28672
7993 135/300 Write block 1748 len 4096 data: 4
7994 136/300 Flush
7995 137/300 Read block 927 len 32768
7996 138/300 Read block 2919 len 24576
7997 139/300 Read block 3937 len 36864
7998 140/300 Read block 961 len 4096
7999 141/300 Write block 3148 len 24576 data: 5 4 4 5 5 4
8000 142/300 Flush
8001 143/300 Write block 1158 len 36864 data: 3 2 2 2 2 2 4 4 4
8002 144/300 Write block 4167 len 12288 data: 5 5 4
8003 145/300 Write block 1387 len 40960 data: 4 4 4 3 3 4 4 4 3 3
8004 146/300 Flush
8005 147/300 Write block 3467 len 24576 data: 3 3 2 3 2 2
8006 148/300 Flush
8007 149/300 Read block 2872 len 36864
8008 150/300 Flush
8009 151/300 Write block 1419 len 24576 data: 3 3 3 3 4 4
8010 152/300 Read block 3869 len 28672
8011 153/300 Write block 1594 len 40960 data: 3 3 2 2 2 4 4 4 3 4
8012 154/300 Flush
8013 155/300 Read block 3216 len 24576
8014 156/300 Write block 313 len 24576 data: 3 3 3 3 4 3
8015 157/300 Write block 524 len 32768 data: 3 2 2 2 2 2 2 3
8016 158/300 Read block 2131 len 20480
8017 159/300 Read block 1430 len 28672
8018 160/300 Read block 1869 len 12288
8019 161/300 Flush
8020 162/300 Write block 84 len 12288 data: 3 3 3
8021 163/300 Read block 4446 len 24576
8022 164/300 Read block 340 len 28672
8023 165/300 Read block 1922 len 20480
8024 166/300 Read block 1965 len 12288
8025 167/300 Read block 1900 len 20480
8026 168/300 Read block 3261 len 24576
8027 169/300 Read block 638 len 12288
8028 170/300 Read block 2781 len 32768
8029 171/300 Read block 4896 len 36864
8030 172/300 Read block 240 len 20480
8031 173/300 Write block 2527 len 16384 data: 3 3 3 2
8032 174/300 Write block 4563 len 4096 data: 3
8033 175/300 Read block 1304 len 4096
8034 176/300 Write block 4098 len 28672 data: 2 2 2 2 3 3 4
8035 177/300 Write block 4929 len 16384 data: 3 3 3 2
8036 178/300 Write block 2202 len 12288 data: 4 3 3
8037 179/300 Write block 2053 len 24576 data: 6 5 5 4 3 2
8038 180/300 Write block 4959 len 40960 data: 3 3 3 2 2 2 2 2 2 2
8039 181/300 Write block 4545 len 20480 data: 2 2 3 4 4
8040 182/300 Read block 3452 len 8192
8041 183/300 Write block 4496 len 16384 data: 6 5 6 8
8042 184/300 Write block 2394 len 36864 data: 3 4 4 4 3 3 3 3 4
8043 185/300 Read block 318 len 32768
8044 186/300 Write block 505 len 36864 data: 3 4 4 5 5 4 4 5 4
8045 187/300 Read block 2305 len 36864
8046 188/300 Read block 4880 len 8192
8047 189/300 Flush
8048 190/300 Flush
8049 191/300 Write block 2642 len 4096 data: 2
8050 192/300 Read block 1084 len 36864
8051 193/300 Read block 3926 len 8192
8052 194/300 Read block 3275 len 28672
8053 195/300 Write block 472 len 36864 data: 4 3 4 4 4 3 3 3 3
8054 196/300 Read block 3411 len 24576
8055 197/300 Flush
8056 198/300 Read block 2640 len 4096
8057 199/300 Flush
8058 200/300 Read block 2147 len 12288
8059 201/300 Flush
8060 202/300 Write block 2031 len 4096 data: 5
8061 203/300 Flush
8062 204/300 Read block 1892 len 36864
8063 205/300 Flush
8064 206/300 Read block 791 len 28672
8065 207/300 Read block 2448 len 28672
8066 208/300 Read block 2707 len 32768
8067 209/300 Write block 411 len 40960 data: 4 4 3 3 3 3 3 3 3 4
8068 210/300 Read block 4812 len 20480
8069 211/300 Read block 3777 len 36864
8070 212/300 Read block 1988 len 20480
8071 213/300 Read block 697 len 28672
8072 214/300 Read block 1347 len 28672
8073 215/300 Read block 1656 len 24576
8074 216/300 Read block 353 len 8192
8075 217/300 Write block 1190 len 28672 data: 3 4 4 3 3 4 4
8076 218/300 Read block 3515 len 36864
8077 219/300 Write block 68 len 28672 data: 2 2 2 2 2 2 2
8078 220/300 Write block 1978 len 20480 data: 4 4 2 3 3
8079 221/300 Read block 1149 len 12288
8080 222/300 Read block 323 len 20480
8081 223/300 Read block 100 len 28672
8082 224/300 Write block 4238 len 32768 data: 4 4 4 4 2 2 3 3
8083 225/300 Write block 3197 len 28672 data: 2 2 2 2 3 3 4
8084 226/300 Flush
8085 227/300 Write block 1624 len 20480 data: 5 5 4 3 3
8086 228/300 Write block 663 len 4096 data: 5
8087 229/300 Read block 3871 len 32768
8088 230/300 Write block 2214 len 20480 data: 3 3 3 3 3
8089 231/300 Flush
8090 232/300 Read block 4456 len 16384
8091 233/300 Read block 3104 len 40960
8092 234/300 Read block 1842 len 24576
8093 235/300 Write block 618 len 40960 data: 2 2 2 2 2 3 3 3 3 3
8094 236/300 Read block 586 len 28672
8095 237/300 Read block 3138 len 16384
8096 238/300 Write block 1190 len 12288 data: 4 5 5
8097 239/300 Read block 1220 len 40960
8098 240/300 Write block 3874 len 16384 data: 3 3 4 4
8099 241/300 Flush
8100 242/300 Write block 3420 len 4096 data: 2
8101 243/300 Read block 852 len 32768
8102 244/300 Write block 3617 len 20480 data: 2 2 2 2 3
8103 245/300 Read block 2555 len 16384
8104 246/300 Flush
8105 247/300 Read block 1716 len 32768
8106 248/300 Flush
8107 249/300 Read block 4536 len 24576
8108 250/300 Read block 1435 len 8192
8109 251/300 Read block 4771 len 28672
8110 252/300 Read block 462 len 4096
8111 253/300 Read block 494 len 20480
8112 254/300 Read block 369 len 4096
8113 255/300 Write block 249 len 16384 data: 4 4 3 2
8114 256/300 Write block 1389 len 20480 data: 5 4 4 5 5
8115 257/300 Read block 3521 len 24576
8116 258/300 Write block 3994 len 28672 data: 2 2 2 3 3 3 3
8117 259/300 Flush
8118 260/300 Write block 2220 len 24576 data: 3 2 2 2 2 2
8119 261/300 Write block 2712 len 12288 data: 2 2 2
8120 262/300 Write block 3118 len 40960 data: 2 2 3 3 5 4 4 2 2 2
8121 263/300 Read block 2740 len 36864
8122 264/300 Write block 415 len 28672 data: 4 4 4 4 4 5 3
8123 265/300 Read block 474 len 20480
8124 266/300 Read block 3717 len 16384
8125 267/300 Write block 1948 len 28672 data: 3 3 3 3 2 3 3
8126 268/300 Read block 1475 len 40960
8127 269/300 Read block 4173 len 32768
8128 270/300 Read block 2880 len 40960
8129 271/300 Read block 1959 len 36864
8130 272/300 Flush
8131 273/300 Write block 1713 len 24576 data: 6 7 7 8 7 5
8132 274/300 Read block 2373 len 36864
8133 275/300 Read block 4459 len 16384
8134 276/300 Write block 4149 len 20480 data: 4 3 3 3 3
8135 277/300 Write block 2158 len 12288 data: 2 2 2
8136 278/300 Read block 2307 len 28672
8137 279/300 Read block 748 len 28672
8138 280/300 Write block 3708 len 8192 data: 2 3
8139 281/300 Write block 1720 len 4096 data: 5
8140 282/300 Read block 2202 len 12288
8141 283/300 Flush
8142 284/300 Read block 2929 len 32768
8143 285/300 Read block 1936 len 36864
8144 286/300 Write block 3994 len 28672 data: 3 3 3 4 4 4 4
8145 287/300 Write block 1515 len 20480 data: 3 4 5 6 5
8146 288/300 Flush
8147 289/300 Write block 3077 len 28672 data: 2 2 3 3 3 3 4
8148 290/300 Read block 2364 len 8192
8149 291/300 Write block 4983 len 36864 data: 4 4 4 4 4 3 3 3 3
8150 292/300 Read block 1156 len 4096
8151 293/300 Write block 4384 len 4096 data: 3
8152 294/300 Read block 4899 len 32768
8153 295/300 Write block 3556 len 20480 data: 3 3 3 3 3
8154 296/300 Read block 2026 len 12288
8155 297/300 Read block 3313 len 32768
8156 298/300 Read block 4832 len 28672
8157 299/300 Write block 2128 len 24576 data: 5 4 4 5 5 5
8158 300/300 Read block 2701 len 32768
8159 Main task has work to do, go find it
8160 got dsc Start(2)
8161 start 2
8162 Feb 28 22:14:52.656 INFO request completed, latency_us: 202, response_code: 204, uri: /start/cid/2, method: POST, req_id: 29d0267e-b2ff-458c-ac46-6b6cf23f8a63, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
8163 Replay: started 2, returned:Ok(())
8164 ----------------------------------------------------------------
8165 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
8166 States: Active Active Offline
8167 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
8168 2713 Acked 3712 Write 3 Done Done New true
8169 2714 Acked 3713 Write 2 Done Done New true
8170 2715 Acked 3714 Flush 0 Done Done New true
8171 2716 Acked 3715 Flush 0 Done Done New false
8172 2717 Acked 3716 Write 2 Done Done New false
8173 2718 Acked 3717 Write 10 Done Done New false
8174 2719 Acked 3718 Flush 0 Done Done New false
8175 2720 Acked 3719 Read 7 Done Done New false
8176 2721 Acked 3720 Flush 0 Done Done New false
8177 2722 Acked 3721 Read 10 Done Done New false
8178 2723 Acked 3722 Write 10 Done Done New false
8179 2724 Acked 3723 Write 4 Done Done New false
8180 2725 Acked 3724 Read 3 Done Done New false
8181 2726 Acked 3725 Read 7 Done Done New false
8182 2727 Acked 3726 Flush 0 Done Done New false
8183 2728 Acked 3727 Read 10 Done Done New false
8184 2729 Acked 3728 Flush 0 Done Done New false
8185 2730 Acked 3729 Read 9 Done Done New false
8186 2731 Acked 3730 Write 2 Done Done New false
8187 2732 Acked 3731 Write 6 Done Done New false
8188 2733 Acked 3732 Flush 0 Done Done New false
8189 2734 Acked 3733 Write 4 Done Done New false
8190 2735 Acked 3734 Read 3 Done Done New false
8191 2736 Acked 3735 Read 8 Done Done New false
8192 2737 Acked 3736 Read 8 Done Done New false
8193 2738 Acked 3737 Read 5 Done Done New false
8194 2739 Acked 3738 Write 3 Done Done New false
8195 2740 Acked 3739 Read 6 Done Done New false
8196 2741 Acked 3740 Flush 0 Done Done New false
8197 2742 Acked 3741 Write 2 Done Done New false
8198 2743 Acked 3742 Flush 0 Done Done New false
8199 2744 Acked 3743 Read 2 Done Done New false
8200 2745 Acked 3744 Write 5 Done Done New false
8201 2746 Acked 3745 Flush 0 Done Done New false
8202 2747 Acked 3746 Read 2 Done Done New false
8203 2748 Acked 3747 Flush 0 Done Done New false
8204 2749 Acked 3748 Read 3 Done Done New false
8205 2750 Acked 3749 Read 1 Done Done New false
8206 2751 Acked 3750 Write 8 Done Done New false
8207 2752 Acked 3751 Read 1 Done Done New false
8208 2753 Acked 3752 Read 3 Done Done New false
8209 2754 Acked 3753 Write 2 Done Done New false
8210 2755 Acked 3754 Read 1 Done Done New false
8211 2756 Acked 3755 Write 10 Done Done New false
8212 2757 Acked 3756 Write 3 Done Done New false
8213 2758 Acked 3757 Read 7 Done Done New false
8214 2759 Acked 3758 Read 2 Done Done New false
8215 2760 Acked 3759 Read 10 Done Done New false
8216 2761 Acked 3760 Flush 0 Done Done New false
8217 2762 Acked 3761 Read 6 Done Done New false
8218 2763 Acked 3762 Read 2 Done Done New false
8219 2764 Acked 3763 Read 5 Done Done New false
8220 2765 Acked 3764 Write 7 Done Done New false
8221 2766 Acked 3765 Read 5 Done Done New false
8222 2767 Acked 3766 Read 4 Done Done New false
8223 2768 Acked 3767 Read 7 Done Done New false
8224 2769 Acked 3768 Write 3 Done Done New false
8225 2770 Acked 3769 Read 1 Done Done New false
8226 2771 Acked 3770 Write 1 Done Done New false
8227 2772 Acked 3771 Write 10 Done Done New false
8228 2773 Acked 3772 Write 1 Done Done New false
8229 2774 Acked 3773 Write 7 Done Done New false
8230 2775 Acked 3774 Write 8 Done Done New false
8231 2776 Acked 3775 Write 10 Done Done New false
8232 2777 Acked 3776 Write 1 Done Done New false
8233 2778 Acked 3777 Write 10 Done Done New false
8234 2779 Acked 3778 Read 1 Done Done New false
8235 2780 Acked 3779 Write 4 Done Done New false
8236 2781 Acked 3780 Write 3 Done Done New false
8237 2782 Acked 3781 Read 8 Done Done New false
8238 2783 Acked 3782 Read 5 Done Done New false
8239 2784 Acked 3783 Write 6 Done Done New false
8240 2785 Acked 3784 Flush 0 Done Done New false
8241 2786 Acked 3785 Read 7 Done Done New false
8242 2787 Acked 3786 Read 5 Done Done New false
8243 2788 Acked 3787 Flush 0 Done Done New false
8244 2789 Acked 3788 Write 9 Done Done New false
8245 2790 Acked 3789 Read 9 Done Done New false
8246 2791 Acked 3790 Write 5 Done Done New false
8247 2792 Acked 3791 Write 7 Done Done New false
8248 2793 Acked 3792 Write 1 Done Done New false
8249 2794 Acked 3793 Write 7 Done Done New false
8250 2795 Acked 3794 Write 3 Done Done New false
8251 2796 Acked 3795 Read 7 Done Done New false
8252 2797 Acked 3796 Write 4 Done Done New false
8253 2798 Acked 3797 Write 1 Done Done New false
8254 2799 Acked 3798 Read 10 Done Done New false
8255 2800 Acked 3799 Write 1 Done Done New false
8256 2801 Acked 3800 Write 7 Done Done New false
8257 2802 Acked 3801 Read 2 Done Done New false
8258 2803 Acked 3802 Read 10 Done Done New false
8259 2804 Acked 3803 Read 9 Done Done New false
8260 2805 Acked 3804 Write 10 Done Done New false
8261 2806 Acked 3805 Read 9 Done Done New false
8262 2807 Acked 3806 Write 3 Done Done New false
8263 2808 Acked 3807 Read 6 Done Done New false
8264 2809 Acked 3808 Read 8 Done Done New false
8265 2810 Acked 3809 Read 5 Done Done New false
8266 2811 Acked 3810 Read 1 Done Done New false
8267 2812 Acked 3811 Write 2 Done Done New false
8268 2813 Acked 3812 Write 2 Done Done New false
8269 2814 Acked 3813 Write 3 Done Done New false
8270 2815 Acked 3814 Read 6 Done Done New false
8271 2816 Acked 3815 Write 4 Done Done New false
8272 2817 Acked 3816 Write 2 Done Done New false
8273 2818 Acked 3817 Read 10 Done Done New false
8274 2819 Acked 3818 Read 10 Done Done New false
8275 2820 Acked 3819 Read 10 Done Done New false
8276 2821 Acked 3820 Flush 0 Done Done New false
8277 2822 Acked 3821 Read 3 Done Done New false
8278 2823 Acked 3822 Write 3 Done Done New false
8279 2824 Acked 3823 Write 2 Done Done New false
8280 2825 Acked 3824 Read 5 Done Done New false
8281 2826 Acked 3825 Write 8 Done Done New false
8282 2827 Acked 3826 Read 4 Done Done New false
8283 2828 Acked 3827 Write 6 Done Done New false
8284 2829 Acked 3828 Read 7 Done Done New false
8285 2830 Acked 3829 Write 6 Done Done New false
8286 2831 Acked 3830 Write 10 Done Done New false
8287 2832 Acked 3831 Write 1 Done Done New false
8288 2833 Acked 3832 Write 9 Done Done New false
8289 2834 Acked 3833 Read 1 Done Done New false
8290 2835 Acked 3834 Read 1 Done Done New false
8291 2836 Acked 3835 Read 4 Done Done New false
8292 2837 Acked 3836 Write 1 Done Done New false
8293 2838 Acked 3837 Write 10 Done Done New false
8294 2839 Acked 3838 Read 3 Done Done New false
8295 2840 Acked 3839 Read 2 Done Done New false
8296 2841 Acked 3840 Read 1 Done Done New false
8297 2842 Acked 3841 Write 4 Done Done New false
8298 2843 Acked 3842 Write 3 Done Done New false
8299 2844 Acked 3843 Write 10 Done Done New false
8300 2845 Acked 3844 Write 10 Done Done New false
8301 2846 Acked 3845 Read 7 Done Done New false
8302 2847 Acked 3846 Write 1 Done Done New false
8303 2848 Acked 3847 Flush 0 Done Done New false
8304 2849 Acked 3848 Read 8 Done Done New false
8305 2850 Acked 3849 Read 6 Done Done New false
8306 2851 Acked 3850 Read 9 Done Done New false
8307 2852 Acked 3851 Read 1 Done Done New false
8308 2853 Acked 3852 Write 6 Done Done New false
8309 2854 Acked 3853 Flush 0 Done Done New false
8310 2855 Acked 3854 Write 9 Done Done New false
8311 2856 Acked 3855 Write 3 Done Done New false
8312 2857 Acked 3856 Write 10 Done Done New false
8313 2858 Acked 3857 Flush 0 Done Done New false
8314 2859 Acked 3858 Write 6 Done Done New false
8315 2860 Acked 3859 Flush 0 Done Done New false
8316 2861 Acked 3860 Read 9 Done Done New false
8317 2862 Acked 3861 Flush 0 Done Done New false
8318 2863 Acked 3862 Write 6 Done Done New false
8319 2864 Acked 3863 Read 7 Done Done New false
8320 2865 Acked 3864 Write 10 Done Done New false
8321 2866 Acked 3865 Flush 0 Done Done New false
8322 2867 Acked 3866 Read 6 Done Done New false
8323 2868 Acked 3867 Write 6 Done Done New false
8324 2869 Acked 3868 Write 8 Done Done New false
8325 2870 Acked 3869 Read 5 Done Done New false
8326 2871 Acked 3870 Read 7 Done Done New false
8327 2872 Acked 3871 Read 3 Done Done New false
8328 2873 Acked 3872 Flush 0 Done Done New false
8329 2874 Acked 3873 Write 3 Done Done New false
8330 2875 Acked 3874 Read 6 Done Done New false
8331 2876 Acked 3875 Read 7 Done Done New false
8332 2877 Acked 3876 Read 5 Done Done New false
8333 2878 Acked 3877 Read 3 Done Done New false
8334 2879 Acked 3878 Read 5 Done Done New false
8335 2880 Acked 3879 Read 6 Done Done New false
8336 2881 Acked 3880 Read 3 Done Done New false
8337 2882 Acked 3881 Read 8 Done Done New false
8338 2883 Acked 3882 Read 9 Done Done New false
8339 2884 Acked 3883 Read 5 Done Done New false
8340 2885 Acked 3884 Write 4 Done Done New false
8341 2886 Acked 3885 Write 1 Done Done New false
8342 2887 Acked 3886 Read 1 Done Done New false
8343 2888 Acked 3887 Write 7 Done Done New false
8344 2889 Acked 3888 Write 4 Done Done New false
8345 2890 Acked 3889 Write 3 Done Done New false
8346 2891 Acked 3890 Write 6 Done Done New false
8347 2892 Acked 3891 Write 10 Done Done New false
8348 2893 Acked 3892 Write 5 Done Done New false
8349 2894 Acked 3893 Read 2 Done Done New false
8350 2895 Acked 3894 Write 4 Done Done New false
8351 2896 Acked 3895 Write 9 Done Done New false
8352 2897 Acked 3896 Read 8 Done Done New false
8353 2898 Acked 3897 Write 9 Done Done New false
8354 2899 Acked 3898 Read 9 Done Done New false
8355 2900 Acked 3899 Read 2 Done Done New false
8356 2901 Acked 3900 Flush 0 Done Done New false
8357 2902 Acked 3901 Flush 0 Done Done New false
8358 2903 Acked 3902 Write 1 Done Done New false
8359 2904 Acked 3903 Read 9 Done Done New false
8360 2905 Acked 3904 Read 2 Done Done New false
8361 2906 Acked 3905 Read 7 Done Done New false
8362 2907 Acked 3906 Write 9 Done Done New false
8363 2908 Acked 3907 Read 6 Done Done New false
8364 2909 Acked 3908 Flush 0 Done Done New false
8365 2910 Acked 3909 Read 1 Done Done New false
8366 2911 Acked 3910 Flush 0 Done Done New false
8367 2912 Acked 3911 Read 3 Done Done New false
8368 2913 Acked 3912 Flush 0 Done Done New false
8369 2914 Acked 3913 Write 1 Done Done New false
8370 2915 Acked 3914 Flush 0 Done Done New false
8371 2916 Acked 3915 Read 9 Done Done New false
8372 2917 Acked 3916 Flush 0 Done Done New false
8373 2918 Acked 3917 Read 7 Done Done New false
8374 2919 Acked 3918 Read 7 Done Done New false
8375 2920 Acked 3919 Read 8 Done Done New false
8376 2921 Acked 3920 Write 10 Done Done New false
8377 2922 Acked 3921 Read 5 Done Done New false
8378 2923 Acked 3922 Read 9 Done Done New false
8379 2924 Acked 3923 Read 5 Done Done New false
8380 2925 Acked 3924 Read 7 Done Done New false
8381 2926 Acked 3925 Read 7 Done Done New false
8382 2927 Acked 3926 Read 6 Done Done New false
8383 2928 Acked 3927 Read 2 Done Done New false
8384 2929 Acked 3928 Write 7 Done Done New false
8385 2930 Acked 3929 Read 9 Done Done New false
8386 2931 Acked 3930 Write 7 Done Done New false
8387 2932 Acked 3931 Write 5 Done Done New false
8388 2933 Acked 3932 Read 3 Done Done New false
8389 2934 Acked 3933 Read 5 Done Done New false
8390 2935 Acked 3934 Read 7 Done Done New false
8391 2936 Acked 3935 Write 8 Done Done New false
8392 2937 Acked 3936 Write 7 Done Done New false
8393 2938 Acked 3937 Flush 0 Done Done New false
8394 2939 Acked 3938 Write 5 Done Done New false
8395 2940 Acked 3939 Write 1 Done Done New false
8396 2941 Acked 3940 Read 8 Done Done New false
8397 2942 Acked 3941 Write 5 Done Done New false
8398 2943 Acked 3942 Flush 0 Done Done New false
8399 2944 Acked 3943 Read 4 Done Done New false
8400 2945 Acked 3944 Read 10 Done Done New false
8401 2946 Acked 3945 Read 6 Done Done New false
8402 2947 Acked 3946 Write 10 Done Done New false
8403 2948 Acked 3947 Read 7 Done Done New false
8404 2949 Acked 3948 Read 4 Done Done New false
8405 2950 Acked 3949 Write 3 Done Done New false
8406 2951 Acked 3950 Read 10 Done Done New false
8407 2952 Acked 3951 Write 4 Done Done New false
8408 2953 Acked 3952 Flush 0 Done Done New false
8409 2954 Acked 3953 Write 1 Done Done New false
8410 2955 Acked 3954 Read 8 Done Done New false
8411 2956 Acked 3955 Write 5 Done Done New false
8412 2957 Acked 3956 Read 4 Done Done New false
8413 2958 Acked 3957 Flush 0 Done Done New false
8414 2959 Acked 3958 Read 8 Done Done New false
8415 2960 Acked 3959 Flush 0 Done Done New false
8416 2961 Acked 3960 Read 6 Done Done New false
8417 2962 Acked 3961 Read 2 Done Done New false
8418 2963 Acked 3962 Read 7 Done Done New false
8419 2964 Acked 3963 Read 1 Done Done New false
8420 2965 Acked 3964 Read 5 Done Done New false
8421 2966 Acked 3965 Read 1 Done Done New false
8422 2967 Acked 3966 Write 4 Done Done New false
8423 2968 Acked 3967 Write 5 Done Done New false
8424 2969 Acked 3968 Read 6 Done Done New false
8425 2970 Acked 3969 Write 7 Done Done New false
8426 2971 Acked 3970 Flush 0 Done Done New false
8427 2972 Acked 3971 Write 6 Done Done New false
8428 2973 Acked 3972 Write 3 Done Done New false
8429 2974 Acked 3973 Write 10 Done Done New false
8430 2975 Acked 3974 Read 9 Done Done New false
8431 2976 Acked 3975 Write 7 Done Done New false
8432 2977 Acked 3976 Read 5 Done Done New false
8433 2978 Acked 3977 Read 4 Done Done New false
8434 2979 Acked 3978 Write 7 Done Done New false
8435 2980 Acked 3979 Read 10 Done Done New false
8436 2981 Acked 3980 Read 8 Done Done New false
8437 2982 Acked 3981 Read 10 Done Done New false
8438 2983 Acked 3982 Read 9 Done Done New false
8439 2984 Acked 3983 Flush 0 Done Done New false
8440 2985 Acked 3984 Write 6 Done Done New false
8441 2986 Acked 3985 Read 9 Done Done New false
8442 2987 Acked 3986 Read 4 Done Done New false
8443 2988 Acked 3987 Write 5 Done Done New false
8444 2989 Acked 3988 Write 3 Done Done New false
8445 2990 Acked 3989 Read 7 Done Done New false
8446 2991 Acked 3990 Read 7 Done Done New false
8447 2992 Acked 3991 Write 2 Done Done New false
8448 2993 Acked 3992 Write 1 Done Done New false
8449 2994 Acked 3993 Read 3 Done Done New false
8450 2995 Acked 3994 Flush 0 Done Done New false
8451 2996 Acked 3995 Read 8 Done Done New false
8452 2997 Acked 3996 Read 9 Done Done New false
8453 2998 Acked 3997 Write 7 Done Done New false
8454 2999 Acked 3998 Write 5 Done Done New false
8455 3000 Acked 3999 Flush 0 Done Done New false
8456 3001 Acked 4000 Write 7 Done Done New false
8457 3002 Acked 4001 Read 2 Done Done New false
8458 3003 Acked 4002 Write 9 Done Done New false
8459 3004 Acked 4003 Read 1 Done Done New false
8460 3005 Acked 4004 Write 1 Done Done New false
8461 3006 Acked 4005 Read 8 Done Done New false
8462 3007 Acked 4006 Write 5 Done Done New false
8463 3008 Acked 4007 Read 3 Done Done New false
8464 3009 Acked 4008 Read 8 Done Done New false
8465 3010 Acked 4009 Read 7 Done Done New false
8466 3011 Acked 4010 Write 6 Done Done New false
8467 3012 Acked 4011 Read 8 Done Done New false
8468 STATES DS:0 DS:1 DS:2 TOTAL
8469 New 0 0 300 300
8470 Sent 0 0 0 0
8471 Done 300 300 0 600
8472 Skipped 0 0 0 0
8473 Error 0 0 0 0
8474 Last Flush: 3999 3999 3711
8475 Downstairs last five completed: 3711 3710 3709 3708 3707
8476 Upstairs last five completed: 3012 3011 3010 3009 3008
8477 CLIENT: Up:0 ds:300 act:2
84782024-02-28T22:14:53.019ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
84792024-02-28T22:14:53.019ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
84802024-02-28T22:14:53.019ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
84812024-02-28T22:14:53.019ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
84822024-02-28T22:14:53.019ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
84832024-02-28T22:14:53.019ZINFOcrucible: [2] client re-new 301 jobs since flush 3711 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
84842024-02-28T22:14:53.019ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
84852024-02-28T22:14:54.020ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
84862024-02-28T22:14:54.020ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
84872024-02-28T22:14:54.020ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
84882024-02-28T22:14:54.020ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
84892024-02-28T22:14:54.020ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
84902024-02-28T22:14:54.020ZINFOcrucible: [2] client re-new 301 jobs since flush 3711 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
84912024-02-28T22:14:54.020ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
84922024-02-28T22:14:55.021ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
84932024-02-28T22:14:55.021ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
84942024-02-28T22:14:55.021ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
84952024-02-28T22:14:55.021ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
84962024-02-28T22:14:55.021ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
84972024-02-28T22:14:55.021ZINFOcrucible: [2] client re-new 301 jobs since flush 3711 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
84982024-02-28T22:14:55.021ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8499 [8830] got start action
8500 [8830] I am going to restart
8501 Starting downstairs at port 8830
8502 Make output file at "/tmp/dsc/downstairs-8830.txt"
85032024-02-28T22:14:56.022ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8504 [8830][2] reports Starting
85052024-02-28T22:14:56.022ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
85062024-02-28T22:14:56.023ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
85072024-02-28T22:14:56.023ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
85082024-02-28T22:14:56.023ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
85092024-02-28T22:14:56.023ZINFOcrucible: [2] client re-new 301 jobs since flush 3711 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
85102024-02-28T22:14:56.023ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8511 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1174)
8512 [8830][2] reports Running
8513 ----------------------------------------------------------------
8514 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8515 States: Active Active Offline
8516 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
8517 2713 Acked 3712 Write 3 Done Done New true
8518 2714 Acked 3713 Write 2 Done Done New true
8519 2715 Acked 3714 Flush 0 Done Done New true
8520 2716 Acked 3715 Flush 0 Done Done New true
8521 2717 Acked 3716 Write 2 Done Done New true
8522 2718 Acked 3717 Write 10 Done Done New true
8523 2719 Acked 3718 Flush 0 Done Done New true
8524 2720 Acked 3719 Read 7 Done Done New true
8525 2721 Acked 3720 Flush 0 Done Done New true
8526 2722 Acked 3721 Read 10 Done Done New true
8527 2723 Acked 3722 Write 10 Done Done New true
8528 2724 Acked 3723 Write 4 Done Done New true
8529 2725 Acked 3724 Read 3 Done Done New true
8530 2726 Acked 3725 Read 7 Done Done New true
8531 2727 Acked 3726 Flush 0 Done Done New true
8532 2728 Acked 3727 Read 10 Done Done New true
8533 2729 Acked 3728 Flush 0 Done Done New true
8534 2730 Acked 3729 Read 9 Done Done New true
8535 2731 Acked 3730 Write 2 Done Done New true
8536 2732 Acked 3731 Write 6 Done Done New true
8537 2733 Acked 3732 Flush 0 Done Done New true
8538 2734 Acked 3733 Write 4 Done Done New true
8539 2735 Acked 3734 Read 3 Done Done New true
8540 2736 Acked 3735 Read 8 Done Done New true
8541 2737 Acked 3736 Read 8 Done Done New true
8542 2738 Acked 3737 Read 5 Done Done New true
8543 2739 Acked 3738 Write 3 Done Done New true
8544 2740 Acked 3739 Read 6 Done Done New true
8545 2741 Acked 3740 Flush 0 Done Done New true
8546 2742 Acked 3741 Write 2 Done Done New true
8547 2743 Acked 3742 Flush 0 Done Done New true
8548 2744 Acked 3743 Read 2 Done Done New true
8549 2745 Acked 3744 Write 5 Done Done New true
8550 2746 Acked 3745 Flush 0 Done Done New true
8551 2747 Acked 3746 Read 2 Done Done New true
8552 2748 Acked 3747 Flush 0 Done Done New true
8553 2749 Acked 3748 Read 3 Done Done New true
8554 2750 Acked 3749 Read 1 Done Done New true
8555 2751 Acked 3750 Write 8 Done Done New true
8556 2752 Acked 3751 Read 1 Done Done New true
8557 2753 Acked 3752 Read 3 Done Done New true
8558 2754 Acked 3753 Write 2 Done Done New true
8559 2755 Acked 3754 Read 1 Done Done New true
8560 2756 Acked 3755 Write 10 Done Done New true
8561 2757 Acked 3756 Write 3 Done Done New true
8562 2758 Acked 3757 Read 7 Done Done New true
8563 2759 Acked 3758 Read 2 Done Done New true
8564 2760 Acked 3759 Read 10 Done Done New true
8565 2761 Acked 3760 Flush 0 Done Done New true
8566 2762 Acked 3761 Read 6 Done Done New true
8567 2763 Acked 3762 Read 2 Done Done New true
8568 2764 Acked 3763 Read 5 Done Done New true
8569 2765 Acked 3764 Write 7 Done Done New true
8570 2766 Acked 3765 Read 5 Done Done New true
8571 2767 Acked 3766 Read 4 Done Done New true
8572 2768 Acked 3767 Read 7 Done Done New true
8573 2769 Acked 3768 Write 3 Done Done New true
8574 2770 Acked 3769 Read 1 Done Done New true
8575 2771 Acked 3770 Write 1 Done Done New true
8576 2772 Acked 3771 Write 10 Done Done New true
8577 2773 Acked 3772 Write 1 Done Done New true
8578 2774 Acked 3773 Write 7 Done Done New true
8579 2775 Acked 3774 Write 8 Done Done New true
8580 2776 Acked 3775 Write 10 Done Done New true
8581 2777 Acked 3776 Write 1 Done Done New true
8582 2778 Acked 3777 Write 10 Done Done New true
8583 2779 Acked 3778 Read 1 Done Done New true
8584 2780 Acked 3779 Write 4 Done Done New true
8585 2781 Acked 3780 Write 3 Done Done New true
8586 2782 Acked 3781 Read 8 Done Done New true
8587 2783 Acked 3782 Read 5 Done Done New true
8588 2784 Acked 3783 Write 6 Done Done New true
8589 2785 Acked 3784 Flush 0 Done Done New true
8590 2786 Acked 3785 Read 7 Done Done New true
8591 2787 Acked 3786 Read 5 Done Done New true
8592 2788 Acked 3787 Flush 0 Done Done New true
8593 2789 Acked 3788 Write 9 Done Done New true
8594 2790 Acked 3789 Read 9 Done Done New true
8595 2791 Acked 3790 Write 5 Done Done New true
8596 2792 Acked 3791 Write 7 Done Done New true
8597 2793 Acked 3792 Write 1 Done Done New true
8598 2794 Acked 3793 Write 7 Done Done New true
8599 2795 Acked 3794 Write 3 Done Done New true
8600 2796 Acked 3795 Read 7 Done Done New true
8601 2797 Acked 3796 Write 4 Done Done New true
8602 2798 Acked 3797 Write 1 Done Done New true
8603 2799 Acked 3798 Read 10 Done Done New true
8604 2800 Acked 3799 Write 1 Done Done New true
8605 2801 Acked 3800 Write 7 Done Done New true
8606 2802 Acked 3801 Read 2 Done Done New true
8607 2803 Acked 3802 Read 10 Done Done New true
8608 2804 Acked 3803 Read 9 Done Done New true
8609 2805 Acked 3804 Write 10 Done Done New true
8610 2806 Acked 3805 Read 9 Done Done New true
8611 2807 Acked 3806 Write 3 Done Done New true
8612 2808 Acked 3807 Read 6 Done Done New true
8613 2809 Acked 3808 Read 8 Done Done New true
8614 2810 Acked 3809 Read 5 Done Done New true
8615 2811 Acked 3810 Read 1 Done Done New true
8616 2812 Acked 3811 Write 2 Done Done New true
8617 2813 Acked 3812 Write 2 Done Done New true
8618 2814 Acked 3813 Write 3 Done Done New true
8619 2815 Acked 3814 Read 6 Done Done New true
8620 2816 Acked 3815 Write 4 Done Done New true
8621 2817 Acked 3816 Write 2 Done Done New true
8622 2818 Acked 3817 Read 10 Done Done New true
8623 2819 Acked 3818 Read 10 Done Done New true
8624 2820 Acked 3819 Read 10 Done Done New true
8625 2821 Acked 3820 Flush 0 Done Done New true
8626 2822 Acked 3821 Read 3 Done Done New true
8627 2823 Acked 3822 Write 3 Done Done New true
8628 2824 Acked 3823 Write 2 Done Done New true
8629 2825 Acked 3824 Read 5 Done Done New true
8630 2826 Acked 3825 Write 8 Done Done New true
8631 2827 Acked 3826 Read 4 Done Done New true
8632 2828 Acked 3827 Write 6 Done Done New true
8633 2829 Acked 3828 Read 7 Done Done New true
8634 2830 Acked 3829 Write 6 Done Done New true
8635 2831 Acked 3830 Write 10 Done Done New true
8636 2832 Acked 3831 Write 1 Done Done New true
8637 2833 Acked 3832 Write 9 Done Done New true
8638 2834 Acked 3833 Read 1 Done Done New true
8639 2835 Acked 3834 Read 1 Done Done New true
8640 2836 Acked 3835 Read 4 Done Done New true
8641 2837 Acked 3836 Write 1 Done Done New true
8642 2838 Acked 3837 Write 10 Done Done New true
8643 2839 Acked 3838 Read 3 Done Done New true
8644 2840 Acked 3839 Read 2 Done Done New true
8645 2841 Acked 3840 Read 1 Done Done New true
8646 2842 Acked 3841 Write 4 Done Done New true
8647 2843 Acked 3842 Write 3 Done Done New true
8648 2844 Acked 3843 Write 10 Done Done New true
8649 2845 Acked 3844 Write 10 Done Done New true
8650 2846 Acked 3845 Read 7 Done Done New true
8651 2847 Acked 3846 Write 1 Done Done New true
8652 2848 Acked 3847 Flush 0 Done Done New true
8653 2849 Acked 3848 Read 8 Done Done New true
8654 2850 Acked 3849 Read 6 Done Done New true
8655 2851 Acked 3850 Read 9 Done Done New true
8656 2852 Acked 3851 Read 1 Done Done New true
8657 2853 Acked 3852 Write 6 Done Done New true
8658 2854 Acked 3853 Flush 0 Done Done New true
8659 2855 Acked 3854 Write 9 Done Done New true
8660 2856 Acked 3855 Write 3 Done Done New true
8661 2857 Acked 3856 Write 10 Done Done New true
8662 2858 Acked 3857 Flush 0 Done Done New true
8663 2859 Acked 3858 Write 6 Done Done New true
8664 2860 Acked 3859 Flush 0 Done Done New true
8665 2861 Acked 3860 Read 9 Done Done New true
8666 2862 Acked 3861 Flush 0 Done Done New true
8667 2863 Acked 3862 Write 6 Done Done New true
8668 2864 Acked 3863 Read 7 Done Done New true
8669 2865 Acked 3864 Write 10 Done Done New true
8670 2866 Acked 3865 Flush 0 Done Done New true
8671 2867 Acked 3866 Read 6 Done Done New true
8672 2868 Acked 3867 Write 6 Done Done New true
8673 2869 Acked 3868 Write 8 Done Done New true
8674 2870 Acked 3869 Read 5 Done Done New true
8675 2871 Acked 3870 Read 7 Done Done New true
8676 2872 Acked 3871 Read 3 Done Done New true
8677 2873 Acked 3872 Flush 0 Done Done New true
8678 2874 Acked 3873 Write 3 Done Done New true
8679 2875 Acked 3874 Read 6 Done Done New true
8680 2876 Acked 3875 Read 7 Done Done New true
8681 2877 Acked 3876 Read 5 Done Done New true
8682 2878 Acked 3877 Read 3 Done Done New true
8683 2879 Acked 3878 Read 5 Done Done New true
8684 2880 Acked 3879 Read 6 Done Done New true
8685 2881 Acked 3880 Read 3 Done Done New true
8686 2882 Acked 3881 Read 8 Done Done New true
8687 2883 Acked 3882 Read 9 Done Done New true
8688 2884 Acked 3883 Read 5 Done Done New true
8689 2885 Acked 3884 Write 4 Done Done New true
8690 2886 Acked 3885 Write 1 Done Done New true
8691 2887 Acked 3886 Read 1 Done Done New true
8692 2888 Acked 3887 Write 7 Done Done New true
8693 2889 Acked 3888 Write 4 Done Done New true
8694 2890 Acked 3889 Write 3 Done Done New true
8695 2891 Acked 3890 Write 6 Done Done New true
8696 2892 Acked 3891 Write 10 Done Done New true
8697 2893 Acked 3892 Write 5 Done Done New true
8698 2894 Acked 3893 Read 2 Done Done New true
8699 2895 Acked 3894 Write 4 Done Done New true
8700 2896 Acked 3895 Write 9 Done Done New true
8701 2897 Acked 3896 Read 8 Done Done New true
8702 2898 Acked 3897 Write 9 Done Done New true
8703 2899 Acked 3898 Read 9 Done Done New true
8704 2900 Acked 3899 Read 2 Done Done New true
8705 2901 Acked 3900 Flush 0 Done Done New true
8706 2902 Acked 3901 Flush 0 Done Done New true
8707 2903 Acked 3902 Write 1 Done Done New true
8708 2904 Acked 3903 Read 9 Done Done New true
8709 2905 Acked 3904 Read 2 Done Done New true
8710 2906 Acked 3905 Read 7 Done Done New true
8711 2907 Acked 3906 Write 9 Done Done New true
8712 2908 Acked 3907 Read 6 Done Done New true
8713 2909 Acked 3908 Flush 0 Done Done New true
8714 2910 Acked 3909 Read 1 Done Done New true
8715 2911 Acked 3910 Flush 0 Done Done New true
8716 2912 Acked 3911 Read 3 Done Done New true
8717 2913 Acked 3912 Flush 0 Done Done New true
8718 2914 Acked 3913 Write 1 Done Done New true
8719 2915 Acked 3914 Flush 0 Done Done New true
8720 2916 Acked 3915 Read 9 Done Done New true
8721 2917 Acked 3916 Flush 0 Done Done New true
8722 2918 Acked 3917 Read 7 Done Done New true
8723 2919 Acked 3918 Read 7 Done Done New true
8724 2920 Acked 3919 Read 8 Done Done New true
8725 2921 Acked 3920 Write 10 Done Done New true
8726 2922 Acked 3921 Read 5 Done Done New true
8727 2923 Acked 3922 Read 9 Done Done New true
8728 2924 Acked 3923 Read 5 Done Done New true
8729 2925 Acked 3924 Read 7 Done Done New true
8730 2926 Acked 3925 Read 7 Done Done New true
8731 2927 Acked 3926 Read 6 Done Done New true
8732 2928 Acked 3927 Read 2 Done Done New true
8733 2929 Acked 3928 Write 7 Done Done New true
8734 2930 Acked 3929 Read 9 Done Done New true
8735 2931 Acked 3930 Write 7 Done Done New true
8736 2932 Acked 3931 Write 5 Done Done New true
8737 2933 Acked 3932 Read 3 Done Done New true
8738 2934 Acked 3933 Read 5 Done Done New true
8739 2935 Acked 3934 Read 7 Done Done New true
8740 2936 Acked 3935 Write 8 Done Done New true
8741 2937 Acked 3936 Write 7 Done Done New true
8742 2938 Acked 3937 Flush 0 Done Done New true
8743 2939 Acked 3938 Write 5 Done Done New true
8744 2940 Acked 3939 Write 1 Done Done New true
8745 2941 Acked 3940 Read 8 Done Done New true
8746 2942 Acked 3941 Write 5 Done Done New true
8747 2943 Acked 3942 Flush 0 Done Done New true
8748 2944 Acked 3943 Read 4 Done Done New true
8749 2945 Acked 3944 Read 10 Done Done New true
8750 2946 Acked 3945 Read 6 Done Done New true
8751 2947 Acked 3946 Write 10 Done Done New true
8752 2948 Acked 3947 Read 7 Done Done New true
8753 2949 Acked 3948 Read 4 Done Done New true
8754 2950 Acked 3949 Write 3 Done Done New true
8755 2951 Acked 3950 Read 10 Done Done New true
8756 2952 Acked 3951 Write 4 Done Done New true
8757 2953 Acked 3952 Flush 0 Done Done New true
8758 2954 Acked 3953 Write 1 Done Done New true
8759 2955 Acked 3954 Read 8 Done Done New true
8760 2956 Acked 3955 Write 5 Done Done New true
8761 2957 Acked 3956 Read 4 Done Done New true
8762 2958 Acked 3957 Flush 0 Done Done New true
8763 2959 Acked 3958 Read 8 Done Done New true
8764 2960 Acked 3959 Flush 0 Done Done New true
8765 2961 Acked 3960 Read 6 Done Done New true
8766 2962 Acked 3961 Read 2 Done Done New true
8767 2963 Acked 3962 Read 7 Done Done New true
8768 2964 Acked 3963 Read 1 Done Done New true
8769 2965 Acked 3964 Read 5 Done Done New true
8770 2966 Acked 3965 Read 1 Done Done New true
8771 2967 Acked 3966 Write 4 Done Done New true
8772 2968 Acked 3967 Write 5 Done Done New true
8773 2969 Acked 3968 Read 6 Done Done New true
8774 2970 Acked 3969 Write 7 Done Done New true
8775 2971 Acked 3970 Flush 0 Done Done New true
8776 2972 Acked 3971 Write 6 Done Done New true
8777 2973 Acked 3972 Write 3 Done Done New true
8778 2974 Acked 3973 Write 10 Done Done New true
8779 2975 Acked 3974 Read 9 Done Done New true
8780 2976 Acked 3975 Write 7 Done Done New true
8781 2977 Acked 3976 Read 5 Done Done New true
8782 2978 Acked 3977 Read 4 Done Done New true
8783 2979 Acked 3978 Write 7 Done Done New true
8784 2980 Acked 3979 Read 10 Done Done New true
8785 2981 Acked 3980 Read 8 Done Done New true
8786 2982 Acked 3981 Read 10 Done Done New true
8787 2983 Acked 3982 Read 9 Done Done New true
8788 2984 Acked 3983 Flush 0 Done Done New true
8789 2985 Acked 3984 Write 6 Done Done New true
8790 2986 Acked 3985 Read 9 Done Done New true
8791 2987 Acked 3986 Read 4 Done Done New true
8792 2988 Acked 3987 Write 5 Done Done New true
8793 2989 Acked 3988 Write 3 Done Done New true
8794 2990 Acked 3989 Read 7 Done Done New true
8795 2991 Acked 3990 Read 7 Done Done New true
8796 2992 Acked 3991 Write 2 Done Done New true
8797 2993 Acked 3992 Write 1 Done Done New true
8798 2994 Acked 3993 Read 3 Done Done New true
8799 2995 Acked 3994 Flush 0 Done Done New true
8800 2996 Acked 3995 Read 8 Done Done New true
8801 2997 Acked 3996 Read 9 Done Done New true
8802 2998 Acked 3997 Write 7 Done Done New true
8803 2999 Acked 3998 Write 5 Done Done New true
8804 3000 Acked 3999 Flush 0 Done Done New true
8805 3001 Acked 4000 Write 7 Done Done New true
8806 3002 Acked 4001 Read 2 Done Done New true
8807 3003 Acked 4002 Write 9 Done Done New true
8808 3004 Acked 4003 Read 1 Done Done New true
8809 3005 Acked 4004 Write 1 Done Done New true
8810 3006 Acked 4005 Read 8 Done Done New true
8811 3007 Acked 4006 Write 5 Done Done New true
8812 3008 Acked 4007 Read 3 Done Done New true
8813 3009 Acked 4008 Read 8 Done Done New true
8814 3010 Acked 4009 Read 7 Done Done New true
8815 3011 Acked 4010 Write 6 Done Done New true
8816 3012 Acked 4011 Read 8 Done Done New true
8817 3013 Acked 4012 Flush 0 Done Done New true
8818 STATES DS:0 DS:1 DS:2 TOTAL
8819 New 0 0 301 301
8820 Sent 0 0 0 0
8821 Done 301 301 0 602
8822 Skipped 0 0 0 0
8823 Error 0 0 0 0
8824 Last Flush: 4012 4012 3711
8825 Downstairs last five completed: 3711 3710 3709 3708 3707
8826 Upstairs last five completed: 3013 3012 3011 3010 3009
8827 CLIENT: Up:0 ds:301 act:2
88282024-02-28T22:14:57.024ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88292024-02-28T22:14:57.024ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88302024-02-28T22:14:57.025ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88312024-02-28T22:14:57.025ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88322024-02-28T22:14:57.025ZINFOcrucible: send last flush ID to this DS: 3711 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88332024-02-28T22:14:57.026ZINFOcrucible: Replied this last flush ID: 3711 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88342024-02-28T22:14:57.026ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88352024-02-28T22:14:57.026ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88362024-02-28T22:14:57.026ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88372024-02-28T22:14:57.026ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8838 ----------------------------------------------------------------
8839 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
8840 Downstairs last five completed: 4012 4011 4010 4009 4008
8841 Upstairs last five completed: 3013 3012 3011 3010 3009
8842 CLIENT: Up:0 ds:0 act:3
8843 Replay: All jobs finished, all DS active.
8844 Main task has work to do, go find it
8845 got dsc Stop(2)
8846 stop 2
8847 Feb 28 22:15:00.662 INFO request completed[8830] Got stop action so:false kr:false
8848 , latency_us: 327, response_code: 204, uri: /stop/cid/2, method: POST, req_id: c20990c8-0825-4a69-992c-70f5546bfa26, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
88492024-02-28T22:15:00.662ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88502024-02-28T22:15:00.662ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88512024-02-28T22:15:00.662ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88522024-02-28T22:15:00.662ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88532024-02-28T22:15:00.662ZINFOcrucible: [2] client re-new 0 jobs since flush 4012 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88542024-02-28T22:15:00.662ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8855 Feb 28 22:15:00.662 INFO request completed, latency_us: 156, response_code: 200, uri: /state/cid/2, method: GET, req_id: 3d97ec96-3315-4346-8195-867d7a159843, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
8856 [8830] Exited with: ExitStatus(unix_wait_status(9))
8857 [8830][2] reports Exit
88582024-02-28T22:15:01.663ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88592024-02-28T22:15:01.663ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88602024-02-28T22:15:01.663ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88612024-02-28T22:15:01.663ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88622024-02-28T22:15:01.663ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88632024-02-28T22:15:01.663ZINFOcrucible: [2] client re-new 0 jobs since flush 4012 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88642024-02-28T22:15:01.663ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88652024-02-28T22:15:02.664ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88662024-02-28T22:15:02.664ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88672024-02-28T22:15:02.664ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88682024-02-28T22:15:02.664ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88692024-02-28T22:15:02.664ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88702024-02-28T22:15:02.664ZINFOcrucible: [2] client re-new 0 jobs since flush 4012 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88712024-02-28T22:15:02.664ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88722024-02-28T22:15:03.667ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88732024-02-28T22:15:03.667ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88742024-02-28T22:15:03.667ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88752024-02-28T22:15:03.667ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88762024-02-28T22:15:03.667ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88772024-02-28T22:15:03.667ZINFOcrucible: [2] client re-new 0 jobs since flush 4012 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88782024-02-28T22:15:03.667ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8879 Feb 28 22:15:04.665 INFO request completed, latency_us: 209, response_code: 200, uri: /state/cid/2, method: GET, req_id: 39eba1e4-4232-49f1-b081-711901568c6d, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
8880 001/300 Write block 2485 len 24576 data: 2 2 2 2 2 2
8881 002/300 Write block 1615 len 24576 data: 4 3 2 2 2 2
8882 003/300 Read block 2242 len 24576
88832024-02-28T22:15:04.668ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88842024-02-28T22:15:04.668ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88852024-02-28T22:15:04.668ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88862024-02-28T22:15:04.668ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88872024-02-28T22:15:04.668ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88882024-02-28T22:15:04.668ZINFOcrucible: [2] client re-new 3 jobs since flush 4012 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
88892024-02-28T22:15:04.668ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
8890 004/300 Write block 1996 len 4096 data: 3
8891 005/300 Write block 4313 len 40960 data: 3 3 3 2 2 4 3 4 4 3
8892 006/300 Read block 2165 len 24576
8893 007/300 Read block 2215 len 40960
8894 008/300 Write block 4792 len 32768 data: 2 2 3 3 4 4 5 4
8895 009/300 Read block 384 len 16384
8896 010/300 Write block 1753 len 8192 data: 6 5
8897 011/300 Flush
8898 012/300 Write block 12 len 8192 data: 3 3
8899 013/300 Write block 3116 len 40960 data: 2 2 3 3 4 4 6 5 5 3
8900 014/300 Flush
8901 015/300 Read block 2721 len 36864
8902 016/300 Write block 2798 len 32768 data: 4 4 4 3 3 3 2 2
8903 017/300 Write block 4411 len 36864 data: 2 2 2 2 2 3 3 4 4
8904 018/300 Read block 3496 len 40960
8905 019/300 Read block 2248 len 32768
8906 020/300 Write block 904 len 8192 data: 2 2
8907 021/300 Read block 1779 len 32768
8908 022/300 Write block 3626 len 28672 data: 3 4 4 4 4 3 4
8909 023/300 Write block 3813 len 16384 data: 2 2 4 4
8910 024/300 Write block 4106 len 20480 data: 3 3 2 2 2
8911 025/300 Read block 4017 len 12288
8912 026/300 Write block 1411 len 12288 data: 2 2 2
8913 027/300 Read block 2422 len 4096
8914 028/300 Flush
8915 029/300 Write block 1036 len 24576 data: 3 3 3 2 3 3
8916 030/300 Read block 1220 len 24576
8917 031/300 Read block 742 len 20480
8918 032/300 Write block 3327 len 40960 data: 3 3 3 2 2 2 2 2 3 2
8919 033/300 Write block 2613 len 12288 data: 2 2 3
8920 034/300 Write block 3833 len 8192 data: 2 2
8921 035/300 Read block 2550 len 20480
8922 036/300 Read block 939 len 32768
8923 037/300 Read block 2573 len 32768
8924 038/300 Read block 1912 len 36864
8925 039/300 Write block 4700 len 8192 data: 3 3
8926 040/300 Write block 2407 len 36864 data: 2 2 3 3 3 5 6 6 7
8927 041/300 Read block 4373 len 40960
8928 042/300 Write block 2109 len 32768 data: 3 3 2 2 2 3 3 4
8929 043/300 Read block 275 len 12288
8930 044/300 Read block 3476 len 20480
8931 045/300 Read block 1400 len 20480
8932 046/300 Read block 2064 len 36864
8933 047/300 Write block 1121 len 28672 data: 4 2 2 5 7 7 8
8934 048/300 Read block 270 len 4096
8935 049/300 Read block 789 len 36864
8936 050/300 Write block 941 len 32768 data: 3 3 3 2 3 3 3 3
8937 051/300 Write block 446 len 24576 data: 4 4 4 4 4 4
8938 052/300 Read block 4416 len 20480
8939 053/300 Flush
8940 054/300 Read block 4098 len 20480
8941 055/300 Write block 3733 len 28672 data: 4 5 5 4 3 3 3
8942 056/300 Read block 1052 len 16384
8943 057/300 Read block 3628 len 20480
8944 058/300 Read block 2222 len 24576
8945 059/300 Read block 3219 len 4096
8946 060/300 Write block 4137 len 8192 data: 5 4
8947 061/300 Read block 403 len 36864
8948 062/300 Read block 1364 len 16384
8949 063/300 Read block 3873 len 20480
8950 064/300 Write block 1196 len 40960 data: 5 4 3 3 3 4 5 3 5 4
8951 065/300 Write block 4536 len 36864 data: 3 3 2 2 2 2 2 2 2
8952 066/300 Write block 1699 len 4096 data: 3
8953 067/300 Read block 208 len 36864
8954 068/300 Write block 2886 len 28672 data: 5 5 5 6 8 7 6
8955 069/300 Read block 328 len 32768
8956 070/300 Read block 1191 len 16384
8957 071/300 Read block 3933 len 12288
8958 072/300 Flush
8959 073/300 Write block 1445 len 16384 data: 2 2 2 2
8960 074/300 Read block 4905 len 16384
8961 075/300 Write block 2863 len 32768 data: 3 2 2 2 2 2 2 2
8962 076/300 Read block 2847 len 36864
8963 077/300 Read block 4243 len 36864
8964 078/300 Read block 3496 len 20480
8965 079/300 Write block 2012 len 12288 data: 4 4 4
8966 080/300 Write block 2273 len 28672 data: 2 3 3 3 3 3 3
8967 081/300 Read block 2650 len 40960
8968 082/300 Read block 1530 len 12288
8969 083/300 Read block 1138 len 24576
8970 084/300 Read block 2043 len 24576
8971 085/300 Write block 2853 len 32768 data: 2 2 2 2 2 2 3 3
8972 086/300 Read block 2781 len 40960
8973 087/300 Write block 4429 len 40960 data: 3 2 3 4 4 4 5 5 4 4
8974 088/300 Read block 603 len 12288
8975 089/300 Flush
8976 090/300 Write block 1307 len 20480 data: 3 3 3 3 3
8977 091/300 Read block 1060 len 36864
8978 092/300 Read block 770 len 20480
8979 093/300 Write block 4788 len 8192 data: 4 3
8980 094/300 Read block 4597 len 28672
8981 095/300 Write block 112 len 32768 data: 2 2 3 3 3 5 5 5
8982 096/300 Flush
8983 097/300 Write block 460 len 40960 data: 4 5 4 2 3 3 3 5 5 5
8984 098/300 Write block 3554 len 16384 data: 4 5 4 4
8985 099/300 Read block 3871 len 40960
8986 100/300 Read block 3828 len 12288
8987 101/300 Read block 2737 len 16384
8988 102/300 Read block 3693 len 28672
8989 103/300 Write block 4949 len 8192 data: 3 3
8990 104/300 Read block 2430 len 12288
8991 105/300 Write block 2205 len 28672 data: 3 3 3 2 2 2 3
8992 106/300 Read block 4144 len 8192
8993 107/300 Write block 146 len 32768 data: 3 3 3 3 3 3 3 3
8994 108/300 Read block 467 len 40960
8995 109/300 Write block 1561 len 24576 data: 5 4 4 3 3 3
8996 110/300 Read block 4421 len 28672
8997 111/300 Read block 1535 len 36864
8998 112/300 Write block 4304 len 8192 data: 8 6
8999 113/300 Write block 692 len 40960 data: 3 3 2 2 3 2 2 2 2 2
9000 114/300 Read block 355 len 4096
9001 115/300 Write block 1044 len 36864 data: 2 2 2 2 3 3 3 3 3
9002 116/300 Read block 4383 len 36864
9003 117/300 Write block 3860 len 20480 data: 3 3 3 3 3
9004 118/300 Read block 3856 len 28672
9005 119/300 Read block 2824 len 24576
9006 120/300 Write block 1574 len 4096 data: 2
9007 121/300 Write block 4725 len 28672 data: 3 2 2 2 2 2 2
9008 122/300 Flush
9009 123/300 Read block 438 len 32768
9010 124/300 Read block 3149 len 32768
9011 125/300 Write block 2533 len 12288 data: 3 3 3
9012 126/300 Write block 3407 len 20480 data: 4 5 5 4 4
9013 127/300 Write block 103 len 32768 data: 2 2 2 2 2 2 2 2
9014 128/300 Read block 3251 len 8192
9015 129/300 Write block 1099 len 4096 data: 3
9016 130/300 Write block 415 len 36864 data: 5 5 5 5 5 6 4 4 6
9017 131/300 Read block 4818 len 8192
9018 132/300 Flush
9019 133/300 Write block 4849 len 20480 data: 2 2 2 2 2
9020 134/300 Read block 656 len 40960
9021 135/300 Read block 1826 len 40960
9022 136/300 Read block 4469 len 36864
9023 137/300 Write block 4105 len 16384 data: 3 4 4 3
9024 138/300 Write block 3720 len 36864 data: 3 4 4 3 4 5 5 5 5
9025 139/300 Read block 827 len 16384
9026 140/300 Read block 1683 len 12288
9027 141/300 Write block 3058 len 24576 data: 2 2 4 4 5 5
9028 142/300 Write block 2191 len 16384 data: 4 4 3 3
9029 143/300 Read block 1412 len 4096
9030 144/300 Flush
9031 145/300 Read block 60 len 20480
9032 146/300 Read block 1346 len 4096
9033 147/300 Write block 3689 len 24576 data: 2 2 2 3 3 3
9034 148/300 Write block 2687 len 24576 data: 5 5 4 2 2 4
9035 149/300 Write block 234 len 8192 data: 2 2
9036 150/300 Read block 696 len 12288
9037 151/300 Read block 3309 len 4096
9038 152/300 Read block 1158 len 32768
9039 153/300 Read block 2735 len 20480
9040 154/300 Write block 1791 len 36864 data: 3 3 3 3 4 4 4 5 4
9041 155/300 Write block 2875 len 8192 data: 3 3
9042 156/300 Read block 3694 len 40960
9043 157/300 Read block 2861 len 12288
9044 158/300 Write block 1148 len 24576 data: 2 2 3 4 4 4
9045 159/300 Write block 3058 len 24576 data: 3 3 5 5 6 6
9046 160/300 Write block 4942 len 28672 data: 5 5 4 5 4 3 3
9047 161/300 Write block 1746 len 20480 data: 3 3 5 4 4
9048 162/300 Flush
9049 163/300 Flush
9050 164/300 Write block 839 len 32768 data: 2 2 2 3 3 3 2 3
9051 165/300 Read block 2999 len 28672
9052 166/300 Write block 679 len 28672 data: 4 5 7 5 5 4 3
9053 167/300 Write block 4503 len 32768 data: 4 5 5 5 5 4 3 4
9054 168/300 Read block 2745 len 12288
9055 169/300 Read block 4439 len 20480
9056 170/300 Read block 4853 len 4096
9057 171/300 Read block 1059 len 16384
9058 172/300 Read block 2796 len 28672
9059 173/300 Flush
9060 174/300 Write block 4076 len 24576 data: 5 4 3 3 2 2
9061 175/300 Read block 3741 len 16384
9062 176/300 Read block 2129 len 16384
9063 177/300 Read block 4118 len 32768
9064 178/300 Write block 468 len 36864 data: 6 6 5 4 5 4 5 5 5
9065 179/300 Write block 4621 len 40960 data: 2 2 2 2 2 4 4 3 4 3
9066 180/300 Flush
9067 181/300 Write block 2791 len 20480 data: 4 4 3 4 4
9068 182/300 Read block 1355 len 4096
9069 183/300 Read block 2422 len 16384
9070 184/300 Read block 1386 len 28672
9071 185/300 Read block 4066 len 28672
9072 186/300 Read block 884 len 12288
9073 187/300 Read block 3331 len 20480
9074 188/300 Read block 4970 len 28672
9075 189/300 Write block 4779 len 36864 data: 2 2 2 3 3 3 3 3 3
9076 190/300 Read block 2414 len 32768
9077 191/300 Write block 732 len 36864 data: 2 2 2 2 2 2 2 2 2
9078 192/300 Read block 3063 len 16384
9079 193/300 Write block 4015 len 4096 data: 5
9080 194/300 Read block 820 len 28672
9081 195/300 Write block 2610 len 12288 data: 3 3 2
9082 196/300 Read block 2436 len 12288
9083 197/300 Write block 1000 len 24576 data: 5 5 4 4 4 3
9084 198/300 Write block 392 len 36864 data: 3 3 3 2 2 3 3 3 4
9085 199/300 Write block 778 len 24576 data: 3 3 3 3 3 3
9086 200/300 Flush
9087 201/300 Flush
9088 202/300 Read block 2711 len 8192
9089 203/300 Write block 4786 len 12288 data: 4 4 5
9090 204/300 Write block 4400 len 16384 data: 4 5 5 5
9091 205/300 Write block 3019 len 8192 data: 4 4
9092 206/300 Read block 4904 len 32768
9093 207/300 Read block 4996 len 16384
9094 208/300 Write block 2045 len 32768 data: 2 2 2 2 3 4 6 7
9095 209/300 Read block 4535 len 28672
9096 210/300 Write block 2841 len 28672 data: 3 3 4 4 3 3 3
9097 211/300 Write block 1783 len 16384 data: 2 2 2 2
9098 212/300 Write block 3383 len 8192 data: 4 2
9099 213/300 Read block 2683 len 36864
9100 214/300 Flush
9101 215/300 Read block 2077 len 4096
9102 216/300 Write block 4718 len 20480 data: 4 4 2 2 2
9103 217/300 Flush
9104 218/300 Write block 2081 len 16384 data: 2 2 2 3
9105 219/300 Read block 4017 len 24576
9106 220/300 Read block 4663 len 40960
9107 221/300 Read block 2885 len 28672
9108 222/300 Write block 4785 len 32768 data: 4 5 5 6 4 2 2 3
9109 223/300 Write block 1586 len 20480 data: 4 3 3 2 2
9110 224/300 Write block 4199 len 4096 data: 2
9111 225/300 Read block 59 len 28672
9112 226/300 Write block 1295 len 28672 data: 3 3 3 2 2 2 2
9113 227/300 Write block 962 len 8192 data: 6 6
9114 228/300 Read block 4284 len 16384
9115 229/300 Read block 4415 len 20480
9116 230/300 Write block 3053 len 12288 data: 2 2 2
9117 231/300 Flush
9118 232/300 Read block 4601 len 32768
9119 233/300 Write block 2495 len 8192 data: 2 2
9120 234/300 Read block 2408 len 40960
9121 235/300 Read block 440 len 20480
9122 236/300 Flush
9123 237/300 Read block 3067 len 40960
9124 238/300 Read block 2351 len 12288
9125 239/300 Read block 4164 len 40960
9126 240/300 Write block 3589 len 20480 data: 3 2 2 3 3
9127 241/300 Read block 176 len 40960
9128 242/300 Read block 493 len 16384
9129 243/300 Write block 4199 len 36864 data: 3 2 2 2 2 2 2 2 2
9130 244/300 Write block 4197 len 24576 data: 2 2 4 3 3 3
9131 245/300 Read block 3667 len 12288
9132 246/300 Write block 1977 len 40960 data: 4 5 5 3 4 4 3 3 3 3
9133 247/300 Write block 1896 len 12288 data: 4 4 2
9134 248/300 Read block 4769 len 24576
9135 249/300 Write block 4348 len 24576 data: 2 3 2 2 3 3
9136 250/300 Read block 4769 len 20480
9137 251/300 Flush
9138 252/300 Read block 1956 len 24576
9139 253/300 Flush
9140 254/300 Read block 991 len 28672
9141 255/300 Read block 4860 len 12288
9142 256/300 Read block 2550 len 20480
9143 257/300 Write block 1864 len 40960 data: 4 4 4 3 3 2 2 2 2 2
9144 258/300 Write block 3031 len 40960 data: 3 2 2 2 4 4 4 3 3 4
9145 259/300 Write block 1470 len 36864 data: 6 6 6 7 6 6 5 4 4
9146 260/300 Write block 734 len 8192 data: 3 3
9147 261/300 Read block 3228 len 40960
9148 262/300 Write block 1622 len 16384 data: 3 5 6 6
9149 263/300 Read block 351 len 24576
9150 264/300 Write block 3701 len 16384 data: 4 4 3 3
9151 265/300 Read block 469 len 16384
9152 266/300 Read block 2580 len 28672
9153 267/300 Read block 2459 len 24576
9154 268/300 Write block 1729 len 16384 data: 2 2 2 2
9155 269/300 Flush
9156 270/300 Read block 3444 len 24576
9157 271/300 Read block 3335 len 16384
9158 272/300 Read block 139 len 20480
9159 273/300 Read block 3814 len 28672
9160 274/300 Write block 2983 len 24576 data: 2 3 4 4 4 4
9161 275/300 Write block 979 len 32768 data: 6 5 5 5 3 2 2 2
9162 276/300 Read block 564 len 20480
9163 277/300 Read block 1099 len 24576
9164 278/300 Read block 4923 len 16384
9165 279/300 Read block 4553 len 40960
9166 280/300 Read block 4490 len 36864
9167 281/300 Write block 4332 len 40960 data: 4 3 3 5 4 2 2 2 2 2
9168 282/300 Read block 3909 len 8192
9169 283/300 Write block 2334 len 24576 data: 2 3 4 5 5 5
9170 284/300 Read block 2602 len 12288
9171 285/300 Write block 2522 len 32768 data: 3 3 3 2 3 4 4 4
9172 286/300 Write block 4789 len 8192 data: 5 3
9173 287/300 Read block 3079 len 28672
9174 288/300 Flush
9175 289/300 Write block 3696 len 4096 data: 3
9176 290/300 Read block 1486 len 16384
9177 291/300 Read block 836 len 8192
9178 292/300 Write block 948 len 12288 data: 4 4 4
9179 293/300 Write block 2685 len 12288 data: 6 5 6
9180 294/300 Flush
9181 295/300 Read block 3863 len 8192
9182 296/300 Flush
9183 297/300 Read block 2722 len 12288
9184 298/300 Write block 3815 len 4096 data: 5
9185 299/300 Write block 2718 len 32768 data: 4 4 4 4 4 4 4 4
9186 300/300 Read block 2511 len 16384
9187 Main task has work to do, go find it
9188 got dsc Start(2)
9189 start 2
9190 Feb 28 22:15:05.428 INFO request completed, latency_us: 214, response_code: 204, uri: /start/cid/2, method: POST, req_id: d5acae00-9506-4660-9422-25bae26e565e, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
9191 Replay: started 2, returned:Ok(())
9192 ----------------------------------------------------------------
9193 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
9194 States: Active Active Offline
9195 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
9196 3014 Acked 4013 Write 6 Done Done New true
9197 3015 Acked 4014 Write 6 Done Done New true
9198 3016 Acked 4015 Read 6 Done Done New true
9199 3017 Acked 4016 Write 1 Done Done New false
9200 3018 Acked 4017 Write 10 Done Done New false
9201 3019 Acked 4018 Read 6 Done Done New false
9202 3020 Acked 4019 Read 10 Done Done New false
9203 3021 Acked 4020 Write 8 Done Done New false
9204 3022 Acked 4021 Read 4 Done Done New false
9205 3023 Acked 4022 Write 2 Done Done New false
9206 3024 Acked 4023 Flush 0 Done Done New false
9207 3025 Acked 4024 Write 2 Done Done New false
9208 3026 Acked 4025 Write 10 Done Done New false
9209 3027 Acked 4026 Flush 0 Done Done New false
9210 3028 Acked 4027 Read 9 Done Done New false
9211 3029 Acked 4028 Write 8 Done Done New false
9212 3030 Acked 4029 Write 9 Done Done New false
9213 3031 Acked 4030 Read 10 Done Done New false
9214 3032 Acked 4031 Read 8 Done Done New false
9215 3033 Acked 4032 Write 2 Done Done New false
9216 3034 Acked 4033 Read 8 Done Done New false
9217 3035 Acked 4034 Write 7 Done Done New false
9218 3036 Acked 4035 Write 4 Done Done New false
9219 3037 Acked 4036 Write 5 Done Done New false
9220 3038 Acked 4037 Read 3 Done Done New false
9221 3039 Acked 4038 Write 3 Done Done New false
9222 3040 Acked 4039 Read 1 Done Done New false
9223 3041 Acked 4040 Flush 0 Done Done New false
9224 3042 Acked 4041 Write 6 Done Done New false
9225 3043 Acked 4042 Read 6 Done Done New false
9226 3044 Acked 4043 Read 5 Done Done New false
9227 3045 Acked 4044 Write 10 Done Done New false
9228 3046 Acked 4045 Write 3 Done Done New false
9229 3047 Acked 4046 Write 2 Done Done New false
9230 3048 Acked 4047 Read 5 Done Done New false
9231 3049 Acked 4048 Read 8 Done Done New false
9232 3050 Acked 4049 Read 8 Done Done New false
9233 3051 Acked 4050 Read 9 Done Done New false
9234 3052 Acked 4051 Write 2 Done Done New false
9235 3053 Acked 4052 Write 9 Done Done New false
9236 3054 Acked 4053 Read 10 Done Done New false
9237 3055 Acked 4054 Write 8 Done Done New false
9238 3056 Acked 4055 Read 3 Done Done New false
9239 3057 Acked 4056 Read 5 Done Done New false
9240 3058 Acked 4057 Read 5 Done Done New false
9241 3059 Acked 4058 Read 9 Done Done New false
9242 3060 Acked 4059 Write 7 Done Done New false
9243 3061 Acked 4060 Read 1 Done Done New false
9244 3062 Acked 4061 Read 9 Done Done New false
9245 3063 Acked 4062 Write 8 Done Done New false
9246 3064 Acked 4063 Write 6 Done Done New false
9247 3065 Acked 4064 Read 5 Done Done New false
9248 3066 Acked 4065 Flush 0 Done Done New false
9249 3067 Acked 4066 Read 5 Done Done New false
9250 3068 Acked 4067 Write 7 Done Done New false
9251 3069 Acked 4068 Read 4 Done Done New false
9252 3070 Acked 4069 Read 5 Done Done New false
9253 3071 Acked 4070 Read 6 Done Done New false
9254 3072 Acked 4071 Read 1 Done Done New false
9255 3073 Acked 4072 Write 2 Done Done New false
9256 3074 Acked 4073 Read 9 Done Done New false
9257 3075 Acked 4074 Read 4 Done Done New false
9258 3076 Acked 4075 Read 5 Done Done New false
9259 3077 Acked 4076 Write 10 Done Done New false
9260 3078 Acked 4077 Write 9 Done Done New false
9261 3079 Acked 4078 Write 1 Done Done New false
9262 3080 Acked 4079 Read 9 Done Done New false
9263 3081 Acked 4080 Write 7 Done Done New false
9264 3082 Acked 4081 Read 8 Done Done New false
9265 3083 Acked 4082 Read 4 Done Done New false
9266 3084 Acked 4083 Read 3 Done Done New false
9267 3085 Acked 4084 Flush 0 Done Done New false
9268 3086 Acked 4085 Write 4 Done Done New false
9269 3087 Acked 4086 Read 4 Done Done New false
9270 3088 Acked 4087 Write 8 Done Done New false
9271 3089 Acked 4088 Read 9 Done Done New false
9272 3090 Acked 4089 Read 9 Done Done New false
9273 3091 Acked 4090 Read 5 Done Done New false
9274 3092 Acked 4091 Write 3 Done Done New false
9275 3093 Acked 4092 Write 7 Done Done New false
9276 3094 Acked 4093 Read 10 Done Done New false
9277 3095 Acked 4094 Read 3 Done Done New false
9278 3096 Acked 4095 Read 6 Done Done New false
9279 3097 Acked 4096 Read 6 Done Done New false
9280 3098 Acked 4097 Write 8 Done Done New false
9281 3099 Acked 4098 Read 10 Done Done New false
9282 3100 Acked 4099 Write 10 Done Done New false
9283 3101 Acked 4100 Read 3 Done Done New false
9284 3102 Acked 4101 Flush 0 Done Done New false
9285 3103 Acked 4102 Write 5 Done Done New false
9286 3104 Acked 4103 Read 9 Done Done New false
9287 3105 Acked 4104 Read 5 Done Done New false
9288 3106 Acked 4105 Write 2 Done Done New false
9289 3107 Acked 4106 Read 7 Done Done New false
9290 3108 Acked 4107 Write 8 Done Done New false
9291 3109 Acked 4108 Flush 0 Done Done New false
9292 3110 Acked 4109 Write 10 Done Done New false
9293 3111 Acked 4110 Write 4 Done Done New false
9294 3112 Acked 4111 Read 10 Done Done New false
9295 3113 Acked 4112 Read 3 Done Done New false
9296 3114 Acked 4113 Read 4 Done Done New false
9297 3115 Acked 4114 Read 7 Done Done New false
9298 3116 Acked 4115 Write 2 Done Done New false
9299 3117 Acked 4116 Read 3 Done Done New false
9300 3118 Acked 4117 Write 7 Done Done New false
9301 3119 Acked 4118 Read 2 Done Done New false
9302 3120 Acked 4119 Write 8 Done Done New false
9303 3121 Acked 4120 Read 10 Done Done New false
9304 3122 Acked 4121 Write 6 Done Done New false
9305 3123 Acked 4122 Read 7 Done Done New false
9306 3124 Acked 4123 Read 9 Done Done New false
9307 3125 Acked 4124 Write 2 Done Done New false
9308 3126 Acked 4125 Write 10 Done Done New false
9309 3127 Acked 4126 Read 1 Done Done New false
9310 3128 Acked 4127 Write 9 Done Done New false
9311 3129 Acked 4128 Read 9 Done Done New false
9312 3130 Acked 4129 Write 5 Done Done New false
9313 3131 Acked 4130 Read 7 Done Done New false
9314 3132 Acked 4131 Read 6 Done Done New false
9315 3133 Acked 4132 Write 1 Done Done New false
9316 3134 Acked 4133 Write 7 Done Done New false
9317 3135 Acked 4134 Flush 0 Done Done New false
9318 3136 Acked 4135 Read 8 Done Done New false
9319 3137 Acked 4136 Read 8 Done Done New false
9320 3138 Acked 4137 Write 3 Done Done New false
9321 3139 Acked 4138 Write 5 Done Done New false
9322 3140 Acked 4139 Write 8 Done Done New false
9323 3141 Acked 4140 Read 2 Done Done New false
9324 3142 Acked 4141 Write 1 Done Done New false
9325 3143 Acked 4142 Write 9 Done Done New false
9326 3144 Acked 4143 Read 2 Done Done New false
9327 3145 Acked 4144 Flush 0 Done Done New false
9328 3146 Acked 4145 Write 5 Done Done New false
9329 3147 Acked 4146 Read 10 Done Done New false
9330 3148 Acked 4147 Read 10 Done Done New false
9331 3149 Acked 4148 Read 9 Done Done New false
9332 3150 Acked 4149 Write 4 Done Done New false
9333 3151 Acked 4150 Write 9 Done Done New false
9334 3152 Acked 4151 Read 4 Done Done New false
9335 3153 Acked 4152 Read 3 Done Done New false
9336 3154 Acked 4153 Write 6 Done Done New false
9337 3155 Acked 4154 Write 4 Done Done New false
9338 3156 Acked 4155 Read 1 Done Done New false
9339 3157 Acked 4156 Flush 0 Done Done New false
9340 3158 Acked 4157 Read 5 Done Done New false
9341 3159 Acked 4158 Read 1 Done Done New false
9342 3160 Acked 4159 Write 6 Done Done New false
9343 3161 Acked 4160 Write 6 Done Done New false
9344 3162 Acked 4161 Write 2 Done Done New false
9345 3163 Acked 4162 Read 3 Done Done New false
9346 3164 Acked 4163 Read 1 Done Done New false
9347 3165 Acked 4164 Read 8 Done Done New false
9348 3166 Acked 4165 Read 5 Done Done New false
9349 3167 Acked 4166 Write 9 Done Done New false
9350 3168 Acked 4167 Write 2 Done Done New false
9351 3169 Acked 4168 Read 10 Done Done New false
9352 3170 Acked 4169 Read 3 Done Done New false
9353 3171 Acked 4170 Write 6 Done Done New false
9354 3172 Acked 4171 Write 6 Done Done New false
9355 3173 Acked 4172 Write 7 Done Done New false
9356 3174 Acked 4173 Write 5 Done Done New false
9357 3175 Acked 4174 Flush 0 Done Done New false
9358 3176 Acked 4175 Flush 0 Done Done New false
9359 3177 Acked 4176 Write 8 Done Done New false
9360 3178 Acked 4177 Read 7 Done Done New false
9361 3179 Acked 4178 Write 7 Done Done New false
9362 3180 Acked 4179 Write 8 Done Done New false
9363 3181 Acked 4180 Read 3 Done Done New false
9364 3182 Acked 4181 Read 5 Done Done New false
9365 3183 Acked 4182 Read 1 Done Done New false
9366 3184 Acked 4183 Read 4 Done Done New false
9367 3185 Acked 4184 Read 7 Done Done New false
9368 3186 Acked 4185 Flush 0 Done Done New false
9369 3187 Acked 4186 Write 6 Done Done New false
9370 3188 Acked 4187 Read 4 Done Done New false
9371 3189 Acked 4188 Read 4 Done Done New false
9372 3190 Acked 4189 Read 8 Done Done New false
9373 3191 Acked 4190 Write 9 Done Done New false
9374 3192 Acked 4191 Write 10 Done Done New false
9375 3193 Acked 4192 Flush 0 Done Done New false
9376 3194 Acked 4193 Write 5 Done Done New false
9377 3195 Acked 4194 Read 1 Done Done New false
9378 3196 Acked 4195 Read 4 Done Done New false
9379 3197 Acked 4196 Read 7 Done Done New false
9380 3198 Acked 4197 Read 7 Done Done New false
9381 3199 Acked 4198 Read 3 Done Done New false
9382 3200 Acked 4199 Read 5 Done Done New false
9383 3201 Acked 4200 Read 7 Done Done New false
9384 3202 Acked 4201 Write 9 Done Done New false
9385 3203 Acked 4202 Read 8 Done Done New false
9386 3204 Acked 4203 Write 9 Done Done New false
9387 3205 Acked 4204 Read 4 Done Done New false
9388 3206 Acked 4205 Write 1 Done Done New false
9389 3207 Acked 4206 Read 7 Done Done New false
9390 3208 Acked 4207 Write 3 Done Done New false
9391 3209 Acked 4208 Read 3 Done Done New false
9392 3210 Acked 4209 Write 6 Done Done New false
9393 3211 Acked 4210 Write 9 Done Done New false
9394 3212 Acked 4211 Write 6 Done Done New false
9395 3213 Acked 4212 Flush 0 Done Done New false
9396 3214 Acked 4213 Flush 0 Done Done New false
9397 3215 Acked 4214 Read 2 Done Done New false
9398 3216 Acked 4215 Write 3 Done Done New false
9399 3217 Acked 4216 Write 4 Done Done New false
9400 3218 Acked 4217 Write 2 Done Done New false
9401 3219 Acked 4218 Read 8 Done Done New false
9402 3220 Acked 4219 Read 4 Done Done New false
9403 3221 Acked 4220 Write 8 Done Done New false
9404 3222 Acked 4221 Read 7 Done Done New false
9405 3223 Acked 4222 Write 7 Done Done New false
9406 3224 Acked 4223 Write 4 Done Done New false
9407 3225 Acked 4224 Write 2 Done Done New false
9408 3226 Acked 4225 Read 9 Done Done New false
9409 3227 Acked 4226 Flush 0 Done Done New false
9410 3228 Acked 4227 Read 1 Done Done New false
9411 3229 Acked 4228 Write 5 Done Done New false
9412 3230 Acked 4229 Flush 0 Done Done New false
9413 3231 Acked 4230 Write 4 Done Done New false
9414 3232 Acked 4231 Read 6 Done Done New false
9415 3233 Acked 4232 Read 10 Done Done New false
9416 3234 Acked 4233 Read 7 Done Done New false
9417 3235 Acked 4234 Write 8 Done Done New false
9418 3236 Acked 4235 Write 5 Done Done New false
9419 3237 Acked 4236 Write 1 Done Done New false
9420 3238 Acked 4237 Read 7 Done Done New false
9421 3239 Acked 4238 Write 7 Done Done New false
9422 3240 Acked 4239 Write 2 Done Done New false
9423 3241 Acked 4240 Read 4 Done Done New false
9424 3242 Acked 4241 Read 5 Done Done New false
9425 3243 Acked 4242 Write 3 Done Done New false
9426 3244 Acked 4243 Flush 0 Done Done New false
9427 3245 Acked 4244 Read 8 Done Done New false
9428 3246 Acked 4245 Write 2 Done Done New false
9429 3247 Acked 4246 Read 10 Done Done New false
9430 3248 Acked 4247 Read 5 Done Done New false
9431 3249 Acked 4248 Flush 0 Done Done New false
9432 3250 Acked 4249 Read 10 Done Done New false
9433 3251 Acked 4250 Read 3 Done Done New false
9434 3252 Acked 4251 Read 10 Done Done New false
9435 3253 Acked 4252 Write 5 Done Done New false
9436 3254 Acked 4253 Read 10 Done Done New false
9437 3255 Acked 4254 Read 4 Done Done New false
9438 3256 Acked 4255 Write 9 Done Done New false
9439 3257 Acked 4256 Write 6 Done Done New false
9440 3258 Acked 4257 Read 3 Done Done New false
9441 3259 Acked 4258 Write 10 Done Done New false
9442 3260 Acked 4259 Write 3 Done Done New false
9443 3261 Acked 4260 Read 6 Done Done New false
9444 3262 Acked 4261 Write 6 Done Done New false
9445 3263 Acked 4262 Read 5 Done Done New false
9446 3264 Acked 4263 Flush 0 Done Done New false
9447 3265 Acked 4264 Read 6 Done Done New false
9448 3266 Acked 4265 Flush 0 Done Done New false
9449 3267 Acked 4266 Read 7 Done Done New false
9450 3268 Acked 4267 Read 3 Done Done New false
9451 3269 Acked 4268 Read 5 Done Done New false
9452 3270 Acked 4269 Write 10 Done Done New false
9453 3271 Acked 4270 Write 10 Done Done New false
9454 3272 Acked 4271 Write 9 Done Done New false
9455 3273 Acked 4272 Write 2 Done Done New false
9456 3274 Acked 4273 Read 10 Done Done New false
9457 3275 Acked 4274 Write 4 Done Done New false
9458 3276 Acked 4275 Read 6 Done Done New false
9459 3277 Acked 4276 Write 4 Done Done New false
9460 3278 Acked 4277 Read 4 Done Done New false
9461 3279 Acked 4278 Read 7 Done Done New false
9462 3280 Acked 4279 Read 6 Done Done New false
9463 3281 Acked 4280 Write 4 Done Done New false
9464 3282 Acked 4281 Flush 0 Done Done New false
9465 3283 Acked 4282 Read 6 Done Done New false
9466 3284 Acked 4283 Read 4 Done Done New false
9467 3285 Acked 4284 Read 5 Done Done New false
9468 3286 Acked 4285 Read 7 Done Done New false
9469 3287 Acked 4286 Write 6 Done Done New false
9470 3288 Acked 4287 Write 8 Done Done New false
9471 3289 Acked 4288 Read 5 Done Done New false
9472 3290 Acked 4289 Read 6 Done Done New false
9473 3291 Acked 4290 Read 4 Done Done New false
9474 3292 Acked 4291 Read 10 Done Done New false
9475 3293 Acked 4292 Read 9 Done Done New false
9476 3294 Acked 4293 Write 10 Done Done New false
9477 3295 Acked 4294 Read 2 Done Done New false
9478 3296 Acked 4295 Write 6 Done Done New false
9479 3297 Acked 4296 Read 3 Done Done New false
9480 3298 Acked 4297 Write 8 Done Done New false
9481 3299 Acked 4298 Write 2 Done Done New false
9482 3300 Acked 4299 Read 7 Done Done New false
9483 3301 Acked 4300 Flush 0 Done Done New false
9484 3302 Acked 4301 Write 1 Done Done New false
9485 3303 Acked 4302 Read 4 Done Done New false
9486 3304 Acked 4303 Read 2 Done Done New false
9487 3305 Acked 4304 Write 3 Done Done New false
9488 3306 Acked 4305 Write 3 Done Done New false
9489 3307 Acked 4306 Flush 0 Done Done New false
9490 3308 Acked 4307 Read 2 Done Done New false
9491 3309 Acked 4308 Flush 0 Done Done New false
9492 3310 Acked 4309 Read 3 Done Done New false
9493 3311 Acked 4310 Write 1 Done Done New false
9494 3312 Acked 4311 Write 8 Done Done New false
9495 3313 Acked 4312 Read 4 Done Done New false
9496 STATES DS:0 DS:1 DS:2 TOTAL
9497 New 0 0 300 300
9498 Sent 0 0 0 0
9499 Done 300 300 0 600
9500 Skipped 0 0 0 0
9501 Error 0 0 0 0
9502 Last Flush: 4308 4308 4012
9503 Downstairs last five completed: 4012 4011 4010 4009 4008
9504 Upstairs last five completed: 3313 3312 3311 3310 3309
9505 CLIENT: Up:0 ds:300 act:2
95062024-02-28T22:15:05.669ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95072024-02-28T22:15:05.669ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95082024-02-28T22:15:05.669ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95092024-02-28T22:15:05.669ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95102024-02-28T22:15:05.669ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95112024-02-28T22:15:05.669ZINFOcrucible: [2] client re-new 301 jobs since flush 4012 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95122024-02-28T22:15:05.669ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95132024-02-28T22:15:06.670ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95142024-02-28T22:15:06.670ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95152024-02-28T22:15:06.670ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95162024-02-28T22:15:06.670ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95172024-02-28T22:15:06.670ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95182024-02-28T22:15:06.670ZINFOcrucible: [2] client re-new 301 jobs since flush 4012 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95192024-02-28T22:15:06.670ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95202024-02-28T22:15:07.671ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95212024-02-28T22:15:07.671ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95222024-02-28T22:15:07.671ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95232024-02-28T22:15:07.671ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95242024-02-28T22:15:07.671ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95252024-02-28T22:15:07.671ZINFOcrucible: [2] client re-new 301 jobs since flush 4012 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95262024-02-28T22:15:07.671ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95272024-02-28T22:15:08.672ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95282024-02-28T22:15:08.672ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95292024-02-28T22:15:08.672ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95302024-02-28T22:15:08.672ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95312024-02-28T22:15:08.672ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95322024-02-28T22:15:08.672ZINFOcrucible: [2] client re-new 301 jobs since flush 4012 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
95332024-02-28T22:15:08.673ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9534 ----------------------------------------------------------------
9535 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9536 States: Active Active Offline
9537 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
9538 3014 Acked 4013 Write 6 Done Done New true
9539 3015 Acked 4014 Write 6 Done Done New true
9540 3016 Acked 4015 Read 6 Done Done New true
9541 3017 Acked 4016 Write 1 Done Done New true
9542 3018 Acked 4017 Write 10 Done Done New true
9543 3019 Acked 4018 Read 6 Done Done New true
9544 3020 Acked 4019 Read 10 Done Done New true
9545 3021 Acked 4020 Write 8 Done Done New true
9546 3022 Acked 4021 Read 4 Done Done New true
9547 3023 Acked 4022 Write 2 Done Done New true
9548 3024 Acked 4023 Flush 0 Done Done New true
9549 3025 Acked 4024 Write 2 Done Done New true
9550 3026 Acked 4025 Write 10 Done Done New true
9551 3027 Acked 4026 Flush 0 Done Done New true
9552 3028 Acked 4027 Read 9 Done Done New true
9553 3029 Acked 4028 Write 8 Done Done New true
9554 3030 Acked 4029 Write 9 Done Done New true
9555 3031 Acked 4030 Read 10 Done Done New true
9556 3032 Acked 4031 Read 8 Done Done New true
9557 3033 Acked 4032 Write 2 Done Done New true
9558 3034 Acked 4033 Read 8 Done Done New true
9559 3035 Acked 4034 Write 7 Done Done New true
9560 3036 Acked 4035 Write 4 Done Done New true
9561 3037 Acked 4036 Write 5 Done Done New true
9562 3038 Acked 4037 Read 3 Done Done New true
9563 3039 Acked 4038 Write 3 Done Done New true
9564 3040 Acked 4039 Read 1 Done Done New true
9565 3041 Acked 4040 Flush 0 Done Done New true
9566 3042 Acked 4041 Write 6 Done Done New true
9567 3043 Acked 4042 Read 6 Done Done New true
9568 3044 Acked 4043 Read 5 Done Done New true
9569 3045 Acked 4044 Write 10 Done Done New true
9570 3046 Acked 4045 Write 3 Done Done New true
9571 3047 Acked 4046 Write 2 Done Done New true
9572 3048 Acked 4047 Read 5 Done Done New true
9573 3049 Acked 4048 Read 8 Done Done New true
9574 3050 Acked 4049 Read 8 Done Done New true
9575 3051 Acked 4050 Read 9 Done Done New true
9576 3052 Acked 4051 Write 2 Done Done New true
9577 3053 Acked 4052 Write 9 Done Done New true
9578 3054 Acked 4053 Read 10 Done Done New true
9579 3055 Acked 4054 Write 8 Done Done New true
9580 3056 Acked 4055 Read 3 Done Done New true
9581 3057 Acked 4056 Read 5 Done Done New true
9582 3058 Acked 4057 Read 5 Done Done New true
9583 3059 Acked 4058 Read 9 Done Done New true
9584 3060 Acked 4059 Write 7 Done Done New true
9585 3061 Acked 4060 Read 1 Done Done New true
9586 3062 Acked 4061 Read 9 Done Done New true
9587 3063 Acked 4062 Write 8 Done Done New true
9588 3064 Acked 4063 Write 6 Done Done New true
9589 3065 Acked 4064 Read 5 Done Done New true
9590 3066 Acked 4065 Flush 0 Done Done New true
9591 3067 Acked 4066 Read 5 Done Done New true
9592 3068 Acked 4067 Write 7 Done Done New true
9593 3069 Acked 4068 Read 4 Done Done New true
9594 3070 Acked 4069 Read 5 Done Done New true
9595 3071 Acked 4070 Read 6 Done Done New true
9596 3072 Acked 4071 Read 1 Done Done New true
9597 3073 Acked 4072 Write 2 Done Done New true
9598 3074 Acked 4073 Read 9 Done Done New true
9599 3075 Acked 4074 Read 4 Done Done New true
9600 3076 Acked 4075 Read 5 Done Done New true
9601 3077 Acked 4076 Write 10 Done Done New true
9602 3078 Acked 4077 Write 9 Done Done New true
9603 3079 Acked 4078 Write 1 Done Done New true
9604 3080 Acked 4079 Read 9 Done Done New true
9605 3081 Acked 4080 Write 7 Done Done New true
9606 3082 Acked 4081 Read 8 Done Done New true
9607 3083 Acked 4082 Read 4 Done Done New true
9608 3084 Acked 4083 Read 3 Done Done New true
9609 3085 Acked 4084 Flush 0 Done Done New true
9610 3086 Acked 4085 Write 4 Done Done New true
9611 3087 Acked 4086 Read 4 Done Done New true
9612 3088 Acked 4087 Write 8 Done Done New true
9613 3089 Acked 4088 Read 9 Done Done New true
9614 3090 Acked 4089 Read 9 Done Done New true
9615 3091 Acked 4090 Read 5 Done Done New true
9616 3092 Acked 4091 Write 3 Done Done New true
9617 3093 Acked 4092 Write 7 Done Done New true
9618 3094 Acked 4093 Read 10 Done Done New true
9619 3095 Acked 4094 Read 3 Done Done New true
9620 3096 Acked 4095 Read 6 Done Done New true
9621 3097 Acked 4096 Read 6 Done Done New true
9622 3098 Acked 4097 Write 8 Done Done New true
9623 3099 Acked 4098 Read 10 Done Done New true
9624 3100 Acked 4099 Write 10 Done Done New true
9625 3101 Acked 4100 Read 3 Done Done New true
9626 3102 Acked 4101 Flush 0 Done Done New true
9627 3103 Acked 4102 Write 5 Done Done New true
9628 3104 Acked 4103 Read 9 Done Done New true
9629 3105 Acked 4104 Read 5 Done Done New true
9630 3106 Acked 4105 Write 2 Done Done New true
9631 3107 Acked 4106 Read 7 Done Done New true
9632 3108 Acked 4107 Write 8 Done Done New true
9633 3109 Acked 4108 Flush 0 Done Done New true
9634 3110 Acked 4109 Write 10 Done Done New true
9635 3111 Acked 4110 Write 4 Done Done New true
9636 3112 Acked 4111 Read 10 Done Done New true
9637 3113 Acked 4112 Read 3 Done Done New true
9638 3114 Acked 4113 Read 4 Done Done New true
9639 3115 Acked 4114 Read 7 Done Done New true
9640 3116 Acked 4115 Write 2 Done Done New true
9641 3117 Acked 4116 Read 3 Done Done New true
9642 3118 Acked 4117 Write 7 Done Done New true
9643 3119 Acked 4118 Read 2 Done Done New true
9644 3120 Acked 4119 Write 8 Done Done New true
9645 3121 Acked 4120 Read 10 Done Done New true
9646 3122 Acked 4121 Write 6 Done Done New true
9647 3123 Acked 4122 Read 7 Done Done New true
9648 3124 Acked 4123 Read 9 Done Done New true
9649 3125 Acked 4124 Write 2 Done Done New true
9650 3126 Acked 4125 Write 10 Done Done New true
9651 3127 Acked 4126 Read 1 Done Done New true
9652 3128 Acked 4127 Write 9 Done Done New true
9653 3129 Acked 4128 Read 9 Done Done New true
9654 3130 Acked 4129 Write 5 Done Done New true
9655 3131 Acked 4130 Read 7 Done Done New true
9656 3132 Acked 4131 Read 6 Done Done New true
9657 3133 Acked 4132 Write 1 Done Done New true
9658 3134 Acked 4133 Write 7 Done Done New true
9659 3135 Acked 4134 Flush 0 Done Done New true
9660 3136 Acked 4135 Read 8 Done Done New true
9661 3137 Acked 4136 Read 8 Done Done New true
9662 3138 Acked 4137 Write 3 Done Done New true
9663 3139 Acked 4138 Write 5 Done Done New true
9664 3140 Acked 4139 Write 8 Done Done New true
9665 3141 Acked 4140 Read 2 Done Done New true
9666 3142 Acked 4141 Write 1 Done Done New true
9667 3143 Acked 4142 Write 9 Done Done New true
9668 3144 Acked 4143 Read 2 Done Done New true
9669 3145 Acked 4144 Flush 0 Done Done New true
9670 3146 Acked 4145 Write 5 Done Done New true
9671 3147 Acked 4146 Read 10 Done Done New true
9672 3148 Acked 4147 Read 10 Done Done New true
9673 3149 Acked 4148 Read 9 Done Done New true
9674 3150 Acked 4149 Write 4 Done Done New true
9675 3151 Acked 4150 Write 9 Done Done New true
9676 3152 Acked 4151 Read 4 Done Done New true
9677 3153 Acked 4152 Read 3 Done Done New true
9678 3154 Acked 4153 Write 6 Done Done New true
9679 3155 Acked 4154 Write 4 Done Done New true
9680 3156 Acked 4155 Read 1 Done Done New true
9681 3157 Acked 4156 Flush 0 Done Done New true
9682 3158 Acked 4157 Read 5 Done Done New true
9683 3159 Acked 4158 Read 1 Done Done New true
9684 3160 Acked 4159 Write 6 Done Done New true
9685 3161 Acked 4160 Write 6 Done Done New true
9686 3162 Acked 4161 Write 2 Done Done New true
9687 3163 Acked 4162 Read 3 Done Done New true
9688 3164 Acked 4163 Read 1 Done Done New true
9689 3165 Acked 4164 Read 8 Done Done New true
9690 3166 Acked 4165 Read 5 Done Done New true
9691 3167 Acked 4166 Write 9 Done Done New true
9692 3168 Acked 4167 Write 2 Done Done New true
9693 3169 Acked 4168 Read 10 Done Done New true
9694 3170 Acked 4169 Read 3 Done Done New true
9695 3171 Acked 4170 Write 6 Done Done New true
9696 3172 Acked 4171 Write 6 Done Done New true
9697 3173 Acked 4172 Write 7 Done Done New true
9698 3174 Acked 4173 Write 5 Done Done New true
9699 3175 Acked 4174 Flush 0 Done Done New true
9700 3176 Acked 4175 Flush 0 Done Done New true
9701 3177 Acked 4176 Write 8 Done Done New true
9702 3178 Acked 4177 Read 7 Done Done New true
9703 3179 Acked 4178 Write 7 Done Done New true
9704 3180 Acked 4179 Write 8 Done Done New true
9705 3181 Acked 4180 Read 3 Done Done New true
9706 3182 Acked 4181 Read 5 Done Done New true
9707 3183 Acked 4182 Read 1 Done Done New true
9708 3184 Acked 4183 Read 4 Done Done New true
9709 3185 Acked 4184 Read 7 Done Done New true
9710 3186 Acked 4185 Flush 0 Done Done New true
9711 3187 Acked 4186 Write 6 Done Done New true
9712 3188 Acked 4187 Read 4 Done Done New true
9713 3189 Acked 4188 Read 4 Done Done New true
9714 3190 Acked 4189 Read 8 Done Done New true
9715 3191 Acked 4190 Write 9 Done Done New true
9716 3192 Acked 4191 Write 10 Done Done New true
9717 3193 Acked 4192 Flush 0 Done Done New true
9718 3194 Acked 4193 Write 5 Done Done New true
9719 3195 Acked 4194 Read 1 Done Done New true
9720 3196 Acked 4195 Read 4 Done Done New true
9721 3197 Acked 4196 Read 7 Done Done New true
9722 3198 Acked 4197 Read 7 Done Done New true
9723 3199 Acked 4198 Read 3 Done Done New true
9724 3200 Acked 4199 Read 5 Done Done New true
9725 3201 Acked 4200 Read 7 Done Done New true
9726 3202 Acked 4201 Write 9 Done Done New true
9727 3203 Acked 4202 Read 8 Done Done New true
9728 3204 Acked 4203 Write 9 Done Done New true
9729 3205 Acked 4204 Read 4 Done Done New true
9730 3206 Acked 4205 Write 1 Done Done New true
9731 3207 Acked 4206 Read 7 Done Done New true
9732 3208 Acked 4207 Write 3 Done Done New true
9733 3209 Acked 4208 Read 3 Done Done New true
9734 3210 Acked 4209 Write 6 Done Done New true
9735 3211 Acked 4210 Write 9 Done Done New true
9736 3212 Acked 4211 Write 6 Done Done New true
9737 3213 Acked 4212 Flush 0 Done Done New true
9738 3214 Acked 4213 Flush 0 Done Done New true
9739 3215 Acked 4214 Read 2 Done Done New true
9740 3216 Acked 4215 Write 3 Done Done New true
9741 3217 Acked 4216 Write 4 Done Done New true
9742 3218 Acked 4217 Write 2 Done Done New true
9743 3219 Acked 4218 Read 8 Done Done New true
9744 3220 Acked 4219 Read 4 Done Done New true
9745 3221 Acked 4220 Write 8 Done Done New true
9746 3222 Acked 4221 Read 7 Done Done New true
9747 3223 Acked 4222 Write 7 Done Done New true
9748 3224 Acked 4223 Write 4 Done Done New true
9749 3225 Acked 4224 Write 2 Done Done New true
9750 3226 Acked 4225 Read 9 Done Done New true
9751 3227 Acked 4226 Flush 0 Done Done New true
9752 3228 Acked 4227 Read 1 Done Done New true
9753 3229 Acked 4228 Write 5 Done Done New true
9754 3230 Acked 4229 Flush 0 Done Done New true
9755 3231 Acked 4230 Write 4 Done Done New true
9756 3232 Acked 4231 Read 6 Done Done New true
9757 3233 Acked 4232 Read 10 Done Done New true
9758 3234 Acked 4233 Read 7 Done Done New true
9759 3235 Acked 4234 Write 8 Done Done New true
9760 3236 Acked 4235 Write 5 Done Done New true
9761 3237 Acked 4236 Write 1 Done Done New true
9762 3238 Acked 4237 Read 7 Done Done New true
9763 3239 Acked 4238 Write 7 Done Done New true
9764 3240 Acked 4239 Write 2 Done Done New true
9765 3241 Acked 4240 Read 4 Done Done New true
9766 3242 Acked 4241 Read 5 Done Done New true
9767 3243 Acked 4242 Write 3 Done Done New true
9768 3244 Acked 4243 Flush 0 Done Done New true
9769 3245 Acked 4244 Read 8 Done Done New true
9770 3246 Acked 4245 Write 2 Done Done New true
9771 3247 Acked 4246 Read 10 Done Done New true
9772 3248 Acked 4247 Read 5 Done Done New true
9773 3249 Acked 4248 Flush 0 Done Done New true
9774 3250 Acked 4249 Read 10 Done Done New true
9775 3251 Acked 4250 Read 3 Done Done New true
9776 3252 Acked 4251 Read 10 Done Done New true
9777 3253 Acked 4252 Write 5 Done Done New true
9778 3254 Acked 4253 Read 10 Done Done New true
9779 3255 Acked 4254 Read 4 Done Done New true
9780 3256 Acked 4255 Write 9 Done Done New true
9781 3257 Acked 4256 Write 6 Done Done New true
9782 3258 Acked 4257 Read 3 Done Done New true
9783 3259 Acked 4258 Write 10 Done Done New true
9784 3260 Acked 4259 Write 3 Done Done New true
9785 3261 Acked 4260 Read 6 Done Done New true
9786 3262 Acked 4261 Write 6 Done Done New true
9787 3263 Acked 4262 Read 5 Done Done New true
9788 3264 Acked 4263 Flush 0 Done Done New true
9789 3265 Acked 4264 Read 6 Done Done New true
9790 3266 Acked 4265 Flush 0 Done Done New true
9791 3267 Acked 4266 Read 7 Done Done New true
9792 3268 Acked 4267 Read 3 Done Done New true
9793 3269 Acked 4268 Read 5 Done Done New true
9794 3270 Acked 4269 Write 10 Done Done New true
9795 3271 Acked 4270 Write 10 Done Done New true
9796 3272 Acked 4271 Write 9 Done Done New true
9797 3273 Acked 4272 Write 2 Done Done New true
9798 3274 Acked 4273 Read 10 Done Done New true
9799 3275 Acked 4274 Write 4 Done Done New true
9800 3276 Acked 4275 Read 6 Done Done New true
9801 3277 Acked 4276 Write 4 Done Done New true
9802 3278 Acked 4277 Read 4 Done Done New true
9803 3279 Acked 4278 Read 7 Done Done New true
9804 3280 Acked 4279 Read 6 Done Done New true
9805 3281 Acked 4280 Write 4 Done Done New true
9806 3282 Acked 4281 Flush 0 Done Done New true
9807 3283 Acked 4282 Read 6 Done Done New true
9808 3284 Acked 4283 Read 4 Done Done New true
9809 3285 Acked 4284 Read 5 Done Done New true
9810 3286 Acked 4285 Read 7 Done Done New true
9811 3287 Acked 4286 Write 6 Done Done New true
9812 3288 Acked 4287 Write 8 Done Done New true
9813 3289 Acked 4288 Read 5 Done Done New true
9814 3290 Acked 4289 Read 6 Done Done New true
9815 3291 Acked 4290 Read 4 Done Done New true
9816 3292 Acked 4291 Read 10 Done Done New true
9817 3293 Acked 4292 Read 9 Done Done New true
9818 3294 Acked 4293 Write 10 Done Done New true
9819 3295 Acked 4294 Read 2 Done Done New true
9820 3296 Acked 4295 Write 6 Done Done New true
9821 3297 Acked 4296 Read 3 Done Done New true
9822 3298 Acked 4297 Write 8 Done Done New true
9823 3299 Acked 4298 Write 2 Done Done New true
9824 3300 Acked 4299 Read 7 Done Done New true
9825 3301 Acked 4300 Flush 0 Done Done New true
9826 3302 Acked 4301 Write 1 Done Done New true
9827 3303 Acked 4302 Read 4 Done Done New true
9828 3304 Acked 4303 Read 2 Done Done New true
9829 3305 Acked 4304 Write 3 Done Done New true
9830 3306 Acked 4305 Write 3 Done Done New true
9831 3307 Acked 4306 Flush 0 Done Done New true
9832 3308 Acked 4307 Read 2 Done Done New true
9833 3309 Acked 4308 Flush 0 Done Done New true
9834 3310 Acked 4309 Read 3 Done Done New true
9835 3311 Acked 4310 Write 1 Done Done New true
9836 3312 Acked 4311 Write 8 Done Done New true
9837 3313 Acked 4312 Read 4 Done Done New true
9838 3314 Acked 4313 Flush 0 Done Done New true
9839 STATES DS:0 DS:1 DS:2 TOTAL
9840 New 0 0 301 301
9841 Sent 0 0 0 0
9842 Done 301 301 0 602
9843 Skipped 0 0 0 0
9844 Error 0 0 0 0
9845 Last Flush: 4313 4313 4012
9846 Downstairs last five completed: 4012 4011 4010 4009 4008
9847 Upstairs last five completed: 3314 3313 3312 3311 3310
9848 CLIENT: Up:0 ds:301 act:2
98492024-02-28T22:15:09.674ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98502024-02-28T22:15:09.674ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98512024-02-28T22:15:09.674ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98522024-02-28T22:15:09.674ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98532024-02-28T22:15:09.674ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98542024-02-28T22:15:09.674ZINFOcrucible: [2] client re-new 301 jobs since flush 4012 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98552024-02-28T22:15:09.674ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9856 [8830] got start action
9857 [8830] I am going to restart
9858 Starting downstairs at port 8830
9859 Make output file at "/tmp/dsc/downstairs-8830.txt"
9860 [8830][2] reports Starting
98612024-02-28T22:15:10.675ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98622024-02-28T22:15:10.675ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98632024-02-28T22:15:10.675ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98642024-02-28T22:15:10.675ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98652024-02-28T22:15:10.675ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98662024-02-28T22:15:10.675ZINFOcrucible: [2] client re-new 301 jobs since flush 4012 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9867 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1175)
98682024-02-28T22:15:10.675ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9869 [8830][2] reports Running
98702024-02-28T22:15:11.676ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98712024-02-28T22:15:11.676ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98722024-02-28T22:15:11.678ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98732024-02-28T22:15:11.678ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98742024-02-28T22:15:11.678ZINFOcrucible: send last flush ID to this DS: 4012 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98752024-02-28T22:15:11.678ZINFOcrucible: Replied this last flush ID: 4012 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98762024-02-28T22:15:11.678ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98772024-02-28T22:15:11.678ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98782024-02-28T22:15:11.678ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98792024-02-28T22:15:11.678ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9880 ----------------------------------------------------------------
9881 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
9882 Downstairs last five completed: 4313 4312 4311 4310 4309
9883 Upstairs last five completed: 3314 3313 3312 3311 3310
9884 CLIENT: Up:0 ds:0 act:3
9885 Replay: All jobs finished, all DS active.
9886 Main task has work to do, go find it
9887 got dsc Stop(0)
9888 stop 0
9889 Feb 28 22:15:13.434 INFO request completed, [8810] Got stop action so:false kr:false
9890 latency_us: 348, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 1de10fdf-ec7b-48f9-8731-933fdba014fc, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
98912024-02-28T22:15:13.434ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98922024-02-28T22:15:13.434ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98932024-02-28T22:15:13.434ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98942024-02-28T22:15:13.434ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98952024-02-28T22:15:13.434ZINFOcrucible: [0] client re-new 0 jobs since flush 4313 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
98962024-02-28T22:15:13.434ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9897 Feb 28 22:15:13.434 INFO request completed, latency_us: 167, response_code: 200, uri: /state/cid/0, method: GET, req_id: f1c85701-fc1c-4848-9c06-69bd96c51996, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
9898 [8810] Exited with: ExitStatus(unix_wait_status(9))
9899 [8810][0] reports Exit
99002024-02-28T22:15:14.435ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99012024-02-28T22:15:14.435ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99022024-02-28T22:15:14.435ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99032024-02-28T22:15:14.435ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99042024-02-28T22:15:14.435ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99052024-02-28T22:15:14.435ZINFOcrucible: [0] client re-new 0 jobs since flush 4313 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99062024-02-28T22:15:14.435ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99072024-02-28T22:15:15.436ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99082024-02-28T22:15:15.436ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99092024-02-28T22:15:15.436ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99102024-02-28T22:15:15.436ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99112024-02-28T22:15:15.436ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99122024-02-28T22:15:15.436ZINFOcrucible: [0] client re-new 0 jobs since flush 4313 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99132024-02-28T22:15:15.436ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99142024-02-28T22:15:16.437ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99152024-02-28T22:15:16.438ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99162024-02-28T22:15:16.438ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99172024-02-28T22:15:16.438ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99182024-02-28T22:15:16.438ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99192024-02-28T22:15:16.438ZINFOcrucible: [0] client re-new 0 jobs since flush 4313 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99202024-02-28T22:15:16.438ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9921 Feb 28 22:15:17.437 INFO request completed, latency_us: 204, response_code: 200, uri: /state/cid/0, method: GET, req_id: 1e85d259-c5f4-43d4-9bca-ba12918342bc, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
9922 001/300 Read block 2466 len 32768
9923 002/300 Read block 1509 len 24576
99242024-02-28T22:15:17.439ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99252024-02-28T22:15:17.439ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99262024-02-28T22:15:17.439ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99272024-02-28T22:15:17.439ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99282024-02-28T22:15:17.439ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99292024-02-28T22:15:17.439ZINFOcrucible: [0] client re-new 2 jobs since flush 4313 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
99302024-02-28T22:15:17.439ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
9931 003/300 Write block 1770 len 24576 data: 3 3 3 3 3 3
9932 004/300 Read block 30 len 4096
9933 005/300 Write block 4167 len 4096 data: 6
9934 006/300 Read block 1403 len 40960
9935 007/300 Read block 2849 len 36864
9936 008/300 Write block 4574 len 28672 data: 3 2 2 2 3 3 3
9937 009/300 Write block 4380 len 32768 data: 2 3 3 3 4 3 3 3
9938 010/300 Read block 355 len 8192
9939 011/300 Read block 4550 len 28672
9940 012/300 Write block 1803 len 4096 data: 5
9941 013/300 Read block 3890 len 8192
9942 014/300 Flush
9943 015/300 Write block 1888 len 20480 data: 4 3 4 5 5
9944 016/300 Read block 1128 len 20480
9945 017/300 Read block 2259 len 24576
9946 018/300 Read block 4221 len 20480
9947 019/300 Read block 1820 len 24576
9948 020/300 Read block 2078 len 24576
9949 021/300 Read block 3398 len 16384
9950 022/300 Read block 4497 len 16384
9951 023/300 Flush
9952 024/300 Write block 3719 len 16384 data: 3 4 5 5
9953 025/300 Read block 641 len 24576
9954 026/300 Write block 1066 len 32768 data: 4 4 3 2 2 2 3 3
9955 027/300 Read block 2427 len 16384
9956 028/300 Write block 3265 len 12288 data: 2 2 2
9957 029/300 Write block 1426 len 12288 data: 3 3 3
9958 030/300 Write block 268 len 4096 data: 3
9959 031/300 Read block 3679 len 20480
9960 032/300 Write block 4196 len 32768 data: 2 3 3 5 4 4 4 3
9961 033/300 Read block 4842 len 24576
9962 034/300 Write block 1602 len 8192 data: 4 5
9963 035/300 Write block 2062 len 20480 data: 3 3 2 2 3
9964 036/300 Write block 4345 len 16384 data: 2 2 3 3
9965 037/300 Write block 301 len 12288 data: 3 3 2
9966 038/300 Write block 687 len 4096 data: 3
9967 039/300 Read block 394 len 8192
9968 040/300 Read block 633 len 36864
9969 041/300 Read block 2916 len 4096
9970 042/300 Read block 2160 len 28672
9971 043/300 Read block 4550 len 40960
9972 044/300 Read block 3933 len 20480
9973 045/300 Read block 607 len 20480
9974 046/300 Read block 2404 len 4096
9975 047/300 Write block 1927 len 12288 data: 2 2 2
9976 048/300 Write block 3826 len 20480 data: 4 4 4 3 2
9977 049/300 Write block 3803 len 20480 data: 2 2 2 2 2
9978 050/300 Read block 729 len 20480
9979 051/300 Read block 4443 len 16384
9980 052/300 Read block 1281 len 20480
9981 053/300 Write block 220 len 28672 data: 3 3 4 3 3 3 3
9982 054/300 Read block 3587 len 20480
9983 055/300 Write block 3036 len 36864 data: 5 5 4 4 5 5 5 5 5
9984 056/300 Write block 1937 len 24576 data: 3 4 4 4 4 3
9985 057/300 Read block 267 len 40960
9986 058/300 Write block 1608 len 36864 data: 4 3 3 4 3 3 3 5 4
9987 059/300 Read block 1549 len 28672
9988 060/300 Read block 2209 len 20480
9989 061/300 Write block 1561 len 28672 data: 6 5 5 4 4 4 2
9990 062/300 Read block 2789 len 40960
9991 063/300 Read block 1166 len 24576
9992 064/300 Read block 1423 len 20480
9993 065/300 Read block 3775 len 16384
9994 066/300 Write block 2715 len 32768 data: 2 2 2 5 5 5 5 5
9995 067/300 Flush
9996 068/300 Flush
9997 069/300 Read block 4842 len 20480
9998 070/300 Write block 2869 len 24576 data: 3 3 3 4 3 2
9999 071/300 Read block 228 len 12288
10000 072/300 Flush
10001 073/300 Read block 3494 len 20480
10002 074/300 Write block 3173 len 8192 data: 3 3
10003 075/300 Flush
10004 076/300 Write block 4460 len 12288 data: 6 7 6
10005 077/300 Flush
10006 078/300 Read block 3393 len 12288
10007 079/300 Read block 1683 len 16384
10008 080/300 Write block 4496 len 36864 data: 7 6 7 9 6 4 4 5 6
10009 081/300 Read block 1010 len 36864
10010 082/300 Read block 3481 len 8192
10011 083/300 Read block 2156 len 16384
10012 084/300 Read block 2824 len 36864
10013 085/300 Write block 3047 len 40960 data: 3 3 2 2 2 2 3 3 3 2
10014 086/300 Write block 4136 len 4096 data: 3
10015 087/300 Read block 791 len 32768
10016 088/300 Read block 3496 len 40960
10017 089/300 Read block 32 len 24576
10018 090/300 Read block 1543 len 8192
10019 091/300 Write block 4114 len 20480 data: 3 4 4 3 3
10020 092/300 Write block 746 len 8192 data: 3 3
10021 093/300 Read block 4236 len 24576
10022 094/300 Write block 1652 len 12288 data: 2 2 2
10023 095/300 Read block 1724 len 32768
10024 096/300 Write block 2333 len 36864 data: 2 3 4 5 6 6 6 4 3
10025 097/300 Flush
10026 098/300 Read block 3603 len 4096
10027 099/300 Write block 789 len 28672 data: 2 3 3 3 3 3 3
10028 100/300 Read block 1394 len 4096
10029 101/300 Flush
10030 102/300 Write block 1646 len 28672 data: 3 2 2 2 2 2 3
10031 103/300 Write block 4132 len 12288 data: 3 3 3
10032 104/300 Read block 2551 len 28672
10033 105/300 Read block 704 len 28672
10034 106/300 Write block 4513 len 24576 data: 3 3 4 3 3 2
10035 107/300 Read block 2818 len 4096
10036 108/300 Write block 1951 len 8192 data: 4 3
10037 109/300 Flush
10038 110/300 Read block 4530 len 40960
10039 111/300 Write block 4077 len 16384 data: 5 4 4 3
10040 112/300 Read block 3954 len 4096
10041 113/300 Write block 671 len 20480 data: 3 3 3 4 4
10042 114/300 Write block 1629 len 4096 data: 4
10043 115/300 Read block 4646 len 24576
10044 116/300 Write block 1520 len 16384 data: 5 4 4 4
10045 117/300 Read block 3017 len 28672
10046 118/300 Write block 4768 len 16384 data: 2 3 3 3
10047 119/300 Flush
10048 120/300 Write block 4928 len 8192 data: 3 4
10049 121/300 Read block 1641 len 32768
10050 122/300 Write block 2035 len 8192 data: 7 8
10051 123/300 Write block 4231 len 40960 data: 3 3 2 2 2 2 3 5 5 5
10052 124/300 Read block 2989 len 4096
10053 125/300 Flush
10054 126/300 Flush
10055 127/300 Write block 145 len 16384 data: 3 4 4 4
10056 128/300 Flush
10057 129/300 Write block 2244 len 8192 data: 5 5
10058 130/300 Read block 3331 len 12288
10059 131/300 Read block 3732 len 4096
10060 132/300 Read block 400 len 20480
10061 133/300 Read block 757 len 8192
10062 134/300 Read block 2325 len 40960
10063 135/300 Read block 4778 len 28672
10064 136/300 Write block 2732 len 40960 data: 3 2 2 2 2 2 2 2 2 2
10065 137/300 Write block 2455 len 36864 data: 3 3 3 4 4 3 3 4 4
10066 138/300 Read block 403 len 4096
10067 139/300 Read block 4955 len 36864
10068 140/300 Read block 3768 len 4096
10069 141/300 Flush
10070 142/300 Flush
10071 143/300 Write block 2620 len 16384 data: 5 6 5 4
10072 144/300 Flush
10073 145/300 Write block 1543 len 4096 data: 3
10074 146/300 Write block 3821 len 36864 data: 3 4 4 4 4 5 5 5 4
10075 147/300 Write block 131 len 24576 data: 4 5 4 6 4 4
10076 148/300 Write block 1856 len 24576 data: 3 3 4 4 4 4
10077 149/300 Write block 2424 len 28672 data: 3 3 2 2 4 5 5
10078 150/300 Read block 1145 len 8192
10079 151/300 Write block 1769 len 12288 data: 3 4 4
10080 152/300 Read block 1700 len 12288
10081 153/300 Read block 2227 len 32768
10082 154/300 Write block 3380 len 8192 data: 3 3
10083 155/300 Read block 1547 len 8192
10084 156/300 Read block 1612 len 16384
10085 157/300 Write block 178 len 16384 data: 2 3 4 3
10086 158/300 Write block 4604 len 16384 data: 2 2 3 3
10087 159/300 Write block 1776 len 32768 data: 3 3 3 3 3 2 2 3
10088 160/300 Read block 2342 len 4096
10089 161/300 Read block 3607 len 8192
10090 162/300 Write block 1135 len 28672 data: 5 4 4 4 4 6 6
10091 163/300 Write block 1905 len 8192 data: 2 3
10092 164/300 Read block 836 len 20480
10093 165/300 Write block 2799 len 24576 data: 5 5 4 4 4 3
10094 166/300 Write block 4839 len 32768 data: 2 2 2 2 2 2 2 2
10095 167/300 Read block 291 len 36864
10096 168/300 Write block 4986 len 24576 data: 5 5 4 4 4 4
10097 169/300 Write block 4144 len 16384 data: 5 4 4 4
10098 170/300 Read block 1365 len 20480
10099 171/300 Read block 1106 len 12288
10100 172/300 Read block 1874 len 12288
10101 173/300 Read block 3876 len 28672
10102 174/300 Read block 1311 len 32768
10103 175/300 Read block 1203 len 28672
10104 176/300 Read block 2528 len 20480
10105 177/300 Read block 755 len 24576
10106 178/300 Read block 4462 len 8192
10107 179/300 Read block 1932 len 8192
10108 180/300 Read block 3969 len 4096
10109 181/300 Read block 2436 len 16384
10110 182/300 Read block 2495 len 36864
10111 183/300 Flush
10112 184/300 Write block 1732 len 28672 data: 3 3 3 3 3 2 3
10113 185/300 Read block 3495 len 4096
10114 186/300 Write block 4564 len 8192 data: 3 4
10115 187/300 Write block 1781 len 16384 data: 3 3 4 3
10116 188/300 Read block 946 len 24576
10117 189/300 Write block 4075 len 24576 data: 5 6 6 5 5 4
10118 190/300 Write block 1541 len 12288 data: 3 3 4
10119 191/300 Read block 4889 len 20480
10120 192/300 Write block 1322 len 20480 data: 3 3 3 4 4
10121 193/300 Read block 3637 len 40960
10122 194/300 Read block 3135 len 28672
10123 195/300 Write block 347 len 32768 data: 4 3 3 3 3 2 4 4
10124 196/300 Write block 1372 len 28672 data: 2 2 2 2 2 2 2
10125 197/300 Read block 143 len 8192
10126 198/300 Read block 1493 len 36864
10127 199/300 Write block 2705 len 4096 data: 3
10128 200/300 Write block 750 len 40960 data: 4 4 4 3 4 4 4 6 5 6
10129 201/300 Write block 2857 len 36864 data: 3 3 4 4 3 3 4 3 3
10130 202/300 Write block 2936 len 40960 data: 3 3 3 5 5 4 4 3 2 2
10131 203/300 Read block 3156 len 4096
10132 204/300 Flush
10133 205/300 Read block 4394 len 32768
10134 206/300 Write block 3177 len 24576 data: 4 6 6 5 4 4
10135 207/300 Flush
10136 208/300 Read block 915 len 36864
10137 209/300 Read block 1211 len 28672
10138 210/300 Write block 2831 len 28672 data: 2 2 2 2 3 3 3
10139 211/300 Write block 72 len 12288 data: 3 3 3
10140 212/300 Read block 3805 len 36864
10141 213/300 Read block 4606 len 32768
10142 214/300 Read block 2946 len 32768
10143 215/300 Write block 2585 len 4096 data: 3
10144 216/300 Write block 3998 len 24576 data: 5 5 5 3 3 3
10145 217/300 Write block 4476 len 40960 data: 2 2 2 2 2 2 2 2 2 2
10146 218/300 Read block 2806 len 4096
10147 219/300 Write block 664 len 40960 data: 5 3 3 3 3 3 5 4 4 4
10148 220/300 Write block 1187 len 28672 data: 3 3 3 5 6 6 4
10149 221/300 Flush
10150 222/300 Write block 4281 len 24576 data: 3 3 3 3 3 2
10151 223/300 Write block 101 len 4096 data: 3
10152 224/300 Read block 2684 len 36864
10153 225/300 Write block 561 len 12288 data: 3 3 3
10154 226/300 Read block 4053 len 28672
10155 227/300 Flush
10156 228/300 Read block 3636 len 36864
10157 229/300 Read block 908 len 36864
10158 230/300 Write block 2579 len 28672 data: 4 4 4 4 3 3 4
10159 231/300 Read block 2816 len 16384
10160 232/300 Write block 1565 len 8192 data: 5 5
10161 233/300 Read block 1616 len 8192
10162 234/300 Write block 3147 len 24576 data: 4 6 5 5 6 6
10163 235/300 Write block 1399 len 32768 data: 3 3 2 2 3 3 3 3
10164 236/300 Read block 1759 len 32768
10165 237/300 Read block 2255 len 16384
10166 238/300 Read block 3939 len 36864
10167 239/300 Write block 545 len 24576 data: 3 3 3 2 2 2
10168 240/300 Read block 1873 len 40960
10169 241/300 Read block 2631 len 36864
10170 242/300 Read block 728 len 8192
10171 243/300 Write block 2049 len 32768 data: 4 5 7 8 7 6 6 5
10172 244/300 Write block 3440 len 40960 data: 5 3 2 3 4 3 4 5 5 5
10173 245/300 Read block 4663 len 32768
10174 246/300 Read block 2914 len 16384
10175 247/300 Write block 4324 len 4096 data: 2
10176 248/300 Read block 976 len 36864
10177 249/300 Read block 4498 len 20480
10178 250/300 Write block 570 len 36864 data: 3 3 3 3 3 4 2 3 2
10179 251/300 Write block 538 len 24576 data: 3 4 4 3 3 3
10180 252/300 Read block 700 len 16384
10181 253/300 Write block 4799 len 36864 data: 5 4 6 6 6 6 5 5 5
10182 254/300 Read block 2009 len 4096
10183 255/300 Write block 4590 len 36864 data: 4 4 3 2 2 2 3 3 3
10184 256/300 Read block 3864 len 12288
10185 257/300 Write block 3207 len 40960 data: 4 4 4 3 4 5 5 5 4 4
10186 258/300 Read block 3450 len 8192
10187 259/300 Read block 4040 len 4096
10188 260/300 Read block 2028 len 4096
10189 261/300 Flush
10190 262/300 Write block 3919 len 40960 data: 2 2 3 3 3 3 3 3 3 3
10191 263/300 Read block 4661 len 4096
10192 264/300 Write block 4047 len 24576 data: 3 3 3 3 3 3
10193 265/300 Write block 4887 len 24576 data: 4 4 4 2 2 3
10194 266/300 Read block 1733 len 28672
10195 267/300 Read block 219 len 40960
10196 268/300 Write block 2190 len 12288 data: 4 5 5
10197 269/300 Read block 3391 len 24576
10198 270/300 Flush
10199 271/300 Read block 2786 len 4096
10200 272/300 Flush
10201 273/300 Write block 186 len 16384 data: 4 4 3 3
10202 274/300 Read block 2660 len 4096
10203 275/300 Write block 317 len 28672 data: 5 4 3 3 3 3 3
10204 276/300 Read block 569 len 4096
10205 277/300 Write block 1334 len 12288 data: 5 4 3
10206 278/300 Write block 433 len 36864 data: 4 3 3 3 2 2 2 4 4
10207 279/300 Read block 3777 len 20480
10208 280/300 Write block 529 len 16384 data: 3 3 4 4
10209 281/300 Write block 1817 len 24576 data: 2 2 3 3 2 2
10210 282/300 Read block 28 len 12288
10211 283/300 Flush
10212 284/300 Write block 412 len 24576 data: 5 4 4 6 6 6
10213 285/300 Write block 2236 len 24576 data: 2 2 3 3 4 4
10214 286/300 Write block 3585 len 28672 data: 3 3 3 3 4 3 3
10215 287/300 Read block 3734 len 40960
10216 288/300 Write block 3470 len 4096 data: 4
10217 289/300 Flush
10218 290/300 Read block 227 len 40960
10219 291/300 Write block 4358 len 8192 data: 2 2
10220 292/300 Flush
10221 293/300 Read block 1027 len 4096
10222 294/300 Write block 1089 len 32768 data: 3 2 2 2 2 3 2 2
10223 295/300 Read block 615 len 16384
10224 296/300 Flush
10225 297/300 Read block 848 len 8192
10226 298/300 Write block 4766 len 20480 data: 2 2 3 4 4
10227 299/300 Flush
10228 300/300 Write block 15 len 4096 data: 3
10229 Main task has work to do, go find it
10230 got dsc Start(0)
10231 start 0
10232 Feb 28 22:15:18.233 INFO request completed, latency_us: 218, response_code: 204, uri: /start/cid/0, method: POST, req_id: d6a3440f-8027-46d1-9c3c-7ee7319ea916, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
10233 Replay: started 0, returned:Ok(())
10234 ----------------------------------------------------------------
10235 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10236 States: Offline Active Active
10237 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
10238 3315 Acked 4314 Read 8 New Done Done true
10239 3316 Acked 4315 Read 6 New Done Done true
10240 3317 Acked 4316 Write 6 New Done Done false
10241 3318 Acked 4317 Read 1 New Done Done false
10242 3319 Acked 4318 Write 1 New Done Done false
10243 3320 Acked 4319 Read 10 New Done Done false
10244 3321 Acked 4320 Read 9 New Done Done false
10245 3322 Acked 4321 Write 7 New Done Done false
10246 3323 Acked 4322 Write 8 New Done Done false
10247 3324 Acked 4323 Read 2 New Done Done false
10248 3325 Acked 4324 Read 7 New Done Done false
10249 3326 Acked 4325 Write 1 New Done Done false
10250 3327 Acked 4326 Read 2 New Done Done false
10251 3328 Acked 4327 Flush 0 New Done Done false
10252 3329 Acked 4328 Write 5 New Done Done false
10253 3330 Acked 4329 Read 5 New Done Done false
10254 3331 Acked 4330 Read 6 New Done Done false
10255 3332 Acked 4331 Read 5 New Done Done false
10256 3333 Acked 4332 Read 6 New Done Done false
10257 3334 Acked 4333 Read 6 New Done Done false
10258 3335 Acked 4334 Read 4 New Done Done false
10259 3336 Acked 4335 Read 4 New Done Done false
10260 3337 Acked 4336 Flush 0 New Done Done false
10261 3338 Acked 4337 Write 4 New Done Done false
10262 3339 Acked 4338 Read 6 New Done Done false
10263 3340 Acked 4339 Write 8 New Done Done false
10264 3341 Acked 4340 Read 4 New Done Done false
10265 3342 Acked 4341 Write 3 New Done Done false
10266 3343 Acked 4342 Write 3 New Done Done false
10267 3344 Acked 4343 Write 1 New Done Done false
10268 3345 Acked 4344 Read 5 New Done Done false
10269 3346 Acked 4345 Write 8 New Done Done false
10270 3347 Acked 4346 Read 6 New Done Done false
10271 3348 Acked 4347 Write 2 New Done Done false
10272 3349 Acked 4348 Write 5 New Done Done false
10273 3350 Acked 4349 Write 4 New Done Done false
10274 3351 Acked 4350 Write 3 New Done Done false
10275 3352 Acked 4351 Write 1 New Done Done false
10276 3353 Acked 4352 Read 2 New Done Done false
10277 3354 Acked 4353 Read 9 New Done Done false
10278 3355 Acked 4354 Read 1 New Done Done false
10279 3356 Acked 4355 Read 7 New Done Done false
10280 3357 Acked 4356 Read 10 New Done Done false
10281 3358 Acked 4357 Read 5 New Done Done false
10282 3359 Acked 4358 Read 5 New Done Done false
10283 3360 Acked 4359 Read 1 New Done Done false
10284 3361 Acked 4360 Write 3 New Done Done false
10285 3362 Acked 4361 Write 5 New Done Done false
10286 3363 Acked 4362 Write 5 New Done Done false
10287 3364 Acked 4363 Read 5 New Done Done false
10288 3365 Acked 4364 Read 4 New Done Done false
10289 3366 Acked 4365 Read 5 New Done Done false
10290 3367 Acked 4366 Write 7 New Done Done false
10291 3368 Acked 4367 Read 5 New Done Done false
10292 3369 Acked 4368 Write 9 New Done Done false
10293 3370 Acked 4369 Write 6 New Done Done false
10294 3371 Acked 4370 Read 10 New Done Done false
10295 3372 Acked 4371 Write 9 New Done Done false
10296 3373 Acked 4372 Read 7 New Done Done false
10297 3374 Acked 4373 Read 5 New Done Done false
10298 3375 Acked 4374 Write 7 New Done Done false
10299 3376 Acked 4375 Read 10 New Done Done false
10300 3377 Acked 4376 Read 6 New Done Done false
10301 3378 Acked 4377 Read 5 New Done Done false
10302 3379 Acked 4378 Read 4 New Done Done false
10303 3380 Acked 4379 Write 8 New Done Done false
10304 3381 Acked 4380 Flush 0 New Done Done false
10305 3382 Acked 4381 Flush 0 New Done Done false
10306 3383 Acked 4382 Read 5 New Done Done false
10307 3384 Acked 4383 Write 6 New Done Done false
10308 3385 Acked 4384 Read 3 New Done Done false
10309 3386 Acked 4385 Flush 0 New Done Done false
10310 3387 Acked 4386 Read 5 New Done Done false
10311 3388 Acked 4387 Write 2 New Done Done false
10312 3389 Acked 4388 Flush 0 New Done Done false
10313 3390 Acked 4389 Write 3 New Done Done false
10314 3391 Acked 4390 Flush 0 New Done Done false
10315 3392 Acked 4391 Read 3 New Done Done false
10316 3393 Acked 4392 Read 4 New Done Done false
10317 3394 Acked 4393 Write 9 New Done Done false
10318 3395 Acked 4394 Read 9 New Done Done false
10319 3396 Acked 4395 Read 2 New Done Done false
10320 3397 Acked 4396 Read 4 New Done Done false
10321 3398 Acked 4397 Read 9 New Done Done false
10322 3399 Acked 4398 Write 10 New Done Done false
10323 3400 Acked 4399 Write 1 New Done Done false
10324 3401 Acked 4400 Read 8 New Done Done false
10325 3402 Acked 4401 Read 10 New Done Done false
10326 3403 Acked 4402 Read 6 New Done Done false
10327 3404 Acked 4403 Read 2 New Done Done false
10328 3405 Acked 4404 Write 5 New Done Done false
10329 3406 Acked 4405 Write 2 New Done Done false
10330 3407 Acked 4406 Read 6 New Done Done false
10331 3408 Acked 4407 Write 3 New Done Done false
10332 3409 Acked 4408 Read 8 New Done Done false
10333 3410 Acked 4409 Write 9 New Done Done false
10334 3411 Acked 4410 Flush 0 New Done Done false
10335 3412 Acked 4411 Read 1 New Done Done false
10336 3413 Acked 4412 Write 7 New Done Done false
10337 3414 Acked 4413 Read 1 New Done Done false
10338 3415 Acked 4414 Flush 0 New Done Done false
10339 3416 Acked 4415 Write 7 New Done Done false
10340 3417 Acked 4416 Write 3 New Done Done false
10341 3418 Acked 4417 Read 7 New Done Done false
10342 3419 Acked 4418 Read 7 New Done Done false
10343 3420 Acked 4419 Write 6 New Done Done false
10344 3421 Acked 4420 Read 1 New Done Done false
10345 3422 Acked 4421 Write 2 New Done Done false
10346 3423 Acked 4422 Flush 0 New Done Done false
10347 3424 Acked 4423 Read 10 New Done Done false
10348 3425 Acked 4424 Write 4 New Done Done false
10349 3426 Acked 4425 Read 1 New Done Done false
10350 3427 Acked 4426 Write 5 New Done Done false
10351 3428 Acked 4427 Write 1 New Done Done false
10352 3429 Acked 4428 Read 6 New Done Done false
10353 3430 Acked 4429 Write 4 New Done Done false
10354 3431 Acked 4430 Read 7 New Done Done false
10355 3432 Acked 4431 Write 4 New Done Done false
10356 3433 Acked 4432 Flush 0 New Done Done false
10357 3434 Acked 4433 Write 2 New Done Done false
10358 3435 Acked 4434 Read 8 New Done Done false
10359 3436 Acked 4435 Write 2 New Done Done false
10360 3437 Acked 4436 Write 10 New Done Done false
10361 3438 Acked 4437 Read 1 New Done Done false
10362 3439 Acked 4438 Flush 0 New Done Done false
10363 3440 Acked 4439 Flush 0 New Done Done false
10364 3441 Acked 4440 Write 4 New Done Done false
10365 3442 Acked 4441 Flush 0 New Done Done false
10366 3443 Acked 4442 Write 2 New Done Done false
10367 3444 Acked 4443 Read 3 New Done Done false
10368 3445 Acked 4444 Read 1 New Done Done false
10369 3446 Acked 4445 Read 5 New Done Done false
10370 3447 Acked 4446 Read 2 New Done Done false
10371 3448 Acked 4447 Read 10 New Done Done false
10372 3449 Acked 4448 Read 7 New Done Done false
10373 3450 Acked 4449 Write 10 New Done Done false
10374 3451 Acked 4450 Write 9 New Done Done false
10375 3452 Acked 4451 Read 1 New Done Done false
10376 3453 Acked 4452 Read 9 New Done Done false
10377 3454 Acked 4453 Read 1 New Done Done false
10378 3455 Acked 4454 Flush 0 New Done Done false
10379 3456 Acked 4455 Flush 0 New Done Done false
10380 3457 Acked 4456 Write 4 New Done Done false
10381 3458 Acked 4457 Flush 0 New Done Done false
10382 3459 Acked 4458 Write 1 New Done Done false
10383 3460 Acked 4459 Write 9 New Done Done false
10384 3461 Acked 4460 Write 6 New Done Done false
10385 3462 Acked 4461 Write 6 New Done Done false
10386 3463 Acked 4462 Write 7 New Done Done false
10387 3464 Acked 4463 Read 2 New Done Done false
10388 3465 Acked 4464 Write 3 New Done Done false
10389 3466 Acked 4465 Read 3 New Done Done false
10390 3467 Acked 4466 Read 8 New Done Done false
10391 3468 Acked 4467 Write 2 New Done Done false
10392 3469 Acked 4468 Read 2 New Done Done false
10393 3470 Acked 4469 Read 4 New Done Done false
10394 3471 Acked 4470 Write 4 New Done Done false
10395 3472 Acked 4471 Write 4 New Done Done false
10396 3473 Acked 4472 Write 8 New Done Done false
10397 3474 Acked 4473 Read 1 New Done Done false
10398 3475 Acked 4474 Read 2 New Done Done false
10399 3476 Acked 4475 Write 7 New Done Done false
10400 3477 Acked 4476 Write 2 New Done Done false
10401 3478 Acked 4477 Read 5 New Done Done false
10402 3479 Acked 4478 Write 6 New Done Done false
10403 3480 Acked 4479 Write 8 New Done Done false
10404 3481 Acked 4480 Read 9 New Done Done false
10405 3482 Acked 4481 Write 6 New Done Done false
10406 3483 Acked 4482 Write 4 New Done Done false
10407 3484 Acked 4483 Read 5 New Done Done false
10408 3485 Acked 4484 Read 3 New Done Done false
10409 3486 Acked 4485 Read 3 New Done Done false
10410 3487 Acked 4486 Read 7 New Done Done false
10411 3488 Acked 4487 Read 8 New Done Done false
10412 3489 Acked 4488 Read 7 New Done Done false
10413 3490 Acked 4489 Read 5 New Done Done false
10414 3491 Acked 4490 Read 6 New Done Done false
10415 3492 Acked 4491 Read 2 New Done Done false
10416 3493 Acked 4492 Read 2 New Done Done false
10417 3494 Acked 4493 Read 1 New Done Done false
10418 3495 Acked 4494 Read 4 New Done Done false
10419 3496 Acked 4495 Read 9 New Done Done false
10420 3497 Acked 4496 Flush 0 New Done Done false
10421 3498 Acked 4497 Write 7 New Done Done false
10422 3499 Acked 4498 Read 1 New Done Done false
10423 3500 Acked 4499 Write 2 New Done Done false
10424 3501 Acked 4500 Write 4 New Done Done false
10425 3502 Acked 4501 Read 6 New Done Done false
10426 3503 Acked 4502 Write 6 New Done Done false
10427 3504 Acked 4503 Write 3 New Done Done false
10428 3505 Acked 4504 Read 5 New Done Done false
10429 3506 Acked 4505 Write 5 New Done Done false
10430 3507 Acked 4506 Read 10 New Done Done false
10431 3508 Acked 4507 Read 7 New Done Done false
10432 3509 Acked 4508 Write 8 New Done Done false
10433 3510 Acked 4509 Write 7 New Done Done false
10434 3511 Acked 4510 Read 2 New Done Done false
10435 3512 Acked 4511 Read 9 New Done Done false
10436 3513 Acked 4512 Write 1 New Done Done false
10437 3514 Acked 4513 Write 10 New Done Done false
10438 3515 Acked 4514 Write 9 New Done Done false
10439 3516 Acked 4515 Write 10 New Done Done false
10440 3517 Acked 4516 Read 1 New Done Done false
10441 3518 Acked 4517 Flush 0 New Done Done false
10442 3519 Acked 4518 Read 8 New Done Done false
10443 3520 Acked 4519 Write 6 New Done Done false
10444 3521 Acked 4520 Flush 0 New Done Done false
10445 3522 Acked 4521 Read 9 New Done Done false
10446 3523 Acked 4522 Read 7 New Done Done false
10447 3524 Acked 4523 Write 7 New Done Done false
10448 3525 Acked 4524 Write 3 New Done Done false
10449 3526 Acked 4525 Read 9 New Done Done false
10450 3527 Acked 4526 Read 8 New Done Done false
10451 3528 Acked 4527 Read 8 New Done Done false
10452 3529 Acked 4528 Write 1 New Done Done false
10453 3530 Acked 4529 Write 6 New Done Done false
10454 3531 Acked 4530 Write 10 New Done Done false
10455 3532 Acked 4531 Read 1 New Done Done false
10456 3533 Acked 4532 Write 10 New Done Done false
10457 3534 Acked 4533 Write 7 New Done Done false
10458 3535 Acked 4534 Flush 0 New Done Done false
10459 3536 Acked 4535 Write 6 New Done Done false
10460 3537 Acked 4536 Write 1 New Done Done false
10461 3538 Acked 4537 Read 9 New Done Done false
10462 3539 Acked 4538 Write 3 New Done Done false
10463 3540 Acked 4539 Read 7 New Done Done false
10464 3541 Acked 4540 Flush 0 New Done Done false
10465 3542 Acked 4541 Read 9 New Done Done false
10466 3543 Acked 4542 Read 9 New Done Done false
10467 3544 Acked 4543 Write 7 New Done Done false
10468 3545 Acked 4544 Read 4 New Done Done false
10469 3546 Acked 4545 Write 2 New Done Done false
10470 3547 Acked 4546 Read 2 New Done Done false
10471 3548 Acked 4547 Write 6 New Done Done false
10472 3549 Acked 4548 Write 8 New Done Done false
10473 3550 Acked 4549 Read 8 New Done Done false
10474 3551 Acked 4550 Read 4 New Done Done false
10475 3552 Acked 4551 Read 9 New Done Done false
10476 3553 Acked 4552 Write 6 New Done Done false
10477 3554 Acked 4553 Read 10 New Done Done false
10478 3555 Acked 4554 Read 9 New Done Done false
10479 3556 Acked 4555 Read 2 New Done Done false
10480 3557 Acked 4556 Write 8 New Done Done false
10481 3558 Acked 4557 Write 10 New Done Done false
10482 3559 Acked 4558 Read 8 New Done Done false
10483 3560 Acked 4559 Read 4 New Done Done false
10484 3561 Acked 4560 Flush 0 New Done Done false
10485 3562 Acked 4561 Write 1 New Done Done false
10486 3563 Acked 4562 Read 9 New Done Done false
10487 3564 Acked 4563 Read 5 New Done Done false
10488 3565 Acked 4564 Write 9 New Done Done false
10489 3566 Acked 4565 Write 6 New Done Done false
10490 3567 Acked 4566 Read 4 New Done Done false
10491 3568 Acked 4567 Write 9 New Done Done false
10492 3569 Acked 4568 Read 1 New Done Done false
10493 3570 Acked 4569 Write 9 New Done Done false
10494 3571 Acked 4570 Read 3 New Done Done false
10495 3572 Acked 4571 Write 10 New Done Done false
10496 3573 Acked 4572 Read 2 New Done Done false
10497 3574 Acked 4573 Read 1 New Done Done false
10498 3575 Acked 4574 Read 1 New Done Done false
10499 3576 Acked 4575 Flush 0 New Done Done false
10500 3577 Acked 4576 Write 10 New Done Done false
10501 3578 Acked 4577 Read 1 New Done Done false
10502 3579 Acked 4578 Write 6 New Done Done false
10503 3580 Acked 4579 Write 6 New Done Done false
10504 3581 Acked 4580 Read 7 New Done Done false
10505 3582 Acked 4581 Read 10 New Done Done false
10506 3583 Acked 4582 Write 3 New Done Done false
10507 3584 Acked 4583 Read 6 New Done Done false
10508 3585 Acked 4584 Flush 0 New Done Done false
10509 3586 Acked 4585 Read 1 New Done Done false
10510 3587 Acked 4586 Flush 0 New Done Done false
10511 3588 Acked 4587 Write 4 New Done Done false
10512 3589 Acked 4588 Read 1 New Done Done false
10513 3590 Acked 4589 Write 7 New Done Done false
10514 3591 Acked 4590 Read 1 New Done Done false
10515 3592 Acked 4591 Write 3 New Done Done false
10516 3593 Acked 4592 Write 9 New Done Done false
10517 3594 Acked 4593 Read 5 New Done Done false
10518 3595 Acked 4594 Write 4 New Done Done false
10519 3596 Acked 4595 Write 6 New Done Done false
10520 3597 Acked 4596 Read 3 New Done Done false
10521 3598 Acked 4597 Flush 0 New Done Done false
10522 3599 Acked 4598 Write 6 New Done Done false
10523 3600 Acked 4599 Write 6 New Done Done false
10524 3601 Acked 4600 Write 7 New Done Done false
10525 3602 Acked 4601 Read 10 New Done Done false
10526 3603 Acked 4602 Write 1 New Done Done false
10527 3604 Acked 4603 Flush 0 New Done Done false
10528 3605 Acked 4604 Read 10 New Done Done false
10529 3606 Acked 4605 Write 2 New Done Done false
10530 3607 Acked 4606 Flush 0 New Done Done false
10531 3608 Acked 4607 Read 1 New Done Done false
10532 3609 Acked 4608 Write 8 New Done Done false
10533 3610 Acked 4609 Read 4 New Done Done false
10534 3611 Acked 4610 Flush 0 New Done Done false
10535 3612 Acked 4611 Read 2 New Done Done false
10536 3613 Acked 4612 Write 5 New Done Done false
10537 3614 Acked 4613 Flush 0 New Done Done false
10538 3615 Acked 4614 Write 1 New Done Done false
10539 STATES DS:0 DS:1 DS:2 TOTAL
10540 New 301 0 0 301
10541 Sent 0 0 0 0
10542 Done 0 301 301 602
10543 Skipped 0 0 0 0
10544 Error 0 0 0 0
10545 Last Flush: 4313 4613 4613
10546 Downstairs last five completed: 4313 4312 4311 4310 4309
10547 Upstairs last five completed: 3615 3614 3613 3612 3611
10548 CLIENT: Up:0 ds:301 act:2
105492024-02-28T22:15:18.440ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105502024-02-28T22:15:18.441ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105512024-02-28T22:15:18.441ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105522024-02-28T22:15:18.441ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105532024-02-28T22:15:18.441ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105542024-02-28T22:15:18.441ZINFOcrucible: [0] client re-new 301 jobs since flush 4313 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105552024-02-28T22:15:18.441ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105562024-02-28T22:15:19.442ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105572024-02-28T22:15:19.442ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105582024-02-28T22:15:19.442ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105592024-02-28T22:15:19.442ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105602024-02-28T22:15:19.442ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105612024-02-28T22:15:19.442ZINFOcrucible: [0] client re-new 302 jobs since flush 4313 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105622024-02-28T22:15:19.442ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105632024-02-28T22:15:20.443ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105642024-02-28T22:15:20.443ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105652024-02-28T22:15:20.444ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105662024-02-28T22:15:20.444ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105672024-02-28T22:15:20.444ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105682024-02-28T22:15:20.444ZINFOcrucible: [0] client re-new 302 jobs since flush 4313 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105692024-02-28T22:15:20.444ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
10570 [8810] got start action
10571 [8810] I am going to restart
10572 Starting downstairs at port 8810
10573 Make output file at "/tmp/dsc/downstairs-8810.txt"
10574 [8810][0] reports Starting
105752024-02-28T22:15:21.445ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105762024-02-28T22:15:21.445ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105772024-02-28T22:15:21.445ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105782024-02-28T22:15:21.445ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105792024-02-28T22:15:21.445ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105802024-02-28T22:15:21.445ZINFOcrucible: [0] client re-new 302 jobs since flush 4313 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
105812024-02-28T22:15:21.445ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
10582 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1176)
10583 [8810][0] reports Running
10584 ----------------------------------------------------------------
10585 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
10586 States: Offline Active Active
10587 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
10588 3315 Acked 4314 Read 8 New Done Done true
10589 3316 Acked 4315 Read 6 New Done Done true
10590 3317 Acked 4316 Write 6 New Done Done true
10591 3318 Acked 4317 Read 1 New Done Done true
10592 3319 Acked 4318 Write 1 New Done Done true
10593 3320 Acked 4319 Read 10 New Done Done true
10594 3321 Acked 4320 Read 9 New Done Done true
10595 3322 Acked 4321 Write 7 New Done Done true
10596 3323 Acked 4322 Write 8 New Done Done true
10597 3324 Acked 4323 Read 2 New Done Done true
10598 3325 Acked 4324 Read 7 New Done Done true
10599 3326 Acked 4325 Write 1 New Done Done true
10600 3327 Acked 4326 Read 2 New Done Done true
10601 3328 Acked 4327 Flush 0 New Done Done true
10602 3329 Acked 4328 Write 5 New Done Done true
10603 3330 Acked 4329 Read 5 New Done Done true
10604 3331 Acked 4330 Read 6 New Done Done true
10605 3332 Acked 4331 Read 5 New Done Done true
10606 3333 Acked 4332 Read 6 New Done Done true
10607 3334 Acked 4333 Read 6 New Done Done true
10608 3335 Acked 4334 Read 4 New Done Done true
10609 3336 Acked 4335 Read 4 New Done Done true
10610 3337 Acked 4336 Flush 0 New Done Done true
10611 3338 Acked 4337 Write 4 New Done Done true
10612 3339 Acked 4338 Read 6 New Done Done true
10613 3340 Acked 4339 Write 8 New Done Done true
10614 3341 Acked 4340 Read 4 New Done Done true
10615 3342 Acked 4341 Write 3 New Done Done true
10616 3343 Acked 4342 Write 3 New Done Done true
10617 3344 Acked 4343 Write 1 New Done Done true
10618 3345 Acked 4344 Read 5 New Done Done true
10619 3346 Acked 4345 Write 8 New Done Done true
10620 3347 Acked 4346 Read 6 New Done Done true
10621 3348 Acked 4347 Write 2 New Done Done true
10622 3349 Acked 4348 Write 5 New Done Done true
10623 3350 Acked 4349 Write 4 New Done Done true
10624 3351 Acked 4350 Write 3 New Done Done true
10625 3352 Acked 4351 Write 1 New Done Done true
10626 3353 Acked 4352 Read 2 New Done Done true
10627 3354 Acked 4353 Read 9 New Done Done true
10628 3355 Acked 4354 Read 1 New Done Done true
10629 3356 Acked 4355 Read 7 New Done Done true
10630 3357 Acked 4356 Read 10 New Done Done true
10631 3358 Acked 4357 Read 5 New Done Done true
10632 3359 Acked 4358 Read 5 New Done Done true
10633 3360 Acked 4359 Read 1 New Done Done true
10634 3361 Acked 4360 Write 3 New Done Done true
10635 3362 Acked 4361 Write 5 New Done Done true
10636 3363 Acked 4362 Write 5 New Done Done true
10637 3364 Acked 4363 Read 5 New Done Done true
10638 3365 Acked 4364 Read 4 New Done Done true
10639 3366 Acked 4365 Read 5 New Done Done true
10640 3367 Acked 4366 Write 7 New Done Done true
10641 3368 Acked 4367 Read 5 New Done Done true
10642 3369 Acked 4368 Write 9 New Done Done true
10643 3370 Acked 4369 Write 6 New Done Done true
10644 3371 Acked 4370 Read 10 New Done Done true
10645 3372 Acked 4371 Write 9 New Done Done true
10646 3373 Acked 4372 Read 7 New Done Done true
10647 3374 Acked 4373 Read 5 New Done Done true
10648 3375 Acked 4374 Write 7 New Done Done true
10649 3376 Acked 4375 Read 10 New Done Done true
10650 3377 Acked 4376 Read 6 New Done Done true
10651 3378 Acked 4377 Read 5 New Done Done true
10652 3379 Acked 4378 Read 4 New Done Done true
10653 3380 Acked 4379 Write 8 New Done Done true
10654 3381 Acked 4380 Flush 0 New Done Done true
10655 3382 Acked 4381 Flush 0 New Done Done true
10656 3383 Acked 4382 Read 5 New Done Done true
10657 3384 Acked 4383 Write 6 New Done Done true
10658 3385 Acked 4384 Read 3 New Done Done true
10659 3386 Acked 4385 Flush 0 New Done Done true
10660 3387 Acked 4386 Read 5 New Done Done true
10661 3388 Acked 4387 Write 2 New Done Done true
10662 3389 Acked 4388 Flush 0 New Done Done true
10663 3390 Acked 4389 Write 3 New Done Done true
10664 3391 Acked 4390 Flush 0 New Done Done true
10665 3392 Acked 4391 Read 3 New Done Done true
10666 3393 Acked 4392 Read 4 New Done Done true
10667 3394 Acked 4393 Write 9 New Done Done true
10668 3395 Acked 4394 Read 9 New Done Done true
10669 3396 Acked 4395 Read 2 New Done Done true
10670 3397 Acked 4396 Read 4 New Done Done true
10671 3398 Acked 4397 Read 9 New Done Done true
10672 3399 Acked 4398 Write 10 New Done Done true
10673 3400 Acked 4399 Write 1 New Done Done true
10674 3401 Acked 4400 Read 8 New Done Done true
10675 3402 Acked 4401 Read 10 New Done Done true
10676 3403 Acked 4402 Read 6 New Done Done true
10677 3404 Acked 4403 Read 2 New Done Done true
10678 3405 Acked 4404 Write 5 New Done Done true
10679 3406 Acked 4405 Write 2 New Done Done true
10680 3407 Acked 4406 Read 6 New Done Done true
10681 3408 Acked 4407 Write 3 New Done Done true
10682 3409 Acked 4408 Read 8 New Done Done true
10683 3410 Acked 4409 Write 9 New Done Done true
10684 3411 Acked 4410 Flush 0 New Done Done true
10685 3412 Acked 4411 Read 1 New Done Done true
10686 3413 Acked 4412 Write 7 New Done Done true
10687 3414 Acked 4413 Read 1 New Done Done true
10688 3415 Acked 4414 Flush 0 New Done Done true
10689 3416 Acked 4415 Write 7 New Done Done true
10690 3417 Acked 4416 Write 3 New Done Done true
10691 3418 Acked 4417 Read 7 New Done Done true
10692 3419 Acked 4418 Read 7 New Done Done true
10693 3420 Acked 4419 Write 6 New Done Done true
10694 3421 Acked 4420 Read 1 New Done Done true
10695 3422 Acked 4421 Write 2 New Done Done true
10696 3423 Acked 4422 Flush 0 New Done Done true
10697 3424 Acked 4423 Read 10 New Done Done true
10698 3425 Acked 4424 Write 4 New Done Done true
10699 3426 Acked 4425 Read 1 New Done Done true
10700 3427 Acked 4426 Write 5 New Done Done true
10701 3428 Acked 4427 Write 1 New Done Done true
10702 3429 Acked 4428 Read 6 New Done Done true
10703 3430 Acked 4429 Write 4 New Done Done true
10704 3431 Acked 4430 Read 7 New Done Done true
10705 3432 Acked 4431 Write 4 New Done Done true
10706 3433 Acked 4432 Flush 0 New Done Done true
10707 3434 Acked 4433 Write 2 New Done Done true
10708 3435 Acked 4434 Read 8 New Done Done true
10709 3436 Acked 4435 Write 2 New Done Done true
10710 3437 Acked 4436 Write 10 New Done Done true
10711 3438 Acked 4437 Read 1 New Done Done true
10712 3439 Acked 4438 Flush 0 New Done Done true
10713 3440 Acked 4439 Flush 0 New Done Done true
10714 3441 Acked 4440 Write 4 New Done Done true
10715 3442 Acked 4441 Flush 0 New Done Done true
10716 3443 Acked 4442 Write 2 New Done Done true
10717 3444 Acked 4443 Read 3 New Done Done true
10718 3445 Acked 4444 Read 1 New Done Done true
10719 3446 Acked 4445 Read 5 New Done Done true
10720 3447 Acked 4446 Read 2 New Done Done true
10721 3448 Acked 4447 Read 10 New Done Done true
10722 3449 Acked 4448 Read 7 New Done Done true
10723 3450 Acked 4449 Write 10 New Done Done true
10724 3451 Acked 4450 Write 9 New Done Done true
10725 3452 Acked 4451 Read 1 New Done Done true
10726 3453 Acked 4452 Read 9 New Done Done true
10727 3454 Acked 4453 Read 1 New Done Done true
10728 3455 Acked 4454 Flush 0 New Done Done true
10729 3456 Acked 4455 Flush 0 New Done Done true
10730 3457 Acked 4456 Write 4 New Done Done true
10731 3458 Acked 4457 Flush 0 New Done Done true
10732 3459 Acked 4458 Write 1 New Done Done true
10733 3460 Acked 4459 Write 9 New Done Done true
10734 3461 Acked 4460 Write 6 New Done Done true
10735 3462 Acked 4461 Write 6 New Done Done true
10736 3463 Acked 4462 Write 7 New Done Done true
10737 3464 Acked 4463 Read 2 New Done Done true
10738 3465 Acked 4464 Write 3 New Done Done true
10739 3466 Acked 4465 Read 3 New Done Done true
10740 3467 Acked 4466 Read 8 New Done Done true
10741 3468 Acked 4467 Write 2 New Done Done true
10742 3469 Acked 4468 Read 2 New Done Done true
10743 3470 Acked 4469 Read 4 New Done Done true
10744 3471 Acked 4470 Write 4 New Done Done true
10745 3472 Acked 4471 Write 4 New Done Done true
10746 3473 Acked 4472 Write 8 New Done Done true
10747 3474 Acked 4473 Read 1 New Done Done true
10748 3475 Acked 4474 Read 2 New Done Done true
10749 3476 Acked 4475 Write 7 New Done Done true
10750 3477 Acked 4476 Write 2 New Done Done true
10751 3478 Acked 4477 Read 5 New Done Done true
10752 3479 Acked 4478 Write 6 New Done Done true
10753 3480 Acked 4479 Write 8 New Done Done true
10754 3481 Acked 4480 Read 9 New Done Done true
10755 3482 Acked 4481 Write 6 New Done Done true
10756 3483 Acked 4482 Write 4 New Done Done true
10757 3484 Acked 4483 Read 5 New Done Done true
10758 3485 Acked 4484 Read 3 New Done Done true
10759 3486 Acked 4485 Read 3 New Done Done true
10760 3487 Acked 4486 Read 7 New Done Done true
10761 3488 Acked 4487 Read 8 New Done Done true
10762 3489 Acked 4488 Read 7 New Done Done true
10763 3490 Acked 4489 Read 5 New Done Done true
10764 3491 Acked 4490 Read 6 New Done Done true
10765 3492 Acked 4491 Read 2 New Done Done true
10766 3493 Acked 4492 Read 2 New Done Done true
10767 3494 Acked 4493 Read 1 New Done Done true
10768 3495 Acked 4494 Read 4 New Done Done true
10769 3496 Acked 4495 Read 9 New Done Done true
10770 3497 Acked 4496 Flush 0 New Done Done true
10771 3498 Acked 4497 Write 7 New Done Done true
10772 3499 Acked 4498 Read 1 New Done Done true
10773 3500 Acked 4499 Write 2 New Done Done true
10774 3501 Acked 4500 Write 4 New Done Done true
10775 3502 Acked 4501 Read 6 New Done Done true
10776 3503 Acked 4502 Write 6 New Done Done true
10777 3504 Acked 4503 Write 3 New Done Done true
10778 3505 Acked 4504 Read 5 New Done Done true
10779 3506 Acked 4505 Write 5 New Done Done true
10780 3507 Acked 4506 Read 10 New Done Done true
10781 3508 Acked 4507 Read 7 New Done Done true
10782 3509 Acked 4508 Write 8 New Done Done true
10783 3510 Acked 4509 Write 7 New Done Done true
10784 3511 Acked 4510 Read 2 New Done Done true
10785 3512 Acked 4511 Read 9 New Done Done true
10786 3513 Acked 4512 Write 1 New Done Done true
10787 3514 Acked 4513 Write 10 New Done Done true
10788 3515 Acked 4514 Write 9 New Done Done true
10789 3516 Acked 4515 Write 10 New Done Done true
10790 3517 Acked 4516 Read 1 New Done Done true
10791 3518 Acked 4517 Flush 0 New Done Done true
10792 3519 Acked 4518 Read 8 New Done Done true
10793 3520 Acked 4519 Write 6 New Done Done true
10794 3521 Acked 4520 Flush 0 New Done Done true
10795 3522 Acked 4521 Read 9 New Done Done true
10796 3523 Acked 4522 Read 7 New Done Done true
10797 3524 Acked 4523 Write 7 New Done Done true
10798 3525 Acked 4524 Write 3 New Done Done true
10799 3526 Acked 4525 Read 9 New Done Done true
10800 3527 Acked 4526 Read 8 New Done Done true
10801 3528 Acked 4527 Read 8 New Done Done true
10802 3529 Acked 4528 Write 1 New Done Done true
10803 3530 Acked 4529 Write 6 New Done Done true
10804 3531 Acked 4530 Write 10 New Done Done true
10805 3532 Acked 4531 Read 1 New Done Done true
10806 3533 Acked 4532 Write 10 New Done Done true
10807 3534 Acked 4533 Write 7 New Done Done true
10808 3535 Acked 4534 Flush 0 New Done Done true
10809 3536 Acked 4535 Write 6 New Done Done true
10810 3537 Acked 4536 Write 1 New Done Done true
10811 3538 Acked 4537 Read 9 New Done Done true
10812 3539 Acked 4538 Write 3 New Done Done true
10813 3540 Acked 4539 Read 7 New Done Done true
10814 3541 Acked 4540 Flush 0 New Done Done true
10815 3542 Acked 4541 Read 9 New Done Done true
10816 3543 Acked 4542 Read 9 New Done Done true
10817 3544 Acked 4543 Write 7 New Done Done true
10818 3545 Acked 4544 Read 4 New Done Done true
10819 3546 Acked 4545 Write 2 New Done Done true
10820 3547 Acked 4546 Read 2 New Done Done true
10821 3548 Acked 4547 Write 6 New Done Done true
10822 3549 Acked 4548 Write 8 New Done Done true
10823 3550 Acked 4549 Read 8 New Done Done true
10824 3551 Acked 4550 Read 4 New Done Done true
10825 3552 Acked 4551 Read 9 New Done Done true
10826 3553 Acked 4552 Write 6 New Done Done true
10827 3554 Acked 4553 Read 10 New Done Done true
10828 3555 Acked 4554 Read 9 New Done Done true
10829 3556 Acked 4555 Read 2 New Done Done true
10830 3557 Acked 4556 Write 8 New Done Done true
10831 3558 Acked 4557 Write 10 New Done Done true
10832 3559 Acked 4558 Read 8 New Done Done true
10833 3560 Acked 4559 Read 4 New Done Done true
10834 3561 Acked 4560 Flush 0 New Done Done true
10835 3562 Acked 4561 Write 1 New Done Done true
10836 3563 Acked 4562 Read 9 New Done Done true
10837 3564 Acked 4563 Read 5 New Done Done true
10838 3565 Acked 4564 Write 9 New Done Done true
10839 3566 Acked 4565 Write 6 New Done Done true
10840 3567 Acked 4566 Read 4 New Done Done true
10841 3568 Acked 4567 Write 9 New Done Done true
10842 3569 Acked 4568 Read 1 New Done Done true
10843 3570 Acked 4569 Write 9 New Done Done true
10844 3571 Acked 4570 Read 3 New Done Done true
10845 3572 Acked 4571 Write 10 New Done Done true
10846 3573 Acked 4572 Read 2 New Done Done true
10847 3574 Acked 4573 Read 1 New Done Done true
10848 3575 Acked 4574 Read 1 New Done Done true
10849 3576 Acked 4575 Flush 0 New Done Done true
10850 3577 Acked 4576 Write 10 New Done Done true
10851 3578 Acked 4577 Read 1 New Done Done true
10852 3579 Acked 4578 Write 6 New Done Done true
10853 3580 Acked 4579 Write 6 New Done Done true
10854 3581 Acked 4580 Read 7 New Done Done true
10855 3582 Acked 4581 Read 10 New Done Done true
10856 3583 Acked 4582 Write 3 New Done Done true
10857 3584 Acked 4583 Read 6 New Done Done true
10858 3585 Acked 4584 Flush 0 New Done Done true
10859 3586 Acked 4585 Read 1 New Done Done true
10860 3587 Acked 4586 Flush 0 New Done Done true
10861 3588 Acked 4587 Write 4 New Done Done true
10862 3589 Acked 4588 Read 1 New Done Done true
10863 3590 Acked 4589 Write 7 New Done Done true
10864 3591 Acked 4590 Read 1 New Done Done true
10865 3592 Acked 4591 Write 3 New Done Done true
10866 3593 Acked 4592 Write 9 New Done Done true
10867 3594 Acked 4593 Read 5 New Done Done true
10868 3595 Acked 4594 Write 4 New Done Done true
10869 3596 Acked 4595 Write 6 New Done Done true
10870 3597 Acked 4596 Read 3 New Done Done true
10871 3598 Acked 4597 Flush 0 New Done Done true
10872 3599 Acked 4598 Write 6 New Done Done true
10873 3600 Acked 4599 Write 6 New Done Done true
10874 3601 Acked 4600 Write 7 New Done Done true
10875 3602 Acked 4601 Read 10 New Done Done true
10876 3603 Acked 4602 Write 1 New Done Done true
10877 3604 Acked 4603 Flush 0 New Done Done true
10878 3605 Acked 4604 Read 10 New Done Done true
10879 3606 Acked 4605 Write 2 New Done Done true
10880 3607 Acked 4606 Flush 0 New Done Done true
10881 3608 Acked 4607 Read 1 New Done Done true
10882 3609 Acked 4608 Write 8 New Done Done true
10883 3610 Acked 4609 Read 4 New Done Done true
10884 3611 Acked 4610 Flush 0 New Done Done true
10885 3612 Acked 4611 Read 2 New Done Done true
10886 3613 Acked 4612 Write 5 New Done Done true
10887 3614 Acked 4613 Flush 0 New Done Done true
10888 3615 Acked 4614 Write 1 New Done Done true
10889 3616 Acked 4615 Flush 0 New Done Done true
10890 STATES DS:0 DS:1 DS:2 TOTAL
10891 New 302 0 0 302
10892 Sent 0 0 0 0
10893 Done 0 302 302 604
10894 Skipped 0 0 0 0
10895 Error 0 0 0 0
10896 Last Flush: 4313 4615 4615
10897 Downstairs last five completed: 4313 4312 4311 4310 4309
10898 Upstairs last five completed: 3616 3615 3614 3613 3612
10899 CLIENT: Up:0 ds:302 act:2
109002024-02-28T22:15:22.446ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109012024-02-28T22:15:22.446ZINFOcrucible: ds_connection connected = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109022024-02-28T22:15:22.448ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109032024-02-28T22:15:22.448ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109042024-02-28T22:15:22.448ZINFOcrucible: send last flush ID to this DS: 4313 = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109052024-02-28T22:15:22.448ZINFOcrucible: Replied this last flush ID: 4313 = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109062024-02-28T22:15:22.448ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109072024-02-28T22:15:22.448ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109082024-02-28T22:15:22.448ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109092024-02-28T22:15:22.448ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
10910 ----------------------------------------------------------------
10911 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10912 Downstairs last five completed: 4615 4614 4613 4612 4611
10913 Upstairs last five completed: 3616 3615 3614 3613 3612
10914 CLIENT: Up:0 ds:0 act:3
10915 Replay: All jobs finished, all DS active.
10916 Main task has work to do, go find it
10917 got dsc Stop(2)
10918 stop 2
10919 Feb 28 22:15:26.239 INFO request completed, latency_us: 359, response_code: 204, uri: /stop/cid/2, method: POST, req_id: b517d773-531c-4215-a78b-d425c5dedeaf, remote_addr: 127.0.0.1:36617, local_addr:[8830] Got stop action so:false kr:false
10920 127.0.0.1:9998
109212024-02-28T22:15:26.240ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109222024-02-28T22:15:26.240ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109232024-02-28T22:15:26.240ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109242024-02-28T22:15:26.240ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109252024-02-28T22:15:26.240ZINFOcrucible: [2] client re-new 0 jobs since flush 4615 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109262024-02-28T22:15:26.240ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
10927 Feb 28 22:15:26.240 INFO request completed, latency_us: 218, response_code: 200, uri: /state/cid/2, method: GET, req_id: a22fe050-1633-4a6e-9965-71d162e1c63b, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
10928 [8830] Exited with: ExitStatus(unix_wait_status(9))
10929 [8830][2] reports Exit
109302024-02-28T22:15:27.241ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109312024-02-28T22:15:27.241ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109322024-02-28T22:15:27.241ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109332024-02-28T22:15:27.241ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109342024-02-28T22:15:27.241ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109352024-02-28T22:15:27.241ZINFOcrucible: [2] client re-new 0 jobs since flush 4615 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109362024-02-28T22:15:27.241ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109372024-02-28T22:15:28.243ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109382024-02-28T22:15:28.243ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109392024-02-28T22:15:28.243ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109402024-02-28T22:15:28.243ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109412024-02-28T22:15:28.244ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109422024-02-28T22:15:28.244ZINFOcrucible: [2] client re-new 0 jobs since flush 4615 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109432024-02-28T22:15:28.244ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109442024-02-28T22:15:29.245ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109452024-02-28T22:15:29.245ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109462024-02-28T22:15:29.245ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109472024-02-28T22:15:29.245ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109482024-02-28T22:15:29.245ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109492024-02-28T22:15:29.245ZINFOcrucible: [2] client re-new 0 jobs since flush 4615 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109502024-02-28T22:15:29.245ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
10951 Feb 28 22:15:30.242 INFO request completed, latency_us: 219, response_code: 200, uri: /state/cid/2, method: GET, req_id: 4434ec3e-53e1-47c7-9f1e-c2499b2b4a2c, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
10952 001/300 Read block 2370 len 4096
10953 002/300 Read block 3041 len 16384
10954 003/300 Read block 1699 len 24576
109552024-02-28T22:15:30.246ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109562024-02-28T22:15:30.246ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109572024-02-28T22:15:30.246ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109582024-02-28T22:15:30.246ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109592024-02-28T22:15:30.246ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109602024-02-28T22:15:30.246ZINFOcrucible: [2] client re-new 3 jobs since flush 4615 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
109612024-02-28T22:15:30.246ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
10962 004/300 Flush
10963 005/300 Read block 3016 len 40960
10964 006/300 Write block 2984 len 24576 data: 4 5 5 5 5 4
10965 007/300 Write block 1472 len 28672 data: 7 8 7 7 6 5 5
10966 008/300 Read block 3574 len 28672
10967 009/300 Read block 3898 len 40960
10968 010/300 Read block 1589 len 24576
10969 011/300 Write block 2953 len 36864 data: 3 3 3 3 3 2 2 3 3
10970 012/300 Write block 2325 len 4096 data: 4
10971 013/300 Read block 1462 len 40960
10972 014/300 Write block 726 len 40960 data: 2 2 2 2 2 2 3 3 4 4
10973 015/300 Read block 3646 len 12288
10974 016/300 Read block 1969 len 36864
10975 017/300 Write block 3820 len 20480 data: 3 4 5 5 5
10976 018/300 Flush
10977 019/300 Write block 4283 len 32768 data: 4 4 4 3 2 2 2 2
10978 020/300 Read block 789 len 40960
10979 021/300 Read block 1822 len 36864
10980 022/300 Write block 1103 len 8192 data: 5 6
10981 023/300 Read block 2742 len 12288
10982 024/300 Read block 231 len 28672
10983 025/300 Flush
10984 026/300 Write block 4675 len 20480 data: 2 2 2 2 2
10985 027/300 Read block 3917 len 12288
10986 028/300 Read block 4771 len 16384
10987 029/300 Write block 266 len 4096 data: 2
10988 030/300 Read block 4394 len 36864
10989 031/300 Read block 3006 len 28672
10990 032/300 Write block 918 len 32768 data: 4 4 4 3 3 2 2 2
10991 033/300 Read block 3363 len 4096
10992 034/300 Read block 2616 len 28672
10993 035/300 Read block 4796 len 16384
10994 036/300 Read block 4156 len 8192
10995 037/300 Read block 101 len 28672
10996 038/300 Read block 63 len 28672
10997 039/300 Read block 601 len 32768
10998 040/300 Read block 1792 len 28672
10999 041/300 Flush
11000 042/300 Read block 3408 len 20480
11001 043/300 Write block 1320 len 20480 data: 4 3 4 4 4
11002 044/300 Read block 2846 len 12288
11003 045/300 Read block 4464 len 16384
11004 046/300 Read block 834 len 40960
11005 047/300 Read block 539 len 32768
11006 048/300 Write block 4499 len 24576 data: 10 7 5 5 6 7
11007 049/300 Read block 3259 len 28672
11008 050/300 Read block 2543 len 36864
11009 051/300 Write block 4174 len 40960 data: 3 2 2 2 2 2 3 3 3 3
11010 052/300 Write block 1214 len 20480 data: 3 4 4 4 4
11011 053/300 Read block 1886 len 28672
11012 054/300 Write block 3704 len 24576 data: 4 3 2 2 3 4
11013 055/300 Read block 2910 len 24576
11014 056/300 Write block 1212 len 36864 data: 3 3 4 5 5 5 5 4 5
11015 057/300 Write block 2449 len 20480 data: 4 3 2 2 3
11016 058/300 Read block 2677 len 4096
11017 059/300 Read block 2042 len 24576
11018 060/300 Read block 2478 len 8192
11019 061/300 Read block 4288 len 36864
11020 062/300 Write block 3431 len 16384 data: 3 5 5 5
11021 063/300 Read block 439 len 4096
11022 064/300 Read block 4796 len 16384
11023 065/300 Read block 2633 len 20480
11024 066/300 Read block 1886 len 28672
11025 067/300 Write block 3476 len 40960 data: 3 3 3 3 3 2 2 2 2 2
11026 068/300 Write block 638 len 16384 data: 2 2 2 2
11027 069/300 Read block 2750 len 8192
11028 070/300 Read block 2136 len 28672
11029 071/300 Write block 430 len 40960 data: 3 3 3 5 4 4 4 3 3 3
11030 072/300 Read block 2662 len 8192
11031 073/300 Read block 1932 len 40960
11032 074/300 Write block 534 len 20480 data: 3 3 3 3 4
11033 075/300 Read block 3146 len 36864
11034 076/300 Write block 3701 len 4096 data: 5
11035 077/300 Write block 1963 len 12288 data: 4 4 4
11036 078/300 Read block 2878 len 32768
11037 079/300 Write block 1983 len 40960 data: 4 4 4 4 3 3 4 4 5 4
11038 080/300 Read block 4037 len 40960
11039 081/300 Read block 1576 len 36864
11040 082/300 Write block 3247 len 28672 data: 3 3 4 4 4 4 4
11041 083/300 Read block 3813 len 40960
11042 084/300 Write block 3835 len 12288 data: 2 2 3
11043 085/300 Write block 4335 len 20480 data: 6 5 3 3 3
11044 086/300 Read block 3776 len 4096
11045 087/300 Read block 229 len 28672
11046 088/300 Write block 3438 len 28672 data: 5 5 6 4 3 4 5
11047 089/300 Read block 4955 len 40960
11048 090/300 Read block 3075 len 4096
11049 091/300 Write block 2066 len 40960 data: 4 3 3 3 3 3 3 3 3 2
11050 092/300 Write block 746 len 36864 data: 4 4 3 4 5 5 5 4 5
11051 093/300 Read block 2535 len 36864
11052 094/300 Read block 4095 len 4096
11053 095/300 Flush
11054 096/300 Write block 4957 len 20480 data: 2 2 4 4 4
11055 097/300 Read block 3045 len 12288
11056 098/300 Read block 1919 len 4096
11057 099/300 Write block 4785 len 32768 data: 5 6 6 7 6 4 3 4
11058 100/300 Write block 865 len 24576 data: 2 2 2 2 2 4
11059 101/300 Read block 1290 len 28672
11060 102/300 Read block 188 len 16384
11061 103/300 Flush
11062 104/300 Write block 2872 len 8192 data: 5 4
11063 105/300 Write block 2541 len 24576 data: 4 4 3 4 4 3
11064 106/300 Read block 2310 len 24576
11065 107/300 Flush
11066 108/300 Flush
11067 109/300 Write block 3474 len 4096 data: 3
11068 110/300 Read block 4631 len 32768
11069 111/300 Read block 1373 len 28672
11070 112/300 Read block 2932 len 4096
11071 113/300 Read block 4820 len 4096
11072 114/300 Flush
11073 115/300 Read block 139 len 40960
11074 116/300 Read block 1151 len 16384
11075 117/300 Write block 376 len 40960 data: 3 3 4 3 2 2 2 2 2 2
11076 118/300 Flush
11077 119/300 Write block 3272 len 36864 data: 2 2 3 3 3 2 2 2 2
11078 120/300 Read block 1156 len 8192
11079 121/300 Write block 2690 len 16384 data: 3 3 5 3
11080 122/300 Write block 4656 len 32768 data: 5 4 4 4 4 3 3 3
11081 123/300 Write block 56 len 36864 data: 4 4 4 4 4 4 4 4 3
11082 124/300 Write block 1581 len 32768 data: 7 7 5 4 4 5 4 4
11083 125/300 Read block 4388 len 40960
11084 126/300 Read block 3308 len 16384
11085 127/300 Read block 2392 len 12288
11086 128/300 Read block 3570 len 24576
11087 129/300 Write block 4347 len 12288 data: 4 4 4
11088 130/300 Read block 1888 len 36864
11089 131/300 Read block 4184 len 40960
11090 132/300 Write block 1619 len 32768 data: 3 3 3 4 6 7 7 5
11091 133/300 Write block 2500 len 8192 data: 2 2
11092 134/300 Read block 1460 len 28672
11093 135/300 Write block 4862 len 28672 data: 3 3 3 3 2 3 3
11094 136/300 Read block 4633 len 24576
11095 137/300 Write block 1860 len 20480 data: 5 5 3 3 5
11096 138/300 Read block 3727 len 8192
11097 139/300 Read block 1754 len 28672
11098 140/300 Read block 4928 len 12288
11099 141/300 Write block 3870 len 4096 data: 4
11100 142/300 Write block 4172 len 16384 data: 4 3 4 3
11101 143/300 Read block 850 len 20480
11102 144/300 Read block 373 len 24576
11103 145/300 Flush
11104 146/300 Write block 1411 len 36864 data: 3 3 3 3 3 3 3 3 4
11105 147/300 Read block 4658 len 40960
11106 148/300 Flush
11107 149/300 Read block 2464 len 16384
11108 150/300 Write block 3527 len 24576 data: 4 4 4 4 4 5
11109 151/300 Read block 3076 len 32768
11110 152/300 Write block 3399 len 36864 data: 4 3 3 3 3 3 3 3 5
11111 153/300 Write block 3274 len 40960 data: 4 4 4 3 3 3 3 3 3 4
11112 154/300 Read block 602 len 32768
11113 155/300 Read block 2768 len 40960
11114 156/300 Read block 1264 len 8192
11115 157/300 Read block 994 len 32768
11116 158/300 Read block 4891 len 24576
11117 159/300 Write block 3709 len 32768 data: 5 3 3 2 2 2 3 2
11118 160/300 Read block 1236 len 40960
11119 161/300 Read block 184 len 16384
11120 162/300 Read block 3115 len 12288
11121 163/300 Write block 3025 len 4096 data: 2
11122 164/300 Read block 218 len 32768
11123 165/300 Read block 1539 len 8192
11124 166/300 Write block 505 len 20480 data: 4 5 5 6 6
11125 167/300 Read block 1029 len 32768
11126 168/300 Write block 4657 len 8192 data: 5 5
11127 169/300 Read block 4345 len 8192
11128 170/300 Write block 1983 len 24576 data: 5 5 5 5 4 4
11129 171/300 Read block 2069 len 16384
11130 172/300 Flush
11131 173/300 Write block 1040 len 32768 data: 4 4 3 2 3 3 3 3
11132 174/300 Read block 1890 len 12288
11133 175/300 Write block 4169 len 28672 data: 5 4 4 5 4 5 4
11134 176/300 Write block 4391 len 8192 data: 2 2
11135 177/300 Flush
11136 178/300 Write block 4165 len 32768 data: 4 4 7 6 6 5 5 6
11137 179/300 Write block 4699 len 16384 data: 3 4 4 3
11138 180/300 Write block 4523 len 40960 data: 3 3 2 2 2 2 4 4 4 3
11139 181/300 Read block 1290 len 40960
11140 182/300 Write block 668 len 16384 data: 4 4 6 5
11141 183/300 Read block 1461 len 24576
11142 184/300 Read block 793 len 24576
11143 185/300 Read block 3098 len 16384
11144 186/300 Write block 2666 len 4096 data: 2
11145 187/300 Read block 3416 len 28672
11146 188/300 Write block 1518 len 36864 data: 7 6 6 5 5 5 3 3 3
11147 189/300 Read block 2390 len 28672
11148 190/300 Read block 2899 len 40960
11149 191/300 Write block 2637 len 16384 data: 2 2 2 2
11150 192/300 Read block 730 len 36864
11151 193/300 Read block 4118 len 32768
11152 194/300 Write block 518 len 16384 data: 3 4 4 4
11153 195/300 Read block 1454 len 4096
11154 196/300 Read block 2968 len 12288
11155 197/300 Write block 133 len 40960 data: 5 7 5 5 4 3 4 5 5 4
11156 198/300 Read block 3537 len 28672
11157 199/300 Write block 3804 len 20480 data: 3 3 3 3 2
11158 200/300 Write block 2935 len 36864 data: 4 4 4 4 6 6 5 5 4
11159 201/300 Write block 2337 len 28672 data: 7 7 7 5 4 3 3
11160 202/300 Read block 3424 len 28672
11161 203/300 Write block 3773 len 32768 data: 3 3 3 3 4 4 4 3
11162 204/300 Write block 3935 len 24576 data: 3 3 3 3 3 4
11163 205/300 Write block 1722 len 40960 data: 5 5 4 3 3 3 3 3 3 3
11164 206/300 Flush
11165 207/300 Read block 2260 len 20480
11166 208/300 Flush
11167 209/300 Write block 4877 len 36864 data: 3 4 3 3 3 4 5 4 4
11168 210/300 Read block 673 len 8192
11169 211/300 Write block 1473 len 8192 data: 9 8
11170 212/300 Write block 4508 len 16384 data: 5 4 5 3
11171 213/300 Read block 655 len 20480
11172 214/300 Flush
11173 215/300 Write block 1439 len 28672 data: 3 2 2 2 2 2 3
11174 216/300 Write block 4949 len 8192 data: 4 4
11175 217/300 Write block 255 len 24576 data: 3 2 2 3 3 3
11176 218/300 Write block 1899 len 32768 data: 2 2 2 2 2 2 3 4
11177 219/300 Read block 3564 len 32768
11178 220/300 Read block 4701 len 36864
11179 221/300 Flush
11180 222/300 Read block 1714 len 32768
11181 223/300 Write block 1892 len 4096 data: 6
11182 224/300 Write block 28 len 24576 data: 3 2 2 2 2 2
11183 225/300 Write block 1828 len 12288 data: 4 4 5
11184 226/300 Read block 1726 len 32768
11185 227/300 Write block 4884 len 32768 data: 5 5 4 5 5 5 3 3
11186 228/300 Flush
11187 229/300 Flush
11188 230/300 Read block 670 len 32768
11189 231/300 Read block 1316 len 8192
11190 232/300 Write block 4850 len 16384 data: 3 3 3 3
11191 233/300 Read block 81 len 20480
11192 234/300 Write block 4859 len 24576 data: 4 4 3 4 4 4
11193 235/300 Write block 3739 len 24576 data: 4 6 5 6 5 4
11194 236/300 Write block 561 len 32768 data: 4 4 4 3 3 2 2 3
11195 237/300 Read block 3745 len 40960
11196 238/300 Read block 2366 len 8192
11197 239/300 Flush
11198 240/300 Read block 2996 len 12288
11199 241/300 Read block 2723 len 40960
11200 242/300 Read block 382 len 32768
11201 243/300 Flush
11202 244/300 Write block 2927 len 24576 data: 4 4 4 4 4 3
11203 245/300 Flush
11204 246/300 Read block 3372 len 16384
11205 247/300 Read block 1873 len 36864
11206 248/300 Write block 1415 len 40960 data: 4 4 4 4 5 4 4 4 5 5
11207 249/300 Read block 722 len 24576
11208 250/300 Write block 3100 len 8192 data: 2 3
11209 251/300 Flush
11210 252/300 Flush
11211 253/300 Read block 1438 len 40960
11212 254/300 Read block 4276 len 16384
11213 255/300 Read block 1253 len 28672
11214 256/300 Read block 3140 len 16384
11215 257/300 Read block 1640 len 32768
11216 258/300 Read block 2463 len 8192
11217 259/300 Read block 3218 len 16384
11218 260/300 Write block 2426 len 8192 data: 3 3
11219 261/300 Write block 4600 len 20480 data: 4 4 3 3 3
11220 262/300 Write block 1082 len 28672 data: 4 4 3 4 3 3 3
11221 263/300 Read block 346 len 32768
11222 264/300 Write block 4329 len 32768 data: 4 4 4 5 4 4 7 6
11223 265/300 Read block 3386 len 12288
11224 266/300 Write block 2188 len 28672 data: 3 4 5 6 6 4 4
11225 267/300 Write block 3014 len 12288 data: 4 3 4
11226 268/300 Read block 4288 len 20480
11227 269/300 Write block 1404 len 24576 data: 4 4 4 3 3 3
11228 270/300 Read block 3063 len 32768
11229 271/300 Write block 1249 len 32768 data: 2 2 3 3 3 3 3 3
11230 272/300 Write block 4647 len 40960 data: 2 2 2 2 3 3 3 4 4 6
11231 273/300 Read block 307 len 36864
11232 274/300 Write block 4983 len 12288 data: 5 5 5
11233 275/300 Write block 2495 len 4096 data: 3
11234 276/300 Write block 4808 len 32768 data: 4 3 2 3 3 2 2 2
11235 277/300 Write block 851 len 28672 data: 3 3 3 3 5 3 3
11236 278/300 Read block 2376 len 16384
11237 279/300 Read block 3 len 32768
11238 280/300 Write block 914 len 36864 data: 4 2 3 3 5 5 5 4 4
11239 281/300 Read block 3467 len 32768
11240 282/300 Flush
11241 283/300 Read block 3829 len 24576
11242 284/300 Write block 3202 len 24576 data: 4 5 4 4 4 5
11243 285/300 Flush
11244 286/300 Write block 4605 len 20480 data: 3 4 4 3 4
11245 287/300 Read block 4379 len 12288
11246 288/300 Read block 1699 len 4096
11247 289/300 Read block 4735 len 24576
11248 290/300 Write block 1987 len 24576 data: 5 5 5 5 6 5
11249 291/300 Read block 506 len 20480
11250 292/300 Flush
11251 293/300 Read block 2238 len 4096
11252 294/300 Read block 3450 len 20480
11253 295/300 Flush
11254 296/300 Write block 496 len 16384 data: 4 3 5 4
11255 297/300 Write block 4088 len 12288 data: 4 5 5
11256 298/300 Write block 4922 len 28672 data: 4 4 3 3 3 3 4
11257 299/300 Write block 953 len 40960 data: 5 5 5 5 6 7 8 7 7 7
11258 300/300 Write block 776 len 4096 data: 3
11259 Main task has work to do, go find it
11260 got dsc Start(2)
11261 start 2
11262 Feb 28 22:15:30.980 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/2, method: POST, req_id: 0ccdecce-22b3-41de-9616-16a7b6788be9, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
11263 Replay: started 2, returned:Ok(())
11264 ----------------------------------------------------------------
11265 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
11266 States: Active Active Offline
11267 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
11268 3617 Acked 4616 Read 1 Done Done New true
11269 3618 Acked 4617 Read 4 Done Done New true
11270 3619 Acked 4618 Read 6 Done Done New true
11271 3620 Acked 4619 Flush 0 Done Done New false
11272 3621 Acked 4620 Read 10 Done Done New false
11273 3622 Acked 4621 Write 6 Done Done New false
11274 3623 Acked 4622 Write 7 Done Done New false
11275 3624 Acked 4623 Read 7 Done Done New false
11276 3625 Acked 4624 Read 10 Done Done New false
11277 3626 Acked 4625 Read 6 Done Done New false
11278 3627 Acked 4626 Write 9 Done Done New false
11279 3628 Acked 4627 Write 1 Done Done New false
11280 3629 Acked 4628 Read 10 Done Done New false
11281 3630 Acked 4629 Write 10 Done Done New false
11282 3631 Acked 4630 Read 3 Done Done New false
11283 3632 Acked 4631 Read 9 Done Done New false
11284 3633 Acked 4632 Write 5 Done Done New false
11285 3634 Acked 4633 Flush 0 Done Done New false
11286 3635 Acked 4634 Write 8 Done Done New false
11287 3636 Acked 4635 Read 10 Done Done New false
11288 3637 Acked 4636 Read 9 Done Done New false
11289 3638 Acked 4637 Write 2 Done Done New false
11290 3639 Acked 4638 Read 3 Done Done New false
11291 3640 Acked 4639 Read 7 Done Done New false
11292 3641 Acked 4640 Flush 0 Done Done New false
11293 3642 Acked 4641 Write 5 Done Done New false
11294 3643 Acked 4642 Read 3 Done Done New false
11295 3644 Acked 4643 Read 4 Done Done New false
11296 3645 Acked 4644 Write 1 Done Done New false
11297 3646 Acked 4645 Read 9 Done Done New false
11298 3647 Acked 4646 Read 7 Done Done New false
11299 3648 Acked 4647 Write 8 Done Done New false
11300 3649 Acked 4648 Read 1 Done Done New false
11301 3650 Acked 4649 Read 7 Done Done New false
11302 3651 Acked 4650 Read 4 Done Done New false
11303 3652 Acked 4651 Read 2 Done Done New false
11304 3653 Acked 4652 Read 7 Done Done New false
11305 3654 Acked 4653 Read 7 Done Done New false
11306 3655 Acked 4654 Read 8 Done Done New false
11307 3656 Acked 4655 Read 7 Done Done New false
11308 3657 Acked 4656 Flush 0 Done Done New false
11309 3658 Acked 4657 Read 5 Done Done New false
11310 3659 Acked 4658 Write 5 Done Done New false
11311 3660 Acked 4659 Read 3 Done Done New false
11312 3661 Acked 4660 Read 4 Done Done New false
11313 3662 Acked 4661 Read 10 Done Done New false
11314 3663 Acked 4662 Read 8 Done Done New false
11315 3664 Acked 4663 Write 6 Done Done New false
11316 3665 Acked 4664 Read 7 Done Done New false
11317 3666 Acked 4665 Read 9 Done Done New false
11318 3667 Acked 4666 Write 10 Done Done New false
11319 3668 Acked 4667 Write 5 Done Done New false
11320 3669 Acked 4668 Read 7 Done Done New false
11321 3670 Acked 4669 Write 6 Done Done New false
11322 3671 Acked 4670 Read 6 Done Done New false
11323 3672 Acked 4671 Write 9 Done Done New false
11324 3673 Acked 4672 Write 5 Done Done New false
11325 3674 Acked 4673 Read 1 Done Done New false
11326 3675 Acked 4674 Read 6 Done Done New false
11327 3676 Acked 4675 Read 2 Done Done New false
11328 3677 Acked 4676 Read 9 Done Done New false
11329 3678 Acked 4677 Write 4 Done Done New false
11330 3679 Acked 4678 Read 1 Done Done New false
11331 3680 Acked 4679 Read 4 Done Done New false
11332 3681 Acked 4680 Read 5 Done Done New false
11333 3682 Acked 4681 Read 7 Done Done New false
11334 3683 Acked 4682 Write 10 Done Done New false
11335 3684 Acked 4683 Write 4 Done Done New false
11336 3685 Acked 4684 Read 2 Done Done New false
11337 3686 Acked 4685 Read 7 Done Done New false
11338 3687 Acked 4686 Write 10 Done Done New false
11339 3688 Acked 4687 Read 2 Done Done New false
11340 3689 Acked 4688 Read 10 Done Done New false
11341 3690 Acked 4689 Write 5 Done Done New false
11342 3691 Acked 4690 Read 9 Done Done New false
11343 3692 Acked 4691 Write 1 Done Done New false
11344 3693 Acked 4692 Write 3 Done Done New false
11345 3694 Acked 4693 Read 8 Done Done New false
11346 3695 Acked 4694 Write 10 Done Done New false
11347 3696 Acked 4695 Read 10 Done Done New false
11348 3697 Acked 4696 Read 9 Done Done New false
11349 3698 Acked 4697 Write 7 Done Done New false
11350 3699 Acked 4698 Read 10 Done Done New false
11351 3700 Acked 4699 Write 3 Done Done New false
11352 3701 Acked 4700 Write 5 Done Done New false
11353 3702 Acked 4701 Read 1 Done Done New false
11354 3703 Acked 4702 Read 7 Done Done New false
11355 3704 Acked 4703 Write 7 Done Done New false
11356 3705 Acked 4704 Read 10 Done Done New false
11357 3706 Acked 4705 Read 1 Done Done New false
11358 3707 Acked 4706 Write 10 Done Done New false
11359 3708 Acked 4707 Write 9 Done Done New false
11360 3709 Acked 4708 Read 9 Done Done New false
11361 3710 Acked 4709 Read 1 Done Done New false
11362 3711 Acked 4710 Flush 0 Done Done New false
11363 3712 Acked 4711 Write 5 Done Done New false
11364 3713 Acked 4712 Read 3 Done Done New false
11365 3714 Acked 4713 Read 1 Done Done New false
11366 3715 Acked 4714 Write 8 Done Done New false
11367 3716 Acked 4715 Write 6 Done Done New false
11368 3717 Acked 4716 Read 7 Done Done New false
11369 3718 Acked 4717 Read 4 Done Done New false
11370 3719 Acked 4718 Flush 0 Done Done New false
11371 3720 Acked 4719 Write 2 Done Done New false
11372 3721 Acked 4720 Write 6 Done Done New false
11373 3722 Acked 4721 Read 6 Done Done New false
11374 3723 Acked 4722 Flush 0 Done Done New false
11375 3724 Acked 4723 Flush 0 Done Done New false
11376 3725 Acked 4724 Write 1 Done Done New false
11377 3726 Acked 4725 Read 8 Done Done New false
11378 3727 Acked 4726 Read 7 Done Done New false
11379 3728 Acked 4727 Read 1 Done Done New false
11380 3729 Acked 4728 Read 1 Done Done New false
11381 3730 Acked 4729 Flush 0 Done Done New false
11382 3731 Acked 4730 Read 10 Done Done New false
11383 3732 Acked 4731 Read 4 Done Done New false
11384 3733 Acked 4732 Write 10 Done Done New false
11385 3734 Acked 4733 Flush 0 Done Done New false
11386 3735 Acked 4734 Write 9 Done Done New false
11387 3736 Acked 4735 Read 2 Done Done New false
11388 3737 Acked 4736 Write 4 Done Done New false
11389 3738 Acked 4737 Write 8 Done Done New false
11390 3739 Acked 4738 Write 9 Done Done New false
11391 3740 Acked 4739 Write 8 Done Done New false
11392 3741 Acked 4740 Read 10 Done Done New false
11393 3742 Acked 4741 Read 4 Done Done New false
11394 3743 Acked 4742 Read 3 Done Done New false
11395 3744 Acked 4743 Read 6 Done Done New false
11396 3745 Acked 4744 Write 3 Done Done New false
11397 3746 Acked 4745 Read 9 Done Done New false
11398 3747 Acked 4746 Read 10 Done Done New false
11399 3748 Acked 4747 Write 8 Done Done New false
11400 3749 Acked 4748 Write 2 Done Done New false
11401 3750 Acked 4749 Read 7 Done Done New false
11402 3751 Acked 4750 Write 7 Done Done New false
11403 3752 Acked 4751 Read 6 Done Done New false
11404 3753 Acked 4752 Write 5 Done Done New false
11405 3754 Acked 4753 Read 2 Done Done New false
11406 3755 Acked 4754 Read 7 Done Done New false
11407 3756 Acked 4755 Read 3 Done Done New false
11408 3757 Acked 4756 Write 1 Done Done New false
11409 3758 Acked 4757 Write 4 Done Done New false
11410 3759 Acked 4758 Read 5 Done Done New false
11411 3760 Acked 4759 Read 6 Done Done New false
11412 3761 Acked 4760 Flush 0 Done Done New false
11413 3762 Acked 4761 Write 9 Done Done New false
11414 3763 Acked 4762 Read 10 Done Done New false
11415 3764 Acked 4763 Flush 0 Done Done New false
11416 3765 Acked 4764 Read 4 Done Done New false
11417 3766 Acked 4765 Write 6 Done Done New false
11418 3767 Acked 4766 Read 8 Done Done New false
11419 3768 Acked 4767 Write 9 Done Done New false
11420 3769 Acked 4768 Write 10 Done Done New false
11421 3770 Acked 4769 Read 8 Done Done New false
11422 3771 Acked 4770 Read 10 Done Done New false
11423 3772 Acked 4771 Read 2 Done Done New false
11424 3773 Acked 4772 Read 8 Done Done New false
11425 3774 Acked 4773 Read 6 Done Done New false
11426 3775 Acked 4774 Write 8 Done Done New false
11427 3776 Acked 4775 Read 10 Done Done New false
11428 3777 Acked 4776 Read 4 Done Done New false
11429 3778 Acked 4777 Read 3 Done Done New false
11430 3779 Acked 4778 Write 1 Done Done New false
11431 3780 Acked 4779 Read 8 Done Done New false
11432 3781 Acked 4780 Read 2 Done Done New false
11433 3782 Acked 4781 Write 5 Done Done New false
11434 3783 Acked 4782 Read 8 Done Done New false
11435 3784 Acked 4783 Write 2 Done Done New false
11436 3785 Acked 4784 Read 2 Done Done New false
11437 3786 Acked 4785 Write 6 Done Done New false
11438 3787 Acked 4786 Read 4 Done Done New false
11439 3788 Acked 4787 Flush 0 Done Done New false
11440 3789 Acked 4788 Write 8 Done Done New false
11441 3790 Acked 4789 Read 3 Done Done New false
11442 3791 Acked 4790 Write 7 Done Done New false
11443 3792 Acked 4791 Write 2 Done Done New false
11444 3793 Acked 4792 Flush 0 Done Done New false
11445 3794 Acked 4793 Write 8 Done Done New false
11446 3795 Acked 4794 Write 4 Done Done New false
11447 3796 Acked 4795 Write 10 Done Done New false
11448 3797 Acked 4796 Read 10 Done Done New false
11449 3798 Acked 4797 Write 4 Done Done New false
11450 3799 Acked 4798 Read 6 Done Done New false
11451 3800 Acked 4799 Read 6 Done Done New false
11452 3801 Acked 4800 Read 4 Done Done New false
11453 3802 Acked 4801 Write 1 Done Done New false
11454 3803 Acked 4802 Read 7 Done Done New false
11455 3804 Acked 4803 Write 9 Done Done New false
11456 3805 Acked 4804 Read 7 Done Done New false
11457 3806 Acked 4805 Read 10 Done Done New false
11458 3807 Acked 4806 Write 4 Done Done New false
11459 3808 Acked 4807 Read 9 Done Done New false
11460 3809 Acked 4808 Read 8 Done Done New false
11461 3810 Acked 4809 Write 4 Done Done New false
11462 3811 Acked 4810 Read 1 Done Done New false
11463 3812 Acked 4811 Read 3 Done Done New false
11464 3813 Acked 4812 Write 10 Done Done New false
11465 3814 Acked 4813 Read 7 Done Done New false
11466 3815 Acked 4814 Write 5 Done Done New false
11467 3816 Acked 4815 Write 9 Done Done New false
11468 3817 Acked 4816 Write 7 Done Done New false
11469 3818 Acked 4817 Read 7 Done Done New false
11470 3819 Acked 4818 Write 8 Done Done New false
11471 3820 Acked 4819 Write 6 Done Done New false
11472 3821 Acked 4820 Write 10 Done Done New false
11473 3822 Acked 4821 Flush 0 Done Done New false
11474 3823 Acked 4822 Read 5 Done Done New false
11475 3824 Acked 4823 Flush 0 Done Done New false
11476 3825 Acked 4824 Write 9 Done Done New false
11477 3826 Acked 4825 Read 2 Done Done New false
11478 3827 Acked 4826 Write 2 Done Done New false
11479 3828 Acked 4827 Write 4 Done Done New false
11480 3829 Acked 4828 Read 5 Done Done New false
11481 3830 Acked 4829 Flush 0 Done Done New false
11482 3831 Acked 4830 Write 7 Done Done New false
11483 3832 Acked 4831 Write 2 Done Done New false
11484 3833 Acked 4832 Write 6 Done Done New false
11485 3834 Acked 4833 Write 8 Done Done New false
11486 3835 Acked 4834 Read 8 Done Done New false
11487 3836 Acked 4835 Read 9 Done Done New false
11488 3837 Acked 4836 Flush 0 Done Done New false
11489 3838 Acked 4837 Read 8 Done Done New false
11490 3839 Acked 4838 Write 1 Done Done New false
11491 3840 Acked 4839 Write 6 Done Done New false
11492 3841 Acked 4840 Write 3 Done Done New false
11493 3842 Acked 4841 Read 8 Done Done New false
11494 3843 Acked 4842 Write 8 Done Done New false
11495 3844 Acked 4843 Flush 0 Done Done New false
11496 3845 Acked 4844 Flush 0 Done Done New false
11497 3846 Acked 4845 Read 8 Done Done New false
11498 3847 Acked 4846 Read 2 Done Done New false
11499 3848 Acked 4847 Write 4 Done Done New false
11500 3849 Acked 4848 Read 5 Done Done New false
11501 3850 Acked 4849 Write 6 Done Done New false
11502 3851 Acked 4850 Write 6 Done Done New false
11503 3852 Acked 4851 Write 8 Done Done New false
11504 3853 Acked 4852 Read 10 Done Done New false
11505 3854 Acked 4853 Read 2 Done Done New false
11506 3855 Acked 4854 Flush 0 Done Done New false
11507 3856 Acked 4855 Read 3 Done Done New false
11508 3857 Acked 4856 Read 10 Done Done New false
11509 3858 Acked 4857 Read 8 Done Done New false
11510 3859 Acked 4858 Flush 0 Done Done New false
11511 3860 Acked 4859 Write 6 Done Done New false
11512 3861 Acked 4860 Flush 0 Done Done New false
11513 3862 Acked 4861 Read 4 Done Done New false
11514 3863 Acked 4862 Read 9 Done Done New false
11515 3864 Acked 4863 Write 10 Done Done New false
11516 3865 Acked 4864 Read 6 Done Done New false
11517 3866 Acked 4865 Write 2 Done Done New false
11518 3867 Acked 4866 Flush 0 Done Done New false
11519 3868 Acked 4867 Flush 0 Done Done New false
11520 3869 Acked 4868 Read 10 Done Done New false
11521 3870 Acked 4869 Read 4 Done Done New false
11522 3871 Acked 4870 Read 7 Done Done New false
11523 3872 Acked 4871 Read 4 Done Done New false
11524 3873 Acked 4872 Read 8 Done Done New false
11525 3874 Acked 4873 Read 2 Done Done New false
11526 3875 Acked 4874 Read 4 Done Done New false
11527 3876 Acked 4875 Write 2 Done Done New false
11528 3877 Acked 4876 Write 5 Done Done New false
11529 3878 Acked 4877 Write 7 Done Done New false
11530 3879 Acked 4878 Read 8 Done Done New false
11531 3880 Acked 4879 Write 8 Done Done New false
11532 3881 Acked 4880 Read 3 Done Done New false
11533 3882 Acked 4881 Write 7 Done Done New false
11534 3883 Acked 4882 Write 3 Done Done New false
11535 3884 Acked 4883 Read 5 Done Done New false
11536 3885 Acked 4884 Write 6 Done Done New false
11537 3886 Acked 4885 Read 8 Done Done New false
11538 3887 Acked 4886 Write 8 Done Done New false
11539 3888 Acked 4887 Write 10 Done Done New false
11540 3889 Acked 4888 Read 9 Done Done New false
11541 3890 Acked 4889 Write 3 Done Done New false
11542 3891 Acked 4890 Write 1 Done Done New false
11543 3892 Acked 4891 Write 8 Done Done New false
11544 3893 Acked 4892 Write 7 Done Done New false
11545 3894 Acked 4893 Read 4 Done Done New false
11546 3895 Acked 4894 Read 8 Done Done New false
11547 3896 Acked 4895 Write 9 Done Done New false
11548 3897 Acked 4896 Read 8 Done Done New false
11549 3898 Acked 4897 Flush 0 Done Done New false
11550 3899 Acked 4898 Read 6 Done Done New false
11551 3900 Acked 4899 Write 6 Done Done New false
11552 3901 Acked 4900 Flush 0 Done Done New false
11553 3902 Acked 4901 Write 5 Done Done New false
11554 3903 Acked 4902 Read 3 Done Done New false
11555 3904 Acked 4903 Read 1 Done Done New false
11556 3905 Acked 4904 Read 6 Done Done New false
11557 3906 Acked 4905 Write 6 Done Done New false
11558 3907 Acked 4906 Read 5 Done Done New false
11559 3908 Acked 4907 Flush 0 Done Done New false
11560 3909 Acked 4908 Read 1 Done Done New false
11561 3910 Acked 4909 Read 5 Done Done New false
11562 3911 Acked 4910 Flush 0 Done Done New false
11563 3912 Acked 4911 Write 4 Done Done New false
11564 3913 Acked 4912 Write 3 Done Done New false
11565 3914 Acked 4913 Write 7 Done Done New false
11566 3915 Acked 4914 Write 10 Done Done New false
11567 3916 Acked 4915 Write 1 Done Done New false
11568 STATES DS:0 DS:1 DS:2 TOTAL
11569 New 0 0 300 300
11570 Sent 0 0 0 0
11571 Done 300 300 0 600
11572 Skipped 0 0 0 0
11573 Error 0 0 0 0
11574 Last Flush: 4910 4910 4615
11575 Downstairs last five completed: 4615 4614 4613 4612 4611
11576 Upstairs last five completed: 3916 3915 3914 3913 3912
11577 CLIENT: Up:0 ds:300 act:2
115782024-02-28T22:15:31.247ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
115792024-02-28T22:15:31.248ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
115802024-02-28T22:15:31.248ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
115812024-02-28T22:15:31.248ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
115822024-02-28T22:15:31.248ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
115832024-02-28T22:15:31.248ZINFOcrucible: [2] client re-new 301 jobs since flush 4615 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
115842024-02-28T22:15:31.248ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
11585 [8830] got start action
11586 [8830] I am going to restart
11587 Starting downstairs at port 8830
11588 Make output file at "/tmp/dsc/downstairs-8830.txt"
11589 [8830][2] reports Starting
115902024-02-28T22:15:32.249ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
115912024-02-28T22:15:32.249ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
115922024-02-28T22:15:32.249ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
115932024-02-28T22:15:32.249ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
115942024-02-28T22:15:32.249ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
115952024-02-28T22:15:32.249ZINFOcrucible: [2] client re-new 301 jobs since flush 4615 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
115962024-02-28T22:15:32.249ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
11597 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1177)
11598 [8830][2] reports Running
115992024-02-28T22:15:33.250ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116002024-02-28T22:15:33.250ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116012024-02-28T22:15:33.252ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116022024-02-28T22:15:33.252ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116032024-02-28T22:15:33.252ZINFOcrucible: send last flush ID to this DS: 4615 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116042024-02-28T22:15:33.252ZINFOcrucible: Replied this last flush ID: 4615 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116052024-02-28T22:15:33.252ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116062024-02-28T22:15:33.252ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116072024-02-28T22:15:33.252ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116082024-02-28T22:15:33.252ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
11609 ----------------------------------------------------------------
11610 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
11611 Downstairs last five completed: 4916 4915 4914 4913 4912
11612 Upstairs last five completed: 3917 3916 3915 3914 3913
11613 CLIENT: Up:0 ds:0 act:3
11614 Replay: All jobs finished, all DS active.
11615 Main task has work to do, go find it
11616 got dsc Stop(0)
11617 stop 0
11618 Feb 28 22:15:34.984 INFO request completed, latency_us: 371, response_code: 204, uri: /stop/cid/0, method: [8810] Got stop action so:false kr:false
11619 POST, req_id: eb549fe6-e192-461a-af3a-ce9934671ba6, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
116202024-02-28T22:15:34.985ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116212024-02-28T22:15:34.985ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116222024-02-28T22:15:34.985ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116232024-02-28T22:15:34.985ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
11624 Feb 28 22:15:34.985 INFO request completed, latency_us: 137, response_code: 200, uri: /state/cid/0, method: GET, req_id: 4634df70-72d0-4c4c-8e74-5831433e76bf, remote_addr: 127.0.0.1:36617, local_addr: 127.0.{"msg":"[0] client re-new 0 jobs since flush 4916","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:15:34.985302464Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
11625 0.1:9998
116262024-02-28T22:15:34.985ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
11627 [8810] Exited with: ExitStatus(unix_wait_status(9))
11628 [8810][0] reports Exit
116292024-02-28T22:15:35.986ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116302024-02-28T22:15:35.986ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116312024-02-28T22:15:35.986ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116322024-02-28T22:15:35.986ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116332024-02-28T22:15:35.986ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116342024-02-28T22:15:35.986ZINFOcrucible: [0] client re-new 0 jobs since flush 4916 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116352024-02-28T22:15:35.986ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116362024-02-28T22:15:36.987ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116372024-02-28T22:15:36.987ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116382024-02-28T22:15:36.987ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116392024-02-28T22:15:36.987ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116402024-02-28T22:15:36.987ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116412024-02-28T22:15:36.987ZINFOcrucible: [0] client re-new 0 jobs since flush 4916 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116422024-02-28T22:15:36.987ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116432024-02-28T22:15:37.988ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116442024-02-28T22:15:37.988ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116452024-02-28T22:15:37.988ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116462024-02-28T22:15:37.988ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116472024-02-28T22:15:37.989ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116482024-02-28T22:15:37.989ZINFOcrucible: [0] client re-new 0 jobs since flush 4916 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116492024-02-28T22:15:37.989ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
11650 Feb 28 22:15:38.987 INFO request completed, latency_us: 319, response_code: 200, uri: /state/cid/0, method: GET, req_id: 4db577cb-914a-4103-9e28-e1d402ab2596, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
11651 001/300 Read block 2994 len 8192
11652 002/300 Read block 793 len 28672
11653 003/300 Read block 1107 len 20480
116542024-02-28T22:15:38.990ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116552024-02-28T22:15:38.990ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116562024-02-28T22:15:38.990ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116572024-02-28T22:15:38.990ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116582024-02-28T22:15:38.990ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116592024-02-28T22:15:38.990ZINFOcrucible: [0] client re-new 3 jobs since flush 4916 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
116602024-02-28T22:15:38.990ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
11661 004/300 Read block 1230 len 36864
11662 005/300 Write block 4842 len 36864 data: 3 3 3 3 3 2 2 3 4
11663 006/300 Read block 1342 len 20480
11664 007/300 Write block 4436 len 4096 data: 6
11665 008/300 Flush
11666 009/300 Write block 4494 len 28672 data: 6 5 8 7 8 11 8
11667 010/300 Read block 3087 len 36864
11668 011/300 Write block 1990 len 4096 data: 6
11669 012/300 Read block 4134 len 24576
11670 013/300 Write block 1924 len 20480 data: 2 2 2 3 3
11671 014/300 Read block 2312 len 36864
11672 015/300 Write block 417 len 20480 data: 7 6 6 7 5
11673 016/300 Read block 1170 len 40960
11674 017/300 Write block 4223 len 40960 data: 3 4 5 5 5 5 4 4 4 4
11675 018/300 Write block 3773 len 20480 data: 4 4 4 4 5
11676 019/300 Write block 4861 len 28672 data: 4 5 5 5 4 3 4
11677 020/300 Read block 2110 len 8192
11678 021/300 Read block 1440 len 28672
11679 022/300 Write block 1417 len 16384 data: 5 5 6 5
11680 023/300 Write block 2935 len 8192 data: 5 5
11681 024/300 Read block 721 len 24576
11682 025/300 Read block 2285 len 28672
11683 026/300 Read block 1168 len 24576
11684 027/300 Read block 3341 len 24576
11685 028/300 Write block 3604 len 28672 data: 2 2 2 2 2 2 2
11686 029/300 Write block 4381 len 28672 data: 4 4 4 5 4 4 4
11687 030/300 Read block 4276 len 40960
11688 031/300 Write block 4088 len 20480 data: 5 6 6 7 6
11689 032/300 Read block 319 len 20480
11690 033/300 Read block 3323 len 16384
11691 034/300 Write block 4451 len 24576 data: 3 3 3 3 3 4
11692 035/300 Write block 3952 len 20480 data: 3 5 5 6 5
11693 036/300 Read block 634 len 28672
11694 037/300 Write block 4311 len 4096 data: 3
11695 038/300 Write block 681 len 32768 data: 8 6 6 5 4 3 4 3
11696 039/300 Write block 2764 len 20480 data: 3 3 3 3 2
11697 040/300 Write block 1896 len 36864 data: 5 5 3 3 3 3 3 3 3
11698 041/300 Read block 975 len 36864
11699 042/300 Read block 214 len 32768
11700 043/300 Read block 6 len 12288
11701 044/300 Read block 3942 len 24576
11702 045/300 Read block 3006 len 20480
11703 046/300 Read block 3485 len 4096
11704 047/300 Read block 3086 len 32768
11705 048/300 Read block 3664 len 16384
11706 049/300 Flush
11707 050/300 Write block 1458 len 8192 data: 4 4
11708 051/300 Write block 2218 len 32768 data: 4 3 4 3 3 3 3 3
11709 052/300 Read block 463 len 16384
11710 053/300 Write block 3510 len 36864 data: 2 3 4 4 4 4 5 4 4
11711 054/300 Write block 3237 len 20480 data: 4 4 4 4 4
11712 055/300 Read block 1603 len 24576
11713 056/300 Write block 2136 len 8192 data: 3 3
11714 057/300 Write block 4252 len 16384 data: 4 4 6 5
11715 058/300 Read block 44 len 20480
11716 059/300 Write block 4909 len 8192 data: 3 4
11717 060/300 Read block 4459 len 12288
11718 061/300 Write block 575 len 16384 data: 5 3 4 3
11719 062/300 Write block 275 len 36864 data: 4 5 5 5 5 4 3 3 3
11720 063/300 Read block 4404 len 12288
11721 064/300 Flush
11722 065/300 Read block 397 len 36864
11723 066/300 Read block 3574 len 36864
11724 067/300 Read block 947 len 32768
11725 068/300 Read block 1299 len 8192
11726 069/300 Read block 48 len 40960
11727 070/300 Read block 2339 len 40960
11728 071/300 Flush
11729 072/300 Read block 2381 len 20480
11730 073/300 Flush
11731 074/300 Read block 3045 len 16384
11732 075/300 Read block 3640 len 4096
11733 076/300 Read block 2582 len 24576
11734 077/300 Read block 4298 len 32768
11735 078/300 Write block 66 len 16384 data: 3 2 3 3
11736 079/300 Write block 2811 len 12288 data: 2 2 3
11737 080/300 Write block 2194 len 8192 data: 5 3
11738 081/300 Read block 2278 len 4096
11739 082/300 Read block 4405 len 24576
11740 083/300 Read block 2883 len 24576
11741 084/300 Read block 1551 len 32768
11742 085/300 Write block 3533 len 12288 data: 5 4 4
11743 086/300 Write block 3216 len 32768 data: 5 4 4 5 5 4 3 3
11744 087/300 Read block 1532 len 8192
11745 088/300 Write block 3266 len 40960 data: 3 3 2 2 2 2 3 3 5 5
11746 089/300 Read block 4818 len 24576
11747 090/300 Write block 3970 len 12288 data: 3 4 3
11748 091/300 Write block 3681 len 12288 data: 3 2 2
11749 092/300 Read block 1729 len 8192
11750 093/300 Write block 93 len 12288 data: 2 4 4
11751 094/300 Write block 4576 len 28672 data: 3 3 4 4 4 3 4
11752 095/300 Write block 2975 len 16384 data: 8 7 7 6
11753 096/300 Flush
11754 097/300 Read block 2439 len 36864
11755 098/300 Read block 342 len 20480
11756 099/300 Write block 389 len 20480 data: 4 3 3 4 4
11757 100/300 Write block 477 len 32768 data: 4 4 4 4 3 3 3 5
11758 101/300 Write block 453 len 12288 data: 4 4 4
11759 102/300 Flush
11760 103/300 Read block 591 len 8192
11761 104/300 Write block 3757 len 8192 data: 4 4
11762 105/300 Write block 2102 len 12288 data: 2 2 2
11763 106/300 Read block 38 len 32768
11764 107/300 Read block 495 len 36864
11765 108/300 Write block 4969 len 12288 data: 2 2 2
11766 109/300 Write block 2721 len 4096 data: 6
11767 110/300 Read block 1359 len 36864
11768 111/300 Read block 4738 len 4096
11769 112/300 Write block 1846 len 16384 data: 3 2 2 2
11770 113/300 Write block 2493 len 36864 data: 2 2 4 3 2 3 2 3 3
11771 114/300 Write block 4019 len 12288 data: 4 4 2
11772 115/300 Read block 1044 len 8192
11773 116/300 Write block 2851 len 24576 data: 3 2 3 3 3 3
11774 117/300 Read block 1071 len 4096
11775 118/300 Read block 2560 len 36864
11776 119/300 Write block 4491 len 24576 data: 4 4 4 7 6 9
11777 120/300 Flush
11778 121/300 Read block 640 len 4096
11779 122/300 Read block 2269 len 32768
11780 123/300 Write block 438 len 32768 data: 4 4 5 5 4 3 3 3
11781 124/300 Read block 3441 len 20480
11782 125/300 Read block 3655 len 32768
11783 126/300 Read block 406 len 28672
11784 127/300 Write block 318 len 12288 data: 5 4 4
11785 128/300 Write block 2983 len 4096 data: 3
11786 129/300 Read block 4936 len 4096
11787 130/300 Write block 2884 len 24576 data: 5 4 6 6 6 7
11788 131/300 Read block 541 len 4096
11789 132/300 Read block 1810 len 4096
11790 133/300 Write block 593 len 8192 data: 4 4
11791 134/300 Flush
11792 135/300 Write block 3391 len 36864 data: 4 4 4 5 6 5 4 4 5
11793 136/300 Read block 1516 len 36864
11794 137/300 Read block 1917 len 8192
11795 138/300 Read block 4971 len 12288
11796 139/300 Read block 133 len 32768
11797 140/300 Read block 4164 len 24576
11798 141/300 Write block 437 len 40960 data: 4 5 5 6 6 5 4 4 4 5
11799 142/300 Write block 3062 len 32768 data: 7 7 3 3 3 2 2 3
11800 143/300 Read block 2607 len 8192
11801 144/300 Flush
11802 145/300 Read block 3351 len 32768
11803 146/300 Read block 1659 len 16384
11804 147/300 Read block 1250 len 40960
11805 148/300 Write block 143 len 16384 data: 4 4 4 5
11806 149/300 Write block 2117 len 4096 data: 4
11807 150/300 Flush
11808 151/300 Read block 806 len 16384
11809 152/300 Flush
11810 153/300 Write block 3987 len 4096 data: 3
11811 154/300 Write block 1144 len 28672 data: 4 2 2 2 3 3 4
11812 155/300 Read block 2384 len 28672
11813 156/300 Read block 1069 len 16384
11814 157/300 Read block 2819 len 28672
11815 158/300 Read block 4135 len 4096
11816 159/300 Read block 534 len 16384
11817 160/300 Write block 3726 len 20480 data: 6 6 6 7 6
11818 161/300 Write block 3602 len 4096 data: 3
11819 162/300 Read block 3969 len 28672
11820 163/300 Read block 658 len 40960
11821 164/300 Read block 4071 len 12288
11822 165/300 Read block 1198 len 24576
11823 166/300 Read block 4250 len 40960
11824 167/300 Read block 3357 len 24576
11825 168/300 Read block 1624 len 28672
11826 169/300 Read block 3543 len 8192
11827 170/300 Write block 2390 len 12288 data: 4 4 4
11828 171/300 Read block 4370 len 32768
11829 172/300 Read block 4134 len 12288
11830 173/300 Write block 3071 len 32768 data: 3 3 3 3 3 2 3 3
11831 174/300 Write block 961 len 20480 data: 8 8 7 5 5
11832 175/300 Read block 3541 len 40960
11833 176/300 Write block 4011 len 24576 data: 4 5 6 7 6 5
11834 177/300 Write block 3572 len 40960 data: 4 5 4 3 3 3 3 2 2 2
11835 178/300 Write block 3590 len 8192 data: 4 4
11836 179/300 Read block 4833 len 12288
11837 180/300 Flush
11838 181/300 Flush
11839 182/300 Write block 102 len 20480 data: 2 3 3 3 3
11840 183/300 Write block 2067 len 40960 data: 4 4 4 4 4 4 4 4 3 2
11841 184/300 Write block 2230 len 32768 data: 3 3 3 2 2 2 3 3
11842 185/300 Write block 860 len 16384 data: 2 2 2 2
11843 186/300 Read block 1767 len 4096
11844 187/300 Read block 3040 len 36864
11845 188/300 Flush
11846 189/300 Flush
11847 190/300 Write block 3536 len 4096 data: 3
11848 191/300 Write block 3876 len 20480 data: 5 5 3 4 4
11849 192/300 Write block 2602 len 40960 data: 5 4 4 4 4 3 3 3 4 4
11850 193/300 Flush
11851 194/300 Read block 4163 len 4096
11852 195/300 Write block 4334 len 8192 data: 5 8
11853 196/300 Write block 611 len 36864 data: 3 3 3 3 3 2 2 3 3
11854 197/300 Read block 4452 len 32768
11855 198/300 Flush
11856 199/300 Read block 4229 len 36864
11857 200/300 Write block 4860 len 16384 data: 5 5 6 6
11858 201/300 Write block 2210 len 20480 data: 3 4 3 3 4
11859 202/300 Read block 968 len 24576
11860 203/300 Read block 2172 len 28672
11861 204/300 Write block 234 len 4096 data: 3
11862 205/300 Write block 4181 len 36864 data: 4 4 4 3 3 3 2 3 3
11863 206/300 Write block 4548 len 40960 data: 5 5 4 3 3 2 2 3 3 3
11864 207/300 Read block 1896 len 28672
11865 208/300 Read block 1251 len 40960
11866 209/300 Read block 1752 len 8192
11867 210/300 Write block 84 len 4096 data: 4
11868 211/300 Read block 553 len 8192
11869 212/300 Flush
11870 213/300 Write block 4037 len 4096 data: 3
11871 214/300 Read block 2674 len 32768
11872 215/300 Flush
11873 216/300 Write block 1033 len 40960 data: 3 3 3 4 4 4 3 5 5 4
11874 217/300 Flush
11875 218/300 Read block 99 len 40960
11876 219/300 Read block 2994 len 4096
11877 220/300 Flush
11878 221/300 Read block 1967 len 32768
11879 222/300 Read block 2589 len 20480
11880 223/300 Flush
11881 224/300 Write block 4864 len 32768 data: 6 5 4 5 4 3 5 4
11882 225/300 Read block 518 len 40960
11883 226/300 Write block 1613 len 16384 data: 4 4 6 5
11884 227/300 Write block 3978 len 24576 data: 3 3 2 3 3 3
11885 228/300 Read block 149 len 4096
11886 229/300 Write block 602 len 12288 data: 3 3 3
11887 230/300 Read block 2440 len 16384
11888 231/300 Write block 4247 len 36864 data: 2 3 3 3 4 5 5 7 6
11889 232/300 Write block 3229 len 20480 data: 4 4 5 4 3
11890 233/300 Write block 2091 len 8192 data: 3 3
11891 234/300 Write block 2049 len 4096 data: 5
11892 235/300 Read block 2298 len 12288
11893 236/300 Read block 2011 len 8192
11894 237/300 Write block 4564 len 28672 data: 4 5 4 4 4 4 4
11895 238/300 Read block 2947 len 24576
11896 239/300 Read block 4925 len 36864
11897 240/300 Write block 4692 len 24576 data: 3 3 3 3 3 3
11898 241/300 Read block 968 len 28672
11899 242/300 Flush
11900 243/300 Write block 3021 len 20480 data: 3 2 2 2 3
11901 244/300 Write block 918 len 40960 data: 6 6 6 5 5 3 3 3 3 3
11902 245/300 Read block 4917 len 20480
11903 246/300 Read block 4766 len 32768
11904 247/300 Read block 1199 len 16384
11905 248/300 Read block 1013 len 4096
11906 249/300 Read block 3515 len 24576
11907 250/300 Write block 3917 len 36864 data: 3 3 3 3 4 4 4 4 4
11908 251/300 Read block 161 len 32768
11909 252/300 Write block 3811 len 20480 data: 2 2 3 3 6
11910 253/300 Read block 3051 len 40960
11911 254/300 Write block 1141 len 8192 data: 7 5
11912 255/300 Write block 3049 len 40960 data: 3 3 3 3 4 4 4 3 3 4
11913 256/300 Read block 905 len 28672
11914 257/300 Read block 793 len 8192
11915 258/300 Write block 4408 len 28672 data: 2 2 2 3 3 3 3
11916 259/300 Read block 1717 len 12288
11917 260/300 Write block 3112 len 16384 data: 5 4 3 2
11918 261/300 Write block 3204 len 20480 data: 5 5 5 6 5
11919 262/300 Read block 4104 len 4096
11920 263/300 Write block 2604 len 32768 data: 5 5 5 4 4 4 5 5
11921 264/300 Write block 2535 len 8192 data: 4 5
11922 265/300 Flush
11923 266/300 Write block 298 len 12288 data: 5 4 3
11924 267/300 Write block 843 len 24576 data: 4 4 3 4 3 3
11925 268/300 Read block 3077 len 20480
11926 269/300 Write block 3096 len 28672 data: 4 3 3 3 3 4 3
11927 270/300 Write block 1549 len 36864 data: 5 4 4 3 3 3 4 4 4
11928 271/300 Read block 3723 len 28672
11929 272/300 Write block 3985 len 32768 data: 3 3 4 3 3 2 2 2
11930 273/300 Flush
11931 274/300 Write block 3595 len 28672 data: 4 3 3 3 3 3 3
11932 275/300 Read block 4108 len 12288
11933 276/300 Read block 3892 len 16384
11934 277/300 Read block 3234 len 32768
11935 278/300 Read block 3836 len 4096
11936 279/300 Flush
11937 280/300 Read block 1341 len 12288
11938 281/300 Write block 2466 len 16384 data: 4 3 3 3
11939 282/300 Write block 3623 len 16384 data: 3 3 3 4
11940 283/300 Flush
11941 284/300 Write block 4364 len 16384 data: 6 6 7 5
11942 285/300 Read block 3927 len 28672
11943 286/300 Read block 613 len 32768
11944 287/300 Read block 4413 len 20480
11945 288/300 Read block 1327 len 24576
11946 289/300 Read block 3535 len 40960
11947 290/300 Write block 4620 len 28672 data: 4 3 3 3 3 3 5
11948 291/300 Read block 1238 len 36864
11949 292/300 Read block 610 len 40960
11950 293/300 Read block 1869 len 4096
11951 294/300 Write block 2492 len 4096 data: 2
11952 295/300 Read block 1152 len 12288
11953 296/300 Read block 700 len 32768
11954 297/300 Write block 4186 len 24576 data: 4 3 4 4 3 3
11955 298/300 Read block 2256 len 40960
11956 299/300 Write block 4590 len 12288 data: 5 5 4
11957 300/300 Read block 4046 len 40960
11958 Main task has work to do, go find it
11959 got dsc Start(0)
11960 start 0
11961 Feb 28 22:15:39.756 INFO request completed, latency_us: 232, response_code: 204, uri: /start/cid/0, method: POST, req_id: 0c7e14e9-d019-413e-b77e-8615011dd1cb, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
11962 Replay: started 0, returned:Ok(())
11963 ----------------------------------------------------------------
11964 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11965 States: Offline Active Active
11966 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
11967 3918 Acked 4917 Read 2 New Done Done true
11968 3919 Acked 4918 Read 7 New Done Done true
11969 3920 Acked 4919 Read 5 New Done Done true
11970 3921 Acked 4920 Read 9 New Done Done false
11971 3922 Acked 4921 Write 9 New Done Done false
11972 3923 Acked 4922 Read 5 New Done Done false
11973 3924 Acked 4923 Write 1 New Done Done false
11974 3925 Acked 4924 Flush 0 New Done Done false
11975 3926 Acked 4925 Write 7 New Done Done false
11976 3927 Acked 4926 Read 9 New Done Done false
11977 3928 Acked 4927 Write 1 New Done Done false
11978 3929 Acked 4928 Read 6 New Done Done false
11979 3930 Acked 4929 Write 5 New Done Done false
11980 3931 Acked 4930 Read 9 New Done Done false
11981 3932 Acked 4931 Write 5 New Done Done false
11982 3933 Acked 4932 Read 10 New Done Done false
11983 3934 Acked 4933 Write 10 New Done Done false
11984 3935 Acked 4934 Write 5 New Done Done false
11985 3936 Acked 4935 Write 7 New Done Done false
11986 3937 Acked 4936 Read 2 New Done Done false
11987 3938 Acked 4937 Read 7 New Done Done false
11988 3939 Acked 4938 Write 4 New Done Done false
11989 3940 Acked 4939 Write 2 New Done Done false
11990 3941 Acked 4940 Read 6 New Done Done false
11991 3942 Acked 4941 Read 7 New Done Done false
11992 3943 Acked 4942 Read 6 New Done Done false
11993 3944 Acked 4943 Read 6 New Done Done false
11994 3945 Acked 4944 Write 7 New Done Done false
11995 3946 Acked 4945 Write 7 New Done Done false
11996 3947 Acked 4946 Read 10 New Done Done false
11997 3948 Acked 4947 Write 5 New Done Done false
11998 3949 Acked 4948 Read 5 New Done Done false
11999 3950 Acked 4949 Read 4 New Done Done false
12000 3951 Acked 4950 Write 6 New Done Done false
12001 3952 Acked 4951 Write 5 New Done Done false
12002 3953 Acked 4952 Read 7 New Done Done false
12003 3954 Acked 4953 Write 1 New Done Done false
12004 3955 Acked 4954 Write 8 New Done Done false
12005 3956 Acked 4955 Write 5 New Done Done false
12006 3957 Acked 4956 Write 9 New Done Done false
12007 3958 Acked 4957 Read 9 New Done Done false
12008 3959 Acked 4958 Read 8 New Done Done false
12009 3960 Acked 4959 Read 3 New Done Done false
12010 3961 Acked 4960 Read 6 New Done Done false
12011 3962 Acked 4961 Read 5 New Done Done false
12012 3963 Acked 4962 Read 1 New Done Done false
12013 3964 Acked 4963 Read 8 New Done Done false
12014 3965 Acked 4964 Read 4 New Done Done false
12015 3966 Acked 4965 Flush 0 New Done Done false
12016 3967 Acked 4966 Write 2 New Done Done false
12017 3968 Acked 4967 Write 8 New Done Done false
12018 3969 Acked 4968 Read 4 New Done Done false
12019 3970 Acked 4969 Write 9 New Done Done false
12020 3971 Acked 4970 Write 5 New Done Done false
12021 3972 Acked 4971 Read 6 New Done Done false
12022 3973 Acked 4972 Write 2 New Done Done false
12023 3974 Acked 4973 Write 4 New Done Done false
12024 3975 Acked 4974 Read 5 New Done Done false
12025 3976 Acked 4975 Write 2 New Done Done false
12026 3977 Acked 4976 Read 3 New Done Done false
12027 3978 Acked 4977 Write 4 New Done Done false
12028 3979 Acked 4978 Write 9 New Done Done false
12029 3980 Acked 4979 Read 3 New Done Done false
12030 3981 Acked 4980 Flush 0 New Done Done false
12031 3982 Acked 4981 Read 9 New Done Done false
12032 3983 Acked 4982 Read 9 New Done Done false
12033 3984 Acked 4983 Read 8 New Done Done false
12034 3985 Acked 4984 Read 2 New Done Done false
12035 3986 Acked 4985 Read 10 New Done Done false
12036 3987 Acked 4986 Read 10 New Done Done false
12037 3988 Acked 4987 Flush 0 New Done Done false
12038 3989 Acked 4988 Read 5 New Done Done false
12039 3990 Acked 4989 Flush 0 New Done Done false
12040 3991 Acked 4990 Read 4 New Done Done false
12041 3992 Acked 4991 Read 1 New Done Done false
12042 3993 Acked 4992 Read 6 New Done Done false
12043 3994 Acked 4993 Read 8 New Done Done false
12044 3995 Acked 4994 Write 4 New Done Done false
12045 3996 Acked 4995 Write 3 New Done Done false
12046 3997 Acked 4996 Write 2 New Done Done false
12047 3998 Acked 4997 Read 1 New Done Done false
12048 3999 Acked 4998 Read 6 New Done Done false
12049 4000 Acked 4999 Read 6 New Done Done false
12050 4001 Acked 5000 Read 8 New Done Done false
12051 4002 Acked 5001 Write 3 New Done Done false
12052 4003 Acked 5002 Write 8 New Done Done false
12053 4004 Acked 5003 Read 2 New Done Done false
12054 4005 Acked 5004 Write 10 New Done Done false
12055 4006 Acked 5005 Read 6 New Done Done false
12056 4007 Acked 5006 Write 3 New Done Done false
12057 4008 Acked 5007 Write 3 New Done Done false
12058 4009 Acked 5008 Read 2 New Done Done false
12059 4010 Acked 5009 Write 3 New Done Done false
12060 4011 Acked 5010 Write 7 New Done Done false
12061 4012 Acked 5011 Write 4 New Done Done false
12062 4013 Acked 5012 Flush 0 New Done Done false
12063 4014 Acked 5013 Read 9 New Done Done false
12064 4015 Acked 5014 Read 5 New Done Done false
12065 4016 Acked 5015 Write 5 New Done Done false
12066 4017 Acked 5016 Write 8 New Done Done false
12067 4018 Acked 5017 Write 3 New Done Done false
12068 4019 Acked 5018 Flush 0 New Done Done false
12069 4020 Acked 5019 Read 2 New Done Done false
12070 4021 Acked 5020 Write 2 New Done Done false
12071 4022 Acked 5021 Write 3 New Done Done false
12072 4023 Acked 5022 Read 8 New Done Done false
12073 4024 Acked 5023 Read 9 New Done Done false
12074 4025 Acked 5024 Write 3 New Done Done false
12075 4026 Acked 5025 Write 1 New Done Done false
12076 4027 Acked 5026 Read 9 New Done Done false
12077 4028 Acked 5027 Read 1 New Done Done false
12078 4029 Acked 5028 Write 4 New Done Done false
12079 4030 Acked 5029 Write 9 New Done Done false
12080 4031 Acked 5030 Write 3 New Done Done false
12081 4032 Acked 5031 Read 2 New Done Done false
12082 4033 Acked 5032 Write 6 New Done Done false
12083 4034 Acked 5033 Read 1 New Done Done false
12084 4035 Acked 5034 Read 9 New Done Done false
12085 4036 Acked 5035 Write 6 New Done Done false
12086 4037 Acked 5036 Flush 0 New Done Done false
12087 4038 Acked 5037 Read 1 New Done Done false
12088 4039 Acked 5038 Read 8 New Done Done false
12089 4040 Acked 5039 Write 8 New Done Done false
12090 4041 Acked 5040 Read 5 New Done Done false
12091 4042 Acked 5041 Read 8 New Done Done false
12092 4043 Acked 5042 Read 7 New Done Done false
12093 4044 Acked 5043 Write 3 New Done Done false
12094 4045 Acked 5044 Write 1 New Done Done false
12095 4046 Acked 5045 Read 1 New Done Done false
12096 4047 Acked 5046 Write 6 New Done Done false
12097 4048 Acked 5047 Read 1 New Done Done false
12098 4049 Acked 5048 Read 1 New Done Done false
12099 4050 Acked 5049 Write 2 New Done Done false
12100 4051 Acked 5050 Flush 0 New Done Done false
12101 4052 Acked 5051 Write 9 New Done Done false
12102 4053 Acked 5052 Read 9 New Done Done false
12103 4054 Acked 5053 Read 2 New Done Done false
12104 4055 Acked 5054 Read 3 New Done Done false
12105 4056 Acked 5055 Read 8 New Done Done false
12106 4057 Acked 5056 Read 6 New Done Done false
12107 4058 Acked 5057 Write 10 New Done Done false
12108 4059 Acked 5058 Write 8 New Done Done false
12109 4060 Acked 5059 Read 2 New Done Done false
12110 4061 Acked 5060 Flush 0 New Done Done false
12111 4062 Acked 5061 Read 8 New Done Done false
12112 4063 Acked 5062 Read 4 New Done Done false
12113 4064 Acked 5063 Read 10 New Done Done false
12114 4065 Acked 5064 Write 4 New Done Done false
12115 4066 Acked 5065 Write 1 New Done Done false
12116 4067 Acked 5066 Flush 0 New Done Done false
12117 4068 Acked 5067 Read 4 New Done Done false
12118 4069 Acked 5068 Flush 0 New Done Done false
12119 4070 Acked 5069 Write 1 New Done Done false
12120 4071 Acked 5070 Write 7 New Done Done false
12121 4072 Acked 5071 Read 7 New Done Done false
12122 4073 Acked 5072 Read 4 New Done Done false
12123 4074 Acked 5073 Read 7 New Done Done false
12124 4075 Acked 5074 Read 1 New Done Done false
12125 4076 Acked 5075 Read 4 New Done Done false
12126 4077 Acked 5076 Write 5 New Done Done false
12127 4078 Acked 5077 Write 1 New Done Done false
12128 4079 Acked 5078 Read 7 New Done Done false
12129 4080 Acked 5079 Read 10 New Done Done false
12130 4081 Acked 5080 Read 3 New Done Done false
12131 4082 Acked 5081 Read 6 New Done Done false
12132 4083 Acked 5082 Read 10 New Done Done false
12133 4084 Acked 5083 Read 6 New Done Done false
12134 4085 Acked 5084 Read 7 New Done Done false
12135 4086 Acked 5085 Read 2 New Done Done false
12136 4087 Acked 5086 Write 3 New Done Done false
12137 4088 Acked 5087 Read 8 New Done Done false
12138 4089 Acked 5088 Read 3 New Done Done false
12139 4090 Acked 5089 Write 8 New Done Done false
12140 4091 Acked 5090 Write 5 New Done Done false
12141 4092 Acked 5091 Read 10 New Done Done false
12142 4093 Acked 5092 Write 6 New Done Done false
12143 4094 Acked 5093 Write 10 New Done Done false
12144 4095 Acked 5094 Write 2 New Done Done false
12145 4096 Acked 5095 Read 3 New Done Done false
12146 4097 Acked 5096 Flush 0 New Done Done false
12147 4098 Acked 5097 Flush 0 New Done Done false
12148 4099 Acked 5098 Write 5 New Done Done false
12149 4100 Acked 5099 Write 10 New Done Done false
12150 4101 Acked 5100 Write 8 New Done Done false
12151 4102 Acked 5101 Write 4 New Done Done false
12152 4103 Acked 5102 Read 1 New Done Done false
12153 4104 Acked 5103 Read 9 New Done Done false
12154 4105 Acked 5104 Flush 0 New Done Done false
12155 4106 Acked 5105 Flush 0 New Done Done false
12156 4107 Acked 5106 Write 1 New Done Done false
12157 4108 Acked 5107 Write 5 New Done Done false
12158 4109 Acked 5108 Write 10 New Done Done false
12159 4110 Acked 5109 Flush 0 New Done Done false
12160 4111 Acked 5110 Read 1 New Done Done false
12161 4112 Acked 5111 Write 2 New Done Done false
12162 4113 Acked 5112 Write 9 New Done Done false
12163 4114 Acked 5113 Read 8 New Done Done false
12164 4115 Acked 5114 Flush 0 New Done Done false
12165 4116 Acked 5115 Read 9 New Done Done false
12166 4117 Acked 5116 Write 4 New Done Done false
12167 4118 Acked 5117 Write 5 New Done Done false
12168 4119 Acked 5118 Read 6 New Done Done false
12169 4120 Acked 5119 Read 7 New Done Done false
12170 4121 Acked 5120 Write 1 New Done Done false
12171 4122 Acked 5121 Write 9 New Done Done false
12172 4123 Acked 5122 Write 10 New Done Done false
12173 4124 Acked 5123 Read 7 New Done Done false
12174 4125 Acked 5124 Read 10 New Done Done false
12175 4126 Acked 5125 Read 2 New Done Done false
12176 4127 Acked 5126 Write 1 New Done Done false
12177 4128 Acked 5127 Read 2 New Done Done false
12178 4129 Acked 5128 Flush 0 New Done Done false
12179 4130 Acked 5129 Write 1 New Done Done false
12180 4131 Acked 5130 Read 8 New Done Done false
12181 4132 Acked 5131 Flush 0 New Done Done false
12182 4133 Acked 5132 Write 10 New Done Done false
12183 4134 Acked 5133 Flush 0 New Done Done false
12184 4135 Acked 5134 Read 10 New Done Done false
12185 4136 Acked 5135 Read 1 New Done Done false
12186 4137 Acked 5136 Flush 0 New Done Done false
12187 4138 Acked 5137 Read 8 New Done Done false
12188 4139 Acked 5138 Read 5 New Done Done false
12189 4140 Acked 5139 Flush 0 New Done Done false
12190 4141 Acked 5140 Write 8 New Done Done false
12191 4142 Acked 5141 Read 10 New Done Done false
12192 4143 Acked 5142 Write 4 New Done Done false
12193 4144 Acked 5143 Write 6 New Done Done false
12194 4145 Acked 5144 Read 1 New Done Done false
12195 4146 Acked 5145 Write 3 New Done Done false
12196 4147 Acked 5146 Read 4 New Done Done false
12197 4148 Acked 5147 Write 9 New Done Done false
12198 4149 Acked 5148 Write 5 New Done Done false
12199 4150 Acked 5149 Write 2 New Done Done false
12200 4151 Acked 5150 Write 1 New Done Done false
12201 4152 Acked 5151 Read 3 New Done Done false
12202 4153 Acked 5152 Read 2 New Done Done false
12203 4154 Acked 5153 Write 7 New Done Done false
12204 4155 Acked 5154 Read 6 New Done Done false
12205 4156 Acked 5155 Read 9 New Done Done false
12206 4157 Acked 5156 Write 6 New Done Done false
12207 4158 Acked 5157 Read 7 New Done Done false
12208 4159 Acked 5158 Flush 0 New Done Done false
12209 4160 Acked 5159 Write 5 New Done Done false
12210 4161 Acked 5160 Write 10 New Done Done false
12211 4162 Acked 5161 Flush 0 New Done Done false
12212 4163 Acked 5162 Read 5 New Done Done false
12213 4164 Acked 5163 Read 8 New Done Done false
12214 4165 Acked 5164 Read 4 New Done Done false
12215 4166 Acked 5165 Read 1 New Done Done false
12216 4167 Acked 5166 Read 6 New Done Done false
12217 4168 Acked 5167 Write 9 New Done Done false
12218 4169 Acked 5168 Read 8 New Done Done false
12219 4170 Acked 5169 Write 5 New Done Done false
12220 4171 Acked 5170 Read 10 New Done Done false
12221 4172 Acked 5171 Write 2 New Done Done false
12222 4173 Acked 5172 Write 10 New Done Done false
12223 4174 Acked 5173 Read 7 New Done Done false
12224 4175 Acked 5174 Read 2 New Done Done false
12225 4176 Acked 5175 Write 7 New Done Done false
12226 4177 Acked 5176 Read 3 New Done Done false
12227 4178 Acked 5177 Write 4 New Done Done false
12228 4179 Acked 5178 Write 5 New Done Done false
12229 4180 Acked 5179 Read 1 New Done Done false
12230 4181 Acked 5180 Write 8 New Done Done false
12231 4182 Acked 5181 Write 2 New Done Done false
12232 4183 Acked 5182 Flush 0 New Done Done false
12233 4184 Acked 5183 Write 3 New Done Done false
12234 4185 Acked 5184 Write 6 New Done Done false
12235 4186 Acked 5185 Read 5 New Done Done false
12236 4187 Acked 5186 Write 7 New Done Done false
12237 4188 Acked 5187 Write 9 New Done Done false
12238 4189 Acked 5188 Read 7 New Done Done false
12239 4190 Acked 5189 Write 8 New Done Done false
12240 4191 Acked 5190 Flush 0 New Done Done false
12241 4192 Acked 5191 Write 7 New Done Done false
12242 4193 Acked 5192 Read 3 New Done Done false
12243 4194 Acked 5193 Read 4 New Done Done false
12244 4195 Acked 5194 Read 8 New Done Done false
12245 4196 Acked 5195 Read 1 New Done Done false
12246 4197 Acked 5196 Flush 0 New Done Done false
12247 4198 Acked 5197 Read 3 New Done Done false
12248 4199 Acked 5198 Write 4 New Done Done false
12249 4200 Acked 5199 Write 4 New Done Done false
12250 4201 Acked 5200 Flush 0 New Done Done false
12251 4202 Acked 5201 Write 4 New Done Done false
12252 4203 Acked 5202 Read 7 New Done Done false
12253 4204 Acked 5203 Read 8 New Done Done false
12254 4205 Acked 5204 Read 5 New Done Done false
12255 4206 Acked 5205 Read 6 New Done Done false
12256 4207 Acked 5206 Read 10 New Done Done false
12257 4208 Acked 5207 Write 7 New Done Done false
12258 4209 Acked 5208 Read 9 New Done Done false
12259 4210 Acked 5209 Read 10 New Done Done false
12260 4211 Acked 5210 Read 1 New Done Done false
12261 4212 Acked 5211 Write 1 New Done Done false
12262 4213 Acked 5212 Read 3 New Done Done false
12263 4214 Acked 5213 Read 8 New Done Done false
12264 4215 Acked 5214 Write 6 New Done Done false
12265 4216 Acked 5215 Read 10 New Done Done false
12266 4217 Acked 5216 Write 3 New Done Done false
12267 4218 Acked 5217 Read 10 New Done Done false
12268 STATES DS:0 DS:1 DS:2 TOTAL
12269 New 301 0 0 301
12270 Sent 0 0 0 0
12271 Done 0 301 301 602
12272 Skipped 0 0 0 0
12273 Error 0 0 0 0
12274 Last Flush: 4916 5200 5200
12275 Downstairs last five completed: 4916 4915 4914 4913 4912
12276 Upstairs last five completed: 4218 4217 4216 4215 4214
12277 CLIENT: Up:0 ds:301 act:2
122782024-02-28T22:15:39.991ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
122792024-02-28T22:15:39.991ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
122802024-02-28T22:15:39.991ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
122812024-02-28T22:15:39.991ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
122822024-02-28T22:15:39.991ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
122832024-02-28T22:15:39.991ZINFOcrucible: [0] client re-new 301 jobs since flush 4916 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
122842024-02-28T22:15:39.992ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
122852024-02-28T22:15:40.992ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
122862024-02-28T22:15:40.992ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
122872024-02-28T22:15:40.992ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
122882024-02-28T22:15:40.992ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
122892024-02-28T22:15:40.992ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
122902024-02-28T22:15:40.992ZINFOcrucible: [0] client re-new 302 jobs since flush 4916 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
122912024-02-28T22:15:40.993ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
122922024-02-28T22:15:41.994ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
122932024-02-28T22:15:41.994ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
122942024-02-28T22:15:41.994ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
122952024-02-28T22:15:41.994ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
122962024-02-28T22:15:41.994ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
122972024-02-28T22:15:41.994ZINFOcrucible: [0] client re-new 302 jobs since flush 4916 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
122982024-02-28T22:15:41.994ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
12299 [8810] got start action
12300 [8810] I am going to restart
12301 Starting downstairs at port 8810
12302 Make output file at "/tmp/dsc/downstairs-8810.txt"
12303 [8810][0] reports Starting
123042024-02-28T22:15:42.995ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
123052024-02-28T22:15:42.995ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
123062024-02-28T22:15:42.995ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
123072024-02-28T22:15:42.995ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
123082024-02-28T22:15:42.995ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
123092024-02-28T22:15:42.995ZINFOcrucible: [0] client re-new 302 jobs since flush 4916 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
123102024-02-28T22:15:42.995ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
12311 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1178)
12312 [8810][0] reports Running
12313 ----------------------------------------------------------------
12314 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
12315 States: Offline Active Active
12316 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
12317 3918 Acked 4917 Read 2 New Done Done true
12318 3919 Acked 4918 Read 7 New Done Done true
12319 3920 Acked 4919 Read 5 New Done Done true
12320 3921 Acked 4920 Read 9 New Done Done true
12321 3922 Acked 4921 Write 9 New Done Done true
12322 3923 Acked 4922 Read 5 New Done Done true
12323 3924 Acked 4923 Write 1 New Done Done true
12324 3925 Acked 4924 Flush 0 New Done Done true
12325 3926 Acked 4925 Write 7 New Done Done true
12326 3927 Acked 4926 Read 9 New Done Done true
12327 3928 Acked 4927 Write 1 New Done Done true
12328 3929 Acked 4928 Read 6 New Done Done true
12329 3930 Acked 4929 Write 5 New Done Done true
12330 3931 Acked 4930 Read 9 New Done Done true
12331 3932 Acked 4931 Write 5 New Done Done true
12332 3933 Acked 4932 Read 10 New Done Done true
12333 3934 Acked 4933 Write 10 New Done Done true
12334 3935 Acked 4934 Write 5 New Done Done true
12335 3936 Acked 4935 Write 7 New Done Done true
12336 3937 Acked 4936 Read 2 New Done Done true
12337 3938 Acked 4937 Read 7 New Done Done true
12338 3939 Acked 4938 Write 4 New Done Done true
12339 3940 Acked 4939 Write 2 New Done Done true
12340 3941 Acked 4940 Read 6 New Done Done true
12341 3942 Acked 4941 Read 7 New Done Done true
12342 3943 Acked 4942 Read 6 New Done Done true
12343 3944 Acked 4943 Read 6 New Done Done true
12344 3945 Acked 4944 Write 7 New Done Done true
12345 3946 Acked 4945 Write 7 New Done Done true
12346 3947 Acked 4946 Read 10 New Done Done true
12347 3948 Acked 4947 Write 5 New Done Done true
12348 3949 Acked 4948 Read 5 New Done Done true
12349 3950 Acked 4949 Read 4 New Done Done true
12350 3951 Acked 4950 Write 6 New Done Done true
12351 3952 Acked 4951 Write 5 New Done Done true
12352 3953 Acked 4952 Read 7 New Done Done true
12353 3954 Acked 4953 Write 1 New Done Done true
12354 3955 Acked 4954 Write 8 New Done Done true
12355 3956 Acked 4955 Write 5 New Done Done true
12356 3957 Acked 4956 Write 9 New Done Done true
12357 3958 Acked 4957 Read 9 New Done Done true
12358 3959 Acked 4958 Read 8 New Done Done true
12359 3960 Acked 4959 Read 3 New Done Done true
12360 3961 Acked 4960 Read 6 New Done Done true
12361 3962 Acked 4961 Read 5 New Done Done true
12362 3963 Acked 4962 Read 1 New Done Done true
12363 3964 Acked 4963 Read 8 New Done Done true
12364 3965 Acked 4964 Read 4 New Done Done true
12365 3966 Acked 4965 Flush 0 New Done Done true
12366 3967 Acked 4966 Write 2 New Done Done true
12367 3968 Acked 4967 Write 8 New Done Done true
12368 3969 Acked 4968 Read 4 New Done Done true
12369 3970 Acked 4969 Write 9 New Done Done true
12370 3971 Acked 4970 Write 5 New Done Done true
12371 3972 Acked 4971 Read 6 New Done Done true
12372 3973 Acked 4972 Write 2 New Done Done true
12373 3974 Acked 4973 Write 4 New Done Done true
12374 3975 Acked 4974 Read 5 New Done Done true
12375 3976 Acked 4975 Write 2 New Done Done true
12376 3977 Acked 4976 Read 3 New Done Done true
12377 3978 Acked 4977 Write 4 New Done Done true
12378 3979 Acked 4978 Write 9 New Done Done true
12379 3980 Acked 4979 Read 3 New Done Done true
12380 3981 Acked 4980 Flush 0 New Done Done true
12381 3982 Acked 4981 Read 9 New Done Done true
12382 3983 Acked 4982 Read 9 New Done Done true
12383 3984 Acked 4983 Read 8 New Done Done true
12384 3985 Acked 4984 Read 2 New Done Done true
12385 3986 Acked 4985 Read 10 New Done Done true
12386 3987 Acked 4986 Read 10 New Done Done true
12387 3988 Acked 4987 Flush 0 New Done Done true
12388 3989 Acked 4988 Read 5 New Done Done true
12389 3990 Acked 4989 Flush 0 New Done Done true
12390 3991 Acked 4990 Read 4 New Done Done true
12391 3992 Acked 4991 Read 1 New Done Done true
12392 3993 Acked 4992 Read 6 New Done Done true
12393 3994 Acked 4993 Read 8 New Done Done true
12394 3995 Acked 4994 Write 4 New Done Done true
12395 3996 Acked 4995 Write 3 New Done Done true
12396 3997 Acked 4996 Write 2 New Done Done true
12397 3998 Acked 4997 Read 1 New Done Done true
12398 3999 Acked 4998 Read 6 New Done Done true
12399 4000 Acked 4999 Read 6 New Done Done true
12400 4001 Acked 5000 Read 8 New Done Done true
12401 4002 Acked 5001 Write 3 New Done Done true
12402 4003 Acked 5002 Write 8 New Done Done true
12403 4004 Acked 5003 Read 2 New Done Done true
12404 4005 Acked 5004 Write 10 New Done Done true
12405 4006 Acked 5005 Read 6 New Done Done true
12406 4007 Acked 5006 Write 3 New Done Done true
12407 4008 Acked 5007 Write 3 New Done Done true
12408 4009 Acked 5008 Read 2 New Done Done true
12409 4010 Acked 5009 Write 3 New Done Done true
12410 4011 Acked 5010 Write 7 New Done Done true
12411 4012 Acked 5011 Write 4 New Done Done true
12412 4013 Acked 5012 Flush 0 New Done Done true
12413 4014 Acked 5013 Read 9 New Done Done true
12414 4015 Acked 5014 Read 5 New Done Done true
12415 4016 Acked 5015 Write 5 New Done Done true
12416 4017 Acked 5016 Write 8 New Done Done true
12417 4018 Acked 5017 Write 3 New Done Done true
12418 4019 Acked 5018 Flush 0 New Done Done true
12419 4020 Acked 5019 Read 2 New Done Done true
12420 4021 Acked 5020 Write 2 New Done Done true
12421 4022 Acked 5021 Write 3 New Done Done true
12422 4023 Acked 5022 Read 8 New Done Done true
12423 4024 Acked 5023 Read 9 New Done Done true
12424 4025 Acked 5024 Write 3 New Done Done true
12425 4026 Acked 5025 Write 1 New Done Done true
12426 4027 Acked 5026 Read 9 New Done Done true
12427 4028 Acked 5027 Read 1 New Done Done true
12428 4029 Acked 5028 Write 4 New Done Done true
12429 4030 Acked 5029 Write 9 New Done Done true
12430 4031 Acked 5030 Write 3 New Done Done true
12431 4032 Acked 5031 Read 2 New Done Done true
12432 4033 Acked 5032 Write 6 New Done Done true
12433 4034 Acked 5033 Read 1 New Done Done true
12434 4035 Acked 5034 Read 9 New Done Done true
12435 4036 Acked 5035 Write 6 New Done Done true
12436 4037 Acked 5036 Flush 0 New Done Done true
12437 4038 Acked 5037 Read 1 New Done Done true
12438 4039 Acked 5038 Read 8 New Done Done true
12439 4040 Acked 5039 Write 8 New Done Done true
12440 4041 Acked 5040 Read 5 New Done Done true
12441 4042 Acked 5041 Read 8 New Done Done true
12442 4043 Acked 5042 Read 7 New Done Done true
12443 4044 Acked 5043 Write 3 New Done Done true
12444 4045 Acked 5044 Write 1 New Done Done true
12445 4046 Acked 5045 Read 1 New Done Done true
12446 4047 Acked 5046 Write 6 New Done Done true
12447 4048 Acked 5047 Read 1 New Done Done true
12448 4049 Acked 5048 Read 1 New Done Done true
12449 4050 Acked 5049 Write 2 New Done Done true
12450 4051 Acked 5050 Flush 0 New Done Done true
12451 4052 Acked 5051 Write 9 New Done Done true
12452 4053 Acked 5052 Read 9 New Done Done true
12453 4054 Acked 5053 Read 2 New Done Done true
12454 4055 Acked 5054 Read 3 New Done Done true
12455 4056 Acked 5055 Read 8 New Done Done true
12456 4057 Acked 5056 Read 6 New Done Done true
12457 4058 Acked 5057 Write 10 New Done Done true
12458 4059 Acked 5058 Write 8 New Done Done true
12459 4060 Acked 5059 Read 2 New Done Done true
12460 4061 Acked 5060 Flush 0 New Done Done true
12461 4062 Acked 5061 Read 8 New Done Done true
12462 4063 Acked 5062 Read 4 New Done Done true
12463 4064 Acked 5063 Read 10 New Done Done true
12464 4065 Acked 5064 Write 4 New Done Done true
12465 4066 Acked 5065 Write 1 New Done Done true
12466 4067 Acked 5066 Flush 0 New Done Done true
12467 4068 Acked 5067 Read 4 New Done Done true
12468 4069 Acked 5068 Flush 0 New Done Done true
12469 4070 Acked 5069 Write 1 New Done Done true
12470 4071 Acked 5070 Write 7 New Done Done true
12471 4072 Acked 5071 Read 7 New Done Done true
12472 4073 Acked 5072 Read 4 New Done Done true
12473 4074 Acked 5073 Read 7 New Done Done true
12474 4075 Acked 5074 Read 1 New Done Done true
12475 4076 Acked 5075 Read 4 New Done Done true
12476 4077 Acked 5076 Write 5 New Done Done true
12477 4078 Acked 5077 Write 1 New Done Done true
12478 4079 Acked 5078 Read 7 New Done Done true
12479 4080 Acked 5079 Read 10 New Done Done true
12480 4081 Acked 5080 Read 3 New Done Done true
12481 4082 Acked 5081 Read 6 New Done Done true
12482 4083 Acked 5082 Read 10 New Done Done true
12483 4084 Acked 5083 Read 6 New Done Done true
12484 4085 Acked 5084 Read 7 New Done Done true
12485 4086 Acked 5085 Read 2 New Done Done true
12486 4087 Acked 5086 Write 3 New Done Done true
12487 4088 Acked 5087 Read 8 New Done Done true
12488 4089 Acked 5088 Read 3 New Done Done true
12489 4090 Acked 5089 Write 8 New Done Done true
12490 4091 Acked 5090 Write 5 New Done Done true
12491 4092 Acked 5091 Read 10 New Done Done true
12492 4093 Acked 5092 Write 6 New Done Done true
12493 4094 Acked 5093 Write 10 New Done Done true
12494 4095 Acked 5094 Write 2 New Done Done true
12495 4096 Acked 5095 Read 3 New Done Done true
12496 4097 Acked 5096 Flush 0 New Done Done true
12497 4098 Acked 5097 Flush 0 New Done Done true
12498 4099 Acked 5098 Write 5 New Done Done true
12499 4100 Acked 5099 Write 10 New Done Done true
12500 4101 Acked 5100 Write 8 New Done Done true
12501 4102 Acked 5101 Write 4 New Done Done true
12502 4103 Acked 5102 Read 1 New Done Done true
12503 4104 Acked 5103 Read 9 New Done Done true
12504 4105 Acked 5104 Flush 0 New Done Done true
12505 4106 Acked 5105 Flush 0 New Done Done true
12506 4107 Acked 5106 Write 1 New Done Done true
12507 4108 Acked 5107 Write 5 New Done Done true
12508 4109 Acked 5108 Write 10 New Done Done true
12509 4110 Acked 5109 Flush 0 New Done Done true
12510 4111 Acked 5110 Read 1 New Done Done true
12511 4112 Acked 5111 Write 2 New Done Done true
12512 4113 Acked 5112 Write 9 New Done Done true
12513 4114 Acked 5113 Read 8 New Done Done true
12514 4115 Acked 5114 Flush 0 New Done Done true
12515 4116 Acked 5115 Read 9 New Done Done true
12516 4117 Acked 5116 Write 4 New Done Done true
12517 4118 Acked 5117 Write 5 New Done Done true
12518 4119 Acked 5118 Read 6 New Done Done true
12519 4120 Acked 5119 Read 7 New Done Done true
12520 4121 Acked 5120 Write 1 New Done Done true
12521 4122 Acked 5121 Write 9 New Done Done true
12522 4123 Acked 5122 Write 10 New Done Done true
12523 4124 Acked 5123 Read 7 New Done Done true
12524 4125 Acked 5124 Read 10 New Done Done true
12525 4126 Acked 5125 Read 2 New Done Done true
12526 4127 Acked 5126 Write 1 New Done Done true
12527 4128 Acked 5127 Read 2 New Done Done true
12528 4129 Acked 5128 Flush 0 New Done Done true
12529 4130 Acked 5129 Write 1 New Done Done true
12530 4131 Acked 5130 Read 8 New Done Done true
12531 4132 Acked 5131 Flush 0 New Done Done true
12532 4133 Acked 5132 Write 10 New Done Done true
12533 4134 Acked 5133 Flush 0 New Done Done true
12534 4135 Acked 5134 Read 10 New Done Done true
12535 4136 Acked 5135 Read 1 New Done Done true
12536 4137 Acked 5136 Flush 0 New Done Done true
12537 4138 Acked 5137 Read 8 New Done Done true
12538 4139 Acked 5138 Read 5 New Done Done true
12539 4140 Acked 5139 Flush 0 New Done Done true
12540 4141 Acked 5140 Write 8 New Done Done true
12541 4142 Acked 5141 Read 10 New Done Done true
12542 4143 Acked 5142 Write 4 New Done Done true
12543 4144 Acked 5143 Write 6 New Done Done true
12544 4145 Acked 5144 Read 1 New Done Done true
12545 4146 Acked 5145 Write 3 New Done Done true
12546 4147 Acked 5146 Read 4 New Done Done true
12547 4148 Acked 5147 Write 9 New Done Done true
12548 4149 Acked 5148 Write 5 New Done Done true
12549 4150 Acked 5149 Write 2 New Done Done true
12550 4151 Acked 5150 Write 1 New Done Done true
12551 4152 Acked 5151 Read 3 New Done Done true
12552 4153 Acked 5152 Read 2 New Done Done true
12553 4154 Acked 5153 Write 7 New Done Done true
12554 4155 Acked 5154 Read 6 New Done Done true
12555 4156 Acked 5155 Read 9 New Done Done true
12556 4157 Acked 5156 Write 6 New Done Done true
12557 4158 Acked 5157 Read 7 New Done Done true
12558 4159 Acked 5158 Flush 0 New Done Done true
12559 4160 Acked 5159 Write 5 New Done Done true
12560 4161 Acked 5160 Write 10 New Done Done true
12561 4162 Acked 5161 Flush 0 New Done Done true
12562 4163 Acked 5162 Read 5 New Done Done true
12563 4164 Acked 5163 Read 8 New Done Done true
12564 4165 Acked 5164 Read 4 New Done Done true
12565 4166 Acked 5165 Read 1 New Done Done true
12566 4167 Acked 5166 Read 6 New Done Done true
12567 4168 Acked 5167 Write 9 New Done Done true
12568 4169 Acked 5168 Read 8 New Done Done true
12569 4170 Acked 5169 Write 5 New Done Done true
12570 4171 Acked 5170 Read 10 New Done Done true
12571 4172 Acked 5171 Write 2 New Done Done true
12572 4173 Acked 5172 Write 10 New Done Done true
12573 4174 Acked 5173 Read 7 New Done Done true
12574 4175 Acked 5174 Read 2 New Done Done true
12575 4176 Acked 5175 Write 7 New Done Done true
12576 4177 Acked 5176 Read 3 New Done Done true
12577 4178 Acked 5177 Write 4 New Done Done true
12578 4179 Acked 5178 Write 5 New Done Done true
12579 4180 Acked 5179 Read 1 New Done Done true
12580 4181 Acked 5180 Write 8 New Done Done true
12581 4182 Acked 5181 Write 2 New Done Done true
12582 4183 Acked 5182 Flush 0 New Done Done true
12583 4184 Acked 5183 Write 3 New Done Done true
12584 4185 Acked 5184 Write 6 New Done Done true
12585 4186 Acked 5185 Read 5 New Done Done true
12586 4187 Acked 5186 Write 7 New Done Done true
12587 4188 Acked 5187 Write 9 New Done Done true
12588 4189 Acked 5188 Read 7 New Done Done true
12589 4190 Acked 5189 Write 8 New Done Done true
12590 4191 Acked 5190 Flush 0 New Done Done true
12591 4192 Acked 5191 Write 7 New Done Done true
12592 4193 Acked 5192 Read 3 New Done Done true
12593 4194 Acked 5193 Read 4 New Done Done true
12594 4195 Acked 5194 Read 8 New Done Done true
12595 4196 Acked 5195 Read 1 New Done Done true
12596 4197 Acked 5196 Flush 0 New Done Done true
12597 4198 Acked 5197 Read 3 New Done Done true
12598 4199 Acked 5198 Write 4 New Done Done true
12599 4200 Acked 5199 Write 4 New Done Done true
12600 4201 Acked 5200 Flush 0 New Done Done true
12601 4202 Acked 5201 Write 4 New Done Done true
12602 4203 Acked 5202 Read 7 New Done Done true
12603 4204 Acked 5203 Read 8 New Done Done true
12604 4205 Acked 5204 Read 5 New Done Done true
12605 4206 Acked 5205 Read 6 New Done Done true
12606 4207 Acked 5206 Read 10 New Done Done true
12607 4208 Acked 5207 Write 7 New Done Done true
12608 4209 Acked 5208 Read 9 New Done Done true
12609 4210 Acked 5209 Read 10 New Done Done true
12610 4211 Acked 5210 Read 1 New Done Done true
12611 4212 Acked 5211 Write 1 New Done Done true
12612 4213 Acked 5212 Read 3 New Done Done true
12613 4214 Acked 5213 Read 8 New Done Done true
12614 4215 Acked 5214 Write 6 New Done Done true
12615 4216 Acked 5215 Read 10 New Done Done true
12616 4217 Acked 5216 Write 3 New Done Done true
12617 4218 Acked 5217 Read 10 New Done Done true
12618 4219 Acked 5218 Flush 0 New Done Done true
12619 STATES DS:0 DS:1 DS:2 TOTAL
12620 New 302 0 0 302
12621 Sent 0 0 0 0
12622 Done 0 302 302 604
12623 Skipped 0 0 0 0
12624 Error 0 0 0 0
12625 Last Flush: 4916 5218 5218
12626 Downstairs last five completed: 4916 4915 4914 4913 4912
12627 Upstairs last five completed: 4219 4218 4217 4216 4215
12628 CLIENT: Up:0 ds:302 act:2
126292024-02-28T22:15:43.996ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126302024-02-28T22:15:43.997ZINFOcrucible: ds_connection connected = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126312024-02-28T22:15:43.998ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126322024-02-28T22:15:43.998ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126332024-02-28T22:15:43.998ZINFOcrucible: send last flush ID to this DS: 4916 = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126342024-02-28T22:15:43.998ZINFOcrucible: Replied this last flush ID: 4916 = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126352024-02-28T22:15:43.998ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126362024-02-28T22:15:43.998ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126372024-02-28T22:15:43.998ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126382024-02-28T22:15:43.998ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
12639 ----------------------------------------------------------------
12640 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12641 Downstairs last five completed: 5218 5217 5216 5215 5214
12642 Upstairs last five completed: 4219 4218 4217 4216 4215
12643 CLIENT: Up:0 ds:0 act:3
12644 Replay: All jobs finished, all DS active.
12645 Main task has work to do, go find it
12646 got dsc Stop(2)
12647 stop 2
12648 Feb 28 22:15:47.761 INFO request completed, latency_us: 373, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 5231bc15-ead0-4348-9180-5e02e127c881, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0[8830] Got stop action so:false kr:false
12649 .1:9998
126502024-02-28T22:15:47.762ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126512024-02-28T22:15:47.762ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126522024-02-28T22:15:47.762ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126532024-02-28T22:15:47.762ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126542024-02-28T22:15:47.762ZINFOcrucible: [2] client re-new 0 jobs since flush 5218 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
12655 Feb 28 22:15:47.762{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:15:47.762801554Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
12656 INFO request completed, latency_us: 137, response_code: 200, uri: /state/cid/2, method: GET, req_id: 49b502fb-9bd1-4e21-94d6-f4f13dcb3560, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
12657 [8830] Exited with: ExitStatus(unix_wait_status(9))
12658 [8830][2] reports Exit
126592024-02-28T22:15:48.764ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126602024-02-28T22:15:48.764ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126612024-02-28T22:15:48.764ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126622024-02-28T22:15:48.764ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126632024-02-28T22:15:48.764ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126642024-02-28T22:15:48.764ZINFOcrucible: [2] client re-new 0 jobs since flush 5218 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126652024-02-28T22:15:48.764ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126662024-02-28T22:15:49.765ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126672024-02-28T22:15:49.765ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126682024-02-28T22:15:49.765ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126692024-02-28T22:15:49.765ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126702024-02-28T22:15:49.765ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126712024-02-28T22:15:49.766ZINFOcrucible: [2] client re-new 0 jobs since flush 5218 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126722024-02-28T22:15:49.766ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126732024-02-28T22:15:50.767ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126742024-02-28T22:15:50.767ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126752024-02-28T22:15:50.767ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126762024-02-28T22:15:50.768ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126772024-02-28T22:15:50.768ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126782024-02-28T22:15:50.768ZINFOcrucible: [2] client re-new 0 jobs since flush 5218 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126792024-02-28T22:15:50.768ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
12680 Feb 28 22:15:51.766 INFO request completed, latency_us: 327, response_code: 200, uri: /state/cid/2, method: GET, req_id: 3cac6dd0-b377-45a7-9217-87ef6d65e2e0, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
12681 001/300 Read block 4114 len 40960
12682 002/300 Write block 2065 len 36864 data: 3 5 5 5 5 5 5 5 5
12683 003/300 Read block 4933 len 12288
126842024-02-28T22:15:51.769ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126852024-02-28T22:15:51.769ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126862024-02-28T22:15:51.769ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126872024-02-28T22:15:51.769ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126882024-02-28T22:15:51.769ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126892024-02-28T22:15:51.769ZINFOcrucible: [2] client re-new 3 jobs since flush 5218 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
126902024-02-28T22:15:51.769ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
12691 004/300 Read block 3139 len 16384
12692 005/300 Read block 2215 len 36864
12693 006/300 Write block 3392 len 12288 data: 5 5 6
12694 007/300 Write block 3909 len 20480 data: 4 3 2 3 3
12695 008/300 Read block 4198 len 24576
12696 009/300 Write block 1106 len 24576 data: 6 5 4 4 4 4
12697 010/300 Write block 4474 len 28672 data: 2 2 3 3 3 3 3
12698 011/300 Write block 4413 len 16384 data: 4 4 3 4
12699 012/300 Read block 608 len 24576
12700 013/300 Read block 2857 len 20480
12701 014/300 Read block 3204 len 8192
12702 015/300 Write block 1163 len 32768 data: 3 5 5 5 5 5 6 6
12703 016/300 Read block 1076 len 20480
12704 017/300 Read block 3122 len 32768
12705 018/300 Read block 2185 len 24576
12706 019/300 Write block 3860 len 12288 data: 4 4 4
12707 020/300 Write block 1311 len 8192 data: 4 3
12708 021/300 Read block 2738 len 24576
12709 022/300 Write block 345 len 16384 data: 4 4 5 4
12710 023/300 Read block 2408 len 28672
12711 024/300 Read block 1634 len 36864
12712 025/300 Write block 1531 len 16384 data: 3 4 4 4
12713 026/300 Read block 1477 len 12288
12714 027/300 Read block 4318 len 28672
12715 028/300 Read block 446 len 28672
12716 029/300 Flush
12717 030/300 Read block 3405 len 36864
12718 031/300 Read block 2749 len 40960
12719 032/300 Write block 3561 len 24576 data: 3 3 3 3 2 3
12720 033/300 Read block 3111 len 4096
12721 034/300 Read block 4338 len 32768
12722 035/300 Write block 3224 len 4096 data: 4
12723 036/300 Write block 1768 len 16384 data: 3 4 5 5
12724 037/300 Write block 3734 len 4096 data: 6
12725 038/300 Read block 3659 len 28672
12726 039/300 Read block 4324 len 32768
12727 040/300 Read block 215 len 28672
12728 041/300 Flush
12729 042/300 Read block 4742 len 32768
12730 043/300 Read block 1005 len 36864
12731 044/300 Read block 4491 len 8192
12732 045/300 Write block 3569 len 20480 data: 3 3 3 5 6
12733 046/300 Write block 3951 len 8192 data: 2 4
12734 047/300 Write block 3077 len 16384 data: 4 4 4 4
12735 048/300 Write block 1583 len 28672 data: 6 5 5 6 5 5 3
12736 049/300 Read block 3026 len 20480
12737 050/300 Read block 4620 len 32768
12738 051/300 Flush
12739 052/300 Write block 823 len 28672 data: 4 4 4 3 5 3 3
12740 053/300 Write block 1561 len 8192 data: 7 6
12741 054/300 Write block 98 len 28672 data: 4 3 3 4 3 4 4
12742 055/300 Flush
12743 056/300 Read block 1605 len 28672
12744 057/300 Read block 4973 len 16384
12745 058/300 Read block 4186 len 8192
12746 059/300 Read block 2068 len 8192
12747 060/300 Write block 4277 len 8192 data: 3 3
12748 061/300 Read block 575 len 16384
12749 062/300 Read block 2624 len 20480
12750 063/300 Read block 4053 len 28672
12751 064/300 Read block 4129 len 16384
12752 065/300 Write block 3256 len 12288 data: 3 2 2
12753 066/300 Write block 1513 len 4096 data: 3
12754 067/300 Read block 4716 len 8192
12755 068/300 Write block 63 len 32768 data: 5 4 3 4 3 4 4 3
12756 069/300 Read block 2907 len 24576
12757 070/300 Write block 2241 len 16384 data: 5 5 5 6
12758 071/300 Write block 2261 len 36864 data: 3 3 3 3 3 3 4 3 2
12759 072/300 Read block 1250 len 16384
12760 073/300 Read block 2468 len 12288
12761 074/300 Read block 3477 len 20480
12762 075/300 Write block 2881 len 8192 data: 3 4
12763 076/300 Read block 2310 len 16384
12764 077/300 Read block 4793 len 16384
12765 078/300 Write block 4925 len 32768 data: 4 4 4 5 5 4 4 3
12766 079/300 Write block 774 len 20480 data: 4 3 4 3 4
12767 080/300 Write block 3381 len 16384 data: 4 4 5 3
12768 081/300 Read block 1100 len 4096
12769 082/300 Write block 1780 len 20480 data: 4 4 4 5 4
12770 083/300 Read block 1243 len 16384
12771 084/300 Read block 2369 len 32768
12772 085/300 Read block 1632 len 40960
12773 086/300 Read block 870 len 8192
12774 087/300 Write block 2768 len 40960 data: 3 2 2 2 3 3 3 3 3 3
12775 088/300 Read block 1443 len 16384
12776 089/300 Write block 1038 len 36864 data: 5 4 6 6 5 3 4 4 4
12777 090/300 Flush
12778 091/300 Flush
12779 092/300 Read block 1532 len 4096
12780 093/300 Read block 4696 len 24576
12781 094/300 Read block 2362 len 12288
12782 095/300 Read block 2506 len 36864
12783 096/300 Write block 3159 len 20480 data: 3 3 4 4 6
12784 097/300 Read block 465 len 36864
12785 098/300 Write block 4140 len 12288 data: 5 5 5
12786 099/300 Flush
12787 100/300 Write block 907 len 12288 data: 2 3 4
12788 101/300 Flush
12789 102/300 Read block 4841 len 32768
12790 103/300 Write block 554 len 8192 data: 2 2
12791 104/300 Write block 1669 len 16384 data: 2 3 2 2
12792 105/300 Read block 268 len 8192
12793 106/300 Write block 3364 len 20480 data: 3 3 4 3 3
12794 107/300 Write block 173 len 16384 data: 4 4 3 3
12795 108/300 Write block 3261 len 28672 data: 2 2 2 2 3 4 4
12796 109/300 Flush
12797 110/300 Read block 3948 len 24576
12798 111/300 Read block 2069 len 28672
12799 112/300 Read block 447 len 36864
12800 113/300 Flush
12801 114/300 Write block 3789 len 40960 data: 3 3 4 3 3 3 3 3 2 3
12802 115/300 Write block 2816 len 20480 data: 6 6 6 6 7
12803 116/300 Flush
12804 117/300 Write block 3452 len 24576 data: 4 2 2 2 2 2
12805 118/300 Read block 3158 len 24576
12806 119/300 Write block 4538 len 16384 data: 3 3 3 3
12807 120/300 Read block 565 len 20480
12808 121/300 Read block 967 len 24576
12809 122/300 Write block 4812 len 40960 data: 4 3 3 3 2 2 2 3 3 3
12810 123/300 Write block 3895 len 12288 data: 5 3 3
12811 124/300 Read block 1726 len 8192
12812 125/300 Write block 757 len 32768 data: 7 6 7 5 5 6 6 4
12813 126/300 Flush
12814 127/300 Write block 1851 len 4096 data: 3
12815 128/300 Read block 935 len 8192
12816 129/300 Read block 3104 len 36864
12817 130/300 Read block 4437 len 40960
12818 131/300 Write block 935 len 24576 data: 4 3 3 3 4 3
12819 132/300 Read block 1113 len 20480
12820 133/300 Read block 3037 len 16384
12821 134/300 Write block 2862 len 24576 data: 4 5 4 4 3 3
12822 135/300 Flush
12823 136/300 Read block 3727 len 12288
12824 137/300 Read block 1384 len 12288
12825 138/300 Write block 2406 len 36864 data: 3 3 3 4 4 4 6 7 7
12826 139/300 Flush
12827 140/300 Write block 1840 len 36864 data: 3 3 3 3 3 4 4 3 3
12828 141/300 Write block 1676 len 20480 data: 4 4 5 3 3
12829 142/300 Write block 1043 len 20480 data: 4 5 5 5 4
12830 143/300 Write block 341 len 28672 data: 3 4 4 4 5 5 6
12831 144/300 Read block 3324 len 40960
12832 145/300 Read block 3371 len 32768
12833 146/300 Flush
12834 147/300 Flush
12835 148/300 Read block 1517 len 24576
12836 149/300 Flush
12837 150/300 Write block 3112 len 20480 data: 6 5 4 3 3
12838 151/300 Read block 3096 len 28672
12839 152/300 Read block 1943 len 40960
12840 153/300 Read block 1016 len 20480
12841 154/300 Read block 277 len 40960
12842 155/300 Read block 3423 len 32768
12843 156/300 Write block 1899 len 12288 data: 4 4 4
12844 157/300 Read block 2280 len 12288
12845 158/300 Read block 4045 len 32768
12846 159/300 Write block 4685 len 8192 data: 3 3
12847 160/300 Write block 1403 len 12288 data: 4 5 5
12848 161/300 Read block 3316 len 36864
12849 162/300 Write block 1771 len 20480 data: 6 4 4 4 4
12850 163/300 Read block 2270 len 12288
12851 164/300 Write block 2588 len 16384 data: 2 2 2 2
12852 165/300 Flush
12853 166/300 Write block 3399 len 40960 data: 6 4 4 4 4 4 4 4 6 6
12854 167/300 Write block 4553 len 40960 data: 3 3 4 4 4 3 3 3 3 3
12855 168/300 Read block 728 len 20480
12856 169/300 Read block 221 len 12288
12857 170/300 Read block 2386 len 20480
12858 171/300 Read block 4638 len 32768
12859 172/300 Write block 4179 len 36864 data: 3 4 5 5 5 4 4 5 4
12860 173/300 Read block 1496 len 24576
12861 174/300 Read block 4670 len 16384
12862 175/300 Read block 2755 len 24576
12863 176/300 Read block 3258 len 16384
12864 177/300 Write block 193 len 8192 data: 2 2
12865 178/300 Write block 3363 len 32768 data: 4 4 4 5 4 4 3 4
12866 179/300 Read block 1243 len 32768
12867 180/300 Flush
12868 181/300 Read block 2130 len 40960
12869 182/300 Write block 4734 len 16384 data: 3 3 3 3
12870 183/300 Read block 3005 len 4096
12871 184/300 Read block 3595 len 20480
12872 185/300 Read block 177 len 20480
12873 186/300 Read block 3354 len 32768
12874 187/300 Write block 159 len 20480 data: 4 5 4 3 3
12875 188/300 Write block 1865 len 24576 data: 5 5 4 4 3 3
12876 189/300 Flush
12877 190/300 Write block 4539 len 28672 data: 4 4 4 3 3 3 3
12878 191/300 Read block 1471 len 28672
12879 192/300 Write block 110 len 28672 data: 3 2 3 3 4 4 4
12880 193/300 Read block 4098 len 16384
12881 194/300 Flush
12882 195/300 Read block 4959 len 12288
12883 196/300 Read block 2688 len 12288
12884 197/300 Write block 2571 len 4096 data: 2
12885 198/300 Read block 2534 len 24576
12886 199/300 Read block 1297 len 40960
12887 200/300 Read block 3745 len 24576
12888 201/300 Read block 4423 len 16384
12889 202/300 Read block 3034 len 8192
12890 203/300 Read block 1559 len 36864
12891 204/300 Write block 1324 len 36864 data: 5 5 5 3 3 4 3 4 5
12892 205/300 Write block 2348 len 24576 data: 3 2 2 2 2 2
12893 206/300 Read block 2264 len 24576
12894 207/300 Write block 4423 len 24576 data: 3 3 3 4 4 3
12895 208/300 Read block 2073 len 28672
12896 209/300 Write block 3291 len 32768 data: 6 5 5 5 4 4 3 2
12897 210/300 Write block 2160 len 40960 data: 3 2 2 2 2 2 2 3 3 3
12898 211/300 Read block 4551 len 16384
12899 212/300 Read block 3308 len 32768
12900 213/300 Read block 2251 len 20480
12901 214/300 Read block 4113 len 12288
12902 215/300 Write block 3058 len 32768 data: 5 4 6 6 8 8 4 4
12903 216/300 Write block 1259 len 40960 data: 4 4 3 3 2 2 2 3 4 4
12904 217/300 Read block 3826 len 32768
12905 218/300 Write block 2778 len 28672 data: 3 4 4 3 4 5 5
12906 219/300 Read block 3596 len 32768
12907 220/300 Read block 2905 len 8192
12908 221/300 Read block 3501 len 4096
12909 222/300 Write block 1833 len 20480 data: 4 4 4 5 3
12910 223/300 Read block 3784 len 40960
12911 224/300 Read block 1817 len 8192
12912 225/300 Write block 3971 len 24576 data: 5 4 4 3 3 3
12913 226/300 Write block 1393 len 20480 data: 6 5 4 4 3
12914 227/300 Read block 4447 len 36864
12915 228/300 Write block 4789 len 16384 data: 7 5 4 5
12916 229/300 Read block 2784 len 8192
12917 230/300 Read block 4104 len 40960
12918 231/300 Flush
12919 232/300 Write block 1625 len 4096 data: 8
12920 233/300 Write block 1882 len 24576 data: 4 4 5 6 6 4
12921 234/300 Read block 2306 len 40960
12922 235/300 Flush
12923 236/300 Flush
12924 237/300 Write block 1083 len 24576 data: 5 4 5 4 4 4
12925 238/300 Write block 3581 len 40960 data: 3 3 3 3 4 4 4 4 5 5
12926 239/300 Read block 3912 len 8192
12927 240/300 Read block 2701 len 16384
12928 241/300 Read block 3246 len 36864
12929 242/300 Write block 2205 len 8192 data: 4 4
12930 243/300 Read block 2060 len 20480
12931 244/300 Read block 3093 len 16384
12932 245/300 Flush
12933 246/300 Write block 4683 len 32768 data: 3 3 4 4 3 3 2 2
12934 247/300 Write block 2191 len 28672 data: 7 7 5 6 4 2 2
12935 248/300 Read block 654 len 20480
12936 249/300 Read block 82 len 24576
12937 250/300 Read block 3950 len 36864
12938 251/300 Read block 1922 len 28672
12939 252/300 Write block 1502 len 28672 data: 2 2 2 2 2 2 2
12940 253/300 Write block 2891 len 40960 data: 8 7 5 4 3 3 3 3 3 4
12941 254/300 Flush
12942 255/300 Write block 1529 len 24576 data: 3 3 4 5 5 5
12943 256/300 Write block 234 len 24576 data: 4 3 2 2 2 2
12944 257/300 Write block 1612 len 24576 data: 4 5 5 7 6 3
12945 258/300 Write block 2516 len 16384 data: 5 5 5 3
12946 259/300 Write block 3943 len 40960 data: 4 4 3 3 3 2 2 2 3 5
12947 260/300 Read block 2461 len 32768
12948 261/300 Write block 1583 len 12288 data: 7 6 6
12949 262/300 Flush
12950 263/300 Read block 3730 len 12288
12951 264/300 Read block 2089 len 36864
12952 265/300 Read block 439 len 40960
12953 266/300 Read block 2948 len 12288
12954 267/300 Read block 2012 len 4096
12955 268/300 Read block 1997 len 28672
12956 269/300 Read block 83 len 4096
12957 270/300 Flush
12958 271/300 Write block 4160 len 20480 data: 4 3 3 3 3
12959 272/300 Read block 3821 len 20480
12960 273/300 Write block 77 len 24576 data: 2 2 2 2 3 3
12961 274/300 Write block 3021 len 40960 data: 4 3 3 3 4 2 2 2 3 3
12962 275/300 Read block 498 len 28672
12963 276/300 Write block 2366 len 4096 data: 3
12964 277/300 Read block 2276 len 36864
12965 278/300 Read block 812 len 36864
12966 279/300 Write block 4977 len 36864 data: 5 4 4 4 4 3 6 6 6
12967 280/300 Read block 4562 len 4096
12968 281/300 Write block 1372 len 8192 data: 3 3
12969 282/300 Write block 2382 len 8192 data: 3 3
12970 283/300 Read block 4940 len 16384
12971 284/300 Write block 3727 len 16384 data: 7 7 8 7
12972 285/300 Read block 3172 len 40960
12973 286/300 Flush
12974 287/300 Write block 897 len 8192 data: 3 3
12975 288/300 Read block 1941 len 36864
12976 289/300 Write block 182 len 4096 data: 3
12977 290/300 Read block 2107 len 40960
12978 291/300 Read block 3542 len 36864
12979 292/300 Read block 2775 len 36864
12980 293/300 Read block 856 len 28672
12981 294/300 Read block 716 len 16384
12982 295/300 Read block 2094 len 20480
12983 296/300 Read block 60 len 28672
12984 297/300 Write block 4483 len 8192 data: 3 3
12985 298/300 Flush
12986 299/300 Write block 3834 len 36864 data: 3 3 3 4 3 3 2 2 4
12987 300/300 Write block 186 len 4096 data: 5
12988 Main task has work to do, go find it
12989 got dsc Start(2)
12990 start 2
12991 Feb 28 22:15:52.499 INFO request completed, latency_us: 220, response_code: 204, uri: /start/cid/2, method: POST, req_id: df3873f2-b190-4857-990c-234a9336dd10, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
12992 Replay: started 2, returned:Ok(())
12993 ----------------------------------------------------------------
12994 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
12995 States: Active Active Offline
12996 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
12997 4220 Acked 5219 Read 10 Done Done New true
12998 4221 Acked 5220 Write 9 Done Done New true
12999 4222 Acked 5221 Read 3 Done Done New true
13000 4223 Acked 5222 Read 4 Done Done New false
13001 4224 Acked 5223 Read 9 Done Done New false
13002 4225 Acked 5224 Write 3 Done Done New false
13003 4226 Acked 5225 Write 5 Done Done New false
13004 4227 Acked 5226 Read 6 Done Done New false
13005 4228 Acked 5227 Write 6 Done Done New false
13006 4229 Acked 5228 Write 7 Done Done New false
13007 4230 Acked 5229 Write 4 Done Done New false
13008 4231 Acked 5230 Read 6 Done Done New false
13009 4232 Acked 5231 Read 5 Done Done New false
13010 4233 Acked 5232 Read 2 Done Done New false
13011 4234 Acked 5233 Write 8 Done Done New false
13012 4235 Acked 5234 Read 5 Done Done New false
13013 4236 Acked 5235 Read 8 Done Done New false
13014 4237 Acked 5236 Read 6 Done Done New false
13015 4238 Acked 5237 Write 3 Done Done New false
13016 4239 Acked 5238 Write 2 Done Done New false
13017 4240 Acked 5239 Read 6 Done Done New false
13018 4241 Acked 5240 Write 4 Done Done New false
13019 4242 Acked 5241 Read 7 Done Done New false
13020 4243 Acked 5242 Read 9 Done Done New false
13021 4244 Acked 5243 Write 4 Done Done New false
13022 4245 Acked 5244 Read 3 Done Done New false
13023 4246 Acked 5245 Read 7 Done Done New false
13024 4247 Acked 5246 Read 7 Done Done New false
13025 4248 Acked 5247 Flush 0 Done Done New false
13026 4249 Acked 5248 Read 9 Done Done New false
13027 4250 Acked 5249 Read 10 Done Done New false
13028 4251 Acked 5250 Write 6 Done Done New false
13029 4252 Acked 5251 Read 1 Done Done New false
13030 4253 Acked 5252 Read 8 Done Done New false
13031 4254 Acked 5253 Write 1 Done Done New false
13032 4255 Acked 5254 Write 4 Done Done New false
13033 4256 Acked 5255 Write 1 Done Done New false
13034 4257 Acked 5256 Read 7 Done Done New false
13035 4258 Acked 5257 Read 8 Done Done New false
13036 4259 Acked 5258 Read 7 Done Done New false
13037 4260 Acked 5259 Flush 0 Done Done New false
13038 4261 Acked 5260 Read 8 Done Done New false
13039 4262 Acked 5261 Read 9 Done Done New false
13040 4263 Acked 5262 Read 2 Done Done New false
13041 4264 Acked 5263 Write 5 Done Done New false
13042 4265 Acked 5264 Write 2 Done Done New false
13043 4266 Acked 5265 Write 4 Done Done New false
13044 4267 Acked 5266 Write 7 Done Done New false
13045 4268 Acked 5267 Read 5 Done Done New false
13046 4269 Acked 5268 Read 8 Done Done New false
13047 4270 Acked 5269 Flush 0 Done Done New false
13048 4271 Acked 5270 Write 7 Done Done New false
13049 4272 Acked 5271 Write 2 Done Done New false
13050 4273 Acked 5272 Write 7 Done Done New false
13051 4274 Acked 5273 Flush 0 Done Done New false
13052 4275 Acked 5274 Read 7 Done Done New false
13053 4276 Acked 5275 Read 4 Done Done New false
13054 4277 Acked 5276 Read 2 Done Done New false
13055 4278 Acked 5277 Read 2 Done Done New false
13056 4279 Acked 5278 Write 2 Done Done New false
13057 4280 Acked 5279 Read 4 Done Done New false
13058 4281 Acked 5280 Read 5 Done Done New false
13059 4282 Acked 5281 Read 7 Done Done New false
13060 4283 Acked 5282 Read 4 Done Done New false
13061 4284 Acked 5283 Write 3 Done Done New false
13062 4285 Acked 5284 Write 1 Done Done New false
13063 4286 Acked 5285 Read 2 Done Done New false
13064 4287 Acked 5286 Write 8 Done Done New false
13065 4288 Acked 5287 Read 6 Done Done New false
13066 4289 Acked 5288 Write 4 Done Done New false
13067 4290 Acked 5289 Write 9 Done Done New false
13068 4291 Acked 5290 Read 4 Done Done New false
13069 4292 Acked 5291 Read 3 Done Done New false
13070 4293 Acked 5292 Read 5 Done Done New false
13071 4294 Acked 5293 Write 2 Done Done New false
13072 4295 Acked 5294 Read 4 Done Done New false
13073 4296 Acked 5295 Read 4 Done Done New false
13074 4297 Acked 5296 Write 8 Done Done New false
13075 4298 Acked 5297 Write 5 Done Done New false
13076 4299 Acked 5298 Write 4 Done Done New false
13077 4300 Acked 5299 Read 1 Done Done New false
13078 4301 Acked 5300 Write 5 Done Done New false
13079 4302 Acked 5301 Read 4 Done Done New false
13080 4303 Acked 5302 Read 8 Done Done New false
13081 4304 Acked 5303 Read 10 Done Done New false
13082 4305 Acked 5304 Read 2 Done Done New false
13083 4306 Acked 5305 Write 10 Done Done New false
13084 4307 Acked 5306 Read 4 Done Done New false
13085 4308 Acked 5307 Write 9 Done Done New false
13086 4309 Acked 5308 Flush 0 Done Done New false
13087 4310 Acked 5309 Flush 0 Done Done New false
13088 4311 Acked 5310 Read 1 Done Done New false
13089 4312 Acked 5311 Read 6 Done Done New false
13090 4313 Acked 5312 Read 3 Done Done New false
13091 4314 Acked 5313 Read 9 Done Done New false
13092 4315 Acked 5314 Write 5 Done Done New false
13093 4316 Acked 5315 Read 9 Done Done New false
13094 4317 Acked 5316 Write 3 Done Done New false
13095 4318 Acked 5317 Flush 0 Done Done New false
13096 4319 Acked 5318 Write 3 Done Done New false
13097 4320 Acked 5319 Flush 0 Done Done New false
13098 4321 Acked 5320 Read 8 Done Done New false
13099 4322 Acked 5321 Write 2 Done Done New false
13100 4323 Acked 5322 Write 4 Done Done New false
13101 4324 Acked 5323 Read 2 Done Done New false
13102 4325 Acked 5324 Write 5 Done Done New false
13103 4326 Acked 5325 Write 4 Done Done New false
13104 4327 Acked 5326 Write 7 Done Done New false
13105 4328 Acked 5327 Flush 0 Done Done New false
13106 4329 Acked 5328 Read 6 Done Done New false
13107 4330 Acked 5329 Read 7 Done Done New false
13108 4331 Acked 5330 Read 9 Done Done New false
13109 4332 Acked 5331 Flush 0 Done Done New false
13110 4333 Acked 5332 Write 10 Done Done New false
13111 4334 Acked 5333 Write 5 Done Done New false
13112 4335 Acked 5334 Flush 0 Done Done New false
13113 4336 Acked 5335 Write 6 Done Done New false
13114 4337 Acked 5336 Read 6 Done Done New false
13115 4338 Acked 5337 Write 4 Done Done New false
13116 4339 Acked 5338 Read 5 Done Done New false
13117 4340 Acked 5339 Read 6 Done Done New false
13118 4341 Acked 5340 Write 10 Done Done New false
13119 4342 Acked 5341 Write 3 Done Done New false
13120 4343 Acked 5342 Read 2 Done Done New false
13121 4344 Acked 5343 Write 8 Done Done New false
13122 4345 Acked 5344 Flush 0 Done Done New false
13123 4346 Acked 5345 Write 1 Done Done New false
13124 4347 Acked 5346 Read 2 Done Done New false
13125 4348 Acked 5347 Read 9 Done Done New false
13126 4349 Acked 5348 Read 10 Done Done New false
13127 4350 Acked 5349 Write 6 Done Done New false
13128 4351 Acked 5350 Read 5 Done Done New false
13129 4352 Acked 5351 Read 4 Done Done New false
13130 4353 Acked 5352 Write 6 Done Done New false
13131 4354 Acked 5353 Flush 0 Done Done New false
13132 4355 Acked 5354 Read 3 Done Done New false
13133 4356 Acked 5355 Read 3 Done Done New false
13134 4357 Acked 5356 Write 9 Done Done New false
13135 4358 Acked 5357 Flush 0 Done Done New false
13136 4359 Acked 5358 Write 9 Done Done New false
13137 4360 Acked 5359 Write 5 Done Done New false
13138 4361 Acked 5360 Write 5 Done Done New false
13139 4362 Acked 5361 Write 7 Done Done New false
13140 4363 Acked 5362 Read 10 Done Done New false
13141 4364 Acked 5363 Read 8 Done Done New false
13142 4365 Acked 5364 Flush 0 Done Done New false
13143 4366 Acked 5365 Flush 0 Done Done New false
13144 4367 Acked 5366 Read 6 Done Done New false
13145 4368 Acked 5367 Flush 0 Done Done New false
13146 4369 Acked 5368 Write 5 Done Done New false
13147 4370 Acked 5369 Read 7 Done Done New false
13148 4371 Acked 5370 Read 10 Done Done New false
13149 4372 Acked 5371 Read 5 Done Done New false
13150 4373 Acked 5372 Read 10 Done Done New false
13151 4374 Acked 5373 Read 8 Done Done New false
13152 4375 Acked 5374 Write 3 Done Done New false
13153 4376 Acked 5375 Read 3 Done Done New false
13154 4377 Acked 5376 Read 8 Done Done New false
13155 4378 Acked 5377 Write 2 Done Done New false
13156 4379 Acked 5378 Write 3 Done Done New false
13157 4380 Acked 5379 Read 9 Done Done New false
13158 4381 Acked 5380 Write 5 Done Done New false
13159 4382 Acked 5381 Read 3 Done Done New false
13160 4383 Acked 5382 Write 4 Done Done New false
13161 4384 Acked 5383 Flush 0 Done Done New false
13162 4385 Acked 5384 Write 10 Done Done New false
13163 4386 Acked 5385 Write 10 Done Done New false
13164 4387 Acked 5386 Read 5 Done Done New false
13165 4388 Acked 5387 Read 3 Done Done New false
13166 4389 Acked 5388 Read 5 Done Done New false
13167 4390 Acked 5389 Read 8 Done Done New false
13168 4391 Acked 5390 Write 9 Done Done New false
13169 4392 Acked 5391 Read 6 Done Done New false
13170 4393 Acked 5392 Read 4 Done Done New false
13171 4394 Acked 5393 Read 6 Done Done New false
13172 4395 Acked 5394 Read 4 Done Done New false
13173 4396 Acked 5395 Write 2 Done Done New false
13174 4397 Acked 5396 Write 8 Done Done New false
13175 4398 Acked 5397 Read 8 Done Done New false
13176 4399 Acked 5398 Flush 0 Done Done New false
13177 4400 Acked 5399 Read 10 Done Done New false
13178 4401 Acked 5400 Write 4 Done Done New false
13179 4402 Acked 5401 Read 1 Done Done New false
13180 4403 Acked 5402 Read 5 Done Done New false
13181 4404 Acked 5403 Read 5 Done Done New false
13182 4405 Acked 5404 Read 8 Done Done New false
13183 4406 Acked 5405 Write 5 Done Done New false
13184 4407 Acked 5406 Write 6 Done Done New false
13185 4408 Acked 5407 Flush 0 Done Done New false
13186 4409 Acked 5408 Write 7 Done Done New false
13187 4410 Acked 5409 Read 7 Done Done New false
13188 4411 Acked 5410 Write 7 Done Done New false
13189 4412 Acked 5411 Read 4 Done Done New false
13190 4413 Acked 5412 Flush 0 Done Done New false
13191 4414 Acked 5413 Read 3 Done Done New false
13192 4415 Acked 5414 Read 3 Done Done New false
13193 4416 Acked 5415 Write 1 Done Done New false
13194 4417 Acked 5416 Read 6 Done Done New false
13195 4418 Acked 5417 Read 10 Done Done New false
13196 4419 Acked 5418 Read 6 Done Done New false
13197 4420 Acked 5419 Read 4 Done Done New false
13198 4421 Acked 5420 Read 2 Done Done New false
13199 4422 Acked 5421 Read 9 Done Done New false
13200 4423 Acked 5422 Write 9 Done Done New false
13201 4424 Acked 5423 Write 6 Done Done New false
13202 4425 Acked 5424 Read 6 Done Done New false
13203 4426 Acked 5425 Write 6 Done Done New false
13204 4427 Acked 5426 Read 7 Done Done New false
13205 4428 Acked 5427 Write 8 Done Done New false
13206 4429 Acked 5428 Write 10 Done Done New false
13207 4430 Acked 5429 Read 4 Done Done New false
13208 4431 Acked 5430 Read 8 Done Done New false
13209 4432 Acked 5431 Read 5 Done Done New false
13210 4433 Acked 5432 Read 3 Done Done New false
13211 4434 Acked 5433 Write 8 Done Done New false
13212 4435 Acked 5434 Write 10 Done Done New false
13213 4436 Acked 5435 Read 8 Done Done New false
13214 4437 Acked 5436 Write 7 Done Done New false
13215 4438 Acked 5437 Read 8 Done Done New false
13216 4439 Acked 5438 Read 2 Done Done New false
13217 4440 Acked 5439 Read 1 Done Done New false
13218 4441 Acked 5440 Write 5 Done Done New false
13219 4442 Acked 5441 Read 10 Done Done New false
13220 4443 Acked 5442 Read 2 Done Done New false
13221 4444 Acked 5443 Write 6 Done Done New false
13222 4445 Acked 5444 Write 5 Done Done New false
13223 4446 Acked 5445 Read 9 Done Done New false
13224 4447 Acked 5446 Write 4 Done Done New false
13225 4448 Acked 5447 Read 2 Done Done New false
13226 4449 Acked 5448 Read 10 Done Done New false
13227 4450 Acked 5449 Flush 0 Done Done New false
13228 4451 Acked 5450 Write 1 Done Done New false
13229 4452 Acked 5451 Write 6 Done Done New false
13230 4453 Acked 5452 Read 10 Done Done New false
13231 4454 Acked 5453 Flush 0 Done Done New false
13232 4455 Acked 5454 Flush 0 Done Done New false
13233 4456 Acked 5455 Write 6 Done Done New false
13234 4457 Acked 5456 Write 10 Done Done New false
13235 4458 Acked 5457 Read 2 Done Done New false
13236 4459 Acked 5458 Read 4 Done Done New false
13237 4460 Acked 5459 Read 9 Done Done New false
13238 4461 Acked 5460 Write 2 Done Done New false
13239 4462 Acked 5461 Read 5 Done Done New false
13240 4463 Acked 5462 Read 4 Done Done New false
13241 4464 Acked 5463 Flush 0 Done Done New false
13242 4465 Acked 5464 Write 8 Done Done New false
13243 4466 Acked 5465 Write 7 Done Done New false
13244 4467 Acked 5466 Read 5 Done Done New false
13245 4468 Acked 5467 Read 6 Done Done New false
13246 4469 Acked 5468 Read 9 Done Done New false
13247 4470 Acked 5469 Read 7 Done Done New false
13248 4471 Acked 5470 Write 7 Done Done New false
13249 4472 Acked 5471 Write 10 Done Done New false
13250 4473 Acked 5472 Flush 0 Done Done New false
13251 4474 Acked 5473 Write 6 Done Done New false
13252 4475 Acked 5474 Write 6 Done Done New false
13253 4476 Acked 5475 Write 6 Done Done New false
13254 4477 Acked 5476 Write 4 Done Done New false
13255 4478 Acked 5477 Write 10 Done Done New false
13256 4479 Acked 5478 Read 8 Done Done New false
13257 4480 Acked 5479 Write 3 Done Done New false
13258 4481 Acked 5480 Flush 0 Done Done New false
13259 4482 Acked 5481 Read 3 Done Done New false
13260 4483 Acked 5482 Read 9 Done Done New false
13261 4484 Acked 5483 Read 10 Done Done New false
13262 4485 Acked 5484 Read 3 Done Done New false
13263 4486 Acked 5485 Read 1 Done Done New false
13264 4487 Acked 5486 Read 7 Done Done New false
13265 4488 Acked 5487 Read 1 Done Done New false
13266 4489 Acked 5488 Flush 0 Done Done New false
13267 4490 Acked 5489 Write 5 Done Done New false
13268 4491 Acked 5490 Read 5 Done Done New false
13269 4492 Acked 5491 Write 6 Done Done New false
13270 4493 Acked 5492 Write 10 Done Done New false
13271 4494 Acked 5493 Read 7 Done Done New false
13272 4495 Acked 5494 Write 1 Done Done New false
13273 4496 Acked 5495 Read 9 Done Done New false
13274 4497 Acked 5496 Read 9 Done Done New false
13275 4498 Acked 5497 Write 9 Done Done New false
13276 4499 Acked 5498 Read 1 Done Done New false
13277 4500 Acked 5499 Write 2 Done Done New false
13278 4501 Acked 5500 Write 2 Done Done New false
13279 4502 Acked 5501 Read 4 Done Done New false
13280 4503 Acked 5502 Write 4 Done Done New false
13281 4504 Acked 5503 Read 10 Done Done New false
13282 4505 Acked 5504 Flush 0 Done Done New false
13283 4506 Acked 5505 Write 2 Done Done New false
13284 4507 Acked 5506 Read 9 Done Done New false
13285 4508 Acked 5507 Write 1 Done Done New false
13286 4509 Acked 5508 Read 10 Done Done New false
13287 4510 Acked 5509 Read 9 Done Done New false
13288 4511 Acked 5510 Read 9 Done Done New false
13289 4512 Acked 5511 Read 7 Done Done New false
13290 4513 Acked 5512 Read 4 Done Done New false
13291 4514 Acked 5513 Read 5 Done Done New false
13292 4515 Acked 5514 Read 7 Done Done New false
13293 4516 Acked 5515 Write 2 Done Done New false
13294 4517 Acked 5516 Flush 0 Done Done New false
13295 4518 Acked 5517 Write 9 Done Done New false
13296 4519 Acked 5518 Write 1 Done Done New false
13297 STATES DS:0 DS:1 DS:2 TOTAL
13298 New 0 0 300 300
13299 Sent 0 0 0 0
13300 Done 300 300 0 600
13301 Skipped 0 0 0 0
13302 Error 0 0 0 0
13303 Last Flush: 5516 5516 5218
13304 Downstairs last five completed: 5218 5217 5216 5215 5214
13305 Upstairs last five completed: 4519 4518 4517 4516 4515
13306 CLIENT: Up:0 ds:300 act:2
133072024-02-28T22:15:52.770ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133082024-02-28T22:15:52.771ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133092024-02-28T22:15:52.771ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133102024-02-28T22:15:52.771ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133112024-02-28T22:15:52.771ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133122024-02-28T22:15:52.771ZINFOcrucible: [2] client re-new 301 jobs since flush 5218 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133132024-02-28T22:15:52.771ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
13314 [8830] got start action
13315 [8830] I am going to restart
13316 Starting downstairs at port 8830
13317 Make output file at "/tmp/dsc/downstairs-8830.txt"
13318 [8830][2] reports Starting
133192024-02-28T22:15:53.772ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133202024-02-28T22:15:53.772ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133212024-02-28T22:15:53.772ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133222024-02-28T22:15:53.772ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133232024-02-28T22:15:53.772ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133242024-02-28T22:15:53.772ZINFOcrucible: [2] client re-new 301 jobs since flush 5218 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133252024-02-28T22:15:53.772ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
13326 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1179)
13327 [8830][2] reports Running
133282024-02-28T22:15:54.773ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133292024-02-28T22:15:54.774ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133302024-02-28T22:15:54.775ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133312024-02-28T22:15:54.775ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133322024-02-28T22:15:54.775ZINFOcrucible: send last flush ID to this DS: 5218 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133332024-02-28T22:15:54.775ZINFOcrucible: Replied this last flush ID: 5218 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133342024-02-28T22:15:54.776ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133352024-02-28T22:15:54.776ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133362024-02-28T22:15:54.776ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133372024-02-28T22:15:54.776ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
13338 ----------------------------------------------------------------
13339 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
13340 Downstairs last five completed: 5519 5518 5517 5516 5515
13341 Upstairs last five completed: 4520 4519 4518 4517 4516
13342 CLIENT: Up:0 ds:0 act:3
13343 Replay: All jobs finished, all DS active.
13344 Main task has work to do, go find it
13345 got dsc Stop(1)
13346 stop 1
13347 Feb 28 22:15:56.502 INFO request completed, latency_us: 349, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 8e771bb9-0da8-4855-873f-d8d81080374a, remote_addr: 127.0.[8820] Got stop action so:false kr:false
13348 0.1:36617, local_addr: 127.0.0.1:9998
133492024-02-28T22:15:56.503ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133502024-02-28T22:15:56.503ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133512024-02-28T22:15:56.503ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133522024-02-28T22:15:56.503ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
13353 Feb 28 22:15:56.503 INFO request completed, latency_us: 169, response_code: 200, uri: /state/cid/1, method: GET, req_id: ee4d1ba9-7d06-4720-91dc-a22e3a2375a4, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
133542024-02-28T22:15:56.503ZINFOcrucible: [1] client re-new 0 jobs since flush 5519 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133552024-02-28T22:15:56.503ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
13356 [8820] Exited with: ExitStatus(unix_wait_status(9))
13357 [8820][1] reports Exit
133582024-02-28T22:15:57.505ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133592024-02-28T22:15:57.505ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133602024-02-28T22:15:57.505ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133612024-02-28T22:15:57.505ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133622024-02-28T22:15:57.505ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133632024-02-28T22:15:57.505ZINFOcrucible: [1] client re-new 0 jobs since flush 5519 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133642024-02-28T22:15:57.505ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133652024-02-28T22:15:58.506ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133662024-02-28T22:15:58.506ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133672024-02-28T22:15:58.506ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133682024-02-28T22:15:58.506ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133692024-02-28T22:15:58.506ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133702024-02-28T22:15:58.506ZINFOcrucible: [1] client re-new 0 jobs since flush 5519 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133712024-02-28T22:15:58.506ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133722024-02-28T22:15:59.508ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133732024-02-28T22:15:59.508ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133742024-02-28T22:15:59.508ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133752024-02-28T22:15:59.508ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133762024-02-28T22:15:59.508ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133772024-02-28T22:15:59.508ZINFOcrucible: [1] client re-new 0 jobs since flush 5519 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133782024-02-28T22:15:59.508ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
13379 Feb 28 22:16:00.506 INFO request completed, latency_us: 211, response_code: 200, uri: /state/cid/1, method: GET, req_id: 6fe87c56-0c3a-4ae6-99cc-81a5a9591e83, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
13380 001/300 Write block 1252 len 36864 data: 4 4 4 4 4 3 3 5 5
13381 002/300 Read block 598 len 20480
133822024-02-28T22:16:00.509ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133832024-02-28T22:16:00.509ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133842024-02-28T22:16:00.509ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133852024-02-28T22:16:00.509ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133862024-02-28T22:16:00.509ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133872024-02-28T22:16:00.509ZINFOcrucible: [1] client re-new 2 jobs since flush 5519 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
133882024-02-28T22:16:00.509ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
13389 003/300 Read block 1528 len 36864
13390 004/300 Write block 857 len 16384 data: 4 2 2 3
13391 005/300 Write block 1636 len 36864 data: 4 5 5 5 5 5 5 5 5
13392 006/300 Read block 2585 len 12288
13393 007/300 Write block 4051 len 12288 data: 4 4 3
13394 008/300 Read block 3042 len 36864
13395 009/300 Read block 3987 len 8192
13396 010/300 Write block 1439 len 8192 data: 4 3
13397 011/300 Read block 4850 len 16384
13398 012/300 Write block 2469 len 8192 data: 4 4
13399 013/300 Write block 3013 len 40960 data: 4 5 4 5 4 4 5 5 5 4
13400 014/300 Write block 1446 len 4096 data: 3
13401 015/300 Read block 1773 len 24576
13402 016/300 Read block 2997 len 40960
13403 017/300 Write block 4287 len 12288 data: 3 3 3
13404 018/300 Read block 960 len 40960
13405 019/300 Write block 1470 len 36864 data: 7 7 8 10 9 8 7 6 6
13406 020/300 Read block 3265 len 16384
13407 021/300 Read block 1027 len 16384
13408 022/300 Read block 3250 len 12288
13409 023/300 Read block 3313 len 24576
13410 024/300 Read block 1221 len 36864
13411 025/300 Write block 932 len 4096 data: 4
13412 026/300 Write block 4949 len 20480 data: 5 5 3 2 2
13413 027/300 Write block 3451 len 8192 data: 4 5
13414 028/300 Write block 2169 len 20480 data: 4 2 2 3 3
13415 029/300 Read block 4014 len 24576
13416 030/300 Write block 2144 len 12288 data: 4 4 4
13417 031/300 Read block 2970 len 24576
13418 032/300 Read block 2827 len 24576
13419 033/300 Write block 2739 len 16384 data: 3 3 3 2
13420 034/300 Read block 4407 len 16384
13421 035/300 Read block 3208 len 12288
13422 036/300 Flush
13423 037/300 Read block 1239 len 40960
13424 038/300 Write block 115 len 36864 data: 5 5 6 6 6 4 4 5 4
13425 039/300 Read block 761 len 28672
13426 040/300 Read block 4694 len 4096
13427 041/300 Read block 3797 len 28672
13428 042/300 Read block 4295 len 36864
13429 043/300 Write block 1091 len 16384 data: 3 3 3 4
13430 044/300 Write block 3858 len 4096 data: 4
13431 045/300 Read block 4159 len 4096
13432 046/300 Write block 3083 len 4096 data: 5
13433 047/300 Write block 314 len 12288 data: 4 4 4
13434 048/300 Read block 3037 len 40960
13435 049/300 Flush
13436 050/300 Write block 2234 len 32768 data: 3 3 4 4 4 4 5 6
13437 051/300 Flush
13438 052/300 Write block 1341 len 20480 data: 2 2 2 2 3
13439 053/300 Flush
13440 054/300 Read block 4903 len 4096
13441 055/300 Write block 303 len 40960 data: 3 3 3 3 3 3 3 3 2 3
13442 056/300 Read block 1364 len 24576
13443 057/300 Write block 764 len 16384 data: 5 3 3 2
13444 058/300 Read block 2117 len 20480
13445 059/300 Write block 232 len 20480 data: 2 2 5 4 3
13446 060/300 Read block 3067 len 16384
13447 061/300 Read block 123 len 24576
13448 062/300 Write block 75 len 40960 data: 2 2 3 3 3 3 4 4 3 5
13449 063/300 Write block 1487 len 32768 data: 4 3 3 4 4 4 3 3
13450 064/300 Write block 1586 len 4096 data: 7
13451 065/300 Flush
13452 066/300 Read block 2217 len 36864
13453 067/300 Read block 3997 len 16384
13454 068/300 Write block 2079 len 4096 data: 2
13455 069/300 Read block 121 len 20480
13456 070/300 Read block 4853 len 40960
13457 071/300 Write block 611 len 28672 data: 4 4 4 4 4 3 3
13458 072/300 Read block 348 len 32768
13459 073/300 Read block 1652 len 24576
13460 074/300 Write block 4059 len 4096 data: 4
13461 075/300 Read block 2710 len 20480
13462 076/300 Flush
13463 077/300 Write block 2556 len 20480 data: 2 2 2 2 2
13464 078/300 Read block 4350 len 36864
13465 079/300 Read block 4578 len 16384
13466 080/300 Write block 77 len 40960 data: 4 4 4 4 5 5 4 6 4 4
13467 081/300 Read block 1505 len 28672
13468 082/300 Write block 4734 len 28672 data: 4 4 4 4 3 3 3
13469 083/300 Write block 364 len 8192 data: 2 2
13470 084/300 Read block 2188 len 4096
13471 085/300 Read block 2423 len 32768
13472 086/300 Read block 3836 len 20480
13473 087/300 Flush
13474 088/300 Write block 3704 len 32768 data: 5 4 3 3 4 6 4 4
13475 089/300 Write block 1905 len 32768 data: 4 5 4 3 3 3 3 3
13476 090/300 Read block 799 len 12288
13477 091/300 Write block 2406 len 40960 data: 4 4 4 5 5 5 7 8 8 8
13478 092/300 Read block 2970 len 28672
13479 093/300 Read block 3106 len 36864
13480 094/300 Write block 1634 len 40960 data: 4 3 5 6 6 6 6 6 6 6
13481 095/300 Write block 3911 len 24576 data: 3 4 4 3 3 3
13482 096/300 Read block 2423 len 12288
13483 097/300 Write block 4781 len 20480 data: 3 4 4 4 6
13484 098/300 Read block 2658 len 16384
13485 099/300 Flush
13486 100/300 Write block 1931 len 12288 data: 2 2 2
13487 101/300 Flush
13488 102/300 Write block 2539 len 16384 data: 4 4 5 5
13489 103/300 Write block 1411 len 12288 data: 4 4 4
13490 104/300 Write block 4744 len 20480 data: 2 2 3 3 5
13491 105/300 Read block 4493 len 16384
13492 106/300 Write block 635 len 28672 data: 2 2 2 3 3 3 3
13493 107/300 Read block 3818 len 24576
13494 108/300 Write block 2986 len 40960 data: 6 6 6 5 5 4 3 3 2 2
13495 109/300 Read block 4011 len 24576
13496 110/300 Read block 82 len 16384
13497 111/300 Write block 2740 len 28672 data: 4 4 3 2 2 2 2
13498 112/300 Write block 4580 len 20480 data: 5 4 5 4 3
13499 113/300 Read block 466 len 16384
13500 114/300 Read block 4384 len 28672
13501 115/300 Flush
13502 116/300 Read block 743 len 8192
13503 117/300 Write block 845 len 28672 data: 4 5 4 4 3 3 4
13504 118/300 Flush
13505 119/300 Read block 2548 len 20480
13506 120/300 Read block 285 len 16384
13507 121/300 Read block 4626 len 32768
13508 122/300 Read block 4888 len 8192
13509 123/300 Write block 4773 len 24576 data: 3 3 2 2 2 2
13510 124/300 Read block 3710 len 20480
13511 125/300 Write block 3241 len 28672 data: 5 3 3 3 3 3 4
13512 126/300 Read block 1388 len 16384
13513 127/300 Write block 2114 len 16384 data: 4 4 5 5
13514 128/300 Write block 599 len 24576 data: 5 2 3 4 4 4
13515 129/300 Read block 2820 len 20480
13516 130/300 Write block 1346 len 24576 data: 3 3 3 5 5 5
13517 131/300 Flush
13518 132/300 Read block 1942 len 40960
13519 133/300 Read block 1142 len 20480
13520 134/300 Flush
13521 135/300 Write block 2200 len 16384 data: 3 4 5 4
13522 136/300 Flush
13523 137/300 Write block 2563 len 16384 data: 4 2 2 2
13524 138/300 Read block 2214 len 8192
13525 139/300 Flush
13526 140/300 Read block 4082 len 28672
13527 141/300 Read block 1890 len 20480
13528 142/300 Write block 4979 len 16384 data: 5 5 5 4
13529 143/300 Read block 3091 len 40960
13530 144/300 Read block 2992 len 4096
13531 145/300 Read block 3763 len 8192
13532 146/300 Read block 4494 len 16384
13533 147/300 Write block 5 len 28672 data: 4 4 4 3 3 3 3
13534 148/300 Read block 54 len 20480
13535 149/300 Read block 1949 len 32768
13536 150/300 Write block 3543 len 28672 data: 6 6 6 6 6 5 5
13537 151/300 Write block 1974 len 36864 data: 4 4 4 5 6 6 4 5 5
13538 152/300 Read block 1556 len 4096
13539 153/300 Read block 1685 len 12288
13540 154/300 Write block 1383 len 20480 data: 2 2 4 4 5
13541 155/300 Read block 753 len 8192
13542 156/300 Write block 3737 len 16384 data: 4 4 5 7
13543 157/300 Write block 1327 len 32768 data: 4 4 5 4 5 6 5 6
13544 158/300 Write block 4430 len 16384 data: 3 4 5 5
13545 159/300 Write block 1037 len 20480 data: 5 6 5 7 7
13546 160/300 Read block 4600 len 20480
13547 161/300 Flush
13548 162/300 Read block 4850 len 16384
13549 163/300 Write block 291 len 28672 data: 2 2 4 6 6 6 5
13550 164/300 Read block 2921 len 20480
13551 165/300 Read block 1944 len 28672
13552 166/300 Write block 3358 len 40960 data: 6 6 6 6 5 5 5 5 6 5
13553 167/300 Write block 915 len 16384 data: 3 4 4 7
13554 168/300 Write block 2982 len 12288 data: 3 4 5
13555 169/300 Read block 868 len 36864
13556 170/300 Write block 805 len 20480 data: 4 4 3 3 3
13557 171/300 Write block 643 len 16384 data: 3 2 2 2
13558 172/300 Write block 4575 len 36864 data: 3 4 4 5 5 6 5 6 5
13559 173/300 Flush
13560 174/300 Flush
13561 175/300 Read block 1789 len 36864
13562 176/300 Read block 2828 len 40960
13563 177/300 Write block 4012 len 36864 data: 6 7 8 7 6 4 4 5 5
13564 178/300 Read block 2480 len 24576
13565 179/300 Write block 4408 len 4096 data: 3
13566 180/300 Read block 1083 len 32768
13567 181/300 Read block 2693 len 32768
13568 182/300 Flush
13569 183/300 Read block 4389 len 24576
13570 184/300 Read block 3038 len 40960
13571 185/300 Read block 2234 len 12288
13572 186/300 Read block 884 len 40960
13573 187/300 Write block 2551 len 16384 data: 3 3 3 3
13574 188/300 Read block 4356 len 24576
13575 189/300 Write block 3203 len 40960 data: 6 6 6 6 7 6 5 4 5 6
13576 190/300 Flush
13577 191/300 Read block 4050 len 40960
13578 192/300 Read block 4171 len 36864
13579 193/300 Read block 4511 len 4096
13580 194/300 Read block 3752 len 28672
13581 195/300 Write block 1718 len 4096 data: 6
13582 196/300 Flush
13583 197/300 Read block 1420 len 4096
13584 198/300 Write block 1310 len 36864 data: 4 5 4 2 2 2 3 3 3
13585 199/300 Write block 1489 len 28672 data: 4 5 5 5 4 4 3
13586 200/300 Read block 1192 len 4096
13587 201/300 Read block 3389 len 4096
13588 202/300 Read block 1995 len 8192
13589 203/300 Read block 170 len 4096
13590 204/300 Read block 3931 len 36864
13591 205/300 Read block 4137 len 8192
13592 206/300 Read block 4083 len 28672
13593 207/300 Read block 842 len 40960
13594 208/300 Write block 3076 len 24576 data: 3 5 5 5 5 4
13595 209/300 Write block 1893 len 32768 data: 5 5 4 6 6 4 5 5
13596 210/300 Write block 28 len 20480 data: 4 3 3 3 3
13597 211/300 Write block 2378 len 8192 data: 2 2
13598 212/300 Read block 1832 len 16384
13599 213/300 Read block 932 len 28672
13600 214/300 Flush
13601 215/300 Read block 4452 len 40960
13602 216/300 Read block 1495 len 16384
13603 217/300 Write block 973 len 36864 data: 2 2 2 3 4 5 7 6 6
13604 218/300 Read block 3528 len 12288
13605 219/300 Write block 1071 len 32768 data: 3 4 4 2 2 2 2 2
13606 220/300 Write block 2090 len 32768 data: 3 4 4 4 4 5 5 5
13607 221/300 Write block 3280 len 36864 data: 4 4 4 5 5 4 4 4 5
13608 222/300 Write block 4339 len 20480 data: 4 3 3 2 2
13609 223/300 Read block 3776 len 16384
13610 224/300 Read block 3537 len 8192
13611 225/300 Read block 4116 len 20480
13612 226/300 Read block 770 len 40960
13613 227/300 Write block 4756 len 16384 data: 2 2 3 4
13614 228/300 Read block 3293 len 28672
13615 229/300 Write block 1273 len 20480 data: 4 4 4 3 3
13616 230/300 Write block 4452 len 16384 data: 4 4 4 4
13617 231/300 Write block 3367 len 8192 data: 6 5
13618 232/300 Read block 2861 len 4096
13619 233/300 Read block 3167 len 36864
13620 234/300 Read block 213 len 40960
13621 235/300 Write block 2515 len 16384 data: 4 6 6 6
13622 236/300 Read block 1115 len 20480
13623 237/300 Read block 1499 len 40960
13624 238/300 Write block 3440 len 8192 data: 7 5
13625 239/300 Write block 3635 len 32768 data: 4 3 3 3 5 5 5 4
13626 240/300 Read block 2092 len 8192
13627 241/300 Read block 4875 len 4096
13628 242/300 Write block 1274 len 8192 data: 5 5
13629 243/300 Read block 4928 len 16384
13630 244/300 Write block 2883 len 32768 data: 5 6 5 7 7 7 8 9
13631 245/300 Write block 706 len 40960 data: 2 2 2 2 2 3 3 3 4 4
13632 246/300 Read block 2070 len 20480
13633 247/300 Write block 1599 len 16384 data: 5 5 5 5
13634 248/300 Read block 3133 len 20480
13635 249/300 Read block 2651 len 36864
13636 250/300 Write block 4330 len 16384 data: 5 5 6 5
13637 251/300 Read block 3900 len 12288
13638 252/300 Read block 140 len 36864
13639 253/300 Read block 3727 len 32768
13640 254/300 Read block 2781 len 28672
13641 255/300 Write block 610 len 12288 data: 4 5 5
13642 256/300 Read block 776 len 8192
13643 257/300 Write block 3887 len 28672 data: 5 5 4 4 5 5 4
13644 258/300 Read block 383 len 16384
13645 259/300 Read block 3994 len 40960
13646 260/300 Read block 2967 len 12288
13647 261/300 Write block 4854 len 8192 data: 2 2
13648 262/300 Read block 1146 len 32768
13649 263/300 Read block 3105 len 12288
13650 264/300 Write block 60 len 36864 data: 5 5 5 6 5 4 5 4 5
13651 265/300 Write block 3769 len 32768 data: 2 2 2 4 5 5 5 5
13652 266/300 Write block 1440 len 4096 data: 4
13653 267/300 Write block 1991 len 36864 data: 7 6 3 4 3 4 3 2 2
13654 268/300 Write block 2286 len 8192 data: 3 3
13655 269/300 Read block 2646 len 28672
13656 270/300 Write block 2269 len 12288 data: 3 3 3
13657 271/300 Write block 248 len 24576 data: 4 5 5 4 3 2
13658 272/300 Read block 1183 len 16384
13659 273/300 Read block 841 len 8192
13660 274/300 Write block 1136 len 40960 data: 5 5 5 5 7 8 6 5 5 3
13661 275/300 Write block 3112 len 12288 data: 7 6 5
13662 276/300 Write block 148 len 8192 data: 5 4
13663 277/300 Write block 1898 len 40960 data: 5 6 6 5 4 4 4 5 6 5
13664 278/300 Read block 4077 len 40960
13665 279/300 Read block 2438 len 40960
13666 280/300 Read block 2848 len 8192
13667 281/300 Write block 2099 len 20480 data: 6 4 3 3 3
13668 282/300 Write block 2567 len 36864 data: 2 2 2 2 3 2 2 2 2
13669 283/300 Write block 4671 len 20480 data: 2 3 2 2 3
13670 284/300 Read block 2821 len 36864
13671 285/300 Flush
13672 286/300 Flush
13673 287/300 Write block 2236 len 20480 data: 5 5 5 5 6
13674 288/300 Read block 1855 len 24576
13675 289/300 Read block 2164 len 24576
13676 290/300 Read block 810 len 8192
13677 291/300 Write block 13 len 40960 data: 4 3 4 3 3 2 2 2 2 2
13678 292/300 Write block 335 len 12288 data: 3 2 2
13679 293/300 Read block 718 len 8192
13680 294/300 Read block 2368 len 4096
13681 295/300 Flush
13682 296/300 Write block 3942 len 36864 data: 5 5 5 4 4 4 3 3 3
13683 297/300 Flush
13684 298/300 Read block 585 len 20480
13685 299/300 Read block 553 len 12288
13686 300/300 Write block 3475 len 24576 data: 3 4 4 4 4 4
13687 Main task has work to do, go find it
13688 got dsc Start(1)
13689 start 1
13690 Feb 28 22:16:01.282 INFO request completed, latency_us: 220, response_code: 204, uri: /start/cid/1, method: POST, req_id: 8e6df5b6-01f9-40c1-87c1-50a5bde93656, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
13691 Replay: started 1, returned:Ok(())
13692 ----------------------------------------------------------------
13693 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13694 States: Active Offline Active
13695 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
13696 4521 Acked 5520 Write 9 Done New Done true
13697 4522 Acked 5521 Read 5 Done New Done true
13698 4523 Acked 5522 Read 9 Done New Done false
13699 4524 Acked 5523 Write 4 Done New Done false
13700 4525 Acked 5524 Write 9 Done New Done false
13701 4526 Acked 5525 Read 3 Done New Done false
13702 4527 Acked 5526 Write 3 Done New Done false
13703 4528 Acked 5527 Read 9 Done New Done false
13704 4529 Acked 5528 Read 2 Done New Done false
13705 4530 Acked 5529 Write 2 Done New Done false
13706 4531 Acked 5530 Read 4 Done New Done false
13707 4532 Acked 5531 Write 2 Done New Done false
13708 4533 Acked 5532 Write 10 Done New Done false
13709 4534 Acked 5533 Write 1 Done New Done false
13710 4535 Acked 5534 Read 6 Done New Done false
13711 4536 Acked 5535 Read 10 Done New Done false
13712 4537 Acked 5536 Write 3 Done New Done false
13713 4538 Acked 5537 Read 10 Done New Done false
13714 4539 Acked 5538 Write 9 Done New Done false
13715 4540 Acked 5539 Read 4 Done New Done false
13716 4541 Acked 5540 Read 4 Done New Done false
13717 4542 Acked 5541 Read 3 Done New Done false
13718 4543 Acked 5542 Read 6 Done New Done false
13719 4544 Acked 5543 Read 9 Done New Done false
13720 4545 Acked 5544 Write 1 Done New Done false
13721 4546 Acked 5545 Write 5 Done New Done false
13722 4547 Acked 5546 Write 2 Done New Done false
13723 4548 Acked 5547 Write 5 Done New Done false
13724 4549 Acked 5548 Read 6 Done New Done false
13725 4550 Acked 5549 Write 3 Done New Done false
13726 4551 Acked 5550 Read 6 Done New Done false
13727 4552 Acked 5551 Read 6 Done New Done false
13728 4553 Acked 5552 Write 4 Done New Done false
13729 4554 Acked 5553 Read 4 Done New Done false
13730 4555 Acked 5554 Read 3 Done New Done false
13731 4556 Acked 5555 Flush 0 Done New Done false
13732 4557 Acked 5556 Read 10 Done New Done false
13733 4558 Acked 5557 Write 9 Done New Done false
13734 4559 Acked 5558 Read 7 Done New Done false
13735 4560 Acked 5559 Read 1 Done New Done false
13736 4561 Acked 5560 Read 7 Done New Done false
13737 4562 Acked 5561 Read 9 Done New Done false
13738 4563 Acked 5562 Write 4 Done New Done false
13739 4564 Acked 5563 Write 1 Done New Done false
13740 4565 Acked 5564 Read 1 Done New Done false
13741 4566 Acked 5565 Write 1 Done New Done false
13742 4567 Acked 5566 Write 3 Done New Done false
13743 4568 Acked 5567 Read 10 Done New Done false
13744 4569 Acked 5568 Flush 0 Done New Done false
13745 4570 Acked 5569 Write 8 Done New Done false
13746 4571 Acked 5570 Flush 0 Done New Done false
13747 4572 Acked 5571 Write 5 Done New Done false
13748 4573 Acked 5572 Flush 0 Done New Done false
13749 4574 Acked 5573 Read 1 Done New Done false
13750 4575 Acked 5574 Write 10 Done New Done false
13751 4576 Acked 5575 Read 6 Done New Done false
13752 4577 Acked 5576 Write 4 Done New Done false
13753 4578 Acked 5577 Read 5 Done New Done false
13754 4579 Acked 5578 Write 5 Done New Done false
13755 4580 Acked 5579 Read 4 Done New Done false
13756 4581 Acked 5580 Read 6 Done New Done false
13757 4582 Acked 5581 Write 10 Done New Done false
13758 4583 Acked 5582 Write 8 Done New Done false
13759 4584 Acked 5583 Write 1 Done New Done false
13760 4585 Acked 5584 Flush 0 Done New Done false
13761 4586 Acked 5585 Read 9 Done New Done false
13762 4587 Acked 5586 Read 4 Done New Done false
13763 4588 Acked 5587 Write 1 Done New Done false
13764 4589 Acked 5588 Read 5 Done New Done false
13765 4590 Acked 5589 Read 10 Done New Done false
13766 4591 Acked 5590 Write 7 Done New Done false
13767 4592 Acked 5591 Read 8 Done New Done false
13768 4593 Acked 5592 Read 6 Done New Done false
13769 4594 Acked 5593 Write 1 Done New Done false
13770 4595 Acked 5594 Read 5 Done New Done false
13771 4596 Acked 5595 Flush 0 Done New Done false
13772 4597 Acked 5596 Write 5 Done New Done false
13773 4598 Acked 5597 Read 9 Done New Done false
13774 4599 Acked 5598 Read 4 Done New Done false
13775 4600 Acked 5599 Write 10 Done New Done false
13776 4601 Acked 5600 Read 7 Done New Done false
13777 4602 Acked 5601 Write 7 Done New Done false
13778 4603 Acked 5602 Write 2 Done New Done false
13779 4604 Acked 5603 Read 1 Done New Done false
13780 4605 Acked 5604 Read 8 Done New Done false
13781 4606 Acked 5605 Read 5 Done New Done false
13782 4607 Acked 5606 Flush 0 Done New Done false
13783 4608 Acked 5607 Write 8 Done New Done false
13784 4609 Acked 5608 Write 8 Done New Done false
13785 4610 Acked 5609 Read 3 Done New Done false
13786 4611 Acked 5610 Write 10 Done New Done false
13787 4612 Acked 5611 Read 7 Done New Done false
13788 4613 Acked 5612 Read 9 Done New Done false
13789 4614 Acked 5613 Write 10 Done New Done false
13790 4615 Acked 5614 Write 6 Done New Done false
13791 4616 Acked 5615 Read 3 Done New Done false
13792 4617 Acked 5616 Write 5 Done New Done false
13793 4618 Acked 5617 Read 4 Done New Done false
13794 4619 Acked 5618 Flush 0 Done New Done false
13795 4620 Acked 5619 Write 3 Done New Done false
13796 4621 Acked 5620 Flush 0 Done New Done false
13797 4622 Acked 5621 Write 4 Done New Done false
13798 4623 Acked 5622 Write 3 Done New Done false
13799 4624 Acked 5623 Write 5 Done New Done false
13800 4625 Acked 5624 Read 4 Done New Done false
13801 4626 Acked 5625 Write 7 Done New Done false
13802 4627 Acked 5626 Read 6 Done New Done false
13803 4628 Acked 5627 Write 10 Done New Done false
13804 4629 Acked 5628 Read 6 Done New Done false
13805 4630 Acked 5629 Read 4 Done New Done false
13806 4631 Acked 5630 Write 7 Done New Done false
13807 4632 Acked 5631 Write 5 Done New Done false
13808 4633 Acked 5632 Read 4 Done New Done false
13809 4634 Acked 5633 Read 7 Done New Done false
13810 4635 Acked 5634 Flush 0 Done New Done false
13811 4636 Acked 5635 Read 2 Done New Done false
13812 4637 Acked 5636 Write 7 Done New Done false
13813 4638 Acked 5637 Flush 0 Done New Done false
13814 4639 Acked 5638 Read 5 Done New Done false
13815 4640 Acked 5639 Read 4 Done New Done false
13816 4641 Acked 5640 Read 8 Done New Done false
13817 4642 Acked 5641 Read 2 Done New Done false
13818 4643 Acked 5642 Write 6 Done New Done false
13819 4644 Acked 5643 Read 5 Done New Done false
13820 4645 Acked 5644 Write 7 Done New Done false
13821 4646 Acked 5645 Read 4 Done New Done false
13822 4647 Acked 5646 Write 4 Done New Done false
13823 4648 Acked 5647 Write 6 Done New Done false
13824 4649 Acked 5648 Read 5 Done New Done false
13825 4650 Acked 5649 Write 6 Done New Done false
13826 4651 Acked 5650 Flush 0 Done New Done false
13827 4652 Acked 5651 Read 10 Done New Done false
13828 4653 Acked 5652 Read 5 Done New Done false
13829 4654 Acked 5653 Flush 0 Done New Done false
13830 4655 Acked 5654 Write 4 Done New Done false
13831 4656 Acked 5655 Flush 0 Done New Done false
13832 4657 Acked 5656 Write 4 Done New Done false
13833 4658 Acked 5657 Read 2 Done New Done false
13834 4659 Acked 5658 Flush 0 Done New Done false
13835 4660 Acked 5659 Read 7 Done New Done false
13836 4661 Acked 5660 Read 5 Done New Done false
13837 4662 Acked 5661 Write 4 Done New Done false
13838 4663 Acked 5662 Read 10 Done New Done false
13839 4664 Acked 5663 Read 1 Done New Done false
13840 4665 Acked 5664 Read 2 Done New Done false
13841 4666 Acked 5665 Read 4 Done New Done false
13842 4667 Acked 5666 Write 7 Done New Done false
13843 4668 Acked 5667 Read 5 Done New Done false
13844 4669 Acked 5668 Read 8 Done New Done false
13845 4670 Acked 5669 Write 7 Done New Done false
13846 4671 Acked 5670 Write 9 Done New Done false
13847 4672 Acked 5671 Read 1 Done New Done false
13848 4673 Acked 5672 Read 3 Done New Done false
13849 4674 Acked 5673 Write 5 Done New Done false
13850 4675 Acked 5674 Read 2 Done New Done false
13851 4676 Acked 5675 Write 4 Done New Done false
13852 4677 Acked 5676 Write 8 Done New Done false
13853 4678 Acked 5677 Write 4 Done New Done false
13854 4679 Acked 5678 Write 5 Done New Done false
13855 4680 Acked 5679 Read 5 Done New Done false
13856 4681 Acked 5680 Flush 0 Done New Done false
13857 4682 Acked 5681 Read 4 Done New Done false
13858 4683 Acked 5682 Write 7 Done New Done false
13859 4684 Acked 5683 Read 5 Done New Done false
13860 4685 Acked 5684 Read 7 Done New Done false
13861 4686 Acked 5685 Write 10 Done New Done false
13862 4687 Acked 5686 Write 4 Done New Done false
13863 4688 Acked 5687 Write 3 Done New Done false
13864 4689 Acked 5688 Read 9 Done New Done false
13865 4690 Acked 5689 Write 5 Done New Done false
13866 4691 Acked 5690 Write 4 Done New Done false
13867 4692 Acked 5691 Write 9 Done New Done false
13868 4693 Acked 5692 Flush 0 Done New Done false
13869 4694 Acked 5693 Flush 0 Done New Done false
13870 4695 Acked 5694 Read 9 Done New Done false
13871 4696 Acked 5695 Read 10 Done New Done false
13872 4697 Acked 5696 Write 9 Done New Done false
13873 4698 Acked 5697 Read 6 Done New Done false
13874 4699 Acked 5698 Write 1 Done New Done false
13875 4700 Acked 5699 Read 8 Done New Done false
13876 4701 Acked 5700 Read 8 Done New Done false
13877 4702 Acked 5701 Flush 0 Done New Done false
13878 4703 Acked 5702 Read 6 Done New Done false
13879 4704 Acked 5703 Read 10 Done New Done false
13880 4705 Acked 5704 Read 3 Done New Done false
13881 4706 Acked 5705 Read 10 Done New Done false
13882 4707 Acked 5706 Write 4 Done New Done false
13883 4708 Acked 5707 Read 6 Done New Done false
13884 4709 Acked 5708 Write 10 Done New Done false
13885 4710 Acked 5709 Flush 0 Done New Done false
13886 4711 Acked 5710 Read 10 Done New Done false
13887 4712 Acked 5711 Read 9 Done New Done false
13888 4713 Acked 5712 Read 1 Done New Done false
13889 4714 Acked 5713 Read 7 Done New Done false
13890 4715 Acked 5714 Write 1 Done New Done false
13891 4716 Acked 5715 Flush 0 Done New Done false
13892 4717 Acked 5716 Read 1 Done New Done false
13893 4718 Acked 5717 Write 9 Done New Done false
13894 4719 Acked 5718 Write 7 Done New Done false
13895 4720 Acked 5719 Read 1 Done New Done false
13896 4721 Acked 5720 Read 1 Done New Done false
13897 4722 Acked 5721 Read 2 Done New Done false
13898 4723 Acked 5722 Read 1 Done New Done false
13899 4724 Acked 5723 Read 9 Done New Done false
13900 4725 Acked 5724 Read 2 Done New Done false
13901 4726 Acked 5725 Read 7 Done New Done false
13902 4727 Acked 5726 Read 10 Done New Done false
13903 4728 Acked 5727 Write 6 Done New Done false
13904 4729 Acked 5728 Write 8 Done New Done false
13905 4730 Acked 5729 Write 5 Done New Done false
13906 4731 Acked 5730 Write 2 Done New Done false
13907 4732 Acked 5731 Read 4 Done New Done false
13908 4733 Acked 5732 Read 7 Done New Done false
13909 4734 Acked 5733 Flush 0 Done New Done false
13910 4735 Acked 5734 Read 10 Done New Done false
13911 4736 Acked 5735 Read 4 Done New Done false
13912 4737 Acked 5736 Write 9 Done New Done false
13913 4738 Acked 5737 Read 3 Done New Done false
13914 4739 Acked 5738 Write 8 Done New Done false
13915 4740 Acked 5739 Write 8 Done New Done false
13916 4741 Acked 5740 Write 9 Done New Done false
13917 4742 Acked 5741 Write 5 Done New Done false
13918 4743 Acked 5742 Read 4 Done New Done false
13919 4744 Acked 5743 Read 2 Done New Done false
13920 4745 Acked 5744 Read 5 Done New Done false
13921 4746 Acked 5745 Read 10 Done New Done false
13922 4747 Acked 5746 Write 4 Done New Done false
13923 4748 Acked 5747 Read 7 Done New Done false
13924 4749 Acked 5748 Write 5 Done New Done false
13925 4750 Acked 5749 Write 4 Done New Done false
13926 4751 Acked 5750 Write 2 Done New Done false
13927 4752 Acked 5751 Read 1 Done New Done false
13928 4753 Acked 5752 Read 9 Done New Done false
13929 4754 Acked 5753 Read 10 Done New Done false
13930 4755 Acked 5754 Write 4 Done New Done false
13931 4756 Acked 5755 Read 5 Done New Done false
13932 4757 Acked 5756 Read 10 Done New Done false
13933 4758 Acked 5757 Write 2 Done New Done false
13934 4759 Acked 5758 Write 8 Done New Done false
13935 4760 Acked 5759 Read 2 Done New Done false
13936 4761 Acked 5760 Read 1 Done New Done false
13937 4762 Acked 5761 Write 2 Done New Done false
13938 4763 Acked 5762 Read 4 Done New Done false
13939 4764 Acked 5763 Write 8 Done New Done false
13940 4765 Acked 5764 Write 10 Done New Done false
13941 4766 Acked 5765 Read 5 Done New Done false
13942 4767 Acked 5766 Write 4 Done New Done false
13943 4768 Acked 5767 Read 5 Done New Done false
13944 4769 Acked 5768 Read 9 Done New Done false
13945 4770 Acked 5769 Write 4 Done New Done false
13946 4771 Acked 5770 Read 3 Done New Done false
13947 4772 Acked 5771 Read 9 Done New Done false
13948 4773 Acked 5772 Read 8 Done New Done false
13949 4774 Acked 5773 Read 7 Done New Done false
13950 4775 Acked 5774 Write 3 Done New Done false
13951 4776 Acked 5775 Read 2 Done New Done false
13952 4777 Acked 5776 Write 7 Done New Done false
13953 4778 Acked 5777 Read 4 Done New Done false
13954 4779 Acked 5778 Read 10 Done New Done false
13955 4780 Acked 5779 Read 3 Done New Done false
13956 4781 Acked 5780 Write 2 Done New Done false
13957 4782 Acked 5781 Read 8 Done New Done false
13958 4783 Acked 5782 Read 3 Done New Done false
13959 4784 Acked 5783 Write 9 Done New Done false
13960 4785 Acked 5784 Write 8 Done New Done false
13961 4786 Acked 5785 Write 1 Done New Done false
13962 4787 Acked 5786 Write 9 Done New Done false
13963 4788 Acked 5787 Write 2 Done New Done false
13964 4789 Acked 5788 Read 7 Done New Done false
13965 4790 Acked 5789 Write 3 Done New Done false
13966 4791 Acked 5790 Write 6 Done New Done false
13967 4792 Acked 5791 Read 4 Done New Done false
13968 4793 Acked 5792 Read 2 Done New Done false
13969 4794 Acked 5793 Write 10 Done New Done false
13970 4795 Acked 5794 Write 3 Done New Done false
13971 4796 Acked 5795 Write 2 Done New Done false
13972 4797 Acked 5796 Write 10 Done New Done false
13973 4798 Acked 5797 Read 10 Done New Done false
13974 4799 Acked 5798 Read 10 Done New Done false
13975 4800 Acked 5799 Read 2 Done New Done false
13976 4801 Acked 5800 Write 5 Done New Done false
13977 4802 Acked 5801 Flush 0 Done New Done false
13978 4803 Acked 5802 Write 9 Done New Done false
13979 4804 Acked 5803 Write 5 Done New Done false
13980 4805 Acked 5804 Read 9 Done New Done false
13981 4806 Acked 5805 Flush 0 Done New Done false
13982 4807 Acked 5806 Flush 0 Done New Done false
13983 4808 Acked 5807 Write 5 Done New Done false
13984 4809 Acked 5808 Read 6 Done New Done false
13985 4810 Acked 5809 Read 6 Done New Done false
13986 4811 Acked 5810 Read 2 Done New Done false
13987 4812 Acked 5811 Write 10 Done New Done false
13988 4813 Acked 5812 Write 3 Done New Done false
13989 4814 Acked 5813 Read 2 Done New Done false
13990 4815 Acked 5814 Read 1 Done New Done false
13991 4816 Acked 5815 Flush 0 Done New Done false
13992 4817 Acked 5816 Write 9 Done New Done false
13993 4818 Acked 5817 Flush 0 Done New Done false
13994 4819 Acked 5818 Read 5 Done New Done false
13995 4820 Acked 5819 Read 3 Done New Done false
13996 4821 Acked 5820 Write 6 Done New Done false
13997 STATES DS:0 DS:1 DS:2 TOTAL
13998 New 0 301 0 301
13999 Sent 0 0 0 0
14000 Done 301 0 301 602
14001 Skipped 0 0 0 0
14002 Error 0 0 0 0
14003 Last Flush: 5817 5519 5817
14004 Downstairs last five completed: 5519 5518 5517 5516 5515
14005 Upstairs last five completed: 4821 4820 4819 4818 4817
14006 CLIENT: Up:0 ds:301 act:2
140072024-02-28T22:16:01.510ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140082024-02-28T22:16:01.510ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140092024-02-28T22:16:01.510ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140102024-02-28T22:16:01.510ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140112024-02-28T22:16:01.510ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140122024-02-28T22:16:01.510ZINFOcrucible: [1] client re-new 301 jobs since flush 5519 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140132024-02-28T22:16:01.510ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140142024-02-28T22:16:02.511ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140152024-02-28T22:16:02.511ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140162024-02-28T22:16:02.511ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140172024-02-28T22:16:02.511ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140182024-02-28T22:16:02.511ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140192024-02-28T22:16:02.511ZINFOcrucible: [1] client re-new 302 jobs since flush 5519 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140202024-02-28T22:16:02.511ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140212024-02-28T22:16:03.512ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140222024-02-28T22:16:03.512ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140232024-02-28T22:16:03.512ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140242024-02-28T22:16:03.512ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140252024-02-28T22:16:03.512ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140262024-02-28T22:16:03.512ZINFOcrucible: [1] client re-new 302 jobs since flush 5519 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140272024-02-28T22:16:03.513ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140282024-02-28T22:16:04.514ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140292024-02-28T22:16:04.514ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140302024-02-28T22:16:04.514ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140312024-02-28T22:16:04.514ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140322024-02-28T22:16:04.514ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140332024-02-28T22:16:04.514ZINFOcrucible: [1] client re-new 302 jobs since flush 5519 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
140342024-02-28T22:16:04.514ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
14035 ----------------------------------------------------------------
14036 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
14037 States: Active Offline Active
14038 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
14039 4521 Acked 5520 Write 9 Done New Done true
14040 4522 Acked 5521 Read 5 Done New Done true
14041 4523 Acked 5522 Read 9 Done New Done true
14042 4524 Acked 5523 Write 4 Done New Done true
14043 4525 Acked 5524 Write 9 Done New Done true
14044 4526 Acked 5525 Read 3 Done New Done true
14045 4527 Acked 5526 Write 3 Done New Done true
14046 4528 Acked 5527 Read 9 Done New Done true
14047 4529 Acked 5528 Read 2 Done New Done true
14048 4530 Acked 5529 Write 2 Done New Done true
14049 4531 Acked 5530 Read 4 Done New Done true
14050 4532 Acked 5531 Write 2 Done New Done true
14051 4533 Acked 5532 Write 10 Done New Done true
14052 4534 Acked 5533 Write 1 Done New Done true
14053 4535 Acked 5534 Read 6 Done New Done true
14054 4536 Acked 5535 Read 10 Done New Done true
14055 4537 Acked 5536 Write 3 Done New Done true
14056 4538 Acked 5537 Read 10 Done New Done true
14057 4539 Acked 5538 Write 9 Done New Done true
14058 4540 Acked 5539 Read 4 Done New Done true
14059 4541 Acked 5540 Read 4 Done New Done true
14060 4542 Acked 5541 Read 3 Done New Done true
14061 4543 Acked 5542 Read 6 Done New Done true
14062 4544 Acked 5543 Read 9 Done New Done true
14063 4545 Acked 5544 Write 1 Done New Done true
14064 4546 Acked 5545 Write 5 Done New Done true
14065 4547 Acked 5546 Write 2 Done New Done true
14066 4548 Acked 5547 Write 5 Done New Done true
14067 4549 Acked 5548 Read 6 Done New Done true
14068 4550 Acked 5549 Write 3 Done New Done true
14069 4551 Acked 5550 Read 6 Done New Done true
14070 4552 Acked 5551 Read 6 Done New Done true
14071 4553 Acked 5552 Write 4 Done New Done true
14072 4554 Acked 5553 Read 4 Done New Done true
14073 4555 Acked 5554 Read 3 Done New Done true
14074 4556 Acked 5555 Flush 0 Done New Done true
14075 4557 Acked 5556 Read 10 Done New Done true
14076 4558 Acked 5557 Write 9 Done New Done true
14077 4559 Acked 5558 Read 7 Done New Done true
14078 4560 Acked 5559 Read 1 Done New Done true
14079 4561 Acked 5560 Read 7 Done New Done true
14080 4562 Acked 5561 Read 9 Done New Done true
14081 4563 Acked 5562 Write 4 Done New Done true
14082 4564 Acked 5563 Write 1 Done New Done true
14083 4565 Acked 5564 Read 1 Done New Done true
14084 4566 Acked 5565 Write 1 Done New Done true
14085 4567 Acked 5566 Write 3 Done New Done true
14086 4568 Acked 5567 Read 10 Done New Done true
14087 4569 Acked 5568 Flush 0 Done New Done true
14088 4570 Acked 5569 Write 8 Done New Done true
14089 4571 Acked 5570 Flush 0 Done New Done true
14090 4572 Acked 5571 Write 5 Done New Done true
14091 4573 Acked 5572 Flush 0 Done New Done true
14092 4574 Acked 5573 Read 1 Done New Done true
14093 4575 Acked 5574 Write 10 Done New Done true
14094 4576 Acked 5575 Read 6 Done New Done true
14095 4577 Acked 5576 Write 4 Done New Done true
14096 4578 Acked 5577 Read 5 Done New Done true
14097 4579 Acked 5578 Write 5 Done New Done true
14098 4580 Acked 5579 Read 4 Done New Done true
14099 4581 Acked 5580 Read 6 Done New Done true
14100 4582 Acked 5581 Write 10 Done New Done true
14101 4583 Acked 5582 Write 8 Done New Done true
14102 4584 Acked 5583 Write 1 Done New Done true
14103 4585 Acked 5584 Flush 0 Done New Done true
14104 4586 Acked 5585 Read 9 Done New Done true
14105 4587 Acked 5586 Read 4 Done New Done true
14106 4588 Acked 5587 Write 1 Done New Done true
14107 4589 Acked 5588 Read 5 Done New Done true
14108 4590 Acked 5589 Read 10 Done New Done true
14109 4591 Acked 5590 Write 7 Done New Done true
14110 4592 Acked 5591 Read 8 Done New Done true
14111 4593 Acked 5592 Read 6 Done New Done true
14112 4594 Acked 5593 Write 1 Done New Done true
14113 4595 Acked 5594 Read 5 Done New Done true
14114 4596 Acked 5595 Flush 0 Done New Done true
14115 4597 Acked 5596 Write 5 Done New Done true
14116 4598 Acked 5597 Read 9 Done New Done true
14117 4599 Acked 5598 Read 4 Done New Done true
14118 4600 Acked 5599 Write 10 Done New Done true
14119 4601 Acked 5600 Read 7 Done New Done true
14120 4602 Acked 5601 Write 7 Done New Done true
14121 4603 Acked 5602 Write 2 Done New Done true
14122 4604 Acked 5603 Read 1 Done New Done true
14123 4605 Acked 5604 Read 8 Done New Done true
14124 4606 Acked 5605 Read 5 Done New Done true
14125 4607 Acked 5606 Flush 0 Done New Done true
14126 4608 Acked 5607 Write 8 Done New Done true
14127 4609 Acked 5608 Write 8 Done New Done true
14128 4610 Acked 5609 Read 3 Done New Done true
14129 4611 Acked 5610 Write 10 Done New Done true
14130 4612 Acked 5611 Read 7 Done New Done true
14131 4613 Acked 5612 Read 9 Done New Done true
14132 4614 Acked 5613 Write 10 Done New Done true
14133 4615 Acked 5614 Write 6 Done New Done true
14134 4616 Acked 5615 Read 3 Done New Done true
14135 4617 Acked 5616 Write 5 Done New Done true
14136 4618 Acked 5617 Read 4 Done New Done true
14137 4619 Acked 5618 Flush 0 Done New Done true
14138 4620 Acked 5619 Write 3 Done New Done true
14139 4621 Acked 5620 Flush 0 Done New Done true
14140 4622 Acked 5621 Write 4 Done New Done true
14141 4623 Acked 5622 Write 3 Done New Done true
14142 4624 Acked 5623 Write 5 Done New Done true
14143 4625 Acked 5624 Read 4 Done New Done true
14144 4626 Acked 5625 Write 7 Done New Done true
14145 4627 Acked 5626 Read 6 Done New Done true
14146 4628 Acked 5627 Write 10 Done New Done true
14147 4629 Acked 5628 Read 6 Done New Done true
14148 4630 Acked 5629 Read 4 Done New Done true
14149 4631 Acked 5630 Write 7 Done New Done true
14150 4632 Acked 5631 Write 5 Done New Done true
14151 4633 Acked 5632 Read 4 Done New Done true
14152 4634 Acked 5633 Read 7 Done New Done true
14153 4635 Acked 5634 Flush 0 Done New Done true
14154 4636 Acked 5635 Read 2 Done New Done true
14155 4637 Acked 5636 Write 7 Done New Done true
14156 4638 Acked 5637 Flush 0 Done New Done true
14157 4639 Acked 5638 Read 5 Done New Done true
14158 4640 Acked 5639 Read 4 Done New Done true
14159 4641 Acked 5640 Read 8 Done New Done true
14160 4642 Acked 5641 Read 2 Done New Done true
14161 4643 Acked 5642 Write 6 Done New Done true
14162 4644 Acked 5643 Read 5 Done New Done true
14163 4645 Acked 5644 Write 7 Done New Done true
14164 4646 Acked 5645 Read 4 Done New Done true
14165 4647 Acked 5646 Write 4 Done New Done true
14166 4648 Acked 5647 Write 6 Done New Done true
14167 4649 Acked 5648 Read 5 Done New Done true
14168 4650 Acked 5649 Write 6 Done New Done true
14169 4651 Acked 5650 Flush 0 Done New Done true
14170 4652 Acked 5651 Read 10 Done New Done true
14171 4653 Acked 5652 Read 5 Done New Done true
14172 4654 Acked 5653 Flush 0 Done New Done true
14173 4655 Acked 5654 Write 4 Done New Done true
14174 4656 Acked 5655 Flush 0 Done New Done true
14175 4657 Acked 5656 Write 4 Done New Done true
14176 4658 Acked 5657 Read 2 Done New Done true
14177 4659 Acked 5658 Flush 0 Done New Done true
14178 4660 Acked 5659 Read 7 Done New Done true
14179 4661 Acked 5660 Read 5 Done New Done true
14180 4662 Acked 5661 Write 4 Done New Done true
14181 4663 Acked 5662 Read 10 Done New Done true
14182 4664 Acked 5663 Read 1 Done New Done true
14183 4665 Acked 5664 Read 2 Done New Done true
14184 4666 Acked 5665 Read 4 Done New Done true
14185 4667 Acked 5666 Write 7 Done New Done true
14186 4668 Acked 5667 Read 5 Done New Done true
14187 4669 Acked 5668 Read 8 Done New Done true
14188 4670 Acked 5669 Write 7 Done New Done true
14189 4671 Acked 5670 Write 9 Done New Done true
14190 4672 Acked 5671 Read 1 Done New Done true
14191 4673 Acked 5672 Read 3 Done New Done true
14192 4674 Acked 5673 Write 5 Done New Done true
14193 4675 Acked 5674 Read 2 Done New Done true
14194 4676 Acked 5675 Write 4 Done New Done true
14195 4677 Acked 5676 Write 8 Done New Done true
14196 4678 Acked 5677 Write 4 Done New Done true
14197 4679 Acked 5678 Write 5 Done New Done true
14198 4680 Acked 5679 Read 5 Done New Done true
14199 4681 Acked 5680 Flush 0 Done New Done true
14200 4682 Acked 5681 Read 4 Done New Done true
14201 4683 Acked 5682 Write 7 Done New Done true
14202 4684 Acked 5683 Read 5 Done New Done true
14203 4685 Acked 5684 Read 7 Done New Done true
14204 4686 Acked 5685 Write 10 Done New Done true
14205 4687 Acked 5686 Write 4 Done New Done true
14206 4688 Acked 5687 Write 3 Done New Done true
14207 4689 Acked 5688 Read 9 Done New Done true
14208 4690 Acked 5689 Write 5 Done New Done true
14209 4691 Acked 5690 Write 4 Done New Done true
14210 4692 Acked 5691 Write 9 Done New Done true
14211 4693 Acked 5692 Flush 0 Done New Done true
14212 4694 Acked 5693 Flush 0 Done New Done true
14213 4695 Acked 5694 Read 9 Done New Done true
14214 4696 Acked 5695 Read 10 Done New Done true
14215 4697 Acked 5696 Write 9 Done New Done true
14216 4698 Acked 5697 Read 6 Done New Done true
14217 4699 Acked 5698 Write 1 Done New Done true
14218 4700 Acked 5699 Read 8 Done New Done true
14219 4701 Acked 5700 Read 8 Done New Done true
14220 4702 Acked 5701 Flush 0 Done New Done true
14221 4703 Acked 5702 Read 6 Done New Done true
14222 4704 Acked 5703 Read 10 Done New Done true
14223 4705 Acked 5704 Read 3 Done New Done true
14224 4706 Acked 5705 Read 10 Done New Done true
14225 4707 Acked 5706 Write 4 Done New Done true
14226 4708 Acked 5707 Read 6 Done New Done true
14227 4709 Acked 5708 Write 10 Done New Done true
14228 4710 Acked 5709 Flush 0 Done New Done true
14229 4711 Acked 5710 Read 10 Done New Done true
14230 4712 Acked 5711 Read 9 Done New Done true
14231 4713 Acked 5712 Read 1 Done New Done true
14232 4714 Acked 5713 Read 7 Done New Done true
14233 4715 Acked 5714 Write 1 Done New Done true
14234 4716 Acked 5715 Flush 0 Done New Done true
14235 4717 Acked 5716 Read 1 Done New Done true
14236 4718 Acked 5717 Write 9 Done New Done true
14237 4719 Acked 5718 Write 7 Done New Done true
14238 4720 Acked 5719 Read 1 Done New Done true
14239 4721 Acked 5720 Read 1 Done New Done true
14240 4722 Acked 5721 Read 2 Done New Done true
14241 4723 Acked 5722 Read 1 Done New Done true
14242 4724 Acked 5723 Read 9 Done New Done true
14243 4725 Acked 5724 Read 2 Done New Done true
14244 4726 Acked 5725 Read 7 Done New Done true
14245 4727 Acked 5726 Read 10 Done New Done true
14246 4728 Acked 5727 Write 6 Done New Done true
14247 4729 Acked 5728 Write 8 Done New Done true
14248 4730 Acked 5729 Write 5 Done New Done true
14249 4731 Acked 5730 Write 2 Done New Done true
14250 4732 Acked 5731 Read 4 Done New Done true
14251 4733 Acked 5732 Read 7 Done New Done true
14252 4734 Acked 5733 Flush 0 Done New Done true
14253 4735 Acked 5734 Read 10 Done New Done true
14254 4736 Acked 5735 Read 4 Done New Done true
14255 4737 Acked 5736 Write 9 Done New Done true
14256 4738 Acked 5737 Read 3 Done New Done true
14257 4739 Acked 5738 Write 8 Done New Done true
14258 4740 Acked 5739 Write 8 Done New Done true
14259 4741 Acked 5740 Write 9 Done New Done true
14260 4742 Acked 5741 Write 5 Done New Done true
14261 4743 Acked 5742 Read 4 Done New Done true
14262 4744 Acked 5743 Read 2 Done New Done true
14263 4745 Acked 5744 Read 5 Done New Done true
14264 4746 Acked 5745 Read 10 Done New Done true
14265 4747 Acked 5746 Write 4 Done New Done true
14266 4748 Acked 5747 Read 7 Done New Done true
14267 4749 Acked 5748 Write 5 Done New Done true
14268 4750 Acked 5749 Write 4 Done New Done true
14269 4751 Acked 5750 Write 2 Done New Done true
14270 4752 Acked 5751 Read 1 Done New Done true
14271 4753 Acked 5752 Read 9 Done New Done true
14272 4754 Acked 5753 Read 10 Done New Done true
14273 4755 Acked 5754 Write 4 Done New Done true
14274 4756 Acked 5755 Read 5 Done New Done true
14275 4757 Acked 5756 Read 10 Done New Done true
14276 4758 Acked 5757 Write 2 Done New Done true
14277 4759 Acked 5758 Write 8 Done New Done true
14278 4760 Acked 5759 Read 2 Done New Done true
14279 4761 Acked 5760 Read 1 Done New Done true
14280 4762 Acked 5761 Write 2 Done New Done true
14281 4763 Acked 5762 Read 4 Done New Done true
14282 4764 Acked 5763 Write 8 Done New Done true
14283 4765 Acked 5764 Write 10 Done New Done true
14284 4766 Acked 5765 Read 5 Done New Done true
14285 4767 Acked 5766 Write 4 Done New Done true
14286 4768 Acked 5767 Read 5 Done New Done true
14287 4769 Acked 5768 Read 9 Done New Done true
14288 4770 Acked 5769 Write 4 Done New Done true
14289 4771 Acked 5770 Read 3 Done New Done true
14290 4772 Acked 5771 Read 9 Done New Done true
14291 4773 Acked 5772 Read 8 Done New Done true
14292 4774 Acked 5773 Read 7 Done New Done true
14293 4775 Acked 5774 Write 3 Done New Done true
14294 4776 Acked 5775 Read 2 Done New Done true
14295 4777 Acked 5776 Write 7 Done New Done true
14296 4778 Acked 5777 Read 4 Done New Done true
14297 4779 Acked 5778 Read 10 Done New Done true
14298 4780 Acked 5779 Read 3 Done New Done true
14299 4781 Acked 5780 Write 2 Done New Done true
14300 4782 Acked 5781 Read 8 Done New Done true
14301 4783 Acked 5782 Read 3 Done New Done true
14302 4784 Acked 5783 Write 9 Done New Done true
14303 4785 Acked 5784 Write 8 Done New Done true
14304 4786 Acked 5785 Write 1 Done New Done true
14305 4787 Acked 5786 Write 9 Done New Done true
14306 4788 Acked 5787 Write 2 Done New Done true
14307 4789 Acked 5788 Read 7 Done New Done true
14308 4790 Acked 5789 Write 3 Done New Done true
14309 4791 Acked 5790 Write 6 Done New Done true
14310 4792 Acked 5791 Read 4 Done New Done true
14311 4793 Acked 5792 Read 2 Done New Done true
14312 4794 Acked 5793 Write 10 Done New Done true
14313 4795 Acked 5794 Write 3 Done New Done true
14314 4796 Acked 5795 Write 2 Done New Done true
14315 4797 Acked 5796 Write 10 Done New Done true
14316 4798 Acked 5797 Read 10 Done New Done true
14317 4799 Acked 5798 Read 10 Done New Done true
14318 4800 Acked 5799 Read 2 Done New Done true
14319 4801 Acked 5800 Write 5 Done New Done true
14320 4802 Acked 5801 Flush 0 Done New Done true
14321 4803 Acked 5802 Write 9 Done New Done true
14322 4804 Acked 5803 Write 5 Done New Done true
14323 4805 Acked 5804 Read 9 Done New Done true
14324 4806 Acked 5805 Flush 0 Done New Done true
14325 4807 Acked 5806 Flush 0 Done New Done true
14326 4808 Acked 5807 Write 5 Done New Done true
14327 4809 Acked 5808 Read 6 Done New Done true
14328 4810 Acked 5809 Read 6 Done New Done true
14329 4811 Acked 5810 Read 2 Done New Done true
14330 4812 Acked 5811 Write 10 Done New Done true
14331 4813 Acked 5812 Write 3 Done New Done true
14332 4814 Acked 5813 Read 2 Done New Done true
14333 4815 Acked 5814 Read 1 Done New Done true
14334 4816 Acked 5815 Flush 0 Done New Done true
14335 4817 Acked 5816 Write 9 Done New Done true
14336 4818 Acked 5817 Flush 0 Done New Done true
14337 4819 Acked 5818 Read 5 Done New Done true
14338 4820 Acked 5819 Read 3 Done New Done true
14339 4821 Acked 5820 Write 6 Done New Done true
14340 4822 Acked 5821 Flush 0 Done New Done true
14341 STATES DS:0 DS:1 DS:2 TOTAL
14342 New 0 302 0 302
14343 Sent 0 0 0 0
14344 Done 302 0 302 604
14345 Skipped 0 0 0 0
14346 Error 0 0 0 0
14347 Last Flush: 5821 5519 5821
14348 Downstairs last five completed: 5519 5518 5517 5516 5515
14349 Upstairs last five completed: 4822 4821 4820 4819 4818
14350 CLIENT: Up:0 ds:302 act:2
143512024-02-28T22:16:05.515ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143522024-02-28T22:16:05.515ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143532024-02-28T22:16:05.515ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143542024-02-28T22:16:05.515ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143552024-02-28T22:16:05.515ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143562024-02-28T22:16:05.515ZINFOcrucible: [1] client re-new 302 jobs since flush 5519 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143572024-02-28T22:16:05.516ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
14358 [8820] got start action
14359 [8820] I am going to restart
14360 Starting downstairs at port 8820
14361 Make output file at "/tmp/dsc/downstairs-8820.txt"
14362 [8820][1] reports Starting
143632024-02-28T22:16:06.516ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143642024-02-28T22:16:06.516ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143652024-02-28T22:16:06.516ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143662024-02-28T22:16:06.516ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143672024-02-28T22:16:06.516ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143682024-02-28T22:16:06.516ZINFOcrucible: [1] client re-new 302 jobs since flush 5519 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143692024-02-28T22:16:06.516ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
14370 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1180)
14371 [8820][1] reports Running
143722024-02-28T22:16:07.517ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143732024-02-28T22:16:07.517ZINFOcrucible: ds_connection connected = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143742024-02-28T22:16:07.519ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143752024-02-28T22:16:07.519ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143762024-02-28T22:16:07.519ZINFOcrucible: send last flush ID to this DS: 5519 = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143772024-02-28T22:16:07.519ZINFOcrucible: Replied this last flush ID: 5519 = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143782024-02-28T22:16:07.519ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143792024-02-28T22:16:07.519ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143802024-02-28T22:16:07.519ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143812024-02-28T22:16:07.519ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
14382 ----------------------------------------------------------------
14383 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
14384 Downstairs last five completed: 5821 5820 5819 5818 5817
14385 Upstairs last five completed: 4822 4821 4820 4819 4818
14386 CLIENT: Up:0 ds:0 act:3
14387 Replay: All jobs finished, all DS active.
14388 Main task has work to do, go find it
14389 got dsc Stop(2)
14390 stop 2
14391 Feb 28 22:16:09.287 INFO request completed[8830] Got stop action so:false kr:false
14392 , latency_us: 304, response_code: 204, uri: /stop/cid/2, method: POST, req_id: a5a40121-ddfb-424e-a0c3-ff6f8f022231, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
143932024-02-28T22:16:09.288ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143942024-02-28T22:16:09.288ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143952024-02-28T22:16:09.288ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
143962024-02-28T22:16:09.288ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
14397 Feb 28 22:16:09.288 INFO request completed, latency_us: 120, response_code: {"msg":"[2] client re-new 0 jobs since flush 5821","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:16:09.288807395Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
14398 200, uri: /state/cid/2, method: GET, req_id: b09c1c2d-9736-425d-841a-e5e94391aec3{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:16:09.288853513Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
14399 , remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
14400 [8830] Exited with: ExitStatus(unix_wait_status(9))
14401 [8830][2] reports Exit
144022024-02-28T22:16:10.289ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144032024-02-28T22:16:10.289ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144042024-02-28T22:16:10.289ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144052024-02-28T22:16:10.289ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144062024-02-28T22:16:10.289ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144072024-02-28T22:16:10.289ZINFOcrucible: [2] client re-new 0 jobs since flush 5821 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144082024-02-28T22:16:10.289ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144092024-02-28T22:16:11.290ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144102024-02-28T22:16:11.290ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144112024-02-28T22:16:11.290ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144122024-02-28T22:16:11.290ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144132024-02-28T22:16:11.290ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144142024-02-28T22:16:11.290ZINFOcrucible: [2] client re-new 0 jobs since flush 5821 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144152024-02-28T22:16:11.290ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144162024-02-28T22:16:12.291ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144172024-02-28T22:16:12.291ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144182024-02-28T22:16:12.291ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144192024-02-28T22:16:12.291ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144202024-02-28T22:16:12.291ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144212024-02-28T22:16:12.291ZINFOcrucible: [2] client re-new 0 jobs since flush 5821 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144222024-02-28T22:16:12.291ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
14423 Feb 28 22:16:13.290 INFO request completed, latency_us: 235, response_code: 200, uri: /state/cid/2, method: GET, req_id: b1ea4172-37dc-4012-b929-f2be70b7f3b2, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
14424 001/300 Read block 4364 len 12288
144252024-02-28T22:16:13.292ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144262024-02-28T22:16:13.292ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144272024-02-28T22:16:13.292ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144282024-02-28T22:16:13.292ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144292024-02-28T22:16:13.292ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144302024-02-28T22:16:13.292ZINFOcrucible: [2] client re-new 1 jobs since flush 5821 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
144312024-02-28T22:16:13.292ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
14432 002/300 Flush
14433 003/300 Write block 3572 len 40960 data: 6 7 5 4 4 4 4 3 3 4
14434 004/300 Read block 1693 len 24576
14435 005/300 Read block 4114 len 28672
14436 006/300 Read block 3739 len 32768
14437 007/300 Read block 1276 len 28672
14438 008/300 Read block 4076 len 28672
14439 009/300 Read block 2786 len 16384
14440 010/300 Write block 3276 len 28672 data: 5 4 4 4 5 5 5
14441 011/300 Read block 4004 len 28672
14442 012/300 Write block 2381 len 4096 data: 3
14443 013/300 Write block 2732 len 4096 data: 4
14444 014/300 Write block 1176 len 40960 data: 3 3 3 3 3 3 3 2 3 3
14445 015/300 Read block 1239 len 12288
14446 016/300 Flush
14447 017/300 Write block 3288 len 20480 data: 6 6 6 7 6
14448 018/300 Read block 4168 len 36864
14449 019/300 Write block 1415 len 32768 data: 5 5 6 6 7 6 5 5
14450 020/300 Write block 1796 len 32768 data: 5 5 6 5 5 5 5 6
14451 021/300 Read block 3276 len 4096
14452 022/300 Write block 3631 len 4096 data: 4
14453 023/300 Flush
14454 024/300 Flush
14455 025/300 Write block 3256 len 4096 data: 4
14456 026/300 Write block 2177 len 16384 data: 3 4 4 4
14457 027/300 Write block 4104 len 32768 data: 5 4 5 5 4 3 3 3
14458 028/300 Read block 3782 len 36864
14459 029/300 Flush
14460 030/300 Read block 751 len 16384
14461 031/300 Read block 4239 len 16384
14462 032/300 Write block 2316 len 40960 data: 6 5 6 4 4 4 4 4 4 5
14463 033/300 Write block 1783 len 20480 data: 6 5 3 3 2
14464 034/300 Flush
14465 035/300 Flush
14466 036/300 Write block 3071 len 24576 data: 4 4 4 4 4 4
14467 037/300 Read block 507 len 28672
14468 038/300 Read block 1693 len 40960
14469 039/300 Write block 2596 len 20480 data: 4 4 4 4 4
14470 040/300 Write block 3116 len 28672 data: 4 3 4 4 5 5 7
14471 041/300 Read block 2160 len 40960
14472 042/300 Read block 292 len 12288
14473 043/300 Write block 3903 len 32768 data: 5 5 5 4 6 5 5 4
14474 044/300 Write block 730 len 40960 data: 3 3 4 4 5 5 3 3 3 3
14475 045/300 Read block 1983 len 28672
14476 046/300 Write block 538 len 32768 data: 5 5 5 4 4 4 3 4
14477 047/300 Write block 4166 len 4096 data: 5
14478 048/300 Read block 1750 len 40960
14479 049/300 Write block 4766 len 40960 data: 3 3 4 5 5 4 3 4 4 3
14480 050/300 Read block 4434 len 36864
14481 051/300 Write block 1914 len 32768 data: 3 3 3 3 3 3 3 2
14482 052/300 Read block 3822 len 32768
14483 053/300 Write block 1809 len 24576 data: 2 2 2 2 2 2
14484 054/300 Read block 3316 len 4096
14485 055/300 Read block 4567 len 12288
14486 056/300 Write block 1295 len 8192 data: 4 4
14487 057/300 Write block 866 len 32768 data: 3 3 3 3 5 4 3 3
14488 058/300 Read block 2745 len 36864
14489 059/300 Flush
14490 060/300 Flush
14491 061/300 Read block 1833 len 40960
14492 062/300 Flush
14493 063/300 Read block 3581 len 16384
14494 064/300 Write block 2551 len 32768 data: 4 4 4 4 3 3 3 3
14495 065/300 Write block 3176 len 40960 data: 4 5 7 7 6 5 5 3 3 3
14496 066/300 Read block 1120 len 12288
14497 067/300 Write block 1782 len 8192 data: 5 7
14498 068/300 Write block 2106 len 20480 data: 2 2 2 4 4
14499 069/300 Read block 430 len 4096
14500 070/300 Write block 2062 len 20480 data: 4 4 3 4 6
14501 071/300 Flush
14502 072/300 Write block 1830 len 12288 data: 6 5 4
14503 073/300 Flush
14504 074/300 Write block 2346 len 40960 data: 2 2 4 3 3 3 3 3 2 2
14505 075/300 Write block 877 len 20480 data: 5 4 4 4 3
14506 076/300 Write block 111 len 4096 data: 3
14507 077/300 Write block 505 len 24576 data: 5 6 6 7 7 5
14508 078/300 Flush
14509 079/300 Write block 4691 len 32768 data: 3 4 4 4 4 4 4 3
14510 080/300 Write block 4538 len 40960 data: 4 5 5 5 4 4 4 4 3 4
14511 081/300 Write block 1295 len 28672 data: 5 5 4 3 3 3 3
14512 082/300 Read block 777 len 4096
14513 083/300 Write block 2095 len 20480 data: 6 6 6 5 7
14514 084/300 Write block 1982 len 24576 data: 6 6 6 6 6 6
14515 085/300 Write block 194 len 12288 data: 3 2 3
14516 086/300 Read block 3707 len 32768
14517 087/300 Read block 2282 len 20480
14518 088/300 Write block 4563 len 12288 data: 4 5 6
14519 089/300 Write block 1423 len 20480 data: 6 6 4 4 4
14520 090/300 Read block 2003 len 28672
14521 091/300 Write block 775 len 16384 data: 4 5 4 5
14522 092/300 Read block 4040 len 20480
14523 093/300 Read block 554 len 8192
14524 094/300 Read block 4295 len 12288
14525 095/300 Read block 2274 len 36864
14526 096/300 Write block 3459 len 12288 data: 2 2 3
14527 097/300 Read block 3880 len 40960
14528 098/300 Write block 1513 len 16384 data: 4 2 4 5
14529 099/300 Read block 885 len 20480
14530 100/300 Read block 4578 len 16384
14531 101/300 Write block 2986 len 12288 data: 7 7 7
14532 102/300 Read block 2351 len 40960
14533 103/300 Write block 21 len 16384 data: 3 3 3 4
14534 104/300 Read block 2133 len 36864
14535 105/300 Read block 3525 len 40960
14536 106/300 Flush
14537 107/300 Write block 4819 len 24576 data: 4 4 4 3 3 3
14538 108/300 Write block 405 len 12288 data: 3 3 3
14539 109/300 Write block 2650 len 8192 data: 5 5
14540 110/300 Write block 403 len 24576 data: 3 3 4 4 4 4
14541 111/300 Flush
14542 112/300 Write block 21 len 16384 data: 4 4 4 5
14543 113/300 Flush
14544 114/300 Read block 211 len 4096
14545 115/300 Read block 2336 len 4096
14546 116/300 Read block 3000 len 36864
14547 117/300 Write block 3407 len 24576 data: 7 7 6 5 5 3
14548 118/300 Read block 1045 len 16384
14549 119/300 Flush
14550 120/300 Read block 1491 len 32768
14551 121/300 Read block 2833 len 24576
14552 122/300 Write block 214 len 8192 data: 3 2
14553 123/300 Write block 2267 len 24576 data: 5 4 4 4 4 3
14554 124/300 Read block 2732 len 20480
14555 125/300 Read block 3296 len 24576
14556 126/300 Read block 4843 len 12288
14557 127/300 Read block 3697 len 4096
14558 128/300 Write block 4953 len 12288 data: 3 2 2
14559 129/300 Write block 3497 len 16384 data: 2 2 3 4
14560 130/300 Read block 3370 len 20480
14561 131/300 Read block 3340 len 28672
14562 132/300 Read block 3615 len 4096
14563 133/300 Write block 4012 len 28672 data: 7 8 9 8 7 5 5
14564 134/300 Read block 1461 len 12288
14565 135/300 Read block 4707 len 24576
14566 136/300 Read block 2958 len 20480
14567 137/300 Write block 582 len 36864 data: 3 3 3 2 2 2 3 3 3
14568 138/300 Read block 2373 len 36864
14569 139/300 Read block 3657 len 8192
14570 140/300 Read block 4400 len 12288
14571 141/300 Write block 1855 len 20480 data: 3 4 4 5 5
14572 142/300 Read block 2548 len 32768
14573 143/300 Write block 4079 len 16384 data: 6 5 3 2
14574 144/300 Read block 249 len 4096
14575 145/300 Write block 4730 len 12288 data: 3 3 3
14576 146/300 Read block 4083 len 4096
14577 147/300 Read block 462 len 32768
14578 148/300 Write block 1033 len 16384 data: 4 4 4 5
14579 149/300 Write block 3047 len 20480 data: 4 4 4 4 4
14580 150/300 Read block 4590 len 4096
14581 151/300 Read block 2799 len 8192
14582 152/300 Read block 4121 len 4096
14583 153/300 Read block 1197 len 20480
14584 154/300 Read block 4482 len 8192
14585 155/300 Write block 1682 len 40960 data: 2 2 4 4 5 5 6 6 6 4
14586 156/300 Read block 3609 len 16384
14587 157/300 Write block 2417 len 20480 data: 7 6 6 5 5
14588 158/300 Write block 3575 len 8192 data: 5 5
14589 159/300 Read block 4614 len 12288
14590 160/300 Read block 2547 len 32768
14591 161/300 Read block 4420 len 12288
14592 162/300 Read block 2156 len 8192
14593 163/300 Read block 3173 len 8192
14594 164/300 Write block 4727 len 32768 data: 3 3 3 4 4 4 3 5
14595 165/300 Read block 2251 len 24576
14596 166/300 Write block 4311 len 40960 data: 4 3 4 4 4 3 3 5 4 5
14597 167/300 Write block 2718 len 32768 data: 6 6 6 7 6 5 5 5
14598 168/300 Read block 4818 len 8192
14599 169/300 Read block 2266 len 36864
14600 170/300 Write block 4205 len 40960 data: 3 3 3 2 3 3 3 3 3 2
14601 171/300 Write block 4055 len 16384 data: 2 3 3 3
14602 172/300 Read block 2719 len 32768
14603 173/300 Read block 444 len 16384
14604 174/300 Flush
14605 175/300 Write block 2961 len 20480 data: 4 3 4 4 3
14606 176/300 Read block 3056 len 32768
14607 177/300 Write block 423 len 12288 data: 7 6 6
14608 178/300 Write block 4330 len 36864 data: 6 6 7 6 6 9 7 4 4
14609 179/300 Read block 2579 len 8192
14610 180/300 Write block 920 len 8192 data: 7 6
14611 181/300 Read block 2083 len 32768
14612 182/300 Read block 1623 len 28672
14613 183/300 Write block 2569 len 24576 data: 3 3 4 3 3 3
14614 184/300 Read block 4365 len 28672
14615 185/300 Write block 2644 len 12288 data: 2 2 3
14616 186/300 Read block 2819 len 24576
14617 187/300 Write block 1665 len 24576 data: 3 3 3 2 3 4
14618 188/300 Write block 3855 len 20480 data: 5 4 5 5 3
14619 189/300 Read block 4915 len 32768
14620 190/300 Read block 167 len 12288
14621 191/300 Read block 4528 len 4096
14622 192/300 Read block 2659 len 32768
14623 193/300 Flush
14624 194/300 Read block 2189 len 28672
14625 195/300 Read block 4851 len 8192
14626 196/300 Flush
14627 197/300 Read block 3017 len 16384
14628 198/300 Read block 3080 len 4096
14629 199/300 Read block 1976 len 16384
14630 200/300 Read block 4045 len 36864
14631 201/300 Read block 4259 len 40960
14632 202/300 Read block 1915 len 28672
14633 203/300 Flush
14634 204/300 Read block 2692 len 16384
14635 205/300 Read block 1460 len 4096
14636 206/300 Write block 4724 len 36864 data: 3 4 3 4 4 4 5 5 5
14637 207/300 Write block 4245 len 40960 data: 4 3 3 4 4 4 5 6 6 8
14638 208/300 Flush
14639 209/300 Read block 544 len 28672
14640 210/300 Read block 4243 len 8192
14641 211/300 Write block 2866 len 32768 data: 4 4 3 4 4 4 6 5
14642 212/300 Read block 1298 len 20480
14643 213/300 Read block 2665 len 4096
14644 214/300 Write block 2080 len 36864 data: 2 3 3 3 4 3 2 2 3
14645 215/300 Read block 2708 len 36864
14646 216/300 Flush
14647 217/300 Read block 2036 len 36864
14648 218/300 Read block 829 len 12288
14649 219/300 Write block 2947 len 12288 data: 3 3 3
14650 220/300 Write block 492 len 8192 data: 4 5
14651 221/300 Read block 1460 len 28672
14652 222/300 Read block 2580 len 28672
14653 223/300 Read block 3541 len 40960
14654 224/300 Read block 545 len 24576
14655 225/300 Flush
14656 226/300 Write block 2729 len 24576 data: 4 4 3 5 3 3
14657 227/300 Write block 719 len 16384 data: 5 4 4 3
14658 228/300 Read block 1498 len 20480
14659 229/300 Read block 4566 len 40960
14660 230/300 Read block 2808 len 4096
14661 231/300 Flush
14662 232/300 Write block 3873 len 32768 data: 4 4 4 6 6 4 5 5
14663 233/300 Read block 2422 len 12288
14664 234/300 Write block 688 len 16384 data: 4 3 2 2
14665 235/300 Write block 366 len 40960 data: 2 2 2 2 2 2 3 3 3 3
14666 236/300 Read block 1983 len 40960
14667 237/300 Flush
14668 238/300 Read block 3933 len 40960
14669 239/300 Read block 59 len 20480
14670 240/300 Write block 4762 len 40960 data: 4 5 4 3 4 4 5 6 6 5
14671 241/300 Read block 380 len 12288
14672 242/300 Write block 2682 len 12288 data: 6 6 6
14673 243/300 Write block 2539 len 24576 data: 5 5 6 6 4 5
14674 244/300 Write block 1914 len 8192 data: 4 4
14675 245/300 Read block 3957 len 12288
14676 246/300 Write block 4879 len 4096 data: 4
14677 247/300 Read block 3335 len 20480
14678 248/300 Write block 800 len 8192 data: 3 3
14679 249/300 Read block 4988 len 4096
14680 250/300 Write block 2695 len 28672 data: 3 3 3 3 3 3 3
14681 251/300 Write block 1397 len 40960 data: 4 3 4 4 3 3 5 6 6 5
14682 252/300 Write block 884 len 24576 data: 3 3 4 3 4 3
14683 253/300 Read block 1428 len 28672
14684 254/300 Read block 4119 len 36864
14685 255/300 Read block 2388 len 4096
14686 256/300 Read block 1208 len 28672
14687 257/300 Write block 59 len 16384 data: 5 6 6 6
14688 258/300 Read block 4089 len 40960
14689 259/300 Read block 1028 len 12288
14690 260/300 Write block 1729 len 4096 data: 4
14691 261/300 Read block 1000 len 16384
14692 262/300 Read block 3914 len 16384
14693 263/300 Read block 208 len 40960
14694 264/300 Write block 4243 len 8192 data: 3 4
14695 265/300 Write block 1268 len 8192 data: 5 4
14696 266/300 Write block 4626 len 20480 data: 6 5 4 5 4
14697 267/300 Read block 545 len 24576
14698 268/300 Write block 2582 len 32768 data: 5 4 4 5 2 2 3 3
14699 269/300 Write block 724 len 36864 data: 2 2 3 3 3 3 4 4 5
14700 270/300 Read block 1065 len 20480
14701 271/300 Flush
14702 272/300 Flush
14703 273/300 Read block 3322 len 20480
14704 274/300 Read block 1668 len 8192
14705 275/300 Write block 3302 len 40960 data: 3 3 3 4 4 3 3 3 3 3
14706 276/300 Read block 1151 len 40960
14707 277/300 Write block 3266 len 36864 data: 5 5 3 3 3 3 4 4 6
14708 278/300 Write block 2752 len 16384 data: 2 2 2 2
14709 279/300 Write block 1671 len 20480 data: 3 3 4 4 4
14710 280/300 Write block 3514 len 36864 data: 5 5 6 5 5 4 5 5 3
14711 281/300 Write block 1085 len 28672 data: 6 5 5 5 4 3 4
14712 282/300 Write block 534 len 36864 data: 4 4 4 4 6 6 6 5 5
14713 283/300 Write block 779 len 40960 data: 4 4 4 4 4 4 4 4 4 3
14714 284/300 Read block 1420 len 20480
14715 285/300 Read block 3879 len 36864
14716 286/300 Write block 770 len 4096 data: 4
14717 287/300 Write block 140 len 28672 data: 6 6 5 5 5 5 6
14718 288/300 Write block 1920 len 32768 data: 4 3 2 2 3 3 3 4
14719 289/300 Flush
14720 290/300 Read block 4967 len 4096
14721 291/300 Read block 1880 len 36864
14722 292/300 Read block 1003 len 12288
14723 293/300 Read block 3015 len 16384
14724 294/300 Write block 4149 len 40960 data: 5 4 4 4 4 4 4 4 3 3
14725 295/300 Flush
14726 296/300 Read block 155 len 8192
14727 297/300 Read block 4008 len 32768
14728 298/300 Read block 1715 len 8192
14729 299/300 Write block 4701 len 24576 data: 5 4 3 3 2 2
14730 300/300 Write block 2577 len 24576 data: 3 3 5 5 5 6
14731 Main task has work to do, go find it
14732 got dsc Start(2)
14733 start 2
14734 Feb 28 22:16:14.000 INFO request completed, latency_us: 217, response_code: 204, uri: /start/cid/2, method: POST, req_id: 2f21291d-f8ca-49d5-961a-5a294d83f325, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
14735 Replay: started 2, returned:Ok(())
14736 ----------------------------------------------------------------
14737 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
14738 States: Active Active Offline
14739 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
14740 4823 Acked 5822 Read 3 Done Done New true
14741 4824 Acked 5823 Flush 0 Done Done New false
14742 4825 Acked 5824 Write 10 Done Done New false
14743 4826 Acked 5825 Read 6 Done Done New false
14744 4827 Acked 5826 Read 7 Done Done New false
14745 4828 Acked 5827 Read 8 Done Done New false
14746 4829 Acked 5828 Read 7 Done Done New false
14747 4830 Acked 5829 Read 7 Done Done New false
14748 4831 Acked 5830 Read 4 Done Done New false
14749 4832 Acked 5831 Write 7 Done Done New false
14750 4833 Acked 5832 Read 7 Done Done New false
14751 4834 Acked 5833 Write 1 Done Done New false
14752 4835 Acked 5834 Write 1 Done Done New false
14753 4836 Acked 5835 Write 10 Done Done New false
14754 4837 Acked 5836 Read 3 Done Done New false
14755 4838 Acked 5837 Flush 0 Done Done New false
14756 4839 Acked 5838 Write 5 Done Done New false
14757 4840 Acked 5839 Read 9 Done Done New false
14758 4841 Acked 5840 Write 8 Done Done New false
14759 4842 Acked 5841 Write 8 Done Done New false
14760 4843 Acked 5842 Read 1 Done Done New false
14761 4844 Acked 5843 Write 1 Done Done New false
14762 4845 Acked 5844 Flush 0 Done Done New false
14763 4846 Acked 5845 Flush 0 Done Done New false
14764 4847 Acked 5846 Write 1 Done Done New false
14765 4848 Acked 5847 Write 4 Done Done New false
14766 4849 Acked 5848 Write 8 Done Done New false
14767 4850 Acked 5849 Read 9 Done Done New false
14768 4851 Acked 5850 Flush 0 Done Done New false
14769 4852 Acked 5851 Read 4 Done Done New false
14770 4853 Acked 5852 Read 4 Done Done New false
14771 4854 Acked 5853 Write 10 Done Done New false
14772 4855 Acked 5854 Write 5 Done Done New false
14773 4856 Acked 5855 Flush 0 Done Done New false
14774 4857 Acked 5856 Flush 0 Done Done New false
14775 4858 Acked 5857 Write 6 Done Done New false
14776 4859 Acked 5858 Read 7 Done Done New false
14777 4860 Acked 5859 Read 10 Done Done New false
14778 4861 Acked 5860 Write 5 Done Done New false
14779 4862 Acked 5861 Write 7 Done Done New false
14780 4863 Acked 5862 Read 10 Done Done New false
14781 4864 Acked 5863 Read 3 Done Done New false
14782 4865 Acked 5864 Write 8 Done Done New false
14783 4866 Acked 5865 Write 10 Done Done New false
14784 4867 Acked 5866 Read 7 Done Done New false
14785 4868 Acked 5867 Write 8 Done Done New false
14786 4869 Acked 5868 Write 1 Done Done New false
14787 4870 Acked 5869 Read 10 Done Done New false
14788 4871 Acked 5870 Write 10 Done Done New false
14789 4872 Acked 5871 Read 9 Done Done New false
14790 4873 Acked 5872 Write 8 Done Done New false
14791 4874 Acked 5873 Read 8 Done Done New false
14792 4875 Acked 5874 Write 6 Done Done New false
14793 4876 Acked 5875 Read 1 Done Done New false
14794 4877 Acked 5876 Read 3 Done Done New false
14795 4878 Acked 5877 Write 2 Done Done New false
14796 4879 Acked 5878 Write 8 Done Done New false
14797 4880 Acked 5879 Read 9 Done Done New false
14798 4881 Acked 5880 Flush 0 Done Done New false
14799 4882 Acked 5881 Flush 0 Done Done New false
14800 4883 Acked 5882 Read 10 Done Done New false
14801 4884 Acked 5883 Flush 0 Done Done New false
14802 4885 Acked 5884 Read 4 Done Done New false
14803 4886 Acked 5885 Write 8 Done Done New false
14804 4887 Acked 5886 Write 10 Done Done New false
14805 4888 Acked 5887 Read 3 Done Done New false
14806 4889 Acked 5888 Write 2 Done Done New false
14807 4890 Acked 5889 Write 5 Done Done New false
14808 4891 Acked 5890 Read 1 Done Done New false
14809 4892 Acked 5891 Write 5 Done Done New false
14810 4893 Acked 5892 Flush 0 Done Done New false
14811 4894 Acked 5893 Write 3 Done Done New false
14812 4895 Acked 5894 Flush 0 Done Done New false
14813 4896 Acked 5895 Write 10 Done Done New false
14814 4897 Acked 5896 Write 5 Done Done New false
14815 4898 Acked 5897 Write 1 Done Done New false
14816 4899 Acked 5898 Write 6 Done Done New false
14817 4900 Acked 5899 Flush 0 Done Done New false
14818 4901 Acked 5900 Write 8 Done Done New false
14819 4902 Acked 5901 Write 10 Done Done New false
14820 4903 Acked 5902 Write 7 Done Done New false
14821 4904 Acked 5903 Read 1 Done Done New false
14822 4905 Acked 5904 Write 5 Done Done New false
14823 4906 Acked 5905 Write 6 Done Done New false
14824 4907 Acked 5906 Write 3 Done Done New false
14825 4908 Acked 5907 Read 8 Done Done New false
14826 4909 Acked 5908 Read 5 Done Done New false
14827 4910 Acked 5909 Write 3 Done Done New false
14828 4911 Acked 5910 Write 5 Done Done New false
14829 4912 Acked 5911 Read 7 Done Done New false
14830 4913 Acked 5912 Write 4 Done Done New false
14831 4914 Acked 5913 Read 5 Done Done New false
14832 4915 Acked 5914 Read 2 Done Done New false
14833 4916 Acked 5915 Read 3 Done Done New false
14834 4917 Acked 5916 Read 9 Done Done New false
14835 4918 Acked 5917 Write 3 Done Done New false
14836 4919 Acked 5918 Read 10 Done Done New false
14837 4920 Acked 5919 Write 4 Done Done New false
14838 4921 Acked 5920 Read 5 Done Done New false
14839 4922 Acked 5921 Read 4 Done Done New false
14840 4923 Acked 5922 Write 3 Done Done New false
14841 4924 Acked 5923 Read 10 Done Done New false
14842 4925 Acked 5924 Write 4 Done Done New false
14843 4926 Acked 5925 Read 9 Done Done New false
14844 4927 Acked 5926 Read 10 Done Done New false
14845 4928 Acked 5927 Flush 0 Done Done New false
14846 4929 Acked 5928 Write 6 Done Done New false
14847 4930 Acked 5929 Write 3 Done Done New false
14848 4931 Acked 5930 Write 2 Done Done New false
14849 4932 Acked 5931 Write 6 Done Done New false
14850 4933 Acked 5932 Flush 0 Done Done New false
14851 4934 Acked 5933 Write 4 Done Done New false
14852 4935 Acked 5934 Flush 0 Done Done New false
14853 4936 Acked 5935 Read 1 Done Done New false
14854 4937 Acked 5936 Read 1 Done Done New false
14855 4938 Acked 5937 Read 9 Done Done New false
14856 4939 Acked 5938 Write 6 Done Done New false
14857 4940 Acked 5939 Read 4 Done Done New false
14858 4941 Acked 5940 Flush 0 Done Done New false
14859 4942 Acked 5941 Read 8 Done Done New false
14860 4943 Acked 5942 Read 6 Done Done New false
14861 4944 Acked 5943 Write 2 Done Done New false
14862 4945 Acked 5944 Write 6 Done Done New false
14863 4946 Acked 5945 Read 5 Done Done New false
14864 4947 Acked 5946 Read 6 Done Done New false
14865 4948 Acked 5947 Read 3 Done Done New false
14866 4949 Acked 5948 Read 1 Done Done New false
14867 4950 Acked 5949 Write 3 Done Done New false
14868 4951 Acked 5950 Write 4 Done Done New false
14869 4952 Acked 5951 Read 5 Done Done New false
14870 4953 Acked 5952 Read 7 Done Done New false
14871 4954 Acked 5953 Read 1 Done Done New false
14872 4955 Acked 5954 Write 7 Done Done New false
14873 4956 Acked 5955 Read 3 Done Done New false
14874 4957 Acked 5956 Read 6 Done Done New false
14875 4958 Acked 5957 Read 5 Done Done New false
14876 4959 Acked 5958 Write 9 Done Done New false
14877 4960 Acked 5959 Read 9 Done Done New false
14878 4961 Acked 5960 Read 2 Done Done New false
14879 4962 Acked 5961 Read 3 Done Done New false
14880 4963 Acked 5962 Write 5 Done Done New false
14881 4964 Acked 5963 Read 8 Done Done New false
14882 4965 Acked 5964 Write 4 Done Done New false
14883 4966 Acked 5965 Read 1 Done Done New false
14884 4967 Acked 5966 Write 3 Done Done New false
14885 4968 Acked 5967 Read 1 Done Done New false
14886 4969 Acked 5968 Read 8 Done Done New false
14887 4970 Acked 5969 Write 4 Done Done New false
14888 4971 Acked 5970 Write 5 Done Done New false
14889 4972 Acked 5971 Read 1 Done Done New false
14890 4973 Acked 5972 Read 2 Done Done New false
14891 4974 Acked 5973 Read 1 Done Done New false
14892 4975 Acked 5974 Read 5 Done Done New false
14893 4976 Acked 5975 Read 2 Done Done New false
14894 4977 Acked 5976 Write 10 Done Done New false
14895 4978 Acked 5977 Read 4 Done Done New false
14896 4979 Acked 5978 Write 5 Done Done New false
14897 4980 Acked 5979 Write 2 Done Done New false
14898 4981 Acked 5980 Read 3 Done Done New false
14899 4982 Acked 5981 Read 8 Done Done New false
14900 4983 Acked 5982 Read 3 Done Done New false
14901 4984 Acked 5983 Read 2 Done Done New false
14902 4985 Acked 5984 Read 2 Done Done New false
14903 4986 Acked 5985 Write 8 Done Done New false
14904 4987 Acked 5986 Read 6 Done Done New false
14905 4988 Acked 5987 Write 10 Done Done New false
14906 4989 Acked 5988 Write 8 Done Done New false
14907 4990 Acked 5989 Read 2 Done Done New false
14908 4991 Acked 5990 Read 9 Done Done New false
14909 4992 Acked 5991 Write 10 Done Done New false
14910 4993 Acked 5992 Write 4 Done Done New false
14911 4994 Acked 5993 Read 8 Done Done New false
14912 4995 Acked 5994 Read 4 Done Done New false
14913 4996 Acked 5995 Flush 0 Done Done New false
14914 4997 Acked 5996 Write 5 Done Done New false
14915 4998 Acked 5997 Read 8 Done Done New false
14916 4999 Acked 5998 Write 3 Done Done New false
14917 5000 Acked 5999 Write 9 Done Done New false
14918 5001 Acked 6000 Read 2 Done Done New false
14919 5002 Acked 6001 Write 2 Done Done New false
14920 5003 Acked 6002 Read 8 Done Done New false
14921 5004 Acked 6003 Read 7 Done Done New false
14922 5005 Acked 6004 Write 6 Done Done New false
14923 5006 Acked 6005 Read 7 Done Done New false
14924 5007 Acked 6006 Write 3 Done Done New false
14925 5008 Acked 6007 Read 6 Done Done New false
14926 5009 Acked 6008 Write 6 Done Done New false
14927 5010 Acked 6009 Write 5 Done Done New false
14928 5011 Acked 6010 Read 8 Done Done New false
14929 5012 Acked 6011 Read 3 Done Done New false
14930 5013 Acked 6012 Read 1 Done Done New false
14931 5014 Acked 6013 Read 8 Done Done New false
14932 5015 Acked 6014 Flush 0 Done Done New false
14933 5016 Acked 6015 Read 7 Done Done New false
14934 5017 Acked 6016 Read 2 Done Done New false
14935 5018 Acked 6017 Flush 0 Done Done New false
14936 5019 Acked 6018 Read 4 Done Done New false
14937 5020 Acked 6019 Read 1 Done Done New false
14938 5021 Acked 6020 Read 4 Done Done New false
14939 5022 Acked 6021 Read 9 Done Done New false
14940 5023 Acked 6022 Read 10 Done Done New false
14941 5024 Acked 6023 Read 7 Done Done New false
14942 5025 Acked 6024 Flush 0 Done Done New false
14943 5026 Acked 6025 Read 4 Done Done New false
14944 5027 Acked 6026 Read 1 Done Done New false
14945 5028 Acked 6027 Write 9 Done Done New false
14946 5029 Acked 6028 Write 10 Done Done New false
14947 5030 Acked 6029 Flush 0 Done Done New false
14948 5031 Acked 6030 Read 7 Done Done New false
14949 5032 Acked 6031 Read 2 Done Done New false
14950 5033 Acked 6032 Write 8 Done Done New false
14951 5034 Acked 6033 Read 5 Done Done New false
14952 5035 Acked 6034 Read 1 Done Done New false
14953 5036 Acked 6035 Write 9 Done Done New false
14954 5037 Acked 6036 Read 9 Done Done New false
14955 5038 Acked 6037 Flush 0 Done Done New false
14956 5039 Acked 6038 Read 9 Done Done New false
14957 5040 Acked 6039 Read 3 Done Done New false
14958 5041 Acked 6040 Write 3 Done Done New false
14959 5042 Acked 6041 Write 2 Done Done New false
14960 5043 Acked 6042 Read 7 Done Done New false
14961 5044 Acked 6043 Read 7 Done Done New false
14962 5045 Acked 6044 Read 10 Done Done New false
14963 5046 Acked 6045 Read 6 Done Done New false
14964 5047 Acked 6046 Flush 0 Done Done New false
14965 5048 Acked 6047 Write 6 Done Done New false
14966 5049 Acked 6048 Write 4 Done Done New false
14967 5050 Acked 6049 Read 5 Done Done New false
14968 5051 Acked 6050 Read 10 Done Done New false
14969 5052 Acked 6051 Read 1 Done Done New false
14970 5053 Acked 6052 Flush 0 Done Done New false
14971 5054 Acked 6053 Write 8 Done Done New false
14972 5055 Acked 6054 Read 3 Done Done New false
14973 5056 Acked 6055 Write 4 Done Done New false
14974 5057 Acked 6056 Write 10 Done Done New false
14975 5058 Acked 6057 Read 10 Done Done New false
14976 5059 Acked 6058 Flush 0 Done Done New false
14977 5060 Acked 6059 Read 10 Done Done New false
14978 5061 Acked 6060 Read 5 Done Done New false
14979 5062 Acked 6061 Write 10 Done Done New false
14980 5063 Acked 6062 Read 3 Done Done New false
14981 5064 Acked 6063 Write 3 Done Done New false
14982 5065 Acked 6064 Write 6 Done Done New false
14983 5066 Acked 6065 Write 2 Done Done New false
14984 5067 Acked 6066 Read 3 Done Done New false
14985 5068 Acked 6067 Write 1 Done Done New false
14986 5069 Acked 6068 Read 5 Done Done New false
14987 5070 Acked 6069 Write 2 Done Done New false
14988 5071 Acked 6070 Read 1 Done Done New false
14989 5072 Acked 6071 Write 7 Done Done New false
14990 5073 Acked 6072 Write 10 Done Done New false
14991 5074 Acked 6073 Write 6 Done Done New false
14992 5075 Acked 6074 Read 7 Done Done New false
14993 5076 Acked 6075 Read 9 Done Done New false
14994 5077 Acked 6076 Read 1 Done Done New false
14995 5078 Acked 6077 Read 7 Done Done New false
14996 5079 Acked 6078 Write 4 Done Done New false
14997 5080 Acked 6079 Read 10 Done Done New false
14998 5081 Acked 6080 Read 3 Done Done New false
14999 5082 Acked 6081 Write 1 Done Done New false
15000 5083 Acked 6082 Read 4 Done Done New false
15001 5084 Acked 6083 Read 4 Done Done New false
15002 5085 Acked 6084 Read 10 Done Done New false
15003 5086 Acked 6085 Write 2 Done Done New false
15004 5087 Acked 6086 Write 2 Done Done New false
15005 5088 Acked 6087 Write 5 Done Done New false
15006 5089 Acked 6088 Read 6 Done Done New false
15007 5090 Acked 6089 Write 8 Done Done New false
15008 5091 Acked 6090 Write 9 Done Done New false
15009 5092 Acked 6091 Read 5 Done Done New false
15010 5093 Acked 6092 Flush 0 Done Done New false
15011 5094 Acked 6093 Flush 0 Done Done New false
15012 5095 Acked 6094 Read 5 Done Done New false
15013 5096 Acked 6095 Read 2 Done Done New false
15014 5097 Acked 6096 Write 10 Done Done New false
15015 5098 Acked 6097 Read 10 Done Done New false
15016 5099 Acked 6098 Write 9 Done Done New false
15017 5100 Acked 6099 Write 4 Done Done New false
15018 5101 Acked 6100 Write 5 Done Done New false
15019 5102 Acked 6101 Write 9 Done Done New false
15020 5103 Acked 6102 Write 7 Done Done New false
15021 5104 Acked 6103 Write 9 Done Done New false
15022 5105 Acked 6104 Write 10 Done Done New false
15023 5106 Acked 6105 Read 5 Done Done New false
15024 5107 Acked 6106 Read 9 Done Done New false
15025 5108 Acked 6107 Write 1 Done Done New false
15026 5109 Acked 6108 Write 7 Done Done New false
15027 5110 Acked 6109 Write 8 Done Done New false
15028 5111 Acked 6110 Flush 0 Done Done New false
15029 5112 Acked 6111 Read 1 Done Done New false
15030 5113 Acked 6112 Read 9 Done Done New false
15031 5114 Acked 6113 Read 3 Done Done New false
15032 5115 Acked 6114 Read 4 Done Done New false
15033 5116 Acked 6115 Write 10 Done Done New false
15034 5117 Acked 6116 Flush 0 Done Done New false
15035 5118 Acked 6117 Read 2 Done Done New false
15036 5119 Acked 6118 Read 8 Done Done New false
15037 5120 Acked 6119 Read 2 Done Done New false
15038 5121 Acked 6120 Write 6 Done Done New false
15039 5122 Acked 6121 Write 6 Done Done New false
15040 STATES DS:0 DS:1 DS:2 TOTAL
15041 New 0 0 300 300
15042 Sent 0 0 0 0
15043 Done 300 300 0 600
15044 Skipped 0 0 0 0
15045 Error 0 0 0 0
15046 Last Flush: 6116 6116 5821
15047 Downstairs last five completed: 5821 5820 5819 5818 5817
15048 Upstairs last five completed: 5122 5121 5120 5119 5118
15049 CLIENT: Up:0 ds:300 act:2
150502024-02-28T22:16:14.293ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
150512024-02-28T22:16:14.293ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
150522024-02-28T22:16:14.293ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
150532024-02-28T22:16:14.293ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
150542024-02-28T22:16:14.293ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
150552024-02-28T22:16:14.293ZINFOcrucible: [2] client re-new 301 jobs since flush 5821 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
150562024-02-28T22:16:14.293ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
150572024-02-28T22:16:15.294ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
150582024-02-28T22:16:15.294ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
150592024-02-28T22:16:15.294ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
150602024-02-28T22:16:15.294ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
150612024-02-28T22:16:15.294ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
150622024-02-28T22:16:15.294ZINFOcrucible: [2] client re-new 301 jobs since flush 5821 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
150632024-02-28T22:16:15.294ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15064 [8830] got start action
15065 [8830] I am going to restart
15066 Starting downstairs at port 8830
15067 Make output file at "/tmp/dsc/downstairs-8830.txt"
15068 [8830][2] reports Starting
15069 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1181)
15070 [8830][2] reports Running
150712024-02-28T22:16:16.295ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
150722024-02-28T22:16:16.295ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
150732024-02-28T22:16:16.297ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
150742024-02-28T22:16:16.297ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
150752024-02-28T22:16:16.297ZINFOcrucible: send last flush ID to this DS: 5821 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
150762024-02-28T22:16:16.297ZINFOcrucible: Replied this last flush ID: 5821 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
150772024-02-28T22:16:16.297ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
150782024-02-28T22:16:16.297ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
150792024-02-28T22:16:16.297ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
150802024-02-28T22:16:16.297ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15081 ----------------------------------------------------------------
15082 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
15083 Downstairs last five completed: 6122 6121 6120 6119 6118
15084 Upstairs last five completed: 5123 5122 5121 5120 5119
15085 CLIENT: Up:0 ds:0 act:3
15086 Replay: All jobs finished, all DS active.
15087 Main task has work to do, go find it
15088 got dsc Stop(0)
15089 stop 0
15090 Feb 28 22:16:18.003 INFO request completed, latency_us: 364, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 28cda369-f767-468a-a406-a9d3108a21d5, remote_addr: 127[8810] Got stop action so:false kr:false
15091 .0.0.1:36617, local_addr: 127.0.0.1:9998
15092 Feb 28 22:16:18.004 INFO request completed, latency_us: 130{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-28T22:16:18.004385612Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
15093 , {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-28T22:16:18.004496508Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
15094 response_code: 200, {"msg":"downstairs task for 0 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-28T22:16:18.004568215Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
15095 uri: /state/cid/0, method: GET, req_id: {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:16:18.004620873Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"client":"0","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
15096 29890963-3caa-4dd4-82bb-553339bd6cab, remote_addr: 127.0.0.1:36617{"msg":"[0] client re-new 0 jobs since flush 6122","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:16:18.004640042Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
15097 , local_addr: 127.0.0.1:9998
150982024-02-28T22:16:18.004ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15099 [8810] Exited with: ExitStatus(unix_wait_status(9))
15100 [8810][0] reports Exit
151012024-02-28T22:16:19.005ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151022024-02-28T22:16:19.005ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151032024-02-28T22:16:19.005ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151042024-02-28T22:16:19.006ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151052024-02-28T22:16:19.006ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151062024-02-28T22:16:19.006ZINFOcrucible: [0] client re-new 0 jobs since flush 6122 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151072024-02-28T22:16:19.006ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151082024-02-28T22:16:20.008ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151092024-02-28T22:16:20.008ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151102024-02-28T22:16:20.008ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151112024-02-28T22:16:20.008ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151122024-02-28T22:16:20.008ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151132024-02-28T22:16:20.008ZINFOcrucible: [0] client re-new 0 jobs since flush 6122 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151142024-02-28T22:16:20.008ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151152024-02-28T22:16:21.009ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151162024-02-28T22:16:21.009ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151172024-02-28T22:16:21.009ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151182024-02-28T22:16:21.009ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151192024-02-28T22:16:21.009ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151202024-02-28T22:16:21.009ZINFOcrucible: [0] client re-new 0 jobs since flush 6122 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151212024-02-28T22:16:21.009ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15122 Feb 28 22:16:22.006 INFO request completed, latency_us: 207, response_code: 200, uri: /state/cid/0, method: GET, req_id: 31d023a2-2524-4d66-8464-95c0d902cdeb, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
15123 001/300 Read block 2167 len 36864
15124 002/300 Read block 1048 len 16384
15125 003/300 Write block 1229 len 8192 data: 7 6
15126 004/300 Write block 2787 len 8192 data: 3 3
151272024-02-28T22:16:22.010ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151282024-02-28T22:16:22.010ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151292024-02-28T22:16:22.010ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151302024-02-28T22:16:22.010ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151312024-02-28T22:16:22.010ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151322024-02-28T22:16:22.010ZINFOcrucible: [0] client re-new 3 jobs since flush 6122 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
151332024-02-28T22:16:22.010ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15134 005/300 Read block 3728 len 24576
15135 006/300 Read block 4035 len 40960
15136 007/300 Read block 2800 len 12288
15137 008/300 Read block 2124 len 20480
15138 009/300 Write block 1858 len 40960 data: 6 6 6 6 4 4 6 6 6 5
15139 010/300 Read block 239 len 36864
15140 011/300 Write block 3032 len 16384 data: 3 3 3 5
15141 012/300 Write block 2428 len 12288 data: 5 6 6
15142 013/300 Write block 343 len 32768 data: 5 5 6 6 7 5 4 4
15143 014/300 Write block 752 len 16384 data: 6 5 6 5
15144 015/300 Write block 1431 len 12288 data: 3 4 5
15145 016/300 Read block 4136 len 12288
15146 017/300 Write block 690 len 20480 data: 3 3 4 4 3
15147 018/300 Read block 3897 len 12288
15148 019/300 Write block 4205 len 4096 data: 4
15149 020/300 Flush
15150 021/300 Write block 3404 len 8192 data: 5 5
15151 022/300 Read block 4477 len 20480
15152 023/300 Write block 3191 len 8192 data: 4 4
15153 024/300 Read block 1699 len 12288
15154 025/300 Read block 2636 len 4096
15155 026/300 Read block 2535 len 4096
15156 027/300 Read block 1398 len 4096
15157 028/300 Write block 4529 len 36864 data: 5 5 5 4 3 3 3 4 4
15158 029/300 Write block 4261 len 28672 data: 5 5 5 4 4 4 4
15159 030/300 Read block 2947 len 24576
15160 031/300 Read block 3902 len 28672
15161 032/300 Read block 3820 len 28672
15162 033/300 Write block 1487 len 8192 data: 5 4
15163 034/300 Read block 4072 len 32768
15164 035/300 Write block 89 len 40960 data: 2 2 2 2 3 5 5 4 3 5
15165 036/300 Flush
15166 037/300 Write block 2999 len 16384 data: 2 2 2 2
15167 038/300 Write block 3009 len 4096 data: 3
15168 039/300 Read block 867 len 40960
15169 040/300 Write block 1931 len 12288 data: 3 3 3
15170 041/300 Write block 1507 len 4096 data: 3
15171 042/300 Flush
15172 043/300 Write block 633 len 4096 data: 3
15173 044/300 Read block 3837 len 36864
15174 045/300 Read block 21 len 24576
15175 046/300 Read block 2659 len 32768
15176 047/300 Read block 1871 len 16384
15177 048/300 Write block 793 len 16384 data: 4 4 4 3
15178 049/300 Flush
15179 050/300 Write block 1089 len 12288 data: 5 4 5
15180 051/300 Read block 2863 len 36864
15181 052/300 Read block 1866 len 24576
15182 053/300 Write block 1056 len 8192 data: 5 5
15183 054/300 Read block 7 len 16384
15184 055/300 Write block 699 len 36864 data: 3 3 3 2 2 2 2 3 3
15185 056/300 Write block 856 len 32768 data: 4 5 3 3 4 3 3 3
15186 057/300 Write block 810 len 28672 data: 4 4 3 3 2 2 2
15187 058/300 Read block 142 len 28672
15188 059/300 Flush
15189 060/300 Write block 1192 len 16384 data: 7 5 4 5
15190 061/300 Write block 1799 len 40960 data: 6 6 6 6 7 3 2 2 2 2
15191 062/300 Flush
15192 063/300 Read block 2881 len 24576
15193 064/300 Read block 4569 len 20480
15194 065/300 Write block 3516 len 16384 data: 7 6 6 5
15195 066/300 Read block 1341 len 8192
15196 067/300 Write block 3573 len 24576 data: 8 6 6 6 5 5
15197 068/300 Write block 21 len 4096 data: 5
15198 069/300 Write block 195 len 24576 data: 3 4 3 3 2 2
15199 070/300 Flush
15200 071/300 Read block 1699 len 36864
15201 072/300 Flush
15202 073/300 Write block 2675 len 20480 data: 4 4 5 5 6
15203 074/300 Read block 3937 len 40960
15204 075/300 Read block 4690 len 24576
15205 076/300 Write block 1791 len 12288 data: 4 4 4
15206 077/300 Flush
15207 078/300 Read block 995 len 32768
15208 079/300 Read block 2565 len 24576
15209 080/300 Read block 2644 len 16384
15210 081/300 Write block 160 len 4096 data: 6
15211 082/300 Read block 2988 len 20480
15212 083/300 Write block 731 len 16384 data: 5 6 5 6
15213 084/300 Write block 31 len 40960 data: 4 4 3 2 2 2 3 3 3 3
15214 085/300 Write block 1900 len 32768 data: 7 6 5 5 5 6 7 6
15215 086/300 Write block 4566 len 24576 data: 5 5 5 5 5 4
15216 087/300 Read block 2152 len 20480
15217 088/300 Read block 3623 len 24576
15218 089/300 Write block 26 len 20480 data: 3 3 5 4 4
15219 090/300 Read block 2014 len 20480
15220 091/300 Read block 2713 len 12288
15221 092/300 Read block 4353 len 28672
15222 093/300 Read block 4589 len 28672
15223 094/300 Write block 3826 len 28672 data: 6 6 6 5 3 2 2
15224 095/300 Read block 1465 len 32768
15225 096/300 Write block 4446 len 20480 data: 2 2 2 2 3
15226 097/300 Read block 4380 len 12288
15227 098/300 Read block 2700 len 20480
15228 099/300 Flush
15229 100/300 Read block 3374 len 20480
15230 101/300 Read block 112 len 40960
15231 102/300 Read block 4487 len 4096
15232 103/300 Read block 4965 len 28672
15233 104/300 Read block 410 len 12288
15234 105/300 Write block 3747 len 24576 data: 4 3 4 4 4 5
15235 106/300 Read block 385 len 32768
15236 107/300 Read block 2470 len 4096
15237 108/300 Read block 409 len 8192
15238 109/300 Write block 995 len 32768 data: 3 3 4 4 5 6 6 5
15239 110/300 Read block 1611 len 8192
15240 111/300 Read block 2038 len 16384
15241 112/300 Write block 2294 len 16384 data: 3 3 3 3
15242 113/300 Write block 2467 len 32768 data: 4 4 5 5 4 4 3 4
15243 114/300 Read block 4751 len 12288
15244 115/300 Write block 3014 len 12288 data: 6 5 6
15245 116/300 Read block 1551 len 32768
15246 117/300 Read block 1085 len 28672
15247 118/300 Write block 56 len 36864 data: 5 5 5 6 7 7 7 7 6
15248 119/300 Write block 4564 len 32768 data: 6 7 6 6 6 6 6 5
15249 120/300 Write block 3637 len 32768 data: 4 4 6 6 6 5 4 3
15250 121/300 Read block 10 len 36864
15251 122/300 Read block 3616 len 32768
15252 123/300 Read block 2638 len 8192
15253 124/300 Write block 4002 len 40960 data: 4 4 3 3 2 2 3 4 4 5
15254 125/300 Read block 1036 len 16384
15255 126/300 Read block 3788 len 28672
15256 127/300 Read block 3462 len 28672
15257 128/300 Write block 4188 len 24576 data: 5 5 4 4 3 3
15258 129/300 Write block 1722 len 8192 data: 6 6
15259 130/300 Read block 1222 len 16384
15260 131/300 Write block 4170 len 32768 data: 6 6 7 5 6 5 3 3
15261 132/300 Flush
15262 133/300 Write block 4745 len 4096 data: 3
15263 134/300 Read block 4342 len 24576
15264 135/300 Read block 2252 len 4096
15265 136/300 Write block 1353 len 32768 data: 4 3 2 2 2 2 2 3
15266 137/300 Read block 3999 len 20480
15267 138/300 Read block 4248 len 32768
15268 139/300 Read block 2525 len 24576
15269 140/300 Read block 2556 len 28672
15270 141/300 Write block 1902 len 8192 data: 6 6
15271 142/300 Read block 2223 len 20480
15272 143/300 Read block 3895 len 4096
15273 144/300 Write block 4791 len 8192 data: 5 6
15274 145/300 Read block 4954 len 36864
15275 146/300 Read block 1575 len 12288
15276 147/300 Read block 4368 len 16384
15277 148/300 Write block 956 len 24576 data: 6 7 8 9 8 9
15278 149/300 Read block 4296 len 12288
15279 150/300 Read block 2437 len 20480
15280 151/300 Read block 3464 len 20480
15281 152/300 Read block 772 len 24576
15282 153/300 Read block 3071 len 32768
15283 154/300 Read block 4891 len 28672
15284 155/300 Read block 1683 len 8192
15285 156/300 Write block 1047 len 36864 data: 5 4 4 4 4 4 4 4 4
15286 157/300 Write block 2073 len 12288 data: 6 5 4
15287 158/300 Flush
15288 159/300 Read block 160 len 12288
15289 160/300 Write block 177 len 32768 data: 2 3 4 5 4 4 3 3
15290 161/300 Write block 2923 len 8192 data: 5 5
15291 162/300 Flush
15292 163/300 Read block 3535 len 4096
15293 164/300 Read block 1646 len 40960
15294 165/300 Read block 3011 len 28672
15295 166/300 Read block 3136 len 8192
15296 167/300 Read block 3441 len 8192
15297 168/300 Write block 2597 len 40960 data: 5 5 5 5 4 6 5 6 6 6
15298 169/300 Read block 4569 len 20480
15299 170/300 Write block 3644 len 32768 data: 4 4 4 3 5 5 4 3
15300 171/300 Write block 4590 len 36864 data: 6 6 5 3 3 3 4 4 4
15301 172/300 Read block 4616 len 20480
15302 173/300 Write block 1952 len 32768 data: 4 4 4 3 3 3 3 3
15303 174/300 Write block 3900 len 8192 data: 3 4
15304 175/300 Read block 841 len 32768
15305 176/300 Read block 1893 len 24576
15306 177/300 Flush
15307 178/300 Read block 4633 len 20480
15308 179/300 Read block 1891 len 24576
15309 180/300 Read block 1509 len 8192
15310 181/300 Read block 442 len 16384
15311 182/300 Read block 48 len 32768
15312 183/300 Read block 2085 len 16384
15313 184/300 Write block 1809 len 8192 data: 3 3
15314 185/300 Flush
15315 186/300 Read block 4750 len 40960
15316 187/300 Read block 410 len 20480
15317 188/300 Read block 2562 len 16384
15318 189/300 Write block 2138 len 24576 data: 2 3 3 3 3 3
15319 190/300 Write block 1518 len 4096 data: 8
15320 191/300 Write block 228 len 40960 data: 2 2 2 2 3 3 6 5 4 3
15321 192/300 Write block 4171 len 32768 data: 7 8 6 7 6 4 4 3
15322 193/300 Read block 3869 len 12288
15323 194/300 Read block 4333 len 12288
15324 195/300 Flush
15325 196/300 Read block 1438 len 20480
15326 197/300 Read block 4194 len 12288
15327 198/300 Read block 172 len 24576
15328 199/300 Write block 1259 len 20480 data: 6 6 4 4 3
15329 200/300 Flush
15330 201/300 Write block 658 len 4096 data: 5
15331 202/300 Read block 2332 len 28672
15332 203/300 Flush
15333 204/300 Flush
15334 205/300 Read block 1730 len 32768
15335 206/300 Write block 3556 len 4096 data: 5
15336 207/300 Read block 3815 len 28672
15337 208/300 Write block 1834 len 20480 data: 5 5 6 4 4
15338 209/300 Write block 4201 len 20480 data: 5 5 4 3 5
15339 210/300 Write block 1038 len 40960 data: 7 6 8 8 6 5 6 6 6 6
15340 211/300 Read block 1882 len 8192
15341 212/300 Write block 2112 len 12288 data: 3 3 5
15342 213/300 Read block 948 len 28672
15343 214/300 Read block 1187 len 40960
15344 215/300 Read block 3176 len 12288
15345 216/300 Read block 1175 len 24576
15346 217/300 Read block 3555 len 4096
15347 218/300 Read block 1096 len 28672
15348 219/300 Write block 473 len 36864 data: 5 6 6 6 5 5 5 5 4
15349 220/300 Read block 513 len 16384
15350 221/300 Flush
15351 222/300 Write block 1979 len 40960 data: 7 5 6 7 7 7 7 7 7 6
15352 223/300 Read block 4240 len 24576
15353 224/300 Read block 2556 len 20480
15354 225/300 Read block 3967 len 24576
15355 226/300 Write block 3616 len 40960 data: 2 3 3 3 3 4 3 4 4 4
15356 227/300 Read block 271 len 36864
15357 228/300 Flush
15358 229/300 Read block 313 len 16384
15359 230/300 Read block 533 len 4096
15360 231/300 Write block 4349 len 32768 data: 5 3 3 4 4 3 3 3
15361 232/300 Write block 23 len 8192 data: 5 6
15362 233/300 Read block 1584 len 24576
15363 234/300 Read block 194 len 4096
15364 235/300 Read block 3454 len 28672
15365 236/300 Write block 4542 len 24576 data: 5 5 5 5 4 5
15366 237/300 Read block 2962 len 12288
15367 238/300 Read block 4656 len 12288
15368 239/300 Read block 361 len 16384
15369 240/300 Read block 317 len 36864
15370 241/300 Read block 724 len 24576
15371 242/300 Flush
15372 243/300 Write block 1754 len 40960 data: 6 5 5 3 2 2 2 3 6 6
15373 244/300 Write block 2096 len 4096 data: 7
15374 245/300 Flush
15375 246/300 Read block 3937 len 4096
15376 247/300 Flush
15377 248/300 Read block 2423 len 16384
15378 249/300 Write block 2430 len 8192 data: 7 5
15379 250/300 Write block 2948 len 36864 data: 4 4 3 3 3 4 4 4 4
15380 251/300 Read block 4878 len 24576
15381 252/300 Read block 527 len 20480
15382 253/300 Read block 3618 len 32768
15383 254/300 Read block 63 len 32768
15384 255/300 Write block 3482 len 8192 data: 3 3
15385 256/300 Write block 2497 len 12288 data: 3 4 3
15386 257/300 Write block 3669 len 8192 data: 3 2
15387 258/300 Write block 1229 len 4096 data: 8
15388 259/300 Write block 3817 len 20480 data: 3 2 2 4 5
15389 260/300 Write block 1851 len 24576 data: 4 3 2 2 4 5
15390 261/300 Read block 4074 len 4096
15391 262/300 Read block 1090 len 40960
15392 263/300 Read block 655 len 12288
15393 264/300 Write block 4180 len 12288 data: 5 6 6
15394 265/300 Write block 3622 len 40960 data: 4 5 5 5 5 5 5 5 5 5
15395 266/300 Write block 3788 len 16384 data: 3 4 4 5
15396 267/300 Flush
15397 268/300 Read block 366 len 28672
15398 269/300 Write block 2588 len 8192 data: 4 4
15399 270/300 Write block 2277 len 40960 data: 4 4 4 3 3 3 4 3 3 4
15400 271/300 Write block 4243 len 20480 data: 4 5 5 4 4
15401 272/300 Read block 1553 len 4096
15402 273/300 Read block 1079 len 28672
15403 274/300 Read block 3971 len 24576
15404 275/300 Write block 3970 len 8192 data: 4 6
15405 276/300 Write block 1590 len 8192 data: 3 3
15406 277/300 Read block 4767 len 8192
15407 278/300 Read block 3604 len 32768
15408 279/300 Flush
15409 280/300 Read block 2257 len 24576
15410 281/300 Flush
15411 282/300 Write block 1758 len 16384 data: 3 3 3 4
15412 283/300 Read block 4316 len 24576
15413 284/300 Read block 1024 len 20480
15414 285/300 Flush
15415 286/300 Write block 910 len 24576 data: 4 4 4 4 5 4
15416 287/300 Read block 3926 len 32768
15417 288/300 Write block 1770 len 12288 data: 6 7 5
15418 289/300 Read block 1112 len 36864
15419 290/300 Write block 2946 len 20480 data: 2 4 5 5 4
15420 291/300 Read block 4345 len 32768
15421 292/300 Read block 349 len 16384
15422 293/300 Write block 1496 len 12288 data: 3 3 3
15423 294/300 Read block 909 len 40960
15424 295/300 Read block 3123 len 20480
15425 296/300 Read block 1668 len 36864
15426 297/300 Read block 574 len 28672
15427 298/300 Write block 4735 len 8192 data: 5 5
15428 299/300 Read block 411 len 4096
15429 300/300 Read block 2757 len 12288
15430 Main task has work to do, go find it
15431 got dsc Start(0)
15432 start 0
15433 Feb 28 22:16:22.672 INFO request completed, latency_us: 295, response_code: 204, uri: /start/cid/0, method: POST, req_id: dbf533fd-f620-42e8-be17-3b14d1cc5f6c, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
15434 Replay: started 0, returned:Ok(())
15435 ----------------------------------------------------------------
15436 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
15437 States: Offline Active Active
15438 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
15439 5124 Acked 6123 Read 9 New Done Done true
15440 5125 Acked 6124 Read 4 New Done Done true
15441 5126 Acked 6125 Write 2 New Done Done true
15442 5127 Acked 6126 Write 2 New Done Done false
15443 5128 Acked 6127 Read 6 New Done Done false
15444 5129 Acked 6128 Read 10 New Done Done false
15445 5130 Acked 6129 Read 3 New Done Done false
15446 5131 Acked 6130 Read 5 New Done Done false
15447 5132 Acked 6131 Write 10 New Done Done false
15448 5133 Acked 6132 Read 9 New Done Done false
15449 5134 Acked 6133 Write 4 New Done Done false
15450 5135 Acked 6134 Write 3 New Done Done false
15451 5136 Acked 6135 Write 8 New Done Done false
15452 5137 Acked 6136 Write 4 New Done Done false
15453 5138 Acked 6137 Write 3 New Done Done false
15454 5139 Acked 6138 Read 3 New Done Done false
15455 5140 Acked 6139 Write 5 New Done Done false
15456 5141 Acked 6140 Read 3 New Done Done false
15457 5142 Acked 6141 Write 1 New Done Done false
15458 5143 Acked 6142 Flush 0 New Done Done false
15459 5144 Acked 6143 Write 2 New Done Done false
15460 5145 Acked 6144 Read 5 New Done Done false
15461 5146 Acked 6145 Write 2 New Done Done false
15462 5147 Acked 6146 Read 3 New Done Done false
15463 5148 Acked 6147 Read 1 New Done Done false
15464 5149 Acked 6148 Read 1 New Done Done false
15465 5150 Acked 6149 Read 1 New Done Done false
15466 5151 Acked 6150 Write 9 New Done Done false
15467 5152 Acked 6151 Write 7 New Done Done false
15468 5153 Acked 6152 Read 6 New Done Done false
15469 5154 Acked 6153 Read 7 New Done Done false
15470 5155 Acked 6154 Read 7 New Done Done false
15471 5156 Acked 6155 Write 2 New Done Done false
15472 5157 Acked 6156 Read 8 New Done Done false
15473 5158 Acked 6157 Write 10 New Done Done false
15474 5159 Acked 6158 Flush 0 New Done Done false
15475 5160 Acked 6159 Write 4 New Done Done false
15476 5161 Acked 6160 Write 1 New Done Done false
15477 5162 Acked 6161 Read 10 New Done Done false
15478 5163 Acked 6162 Write 3 New Done Done false
15479 5164 Acked 6163 Write 1 New Done Done false
15480 5165 Acked 6164 Flush 0 New Done Done false
15481 5166 Acked 6165 Write 1 New Done Done false
15482 5167 Acked 6166 Read 9 New Done Done false
15483 5168 Acked 6167 Read 6 New Done Done false
15484 5169 Acked 6168 Read 8 New Done Done false
15485 5170 Acked 6169 Read 4 New Done Done false
15486 5171 Acked 6170 Write 4 New Done Done false
15487 5172 Acked 6171 Flush 0 New Done Done false
15488 5173 Acked 6172 Write 3 New Done Done false
15489 5174 Acked 6173 Read 9 New Done Done false
15490 5175 Acked 6174 Read 6 New Done Done false
15491 5176 Acked 6175 Write 2 New Done Done false
15492 5177 Acked 6176 Read 4 New Done Done false
15493 5178 Acked 6177 Write 9 New Done Done false
15494 5179 Acked 6178 Write 8 New Done Done false
15495 5180 Acked 6179 Write 7 New Done Done false
15496 5181 Acked 6180 Read 7 New Done Done false
15497 5182 Acked 6181 Flush 0 New Done Done false
15498 5183 Acked 6182 Write 4 New Done Done false
15499 5184 Acked 6183 Write 10 New Done Done false
15500 5185 Acked 6184 Flush 0 New Done Done false
15501 5186 Acked 6185 Read 6 New Done Done false
15502 5187 Acked 6186 Read 5 New Done Done false
15503 5188 Acked 6187 Write 4 New Done Done false
15504 5189 Acked 6188 Read 2 New Done Done false
15505 5190 Acked 6189 Write 6 New Done Done false
15506 5191 Acked 6190 Write 1 New Done Done false
15507 5192 Acked 6191 Write 6 New Done Done false
15508 5193 Acked 6192 Flush 0 New Done Done false
15509 5194 Acked 6193 Read 9 New Done Done false
15510 5195 Acked 6194 Flush 0 New Done Done false
15511 5196 Acked 6195 Write 5 New Done Done false
15512 5197 Acked 6196 Read 10 New Done Done false
15513 5198 Acked 6197 Read 6 New Done Done false
15514 5199 Acked 6198 Write 3 New Done Done false
15515 5200 Acked 6199 Flush 0 New Done Done false
15516 5201 Acked 6200 Read 8 New Done Done false
15517 5202 Acked 6201 Read 6 New Done Done false
15518 5203 Acked 6202 Read 4 New Done Done false
15519 5204 Acked 6203 Write 1 New Done Done false
15520 5205 Acked 6204 Read 5 New Done Done false
15521 5206 Acked 6205 Write 4 New Done Done false
15522 5207 Acked 6206 Write 10 New Done Done false
15523 5208 Acked 6207 Write 8 New Done Done false
15524 5209 Acked 6208 Write 6 New Done Done false
15525 5210 Acked 6209 Read 5 New Done Done false
15526 5211 Acked 6210 Read 6 New Done Done false
15527 5212 Acked 6211 Write 5 New Done Done false
15528 5213 Acked 6212 Read 5 New Done Done false
15529 5214 Acked 6213 Read 3 New Done Done false
15530 5215 Acked 6214 Read 7 New Done Done false
15531 5216 Acked 6215 Read 7 New Done Done false
15532 5217 Acked 6216 Write 7 New Done Done false
15533 5218 Acked 6217 Read 8 New Done Done false
15534 5219 Acked 6218 Write 5 New Done Done false
15535 5220 Acked 6219 Read 3 New Done Done false
15536 5221 Acked 6220 Read 5 New Done Done false
15537 5222 Acked 6221 Flush 0 New Done Done false
15538 5223 Acked 6222 Read 5 New Done Done false
15539 5224 Acked 6223 Read 10 New Done Done false
15540 5225 Acked 6224 Read 1 New Done Done false
15541 5226 Acked 6225 Read 7 New Done Done false
15542 5227 Acked 6226 Read 3 New Done Done false
15543 5228 Acked 6227 Write 6 New Done Done false
15544 5229 Acked 6228 Read 8 New Done Done false
15545 5230 Acked 6229 Read 1 New Done Done false
15546 5231 Acked 6230 Read 2 New Done Done false
15547 5232 Acked 6231 Write 8 New Done Done false
15548 5233 Acked 6232 Read 2 New Done Done false
15549 5234 Acked 6233 Read 4 New Done Done false
15550 5235 Acked 6234 Write 4 New Done Done false
15551 5236 Acked 6235 Write 8 New Done Done false
15552 5237 Acked 6236 Read 3 New Done Done false
15553 5238 Acked 6237 Write 3 New Done Done false
15554 5239 Acked 6238 Read 8 New Done Done false
15555 5240 Acked 6239 Read 7 New Done Done false
15556 5241 Acked 6240 Write 9 New Done Done false
15557 5242 Acked 6241 Write 8 New Done Done false
15558 5243 Acked 6242 Write 8 New Done Done false
15559 5244 Acked 6243 Read 9 New Done Done false
15560 5245 Acked 6244 Read 8 New Done Done false
15561 5246 Acked 6245 Read 2 New Done Done false
15562 5247 Acked 6246 Write 10 New Done Done false
15563 5248 Acked 6247 Read 4 New Done Done false
15564 5249 Acked 6248 Read 7 New Done Done false
15565 5250 Acked 6249 Read 7 New Done Done false
15566 5251 Acked 6250 Write 6 New Done Done false
15567 5252 Acked 6251 Write 2 New Done Done false
15568 5253 Acked 6252 Read 4 New Done Done false
15569 5254 Acked 6253 Write 8 New Done Done false
15570 5255 Acked 6254 Flush 0 New Done Done false
15571 5256 Acked 6255 Write 1 New Done Done false
15572 5257 Acked 6256 Read 6 New Done Done false
15573 5258 Acked 6257 Read 1 New Done Done false
15574 5259 Acked 6258 Write 8 New Done Done false
15575 5260 Acked 6259 Read 5 New Done Done false
15576 5261 Acked 6260 Read 8 New Done Done false
15577 5262 Acked 6261 Read 6 New Done Done false
15578 5263 Acked 6262 Read 7 New Done Done false
15579 5264 Acked 6263 Write 2 New Done Done false
15580 5265 Acked 6264 Read 5 New Done Done false
15581 5266 Acked 6265 Read 1 New Done Done false
15582 5267 Acked 6266 Write 2 New Done Done false
15583 5268 Acked 6267 Read 9 New Done Done false
15584 5269 Acked 6268 Read 3 New Done Done false
15585 5270 Acked 6269 Read 4 New Done Done false
15586 5271 Acked 6270 Write 6 New Done Done false
15587 5272 Acked 6271 Read 3 New Done Done false
15588 5273 Acked 6272 Read 5 New Done Done false
15589 5274 Acked 6273 Read 5 New Done Done false
15590 5275 Acked 6274 Read 6 New Done Done false
15591 5276 Acked 6275 Read 8 New Done Done false
15592 5277 Acked 6276 Read 7 New Done Done false
15593 5278 Acked 6277 Read 2 New Done Done false
15594 5279 Acked 6278 Write 9 New Done Done false
15595 5280 Acked 6279 Write 3 New Done Done false
15596 5281 Acked 6280 Flush 0 New Done Done false
15597 5282 Acked 6281 Read 3 New Done Done false
15598 5283 Acked 6282 Write 8 New Done Done false
15599 5284 Acked 6283 Write 2 New Done Done false
15600 5285 Acked 6284 Flush 0 New Done Done false
15601 5286 Acked 6285 Read 1 New Done Done false
15602 5287 Acked 6286 Read 10 New Done Done false
15603 5288 Acked 6287 Read 7 New Done Done false
15604 5289 Acked 6288 Read 2 New Done Done false
15605 5290 Acked 6289 Read 2 New Done Done false
15606 5291 Acked 6290 Write 10 New Done Done false
15607 5292 Acked 6291 Read 5 New Done Done false
15608 5293 Acked 6292 Write 8 New Done Done false
15609 5294 Acked 6293 Write 9 New Done Done false
15610 5295 Acked 6294 Read 5 New Done Done false
15611 5296 Acked 6295 Write 8 New Done Done false
15612 5297 Acked 6296 Write 2 New Done Done false
15613 5298 Acked 6297 Read 8 New Done Done false
15614 5299 Acked 6298 Read 6 New Done Done false
15615 5300 Acked 6299 Flush 0 New Done Done false
15616 5301 Acked 6300 Read 5 New Done Done false
15617 5302 Acked 6301 Read 6 New Done Done false
15618 5303 Acked 6302 Read 2 New Done Done false
15619 5304 Acked 6303 Read 4 New Done Done false
15620 5305 Acked 6304 Read 8 New Done Done false
15621 5306 Acked 6305 Read 4 New Done Done false
15622 5307 Acked 6306 Write 2 New Done Done false
15623 5308 Acked 6307 Flush 0 New Done Done false
15624 5309 Acked 6308 Read 10 New Done Done false
15625 5310 Acked 6309 Read 5 New Done Done false
15626 5311 Acked 6310 Read 4 New Done Done false
15627 5312 Acked 6311 Write 6 New Done Done false
15628 5313 Acked 6312 Write 1 New Done Done false
15629 5314 Acked 6313 Write 10 New Done Done false
15630 5315 Acked 6314 Write 8 New Done Done false
15631 5316 Acked 6315 Read 3 New Done Done false
15632 5317 Acked 6316 Read 3 New Done Done false
15633 5318 Acked 6317 Flush 0 New Done Done false
15634 5319 Acked 6318 Read 5 New Done Done false
15635 5320 Acked 6319 Read 3 New Done Done false
15636 5321 Acked 6320 Read 6 New Done Done false
15637 5322 Acked 6321 Write 5 New Done Done false
15638 5323 Acked 6322 Flush 0 New Done Done false
15639 5324 Acked 6323 Write 1 New Done Done false
15640 5325 Acked 6324 Read 7 New Done Done false
15641 5326 Acked 6325 Flush 0 New Done Done false
15642 5327 Acked 6326 Flush 0 New Done Done false
15643 5328 Acked 6327 Read 8 New Done Done false
15644 5329 Acked 6328 Write 1 New Done Done false
15645 5330 Acked 6329 Read 7 New Done Done false
15646 5331 Acked 6330 Write 5 New Done Done false
15647 5332 Acked 6331 Write 5 New Done Done false
15648 5333 Acked 6332 Write 10 New Done Done false
15649 5334 Acked 6333 Read 2 New Done Done false
15650 5335 Acked 6334 Write 3 New Done Done false
15651 5336 Acked 6335 Read 7 New Done Done false
15652 5337 Acked 6336 Read 10 New Done Done false
15653 5338 Acked 6337 Read 3 New Done Done false
15654 5339 Acked 6338 Read 6 New Done Done false
15655 5340 Acked 6339 Read 1 New Done Done false
15656 5341 Acked 6340 Read 7 New Done Done false
15657 5342 Acked 6341 Write 9 New Done Done false
15658 5343 Acked 6342 Read 4 New Done Done false
15659 5344 Acked 6343 Flush 0 New Done Done false
15660 5345 Acked 6344 Write 10 New Done Done false
15661 5346 Acked 6345 Read 6 New Done Done false
15662 5347 Acked 6346 Read 5 New Done Done false
15663 5348 Acked 6347 Read 6 New Done Done false
15664 5349 Acked 6348 Write 10 New Done Done false
15665 5350 Acked 6349 Read 9 New Done Done false
15666 5351 Acked 6350 Flush 0 New Done Done false
15667 5352 Acked 6351 Read 4 New Done Done false
15668 5353 Acked 6352 Read 1 New Done Done false
15669 5354 Acked 6353 Write 8 New Done Done false
15670 5355 Acked 6354 Write 2 New Done Done false
15671 5356 Acked 6355 Read 6 New Done Done false
15672 5357 Acked 6356 Read 1 New Done Done false
15673 5358 Acked 6357 Read 7 New Done Done false
15674 5359 Acked 6358 Write 6 New Done Done false
15675 5360 Acked 6359 Read 3 New Done Done false
15676 5361 Acked 6360 Read 3 New Done Done false
15677 5362 Acked 6361 Read 4 New Done Done false
15678 5363 Acked 6362 Read 9 New Done Done false
15679 5364 Acked 6363 Read 6 New Done Done false
15680 5365 Acked 6364 Flush 0 New Done Done false
15681 5366 Acked 6365 Write 10 New Done Done false
15682 5367 Acked 6366 Write 1 New Done Done false
15683 5368 Acked 6367 Flush 0 New Done Done false
15684 5369 Acked 6368 Read 1 New Done Done false
15685 5370 Acked 6369 Flush 0 New Done Done false
15686 5371 Acked 6370 Read 4 New Done Done false
15687 5372 Acked 6371 Write 2 New Done Done false
15688 5373 Acked 6372 Write 9 New Done Done false
15689 5374 Acked 6373 Read 6 New Done Done false
15690 5375 Acked 6374 Read 5 New Done Done false
15691 5376 Acked 6375 Read 8 New Done Done false
15692 5377 Acked 6376 Read 8 New Done Done false
15693 5378 Acked 6377 Write 2 New Done Done false
15694 5379 Acked 6378 Write 3 New Done Done false
15695 5380 Acked 6379 Write 2 New Done Done false
15696 5381 Acked 6380 Write 1 New Done Done false
15697 5382 Acked 6381 Write 5 New Done Done false
15698 5383 Acked 6382 Write 6 New Done Done false
15699 5384 Acked 6383 Read 1 New Done Done false
15700 5385 Acked 6384 Read 10 New Done Done false
15701 5386 Acked 6385 Read 3 New Done Done false
15702 5387 Acked 6386 Write 3 New Done Done false
15703 5388 Acked 6387 Write 10 New Done Done false
15704 5389 Acked 6388 Write 4 New Done Done false
15705 5390 Acked 6389 Flush 0 New Done Done false
15706 5391 Acked 6390 Read 7 New Done Done false
15707 5392 Acked 6391 Write 2 New Done Done false
15708 5393 Acked 6392 Write 10 New Done Done false
15709 5394 Acked 6393 Write 5 New Done Done false
15710 5395 Acked 6394 Read 1 New Done Done false
15711 5396 Acked 6395 Read 7 New Done Done false
15712 5397 Acked 6396 Read 6 New Done Done false
15713 5398 Acked 6397 Write 2 New Done Done false
15714 5399 Acked 6398 Write 2 New Done Done false
15715 5400 Acked 6399 Read 2 New Done Done false
15716 5401 Acked 6400 Read 8 New Done Done false
15717 5402 Acked 6401 Flush 0 New Done Done false
15718 5403 Acked 6402 Read 6 New Done Done false
15719 5404 Acked 6403 Flush 0 New Done Done false
15720 5405 Acked 6404 Write 4 New Done Done false
15721 5406 Acked 6405 Read 6 New Done Done false
15722 5407 Acked 6406 Read 5 New Done Done false
15723 5408 Acked 6407 Flush 0 New Done Done false
15724 5409 Acked 6408 Write 6 New Done Done false
15725 5410 Acked 6409 Read 8 New Done Done false
15726 5411 Acked 6410 Write 3 New Done Done false
15727 5412 Acked 6411 Read 9 New Done Done false
15728 5413 Acked 6412 Write 5 New Done Done false
15729 5414 Acked 6413 Read 8 New Done Done false
15730 5415 Acked 6414 Read 4 New Done Done false
15731 5416 Acked 6415 Write 3 New Done Done false
15732 5417 Acked 6416 Read 10 New Done Done false
15733 5418 Acked 6417 Read 5 New Done Done false
15734 5419 Acked 6418 Read 9 New Done Done false
15735 5420 Acked 6419 Read 7 New Done Done false
15736 5421 Acked 6420 Write 2 New Done Done false
15737 5422 Acked 6421 Read 1 New Done Done false
15738 5423 Acked 6422 Read 3 New Done Done false
15739 STATES DS:0 DS:1 DS:2 TOTAL
15740 New 300 0 0 300
15741 Sent 0 0 0 0
15742 Done 0 300 300 600
15743 Skipped 0 0 0 0
15744 Error 0 0 0 0
15745 Last Flush: 6122 6407 6407
15746 Downstairs last five completed: 6122 6121 6120 6119 6118
15747 Upstairs last five completed: 5423 5422 5421 5420 5419
15748 CLIENT: Up:0 ds:300 act:2
157492024-02-28T22:16:23.011ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157502024-02-28T22:16:23.011ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157512024-02-28T22:16:23.011ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157522024-02-28T22:16:23.011ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157532024-02-28T22:16:23.011ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157542024-02-28T22:16:23.011ZINFOcrucible: [0] client re-new 301 jobs since flush 6122 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157552024-02-28T22:16:23.011ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157562024-02-28T22:16:24.012ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157572024-02-28T22:16:24.012ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157582024-02-28T22:16:24.012ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157592024-02-28T22:16:24.012ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157602024-02-28T22:16:24.012ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157612024-02-28T22:16:24.012ZINFOcrucible: [0] client re-new 301 jobs since flush 6122 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157622024-02-28T22:16:24.013ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157632024-02-28T22:16:25.015ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157642024-02-28T22:16:25.015ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157652024-02-28T22:16:25.015ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157662024-02-28T22:16:25.015ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157672024-02-28T22:16:25.015ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157682024-02-28T22:16:25.015ZINFOcrucible: [0] client re-new 301 jobs since flush 6122 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157692024-02-28T22:16:25.015ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157702024-02-28T22:16:26.016ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157712024-02-28T22:16:26.016ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157722024-02-28T22:16:26.016ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157732024-02-28T22:16:26.016ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157742024-02-28T22:16:26.016ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157752024-02-28T22:16:26.016ZINFOcrucible: [0] client re-new 301 jobs since flush 6122 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
157762024-02-28T22:16:26.016ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
15777 ----------------------------------------------------------------
15778 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15779 States: Offline Active Active
15780 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
15781 5124 Acked 6123 Read 9 New Done Done true
15782 5125 Acked 6124 Read 4 New Done Done true
15783 5126 Acked 6125 Write 2 New Done Done true
15784 5127 Acked 6126 Write 2 New Done Done true
15785 5128 Acked 6127 Read 6 New Done Done true
15786 5129 Acked 6128 Read 10 New Done Done true
15787 5130 Acked 6129 Read 3 New Done Done true
15788 5131 Acked 6130 Read 5 New Done Done true
15789 5132 Acked 6131 Write 10 New Done Done true
15790 5133 Acked 6132 Read 9 New Done Done true
15791 5134 Acked 6133 Write 4 New Done Done true
15792 5135 Acked 6134 Write 3 New Done Done true
15793 5136 Acked 6135 Write 8 New Done Done true
15794 5137 Acked 6136 Write 4 New Done Done true
15795 5138 Acked 6137 Write 3 New Done Done true
15796 5139 Acked 6138 Read 3 New Done Done true
15797 5140 Acked 6139 Write 5 New Done Done true
15798 5141 Acked 6140 Read 3 New Done Done true
15799 5142 Acked 6141 Write 1 New Done Done true
15800 5143 Acked 6142 Flush 0 New Done Done true
15801 5144 Acked 6143 Write 2 New Done Done true
15802 5145 Acked 6144 Read 5 New Done Done true
15803 5146 Acked 6145 Write 2 New Done Done true
15804 5147 Acked 6146 Read 3 New Done Done true
15805 5148 Acked 6147 Read 1 New Done Done true
15806 5149 Acked 6148 Read 1 New Done Done true
15807 5150 Acked 6149 Read 1 New Done Done true
15808 5151 Acked 6150 Write 9 New Done Done true
15809 5152 Acked 6151 Write 7 New Done Done true
15810 5153 Acked 6152 Read 6 New Done Done true
15811 5154 Acked 6153 Read 7 New Done Done true
15812 5155 Acked 6154 Read 7 New Done Done true
15813 5156 Acked 6155 Write 2 New Done Done true
15814 5157 Acked 6156 Read 8 New Done Done true
15815 5158 Acked 6157 Write 10 New Done Done true
15816 5159 Acked 6158 Flush 0 New Done Done true
15817 5160 Acked 6159 Write 4 New Done Done true
15818 5161 Acked 6160 Write 1 New Done Done true
15819 5162 Acked 6161 Read 10 New Done Done true
15820 5163 Acked 6162 Write 3 New Done Done true
15821 5164 Acked 6163 Write 1 New Done Done true
15822 5165 Acked 6164 Flush 0 New Done Done true
15823 5166 Acked 6165 Write 1 New Done Done true
15824 5167 Acked 6166 Read 9 New Done Done true
15825 5168 Acked 6167 Read 6 New Done Done true
15826 5169 Acked 6168 Read 8 New Done Done true
15827 5170 Acked 6169 Read 4 New Done Done true
15828 5171 Acked 6170 Write 4 New Done Done true
15829 5172 Acked 6171 Flush 0 New Done Done true
15830 5173 Acked 6172 Write 3 New Done Done true
15831 5174 Acked 6173 Read 9 New Done Done true
15832 5175 Acked 6174 Read 6 New Done Done true
15833 5176 Acked 6175 Write 2 New Done Done true
15834 5177 Acked 6176 Read 4 New Done Done true
15835 5178 Acked 6177 Write 9 New Done Done true
15836 5179 Acked 6178 Write 8 New Done Done true
15837 5180 Acked 6179 Write 7 New Done Done true
15838 5181 Acked 6180 Read 7 New Done Done true
15839 5182 Acked 6181 Flush 0 New Done Done true
15840 5183 Acked 6182 Write 4 New Done Done true
15841 5184 Acked 6183 Write 10 New Done Done true
15842 5185 Acked 6184 Flush 0 New Done Done true
15843 5186 Acked 6185 Read 6 New Done Done true
15844 5187 Acked 6186 Read 5 New Done Done true
15845 5188 Acked 6187 Write 4 New Done Done true
15846 5189 Acked 6188 Read 2 New Done Done true
15847 5190 Acked 6189 Write 6 New Done Done true
15848 5191 Acked 6190 Write 1 New Done Done true
15849 5192 Acked 6191 Write 6 New Done Done true
15850 5193 Acked 6192 Flush 0 New Done Done true
15851 5194 Acked 6193 Read 9 New Done Done true
15852 5195 Acked 6194 Flush 0 New Done Done true
15853 5196 Acked 6195 Write 5 New Done Done true
15854 5197 Acked 6196 Read 10 New Done Done true
15855 5198 Acked 6197 Read 6 New Done Done true
15856 5199 Acked 6198 Write 3 New Done Done true
15857 5200 Acked 6199 Flush 0 New Done Done true
15858 5201 Acked 6200 Read 8 New Done Done true
15859 5202 Acked 6201 Read 6 New Done Done true
15860 5203 Acked 6202 Read 4 New Done Done true
15861 5204 Acked 6203 Write 1 New Done Done true
15862 5205 Acked 6204 Read 5 New Done Done true
15863 5206 Acked 6205 Write 4 New Done Done true
15864 5207 Acked 6206 Write 10 New Done Done true
15865 5208 Acked 6207 Write 8 New Done Done true
15866 5209 Acked 6208 Write 6 New Done Done true
15867 5210 Acked 6209 Read 5 New Done Done true
15868 5211 Acked 6210 Read 6 New Done Done true
15869 5212 Acked 6211 Write 5 New Done Done true
15870 5213 Acked 6212 Read 5 New Done Done true
15871 5214 Acked 6213 Read 3 New Done Done true
15872 5215 Acked 6214 Read 7 New Done Done true
15873 5216 Acked 6215 Read 7 New Done Done true
15874 5217 Acked 6216 Write 7 New Done Done true
15875 5218 Acked 6217 Read 8 New Done Done true
15876 5219 Acked 6218 Write 5 New Done Done true
15877 5220 Acked 6219 Read 3 New Done Done true
15878 5221 Acked 6220 Read 5 New Done Done true
15879 5222 Acked 6221 Flush 0 New Done Done true
15880 5223 Acked 6222 Read 5 New Done Done true
15881 5224 Acked 6223 Read 10 New Done Done true
15882 5225 Acked 6224 Read 1 New Done Done true
15883 5226 Acked 6225 Read 7 New Done Done true
15884 5227 Acked 6226 Read 3 New Done Done true
15885 5228 Acked 6227 Write 6 New Done Done true
15886 5229 Acked 6228 Read 8 New Done Done true
15887 5230 Acked 6229 Read 1 New Done Done true
15888 5231 Acked 6230 Read 2 New Done Done true
15889 5232 Acked 6231 Write 8 New Done Done true
15890 5233 Acked 6232 Read 2 New Done Done true
15891 5234 Acked 6233 Read 4 New Done Done true
15892 5235 Acked 6234 Write 4 New Done Done true
15893 5236 Acked 6235 Write 8 New Done Done true
15894 5237 Acked 6236 Read 3 New Done Done true
15895 5238 Acked 6237 Write 3 New Done Done true
15896 5239 Acked 6238 Read 8 New Done Done true
15897 5240 Acked 6239 Read 7 New Done Done true
15898 5241 Acked 6240 Write 9 New Done Done true
15899 5242 Acked 6241 Write 8 New Done Done true
15900 5243 Acked 6242 Write 8 New Done Done true
15901 5244 Acked 6243 Read 9 New Done Done true
15902 5245 Acked 6244 Read 8 New Done Done true
15903 5246 Acked 6245 Read 2 New Done Done true
15904 5247 Acked 6246 Write 10 New Done Done true
15905 5248 Acked 6247 Read 4 New Done Done true
15906 5249 Acked 6248 Read 7 New Done Done true
15907 5250 Acked 6249 Read 7 New Done Done true
15908 5251 Acked 6250 Write 6 New Done Done true
15909 5252 Acked 6251 Write 2 New Done Done true
15910 5253 Acked 6252 Read 4 New Done Done true
15911 5254 Acked 6253 Write 8 New Done Done true
15912 5255 Acked 6254 Flush 0 New Done Done true
15913 5256 Acked 6255 Write 1 New Done Done true
15914 5257 Acked 6256 Read 6 New Done Done true
15915 5258 Acked 6257 Read 1 New Done Done true
15916 5259 Acked 6258 Write 8 New Done Done true
15917 5260 Acked 6259 Read 5 New Done Done true
15918 5261 Acked 6260 Read 8 New Done Done true
15919 5262 Acked 6261 Read 6 New Done Done true
15920 5263 Acked 6262 Read 7 New Done Done true
15921 5264 Acked 6263 Write 2 New Done Done true
15922 5265 Acked 6264 Read 5 New Done Done true
15923 5266 Acked 6265 Read 1 New Done Done true
15924 5267 Acked 6266 Write 2 New Done Done true
15925 5268 Acked 6267 Read 9 New Done Done true
15926 5269 Acked 6268 Read 3 New Done Done true
15927 5270 Acked 6269 Read 4 New Done Done true
15928 5271 Acked 6270 Write 6 New Done Done true
15929 5272 Acked 6271 Read 3 New Done Done true
15930 5273 Acked 6272 Read 5 New Done Done true
15931 5274 Acked 6273 Read 5 New Done Done true
15932 5275 Acked 6274 Read 6 New Done Done true
15933 5276 Acked 6275 Read 8 New Done Done true
15934 5277 Acked 6276 Read 7 New Done Done true
15935 5278 Acked 6277 Read 2 New Done Done true
15936 5279 Acked 6278 Write 9 New Done Done true
15937 5280 Acked 6279 Write 3 New Done Done true
15938 5281 Acked 6280 Flush 0 New Done Done true
15939 5282 Acked 6281 Read 3 New Done Done true
15940 5283 Acked 6282 Write 8 New Done Done true
15941 5284 Acked 6283 Write 2 New Done Done true
15942 5285 Acked 6284 Flush 0 New Done Done true
15943 5286 Acked 6285 Read 1 New Done Done true
15944 5287 Acked 6286 Read 10 New Done Done true
15945 5288 Acked 6287 Read 7 New Done Done true
15946 5289 Acked 6288 Read 2 New Done Done true
15947 5290 Acked 6289 Read 2 New Done Done true
15948 5291 Acked 6290 Write 10 New Done Done true
15949 5292 Acked 6291 Read 5 New Done Done true
15950 5293 Acked 6292 Write 8 New Done Done true
15951 5294 Acked 6293 Write 9 New Done Done true
15952 5295 Acked 6294 Read 5 New Done Done true
15953 5296 Acked 6295 Write 8 New Done Done true
15954 5297 Acked 6296 Write 2 New Done Done true
15955 5298 Acked 6297 Read 8 New Done Done true
15956 5299 Acked 6298 Read 6 New Done Done true
15957 5300 Acked 6299 Flush 0 New Done Done true
15958 5301 Acked 6300 Read 5 New Done Done true
15959 5302 Acked 6301 Read 6 New Done Done true
15960 5303 Acked 6302 Read 2 New Done Done true
15961 5304 Acked 6303 Read 4 New Done Done true
15962 5305 Acked 6304 Read 8 New Done Done true
15963 5306 Acked 6305 Read 4 New Done Done true
15964 5307 Acked 6306 Write 2 New Done Done true
15965 5308 Acked 6307 Flush 0 New Done Done true
15966 5309 Acked 6308 Read 10 New Done Done true
15967 5310 Acked 6309 Read 5 New Done Done true
15968 5311 Acked 6310 Read 4 New Done Done true
15969 5312 Acked 6311 Write 6 New Done Done true
15970 5313 Acked 6312 Write 1 New Done Done true
15971 5314 Acked 6313 Write 10 New Done Done true
15972 5315 Acked 6314 Write 8 New Done Done true
15973 5316 Acked 6315 Read 3 New Done Done true
15974 5317 Acked 6316 Read 3 New Done Done true
15975 5318 Acked 6317 Flush 0 New Done Done true
15976 5319 Acked 6318 Read 5 New Done Done true
15977 5320 Acked 6319 Read 3 New Done Done true
15978 5321 Acked 6320 Read 6 New Done Done true
15979 5322 Acked 6321 Write 5 New Done Done true
15980 5323 Acked 6322 Flush 0 New Done Done true
15981 5324 Acked 6323 Write 1 New Done Done true
15982 5325 Acked 6324 Read 7 New Done Done true
15983 5326 Acked 6325 Flush 0 New Done Done true
15984 5327 Acked 6326 Flush 0 New Done Done true
15985 5328 Acked 6327 Read 8 New Done Done true
15986 5329 Acked 6328 Write 1 New Done Done true
15987 5330 Acked 6329 Read 7 New Done Done true
15988 5331 Acked 6330 Write 5 New Done Done true
15989 5332 Acked 6331 Write 5 New Done Done true
15990 5333 Acked 6332 Write 10 New Done Done true
15991 5334 Acked 6333 Read 2 New Done Done true
15992 5335 Acked 6334 Write 3 New Done Done true
15993 5336 Acked 6335 Read 7 New Done Done true
15994 5337 Acked 6336 Read 10 New Done Done true
15995 5338 Acked 6337 Read 3 New Done Done true
15996 5339 Acked 6338 Read 6 New Done Done true
15997 5340 Acked 6339 Read 1 New Done Done true
15998 5341 Acked 6340 Read 7 New Done Done true
15999 5342 Acked 6341 Write 9 New Done Done true
16000 5343 Acked 6342 Read 4 New Done Done true
16001 5344 Acked 6343 Flush 0 New Done Done true
16002 5345 Acked 6344 Write 10 New Done Done true
16003 5346 Acked 6345 Read 6 New Done Done true
16004 5347 Acked 6346 Read 5 New Done Done true
16005 5348 Acked 6347 Read 6 New Done Done true
16006 5349 Acked 6348 Write 10 New Done Done true
16007 5350 Acked 6349 Read 9 New Done Done true
16008 5351 Acked 6350 Flush 0 New Done Done true
16009 5352 Acked 6351 Read 4 New Done Done true
16010 5353 Acked 6352 Read 1 New Done Done true
16011 5354 Acked 6353 Write 8 New Done Done true
16012 5355 Acked 6354 Write 2 New Done Done true
16013 5356 Acked 6355 Read 6 New Done Done true
16014 5357 Acked 6356 Read 1 New Done Done true
16015 5358 Acked 6357 Read 7 New Done Done true
16016 5359 Acked 6358 Write 6 New Done Done true
16017 5360 Acked 6359 Read 3 New Done Done true
16018 5361 Acked 6360 Read 3 New Done Done true
16019 5362 Acked 6361 Read 4 New Done Done true
16020 5363 Acked 6362 Read 9 New Done Done true
16021 5364 Acked 6363 Read 6 New Done Done true
16022 5365 Acked 6364 Flush 0 New Done Done true
16023 5366 Acked 6365 Write 10 New Done Done true
16024 5367 Acked 6366 Write 1 New Done Done true
16025 5368 Acked 6367 Flush 0 New Done Done true
16026 5369 Acked 6368 Read 1 New Done Done true
16027 5370 Acked 6369 Flush 0 New Done Done true
16028 5371 Acked 6370 Read 4 New Done Done true
16029 5372 Acked 6371 Write 2 New Done Done true
16030 5373 Acked 6372 Write 9 New Done Done true
16031 5374 Acked 6373 Read 6 New Done Done true
16032 5375 Acked 6374 Read 5 New Done Done true
16033 5376 Acked 6375 Read 8 New Done Done true
16034 5377 Acked 6376 Read 8 New Done Done true
16035 5378 Acked 6377 Write 2 New Done Done true
16036 5379 Acked 6378 Write 3 New Done Done true
16037 5380 Acked 6379 Write 2 New Done Done true
16038 5381 Acked 6380 Write 1 New Done Done true
16039 5382 Acked 6381 Write 5 New Done Done true
16040 5383 Acked 6382 Write 6 New Done Done true
16041 5384 Acked 6383 Read 1 New Done Done true
16042 5385 Acked 6384 Read 10 New Done Done true
16043 5386 Acked 6385 Read 3 New Done Done true
16044 5387 Acked 6386 Write 3 New Done Done true
16045 5388 Acked 6387 Write 10 New Done Done true
16046 5389 Acked 6388 Write 4 New Done Done true
16047 5390 Acked 6389 Flush 0 New Done Done true
16048 5391 Acked 6390 Read 7 New Done Done true
16049 5392 Acked 6391 Write 2 New Done Done true
16050 5393 Acked 6392 Write 10 New Done Done true
16051 5394 Acked 6393 Write 5 New Done Done true
16052 5395 Acked 6394 Read 1 New Done Done true
16053 5396 Acked 6395 Read 7 New Done Done true
16054 5397 Acked 6396 Read 6 New Done Done true
16055 5398 Acked 6397 Write 2 New Done Done true
16056 5399 Acked 6398 Write 2 New Done Done true
16057 5400 Acked 6399 Read 2 New Done Done true
16058 5401 Acked 6400 Read 8 New Done Done true
16059 5402 Acked 6401 Flush 0 New Done Done true
16060 5403 Acked 6402 Read 6 New Done Done true
16061 5404 Acked 6403 Flush 0 New Done Done true
16062 5405 Acked 6404 Write 4 New Done Done true
16063 5406 Acked 6405 Read 6 New Done Done true
16064 5407 Acked 6406 Read 5 New Done Done true
16065 5408 Acked 6407 Flush 0 New Done Done true
16066 5409 Acked 6408 Write 6 New Done Done true
16067 5410 Acked 6409 Read 8 New Done Done true
16068 5411 Acked 6410 Write 3 New Done Done true
16069 5412 Acked 6411 Read 9 New Done Done true
16070 5413 Acked 6412 Write 5 New Done Done true
16071 5414 Acked 6413 Read 8 New Done Done true
16072 5415 Acked 6414 Read 4 New Done Done true
16073 5416 Acked 6415 Write 3 New Done Done true
16074 5417 Acked 6416 Read 10 New Done Done true
16075 5418 Acked 6417 Read 5 New Done Done true
16076 5419 Acked 6418 Read 9 New Done Done true
16077 5420 Acked 6419 Read 7 New Done Done true
16078 5421 Acked 6420 Write 2 New Done Done true
16079 5422 Acked 6421 Read 1 New Done Done true
16080 5423 Acked 6422 Read 3 New Done Done true
16081 5424 Acked 6423 Flush 0 New Done Done true
16082 STATES DS:0 DS:1 DS:2 TOTAL
16083 New 301 0 0 301
16084 Sent 0 0 0 0
16085 Done 0 301 301 602
16086 Skipped 0 0 0 0
16087 Error 0 0 0 0
16088 Last Flush: 6122 6423 6423
16089 Downstairs last five completed: 6122 6121 6120 6119 6118
16090 Upstairs last five completed: 5424 5423 5422 5421 5420
16091 CLIENT: Up:0 ds:301 act:2
160922024-02-28T22:16:27.018ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
160932024-02-28T22:16:27.018ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
160942024-02-28T22:16:27.018ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
160952024-02-28T22:16:27.018ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
160962024-02-28T22:16:27.018ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
160972024-02-28T22:16:27.018ZINFOcrucible: [0] client re-new 301 jobs since flush 6122 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
160982024-02-28T22:16:27.018ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16099 [8810] got start action
16100 [8810] I am going to restart
16101 Starting downstairs at port 8810
16102 Make output file at "/tmp/dsc/downstairs-8810.txt"
16103 [8810][0] reports Starting
16104 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1182)
16105 [8810][0] reports Running
161062024-02-28T22:16:28.019ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161072024-02-28T22:16:28.019ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161082024-02-28T22:16:28.020ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161092024-02-28T22:16:28.020ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161102024-02-28T22:16:28.020ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161112024-02-28T22:16:28.020ZINFOcrucible: [0] client re-new 301 jobs since flush 6122 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161122024-02-28T22:16:28.020ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161132024-02-28T22:16:29.021ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161142024-02-28T22:16:29.022ZINFOcrucible: ds_connection connected = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161152024-02-28T22:16:29.023ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161162024-02-28T22:16:29.023ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161172024-02-28T22:16:29.023ZINFOcrucible: send last flush ID to this DS: 6122 = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161182024-02-28T22:16:29.023ZINFOcrucible: Replied this last flush ID: 6122 = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161192024-02-28T22:16:29.023ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161202024-02-28T22:16:29.023ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161212024-02-28T22:16:29.023ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161222024-02-28T22:16:29.023ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16123 ----------------------------------------------------------------
16124 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
16125 Downstairs last five completed: 6423 6422 6421 6420 6419
16126 Upstairs last five completed: 5424 5423 5422 5421 5420
16127 CLIENT: Up:0 ds:0 act:3
16128 Replay: All jobs finished, all DS active.
16129 Main task has work to do, go find it
16130 got dsc Stop(2)
16131 stop 2
16132 Feb 28 22:16:30.677 INFO request completed[8830] Got stop action so:false kr:false
16133 , latency_us: 372, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 8f8b13cd-78aa-4290-9e37-44fde18452f0, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
161342024-02-28T22:16:30.678ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161352024-02-28T22:16:30.678ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16136 Feb 28 22:16:30.678 INFO request completed, latency_us:{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-28T22:16:30.678681472Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
16137 145, response_code: 200, {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:16:30.678701171Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"client":"2","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
16138 uri: /state/cid/2, method{"msg":"[2] client re-new 0 jobs since flush 6423","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:16:30.67873976Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
16139 :{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:16:30.678818007Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
16140 GET, req_id: 335ef7fd-599f-4039-a506-c1c4b94d503c, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
16141 [8830] Exited with: ExitStatus(unix_wait_status(9))
16142 [8830][2] reports Exit
161432024-02-28T22:16:31.679ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161442024-02-28T22:16:31.679ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161452024-02-28T22:16:31.679ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161462024-02-28T22:16:31.679ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161472024-02-28T22:16:31.679ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161482024-02-28T22:16:31.679ZINFOcrucible: [2] client re-new 0 jobs since flush 6423 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161492024-02-28T22:16:31.679ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161502024-02-28T22:16:32.680ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161512024-02-28T22:16:32.680ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161522024-02-28T22:16:32.680ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161532024-02-28T22:16:32.680ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161542024-02-28T22:16:32.680ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161552024-02-28T22:16:32.680ZINFOcrucible: [2] client re-new 0 jobs since flush 6423 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161562024-02-28T22:16:32.680ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161572024-02-28T22:16:33.681ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161582024-02-28T22:16:33.681ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161592024-02-28T22:16:33.681ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161602024-02-28T22:16:33.681ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161612024-02-28T22:16:33.681ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161622024-02-28T22:16:33.681ZINFOcrucible: [2] client re-new 0 jobs since flush 6423 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161632024-02-28T22:16:33.681ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16164 Feb 28 22:16:34.681 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/2, method: GET, req_id: d800d7d7-12fa-4f70-8451-4309598a7185, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
16165 001/300 Read block 594 len 4096
161662024-02-28T22:16:34.682ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161672024-02-28T22:16:34.682ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161682024-02-28T22:16:34.682ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161692024-02-28T22:16:34.682ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161702024-02-28T22:16:34.682ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161712024-02-28T22:16:34.682ZINFOcrucible: [2] client re-new 1 jobs since flush 6423 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
161722024-02-28T22:16:34.682ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16173 002/300 Write block 2779 len 36864 data: 5 5 4 5 6 6 4 4 4
16174 003/300 Read block 1831 len 28672
16175 004/300 Write block 699 len 20480 data: 4 4 4 3 3
16176 005/300 Write block 1701 len 20480 data: 3 2 2 2 2
16177 006/300 Read block 867 len 36864
16178 007/300 Write block 2715 len 24576 data: 3 3 3 7 7 7
16179 008/300 Read block 1894 len 32768
16180 009/300 Read block 4110 len 20480
16181 010/300 Read block 3020 len 24576
16182 011/300 Read block 4805 len 28672
16183 012/300 Read block 443 len 12288
16184 013/300 Read block 2826 len 32768
16185 014/300 Write block 2539 len 8192 data: 6 6
16186 015/300 Write block 873 len 4096 data: 4
16187 016/300 Write block 4415 len 20480 data: 4 5 4 5 5
16188 017/300 Write block 3758 len 40960 data: 5 3 3 3 3 3 2 2 3 2
16189 018/300 Flush
16190 019/300 Write block 717 len 20480 data: 4 4 6 5 5
16191 020/300 Write block 2612 len 36864 data: 3 3 3 4 3 5 5 5 6
16192 021/300 Write block 4001 len 24576 data: 4 5 5 4 4 3
16193 022/300 Read block 4316 len 4096
16194 023/300 Read block 4669 len 20480
16195 024/300 Write block 226 len 24576 data: 4 2 3 3 3 3
16196 025/300 Read block 3336 len 4096
16197 026/300 Read block 2668 len 20480
16198 027/300 Write block 2130 len 40960 data: 5 6 6 6 4 3 4 4 3 4
16199 028/300 Read block 568 len 12288
16200 029/300 Read block 1011 len 24576
16201 030/300 Read block 2175 len 12288
16202 031/300 Write block 2044 len 20480 data: 2 3 3 3 3
16203 032/300 Read block 2935 len 32768
16204 033/300 Read block 3755 len 12288
16205 034/300 Write block 2768 len 24576 data: 4 3 3 3 4 4
16206 035/300 Read block 3833 len 32768
16207 036/300 Read block 1987 len 4096
16208 037/300 Write block 394 len 24576 data: 4 3 3 4 4 4
16209 038/300 Read block 3589 len 32768
16210 039/300 Flush
16211 040/300 Read block 4743 len 20480
16212 041/300 Flush
16213 042/300 Read block 3938 len 28672
16214 043/300 Write block 183 len 8192 data: 4 4
16215 044/300 Read block 423 len 12288
16216 045/300 Flush
16217 046/300 Read block 4428 len 12288
16218 047/300 Read block 2755 len 8192
16219 048/300 Read block 2446 len 4096
16220 049/300 Write block 67 len 4096 data: 5
16221 050/300 Read block 1518 len 36864
16222 051/300 Read block 2396 len 12288
16223 052/300 Write block 2769 len 16384 data: 4 4 4 5
16224 053/300 Read block 2318 len 12288
16225 054/300 Read block 3870 len 32768
16226 055/300 Write block 867 len 32768 data: 4 4 4 6 5 4 5 4
16227 056/300 Write block 2336 len 36864 data: 6 8 8 8 6 5 4 4 2
16228 057/300 Write block 4050 len 20480 data: 4 5 5 4 2
16229 058/300 Flush
16230 059/300 Read block 3152 len 28672
16231 060/300 Write block 4260 len 20480 data: 4 6 6 6 5
16232 061/300 Read block 3377 len 24576
16233 062/300 Read block 1384 len 40960
16234 063/300 Read block 4670 len 20480
16235 064/300 Read block 1926 len 28672
16236 065/300 Flush
16237 066/300 Flush
16238 067/300 Flush
16239 068/300 Flush
16240 069/300 Write block 3855 len 12288 data: 6 5 6
16241 070/300 Write block 2677 len 8192 data: 6 6
16242 071/300 Write block 4159 len 28672 data: 4 5 4 4 4 4 5
16243 072/300 Read block 787 len 20480
16244 073/300 Write block 2515 len 32768 data: 5 7 7 7 4 3 3 4
16245 074/300 Flush
16246 075/300 Read block 4387 len 8192
16247 076/300 Write block 654 len 8192 data: 5 5
16248 077/300 Write block 4827 len 8192 data: 4 4
16249 078/300 Read block 692 len 36864
16250 079/300 Read block 31 len 4096
16251 080/300 Write block 4688 len 24576 data: 4 3 3 4 5 5
16252 081/300 Write block 4221 len 36864 data: 3 3 4 5 6 6 6 6 5
16253 082/300 Flush
16254 083/300 Write block 2671 len 4096 data: 3
16255 084/300 Write block 3390 len 32768 data: 4 5 6 6 7 7 6 5
16256 085/300 Read block 103 len 20480
16257 086/300 Read block 2377 len 8192
16258 087/300 Read block 940 len 36864
16259 088/300 Write block 3047 len 12288 data: 5 5 5
16260 089/300 Read block 137 len 4096
16261 090/300 Read block 4971 len 32768
16262 091/300 Read block 715 len 16384
16263 092/300 Write block 1704 len 12288 data: 3 3 2
16264 093/300 Read block 2944 len 20480
16265 094/300 Read block 3560 len 28672
16266 095/300 Write block 2340 len 32768 data: 7 6 5 5 3 2 3 3
16267 096/300 Read block 3605 len 28672
16268 097/300 Read block 325 len 4096
16269 098/300 Read block 1823 len 4096
16270 099/300 Flush
16271 100/300 Flush
16272 101/300 Read block 1804 len 36864
16273 102/300 Read block 4553 len 4096
16274 103/300 Write block 3821 len 4096 data: 6
16275 104/300 Write block 4737 len 16384 data: 5 4 4 4
16276 105/300 Read block 2292 len 12288
16277 106/300 Write block 2226 len 8192 data: 2 2
16278 107/300 Flush
16279 108/300 Write block 752 len 8192 data: 7 6
16280 109/300 Read block 1075 len 36864
16281 110/300 Write block 186 len 20480 data: 6 5 4 4 3
16282 111/300 Read block 123 len 8192
16283 112/300 Flush
16284 113/300 Write block 1507 len 8192 data: 4 3
16285 114/300 Write block 1540 len 36864 data: 3 4 4 5 3 3 4 4 4
16286 115/300 Read block 4167 len 8192
16287 116/300 Read block 4193 len 24576
16288 117/300 Read block 717 len 20480
16289 118/300 Read block 3119 len 36864
16290 119/300 Write block 1407 len 4096 data: 4
16291 120/300 Write block 3624 len 40960 data: 6 6 6 6 6 6 6 6 5 4
16292 121/300 Flush
16293 122/300 Read block 134 len 8192
16294 123/300 Read block 1487 len 16384
16295 124/300 Write block 235 len 4096 data: 6
16296 125/300 Read block 808 len 20480
16297 126/300 Write block 2179 len 36864 data: 5 5 3 3 3 3 3 3 3
16298 127/300 Write block 2847 len 4096 data: 4
16299 128/300 Read block 814 len 20480
16300 129/300 Write block 618 len 36864 data: 4 4 3 3 3 4 4 4 4
16301 130/300 Read block 3669 len 24576
16302 131/300 Flush
16303 132/300 Write block 4304 len 36864 data: 9 7 6 7 7 4 4 5 4
16304 133/300 Write block 2466 len 36864 data: 5 5 5 6 6 5 5 4 5
16305 134/300 Read block 622 len 16384
16306 135/300 Flush
16307 136/300 Write block 4543 len 20480 data: 6 6 6 5 6
16308 137/300 Read block 4954 len 40960
16309 138/300 Read block 4337 len 36864
16310 139/300 Write block 785 len 20480 data: 5 5 5 4 3
16311 140/300 Read block 6 len 36864
16312 141/300 Write block 4544 len 4096 data: 7
16313 142/300 Read block 681 len 8192
16314 143/300 Flush
16315 144/300 Write block 4679 len 16384 data: 3 2 4 3
16316 145/300 Write block 2118 len 20480 data: 4 4 3 3 3
16317 146/300 Read block 1385 len 36864
16318 147/300 Read block 4615 len 16384
16319 148/300 Read block 4409 len 16384
16320 149/300 Read block 2914 len 36864
16321 150/300 Read block 1641 len 20480
16322 151/300 Read block 2632 len 32768
16323 152/300 Read block 2645 len 32768
16324 153/300 Flush
16325 154/300 Write block 4815 len 16384 data: 4 3 3 3
16326 155/300 Read block 4587 len 28672
16327 156/300 Write block 4019 len 40960 data: 6 6 3 2 2 2 2 2 2 2
16328 157/300 Flush
16329 158/300 Write block 2128 len 36864 data: 6 5 6 7 7 7 5 4 5
16330 159/300 Flush
16331 160/300 Read block 3949 len 12288
16332 161/300 Read block 4904 len 16384
16333 162/300 Read block 353 len 36864
16334 163/300 Write block 3602 len 4096 data: 4
16335 164/300 Write block 4210 len 20480 data: 4 4 4 4 3
16336 165/300 Write block 4304 len 8192 data: 10 8
16337 166/300 Write block 892 len 8192 data: 3 3
16338 167/300 Read block 3810 len 8192
16339 168/300 Write block 1347 len 20480 data: 4 4 6 6 6
16340 169/300 Read block 3148 len 24576
16341 170/300 Read block 3219 len 36864
16342 171/300 Read block 1713 len 24576
16343 172/300 Write block 1691 len 8192 data: 5 4
16344 173/300 Write block 143 len 28672 data: 6 6 6 7 5 6 5
16345 174/300 Write block 154 len 32768 data: 5 4 4 4 4 5 7 5
16346 175/300 Read block 1521 len 40960
16347 176/300 Read block 1552 len 12288
16348 177/300 Read block 2468 len 20480
16349 178/300 Write block 1256 len 40960 data: 5 4 4 7 7 5 5 4 3 3
16350 179/300 Read block 722 len 8192
16351 180/300 Write block 4877 len 36864 data: 4 5 5 4 4 5 6 6 6
16352 181/300 Write block 4342 len 16384 data: 3 3 2 3
16353 182/300 Write block 2195 len 40960 data: 5 3 3 2 3 4 5 6 5 4
16354 183/300 Read block 2350 len 28672
16355 184/300 Write block 3033 len 40960 data: 4 4 6 6 6 5 5 6 6 6
16356 185/300 Write block 4612 len 40960 data: 5 3 3 4 4 4 4 4 5 4
16357 186/300 Read block 4899 len 28672
16358 187/300 Read block 2413 len 40960
16359 188/300 Write block 3258 len 28672 data: 3 3 3 3 3 3 3
16360 189/300 Read block 160 len 32768
16361 190/300 Read block 2752 len 28672
16362 191/300 Read block 696 len 24576
16363 192/300 Write block 4249 len 24576 data: 5 5 6 7 7 9
16364 193/300 Read block 4863 len 40960
16365 194/300 Read block 4382 len 4096
16366 195/300 Read block 1736 len 8192
16367 196/300 Read block 3399 len 24576
16368 197/300 Read block 2098 len 8192
16369 198/300 Read block 1973 len 40960
16370 199/300 Write block 2655 len 16384 data: 4 2 2 2
16371 200/300 Write block 4582 len 16384 data: 7 6 4 3
16372 201/300 Write block 2202 len 16384 data: 7 6 5 5
16373 202/300 Read block 4205 len 24576
16374 203/300 Write block 687 len 24576 data: 5 5 4 4 4 5
16375 204/300 Write block 1150 len 16384 data: 5 5 5 5
16376 205/300 Write block 571 len 20480 data: 4 4 4 4 6
16377 206/300 Write block 4619 len 20480 data: 5 6 5 4 4
16378 207/300 Write block 2019 len 20480 data: 2 2 3 3 4
16379 208/300 Read block 3811 len 40960
16380 209/300 Flush
16381 210/300 Read block 1481 len 28672
16382 211/300 Read block 4137 len 28672
16383 212/300 Read block 4354 len 40960
16384 213/300 Read block 1390 len 8192
16385 214/300 Write block 2467 len 32768 data: 6 6 7 7 6 6 5 6
16386 215/300 Read block 3173 len 28672
16387 216/300 Read block 3450 len 4096
16388 217/300 Read block 3145 len 28672
16389 218/300 Read block 1108 len 40960
16390 219/300 Write block 4532 len 24576 data: 5 4 4 4 5 5
16391 220/300 Write block 66 len 12288 data: 6 6 6
16392 221/300 Write block 4719 len 16384 data: 5 3 3 3
16393 222/300 Read block 2003 len 24576
16394 223/300 Write block 602 len 4096 data: 5
16395 224/300 Write block 3243 len 28672 data: 4 4 4 4 5 4 5
16396 225/300 Write block 4135 len 28672 data: 4 4 6 5 4 6 6
16397 226/300 Write block 4373 len 24576 data: 2 2 2 2 2 2
16398 227/300 Read block 4749 len 24576
16399 228/300 Write block 3607 len 8192 data: 3 3
16400 229/300 Read block 629 len 20480
16401 230/300 Read block 591 len 32768
16402 231/300 Write block 2066 len 32768 data: 7 6 6 6 6 6 6 7
16403 232/300 Read block 1426 len 12288
16404 233/300 Read block 4037 len 28672
16405 234/300 Write block 429 len 36864 data: 2 4 4 4 6 5 5 5 5
16406 235/300 Write block 4985 len 24576 data: 7 6 6 5 5 5
16407 236/300 Flush
16408 237/300 Write block 2453 len 28672 data: 4 3 4 4 4 5 5
16409 238/300 Read block 1754 len 40960
16410 239/300 Read block 3329 len 4096
16411 240/300 Write block 4137 len 28672 data: 7 6 5 7 7 6 5
16412 241/300 Read block 2289 len 20480
16413 242/300 Read block 3712 len 24576
16414 243/300 Read block 1446 len 4096
16415 244/300 Write block 4476 len 8192 data: 4 4
16416 245/300 Write block 1343 len 32768 data: 3 3 4 4 5 5 7 7
16417 246/300 Write block 4665 len 32768 data: 3 3 3 3 3 3 3 4
16418 247/300 Write block 432 len 4096 data: 5
16419 248/300 Read block 4549 len 28672
16420 249/300 Read block 2941 len 28672
16421 250/300 Read block 1251 len 4096
16422 251/300 Read block 3900 len 16384
16423 252/300 Write block 478 len 40960 data: 6 6 6 5 4 4 6 4 4 4
16424 253/300 Write block 1713 len 28672 data: 7 8 8 9 8 7 4
16425 254/300 Read block 2779 len 24576
16426 255/300 Read block 1680 len 12288
16427 256/300 Write block 2390 len 24576 data: 5 5 5 4 4 5
16428 257/300 Write block 1899 len 36864 data: 7 8 7 7 7 6 7 8 7
16429 258/300 Write block 3249 len 20480 data: 6 5 5 5 5
16430 259/300 Flush
16431 260/300 Read block 2475 len 32768
16432 261/300 Flush
16433 262/300 Read block 3154 len 12288
16434 263/300 Write block 4878 len 24576 data: 6 6 5 5 6 7
16435 264/300 Read block 2625 len 40960
16436 265/300 Write block 4104 len 24576 data: 6 5 6 6 5 4
16437 266/300 Read block 3200 len 32768
16438 267/300 Read block 1182 len 24576
16439 268/300 Read block 1011 len 20480
16440 269/300 Flush
16441 270/300 Read block 2636 len 8192
16442 271/300 Read block 4740 len 36864
16443 272/300 Write block 4599 len 36864 data: 4 5 5 4 4 4 4 5 5
16444 273/300 Read block 3810 len 4096
16445 274/300 Read block 4073 len 20480
16446 275/300 Write block 3642 len 36864 data: 6 5 5 5 5 4 6 6 5
16447 276/300 Write block 4122 len 36864 data: 4 4 3 3 3 5 5 5 4
16448 277/300 Write block 4428 len 40960 data: 4 4 4 5 6 6 5 6 7 5
16449 278/300 Read block 708 len 12288
16450 279/300 Read block 924 len 4096
16451 280/300 Write block 4712 len 32768 data: 5 5 5 5 5 4 5 6
16452 281/300 Write block 3592 len 36864 data: 4 4 3 5 4 4 4 4 4
16453 282/300 Read block 1335 len 28672
16454 283/300 Write block 2733 len 12288 data: 4 4 3
16455 284/300 Read block 2442 len 36864
16456 285/300 Read block 2665 len 36864
16457 286/300 Write block 185 len 40960 data: 3 7 6 5 5 4 3 3 3 4
16458 287/300 Write block 4717 len 8192 data: 5 6
16459 288/300 Write block 4472 len 40960 data: 2 2 3 3 5 5 4 4 4 3
16460 289/300 Flush
16461 290/300 Read block 1241 len 28672
16462 291/300 Read block 4734 len 32768
16463 292/300 Write block 3584 len 40960 data: 4 5 5 5 5 6 6 5 5 5
16464 293/300 Write block 4393 len 40960 data: 2 2 2 3 3 3 4 5 6 6
16465 294/300 Read block 1795 len 4096
16466 295/300 Read block 685 len 24576
16467 296/300 Write block 3166 len 8192 data: 4 4
16468 297/300 Write block 2743 len 24576 data: 3 3 3 3 2 3
16469 298/300 Write block 390 len 28672 data: 4 4 5 5 5 4 4
16470 299/300 Read block 3562 len 20480
16471 300/300 Read block 2545 len 28672
16472 Main task has work to do, go find it
16473 got dsc Start(2)
16474 start 2
16475 Feb 28 22:16:35.022 INFO request completed, latency_us: 208, response_code: 204, uri: /start/cid/2, method: POST, req_id: 22321215-9e46-4d5f-9e41-1ad55f59d958, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
16476 Replay: started 2, returned:Ok(())
16477 ----------------------------------------------------------------
16478 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16479 States: Active Active Offline
16480 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
16481 5425 Acked 6424 Read 1 Done Done New true
16482 5426 Acked 6425 Write 9 Done Done New false
16483 5427 Acked 6426 Read 7 Done Done New false
16484 5428 Acked 6427 Write 5 Done Done New false
16485 5429 Acked 6428 Write 5 Done Done New false
16486 5430 Acked 6429 Read 9 Done Done New false
16487 5431 Acked 6430 Write 6 Done Done New false
16488 5432 Acked 6431 Read 8 Done Done New false
16489 5433 Acked 6432 Read 5 Done Done New false
16490 5434 Acked 6433 Read 6 Done Done New false
16491 5435 Acked 6434 Read 7 Done Done New false
16492 5436 Acked 6435 Read 3 Done Done New false
16493 5437 Acked 6436 Read 8 Done Done New false
16494 5438 Acked 6437 Write 2 Done Done New false
16495 5439 Acked 6438 Write 1 Done Done New false
16496 5440 Acked 6439 Write 5 Done Done New false
16497 5441 Acked 6440 Write 10 Done Done New false
16498 5442 Acked 6441 Flush 0 Done Done New false
16499 5443 Acked 6442 Write 5 Done Done New false
16500 5444 Acked 6443 Write 9 Done Done New false
16501 5445 Acked 6444 Write 6 Done Done New false
16502 5446 Acked 6445 Read 1 Done Done New false
16503 5447 Acked 6446 Read 5 Done Done New false
16504 5448 Acked 6447 Write 6 Done Done New false
16505 5449 Acked 6448 Read 1 Done Done New false
16506 5450 Acked 6449 Read 5 Done Done New false
16507 5451 Acked 6450 Write 10 Done Done New false
16508 5452 Acked 6451 Read 3 Done Done New false
16509 5453 Acked 6452 Read 6 Done Done New false
16510 5454 Acked 6453 Read 3 Done Done New false
16511 5455 Acked 6454 Write 5 Done Done New false
16512 5456 Acked 6455 Read 8 Done Done New false
16513 5457 Acked 6456 Read 3 Done Done New false
16514 5458 Acked 6457 Write 6 Done Done New false
16515 5459 Acked 6458 Read 8 Done Done New false
16516 5460 Acked 6459 Flush 0 Done Done New false
16517 5461 Acked 6460 Read 1 Done Done New false
16518 5462 Acked 6461 Write 6 Done Done New false
16519 5463 Acked 6462 Read 8 Done Done New false
16520 5464 Acked 6463 Flush 0 Done Done New false
16521 5465 Acked 6464 Read 5 Done Done New false
16522 5466 Acked 6465 Flush 0 Done Done New false
16523 5467 Acked 6466 Read 7 Done Done New false
16524 5468 Acked 6467 Write 2 Done Done New false
16525 5469 Acked 6468 Read 3 Done Done New false
16526 5470 Acked 6469 Flush 0 Done Done New false
16527 5471 Acked 6470 Read 3 Done Done New false
16528 5472 Acked 6471 Read 2 Done Done New false
16529 5473 Acked 6472 Read 1 Done Done New false
16530 5474 Acked 6473 Write 1 Done Done New false
16531 5475 Acked 6474 Read 9 Done Done New false
16532 5476 Acked 6475 Read 3 Done Done New false
16533 5477 Acked 6476 Write 4 Done Done New false
16534 5478 Acked 6477 Read 3 Done Done New false
16535 5479 Acked 6478 Read 8 Done Done New false
16536 5480 Acked 6479 Write 8 Done Done New false
16537 5481 Acked 6480 Write 9 Done Done New false
16538 5482 Acked 6481 Write 5 Done Done New false
16539 5483 Acked 6482 Flush 0 Done Done New false
16540 5484 Acked 6483 Read 7 Done Done New false
16541 5485 Acked 6484 Write 5 Done Done New false
16542 5486 Acked 6485 Read 6 Done Done New false
16543 5487 Acked 6486 Read 10 Done Done New false
16544 5488 Acked 6487 Read 5 Done Done New false
16545 5489 Acked 6488 Read 7 Done Done New false
16546 5490 Acked 6489 Flush 0 Done Done New false
16547 5491 Acked 6490 Flush 0 Done Done New false
16548 5492 Acked 6491 Flush 0 Done Done New false
16549 5493 Acked 6492 Flush 0 Done Done New false
16550 5494 Acked 6493 Write 3 Done Done New false
16551 5495 Acked 6494 Write 2 Done Done New false
16552 5496 Acked 6495 Write 7 Done Done New false
16553 5497 Acked 6496 Read 5 Done Done New false
16554 5498 Acked 6497 Write 8 Done Done New false
16555 5499 Acked 6498 Flush 0 Done Done New false
16556 5500 Acked 6499 Read 2 Done Done New false
16557 5501 Acked 6500 Write 2 Done Done New false
16558 5502 Acked 6501 Write 2 Done Done New false
16559 5503 Acked 6502 Read 9 Done Done New false
16560 5504 Acked 6503 Read 1 Done Done New false
16561 5505 Acked 6504 Write 6 Done Done New false
16562 5506 Acked 6505 Write 9 Done Done New false
16563 5507 Acked 6506 Flush 0 Done Done New false
16564 5508 Acked 6507 Write 1 Done Done New false
16565 5509 Acked 6508 Write 8 Done Done New false
16566 5510 Acked 6509 Read 5 Done Done New false
16567 5511 Acked 6510 Read 2 Done Done New false
16568 5512 Acked 6511 Read 9 Done Done New false
16569 5513 Acked 6512 Write 3 Done Done New false
16570 5514 Acked 6513 Read 1 Done Done New false
16571 5515 Acked 6514 Read 8 Done Done New false
16572 5516 Acked 6515 Read 4 Done Done New false
16573 5517 Acked 6516 Write 3 Done Done New false
16574 5518 Acked 6517 Read 5 Done Done New false
16575 5519 Acked 6518 Read 7 Done Done New false
16576 5520 Acked 6519 Write 8 Done Done New false
16577 5521 Acked 6520 Read 7 Done Done New false
16578 5522 Acked 6521 Read 1 Done Done New false
16579 5523 Acked 6522 Read 1 Done Done New false
16580 5524 Acked 6523 Flush 0 Done Done New false
16581 5525 Acked 6524 Flush 0 Done Done New false
16582 5526 Acked 6525 Read 9 Done Done New false
16583 5527 Acked 6526 Read 1 Done Done New false
16584 5528 Acked 6527 Write 1 Done Done New false
16585 5529 Acked 6528 Write 4 Done Done New false
16586 5530 Acked 6529 Read 3 Done Done New false
16587 5531 Acked 6530 Write 2 Done Done New false
16588 5532 Acked 6531 Flush 0 Done Done New false
16589 5533 Acked 6532 Write 2 Done Done New false
16590 5534 Acked 6533 Read 9 Done Done New false
16591 5535 Acked 6534 Write 5 Done Done New false
16592 5536 Acked 6535 Read 2 Done Done New false
16593 5537 Acked 6536 Flush 0 Done Done New false
16594 5538 Acked 6537 Write 2 Done Done New false
16595 5539 Acked 6538 Write 9 Done Done New false
16596 5540 Acked 6539 Read 2 Done Done New false
16597 5541 Acked 6540 Read 6 Done Done New false
16598 5542 Acked 6541 Read 5 Done Done New false
16599 5543 Acked 6542 Read 9 Done Done New false
16600 5544 Acked 6543 Write 1 Done Done New false
16601 5545 Acked 6544 Write 10 Done Done New false
16602 5546 Acked 6545 Flush 0 Done Done New false
16603 5547 Acked 6546 Read 2 Done Done New false
16604 5548 Acked 6547 Read 4 Done Done New false
16605 5549 Acked 6548 Write 1 Done Done New false
16606 5550 Acked 6549 Read 5 Done Done New false
16607 5551 Acked 6550 Write 9 Done Done New false
16608 5552 Acked 6551 Write 1 Done Done New false
16609 5553 Acked 6552 Read 5 Done Done New false
16610 5554 Acked 6553 Write 9 Done Done New false
16611 5555 Acked 6554 Read 6 Done Done New false
16612 5556 Acked 6555 Flush 0 Done Done New false
16613 5557 Acked 6556 Write 9 Done Done New false
16614 5558 Acked 6557 Write 9 Done Done New false
16615 5559 Acked 6558 Read 4 Done Done New false
16616 5560 Acked 6559 Flush 0 Done Done New false
16617 5561 Acked 6560 Write 5 Done Done New false
16618 5562 Acked 6561 Read 10 Done Done New false
16619 5563 Acked 6562 Read 9 Done Done New false
16620 5564 Acked 6563 Write 5 Done Done New false
16621 5565 Acked 6564 Read 9 Done Done New false
16622 5566 Acked 6565 Write 1 Done Done New false
16623 5567 Acked 6566 Read 2 Done Done New false
16624 5568 Acked 6567 Flush 0 Done Done New false
16625 5569 Acked 6568 Write 4 Done Done New false
16626 5570 Acked 6569 Write 5 Done Done New false
16627 5571 Acked 6570 Read 9 Done Done New false
16628 5572 Acked 6571 Read 4 Done Done New false
16629 5573 Acked 6572 Read 4 Done Done New false
16630 5574 Acked 6573 Read 9 Done Done New false
16631 5575 Acked 6574 Read 5 Done Done New false
16632 5576 Acked 6575 Read 8 Done Done New false
16633 5577 Acked 6576 Read 8 Done Done New false
16634 5578 Acked 6577 Flush 0 Done Done New false
16635 5579 Acked 6578 Write 4 Done Done New false
16636 5580 Acked 6579 Read 7 Done Done New false
16637 5581 Acked 6580 Write 10 Done Done New false
16638 5582 Acked 6581 Flush 0 Done Done New false
16639 5583 Acked 6582 Write 9 Done Done New false
16640 5584 Acked 6583 Flush 0 Done Done New false
16641 5585 Acked 6584 Read 3 Done Done New false
16642 5586 Acked 6585 Read 4 Done Done New false
16643 5587 Acked 6586 Read 9 Done Done New false
16644 5588 Acked 6587 Write 1 Done Done New false
16645 5589 Acked 6588 Write 5 Done Done New false
16646 5590 Acked 6589 Write 2 Done Done New false
16647 5591 Acked 6590 Write 2 Done Done New false
16648 5592 Acked 6591 Read 2 Done Done New false
16649 5593 Acked 6592 Write 5 Done Done New false
16650 5594 Acked 6593 Read 6 Done Done New false
16651 5595 Acked 6594 Read 9 Done Done New false
16652 5596 Acked 6595 Read 6 Done Done New false
16653 5597 Acked 6596 Write 2 Done Done New false
16654 5598 Acked 6597 Write 7 Done Done New false
16655 5599 Acked 6598 Write 8 Done Done New false
16656 5600 Acked 6599 Read 10 Done Done New false
16657 5601 Acked 6600 Read 3 Done Done New false
16658 5602 Acked 6601 Read 5 Done Done New false
16659 5603 Acked 6602 Write 10 Done Done New false
16660 5604 Acked 6603 Read 2 Done Done New false
16661 5605 Acked 6604 Write 9 Done Done New false
16662 5606 Acked 6605 Write 4 Done Done New false
16663 5607 Acked 6606 Write 10 Done Done New false
16664 5608 Acked 6607 Read 7 Done Done New false
16665 5609 Acked 6608 Write 10 Done Done New false
16666 5610 Acked 6609 Write 10 Done Done New false
16667 5611 Acked 6610 Read 7 Done Done New false
16668 5612 Acked 6611 Read 10 Done Done New false
16669 5613 Acked 6612 Write 7 Done Done New false
16670 5614 Acked 6613 Read 8 Done Done New false
16671 5615 Acked 6614 Read 7 Done Done New false
16672 5616 Acked 6615 Read 6 Done Done New false
16673 5617 Acked 6616 Write 6 Done Done New false
16674 5618 Acked 6617 Read 10 Done Done New false
16675 5619 Acked 6618 Read 1 Done Done New false
16676 5620 Acked 6619 Read 2 Done Done New false
16677 5621 Acked 6620 Read 6 Done Done New false
16678 5622 Acked 6621 Read 2 Done Done New false
16679 5623 Acked 6622 Read 10 Done Done New false
16680 5624 Acked 6623 Write 4 Done Done New false
16681 5625 Acked 6624 Write 4 Done Done New false
16682 5626 Acked 6625 Write 4 Done Done New false
16683 5627 Acked 6626 Read 6 Done Done New false
16684 5628 Acked 6627 Write 6 Done Done New false
16685 5629 Acked 6628 Write 4 Done Done New false
16686 5630 Acked 6629 Write 5 Done Done New false
16687 5631 Acked 6630 Write 5 Done Done New false
16688 5632 Acked 6631 Write 5 Done Done New false
16689 5633 Acked 6632 Read 10 Done Done New false
16690 5634 Acked 6633 Flush 0 Done Done New false
16691 5635 Acked 6634 Read 7 Done Done New false
16692 5636 Acked 6635 Read 7 Done Done New false
16693 5637 Acked 6636 Read 10 Done Done New false
16694 5638 Acked 6637 Read 2 Done Done New false
16695 5639 Acked 6638 Write 8 Done Done New false
16696 5640 Acked 6639 Read 7 Done Done New false
16697 5641 Acked 6640 Read 1 Done Done New false
16698 5642 Acked 6641 Read 7 Done Done New false
16699 5643 Acked 6642 Read 10 Done Done New false
16700 5644 Acked 6643 Write 6 Done Done New false
16701 5645 Acked 6644 Write 3 Done Done New false
16702 5646 Acked 6645 Write 4 Done Done New false
16703 5647 Acked 6646 Read 6 Done Done New false
16704 5648 Acked 6647 Write 1 Done Done New false
16705 5649 Acked 6648 Write 7 Done Done New false
16706 5650 Acked 6649 Write 7 Done Done New false
16707 5651 Acked 6650 Write 6 Done Done New false
16708 5652 Acked 6651 Read 6 Done Done New false
16709 5653 Acked 6652 Write 2 Done Done New false
16710 5654 Acked 6653 Read 5 Done Done New false
16711 5655 Acked 6654 Read 8 Done Done New false
16712 5656 Acked 6655 Write 8 Done Done New false
16713 5657 Acked 6656 Read 3 Done Done New false
16714 5658 Acked 6657 Read 7 Done Done New false
16715 5659 Acked 6658 Write 9 Done Done New false
16716 5660 Acked 6659 Write 6 Done Done New false
16717 5661 Acked 6660 Flush 0 Done Done New false
16718 5662 Acked 6661 Write 7 Done Done New false
16719 5663 Acked 6662 Read 10 Done Done New false
16720 5664 Acked 6663 Read 1 Done Done New false
16721 5665 Acked 6664 Write 7 Done Done New false
16722 5666 Acked 6665 Read 5 Done Done New false
16723 5667 Acked 6666 Read 6 Done Done New false
16724 5668 Acked 6667 Read 1 Done Done New false
16725 5669 Acked 6668 Write 2 Done Done New false
16726 5670 Acked 6669 Write 8 Done Done New false
16727 5671 Acked 6670 Write 8 Done Done New false
16728 5672 Acked 6671 Write 1 Done Done New false
16729 5673 Acked 6672 Read 7 Done Done New false
16730 5674 Acked 6673 Read 7 Done Done New false
16731 5675 Acked 6674 Read 1 Done Done New false
16732 5676 Acked 6675 Read 4 Done Done New false
16733 5677 Acked 6676 Write 10 Done Done New false
16734 5678 Acked 6677 Write 7 Done Done New false
16735 5679 Acked 6678 Read 6 Done Done New false
16736 5680 Acked 6679 Read 3 Done Done New false
16737 5681 Acked 6680 Write 6 Done Done New false
16738 5682 Acked 6681 Write 9 Done Done New false
16739 5683 Acked 6682 Write 5 Done Done New false
16740 5684 Acked 6683 Flush 0 Done Done New false
16741 5685 Acked 6684 Read 8 Done Done New false
16742 5686 Acked 6685 Flush 0 Done Done New false
16743 5687 Acked 6686 Read 3 Done Done New false
16744 5688 Acked 6687 Write 6 Done Done New false
16745 5689 Acked 6688 Read 10 Done Done New false
16746 5690 Acked 6689 Write 6 Done Done New false
16747 5691 Acked 6690 Read 8 Done Done New false
16748 5692 Acked 6691 Read 6 Done Done New false
16749 5693 Acked 6692 Read 5 Done Done New false
16750 5694 Acked 6693 Flush 0 Done Done New false
16751 5695 Acked 6694 Read 2 Done Done New false
16752 5696 Acked 6695 Read 9 Done Done New false
16753 5697 Acked 6696 Write 9 Done Done New false
16754 5698 Acked 6697 Read 1 Done Done New false
16755 5699 Acked 6698 Read 5 Done Done New false
16756 5700 Acked 6699 Write 9 Done Done New false
16757 5701 Acked 6700 Write 9 Done Done New false
16758 5702 Acked 6701 Write 10 Done Done New false
16759 5703 Acked 6702 Read 3 Done Done New false
16760 5704 Acked 6703 Read 1 Done Done New false
16761 5705 Acked 6704 Write 8 Done Done New false
16762 5706 Acked 6705 Write 9 Done Done New false
16763 5707 Acked 6706 Read 7 Done Done New false
16764 5708 Acked 6707 Write 3 Done Done New false
16765 5709 Acked 6708 Read 9 Done Done New false
16766 5710 Acked 6709 Read 9 Done Done New false
16767 5711 Acked 6710 Write 10 Done Done New false
16768 5712 Acked 6711 Write 2 Done Done New false
16769 5713 Acked 6712 Write 10 Done Done New false
16770 5714 Acked 6713 Flush 0 Done Done New false
16771 5715 Acked 6714 Read 7 Done Done New false
16772 5716 Acked 6715 Read 8 Done Done New false
16773 5717 Acked 6716 Write 10 Done Done New false
16774 5718 Acked 6717 Write 10 Done Done New false
16775 5719 Acked 6718 Read 1 Done Done New false
16776 5720 Acked 6719 Read 6 Done Done New false
16777 5721 Acked 6720 Write 2 Done Done New false
16778 5722 Acked 6721 Write 6 Done Done New false
16779 5723 Acked 6722 Write 7 Done Done New false
16780 5724 Acked 6723 Read 5 Done Done New false
16781 5725 Acked 6724 Read 7 Done Done New false
16782 STATES DS:0 DS:1 DS:2 TOTAL
16783 New 0 0 301 301
16784 Sent 0 0 0 0
16785 Done 301 301 0 602
16786 Skipped 0 0 0 0
16787 Error 0 0 0 0
16788 Last Flush: 6713 6713 6423
16789 Downstairs last five completed: 6423 6422 6421 6420 6419
16790 Upstairs last five completed: 5725 5724 5723 5722 5721
16791 CLIENT: Up:0 ds:301 act:2
167922024-02-28T22:16:35.683ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
167932024-02-28T22:16:35.683ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
167942024-02-28T22:16:35.683ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
167952024-02-28T22:16:35.683ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
167962024-02-28T22:16:35.683ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
167972024-02-28T22:16:35.684ZINFOcrucible: [2] client re-new 302 jobs since flush 6423 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
167982024-02-28T22:16:35.684ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
167992024-02-28T22:16:36.685ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168002024-02-28T22:16:36.685ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168012024-02-28T22:16:36.685ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168022024-02-28T22:16:36.685ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168032024-02-28T22:16:36.685ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168042024-02-28T22:16:36.685ZINFOcrucible: [2] client re-new 302 jobs since flush 6423 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168052024-02-28T22:16:36.685ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16806 [8830] got start action
16807 [8830] I am going to restart
16808 Starting downstairs at port 8830
16809 Make output file at "/tmp/dsc/downstairs-8830.txt"
16810 [8830][2] reports Starting
16811 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1183)
16812 [8830][2] reports Running
168132024-02-28T22:16:37.687ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168142024-02-28T22:16:37.687ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168152024-02-28T22:16:37.688ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168162024-02-28T22:16:37.688ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168172024-02-28T22:16:37.688ZINFOcrucible: send last flush ID to this DS: 6423 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168182024-02-28T22:16:37.688ZINFOcrucible: Replied this last flush ID: 6423 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168192024-02-28T22:16:37.689ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168202024-02-28T22:16:37.689ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168212024-02-28T22:16:37.689ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168222024-02-28T22:16:37.689ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16823 ----------------------------------------------------------------
16824 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
16825 Downstairs last five completed: 6725 6724 6723 6722 6721
16826 Upstairs last five completed: 5726 5725 5724 5723 5722
16827 CLIENT: Up:0 ds:0 act:3
16828 Replay: All jobs finished, all DS active.
16829 Main task has work to do, go find it
16830 got dsc Stop(2)
16831 stop 2
16832 Feb 28 22:16:39.025 INFO request completed[8830] Got stop action so:false kr:false
16833 , latency_us: 374, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 19cd11c9-76cc-4885-9faf-a75f9d8a4cc1, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
168342024-02-28T22:16:39.026ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168352024-02-28T22:16:39.026ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168362024-02-28T22:16:39.026ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168372024-02-28T22:16:39.026ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168382024-02-28T22:16:39.026ZINFOcrucible: [2] client re-new 0 jobs since flush 6725 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168392024-02-28T22:16:39.026ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16840 Feb 28 22:16:39.026 INFO request completed, latency_us: 151, response_code: 200, uri: /state/cid/2, method: GET, req_id: 49553223-9271-489d-b637-a47d3e4c0c5f, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
16841 [8830] Exited with: ExitStatus(unix_wait_status(9))
16842 [8830][2] reports Exit
168432024-02-28T22:16:40.027ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168442024-02-28T22:16:40.027ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168452024-02-28T22:16:40.027ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168462024-02-28T22:16:40.028ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168472024-02-28T22:16:40.028ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168482024-02-28T22:16:40.028ZINFOcrucible: [2] client re-new 0 jobs since flush 6725 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168492024-02-28T22:16:40.028ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168502024-02-28T22:16:41.029ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168512024-02-28T22:16:41.030ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168522024-02-28T22:16:41.030ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168532024-02-28T22:16:41.030ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168542024-02-28T22:16:41.030ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168552024-02-28T22:16:41.030ZINFOcrucible: [2] client re-new 0 jobs since flush 6725 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168562024-02-28T22:16:41.030ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168572024-02-28T22:16:42.031ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168582024-02-28T22:16:42.031ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168592024-02-28T22:16:42.031ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168602024-02-28T22:16:42.031ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168612024-02-28T22:16:42.031ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168622024-02-28T22:16:42.032ZINFOcrucible: [2] client re-new 0 jobs since flush 6725 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168632024-02-28T22:16:42.032ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16864 Feb 28 22:16:43.028 INFO request completed, latency_us: 322, response_code: 200, uri: /state/cid/2, method: GET, req_id: 0fc08a9d-6390-4193-99f3-39f348b391a2, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
16865 001/300 Write block 789 len 8192 data: 4 4
16866 002/300 Read block 1528 len 4096
16867 003/300 Write block 1329 len 12288 data: 6 5 6
16868 004/300 Flush
168692024-02-28T22:16:43.033ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168702024-02-28T22:16:43.033ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168712024-02-28T22:16:43.033ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168722024-02-28T22:16:43.033ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168732024-02-28T22:16:43.033ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168742024-02-28T22:16:43.033ZINFOcrucible: [2] client re-new 4 jobs since flush 6725 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
168752024-02-28T22:16:43.033ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
16876 005/300 Read block 2712 len 8192
16877 006/300 Write block 129 len 4096 data: 2
16878 007/300 Read block 2116 len 36864
16879 008/300 Read block 2673 len 36864
16880 009/300 Write block 2194 len 36864 data: 7 6 4 4 3 4 5 6 8
16881 010/300 Write block 3037 len 40960 data: 7 6 6 7 7 7 6 6 4 4
16882 011/300 Read block 1402 len 8192
16883 012/300 Write block 1414 len 24576 data: 4 6 6 7 7 8
16884 013/300 Read block 3415 len 28672
16885 014/300 Write block 220 len 16384 data: 4 4 5 4
16886 015/300 Write block 1751 len 20480 data: 6 5 7 7 6
16887 016/300 Read block 1997 len 40960
16888 017/300 Write block 3614 len 16384 data: 4 3 3 4
16889 018/300 Write block 1092 len 32768 data: 4 4 5 3 3 2 2 4
16890 019/300 Read block 3125 len 28672
16891 020/300 Write block 1048 len 12288 data: 5 5 5
16892 021/300 Read block 1674 len 8192
16893 022/300 Write block 3924 len 16384 data: 5 5 4 4
16894 023/300 Read block 2256 len 24576
16895 024/300 Write block 112 len 4096 data: 4
16896 025/300 Write block 3893 len 16384 data: 5 5 6 4
16897 026/300 Read block 3743 len 32768
16898 027/300 Write block 4412 len 36864 data: 4 5 5 5 6 5 6 6 3
16899 028/300 Read block 2850 len 28672
16900 029/300 Write block 2161 len 32768 data: 3 3 3 3 3 3 4 4
16901 030/300 Write block 2496 len 32768 data: 4 4 5 4 4 4 2 2
16902 031/300 Read block 3154 len 12288
16903 032/300 Flush
16904 033/300 Read block 3565 len 16384
16905 034/300 Read block 1152 len 40960
16906 035/300 Write block 2391 len 16384 data: 6 6 5 5
16907 036/300 Read block 2549 len 32768
16908 037/300 Flush
16909 038/300 Write block 2875 len 32768 data: 4 4 4 4 3 3 4 5
16910 039/300 Flush
16911 040/300 Read block 228 len 8192
16912 041/300 Read block 4722 len 32768
16913 042/300 Read block 590 len 8192
16914 043/300 Read block 263 len 12288
16915 044/300 Read block 1554 len 36864
16916 045/300 Read block 354 len 40960
16917 046/300 Read block 884 len 12288
16918 047/300 Read block 1662 len 12288
16919 048/300 Flush
16920 049/300 Write block 2722 len 28672 data: 7 6 6 6 4 5 5
16921 050/300 Read block 4606 len 40960
16922 051/300 Read block 2610 len 4096
16923 052/300 Read block 2632 len 8192
16924 053/300 Write block 688 len 32768 data: 6 5 5 5 6 5 4 3
16925 054/300 Write block 185 len 20480 data: 4 8 7 6 6
16926 055/300 Read block 3934 len 20480
16927 056/300 Read block 3570 len 28672
16928 057/300 Read block 1645 len 4096
16929 058/300 Read block 4662 len 12288
16930 059/300 Read block 2427 len 36864
16931 060/300 Read block 4503 len 40960
16932 061/300 Read block 2882 len 36864
16933 062/300 Flush
16934 063/300 Write block 3186 len 4096 data: 3
16935 064/300 Read block 335 len 40960
16936 065/300 Read block 3756 len 32768
16937 066/300 Read block 4540 len 32768
16938 067/300 Write block 305 len 16384 data: 4 4 4 4
16939 068/300 Write block 4805 len 8192 data: 6 6
16940 069/300 Read block 4260 len 24576
16941 070/300 Write block 1224 len 24576 data: 5 4 4 5 7 9
16942 071/300 Read block 2061 len 12288
16943 072/300 Write block 1686 len 24576 data: 6 6 7 7 7 6
16944 073/300 Write block 2218 len 4096 data: 5
16945 074/300 Write block 541 len 8192 data: 6 6
16946 075/300 Write block 3035 len 4096 data: 7
16947 076/300 Read block 440 len 16384
16948 077/300 Write block 188 len 32768 data: 7 7 5 4 4 4 5 4
16949 078/300 Read block 2981 len 24576
16950 079/300 Read block 4233 len 20480
16951 080/300 Read block 3900 len 16384
16952 081/300 Write block 1019 len 20480 data: 2 2 2 3 3
16953 082/300 Read block 4294 len 8192
16954 083/300 Write block 3062 len 36864 data: 9 9 5 5 4 3 3 4 3
16955 084/300 Read block 2173 len 40960
16956 085/300 Write block 415 len 32768 data: 7 7 8 7 7 8 6 5
16957 086/300 Read block 1272 len 4096
16958 087/300 Read block 2550 len 8192
16959 088/300 Read block 653 len 16384
16960 089/300 Read block 230 len 4096
16961 090/300 Write block 4437 len 4096 data: 6
16962 091/300 Flush
16963 092/300 Read block 243 len 16384
16964 093/300 Read block 2999 len 24576
16965 094/300 Flush
16966 095/300 Read block 4256 len 8192
16967 096/300 Read block 641 len 32768
16968 097/300 Write block 3565 len 28672 data: 3 4 3 4 4 4 4
16969 098/300 Read block 2852 len 12288
16970 099/300 Flush
16971 100/300 Write block 1005 len 8192 data: 4 3
16972 101/300 Write block 3073 len 20480 data: 5 5 5 5 6
16973 102/300 Read block 2350 len 24576
16974 103/300 Read block 650 len 24576
16975 104/300 Read block 1601 len 8192
16976 105/300 Write block 4371 len 40960 data: 3 2 3 3 3 3 3 3 2 3
16977 106/300 Write block 3585 len 36864 data: 6 6 6 6 7 7 6 6 6
16978 107/300 Read block 2004 len 32768
16979 108/300 Write block 2811 len 36864 data: 3 3 4 4 6 7 7 7 7
16980 109/300 Read block 1306 len 8192
16981 110/300 Read block 3196 len 32768
16982 111/300 Read block 4859 len 32768
16983 112/300 Read block 1893 len 16384
16984 113/300 Read block 4418 len 12288
16985 114/300 Read block 4511 len 12288
16986 115/300 Flush
16987 116/300 Read block 3543 len 36864
16988 117/300 Read block 1648 len 8192
16989 118/300 Read block 1494 len 32768
16990 119/300 Read block 3718 len 24576
16991 120/300 Flush
16992 121/300 Read block 3843 len 28672
16993 122/300 Read block 4761 len 16384
16994 123/300 Read block 1524 len 16384
16995 124/300 Write block 4862 len 28672 data: 7 7 7 6 5 6 5
16996 125/300 Write block 280 len 36864 data: 5 4 4 4 3 2 2 2 2
16997 126/300 Flush
16998 127/300 Read block 2277 len 32768
16999 128/300 Flush
17000 129/300 Write block 4019 len 40960 data: 7 7 4 3 3 3 3 3 3 3
17001 130/300 Write block 3981 len 4096 data: 4
17002 131/300 Write block 2484 len 32768 data: 2 3 3 3 3 3 3 2
17003 132/300 Read block 1703 len 24576
17004 133/300 Read block 347 len 24576
17005 134/300 Flush
17006 135/300 Read block 944 len 20480
17007 136/300 Read block 2824 len 16384
17008 137/300 Write block 4304 len 12288 data: 11 9 7
17009 138/300 Write block 637 len 20480 data: 3 4 4 4 4
17010 139/300 Read block 1557 len 40960
17011 140/300 Read block 2332 len 16384
17012 141/300 Write block 3347 len 24576 data: 2 2 2 2 2 2
17013 142/300 Write block 2211 len 32768 data: 5 4 4 5 4 4 4 6
17014 143/300 Write block 2065 len 12288 data: 5 8 7
17015 144/300 Read block 1220 len 16384
17016 145/300 Read block 2483 len 20480
17017 146/300 Read block 4583 len 4096
17018 147/300 Read block 4195 len 40960
17019 148/300 Write block 2335 len 20480 data: 5 7 9 9 9
17020 149/300 Write block 987 len 24576 data: 3 3 3 3 3 4
17021 150/300 Read block 3046 len 32768
17022 151/300 Write block 4233 len 28672 data: 3 3 3 3 4 6 6
17023 152/300 Read block 3876 len 8192
17024 153/300 Read block 4689 len 4096
17025 154/300 Write block 4780 len 8192 data: 3 4
17026 155/300 Read block 3016 len 16384
17027 156/300 Write block 831 len 40960 data: 3 3 3 3 3 3 2 2 3 3
17028 157/300 Write block 1623 len 40960 data: 7 8 9 6 4 4 5 5 4 4
17029 158/300 Read block 1947 len 8192
17030 159/300 Read block 64 len 28672
17031 160/300 Write block 1596 len 40960 data: 3 3 3 6 6 6 6 6 4 4
17032 161/300 Flush
17033 162/300 Read block 2042 len 12288
17034 163/300 Flush
17035 164/300 Read block 3232 len 16384
17036 165/300 Flush
17037 166/300 Write block 254 len 28672 data: 3 4 3 3 4 4 4
17038 167/300 Read block 2615 len 40960
17039 168/300 Write block 4448 len 16384 data: 3 3 4 4
17040 169/300 Write block 981 len 8192 data: 7 6
17041 170/300 Write block 4668 len 40960 data: 4 4 4 4 5 3 3 4 3 3
17042 171/300 Write block 4191 len 40960 data: 5 4 4 3 2 3 4 4 6 5
17043 172/300 Read block 1778 len 40960
17044 173/300 Write block 4291 len 32768 data: 3 3 3 3 3 3 3 3
17045 174/300 Read block 4162 len 24576
17046 175/300 Read block 3670 len 8192
17047 176/300 Write block 2830 len 20480 data: 2 3 3 3 3
17048 177/300 Write block 2529 len 8192 data: 5 3
17049 178/300 Read block 1168 len 12288
17050 179/300 Read block 3394 len 16384
17051 180/300 Read block 4467 len 12288
17052 181/300 Flush
17053 182/300 Read block 410 len 32768
17054 183/300 Read block 2737 len 16384
17055 184/300 Read block 3139 len 40960
17056 185/300 Write block 985 len 4096 data: 3
17057 186/300 Read block 4403 len 24576
17058 187/300 Read block 1017 len 40960
17059 188/300 Write block 4478 len 32768 data: 5 5 5 4 3 4 4 3
17060 189/300 Read block 504 len 28672
17061 190/300 Write block 3712 len 24576 data: 3 3 3 4 3 2
17062 191/300 Read block 283 len 24576
17063 192/300 Read block 1694 len 24576
17064 193/300 Write block 4215 len 12288 data: 2 2 3
17065 194/300 Write block 1501 len 4096 data: 2
17066 195/300 Write block 300 len 16384 data: 4 4 4 4
17067 196/300 Write block 1368 len 8192 data: 2 2
17068 197/300 Write block 1275 len 40960 data: 6 4 4 3 3 4 4 4 4 4
17069 198/300 Flush
17070 199/300 Write block 4321 len 40960 data: 5 4 3 3 2 2 3 4 5 7
17071 200/300 Read block 1569 len 8192
17072 201/300 Read block 2577 len 32768
17073 202/300 Read block 3333 len 40960
17074 203/300 Read block 3292 len 24576
17075 204/300 Read block 3396 len 28672
17076 205/300 Read block 1363 len 12288
17077 206/300 Write block 4086 len 28672 data: 3 3 6 7 7 8 7
17078 207/300 Write block 1362 len 8192 data: 3 3
17079 208/300 Write block 2178 len 24576 data: 5 6 6 4 4 4
17080 209/300 Read block 2707 len 36864
17081 210/300 Write block 3311 len 24576 data: 4 3 2 2 2 2
17082 211/300 Read block 745 len 16384
17083 212/300 Read block 2580 len 16384
17084 213/300 Read block 1219 len 28672
17085 214/300 Write block 4630 len 24576 data: 5 3 3 3 3 3
17086 215/300 Read block 540 len 36864
17087 216/300 Write block 3197 len 40960 data: 3 3 3 3 4 5 7 7 7 7
17088 217/300 Read block 3476 len 28672
17089 218/300 Write block 4612 len 28672 data: 6 4 4 5 5 5 5
17090 219/300 Write block 493 len 36864 data: 6 5 5 5 4 6 5 4 3
17091 220/300 Write block 1475 len 40960 data: 9 8 7 7 3 3 3 3 4 5
17092 221/300 Write block 4173 len 20480 data: 7 8 7 5 5
17093 222/300 Read block 4689 len 16384
17094 223/300 Read block 4824 len 24576
17095 224/300 Read block 1409 len 20480
17096 225/300 Write block 4864 len 8192 data: 8 7
17097 226/300 Read block 4462 len 8192
17098 227/300 Write block 3089 len 24576 data: 3 3 3 3 3 3
17099 228/300 Read block 4478 len 20480
17100 229/300 Read block 4490 len 40960
17101 230/300 Read block 945 len 24576
17102 231/300 Write block 4067 len 32768 data: 3 4 4 5 6 5 6 5
17103 232/300 Write block 232 len 40960 data: 4 4 7 7 5 4 3 3 2 3
17104 233/300 Write block 1607 len 36864 data: 4 5 4 4 5 5 6 6 8
17105 234/300 Flush
17106 235/300 Read block 731 len 20480
17107 236/300 Read block 3562 len 12288
17108 237/300 Write block 3013 len 12288 data: 5 7 6
17109 238/300 Read block 4736 len 8192
17110 239/300 Write block 3147 len 40960 data: 5 7 6 6 7 7 5 4 5 4
17111 240/300 Read block 3818 len 36864
17112 241/300 Read block 21 len 36864
17113 242/300 Read block 3808 len 4096
17114 243/300 Flush
17115 244/300 Write block 2167 len 8192 data: 5 5
17116 245/300 Read block 2790 len 16384
17117 246/300 Write block 4010 len 20480 data: 5 6 8 9 10
17118 247/300 Read block 3966 len 28672
17119 248/300 Write block 2285 len 36864 data: 4 5 4 3 3 3 2 2 3
17120 249/300 Read block 4685 len 40960
17121 250/300 Read block 1857 len 4096
17122 251/300 Write block 630 len 32768 data: 3 3 3 4 2 3 3 4
17123 252/300 Write block 23 len 16384 data: 6 7 4 4
17124 253/300 Write block 532 len 32768 data: 5 3 5 5 5 5 7 7
17125 254/300 Write block 2204 len 20480 data: 6 6 5 4 3
17126 255/300 Write block 2159 len 28672 data: 3 4 4 4 4 4 4
17127 256/300 Read block 11 len 8192
17128 257/300 Read block 3769 len 12288
17129 258/300 Read block 2330 len 12288
17130 259/300 Flush
17131 260/300 Read block 1357 len 20480
17132 261/300 Write block 1273 len 40960 data: 5 6 7 5 5 4 4 5 5 5
17133 262/300 Read block 1904 len 4096
17134 263/300 Flush
17135 264/300 Write block 4723 len 36864 data: 3 4 5 4 5 5 5 6 6
17136 265/300 Flush
17137 266/300 Read block 3763 len 36864
17138 267/300 Read block 1533 len 8192
17139 268/300 Read block 2505 len 32768
17140 269/300 Read block 3358 len 8192
17141 270/300 Write block 3965 len 36864 data: 2 2 2 2 2 5 7 5 5
17142 271/300 Read block 1562 len 12288
17143 272/300 Read block 3543 len 12288
17144 273/300 Write block 2502 len 24576 data: 3 3 2 2 2 3
17145 274/300 Flush
17146 275/300 Read block 3533 len 36864
17147 276/300 Write block 732 len 20480 data: 7 6 7 6 4
17148 277/300 Read block 4032 len 32768
17149 278/300 Read block 2126 len 40960
17150 279/300 Read block 432 len 4096
17151 280/300 Flush
17152 281/300 Read block 4649 len 28672
17153 282/300 Read block 2375 len 36864
17154 283/300 Read block 4740 len 16384
17155 284/300 Read block 473 len 4096
17156 285/300 Write block 3798 len 8192 data: 4 3
17157 286/300 Flush
17158 287/300 Read block 4069 len 32768
17159 288/300 Write block 670 len 16384 data: 7 6 5 5
17160 289/300 Write block 3251 len 32768 data: 6 6 6 5 3 5 3 4
17161 290/300 Read block 2915 len 24576
17162 291/300 Flush
17163 292/300 Read block 1540 len 40960
17164 293/300 Write block 2146 len 24576 data: 5 4 4 4 3 3
17165 294/300 Flush
17166 295/300 Flush
17167 296/300 Flush
17168 297/300 Read block 4138 len 12288
17169 298/300 Flush
17170 299/300 Read block 2069 len 32768
17171 300/300 Read block 1812 len 20480
17172 Main task has work to do, go find it
17173 got dsc Start(2)
17174 start 2
17175 Feb 28 22:16:43.366 INFO request completed, latency_us: 218, response_code: 204, uri: /start/cid/2, method: POST, req_id: db9c5ab6-7558-4965-ae4d-8a738678f6b1, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
17176 Replay: started 2, returned:Ok(())
17177 ----------------------------------------------------------------
17178 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17179 States: Active Active Offline
17180 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
17181 5727 Acked 6726 Write 2 Done Done New true
17182 5728 Acked 6727 Read 1 Done Done New true
17183 5729 Acked 6728 Write 3 Done Done New true
17184 5730 Acked 6729 Flush 0 Done Done New true
17185 5731 Acked 6730 Read 2 Done Done New false
17186 5732 Acked 6731 Write 1 Done Done New false
17187 5733 Acked 6732 Read 9 Done Done New false
17188 5734 Acked 6733 Read 9 Done Done New false
17189 5735 Acked 6734 Write 9 Done Done New false
17190 5736 Acked 6735 Write 10 Done Done New false
17191 5737 Acked 6736 Read 2 Done Done New false
17192 5738 Acked 6737 Write 6 Done Done New false
17193 5739 Acked 6738 Read 7 Done Done New false
17194 5740 Acked 6739 Write 4 Done Done New false
17195 5741 Acked 6740 Write 5 Done Done New false
17196 5742 Acked 6741 Read 10 Done Done New false
17197 5743 Acked 6742 Write 4 Done Done New false
17198 5744 Acked 6743 Write 8 Done Done New false
17199 5745 Acked 6744 Read 7 Done Done New false
17200 5746 Acked 6745 Write 3 Done Done New false
17201 5747 Acked 6746 Read 2 Done Done New false
17202 5748 Acked 6747 Write 4 Done Done New false
17203 5749 Acked 6748 Read 6 Done Done New false
17204 5750 Acked 6749 Write 1 Done Done New false
17205 5751 Acked 6750 Write 4 Done Done New false
17206 5752 Acked 6751 Read 8 Done Done New false
17207 5753 Acked 6752 Write 9 Done Done New false
17208 5754 Acked 6753 Read 7 Done Done New false
17209 5755 Acked 6754 Write 8 Done Done New false
17210 5756 Acked 6755 Write 8 Done Done New false
17211 5757 Acked 6756 Read 3 Done Done New false
17212 5758 Acked 6757 Flush 0 Done Done New false
17213 5759 Acked 6758 Read 4 Done Done New false
17214 5760 Acked 6759 Read 10 Done Done New false
17215 5761 Acked 6760 Write 4 Done Done New false
17216 5762 Acked 6761 Read 8 Done Done New false
17217 5763 Acked 6762 Flush 0 Done Done New false
17218 5764 Acked 6763 Write 8 Done Done New false
17219 5765 Acked 6764 Flush 0 Done Done New false
17220 5766 Acked 6765 Read 2 Done Done New false
17221 5767 Acked 6766 Read 8 Done Done New false
17222 5768 Acked 6767 Read 2 Done Done New false
17223 5769 Acked 6768 Read 3 Done Done New false
17224 5770 Acked 6769 Read 9 Done Done New false
17225 5771 Acked 6770 Read 10 Done Done New false
17226 5772 Acked 6771 Read 3 Done Done New false
17227 5773 Acked 6772 Read 3 Done Done New false
17228 5774 Acked 6773 Flush 0 Done Done New false
17229 5775 Acked 6774 Write 7 Done Done New false
17230 5776 Acked 6775 Read 10 Done Done New false
17231 5777 Acked 6776 Read 1 Done Done New false
17232 5778 Acked 6777 Read 2 Done Done New false
17233 5779 Acked 6778 Write 8 Done Done New false
17234 5780 Acked 6779 Write 5 Done Done New false
17235 5781 Acked 6780 Read 5 Done Done New false
17236 5782 Acked 6781 Read 7 Done Done New false
17237 5783 Acked 6782 Read 1 Done Done New false
17238 5784 Acked 6783 Read 3 Done Done New false
17239 5785 Acked 6784 Read 9 Done Done New false
17240 5786 Acked 6785 Read 10 Done Done New false
17241 5787 Acked 6786 Read 9 Done Done New false
17242 5788 Acked 6787 Flush 0 Done Done New false
17243 5789 Acked 6788 Write 1 Done Done New false
17244 5790 Acked 6789 Read 10 Done Done New false
17245 5791 Acked 6790 Read 8 Done Done New false
17246 5792 Acked 6791 Read 8 Done Done New false
17247 5793 Acked 6792 Write 4 Done Done New false
17248 5794 Acked 6793 Write 2 Done Done New false
17249 5795 Acked 6794 Read 6 Done Done New false
17250 5796 Acked 6795 Write 6 Done Done New false
17251 5797 Acked 6796 Read 3 Done Done New false
17252 5798 Acked 6797 Write 6 Done Done New false
17253 5799 Acked 6798 Write 1 Done Done New false
17254 5800 Acked 6799 Write 2 Done Done New false
17255 5801 Acked 6800 Write 1 Done Done New false
17256 5802 Acked 6801 Read 4 Done Done New false
17257 5803 Acked 6802 Write 8 Done Done New false
17258 5804 Acked 6803 Read 6 Done Done New false
17259 5805 Acked 6804 Read 5 Done Done New false
17260 5806 Acked 6805 Read 4 Done Done New false
17261 5807 Acked 6806 Write 5 Done Done New false
17262 5808 Acked 6807 Read 2 Done Done New false
17263 5809 Acked 6808 Write 9 Done Done New false
17264 5810 Acked 6809 Read 10 Done Done New false
17265 5811 Acked 6810 Write 8 Done Done New false
17266 5812 Acked 6811 Read 1 Done Done New false
17267 5813 Acked 6812 Read 2 Done Done New false
17268 5814 Acked 6813 Read 4 Done Done New false
17269 5815 Acked 6814 Read 1 Done Done New false
17270 5816 Acked 6815 Write 1 Done Done New false
17271 5817 Acked 6816 Flush 0 Done Done New false
17272 5818 Acked 6817 Read 4 Done Done New false
17273 5819 Acked 6818 Read 6 Done Done New false
17274 5820 Acked 6819 Flush 0 Done Done New false
17275 5821 Acked 6820 Read 2 Done Done New false
17276 5822 Acked 6821 Read 8 Done Done New false
17277 5823 Acked 6822 Write 7 Done Done New false
17278 5824 Acked 6823 Read 3 Done Done New false
17279 5825 Acked 6824 Flush 0 Done Done New false
17280 5826 Acked 6825 Write 2 Done Done New false
17281 5827 Acked 6826 Write 5 Done Done New false
17282 5828 Acked 6827 Read 6 Done Done New false
17283 5829 Acked 6828 Read 6 Done Done New false
17284 5830 Acked 6829 Read 2 Done Done New false
17285 5831 Acked 6830 Write 10 Done Done New false
17286 5832 Acked 6831 Write 9 Done Done New false
17287 5833 Acked 6832 Read 8 Done Done New false
17288 5834 Acked 6833 Write 9 Done Done New false
17289 5835 Acked 6834 Read 2 Done Done New false
17290 5836 Acked 6835 Read 8 Done Done New false
17291 5837 Acked 6836 Read 8 Done Done New false
17292 5838 Acked 6837 Read 4 Done Done New false
17293 5839 Acked 6838 Read 3 Done Done New false
17294 5840 Acked 6839 Read 3 Done Done New false
17295 5841 Acked 6840 Flush 0 Done Done New false
17296 5842 Acked 6841 Read 9 Done Done New false
17297 5843 Acked 6842 Read 2 Done Done New false
17298 5844 Acked 6843 Read 8 Done Done New false
17299 5845 Acked 6844 Read 6 Done Done New false
17300 5846 Acked 6845 Flush 0 Done Done New false
17301 5847 Acked 6846 Read 7 Done Done New false
17302 5848 Acked 6847 Read 4 Done Done New false
17303 5849 Acked 6848 Read 4 Done Done New false
17304 5850 Acked 6849 Write 7 Done Done New false
17305 5851 Acked 6850 Write 9 Done Done New false
17306 5852 Acked 6851 Flush 0 Done Done New false
17307 5853 Acked 6852 Read 8 Done Done New false
17308 5854 Acked 6853 Flush 0 Done Done New false
17309 5855 Acked 6854 Write 10 Done Done New false
17310 5856 Acked 6855 Write 1 Done Done New false
17311 5857 Acked 6856 Write 8 Done Done New false
17312 5858 Acked 6857 Read 6 Done Done New false
17313 5859 Acked 6858 Read 6 Done Done New false
17314 5860 Acked 6859 Flush 0 Done Done New false
17315 5861 Acked 6860 Read 5 Done Done New false
17316 5862 Acked 6861 Read 4 Done Done New false
17317 5863 Acked 6862 Write 3 Done Done New false
17318 5864 Acked 6863 Write 5 Done Done New false
17319 5865 Acked 6864 Read 10 Done Done New false
17320 5866 Acked 6865 Read 4 Done Done New false
17321 5867 Acked 6866 Write 6 Done Done New false
17322 5868 Acked 6867 Write 8 Done Done New false
17323 5869 Acked 6868 Write 3 Done Done New false
17324 5870 Acked 6869 Read 4 Done Done New false
17325 5871 Acked 6870 Read 5 Done Done New false
17326 5872 Acked 6871 Read 1 Done Done New false
17327 5873 Acked 6872 Read 10 Done Done New false
17328 5874 Acked 6873 Write 5 Done Done New false
17329 5875 Acked 6874 Write 6 Done Done New false
17330 5876 Acked 6875 Read 8 Done Done New false
17331 5877 Acked 6876 Write 7 Done Done New false
17332 5878 Acked 6877 Read 2 Done Done New false
17333 5879 Acked 6878 Read 1 Done Done New false
17334 5880 Acked 6879 Write 2 Done Done New false
17335 5881 Acked 6880 Read 4 Done Done New false
17336 5882 Acked 6881 Write 10 Done Done New false
17337 5883 Acked 6882 Write 10 Done Done New false
17338 5884 Acked 6883 Read 2 Done Done New false
17339 5885 Acked 6884 Read 7 Done Done New false
17340 5886 Acked 6885 Write 10 Done Done New false
17341 5887 Acked 6886 Flush 0 Done Done New false
17342 5888 Acked 6887 Read 3 Done Done New false
17343 5889 Acked 6888 Flush 0 Done Done New false
17344 5890 Acked 6889 Read 4 Done Done New false
17345 5891 Acked 6890 Flush 0 Done Done New false
17346 5892 Acked 6891 Write 7 Done Done New false
17347 5893 Acked 6892 Read 10 Done Done New false
17348 5894 Acked 6893 Write 4 Done Done New false
17349 5895 Acked 6894 Write 2 Done Done New false
17350 5896 Acked 6895 Write 10 Done Done New false
17351 5897 Acked 6896 Write 10 Done Done New false
17352 5898 Acked 6897 Read 10 Done Done New false
17353 5899 Acked 6898 Write 8 Done Done New false
17354 5900 Acked 6899 Read 6 Done Done New false
17355 5901 Acked 6900 Read 2 Done Done New false
17356 5902 Acked 6901 Write 5 Done Done New false
17357 5903 Acked 6902 Write 2 Done Done New false
17358 5904 Acked 6903 Read 3 Done Done New false
17359 5905 Acked 6904 Read 4 Done Done New false
17360 5906 Acked 6905 Read 3 Done Done New false
17361 5907 Acked 6906 Flush 0 Done Done New false
17362 5908 Acked 6907 Read 8 Done Done New false
17363 5909 Acked 6908 Read 4 Done Done New false
17364 5910 Acked 6909 Read 10 Done Done New false
17365 5911 Acked 6910 Write 1 Done Done New false
17366 5912 Acked 6911 Read 6 Done Done New false
17367 5913 Acked 6912 Read 10 Done Done New false
17368 5914 Acked 6913 Write 8 Done Done New false
17369 5915 Acked 6914 Read 7 Done Done New false
17370 5916 Acked 6915 Flush 0 Done Done New false
17371 5917 Acked 6916 Write 6 Done Done New false
17372 5918 Acked 6917 Read 6 Done Done New false
17373 5919 Acked 6918 Read 6 Done Done New false
17374 5920 Acked 6919 Write 3 Done Done New false
17375 5921 Acked 6920 Write 1 Done Done New false
17376 5922 Acked 6921 Write 4 Done Done New false
17377 5923 Acked 6922 Write 2 Done Done New false
17378 5924 Acked 6923 Write 10 Done Done New false
17379 5925 Acked 6924 Flush 0 Done Done New false
17380 5926 Acked 6925 Write 10 Done Done New false
17381 5927 Acked 6926 Read 2 Done Done New false
17382 5928 Acked 6927 Read 8 Done Done New false
17383 5929 Acked 6928 Read 10 Done Done New false
17384 5930 Acked 6929 Read 6 Done Done New false
17385 5931 Acked 6930 Read 7 Done Done New false
17386 5932 Acked 6931 Read 3 Done Done New false
17387 5933 Acked 6932 Write 7 Done Done New false
17388 5934 Acked 6933 Write 2 Done Done New false
17389 5935 Acked 6934 Write 6 Done Done New false
17390 5936 Acked 6935 Read 9 Done Done New false
17391 5937 Acked 6936 Write 6 Done Done New false
17392 5938 Acked 6937 Read 4 Done Done New false
17393 5939 Acked 6938 Read 4 Done Done New false
17394 5940 Acked 6939 Read 7 Done Done New false
17395 5941 Acked 6940 Write 6 Done Done New false
17396 5942 Acked 6941 Read 9 Done Done New false
17397 5943 Acked 6942 Write 10 Done Done New false
17398 5944 Acked 6943 Read 7 Done Done New false
17399 5945 Acked 6944 Write 7 Done Done New false
17400 5946 Acked 6945 Write 9 Done Done New false
17401 5947 Acked 6946 Write 10 Done Done New false
17402 5948 Acked 6947 Write 5 Done Done New false
17403 5949 Acked 6948 Read 4 Done Done New false
17404 5950 Acked 6949 Read 6 Done Done New false
17405 5951 Acked 6950 Read 5 Done Done New false
17406 5952 Acked 6951 Write 2 Done Done New false
17407 5953 Acked 6952 Read 2 Done Done New false
17408 5954 Acked 6953 Write 6 Done Done New false
17409 5955 Acked 6954 Read 5 Done Done New false
17410 5956 Acked 6955 Read 10 Done Done New false
17411 5957 Acked 6956 Read 6 Done Done New false
17412 5958 Acked 6957 Write 8 Done Done New false
17413 5959 Acked 6958 Write 10 Done Done New false
17414 5960 Acked 6959 Write 9 Done Done New false
17415 5961 Acked 6960 Flush 0 Done Done New false
17416 5962 Acked 6961 Read 5 Done Done New false
17417 5963 Acked 6962 Read 3 Done Done New false
17418 5964 Acked 6963 Write 3 Done Done New false
17419 5965 Acked 6964 Read 2 Done Done New false
17420 5966 Acked 6965 Write 10 Done Done New false
17421 5967 Acked 6966 Read 9 Done Done New false
17422 5968 Acked 6967 Read 9 Done Done New false
17423 5969 Acked 6968 Read 1 Done Done New false
17424 5970 Acked 6969 Flush 0 Done Done New false
17425 5971 Acked 6970 Write 2 Done Done New false
17426 5972 Acked 6971 Read 4 Done Done New false
17427 5973 Acked 6972 Write 5 Done Done New false
17428 5974 Acked 6973 Read 7 Done Done New false
17429 5975 Acked 6974 Write 9 Done Done New false
17430 5976 Acked 6975 Read 10 Done Done New false
17431 5977 Acked 6976 Read 1 Done Done New false
17432 5978 Acked 6977 Write 8 Done Done New false
17433 5979 Acked 6978 Write 4 Done Done New false
17434 5980 Acked 6979 Write 8 Done Done New false
17435 5981 Acked 6980 Write 5 Done Done New false
17436 5982 Acked 6981 Write 7 Done Done New false
17437 5983 Acked 6982 Read 2 Done Done New false
17438 5984 Acked 6983 Read 3 Done Done New false
17439 5985 Acked 6984 Read 3 Done Done New false
17440 5986 Acked 6985 Flush 0 Done Done New false
17441 5987 Acked 6986 Read 5 Done Done New false
17442 5988 Acked 6987 Write 10 Done Done New false
17443 5989 Acked 6988 Read 1 Done Done New false
17444 5990 Acked 6989 Flush 0 Done Done New false
17445 5991 Acked 6990 Write 9 Done Done New false
17446 5992 Acked 6991 Flush 0 Done Done New false
17447 5993 Acked 6992 Read 9 Done Done New false
17448 5994 Acked 6993 Read 2 Done Done New false
17449 5995 Acked 6994 Read 8 Done Done New false
17450 5996 Acked 6995 Read 2 Done Done New false
17451 5997 Acked 6996 Write 9 Done Done New false
17452 5998 Acked 6997 Read 3 Done Done New false
17453 5999 Acked 6998 Read 3 Done Done New false
17454 6000 Acked 6999 Write 6 Done Done New false
17455 6001 Acked 7000 Flush 0 Done Done New false
17456 6002 Acked 7001 Read 9 Done Done New false
17457 6003 Acked 7002 Write 5 Done Done New false
17458 6004 Acked 7003 Read 8 Done Done New false
17459 6005 Acked 7004 Read 10 Done Done New false
17460 6006 Acked 7005 Read 1 Done Done New false
17461 6007 Acked 7006 Flush 0 Done Done New false
17462 6008 Acked 7007 Read 7 Done Done New false
17463 6009 Acked 7008 Read 9 Done Done New false
17464 6010 Acked 7009 Read 4 Done Done New false
17465 6011 Acked 7010 Read 1 Done Done New false
17466 6012 Acked 7011 Write 2 Done Done New false
17467 6013 Acked 7012 Flush 0 Done Done New false
17468 6014 Acked 7013 Read 8 Done Done New false
17469 6015 Acked 7014 Write 4 Done Done New false
17470 6016 Acked 7015 Write 8 Done Done New false
17471 6017 Acked 7016 Read 6 Done Done New false
17472 6018 Acked 7017 Flush 0 Done Done New false
17473 6019 Acked 7018 Read 10 Done Done New false
17474 6020 Acked 7019 Write 6 Done Done New false
17475 6021 Acked 7020 Flush 0 Done Done New false
17476 6022 Acked 7021 Flush 0 Done Done New false
17477 6023 Acked 7022 Flush 0 Done Done New false
17478 6024 Acked 7023 Read 3 Done Done New false
17479 6025 Acked 7024 Flush 0 Done Done New false
17480 6026 Acked 7025 Read 8 Done Done New false
17481 6027 Acked 7026 Read 5 Done Done New false
17482 STATES DS:0 DS:1 DS:2 TOTAL
17483 New 0 0 301 301
17484 Sent 0 0 0 0
17485 Done 301 301 0 602
17486 Skipped 0 0 0 0
17487 Error 0 0 0 0
17488 Last Flush: 7024 7024 6725
17489 Downstairs last five completed: 6725 6724 6723 6722 6721
17490 Upstairs last five completed: 6027 6026 6025 6024 6023
17491 CLIENT: Up:0 ds:301 act:2
174922024-02-28T22:16:44.034ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
174932024-02-28T22:16:44.034ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
174942024-02-28T22:16:44.034ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
174952024-02-28T22:16:44.034ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
174962024-02-28T22:16:44.034ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
174972024-02-28T22:16:44.034ZINFOcrucible: [2] client re-new 302 jobs since flush 6725 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
174982024-02-28T22:16:44.034ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
17499 [8830] got start action
17500 [8830] I am going to restart
17501 Starting downstairs at port 8830
17502 Make output file at "/tmp/dsc/downstairs-8830.txt"
17503 [8830][2] reports Starting
175042024-02-28T22:16:45.035ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175052024-02-28T22:16:45.035ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175062024-02-28T22:16:45.035ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175072024-02-28T22:16:45.035ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175082024-02-28T22:16:45.035ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175092024-02-28T22:16:45.035ZINFOcrucible: [2] client re-new 302 jobs since flush 6725 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175102024-02-28T22:16:45.035ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
17511 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1184)
17512 [8830][2] reports Running
175132024-02-28T22:16:46.036ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175142024-02-28T22:16:46.037ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175152024-02-28T22:16:46.038ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175162024-02-28T22:16:46.038ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175172024-02-28T22:16:46.038ZINFOcrucible: send last flush ID to this DS: 6725 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175182024-02-28T22:16:46.038ZINFOcrucible: Replied this last flush ID: 6725 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175192024-02-28T22:16:46.038ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175202024-02-28T22:16:46.038ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175212024-02-28T22:16:46.038ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175222024-02-28T22:16:46.038ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
17523 ----------------------------------------------------------------
17524 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
17525 Downstairs last five completed: 7027 7026 7025 7024 7023
17526 Upstairs last five completed: 6028 6027 6026 6025 6024
17527 CLIENT: Up:0 ds:0 act:3
17528 Replay: All jobs finished, all DS active.
17529 Main task has work to do, go find it
17530 got dsc Stop(1)
17531 stop 1
17532 Feb 28 22:16:47.370 INFO request completed, latency_us: [8820] Got stop action so:false kr:false
17533 572, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 48c110ef-4bdf-4d18-a69f-ad383ef7ebe6, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
175342024-02-28T22:16:47.370ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175352024-02-28T22:16:47.370ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175362024-02-28T22:16:47.370ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175372024-02-28T22:16:47.370ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175382024-02-28T22:16:47.370ZINFOcrucible: [1] client re-new 0 jobs since flush 7027 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
17539 Feb 28 22:16:47.370 INFO request completed{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:16:47.370972798Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"io task","client":"1","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
17540 , latency_us: 155, response_code: 200, uri: /state/cid/1, method: GET, req_id: f3698c5b-bc5e-4d25-9b39-e69b309fb20b, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
17541 [8820] Exited with: ExitStatus(unix_wait_status(9))
17542 [8820][1] reports Exit
175432024-02-28T22:16:48.373ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175442024-02-28T22:16:48.373ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175452024-02-28T22:16:48.373ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175462024-02-28T22:16:48.373ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175472024-02-28T22:16:48.373ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175482024-02-28T22:16:48.373ZINFOcrucible: [1] client re-new 0 jobs since flush 7027 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175492024-02-28T22:16:48.373ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175502024-02-28T22:16:49.375ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175512024-02-28T22:16:49.375ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175522024-02-28T22:16:49.375ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175532024-02-28T22:16:49.375ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175542024-02-28T22:16:49.375ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175552024-02-28T22:16:49.375ZINFOcrucible: [1] client re-new 0 jobs since flush 7027 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175562024-02-28T22:16:49.375ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175572024-02-28T22:16:50.376ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175582024-02-28T22:16:50.376ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175592024-02-28T22:16:50.376ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175602024-02-28T22:16:50.376ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175612024-02-28T22:16:50.376ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175622024-02-28T22:16:50.376ZINFOcrucible: [1] client re-new 0 jobs since flush 7027 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175632024-02-28T22:16:50.376ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
17564 Feb 28 22:16:51.373 INFO request completed, latency_us: 218, response_code: 200, uri: /state/cid/1, method: GET, req_id: 54619291-d86a-42dd-a5c7-78939a743544, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
17565 001/300 Read block 1834 len 12288
17566 002/300 Write block 4737 len 20480 data: 6 5 5 5 2
17567 003/300 Flush
175682024-02-28T22:16:51.377ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175692024-02-28T22:16:51.377ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175702024-02-28T22:16:51.377ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175712024-02-28T22:16:51.377ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175722024-02-28T22:16:51.377ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175732024-02-28T22:16:51.377ZINFOcrucible: [1] client re-new 3 jobs since flush 7027 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
175742024-02-28T22:16:51.377ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
17575 004/300 Write block 1578 len 4096 data: 4
17576 005/300 Write block 4846 len 16384 data: 4 3 3 4
17577 006/300 Write block 466 len 24576 data: 4 6 7 7 6 5
17578 007/300 Flush
17579 008/300 Flush
17580 009/300 Read block 4198 len 4096
17581 010/300 Read block 4779 len 4096
17582 011/300 Read block 3783 len 24576
17583 012/300 Write block 2504 len 28672 data: 3 3 3 4 2 2 2
17584 013/300 Write block 991 len 4096 data: 4
17585 014/300 Write block 1852 len 20480 data: 4 3 3 5 6
17586 015/300 Read block 1240 len 36864
17587 016/300 Read block 2563 len 36864
17588 017/300 Read block 197 len 12288
17589 018/300 Read block 1189 len 20480
17590 019/300 Write block 3774 len 4096 data: 6
17591 020/300 Read block 3212 len 8192
17592 021/300 Read block 4018 len 40960
17593 022/300 Read block 433 len 12288
17594 023/300 Write block 1274 len 8192 data: 7 8
17595 024/300 Read block 404 len 28672
17596 025/300 Read block 693 len 36864
17597 026/300 Read block 625 len 12288
17598 027/300 Read block 2911 len 4096
17599 028/300 Read block 536 len 8192
17600 029/300 Write block 2989 len 4096 data: 6
17601 030/300 Read block 386 len 12288
17602 031/300 Read block 4424 len 12288
17603 032/300 Read block 1955 len 12288
17604 033/300 Read block 3131 len 40960
17605 034/300 Write block 2925 len 4096 data: 5
17606 035/300 Flush
17607 036/300 Read block 4907 len 40960
17608 037/300 Write block 1407 len 28672 data: 5 4 4 2 5 5 5
17609 038/300 Read block 3540 len 4096
17610 039/300 Write block 3125 len 16384 data: 4 3 3 2
17611 040/300 Read block 4765 len 28672
17612 041/300 Write block 1479 len 12288 data: 4 4 4
17613 042/300 Read block 4519 len 8192
17614 043/300 Write block 1181 len 28672 data: 4 4 3 4 4 3 4
17615 044/300 Write block 2336 len 16384 data: 8 10 10 10
17616 045/300 Read block 4795 len 20480
17617 046/300 Write block 4855 len 40960 data: 3 2 2 4 5 6 6 8 8 9
17618 047/300 Read block 2606 len 28672
17619 048/300 Write block 1584 len 12288 data: 7 7 8
17620 049/300 Write block 1205 len 24576 data: 5 3 3 3 3 4
17621 050/300 Write block 1062 len 28672 data: 4 4 5 4 5 5 4
17622 051/300 Write block 4344 len 16384 data: 3 4 3 5
17623 052/300 Read block 3771 len 36864
17624 053/300 Write block 438 len 8192 data: 6 6
17625 054/300 Read block 4515 len 28672
17626 055/300 Read block 2337 len 32768
17627 056/300 Flush
17628 057/300 Read block 2999 len 8192
17629 058/300 Read block 2717 len 32768
17630 059/300 Read block 4557 len 24576
17631 060/300 Read block 983 len 12288
17632 061/300 Read block 2730 len 8192
17633 062/300 Write block 317 len 28672 data: 6 6 5 5 4 4 4
17634 063/300 Read block 3962 len 20480
17635 064/300 Write block 41 len 12288 data: 4 2 2
17636 065/300 Write block 2558 len 32768 data: 4 3 3 2 2 5 3 3
17637 066/300 Read block 30 len 20480
17638 067/300 Read block 1040 len 8192
17639 068/300 Read block 469 len 20480
17640 069/300 Write block 4476 len 28672 data: 6 6 6 6 6 5 4
17641 070/300 Read block 4389 len 32768
17642 071/300 Flush
17643 072/300 Read block 4529 len 28672
17644 073/300 Write block 3753 len 8192 data: 5 5
17645 074/300 Write block 3059 len 20480 data: 5 7 7 10 10
17646 075/300 Read block 4981 len 4096
17647 076/300 Read block 2379 len 40960
17648 077/300 Flush
17649 078/300 Read block 2585 len 24576
17650 079/300 Flush
17651 080/300 Write block 2365 len 12288 data: 3 4 3
17652 081/300 Write block 4688 len 32768 data: 5 4 4 5 6 6 5 5
17653 082/300 Read block 4982 len 32768
17654 083/300 Read block 4400 len 24576
17655 084/300 Write block 4215 len 36864 data: 3 3 4 2 3 3 4 4 5
17656 085/300 Write block 3578 len 36864 data: 6 4 4 5 4 4 5 7 7
17657 086/300 Flush
17658 087/300 Read block 2601 len 20480
17659 088/300 Read block 4554 len 36864
17660 089/300 Read block 103 len 24576
17661 090/300 Read block 2007 len 36864
17662 091/300 Read block 4695 len 24576
17663 092/300 Write block 7 len 24576 data: 5 4 4 4 4 4
17664 093/300 Write block 794 len 32768 data: 5 5 4 3 3 3 4 4
17665 094/300 Write block 310 len 20480 data: 4 3 4 4 5
17666 095/300 Flush
17667 096/300 Flush
17668 097/300 Read block 727 len 24576
17669 098/300 Read block 2063 len 24576
17670 099/300 Write block 2667 len 28672 data: 3 2 2 2 4 4 5
17671 100/300 Read block 2107 len 20480
17672 101/300 Write block 4387 len 4096 data: 5
17673 102/300 Write block 66 len 8192 data: 7 7
17674 103/300 Flush
17675 104/300 Write block 266 len 16384 data: 3 2 4 3
17676 105/300 Read block 4172 len 36864
17677 106/300 Flush
17678 107/300 Read block 438 len 12288
17679 108/300 Write block 2794 len 32768 data: 5 5 4 4 5 6 6 5
17680 109/300 Read block 3645 len 28672
17681 110/300 Read block 130 len 28672
17682 111/300 Read block 168 len 8192
17683 112/300 Write block 3316 len 36864 data: 3 2 2 2 2 2 3 3 3
17684 113/300 Write block 534 len 28672 data: 6 6 6 6 8 8 7
17685 114/300 Read block 4376 len 4096
17686 115/300 Flush
17687 116/300 Read block 544 len 8192
17688 117/300 Write block 1516 len 28672 data: 6 6 9 7 7 6 6
17689 118/300 Read block 1201 len 40960
17690 119/300 Flush
17691 120/300 Write block 3331 len 20480 data: 3 3 3 3 4
17692 121/300 Read block 99 len 24576
17693 122/300 Read block 97 len 28672
17694 123/300 Read block 4343 len 24576
17695 124/300 Read block 3812 len 8192
17696 125/300 Read block 3879 len 24576
17697 126/300 Write block 2929 len 36864 data: 5 5 5 4 4 4 6 6 5
17698 127/300 Read block 2027 len 32768
17699 128/300 Flush
17700 129/300 Read block 240 len 16384
17701 130/300 Read block 4561 len 32768
17702 131/300 Write block 204 len 16384 data: 5 5 5 4
17703 132/300 Read block 1186 len 12288
17704 133/300 Read block 4548 len 8192
17705 134/300 Write block 3472 len 32768 data: 3 3 4 4 5 5 5 5
17706 135/300 Read block 4333 len 24576
17707 136/300 Read block 1074 len 28672
17708 137/300 Read block 43 len 40960
17709 138/300 Read block 4534 len 16384
17710 139/300 Write block 3505 len 28672 data: 5 5 4 3 3 3 4
17711 140/300 Flush
17712 141/300 Write block 4145 len 8192 data: 5 5
17713 142/300 Write block 4544 len 16384 data: 8 7 6 7
17714 143/300 Read block 756 len 12288
17715 144/300 Write block 3292 len 20480 data: 7 6 6 5 5
17716 145/300 Flush
17717 146/300 Write block 2130 len 4096 data: 7
17718 147/300 Read block 4830 len 12288
17719 148/300 Write block 2795 len 12288 data: 6 5 5
17720 149/300 Read block 1287 len 20480
17721 150/300 Write block 958 len 40960 data: 9 10 9 10 9 8 6 6 5 2
17722 151/300 Flush
17723 152/300 Write block 1969 len 8192 data: 3 3
17724 153/300 Read block 4617 len 32768
17725 154/300 Flush
17726 155/300 Read block 308 len 28672
17727 156/300 Write block 160 len 28672 data: 8 6 4 4 3 3 2
17728 157/300 Read block 2251 len 28672
17729 158/300 Write block 4582 len 16384 data: 8 7 5 4
17730 159/300 Write block 2484 len 32768 data: 3 4 4 4 4 4 4 3
17731 160/300 Read block 2863 len 24576
17732 161/300 Write block 2190 len 4096 data: 6
17733 162/300 Read block 4266 len 16384
17734 163/300 Flush
17735 164/300 Flush
17736 165/300 Write block 4722 len 28672 data: 4 4 5 6 5 6 6
17737 166/300 Read block 4267 len 20480
17738 167/300 Flush
17739 168/300 Flush
17740 169/300 Flush
17741 170/300 Read block 1023 len 40960
17742 171/300 Write block 3873 len 4096 data: 5
17743 172/300 Write block 1027 len 20480 data: 3 3 3 4 4
17744 173/300 Read block 266 len 32768
17745 174/300 Write block 2853 len 24576 data: 4 4 4 4 4 4
17746 175/300 Write block 4746 len 40960 data: 4 4 6 4 4 4 4 2 2 2
17747 176/300 Write block 4036 len 8192 data: 3 4
17748 177/300 Write block 1272 len 40960 data: 4 6 8 9 6 6 5 5 6 6
17749 178/300 Read block 415 len 24576
17750 179/300 Read block 726 len 8192
17751 180/300 Read block 3660 len 4096
17752 181/300 Flush
17753 182/300 Write block 3535 len 28672 data: 5 4 3 3 2 2 3
17754 183/300 Write block 4210 len 36864 data: 5 5 5 5 4 4 4 5 3
17755 184/300 Write block 3428 len 32768 data: 3 3 4 4 6 6 6 5
17756 185/300 Write block 2929 len 4096 data: 6
17757 186/300 Read block 3586 len 4096
17758 187/300 Write block 3494 len 24576 data: 3 3 3 3 3 4
17759 188/300 Read block 1128 len 16384
17760 189/300 Read block 3551 len 36864
17761 190/300 Read block 2252 len 12288
17762 191/300 Write block 4471 len 20480 data: 2 3 3 4 4
17763 192/300 Flush
17764 193/300 Read block 4337 len 20480
17765 194/300 Flush
17766 195/300 Write block 233 len 28672 data: 5 8 8 6 5 4 4
17767 196/300 Read block 1469 len 4096
17768 197/300 Write block 3860 len 36864 data: 5 5 5 4 4 3 3 3 3
17769 198/300 Write block 490 len 32768 data: 3 3 5 7 6 6 6 5
17770 199/300 Write block 2297 len 24576 data: 4 4 4 5 5 4
17771 200/300 Read block 3054 len 36864
17772 201/300 Write block 3586 len 24576 data: 8 7 7 8 8 7
17773 202/300 Write block 875 len 16384 data: 4 5 6 5
17774 203/300 Write block 3479 len 32768 data: 6 5 3 4 4 3 3 2
17775 204/300 Read block 3482 len 20480
17776 205/300 Read block 1693 len 36864
17777 206/300 Read block 2478 len 12288
17778 207/300 Write block 146 len 32768 data: 8 6 7 6 4 4 4 4
17779 208/300 Read block 2546 len 40960
17780 209/300 Write block 4336 len 8192 data: 8 5
17781 210/300 Read block 196 len 4096
17782 211/300 Read block 3801 len 16384
17783 212/300 Read block 1098 len 36864
17784 213/300 Read block 3733 len 20480
17785 214/300 Write block 2868 len 32768 data: 4 5 5 5 7 6 3 5
17786 215/300 Read block 748 len 20480
17787 216/300 Write block 1175 len 36864 data: 3 4 4 4 4 4 5 5 4
17788 217/300 Write block 3911 len 36864 data: 4 5 5 4 4 4 4 4 4
17789 218/300 Write block 3268 len 28672 data: 4 4 4 4 5 5 7
17790 219/300 Flush
17791 220/300 Write block 2087 len 28672 data: 3 4 3 4 5 5 5
17792 221/300 Write block 2704 len 12288 data: 3 4 4
17793 222/300 Write block 4285 len 32768 data: 5 4 4 4 4 3 4 4
17794 223/300 Write block 2518 len 28672 data: 8 5 4 4 5 4 4
17795 224/300 Read block 2662 len 28672
17796 225/300 Read block 4099 len 24576
17797 226/300 Write block 1771 len 40960 data: 8 6 5 5 5 4 4 4 4 5
17798 227/300 Write block 3740 len 20480 data: 8 6 7 6 5
17799 228/300 Write block 1318 len 16384 data: 4 3 5 4
17800 229/300 Write block 1403 len 4096 data: 6
17801 230/300 Flush
17802 231/300 Flush
17803 232/300 Read block 1467 len 16384
17804 233/300 Read block 1707 len 32768
17805 234/300 Read block 4475 len 12288
17806 235/300 Write block 675 len 16384 data: 5 4 4 5
17807 236/300 Write block 1303 len 12288 data: 2 2 2
17808 237/300 Read block 4835 len 40960
17809 238/300 Write block 4882 len 36864 data: 7 8 7 7 5 6 6 6 4
17810 239/300 Read block 2405 len 12288
17811 240/300 Read block 3983 len 36864
17812 241/300 Read block 4935 len 36864
17813 242/300 Write block 4382 len 20480 data: 5 5 6 5 5
17814 243/300 Read block 4477 len 4096
17815 244/300 Read block 2964 len 20480
17816 245/300 Write block 333 len 36864 data: 3 3 4 3 3 4 4 3 4
17817 246/300 Read block 814 len 8192
17818 247/300 Read block 547 len 20480
17819 248/300 Write block 678 len 8192 data: 6 5
17820 249/300 Read block 2566 len 16384
17821 250/300 Write block 1797 len 24576 data: 6 7 7 7 7 7
17822 251/300 Write block 2203 len 4096 data: 7
17823 252/300 Write block 3234 len 4096 data: 3
17824 253/300 Read block 1925 len 24576
17825 254/300 Read block 1661 len 24576
17826 255/300 Write block 915 len 24576 data: 5 5 5 8 7 8
17827 256/300 Read block 2132 len 16384
17828 257/300 Write block 3654 len 8192 data: 3 3
17829 258/300 Flush
17830 259/300 Read block 1015 len 24576
17831 260/300 Flush
17832 261/300 Read block 2508 len 40960
17833 262/300 Write block 4429 len 12288 data: 5 5 6
17834 263/300 Read block 2998 len 24576
17835 264/300 Write block 740 len 20480 data: 3 2 2 2 2
17836 265/300 Write block 3738 len 36864 data: 5 6 9 7 8 7 6 6 5
17837 266/300 Flush
17838 267/300 Read block 3103 len 28672
17839 268/300 Read block 4920 len 32768
17840 269/300 Write block 1961 len 32768 data: 5 5 5 5 5 3 4 3
17841 270/300 Read block 3334 len 20480
17842 271/300 Write block 1915 len 24576 data: 5 4 4 4 4 5
17843 272/300 Read block 172 len 32768
17844 273/300 Write block 4110 len 12288 data: 4 4 3
17845 274/300 Write block 1346 len 28672 data: 5 6 6 8 8 7 4
17846 275/300 Write block 1383 len 40960 data: 3 3 5 5 6 5 6 5 5 6
17847 276/300 Read block 3397 len 28672
17848 277/300 Read block 1965 len 12288
17849 278/300 Flush
17850 279/300 Write block 1979 len 4096 data: 8
17851 280/300 Write block 813 len 20480 data: 4 3 3 3 2
17852 281/300 Read block 1693 len 20480
17853 282/300 Write block 2720 len 24576 data: 8 8 8 7 7 7
17854 283/300 Read block 3716 len 36864
17855 284/300 Read block 1146 len 16384
17856 285/300 Read block 670 len 16384
17857 286/300 Read block 3389 len 16384
17858 287/300 Read block 2337 len 8192
17859 288/300 Write block 1232 len 12288 data: 5 5 5
17860 289/300 Write block 1715 len 32768 data: 9 10 9 8 5 6 4 7
17861 290/300 Write block 4018 len 12288 data: 6 8 8
17862 291/300 Read block 2423 len 40960
17863 292/300 Read block 4010 len 8192
17864 293/300 Read block 437 len 28672
17865 294/300 Write block 928 len 40960 data: 3 3 4 4 5 4 4 5 4 4
17866 295/300 Write block 2719 len 4096 data: 8
17867 296/300 Write block 2292 len 32768 data: 3 4 4 4 4 5 5 5
17868 297/300 Write block 1313 len 40960 data: 3 3 3 4 4 5 4 6 5 5
17869 298/300 Read block 1066 len 20480
17870 299/300 Write block 4701 len 32768 data: 6 5 4 4 3 3 2 2
17871 300/300 Write block 4482 len 28672 data: 5 5 5 4 2 2 2
17872 Main task has work to do, go find it
17873 got dsc Start(1)
17874 start 1
17875 Feb 28 22:16:51.705 INFO request completed, latency_us: 253, response_code: 204, uri: /start/cid/1, method: POST, req_id: d633ba36-eb03-4d34-94f4-8b08a339d6dd, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
17876 Replay: started 1, returned:Ok(())
17877 ----------------------------------------------------------------
17878 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
17879 States: Active Offline Active
17880 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
17881 6029 Acked 7028 Read 3 Done New Done true
17882 6030 Acked 7029 Write 5 Done New Done true
17883 6031 Acked 7030 Flush 0 Done New Done true
17884 6032 Acked 7031 Write 1 Done New Done false
17885 6033 Acked 7032 Write 4 Done New Done false
17886 6034 Acked 7033 Write 6 Done New Done false
17887 6035 Acked 7034 Flush 0 Done New Done false
17888 6036 Acked 7035 Flush 0 Done New Done false
17889 6037 Acked 7036 Read 1 Done New Done false
17890 6038 Acked 7037 Read 1 Done New Done false
17891 6039 Acked 7038 Read 6 Done New Done false
17892 6040 Acked 7039 Write 7 Done New Done false
17893 6041 Acked 7040 Write 1 Done New Done false
17894 6042 Acked 7041 Write 5 Done New Done false
17895 6043 Acked 7042 Read 9 Done New Done false
17896 6044 Acked 7043 Read 9 Done New Done false
17897 6045 Acked 7044 Read 3 Done New Done false
17898 6046 Acked 7045 Read 5 Done New Done false
17899 6047 Acked 7046 Write 1 Done New Done false
17900 6048 Acked 7047 Read 2 Done New Done false
17901 6049 Acked 7048 Read 10 Done New Done false
17902 6050 Acked 7049 Read 3 Done New Done false
17903 6051 Acked 7050 Write 2 Done New Done false
17904 6052 Acked 7051 Read 7 Done New Done false
17905 6053 Acked 7052 Read 9 Done New Done false
17906 6054 Acked 7053 Read 3 Done New Done false
17907 6055 Acked 7054 Read 1 Done New Done false
17908 6056 Acked 7055 Read 2 Done New Done false
17909 6057 Acked 7056 Write 1 Done New Done false
17910 6058 Acked 7057 Read 3 Done New Done false
17911 6059 Acked 7058 Read 3 Done New Done false
17912 6060 Acked 7059 Read 3 Done New Done false
17913 6061 Acked 7060 Read 10 Done New Done false
17914 6062 Acked 7061 Write 1 Done New Done false
17915 6063 Acked 7062 Flush 0 Done New Done false
17916 6064 Acked 7063 Read 10 Done New Done false
17917 6065 Acked 7064 Write 7 Done New Done false
17918 6066 Acked 7065 Read 1 Done New Done false
17919 6067 Acked 7066 Write 4 Done New Done false
17920 6068 Acked 7067 Read 7 Done New Done false
17921 6069 Acked 7068 Write 3 Done New Done false
17922 6070 Acked 7069 Read 2 Done New Done false
17923 6071 Acked 7070 Write 7 Done New Done false
17924 6072 Acked 7071 Write 4 Done New Done false
17925 6073 Acked 7072 Read 5 Done New Done false
17926 6074 Acked 7073 Write 10 Done New Done false
17927 6075 Acked 7074 Read 7 Done New Done false
17928 6076 Acked 7075 Write 3 Done New Done false
17929 6077 Acked 7076 Write 6 Done New Done false
17930 6078 Acked 7077 Write 7 Done New Done false
17931 6079 Acked 7078 Write 4 Done New Done false
17932 6080 Acked 7079 Read 9 Done New Done false
17933 6081 Acked 7080 Write 2 Done New Done false
17934 6082 Acked 7081 Read 7 Done New Done false
17935 6083 Acked 7082 Read 8 Done New Done false
17936 6084 Acked 7083 Flush 0 Done New Done false
17937 6085 Acked 7084 Read 2 Done New Done false
17938 6086 Acked 7085 Read 8 Done New Done false
17939 6087 Acked 7086 Read 6 Done New Done false
17940 6088 Acked 7087 Read 3 Done New Done false
17941 6089 Acked 7088 Read 2 Done New Done false
17942 6090 Acked 7089 Write 7 Done New Done false
17943 6091 Acked 7090 Read 5 Done New Done false
17944 6092 Acked 7091 Write 3 Done New Done false
17945 6093 Acked 7092 Write 8 Done New Done false
17946 6094 Acked 7093 Read 5 Done New Done false
17947 6095 Acked 7094 Read 2 Done New Done false
17948 6096 Acked 7095 Read 5 Done New Done false
17949 6097 Acked 7096 Write 7 Done New Done false
17950 6098 Acked 7097 Read 8 Done New Done false
17951 6099 Acked 7098 Flush 0 Done New Done false
17952 6100 Acked 7099 Read 7 Done New Done false
17953 6101 Acked 7100 Write 2 Done New Done false
17954 6102 Acked 7101 Write 5 Done New Done false
17955 6103 Acked 7102 Read 1 Done New Done false
17956 6104 Acked 7103 Read 10 Done New Done false
17957 6105 Acked 7104 Flush 0 Done New Done false
17958 6106 Acked 7105 Read 6 Done New Done false
17959 6107 Acked 7106 Flush 0 Done New Done false
17960 6108 Acked 7107 Write 3 Done New Done false
17961 6109 Acked 7108 Write 8 Done New Done false
17962 6110 Acked 7109 Read 8 Done New Done false
17963 6111 Acked 7110 Read 6 Done New Done false
17964 6112 Acked 7111 Write 9 Done New Done false
17965 6113 Acked 7112 Write 9 Done New Done false
17966 6114 Acked 7113 Flush 0 Done New Done false
17967 6115 Acked 7114 Read 5 Done New Done false
17968 6116 Acked 7115 Read 9 Done New Done false
17969 6117 Acked 7116 Read 6 Done New Done false
17970 6118 Acked 7117 Read 9 Done New Done false
17971 6119 Acked 7118 Read 6 Done New Done false
17972 6120 Acked 7119 Write 6 Done New Done false
17973 6121 Acked 7120 Write 8 Done New Done false
17974 6122 Acked 7121 Write 5 Done New Done false
17975 6123 Acked 7122 Flush 0 Done New Done false
17976 6124 Acked 7123 Flush 0 Done New Done false
17977 6125 Acked 7124 Read 6 Done New Done false
17978 6126 Acked 7125 Read 6 Done New Done false
17979 6127 Acked 7126 Write 7 Done New Done false
17980 6128 Acked 7127 Read 5 Done New Done false
17981 6129 Acked 7128 Write 1 Done New Done false
17982 6130 Acked 7129 Write 2 Done New Done false
17983 6131 Acked 7130 Flush 0 Done New Done false
17984 6132 Acked 7131 Write 4 Done New Done false
17985 6133 Acked 7132 Read 9 Done New Done false
17986 6134 Acked 7133 Flush 0 Done New Done false
17987 6135 Acked 7134 Read 3 Done New Done false
17988 6136 Acked 7135 Write 8 Done New Done false
17989 6137 Acked 7136 Read 7 Done New Done false
17990 6138 Acked 7137 Read 7 Done New Done false
17991 6139 Acked 7138 Read 2 Done New Done false
17992 6140 Acked 7139 Write 9 Done New Done false
17993 6141 Acked 7140 Write 7 Done New Done false
17994 6142 Acked 7141 Read 1 Done New Done false
17995 6143 Acked 7142 Flush 0 Done New Done false
17996 6144 Acked 7143 Read 2 Done New Done false
17997 6145 Acked 7144 Write 7 Done New Done false
17998 6146 Acked 7145 Read 10 Done New Done false
17999 6147 Acked 7146 Flush 0 Done New Done false
18000 6148 Acked 7147 Write 5 Done New Done false
18001 6149 Acked 7148 Read 6 Done New Done false
18002 6150 Acked 7149 Read 7 Done New Done false
18003 6151 Acked 7150 Read 6 Done New Done false
18004 6152 Acked 7151 Read 2 Done New Done false
18005 6153 Acked 7152 Read 6 Done New Done false
18006 6154 Acked 7153 Write 9 Done New Done false
18007 6155 Acked 7154 Read 8 Done New Done false
18008 6156 Acked 7155 Flush 0 Done New Done false
18009 6157 Acked 7156 Read 4 Done New Done false
18010 6158 Acked 7157 Read 8 Done New Done false
18011 6159 Acked 7158 Write 4 Done New Done false
18012 6160 Acked 7159 Read 3 Done New Done false
18013 6161 Acked 7160 Read 2 Done New Done false
18014 6162 Acked 7161 Write 8 Done New Done false
18015 6163 Acked 7162 Read 6 Done New Done false
18016 6164 Acked 7163 Read 7 Done New Done false
18017 6165 Acked 7164 Read 10 Done New Done false
18018 6166 Acked 7165 Read 4 Done New Done false
18019 6167 Acked 7166 Write 7 Done New Done false
18020 6168 Acked 7167 Flush 0 Done New Done false
18021 6169 Acked 7168 Write 2 Done New Done false
18022 6170 Acked 7169 Write 4 Done New Done false
18023 6171 Acked 7170 Read 3 Done New Done false
18024 6172 Acked 7171 Write 5 Done New Done false
18025 6173 Acked 7172 Flush 0 Done New Done false
18026 6174 Acked 7173 Write 1 Done New Done false
18027 6175 Acked 7174 Read 3 Done New Done false
18028 6176 Acked 7175 Write 3 Done New Done false
18029 6177 Acked 7176 Read 5 Done New Done false
18030 6178 Acked 7177 Write 10 Done New Done false
18031 6179 Acked 7178 Flush 0 Done New Done false
18032 6180 Acked 7179 Write 2 Done New Done false
18033 6181 Acked 7180 Read 8 Done New Done false
18034 6182 Acked 7181 Flush 0 Done New Done false
18035 6183 Acked 7182 Read 7 Done New Done false
18036 6184 Acked 7183 Write 7 Done New Done false
18037 6185 Acked 7184 Read 7 Done New Done false
18038 6186 Acked 7185 Write 4 Done New Done false
18039 6187 Acked 7186 Write 8 Done New Done false
18040 6188 Acked 7187 Read 6 Done New Done false
18041 6189 Acked 7188 Write 1 Done New Done false
18042 6190 Acked 7189 Read 4 Done New Done false
18043 6191 Acked 7190 Flush 0 Done New Done false
18044 6192 Acked 7191 Flush 0 Done New Done false
18045 6193 Acked 7192 Write 7 Done New Done false
18046 6194 Acked 7193 Read 5 Done New Done false
18047 6195 Acked 7194 Flush 0 Done New Done false
18048 6196 Acked 7195 Flush 0 Done New Done false
18049 6197 Acked 7196 Flush 0 Done New Done false
18050 6198 Acked 7197 Read 10 Done New Done false
18051 6199 Acked 7198 Write 1 Done New Done false
18052 6200 Acked 7199 Write 5 Done New Done false
18053 6201 Acked 7200 Read 8 Done New Done false
18054 6202 Acked 7201 Write 6 Done New Done false
18055 6203 Acked 7202 Write 10 Done New Done false
18056 6204 Acked 7203 Write 2 Done New Done false
18057 6205 Acked 7204 Write 10 Done New Done false
18058 6206 Acked 7205 Read 6 Done New Done false
18059 6207 Acked 7206 Read 2 Done New Done false
18060 6208 Acked 7207 Read 1 Done New Done false
18061 6209 Acked 7208 Flush 0 Done New Done false
18062 6210 Acked 7209 Write 7 Done New Done false
18063 6211 Acked 7210 Write 9 Done New Done false
18064 6212 Acked 7211 Write 8 Done New Done false
18065 6213 Acked 7212 Write 1 Done New Done false
18066 6214 Acked 7213 Read 1 Done New Done false
18067 6215 Acked 7214 Write 6 Done New Done false
18068 6216 Acked 7215 Read 4 Done New Done false
18069 6217 Acked 7216 Read 9 Done New Done false
18070 6218 Acked 7217 Read 3 Done New Done false
18071 6219 Acked 7218 Write 5 Done New Done false
18072 6220 Acked 7219 Flush 0 Done New Done false
18073 6221 Acked 7220 Read 5 Done New Done false
18074 6222 Acked 7221 Flush 0 Done New Done false
18075 6223 Acked 7222 Write 7 Done New Done false
18076 6224 Acked 7223 Read 1 Done New Done false
18077 6225 Acked 7224 Write 9 Done New Done false
18078 6226 Acked 7225 Write 8 Done New Done false
18079 6227 Acked 7226 Write 6 Done New Done false
18080 6228 Acked 7227 Read 9 Done New Done false
18081 6229 Acked 7228 Write 6 Done New Done false
18082 6230 Acked 7229 Write 4 Done New Done false
18083 6231 Acked 7230 Write 8 Done New Done false
18084 6232 Acked 7231 Read 5 Done New Done false
18085 6233 Acked 7232 Read 9 Done New Done false
18086 6234 Acked 7233 Read 3 Done New Done false
18087 6235 Acked 7234 Write 8 Done New Done false
18088 6236 Acked 7235 Read 10 Done New Done false
18089 6237 Acked 7236 Write 2 Done New Done false
18090 6238 Acked 7237 Read 1 Done New Done false
18091 6239 Acked 7238 Read 4 Done New Done false
18092 6240 Acked 7239 Read 9 Done New Done false
18093 6241 Acked 7240 Read 5 Done New Done false
18094 6242 Acked 7241 Write 8 Done New Done false
18095 6243 Acked 7242 Read 5 Done New Done false
18096 6244 Acked 7243 Write 9 Done New Done false
18097 6245 Acked 7244 Write 9 Done New Done false
18098 6246 Acked 7245 Write 7 Done New Done false
18099 6247 Acked 7246 Flush 0 Done New Done false
18100 6248 Acked 7247 Write 7 Done New Done false
18101 6249 Acked 7248 Write 3 Done New Done false
18102 6250 Acked 7249 Write 8 Done New Done false
18103 6251 Acked 7250 Write 7 Done New Done false
18104 6252 Acked 7251 Read 7 Done New Done false
18105 6253 Acked 7252 Read 6 Done New Done false
18106 6254 Acked 7253 Write 10 Done New Done false
18107 6255 Acked 7254 Write 5 Done New Done false
18108 6256 Acked 7255 Write 4 Done New Done false
18109 6257 Acked 7256 Write 1 Done New Done false
18110 6258 Acked 7257 Flush 0 Done New Done false
18111 6259 Acked 7258 Flush 0 Done New Done false
18112 6260 Acked 7259 Read 4 Done New Done false
18113 6261 Acked 7260 Read 8 Done New Done false
18114 6262 Acked 7261 Read 3 Done New Done false
18115 6263 Acked 7262 Write 4 Done New Done false
18116 6264 Acked 7263 Write 3 Done New Done false
18117 6265 Acked 7264 Read 10 Done New Done false
18118 6266 Acked 7265 Write 9 Done New Done false
18119 6267 Acked 7266 Read 3 Done New Done false
18120 6268 Acked 7267 Read 9 Done New Done false
18121 6269 Acked 7268 Read 9 Done New Done false
18122 6270 Acked 7269 Write 5 Done New Done false
18123 6271 Acked 7270 Read 1 Done New Done false
18124 6272 Acked 7271 Read 5 Done New Done false
18125 6273 Acked 7272 Write 9 Done New Done false
18126 6274 Acked 7273 Read 2 Done New Done false
18127 6275 Acked 7274 Read 5 Done New Done false
18128 6276 Acked 7275 Write 2 Done New Done false
18129 6277 Acked 7276 Read 4 Done New Done false
18130 6278 Acked 7277 Write 6 Done New Done false
18131 6279 Acked 7278 Write 1 Done New Done false
18132 6280 Acked 7279 Write 1 Done New Done false
18133 6281 Acked 7280 Read 6 Done New Done false
18134 6282 Acked 7281 Read 6 Done New Done false
18135 6283 Acked 7282 Write 6 Done New Done false
18136 6284 Acked 7283 Read 4 Done New Done false
18137 6285 Acked 7284 Write 2 Done New Done false
18138 6286 Acked 7285 Flush 0 Done New Done false
18139 6287 Acked 7286 Read 6 Done New Done false
18140 6288 Acked 7287 Flush 0 Done New Done false
18141 6289 Acked 7288 Read 10 Done New Done false
18142 6290 Acked 7289 Write 3 Done New Done false
18143 6291 Acked 7290 Read 6 Done New Done false
18144 6292 Acked 7291 Write 5 Done New Done false
18145 6293 Acked 7292 Write 9 Done New Done false
18146 6294 Acked 7293 Flush 0 Done New Done false
18147 6295 Acked 7294 Read 7 Done New Done false
18148 6296 Acked 7295 Read 8 Done New Done false
18149 6297 Acked 7296 Write 8 Done New Done false
18150 6298 Acked 7297 Read 5 Done New Done false
18151 6299 Acked 7298 Write 6 Done New Done false
18152 6300 Acked 7299 Read 8 Done New Done false
18153 6301 Acked 7300 Write 3 Done New Done false
18154 6302 Acked 7301 Write 7 Done New Done false
18155 6303 Acked 7302 Write 10 Done New Done false
18156 6304 Acked 7303 Read 7 Done New Done false
18157 6305 Acked 7304 Read 3 Done New Done false
18158 6306 Acked 7305 Flush 0 Done New Done false
18159 6307 Acked 7306 Write 1 Done New Done false
18160 6308 Acked 7307 Write 5 Done New Done false
18161 6309 Acked 7308 Read 5 Done New Done false
18162 6310 Acked 7309 Write 6 Done New Done false
18163 6311 Acked 7310 Read 9 Done New Done false
18164 6312 Acked 7311 Read 4 Done New Done false
18165 6313 Acked 7312 Read 4 Done New Done false
18166 6314 Acked 7313 Read 4 Done New Done false
18167 6315 Acked 7314 Read 2 Done New Done false
18168 6316 Acked 7315 Write 3 Done New Done false
18169 6317 Acked 7316 Write 8 Done New Done false
18170 6318 Acked 7317 Write 3 Done New Done false
18171 6319 Acked 7318 Read 10 Done New Done false
18172 6320 Acked 7319 Read 2 Done New Done false
18173 6321 Acked 7320 Read 7 Done New Done false
18174 6322 Acked 7321 Write 10 Done New Done false
18175 6323 Acked 7322 Write 1 Done New Done false
18176 6324 Acked 7323 Write 8 Done New Done false
18177 6325 Acked 7324 Write 10 Done New Done false
18178 6326 Acked 7325 Read 5 Done New Done false
18179 6327 Acked 7326 Write 8 Done New Done false
18180 6328 Acked 7327 Write 7 Done New Done false
18181 STATES DS:0 DS:1 DS:2 TOTAL
18182 New 0 300 0 300
18183 Sent 0 0 0 0
18184 Done 300 0 300 600
18185 Skipped 0 0 0 0
18186 Error 0 0 0 0
18187 Last Flush: 7305 7027 7305
18188 Downstairs last five completed: 7027 7026 7025 7024 7023
18189 Upstairs last five completed: 6328 6327 6326 6325 6324
18190 CLIENT: Up:0 ds:300 act:2
181912024-02-28T22:16:52.380ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
181922024-02-28T22:16:52.380ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
181932024-02-28T22:16:52.380ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
181942024-02-28T22:16:52.380ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
181952024-02-28T22:16:52.380ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
181962024-02-28T22:16:52.380ZINFOcrucible: [1] client re-new 301 jobs since flush 7027 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
181972024-02-28T22:16:52.380ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
18198 [8820] got start action
18199 [8820] I am going to restart
18200 Starting downstairs at port 8820
18201 Make output file at "/tmp/dsc/downstairs-8820.txt"
18202 [8820][1] reports Starting
182032024-02-28T22:16:53.381ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182042024-02-28T22:16:53.381ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182052024-02-28T22:16:53.381ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182062024-02-28T22:16:53.381ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182072024-02-28T22:16:53.381ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182082024-02-28T22:16:53.381ZINFOcrucible: [1] client re-new 301 jobs since flush 7027 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182092024-02-28T22:16:53.381ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
18210 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1185)
18211 [8820][1] reports Running
182122024-02-28T22:16:54.383ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182132024-02-28T22:16:54.383ZINFOcrucible: ds_connection connected = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182142024-02-28T22:16:54.384ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182152024-02-28T22:16:54.384ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182162024-02-28T22:16:54.384ZINFOcrucible: send last flush ID to this DS: 7027 = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182172024-02-28T22:16:54.385ZINFOcrucible: Replied this last flush ID: 7027 = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182182024-02-28T22:16:54.385ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182192024-02-28T22:16:54.385ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182202024-02-28T22:16:54.385ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182212024-02-28T22:16:54.385ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
18222 ----------------------------------------------------------------
18223 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
18224 Downstairs last five completed: 7328 7327 7326 7325 7324
18225 Upstairs last five completed: 6329 6328 6327 6326 6325
18226 CLIENT: Up:0 ds:0 act:3
18227 Replay: All jobs finished, all DS active.
18228 Main task has work to do, go find it
18229 got dsc Stop(2)
18230 stop 2
18231 Feb 28 22:16:55.709 INFO request completed, latency_us: 397, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 5666ed77-8aae-4b47-a87c-94f6def101fb[8830] Got stop action so:false kr:false
18232 , remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
18233 Feb 28 22:16:55.709 INFO request completed, latency_us: 137, response_code: 200, uri: /state/cid/2, method: GET, req_id: 3e04a8f2-3000-4498-ae12-28068d3c7a79, remote_addr: 127.0{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-28T22:16:55.709905945Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
18234 .0.1:36617, local_addr: 127.0.0.1:9998
182352024-02-28T22:16:55.709ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182362024-02-28T22:16:55.710ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182372024-02-28T22:16:55.710ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182382024-02-28T22:16:55.710ZINFOcrucible: [2] client re-new 0 jobs since flush 7328 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182392024-02-28T22:16:55.710ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
18240 [8830] Exited with: ExitStatus(unix_wait_status(9))
18241 [8830][2] reports Exit
182422024-02-28T22:16:56.711ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182432024-02-28T22:16:56.711ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182442024-02-28T22:16:56.711ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182452024-02-28T22:16:56.711ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182462024-02-28T22:16:56.711ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182472024-02-28T22:16:56.711ZINFOcrucible: [2] client re-new 0 jobs since flush 7328 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182482024-02-28T22:16:56.711ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182492024-02-28T22:16:57.712ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182502024-02-28T22:16:57.712ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182512024-02-28T22:16:57.712ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182522024-02-28T22:16:57.712ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182532024-02-28T22:16:57.712ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182542024-02-28T22:16:57.712ZINFOcrucible: [2] client re-new 0 jobs since flush 7328 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182552024-02-28T22:16:57.712ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182562024-02-28T22:16:58.714ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182572024-02-28T22:16:58.714ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182582024-02-28T22:16:58.714ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182592024-02-28T22:16:58.714ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182602024-02-28T22:16:58.714ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182612024-02-28T22:16:58.714ZINFOcrucible: [2] client re-new 0 jobs since flush 7328 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182622024-02-28T22:16:58.714ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
18263 Feb 28 22:16:59.712 INFO request completed, latency_us: 372, response_code: 200, uri: /state/cid/2, method: GET, req_id: fd54674e-1394-4736-879b-597d635eb861, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
18264 001/300 Read block 4941 len 12288
18265 002/300 Read block 939 len 4096
18266 003/300 Write block 2763 len 12288 data: 3 4 4
18267 004/300 Write block 3297 len 8192 data: 4 3
18268 005/300 Read block 3622 len 20480
182692024-02-28T22:16:59.716ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182702024-02-28T22:16:59.716ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182712024-02-28T22:16:59.716ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182722024-02-28T22:16:59.716ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182732024-02-28T22:16:59.716ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182742024-02-28T22:16:59.716ZINFOcrucible: [2] client re-new 5 jobs since flush 7328 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
182752024-02-28T22:16:59.716ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
18276 006/300 Flush
18277 007/300 Read block 3993 len 12288
18278 008/300 Read block 415 len 24576
18279 009/300 Read block 1652 len 28672
18280 010/300 Read block 2604 len 28672
18281 011/300 Flush
18282 012/300 Flush
18283 013/300 Read block 4729 len 16384
18284 014/300 Read block 4797 len 28672
18285 015/300 Flush
18286 016/300 Write block 1571 len 24576 data: 3 3 3 3 2 4
18287 017/300 Read block 3912 len 36864
18288 018/300 Write block 4034 len 40960 data: 3 3 4 5 4 3 3 3 3 2
18289 019/300 Read block 3994 len 28672
18290 020/300 Read block 4721 len 12288
18291 021/300 Write block 846 len 36864 data: 6 5 5 4 4 5 4 4 4
18292 022/300 Write block 3345 len 4096 data: 3
18293 023/300 Write block 3026 len 12288 data: 3 3 3
18294 024/300 Read block 4710 len 16384
18295 025/300 Read block 519 len 28672
18296 026/300 Write block 3679 len 32768 data: 3 3 4 3 3 2 2 2
18297 027/300 Write block 4848 len 12288 data: 4 5 5
18298 028/300 Write block 3214 len 8192 data: 6 5
18299 029/300 Flush
18300 030/300 Read block 1362 len 8192
18301 031/300 Write block 1833 len 20480 data: 5 6 6 7 5
18302 032/300 Flush
18303 033/300 Flush
18304 034/300 Read block 2847 len 36864
18305 035/300 Read block 3029 len 4096
18306 036/300 Read block 924 len 16384
18307 037/300 Read block 1588 len 36864
18308 038/300 Read block 1412 len 8192
18309 039/300 Read block 3391 len 20480
18310 040/300 Read block 1388 len 8192
18311 041/300 Write block 4835 len 24576 data: 4 4 3 2 3 3
18312 042/300 Write block 946 len 8192 data: 4 4
18313 043/300 Read block 3371 len 12288
18314 044/300 Read block 4645 len 20480
18315 045/300 Read block 4968 len 40960
18316 046/300 Read block 14 len 28672
18317 047/300 Read block 2046 len 32768
18318 048/300 Read block 1056 len 40960
18319 049/300 Write block 368 len 16384 data: 3 3 3 3
18320 050/300 Read block 1985 len 28672
18321 051/300 Flush
18322 052/300 Write block 1752 len 36864 data: 6 8 8 7 6 4 4 4 4
18323 053/300 Write block 1551 len 28672 data: 5 4 4 4 5 5 5
18324 054/300 Read block 2189 len 20480
18325 055/300 Flush
18326 056/300 Write block 3398 len 40960 data: 5 7 5 5 5 5 6 6 5 8
18327 057/300 Read block 4174 len 32768
18328 058/300 Read block 3610 len 32768
18329 059/300 Write block 3328 len 32768 data: 4 4 3 4 4 4 4 5
18330 060/300 Read block 3417 len 36864
18331 061/300 Write block 1757 len 40960 data: 5 5 5 5 5 7 7 6 6 4
18332 062/300 Read block 1591 len 4096
18333 063/300 Read block 2453 len 16384
18334 064/300 Read block 590 len 32768
18335 065/300 Write block 1859 len 12288 data: 7 7 7
18336 066/300 Read block 2621 len 36864
18337 067/300 Write block 780 len 40960 data: 5 5 5 5 5 6 6 6 5 5
18338 068/300 Read block 3687 len 32768
18339 069/300 Read block 3011 len 4096
18340 070/300 Read block 2030 len 16384
18341 071/300 Read block 1535 len 28672
18342 072/300 Write block 4179 len 40960 data: 4 6 7 7 6 5 5 6 5 6
18343 073/300 Read block 4253 len 28672
18344 074/300 Read block 1414 len 12288
18345 075/300 Read block 3475 len 28672
18346 076/300 Read block 4339 len 12288
18347 077/300 Read block 4757 len 8192
18348 078/300 Read block 176 len 8192
18349 079/300 Read block 3025 len 20480
18350 080/300 Read block 3617 len 8192
18351 081/300 Write block 2106 len 32768 data: 3 3 3 5 5 3 4 4
18352 082/300 Write block 1319 len 4096 data: 5
18353 083/300 Read block 3705 len 8192
18354 084/300 Write block 2496 len 32768 data: 5 5 6 5 5 5 4 4
18355 085/300 Read block 1752 len 8192
18356 086/300 Write block 4877 len 40960 data: 5 7 7 6 6 8 9 8 8 6
18357 087/300 Read block 4399 len 12288
18358 088/300 Flush
18359 089/300 Write block 3261 len 16384 data: 4 4 4 4
18360 090/300 Read block 4917 len 28672
18361 091/300 Read block 203 len 36864
18362 092/300 Read block 3257 len 8192
18363 093/300 Write block 4046 len 36864 data: 2 4 4 4 5 6 6 5 3
18364 094/300 Read block 4358 len 4096
18365 095/300 Read block 837 len 28672
18366 096/300 Write block 3739 len 36864 data: 7 10 8 9 8 7 7 6 5
18367 097/300 Read block 4349 len 20480
18368 098/300 Write block 3817 len 4096 data: 4
18369 099/300 Flush
18370 100/300 Read block 4309 len 12288
18371 101/300 Read block 2681 len 40960
18372 102/300 Read block 1838 len 24576
18373 103/300 Write block 1109 len 12288 data: 5 5 5
18374 104/300 Read block 4688 len 40960
18375 105/300 Write block 2212 len 12288 data: 5 5 6
18376 106/300 Read block 2122 len 40960
18377 107/300 Read block 4172 len 28672
18378 108/300 Read block 4020 len 24576
18379 109/300 Write block 3628 len 24576 data: 7 7 7 7 6 5
18380 110/300 Write block 797 len 20480 data: 4 4 4 5 5
18381 111/300 Write block 4460 len 32768 data: 7 8 7 6 4 4 4 3
18382 112/300 Flush
18383 113/300 Read block 3833 len 32768
18384 114/300 Flush
18385 115/300 Read block 91 len 28672
18386 116/300 Read block 2411 len 4096
18387 117/300 Read block 2049 len 40960
18388 118/300 Write block 1998 len 20480 data: 3 3 2 2 2
18389 119/300 Read block 3407 len 36864
18390 120/300 Write block 1138 len 36864 data: 6 6 8 9 7 6 6 4 3
18391 121/300 Flush
18392 122/300 Read block 4874 len 8192
18393 123/300 Read block 233 len 36864
18394 124/300 Read block 189 len 24576
18395 125/300 Write block 2216 len 20480 data: 5 5 7 4 5
18396 126/300 Read block 731 len 36864
18397 127/300 Read block 2975 len 4096
18398 128/300 Write block 3195 len 16384 data: 2 2 4 4
18399 129/300 Read block 203 len 8192
18400 130/300 Read block 3803 len 12288
18401 131/300 Read block 1387 len 36864
18402 132/300 Write block 3963 len 40960 data: 2 2 3 3 3 3 3 6 8 6
18403 133/300 Write block 4571 len 12288 data: 6 4 4
18404 134/300 Read block 2097 len 36864
18405 135/300 Write block 4374 len 36864 data: 4 4 4 4 4 3 4 5 6
18406 136/300 Read block 1919 len 40960
18407 137/300 Write block 2880 len 32768 data: 4 5 6 6 7 6 8 8
18408 138/300 Write block 901 len 40960 data: 2 2 3 3 3 2 3 4 5 5
18409 139/300 Read block 3159 len 32768
18410 140/300 Read block 1541 len 36864
18411 141/300 Read block 3654 len 20480
18412 142/300 Flush
18413 143/300 Flush
18414 144/300 Write block 2032 len 24576 data: 5 7 8 8 9 7
18415 145/300 Read block 1827 len 32768
18416 146/300 Read block 597 len 32768
18417 147/300 Write block 2185 len 20480 data: 4 4 4 4 5
18418 148/300 Write block 1522 len 32768 data: 7 6 4 4 4 2 3 4
18419 149/300 Write block 589 len 24576 data: 4 4 3 3 5 5
18420 150/300 Read block 1353 len 36864
18421 151/300 Flush
18422 152/300 Read block 3983 len 32768
18423 153/300 Read block 230 len 32768
18424 154/300 Read block 2406 len 12288
18425 155/300 Read block 2918 len 16384
18426 156/300 Write block 1739 len 4096 data: 3
18427 157/300 Flush
18428 158/300 Read block 1796 len 20480
18429 159/300 Write block 3433 len 12288 data: 7 7 6
18430 160/300 Read block 208 len 20480
18431 161/300 Read block 648 len 28672
18432 162/300 Flush
18433 163/300 Read block 4237 len 8192
18434 164/300 Write block 3983 len 36864 data: 4 3 4 4 5 4 4 3 3
18435 165/300 Read block 4926 len 4096
18436 166/300 Read block 1166 len 8192
18437 167/300 Write block 2320 len 16384 data: 5 5 5 5
18438 168/300 Write block 2422 len 36864 data: 4 4 4 4 4 4 6 7 8
18439 169/300 Read block 4276 len 16384
18440 170/300 Write block 4056 len 36864 data: 4 4 4 5 4 4 5 5 4
18441 171/300 Flush
18442 172/300 Flush
18443 173/300 Read block 2547 len 8192
18444 174/300 Read block 2448 len 24576
18445 175/300 Read block 3683 len 20480
18446 176/300 Read block 3706 len 12288
18447 177/300 Flush
18448 178/300 Write block 1017 len 20480 data: 2 2 3 3 3
18449 179/300 Read block 1315 len 40960
18450 180/300 Read block 2229 len 28672
18451 181/300 Write block 1063 len 32768 data: 5 6 5 6 6 5 3 3
18452 182/300 Flush
18453 183/300 Write block 2625 len 4096 data: 2
18454 184/300 Write block 3077 len 12288 data: 7 6 6
18455 185/300 Write block 2341 len 4096 data: 7
18456 186/300 Write block 4446 len 24576 data: 3 3 4 4 5 5
18457 187/300 Read block 3618 len 28672
18458 188/300 Read block 2657 len 16384
18459 189/300 Read block 926 len 8192
18460 190/300 Read block 3901 len 12288
18461 191/300 Read block 1563 len 8192
18462 192/300 Write block 739 len 12288 data: 4 4 3
18463 193/300 Read block 1346 len 28672
18464 194/300 Flush
18465 195/300 Read block 4581 len 32768
18466 196/300 Read block 3135 len 4096
18467 197/300 Write block 3346 len 32768 data: 2 3 3 3 3 3 3 2
18468 198/300 Flush
18469 199/300 Write block 3382 len 4096 data: 5
18470 200/300 Read block 3931 len 12288
18471 201/300 Read block 2657 len 4096
18472 202/300 Read block 2532 len 12288
18473 203/300 Write block 3128 len 40960 data: 3 4 4 4 4 4 3 4 4 4
18474 204/300 Flush
18475 205/300 Write block 146 len 32768 data: 9 7 8 7 5 5 5 5
18476 206/300 Flush
18477 207/300 Read block 586 len 20480
18478 208/300 Write block 3839 len 12288 data: 4 3 3
18479 209/300 Flush
18480 210/300 Write block 4005 len 32768 data: 5 4 3 4 5 6 7 9
18481 211/300 Read block 1604 len 28672
18482 212/300 Read block 2408 len 8192
18483 213/300 Write block 3092 len 36864 data: 4 4 4 2 5 4 4 4 4
18484 214/300 Write block 2543 len 32768 data: 5 6 5 4 2 2 2 2
18485 215/300 Read block 2505 len 40960
18486 216/300 Read block 265 len 40960
18487 217/300 Read block 3083 len 40960
18488 218/300 Read block 58 len 12288
18489 219/300 Write block 4153 len 28672 data: 5 5 5 5 4 4 5
18490 220/300 Read block 2822 len 40960
18491 221/300 Read block 2812 len 40960
18492 222/300 Write block 1835 len 12288 data: 7 8 6
18493 223/300 Read block 489 len 40960
18494 224/300 Write block 4907 len 16384 data: 3 3 4 5
18495 225/300 Read block 2215 len 36864
18496 226/300 Write block 2455 len 24576 data: 5 5 5 6 6 4
18497 227/300 Write block 1987 len 36864 data: 8 7 6 7 8 7 4 5 4
18498 228/300 Write block 2319 len 16384 data: 5 6 6 6
18499 229/300 Read block 3276 len 24576
18500 230/300 Write block 1710 len 8192 data: 4 5
18501 231/300 Write block 3890 len 40960 data: 5 6 6 6 6 7 5 4 4 5
18502 232/300 Write block 3490 len 24576 data: 3 3 3 3 4 4
18503 233/300 Read block 1400 len 12288
18504 234/300 Read block 4970 len 4096
18505 235/300 Read block 1016 len 24576
18506 236/300 Read block 2743 len 12288
18507 237/300 Read block 3432 len 4096
18508 238/300 Read block 478 len 32768
18509 239/300 Write block 2367 len 16384 data: 4 4 4 4
18510 240/300 Read block 4326 len 16384
18511 241/300 Write block 4494 len 28672 data: 8 7 10 8 9 12 9
18512 242/300 Read block 4149 len 28672
18513 243/300 Write block 3660 len 20480 data: 3 3 4 3 2
18514 244/300 Read block 2889 len 12288
18515 245/300 Read block 4895 len 4096
18516 246/300 Read block 2056 len 40960
18517 247/300 Read block 800 len 12288
18518 248/300 Write block 2125 len 4096 data: 3
18519 249/300 Read block 2652 len 32768
18520 250/300 Write block 3951 len 12288 data: 4 6 6
18521 251/300 Read block 2110 len 32768
18522 252/300 Write block 1534 len 24576 data: 6 4 5 4 4 4
18523 253/300 Read block 2347 len 16384
18524 254/300 Write block 4584 len 24576 data: 6 5 2 3 3 4
18525 255/300 Write block 2608 len 20480 data: 5 5 6 6 4
18526 256/300 Flush
18527 257/300 Read block 2826 len 8192
18528 258/300 Write block 2677 len 20480 data: 7 7 7 6 5
18529 259/300 Write block 3562 len 36864 data: 4 4 4 4 5 4 5 5 5
18530 260/300 Read block 1347 len 12288
18531 261/300 Flush
18532 262/300 Write block 3512 len 4096 data: 5
18533 263/300 Write block 2531 len 16384 data: 3 3 4 4
18534 264/300 Read block 1900 len 8192
18535 265/300 Write block 384 len 4096 data: 3
18536 266/300 Read block 960 len 28672
18537 267/300 Read block 3865 len 24576
18538 268/300 Write block 2668 len 28672 data: 3 3 3 5 5 6 5
18539 269/300 Write block 65 len 8192 data: 5 8
18540 270/300 Read block 564 len 4096
18541 271/300 Read block 3098 len 32768
18542 272/300 Write block 3115 len 16384 data: 4 5 4 5
18543 273/300 Write block 4009 len 32768 data: 6 7 8 10 10 11 9 8
18544 274/300 Read block 1670 len 8192
18545 275/300 Write block 2382 len 4096 data: 4
18546 276/300 Read block 311 len 8192
18547 277/300 Write block 3108 len 24576 data: 3 3 5 5 8 7
18548 278/300 Write block 3853 len 20480 data: 3 4 7 6 7
18549 279/300 Read block 3684 len 12288
18550 280/300 Read block 304 len 20480
18551 281/300 Read block 604 len 8192
18552 282/300 Write block 2817 len 4096 data: 8
18553 283/300 Read block 30 len 8192
18554 284/300 Write block 4025 len 16384 data: 4 4 4 4
18555 285/300 Write block 4374 len 32768 data: 5 5 5 5 5 4 5 6
18556 286/300 Read block 1065 len 28672
18557 287/300 Write block 1516 len 40960 data: 7 7 10 8 8 7 8 7 5 5
18558 288/300 Read block 127 len 28672
18559 289/300 Read block 1891 len 32768
18560 290/300 Write block 706 len 16384 data: 4 4 3 3
18561 291/300 Write block 846 len 8192 data: 7 6
18562 292/300 Read block 2382 len 16384
18563 293/300 Read block 4947 len 4096
18564 294/300 Read block 1007 len 20480
18565 295/300 Flush
18566 296/300 Read block 4762 len 4096
18567 297/300 Write block 572 len 36864 data: 5 5 5 7 4 5 4 2 3
18568 298/300 Read block 3539 len 36864
18569 299/300 Read block 3587 len 8192
18570 300/300 Read block 3868 len 16384
18571 Main task has work to do, go find it
18572 got dsc Start(2)
18573 start 2
18574 Feb 28 22:17:00.047 INFO request completed, latency_us: 206, response_code: 204, uri: /start/cid/2, method: POST, req_id: 571c0008-ec4c-4374-b2da-03c67f798911, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
18575 Replay: started 2, returned:Ok(())
18576 ----------------------------------------------------------------
18577 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18578 States: Active Active Offline
18579 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
18580 6330 Acked 7329 Read 3 Done Done New true
18581 6331 Acked 7330 Read 1 Done Done New true
18582 6332 Acked 7331 Write 3 Done Done New true
18583 6333 Acked 7332 Write 2 Done Done New true
18584 6334 Acked 7333 Read 5 Done Done New true
18585 6335 Acked 7334 Flush 0 Done Done New false
18586 6336 Acked 7335 Read 3 Done Done New false
18587 6337 Acked 7336 Read 6 Done Done New false
18588 6338 Acked 7337 Read 7 Done Done New false
18589 6339 Acked 7338 Read 7 Done Done New false
18590 6340 Acked 7339 Flush 0 Done Done New false
18591 6341 Acked 7340 Flush 0 Done Done New false
18592 6342 Acked 7341 Read 4 Done Done New false
18593 6343 Acked 7342 Read 7 Done Done New false
18594 6344 Acked 7343 Flush 0 Done Done New false
18595 6345 Acked 7344 Write 6 Done Done New false
18596 6346 Acked 7345 Read 9 Done Done New false
18597 6347 Acked 7346 Write 10 Done Done New false
18598 6348 Acked 7347 Read 7 Done Done New false
18599 6349 Acked 7348 Read 3 Done Done New false
18600 6350 Acked 7349 Write 9 Done Done New false
18601 6351 Acked 7350 Write 1 Done Done New false
18602 6352 Acked 7351 Write 3 Done Done New false
18603 6353 Acked 7352 Read 4 Done Done New false
18604 6354 Acked 7353 Read 7 Done Done New false
18605 6355 Acked 7354 Write 8 Done Done New false
18606 6356 Acked 7355 Write 3 Done Done New false
18607 6357 Acked 7356 Write 2 Done Done New false
18608 6358 Acked 7357 Flush 0 Done Done New false
18609 6359 Acked 7358 Read 2 Done Done New false
18610 6360 Acked 7359 Write 5 Done Done New false
18611 6361 Acked 7360 Flush 0 Done Done New false
18612 6362 Acked 7361 Flush 0 Done Done New false
18613 6363 Acked 7362 Read 9 Done Done New false
18614 6364 Acked 7363 Read 1 Done Done New false
18615 6365 Acked 7364 Read 4 Done Done New false
18616 6366 Acked 7365 Read 9 Done Done New false
18617 6367 Acked 7366 Read 2 Done Done New false
18618 6368 Acked 7367 Read 5 Done Done New false
18619 6369 Acked 7368 Read 2 Done Done New false
18620 6370 Acked 7369 Write 6 Done Done New false
18621 6371 Acked 7370 Write 2 Done Done New false
18622 6372 Acked 7371 Read 3 Done Done New false
18623 6373 Acked 7372 Read 5 Done Done New false
18624 6374 Acked 7373 Read 10 Done Done New false
18625 6375 Acked 7374 Read 7 Done Done New false
18626 6376 Acked 7375 Read 8 Done Done New false
18627 6377 Acked 7376 Read 10 Done Done New false
18628 6378 Acked 7377 Write 4 Done Done New false
18629 6379 Acked 7378 Read 7 Done Done New false
18630 6380 Acked 7379 Flush 0 Done Done New false
18631 6381 Acked 7380 Write 9 Done Done New false
18632 6382 Acked 7381 Write 7 Done Done New false
18633 6383 Acked 7382 Read 5 Done Done New false
18634 6384 Acked 7383 Flush 0 Done Done New false
18635 6385 Acked 7384 Write 10 Done Done New false
18636 6386 Acked 7385 Read 8 Done Done New false
18637 6387 Acked 7386 Flush 0 Done Done New false
18638 6388 Acked 7387 Read 8 Done Done New false
18639 6389 Acked 7388 Write 8 Done Done New false
18640 6390 Acked 7389 Read 9 Done Done New false
18641 6391 Acked 7390 Write 10 Done Done New false
18642 6392 Acked 7391 Read 1 Done Done New false
18643 6393 Acked 7392 Read 4 Done Done New false
18644 6394 Acked 7393 Read 8 Done Done New false
18645 6395 Acked 7394 Write 3 Done Done New false
18646 6396 Acked 7395 Read 9 Done Done New false
18647 6397 Acked 7396 Write 10 Done Done New false
18648 6398 Acked 7397 Read 8 Done Done New false
18649 6399 Acked 7398 Read 1 Done Done New false
18650 6400 Acked 7399 Read 4 Done Done New false
18651 6401 Acked 7400 Read 7 Done Done New false
18652 6402 Acked 7401 Write 10 Done Done New false
18653 6403 Acked 7402 Read 7 Done Done New false
18654 6404 Acked 7403 Read 3 Done Done New false
18655 6405 Acked 7404 Read 7 Done Done New false
18656 6406 Acked 7405 Read 3 Done Done New false
18657 6407 Acked 7406 Read 2 Done Done New false
18658 6408 Acked 7407 Read 2 Done Done New false
18659 6409 Acked 7408 Read 5 Done Done New false
18660 6410 Acked 7409 Read 2 Done Done New false
18661 6411 Acked 7410 Write 8 Done Done New false
18662 6412 Acked 7411 Write 1 Done Done New false
18663 6413 Acked 7412 Read 2 Done Done New false
18664 6414 Acked 7413 Write 8 Done Done New false
18665 6415 Acked 7414 Read 2 Done Done New false
18666 6416 Acked 7415 Write 10 Done Done New false
18667 6417 Acked 7416 Read 3 Done Done New false
18668 6418 Acked 7417 Flush 0 Done Done New false
18669 6419 Acked 7418 Write 4 Done Done New false
18670 6420 Acked 7419 Read 7 Done Done New false
18671 6421 Acked 7420 Read 9 Done Done New false
18672 6422 Acked 7421 Read 2 Done Done New false
18673 6423 Acked 7422 Write 9 Done Done New false
18674 6424 Acked 7423 Read 1 Done Done New false
18675 6425 Acked 7424 Read 7 Done Done New false
18676 6426 Acked 7425 Write 9 Done Done New false
18677 6427 Acked 7426 Read 5 Done Done New false
18678 6428 Acked 7427 Write 1 Done Done New false
18679 6429 Acked 7428 Flush 0 Done Done New false
18680 6430 Acked 7429 Read 3 Done Done New false
18681 6431 Acked 7430 Read 10 Done Done New false
18682 6432 Acked 7431 Read 6 Done Done New false
18683 6433 Acked 7432 Write 3 Done Done New false
18684 6434 Acked 7433 Read 10 Done Done New false
18685 6435 Acked 7434 Write 3 Done Done New false
18686 6436 Acked 7435 Read 10 Done Done New false
18687 6437 Acked 7436 Read 7 Done Done New false
18688 6438 Acked 7437 Read 6 Done Done New false
18689 6439 Acked 7438 Write 6 Done Done New false
18690 6440 Acked 7439 Write 5 Done Done New false
18691 6441 Acked 7440 Write 8 Done Done New false
18692 6442 Acked 7441 Flush 0 Done Done New false
18693 6443 Acked 7442 Read 8 Done Done New false
18694 6444 Acked 7443 Flush 0 Done Done New false
18695 6445 Acked 7444 Read 7 Done Done New false
18696 6446 Acked 7445 Read 1 Done Done New false
18697 6447 Acked 7446 Read 10 Done Done New false
18698 6448 Acked 7447 Write 5 Done Done New false
18699 6449 Acked 7448 Read 9 Done Done New false
18700 6450 Acked 7449 Write 9 Done Done New false
18701 6451 Acked 7450 Flush 0 Done Done New false
18702 6452 Acked 7451 Read 2 Done Done New false
18703 6453 Acked 7452 Read 9 Done Done New false
18704 6454 Acked 7453 Read 6 Done Done New false
18705 6455 Acked 7454 Write 5 Done Done New false
18706 6456 Acked 7455 Read 9 Done Done New false
18707 6457 Acked 7456 Read 1 Done Done New false
18708 6458 Acked 7457 Write 4 Done Done New false
18709 6459 Acked 7458 Read 2 Done Done New false
18710 6460 Acked 7459 Read 3 Done Done New false
18711 6461 Acked 7460 Read 9 Done Done New false
18712 6462 Acked 7461 Write 10 Done Done New false
18713 6463 Acked 7462 Write 3 Done Done New false
18714 6464 Acked 7463 Read 9 Done Done New false
18715 6465 Acked 7464 Write 9 Done Done New false
18716 6466 Acked 7465 Read 10 Done Done New false
18717 6467 Acked 7466 Write 8 Done Done New false
18718 6468 Acked 7467 Write 10 Done Done New false
18719 6469 Acked 7468 Read 8 Done Done New false
18720 6470 Acked 7469 Read 9 Done Done New false
18721 6471 Acked 7470 Read 5 Done Done New false
18722 6472 Acked 7471 Flush 0 Done Done New false
18723 6473 Acked 7472 Flush 0 Done Done New false
18724 6474 Acked 7473 Write 6 Done Done New false
18725 6475 Acked 7474 Read 8 Done Done New false
18726 6476 Acked 7475 Read 8 Done Done New false
18727 6477 Acked 7476 Write 5 Done Done New false
18728 6478 Acked 7477 Write 8 Done Done New false
18729 6479 Acked 7478 Write 6 Done Done New false
18730 6480 Acked 7479 Read 9 Done Done New false
18731 6481 Acked 7480 Flush 0 Done Done New false
18732 6482 Acked 7481 Read 8 Done Done New false
18733 6483 Acked 7482 Read 8 Done Done New false
18734 6484 Acked 7483 Read 3 Done Done New false
18735 6485 Acked 7484 Read 4 Done Done New false
18736 6486 Acked 7485 Write 1 Done Done New false
18737 6487 Acked 7486 Flush 0 Done Done New false
18738 6488 Acked 7487 Read 5 Done Done New false
18739 6489 Acked 7488 Write 3 Done Done New false
18740 6490 Acked 7489 Read 5 Done Done New false
18741 6491 Acked 7490 Read 7 Done Done New false
18742 6492 Acked 7491 Flush 0 Done Done New false
18743 6493 Acked 7492 Read 2 Done Done New false
18744 6494 Acked 7493 Write 9 Done Done New false
18745 6495 Acked 7494 Read 1 Done Done New false
18746 6496 Acked 7495 Read 2 Done Done New false
18747 6497 Acked 7496 Write 4 Done Done New false
18748 6498 Acked 7497 Write 9 Done Done New false
18749 6499 Acked 7498 Read 4 Done Done New false
18750 6500 Acked 7499 Write 9 Done Done New false
18751 6501 Acked 7500 Flush 0 Done Done New false
18752 6502 Acked 7501 Flush 0 Done Done New false
18753 6503 Acked 7502 Read 2 Done Done New false
18754 6504 Acked 7503 Read 6 Done Done New false
18755 6505 Acked 7504 Read 5 Done Done New false
18756 6506 Acked 7505 Read 3 Done Done New false
18757 6507 Acked 7506 Flush 0 Done Done New false
18758 6508 Acked 7507 Write 5 Done Done New false
18759 6509 Acked 7508 Read 10 Done Done New false
18760 6510 Acked 7509 Read 7 Done Done New false
18761 6511 Acked 7510 Write 8 Done Done New false
18762 6512 Acked 7511 Flush 0 Done Done New false
18763 6513 Acked 7512 Write 1 Done Done New false
18764 6514 Acked 7513 Write 3 Done Done New false
18765 6515 Acked 7514 Write 1 Done Done New false
18766 6516 Acked 7515 Write 6 Done Done New false
18767 6517 Acked 7516 Read 7 Done Done New false
18768 6518 Acked 7517 Read 4 Done Done New false
18769 6519 Acked 7518 Read 2 Done Done New false
18770 6520 Acked 7519 Read 3 Done Done New false
18771 6521 Acked 7520 Read 2 Done Done New false
18772 6522 Acked 7521 Write 3 Done Done New false
18773 6523 Acked 7522 Read 7 Done Done New false
18774 6524 Acked 7523 Flush 0 Done Done New false
18775 6525 Acked 7524 Read 8 Done Done New false
18776 6526 Acked 7525 Read 1 Done Done New false
18777 6527 Acked 7526 Write 8 Done Done New false
18778 6528 Acked 7527 Flush 0 Done Done New false
18779 6529 Acked 7528 Write 1 Done Done New false
18780 6530 Acked 7529 Read 3 Done Done New false
18781 6531 Acked 7530 Read 1 Done Done New false
18782 6532 Acked 7531 Read 3 Done Done New false
18783 6533 Acked 7532 Write 10 Done Done New false
18784 6534 Acked 7533 Flush 0 Done Done New false
18785 6535 Acked 7534 Write 8 Done Done New false
18786 6536 Acked 7535 Flush 0 Done Done New false
18787 6537 Acked 7536 Read 5 Done Done New false
18788 6538 Acked 7537 Write 3 Done Done New false
18789 6539 Acked 7538 Flush 0 Done Done New false
18790 6540 Acked 7539 Write 8 Done Done New false
18791 6541 Acked 7540 Read 7 Done Done New false
18792 6542 Acked 7541 Read 2 Done Done New false
18793 6543 Acked 7542 Write 9 Done Done New false
18794 6544 Acked 7543 Write 8 Done Done New false
18795 6545 Acked 7544 Read 10 Done Done New false
18796 6546 Acked 7545 Read 10 Done Done New false
18797 6547 Acked 7546 Read 10 Done Done New false
18798 6548 Acked 7547 Read 3 Done Done New false
18799 6549 Acked 7548 Write 7 Done Done New false
18800 6550 Acked 7549 Read 10 Done Done New false
18801 6551 Acked 7550 Read 10 Done Done New false
18802 6552 Acked 7551 Write 3 Done Done New false
18803 6553 Acked 7552 Read 10 Done Done New false
18804 6554 Acked 7553 Write 4 Done Done New false
18805 6555 Acked 7554 Read 9 Done Done New false
18806 6556 Acked 7555 Write 6 Done Done New false
18807 6557 Acked 7556 Write 9 Done Done New false
18808 6558 Acked 7557 Write 4 Done Done New false
18809 6559 Acked 7558 Read 6 Done Done New false
18810 6560 Acked 7559 Write 2 Done Done New false
18811 6561 Acked 7560 Write 10 Done Done New false
18812 6562 Acked 7561 Write 6 Done Done New false
18813 6563 Acked 7562 Read 3 Done Done New false
18814 6564 Acked 7563 Read 1 Done Done New false
18815 6565 Acked 7564 Read 6 Done Done New false
18816 6566 Acked 7565 Read 3 Done Done New false
18817 6567 Acked 7566 Read 1 Done Done New false
18818 6568 Acked 7567 Read 8 Done Done New false
18819 6569 Acked 7568 Write 4 Done Done New false
18820 6570 Acked 7569 Read 4 Done Done New false
18821 6571 Acked 7570 Write 7 Done Done New false
18822 6572 Acked 7571 Read 7 Done Done New false
18823 6573 Acked 7572 Write 5 Done Done New false
18824 6574 Acked 7573 Read 3 Done Done New false
18825 6575 Acked 7574 Read 1 Done Done New false
18826 6576 Acked 7575 Read 10 Done Done New false
18827 6577 Acked 7576 Read 3 Done Done New false
18828 6578 Acked 7577 Write 1 Done Done New false
18829 6579 Acked 7578 Read 8 Done Done New false
18830 6580 Acked 7579 Write 3 Done Done New false
18831 6581 Acked 7580 Read 8 Done Done New false
18832 6582 Acked 7581 Write 6 Done Done New false
18833 6583 Acked 7582 Read 4 Done Done New false
18834 6584 Acked 7583 Write 6 Done Done New false
18835 6585 Acked 7584 Write 5 Done Done New false
18836 6586 Acked 7585 Flush 0 Done Done New false
18837 6587 Acked 7586 Read 2 Done Done New false
18838 6588 Acked 7587 Write 5 Done Done New false
18839 6589 Acked 7588 Write 9 Done Done New false
18840 6590 Acked 7589 Read 3 Done Done New false
18841 6591 Acked 7590 Flush 0 Done Done New false
18842 6592 Acked 7591 Write 1 Done Done New false
18843 6593 Acked 7592 Write 4 Done Done New false
18844 6594 Acked 7593 Read 2 Done Done New false
18845 6595 Acked 7594 Write 1 Done Done New false
18846 6596 Acked 7595 Read 7 Done Done New false
18847 6597 Acked 7596 Read 6 Done Done New false
18848 6598 Acked 7597 Write 7 Done Done New false
18849 6599 Acked 7598 Write 2 Done Done New false
18850 6600 Acked 7599 Read 1 Done Done New false
18851 6601 Acked 7600 Read 8 Done Done New false
18852 6602 Acked 7601 Write 4 Done Done New false
18853 6603 Acked 7602 Write 8 Done Done New false
18854 6604 Acked 7603 Read 2 Done Done New false
18855 6605 Acked 7604 Write 1 Done Done New false
18856 6606 Acked 7605 Read 2 Done Done New false
18857 6607 Acked 7606 Write 6 Done Done New false
18858 6608 Acked 7607 Write 5 Done Done New false
18859 6609 Acked 7608 Read 3 Done Done New false
18860 6610 Acked 7609 Read 5 Done Done New false
18861 6611 Acked 7610 Read 2 Done Done New false
18862 6612 Acked 7611 Write 1 Done Done New false
18863 6613 Acked 7612 Read 2 Done Done New false
18864 6614 Acked 7613 Write 4 Done Done New false
18865 6615 Acked 7614 Write 8 Done Done New false
18866 6616 Acked 7615 Read 7 Done Done New false
18867 6617 Acked 7616 Write 10 Done Done New false
18868 6618 Acked 7617 Read 7 Done Done New false
18869 6619 Acked 7618 Read 8 Done Done New false
18870 6620 Acked 7619 Write 4 Done Done New false
18871 6621 Acked 7620 Write 2 Done Done New false
18872 6622 Acked 7621 Read 4 Done Done New false
18873 6623 Acked 7622 Read 1 Done Done New false
18874 6624 Acked 7623 Read 5 Done Done New false
18875 6625 Acked 7624 Flush 0 Done Done New false
18876 6626 Acked 7625 Read 1 Done Done New false
18877 6627 Acked 7626 Write 9 Done Done New false
18878 6628 Acked 7627 Read 9 Done Done New false
18879 6629 Acked 7628 Read 2 Done Done New false
18880 6630 Acked 7629 Read 4 Done Done New false
18881 STATES DS:0 DS:1 DS:2 TOTAL
18882 New 0 0 301 301
18883 Sent 0 0 0 0
18884 Done 301 301 0 602
18885 Skipped 0 0 0 0
18886 Error 0 0 0 0
18887 Last Flush: 7624 7624 7328
18888 Downstairs last five completed: 7328 7327 7326 7325 7324
18889 Upstairs last five completed: 6630 6629 6628 6627 6626
18890 CLIENT: Up:0 ds:301 act:2
188912024-02-28T22:17:00.717ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
188922024-02-28T22:17:00.717ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
188932024-02-28T22:17:00.717ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
188942024-02-28T22:17:00.717ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
188952024-02-28T22:17:00.717ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
188962024-02-28T22:17:00.717ZINFOcrucible: [2] client re-new 302 jobs since flush 7328 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
188972024-02-28T22:17:00.717ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
18898 [8830] got start action
18899 [8830] I am going to restart
18900 Starting downstairs at port 8830
18901 Make output file at "/tmp/dsc/downstairs-8830.txt"
18902 [8830][2] reports Starting
189032024-02-28T22:17:01.718ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189042024-02-28T22:17:01.718ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189052024-02-28T22:17:01.718ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189062024-02-28T22:17:01.718ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189072024-02-28T22:17:01.718ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189082024-02-28T22:17:01.718ZINFOcrucible: [2] client re-new 302 jobs since flush 7328 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189092024-02-28T22:17:01.719ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
18910 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1186)
18911 [8830][2] reports Running
189122024-02-28T22:17:02.721ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189132024-02-28T22:17:02.721ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189142024-02-28T22:17:02.722ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189152024-02-28T22:17:02.722ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189162024-02-28T22:17:02.722ZINFOcrucible: send last flush ID to this DS: 7328 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189172024-02-28T22:17:02.722ZINFOcrucible: Replied this last flush ID: 7328 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189182024-02-28T22:17:02.722ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189192024-02-28T22:17:02.722ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189202024-02-28T22:17:02.722ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189212024-02-28T22:17:02.722ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
18922 ----------------------------------------------------------------
18923 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
18924 Downstairs last five completed: 7630 7629 7628 7627 7626
18925 Upstairs last five completed: 6631 6630 6629 6628 6627
18926 CLIENT: Up:0 ds:0 act:3
18927 Replay: All jobs finished, all DS active.
18928 Main task has work to do, go find it
18929 got dsc Stop(1)
18930 stop 1
18931 Feb 28 22:17:04.050 INFO request completed, latency_us: 371[8820] Got stop action so:false kr:false
18932 , response_code: 204, uri: /stop/cid/1, method: POST, req_id: a69a85fa-483b-48ac-b4f8-80754e3fb318, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
18933 Feb 28 22:17:04.051 INFO accepted connection, remote_addr: 127.0.0.1:41656, local_addr: 127.0.0.1:9998
189342024-02-28T22:17:04.051ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189352024-02-28T22:17:04.051ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189362024-02-28T22:17:04.051ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189372024-02-28T22:17:04.051ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189382024-02-28T22:17:04.051ZINFOcrucible: [1] client re-new 0 jobs since flush 7630 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189392024-02-28T22:17:04.051ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
18940 Feb 28 22:17:04.051 INFO request completed, latency_us: 149, response_code: 200, uri: /state/cid/1, method: GET, req_id: ec4ce147-6791-4d44-b7cb-21d367f24134, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
18941 [8820] Exited with: ExitStatus(unix_wait_status(9))
18942 [8820][1] reports Exit
189432024-02-28T22:17:05.053ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189442024-02-28T22:17:05.053ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189452024-02-28T22:17:05.053ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189462024-02-28T22:17:05.053ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189472024-02-28T22:17:05.053ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189482024-02-28T22:17:05.053ZINFOcrucible: [1] client re-new 0 jobs since flush 7630 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189492024-02-28T22:17:05.053ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189502024-02-28T22:17:06.054ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189512024-02-28T22:17:06.054ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189522024-02-28T22:17:06.054ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189532024-02-28T22:17:06.054ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189542024-02-28T22:17:06.054ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189552024-02-28T22:17:06.054ZINFOcrucible: [1] client re-new 0 jobs since flush 7630 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189562024-02-28T22:17:06.054ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189572024-02-28T22:17:07.055ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189582024-02-28T22:17:07.055ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189592024-02-28T22:17:07.055ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189602024-02-28T22:17:07.055ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189612024-02-28T22:17:07.055ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189622024-02-28T22:17:07.055ZINFOcrucible: [1] client re-new 0 jobs since flush 7630 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189632024-02-28T22:17:07.055ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
18964 Feb 28 22:17:08.054 INFO request completed, latency_us: 206, response_code: 200, uri: /state/cid/1, method: GET, req_id: 9fe54d4e-5cba-48e3-96bc-a8fdfeee1b21, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
18965 001/300 Read block 1723 len 36864
18966 002/300 Read block 1782 len 8192
189672024-02-28T22:17:08.056ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189682024-02-28T22:17:08.056ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189692024-02-28T22:17:08.056ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189702024-02-28T22:17:08.056ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189712024-02-28T22:17:08.056ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189722024-02-28T22:17:08.056ZINFOcrucible: [1] client re-new 2 jobs since flush 7630 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
189732024-02-28T22:17:08.056ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
18974 003/300 Read block 4153 len 4096
18975 004/300 Write block 240 len 36864 data: 3 4 3 3 3 3 4 4 5
18976 005/300 Flush
18977 006/300 Read block 3102 len 8192
18978 007/300 Write block 1440 len 24576 data: 5 3 3 3 3 4
18979 008/300 Read block 980 len 36864
18980 009/300 Read block 4428 len 12288
18981 010/300 Write block 4340 len 8192 data: 4 4
18982 011/300 Read block 255 len 16384
18983 012/300 Write block 137 len 40960 data: 5 4 5 7 7 6 7 7 7 10
18984 013/300 Write block 1613 len 8192 data: 7 7
18985 014/300 Flush
18986 015/300 Flush
18987 016/300 Read block 38 len 24576
18988 017/300 Read block 4365 len 28672
18989 018/300 Write block 3593 len 28672 data: 7 4 6 5 5 5 5
18990 019/300 Read block 1045 len 32768
18991 020/300 Write block 1312 len 40960 data: 5 4 4 4 5 5 6 6 7 6
18992 021/300 Write block 206 len 28672 data: 6 5 4 4 3 2 4
18993 022/300 Read block 3886 len 32768
18994 023/300 Read block 97 len 4096
18995 024/300 Flush
18996 025/300 Write block 4749 len 36864 data: 5 5 5 5 3 3 3 3 3
18997 026/300 Read block 1153 len 28672
18998 027/300 Flush
18999 028/300 Write block 4878 len 32768 data: 8 8 7 7 9 10 9 9
19000 029/300 Flush
19001 030/300 Write block 1649 len 4096 data: 3
19002 031/300 Read block 3950 len 4096
19003 032/300 Write block 196 len 4096 data: 5
19004 033/300 Read block 1518 len 12288
19005 034/300 Read block 3528 len 24576
19006 035/300 Write block 46 len 24576 data: 2 2 3 3 4 4
19007 036/300 Read block 15 len 24576
19008 037/300 Write block 812 len 16384 data: 4 5 4 4
19009 038/300 Write block 3052 len 4096 data: 4
19010 039/300 Write block 4129 len 16384 data: 6 5 3 4
19011 040/300 Read block 1959 len 8192
19012 041/300 Write block 997 len 8192 data: 5 5
19013 042/300 Read block 2499 len 28672
19014 043/300 Read block 2406 len 12288
19015 044/300 Write block 4924 len 28672 data: 4 5 5 5 6 6 5
19016 045/300 Read block 288 len 32768
19017 046/300 Write block 3773 len 28672 data: 6 7 6 6 6 5 5
19018 047/300 Read block 4684 len 4096
19019 048/300 Write block 1053 len 24576 data: 5 5 5 6 6 4
19020 049/300 Write block 1370 len 4096 data: 2
19021 050/300 Write block 2239 len 20480 data: 6 7 7 6 6
19022 051/300 Flush
19023 052/300 Flush
19024 053/300 Read block 4385 len 32768
19025 054/300 Flush
19026 055/300 Write block 2153 len 8192 data: 3 3
19027 056/300 Read block 1578 len 40960
19028 057/300 Read block 4405 len 28672
19029 058/300 Read block 4250 len 32768
19030 059/300 Read block 2293 len 28672
19031 060/300 Read block 3611 len 12288
19032 061/300 Read block 429 len 16384
19033 062/300 Flush
19034 063/300 Read block 331 len 24576
19035 064/300 Read block 142 len 28672
19036 065/300 Write block 2904 len 36864 data: 5 4 4 3 3 2 2 2 2
19037 066/300 Read block 179 len 20480
19038 067/300 Read block 4600 len 32768
19039 068/300 Read block 348 len 40960
19040 069/300 Write block 456 len 40960 data: 4 5 5 5 5 6 5 3 4 4
19041 070/300 Write block 3050 len 8192 data: 5 5
19042 071/300 Write block 4105 len 16384 data: 6 7 7 6
19043 072/300 Write block 3318 len 20480 data: 3 3 3 3 4
19044 073/300 Flush
19045 074/300 Write block 3473 len 32768 data: 4 5 5 6 6 6 7 6
19046 075/300 Write block 756 len 28672 data: 5 8 7 8 6 6 7
19047 076/300 Write block 3245 len 12288 data: 5 5 6
19048 077/300 Read block 2056 len 40960
19049 078/300 Read block 4266 len 8192
19050 079/300 Read block 3095 len 28672
19051 080/300 Read block 1743 len 32768
19052 081/300 Read block 1409 len 28672
19053 082/300 Write block 1494 len 8192 data: 5 4
19054 083/300 Read block 3406 len 32768
19055 084/300 Flush
19056 085/300 Read block 2796 len 36864
19057 086/300 Write block 1990 len 24576 data: 8 9 8 5 6 5
19058 087/300 Read block 4712 len 24576
19059 088/300 Flush
19060 089/300 Write block 3389 len 36864 data: 5 5 6 7 7 8 8 7 6
19061 090/300 Write block 2541 len 12288 data: 7 7 6
19062 091/300 Read block 318 len 8192
19063 092/300 Write block 2252 len 12288 data: 2 2 3
19064 093/300 Flush
19065 094/300 Read block 896 len 32768
19066 095/300 Flush
19067 096/300 Read block 4926 len 12288
19068 097/300 Write block 777 len 32768 data: 5 6 5 6 6 6 6 6
19069 098/300 Write block 2213 len 8192 data: 6 7
19070 099/300 Read block 4560 len 24576
19071 100/300 Read block 549 len 24576
19072 101/300 Flush
19073 102/300 Read block 18 len 40960
19074 103/300 Write block 261 len 8192 data: 3 2
19075 104/300 Flush
19076 105/300 Read block 4225 len 16384
19077 106/300 Write block 4085 len 8192 data: 3 4
19078 107/300 Write block 777 len 32768 data: 6 7 6 7 7 7 7 7
19079 108/300 Write block 4156 len 40960 data: 6 5 5 6 6 5 5 5 5 6
19080 109/300 Read block 1815 len 12288
19081 110/300 Flush
19082 111/300 Write block 4714 len 28672 data: 6 6 6 6 7 7 4
19083 112/300 Read block 2980 len 36864
19084 113/300 Read block 3646 len 20480
19085 114/300 Read block 193 len 12288
19086 115/300 Read block 4933 len 8192
19087 116/300 Read block 4168 len 12288
19088 117/300 Read block 2898 len 16384
19089 118/300 Read block 1313 len 8192
19090 119/300 Flush
19091 120/300 Write block 106 len 20480 data: 4 3 3 3 4
19092 121/300 Read block 4689 len 8192
19093 122/300 Write block 2735 len 4096 data: 4
19094 123/300 Read block 4827 len 36864
19095 124/300 Read block 2263 len 36864
19096 125/300 Write block 2081 len 12288 data: 4 4 4
19097 126/300 Write block 671 len 32768 data: 7 6 6 5 6 5 5 7
19098 127/300 Read block 1000 len 20480
19099 128/300 Read block 4732 len 16384
19100 129/300 Write block 3690 len 4096 data: 3
19101 130/300 Read block 4160 len 40960
19102 131/300 Write block 2350 len 4096 data: 4
19103 132/300 Write block 3056 len 4096 data: 4
19104 133/300 Write block 2645 len 36864 data: 3 4 3 5 5 6 6 5 5
19105 134/300 Flush
19106 135/300 Read block 1122 len 40960
19107 136/300 Read block 2579 len 20480
19108 137/300 Read block 247 len 40960
19109 138/300 Write block 3400 len 16384 data: 6 6 6 6
19110 139/300 Read block 396 len 40960
19111 140/300 Read block 4448 len 20480
19112 141/300 Read block 2047 len 16384
19113 142/300 Read block 3401 len 28672
19114 143/300 Read block 3401 len 8192
19115 144/300 Read block 1390 len 24576
19116 145/300 Read block 1747 len 8192
19117 146/300 Read block 3388 len 36864
19118 147/300 Flush
19119 148/300 Read block 3050 len 28672
19120 149/300 Write block 4986 len 4096 data: 7
19121 150/300 Read block 1974 len 20480
19122 151/300 Read block 1929 len 20480
19123 152/300 Flush
19124 153/300 Read block 1587 len 12288
19125 154/300 Flush
19126 155/300 Write block 3060 len 40960 data: 8 8 11 11 6 6 5 4 4 5
19127 156/300 Read block 3243 len 4096
19128 157/300 Write block 4619 len 12288 data: 6 7 6
19129 158/300 Write block 1129 len 36864 data: 8 8 7 5 7 7 6 6 6
19130 159/300 Write block 1608 len 40960 data: 6 5 5 6 6 8 8 9 7 4
19131 160/300 Flush
19132 161/300 Read block 333 len 24576
19133 162/300 Write block 3902 len 4096 data: 4
19134 163/300 Read block 1186 len 28672
19135 164/300 Read block 443 len 32768
19136 165/300 Flush
19137 166/300 Write block 671 len 32768 data: 8 7 7 6 7 6 6 8
19138 167/300 Read block 3960 len 4096
19139 168/300 Read block 912 len 12288
19140 169/300 Write block 1876 len 8192 data: 3 3
19141 170/300 Read block 242 len 36864
19142 171/300 Read block 58 len 8192
19143 172/300 Read block 1552 len 28672
19144 173/300 Write block 4714 len 40960 data: 7 7 7 7 8 8 5 4 5 5
19145 174/300 Flush
19146 175/300 Read block 553 len 20480
19147 176/300 Read block 4159 len 20480
19148 177/300 Read block 3194 len 36864
19149 178/300 Write block 2020 len 20480 data: 3 4 4 5 4
19150 179/300 Read block 3461 len 4096
19151 180/300 Flush
19152 181/300 Read block 1853 len 40960
19153 182/300 Flush
19154 183/300 Read block 2930 len 4096
19155 184/300 Read block 545 len 8192
19156 185/300 Read block 1480 len 36864
19157 186/300 Write block 3708 len 12288 data: 5 7 5
19158 187/300 Write block 2684 len 16384 data: 7 7 6 7
19159 188/300 Write block 4736 len 36864 data: 6 7 6 6 6 3 2 2 3
19160 189/300 Write block 4801 len 16384 data: 7 7 7 7
19161 190/300 Read block 2213 len 32768
19162 191/300 Write block 3019 len 36864 data: 6 6 6 5 4 4 5 4 4
19163 192/300 Read block 3826 len 12288
19164 193/300 Read block 1173 len 40960
19165 194/300 Write block 4643 len 20480 data: 3 2 2 2 3
19166 195/300 Write block 4852 len 24576 data: 4 4 3 4 3 3
19167 196/300 Read block 2732 len 20480
19168 197/300 Write block 1899 len 16384 data: 8 9 8 8
19169 198/300 Read block 2333 len 32768
19170 199/300 Flush
19171 200/300 Read block 1035 len 28672
19172 201/300 Write block 2856 len 8192 data: 5 5
19173 202/300 Write block 617 len 12288 data: 4 5 5
19174 203/300 Write block 413 len 8192 data: 5 5
19175 204/300 Read block 2929 len 8192
19176 205/300 Read block 3054 len 8192
19177 206/300 Read block 2332 len 8192
19178 207/300 Write block 1689 len 4096 data: 8
19179 208/300 Read block 3175 len 36864
19180 209/300 Read block 1297 len 8192
19181 210/300 Read block 3780 len 4096
19182 211/300 Write block 1805 len 28672 data: 3 3 3 3 4 4 3
19183 212/300 Write block 4505 len 8192 data: 6 6
19184 213/300 Write block 4114 len 40960 data: 4 5 5 4 4 4 4 3 5 5
19185 214/300 Write block 1384 len 28672 data: 4 6 6 7 6 7 6
19186 215/300 Write block 87 len 24576 data: 3 3 3 3 3 3
19187 216/300 Write block 1974 len 8192 data: 5 5
19188 217/300 Read block 174 len 40960
19189 218/300 Read block 3041 len 8192
19190 219/300 Read block 4087 len 28672
19191 220/300 Write block 4276 len 36864 data: 2 4 4 3 3 4 4 5 5
19192 221/300 Read block 2000 len 40960
19193 222/300 Read block 78 len 40960
19194 223/300 Read block 1418 len 4096
19195 224/300 Read block 116 len 40960
19196 225/300 Flush
19197 226/300 Read block 4111 len 40960
19198 227/300 Read block 3299 len 36864
19199 228/300 Write block 2147 len 4096 data: 5
19200 229/300 Write block 4227 len 36864 data: 7 7 6 5 5 5 4 4 4
19201 230/300 Read block 3316 len 24576
19202 231/300 Write block 475 len 16384 data: 7 7 6 7
19203 232/300 Flush
19204 233/300 Flush
19205 234/300 Write block 1505 len 20480 data: 3 3 5 4 2
19206 235/300 Read block 3265 len 24576
19207 236/300 Write block 4645 len 20480 data: 3 3 4 3 3
19208 237/300 Write block 3458 len 36864 data: 2 3 3 4 5 5 5 5 4
19209 238/300 Read block 4170 len 40960
19210 239/300 Write block 3246 len 40960 data: 6 7 5 7 6 7 7 7 6 4
19211 240/300 Read block 2777 len 32768
19212 241/300 Read block 4366 len 28672
19213 242/300 Read block 917 len 40960
19214 243/300 Write block 4262 len 40960 data: 7 7 6 5 5 5 4 4 4 4
19215 244/300 Flush
19216 245/300 Read block 3536 len 12288
19217 246/300 Write block 2899 len 40960 data: 4 5 5 5 5 6 5 5 4 4
19218 247/300 Read block 4439 len 16384
19219 248/300 Write block 3845 len 36864 data: 3 3 3 3 4 4 4 3 4
19220 249/300 Read block 3308 len 32768
19221 250/300 Write block 2731 len 8192 data: 4 6
19222 251/300 Read block 3712 len 20480
19223 252/300 Write block 829 len 20480 data: 4 3 4 4 4
19224 253/300 Read block 2328 len 24576
19225 254/300 Write block 4879 len 12288 data: 9 8 8
19226 255/300 Read block 4019 len 16384
19227 256/300 Write block 1807 len 20480 data: 4 4 5 5 4
19228 257/300 Write block 4542 len 32768 data: 6 7 9 8 7 8 6 6
19229 258/300 Write block 3381 len 8192 data: 5 6
19230 259/300 Write block 453 len 36864 data: 5 5 5 5 6 6 6 6 7
19231 260/300 Write block 3909 len 8192 data: 6 5
19232 261/300 Write block 460 len 12288 data: 7 8 6
19233 262/300 Read block 2356 len 32768
19234 263/300 Read block 4461 len 28672
19235 264/300 Read block 2865 len 32768
19236 265/300 Write block 990 len 16384 data: 4 5 5 3
19237 266/300 Read block 3530 len 36864
19238 267/300 Write block 209 len 28672 data: 5 4 3 5 4 4 3
19239 268/300 Flush
19240 269/300 Read block 2322 len 16384
19241 270/300 Write block 4790 len 36864 data: 6 6 7 3 4 4 5 5 6
19242 271/300 Write block 748 len 40960 data: 4 5 6 6 8 7 7 6 6 9
19243 272/300 Read block 973 len 40960
19244 273/300 Write block 3702 len 20480 data: 5 4 6 5 4
19245 274/300 Read block 3710 len 40960
19246 275/300 Read block 46 len 8192
19247 276/300 Write block 926 len 16384 data: 4 4 4 4
19248 277/300 Flush
19249 278/300 Flush
19250 279/300 Flush
19251 280/300 Read block 1937 len 8192
19252 281/300 Write block 4324 len 36864 data: 4 3 3 4 5 6 8 7 8
19253 282/300 Write block 317 len 28672 data: 7 7 6 6 5 5 5
19254 283/300 Read block 2498 len 28672
19255 284/300 Read block 792 len 40960
19256 285/300 Read block 1134 len 8192
19257 286/300 Read block 2446 len 28672
19258 287/300 Write block 3016 len 20480 data: 7 5 5 7 7
19259 288/300 Read block 4814 len 32768
19260 289/300 Write block 4937 len 32768 data: 2 3 3 3 5 6 6 5
19261 290/300 Read block 9 len 20480
19262 291/300 Read block 2129 len 8192
19263 292/300 Write block 1637 len 36864 data: 7 7 7 7 7 7 7 6 3
19264 293/300 Write block 3611 len 12288 data: 2 3 4
19265 294/300 Read block 2551 len 16384
19266 295/300 Read block 722 len 36864
19267 296/300 Read block 1177 len 4096
19268 297/300 Write block 3191 len 40960 data: 5 5 3 3 3 3 5 5 4 4
19269 298/300 Write block 2821 len 24576 data: 6 7 6 4 3 4
19270 299/300 Read block 3392 len 12288
19271 300/300 Read block 1767 len 16384
19272 Main task has work to do, go find it
19273 got dsc Start(1)
19274 start 1
19275 Feb 28 22:17:08.389 INFO request completed, latency_us: 203, response_code: 204, uri: /start/cid/1, method: POST, req_id: 9ff5e9de-b326-4bf3-a89d-b42dce03e041, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
19276 Replay: started 1, returned:Ok(())
19277 ----------------------------------------------------------------
19278 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19279 States: Active Offline Active
19280 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
19281 6632 Acked 7631 Read 9 Done New Done true
19282 6633 Acked 7632 Read 2 Done New Done true
19283 6634 Acked 7633 Read 1 Done New Done false
19284 6635 Acked 7634 Write 9 Done New Done false
19285 6636 Acked 7635 Flush 0 Done New Done false
19286 6637 Acked 7636 Read 2 Done New Done false
19287 6638 Acked 7637 Write 6 Done New Done false
19288 6639 Acked 7638 Read 9 Done New Done false
19289 6640 Acked 7639 Read 3 Done New Done false
19290 6641 Acked 7640 Write 2 Done New Done false
19291 6642 Acked 7641 Read 4 Done New Done false
19292 6643 Acked 7642 Write 10 Done New Done false
19293 6644 Acked 7643 Write 2 Done New Done false
19294 6645 Acked 7644 Flush 0 Done New Done false
19295 6646 Acked 7645 Flush 0 Done New Done false
19296 6647 Acked 7646 Read 6 Done New Done false
19297 6648 Acked 7647 Read 7 Done New Done false
19298 6649 Acked 7648 Write 7 Done New Done false
19299 6650 Acked 7649 Read 8 Done New Done false
19300 6651 Acked 7650 Write 10 Done New Done false
19301 6652 Acked 7651 Write 7 Done New Done false
19302 6653 Acked 7652 Read 8 Done New Done false
19303 6654 Acked 7653 Read 1 Done New Done false
19304 6655 Acked 7654 Flush 0 Done New Done false
19305 6656 Acked 7655 Write 9 Done New Done false
19306 6657 Acked 7656 Read 7 Done New Done false
19307 6658 Acked 7657 Flush 0 Done New Done false
19308 6659 Acked 7658 Write 8 Done New Done false
19309 6660 Acked 7659 Flush 0 Done New Done false
19310 6661 Acked 7660 Write 1 Done New Done false
19311 6662 Acked 7661 Read 1 Done New Done false
19312 6663 Acked 7662 Write 1 Done New Done false
19313 6664 Acked 7663 Read 3 Done New Done false
19314 6665 Acked 7664 Read 6 Done New Done false
19315 6666 Acked 7665 Write 6 Done New Done false
19316 6667 Acked 7666 Read 6 Done New Done false
19317 6668 Acked 7667 Write 4 Done New Done false
19318 6669 Acked 7668 Write 1 Done New Done false
19319 6670 Acked 7669 Write 4 Done New Done false
19320 6671 Acked 7670 Read 2 Done New Done false
19321 6672 Acked 7671 Write 2 Done New Done false
19322 6673 Acked 7672 Read 7 Done New Done false
19323 6674 Acked 7673 Read 3 Done New Done false
19324 6675 Acked 7674 Write 7 Done New Done false
19325 6676 Acked 7675 Read 8 Done New Done false
19326 6677 Acked 7676 Write 7 Done New Done false
19327 6678 Acked 7677 Read 1 Done New Done false
19328 6679 Acked 7678 Write 6 Done New Done false
19329 6680 Acked 7679 Write 1 Done New Done false
19330 6681 Acked 7680 Write 5 Done New Done false
19331 6682 Acked 7681 Flush 0 Done New Done false
19332 6683 Acked 7682 Flush 0 Done New Done false
19333 6684 Acked 7683 Read 8 Done New Done false
19334 6685 Acked 7684 Flush 0 Done New Done false
19335 6686 Acked 7685 Write 2 Done New Done false
19336 6687 Acked 7686 Read 10 Done New Done false
19337 6688 Acked 7687 Read 7 Done New Done false
19338 6689 Acked 7688 Read 8 Done New Done false
19339 6690 Acked 7689 Read 7 Done New Done false
19340 6691 Acked 7690 Read 3 Done New Done false
19341 6692 Acked 7691 Read 4 Done New Done false
19342 6693 Acked 7692 Flush 0 Done New Done false
19343 6694 Acked 7693 Read 6 Done New Done false
19344 6695 Acked 7694 Read 7 Done New Done false
19345 6696 Acked 7695 Write 9 Done New Done false
19346 6697 Acked 7696 Read 5 Done New Done false
19347 6698 Acked 7697 Read 8 Done New Done false
19348 6699 Acked 7698 Read 10 Done New Done false
19349 6700 Acked 7699 Write 10 Done New Done false
19350 6701 Acked 7700 Write 2 Done New Done false
19351 6702 Acked 7701 Write 4 Done New Done false
19352 6703 Acked 7702 Write 5 Done New Done false
19353 6704 Acked 7703 Flush 0 Done New Done false
19354 6705 Acked 7704 Write 8 Done New Done false
19355 6706 Acked 7705 Write 7 Done New Done false
19356 6707 Acked 7706 Write 3 Done New Done false
19357 6708 Acked 7707 Read 10 Done New Done false
19358 6709 Acked 7708 Read 2 Done New Done false
19359 6710 Acked 7709 Read 7 Done New Done false
19360 6711 Acked 7710 Read 8 Done New Done false
19361 6712 Acked 7711 Read 7 Done New Done false
19362 6713 Acked 7712 Write 2 Done New Done false
19363 6714 Acked 7713 Read 8 Done New Done false
19364 6715 Acked 7714 Flush 0 Done New Done false
19365 6716 Acked 7715 Read 9 Done New Done false
19366 6717 Acked 7716 Write 6 Done New Done false
19367 6718 Acked 7717 Read 6 Done New Done false
19368 6719 Acked 7718 Flush 0 Done New Done false
19369 6720 Acked 7719 Write 9 Done New Done false
19370 6721 Acked 7720 Write 3 Done New Done false
19371 6722 Acked 7721 Read 2 Done New Done false
19372 6723 Acked 7722 Write 3 Done New Done false
19373 6724 Acked 7723 Flush 0 Done New Done false
19374 6725 Acked 7724 Read 8 Done New Done false
19375 6726 Acked 7725 Flush 0 Done New Done false
19376 6727 Acked 7726 Read 3 Done New Done false
19377 6728 Acked 7727 Write 8 Done New Done false
19378 6729 Acked 7728 Write 2 Done New Done false
19379 6730 Acked 7729 Read 6 Done New Done false
19380 6731 Acked 7730 Read 6 Done New Done false
19381 6732 Acked 7731 Flush 0 Done New Done false
19382 6733 Acked 7732 Read 10 Done New Done false
19383 6734 Acked 7733 Write 2 Done New Done false
19384 6735 Acked 7734 Flush 0 Done New Done false
19385 6736 Acked 7735 Read 4 Done New Done false
19386 6737 Acked 7736 Write 2 Done New Done false
19387 6738 Acked 7737 Write 8 Done New Done false
19388 6739 Acked 7738 Write 10 Done New Done false
19389 6740 Acked 7739 Read 3 Done New Done false
19390 6741 Acked 7740 Flush 0 Done New Done false
19391 6742 Acked 7741 Write 7 Done New Done false
19392 6743 Acked 7742 Read 9 Done New Done false
19393 6744 Acked 7743 Read 5 Done New Done false
19394 6745 Acked 7744 Read 3 Done New Done false
19395 6746 Acked 7745 Read 2 Done New Done false
19396 6747 Acked 7746 Read 3 Done New Done false
19397 6748 Acked 7747 Read 4 Done New Done false
19398 6749 Acked 7748 Read 2 Done New Done false
19399 6750 Acked 7749 Flush 0 Done New Done false
19400 6751 Acked 7750 Write 5 Done New Done false
19401 6752 Acked 7751 Read 2 Done New Done false
19402 6753 Acked 7752 Write 1 Done New Done false
19403 6754 Acked 7753 Read 9 Done New Done false
19404 6755 Acked 7754 Read 9 Done New Done false
19405 6756 Acked 7755 Write 3 Done New Done false
19406 6757 Acked 7756 Write 8 Done New Done false
19407 6758 Acked 7757 Read 5 Done New Done false
19408 6759 Acked 7758 Read 4 Done New Done false
19409 6760 Acked 7759 Write 1 Done New Done false
19410 6761 Acked 7760 Read 10 Done New Done false
19411 6762 Acked 7761 Write 1 Done New Done false
19412 6763 Acked 7762 Write 1 Done New Done false
19413 6764 Acked 7763 Write 9 Done New Done false
19414 6765 Acked 7764 Flush 0 Done New Done false
19415 6766 Acked 7765 Read 10 Done New Done false
19416 6767 Acked 7766 Read 5 Done New Done false
19417 6768 Acked 7767 Read 10 Done New Done false
19418 6769 Acked 7768 Write 4 Done New Done false
19419 6770 Acked 7769 Read 10 Done New Done false
19420 6771 Acked 7770 Read 5 Done New Done false
19421 6772 Acked 7771 Read 4 Done New Done false
19422 6773 Acked 7772 Read 7 Done New Done false
19423 6774 Acked 7773 Read 2 Done New Done false
19424 6775 Acked 7774 Read 6 Done New Done false
19425 6776 Acked 7775 Read 2 Done New Done false
19426 6777 Acked 7776 Read 9 Done New Done false
19427 6778 Acked 7777 Flush 0 Done New Done false
19428 6779 Acked 7778 Read 7 Done New Done false
19429 6780 Acked 7779 Write 1 Done New Done false
19430 6781 Acked 7780 Read 5 Done New Done false
19431 6782 Acked 7781 Read 5 Done New Done false
19432 6783 Acked 7782 Flush 0 Done New Done false
19433 6784 Acked 7783 Read 3 Done New Done false
19434 6785 Acked 7784 Flush 0 Done New Done false
19435 6786 Acked 7785 Write 10 Done New Done false
19436 6787 Acked 7786 Read 1 Done New Done false
19437 6788 Acked 7787 Write 3 Done New Done false
19438 6789 Acked 7788 Write 9 Done New Done false
19439 6790 Acked 7789 Write 10 Done New Done false
19440 6791 Acked 7790 Flush 0 Done New Done false
19441 6792 Acked 7791 Read 6 Done New Done false
19442 6793 Acked 7792 Write 1 Done New Done false
19443 6794 Acked 7793 Read 7 Done New Done false
19444 6795 Acked 7794 Read 8 Done New Done false
19445 6796 Acked 7795 Flush 0 Done New Done false
19446 6797 Acked 7796 Write 8 Done New Done false
19447 6798 Acked 7797 Read 1 Done New Done false
19448 6799 Acked 7798 Read 3 Done New Done false
19449 6800 Acked 7799 Write 2 Done New Done false
19450 6801 Acked 7800 Read 9 Done New Done false
19451 6802 Acked 7801 Read 2 Done New Done false
19452 6803 Acked 7802 Read 7 Done New Done false
19453 6804 Acked 7803 Write 10 Done New Done false
19454 6805 Acked 7804 Flush 0 Done New Done false
19455 6806 Acked 7805 Read 5 Done New Done false
19456 6807 Acked 7806 Read 5 Done New Done false
19457 6808 Acked 7807 Read 9 Done New Done false
19458 6809 Acked 7808 Write 5 Done New Done false
19459 6810 Acked 7809 Read 1 Done New Done false
19460 6811 Acked 7810 Flush 0 Done New Done false
19461 6812 Acked 7811 Read 10 Done New Done false
19462 6813 Acked 7812 Flush 0 Done New Done false
19463 6814 Acked 7813 Read 1 Done New Done false
19464 6815 Acked 7814 Read 2 Done New Done false
19465 6816 Acked 7815 Read 9 Done New Done false
19466 6817 Acked 7816 Write 3 Done New Done false
19467 6818 Acked 7817 Write 4 Done New Done false
19468 6819 Acked 7818 Write 9 Done New Done false
19469 6820 Acked 7819 Write 4 Done New Done false
19470 6821 Acked 7820 Read 8 Done New Done false
19471 6822 Acked 7821 Write 9 Done New Done false
19472 6823 Acked 7822 Read 3 Done New Done false
19473 6824 Acked 7823 Read 10 Done New Done false
19474 6825 Acked 7824 Write 5 Done New Done false
19475 6826 Acked 7825 Write 6 Done New Done false
19476 6827 Acked 7826 Read 5 Done New Done false
19477 6828 Acked 7827 Write 4 Done New Done false
19478 6829 Acked 7828 Read 8 Done New Done false
19479 6830 Acked 7829 Flush 0 Done New Done false
19480 6831 Acked 7830 Read 7 Done New Done false
19481 6832 Acked 7831 Write 2 Done New Done false
19482 6833 Acked 7832 Write 3 Done New Done false
19483 6834 Acked 7833 Write 2 Done New Done false
19484 6835 Acked 7834 Read 2 Done New Done false
19485 6836 Acked 7835 Read 2 Done New Done false
19486 6837 Acked 7836 Read 2 Done New Done false
19487 6838 Acked 7837 Write 1 Done New Done false
19488 6839 Acked 7838 Read 9 Done New Done false
19489 6840 Acked 7839 Read 2 Done New Done false
19490 6841 Acked 7840 Read 1 Done New Done false
19491 6842 Acked 7841 Write 7 Done New Done false
19492 6843 Acked 7842 Write 2 Done New Done false
19493 6844 Acked 7843 Write 10 Done New Done false
19494 6845 Acked 7844 Write 7 Done New Done false
19495 6846 Acked 7845 Write 6 Done New Done false
19496 6847 Acked 7846 Write 2 Done New Done false
19497 6848 Acked 7847 Read 10 Done New Done false
19498 6849 Acked 7848 Read 2 Done New Done false
19499 6850 Acked 7849 Read 7 Done New Done false
19500 6851 Acked 7850 Flush 0 Done New Done false
19501 6852 Acked 7851 Write 9 Done New Done false
19502 6853 Acked 7852 Read 10 Done New Done false
19503 6854 Acked 7853 Read 10 Done New Done false
19504 6855 Acked 7854 Read 1 Done New Done false
19505 6856 Acked 7855 Read 10 Done New Done false
19506 6857 Acked 7856 Flush 0 Done New Done false
19507 6858 Acked 7857 Read 10 Done New Done false
19508 6859 Acked 7858 Read 9 Done New Done false
19509 6860 Acked 7859 Write 1 Done New Done false
19510 6861 Acked 7860 Write 9 Done New Done false
19511 6862 Acked 7861 Read 6 Done New Done false
19512 6863 Acked 7862 Write 4 Done New Done false
19513 6864 Acked 7863 Flush 0 Done New Done false
19514 6865 Acked 7864 Flush 0 Done New Done false
19515 6866 Acked 7865 Write 5 Done New Done false
19516 6867 Acked 7866 Read 6 Done New Done false
19517 6868 Acked 7867 Write 5 Done New Done false
19518 6869 Acked 7868 Write 9 Done New Done false
19519 6870 Acked 7869 Read 10 Done New Done false
19520 6871 Acked 7870 Write 10 Done New Done false
19521 6872 Acked 7871 Read 8 Done New Done false
19522 6873 Acked 7872 Read 7 Done New Done false
19523 6874 Acked 7873 Read 10 Done New Done false
19524 6875 Acked 7874 Write 10 Done New Done false
19525 6876 Acked 7875 Flush 0 Done New Done false
19526 6877 Acked 7876 Read 3 Done New Done false
19527 6878 Acked 7877 Write 10 Done New Done false
19528 6879 Acked 7878 Read 4 Done New Done false
19529 6880 Acked 7879 Write 9 Done New Done false
19530 6881 Acked 7880 Read 8 Done New Done false
19531 6882 Acked 7881 Write 2 Done New Done false
19532 6883 Acked 7882 Read 5 Done New Done false
19533 6884 Acked 7883 Write 5 Done New Done false
19534 6885 Acked 7884 Read 6 Done New Done false
19535 6886 Acked 7885 Write 3 Done New Done false
19536 6887 Acked 7886 Read 4 Done New Done false
19537 6888 Acked 7887 Write 5 Done New Done false
19538 6889 Acked 7888 Write 8 Done New Done false
19539 6890 Acked 7889 Write 2 Done New Done false
19540 6891 Acked 7890 Write 9 Done New Done false
19541 6892 Acked 7891 Write 2 Done New Done false
19542 6893 Acked 7892 Write 3 Done New Done false
19543 6894 Acked 7893 Read 8 Done New Done false
19544 6895 Acked 7894 Read 7 Done New Done false
19545 6896 Acked 7895 Read 8 Done New Done false
19546 6897 Acked 7896 Write 4 Done New Done false
19547 6898 Acked 7897 Read 9 Done New Done false
19548 6899 Acked 7898 Write 7 Done New Done false
19549 6900 Acked 7899 Flush 0 Done New Done false
19550 6901 Acked 7900 Read 4 Done New Done false
19551 6902 Acked 7901 Write 9 Done New Done false
19552 6903 Acked 7902 Write 10 Done New Done false
19553 6904 Acked 7903 Read 10 Done New Done false
19554 6905 Acked 7904 Write 5 Done New Done false
19555 6906 Acked 7905 Read 10 Done New Done false
19556 6907 Acked 7906 Read 2 Done New Done false
19557 6908 Acked 7907 Write 4 Done New Done false
19558 6909 Acked 7908 Flush 0 Done New Done false
19559 6910 Acked 7909 Flush 0 Done New Done false
19560 6911 Acked 7910 Flush 0 Done New Done false
19561 6912 Acked 7911 Read 2 Done New Done false
19562 6913 Acked 7912 Write 9 Done New Done false
19563 6914 Acked 7913 Write 7 Done New Done false
19564 6915 Acked 7914 Read 7 Done New Done false
19565 6916 Acked 7915 Read 10 Done New Done false
19566 6917 Acked 7916 Read 2 Done New Done false
19567 6918 Acked 7917 Read 7 Done New Done false
19568 6919 Acked 7918 Write 5 Done New Done false
19569 6920 Acked 7919 Read 8 Done New Done false
19570 6921 Acked 7920 Write 8 Done New Done false
19571 6922 Acked 7921 Read 5 Done New Done false
19572 6923 Acked 7922 Read 2 Done New Done false
19573 6924 Acked 7923 Write 9 Done New Done false
19574 6925 Acked 7924 Write 3 Done New Done false
19575 6926 Acked 7925 Read 4 Done New Done false
19576 6927 Acked 7926 Read 9 Done New Done false
19577 6928 Acked 7927 Read 1 Done New Done false
19578 6929 Acked 7928 Write 10 Done New Done false
19579 6930 Acked 7929 Write 6 Done New Done false
19580 6931 Acked 7930 Read 3 Done New Done false
19581 6932 Acked 7931 Read 4 Done New Done false
19582 STATES DS:0 DS:1 DS:2 TOTAL
19583 New 0 301 0 301
19584 Sent 0 0 0 0
19585 Done 301 0 301 602
19586 Skipped 0 0 0 0
19587 Error 0 0 0 0
19588 Last Flush: 7910 7630 7910
19589 Downstairs last five completed: 7630 7629 7628 7627 7626
19590 Upstairs last five completed: 6932 6931 6930 6929 6928
19591 CLIENT: Up:0 ds:301 act:2
195922024-02-28T22:17:09.057ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
195932024-02-28T22:17:09.057ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
195942024-02-28T22:17:09.057ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
195952024-02-28T22:17:09.057ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
195962024-02-28T22:17:09.057ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
195972024-02-28T22:17:09.057ZINFOcrucible: [1] client re-new 302 jobs since flush 7630 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
195982024-02-28T22:17:09.057ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
195992024-02-28T22:17:10.058ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
196002024-02-28T22:17:10.058ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
196012024-02-28T22:17:10.058ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
196022024-02-28T22:17:10.058ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
196032024-02-28T22:17:10.058ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
196042024-02-28T22:17:10.058ZINFOcrucible: [1] client re-new 302 jobs since flush 7630 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
196052024-02-28T22:17:10.058ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
196062024-02-28T22:17:11.059ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
196072024-02-28T22:17:11.059ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
196082024-02-28T22:17:11.060ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
196092024-02-28T22:17:11.060ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
196102024-02-28T22:17:11.060ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
196112024-02-28T22:17:11.060ZINFOcrucible: [1] client re-new 302 jobs since flush 7630 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
196122024-02-28T22:17:11.060ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
196132024-02-28T22:17:12.062ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
196142024-02-28T22:17:12.062ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
196152024-02-28T22:17:12.062ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
196162024-02-28T22:17:12.062ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
196172024-02-28T22:17:12.062ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
196182024-02-28T22:17:12.062ZINFOcrucible: [1] client re-new 302 jobs since flush 7630 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
196192024-02-28T22:17:12.062ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
19620 ----------------------------------------------------------------
19621 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
19622 States: Active Offline Active
19623 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
19624 6632 Acked 7631 Read 9 Done New Done true
19625 6633 Acked 7632 Read 2 Done New Done true
19626 6634 Acked 7633 Read 1 Done New Done true
19627 6635 Acked 7634 Write 9 Done New Done true
19628 6636 Acked 7635 Flush 0 Done New Done true
19629 6637 Acked 7636 Read 2 Done New Done true
19630 6638 Acked 7637 Write 6 Done New Done true
19631 6639 Acked 7638 Read 9 Done New Done true
19632 6640 Acked 7639 Read 3 Done New Done true
19633 6641 Acked 7640 Write 2 Done New Done true
19634 6642 Acked 7641 Read 4 Done New Done true
19635 6643 Acked 7642 Write 10 Done New Done true
19636 6644 Acked 7643 Write 2 Done New Done true
19637 6645 Acked 7644 Flush 0 Done New Done true
19638 6646 Acked 7645 Flush 0 Done New Done true
19639 6647 Acked 7646 Read 6 Done New Done true
19640 6648 Acked 7647 Read 7 Done New Done true
19641 6649 Acked 7648 Write 7 Done New Done true
19642 6650 Acked 7649 Read 8 Done New Done true
19643 6651 Acked 7650 Write 10 Done New Done true
19644 6652 Acked 7651 Write 7 Done New Done true
19645 6653 Acked 7652 Read 8 Done New Done true
19646 6654 Acked 7653 Read 1 Done New Done true
19647 6655 Acked 7654 Flush 0 Done New Done true
19648 6656 Acked 7655 Write 9 Done New Done true
19649 6657 Acked 7656 Read 7 Done New Done true
19650 6658 Acked 7657 Flush 0 Done New Done true
19651 6659 Acked 7658 Write 8 Done New Done true
19652 6660 Acked 7659 Flush 0 Done New Done true
19653 6661 Acked 7660 Write 1 Done New Done true
19654 6662 Acked 7661 Read 1 Done New Done true
19655 6663 Acked 7662 Write 1 Done New Done true
19656 6664 Acked 7663 Read 3 Done New Done true
19657 6665 Acked 7664 Read 6 Done New Done true
19658 6666 Acked 7665 Write 6 Done New Done true
19659 6667 Acked 7666 Read 6 Done New Done true
19660 6668 Acked 7667 Write 4 Done New Done true
19661 6669 Acked 7668 Write 1 Done New Done true
19662 6670 Acked 7669 Write 4 Done New Done true
19663 6671 Acked 7670 Read 2 Done New Done true
19664 6672 Acked 7671 Write 2 Done New Done true
19665 6673 Acked 7672 Read 7 Done New Done true
19666 6674 Acked 7673 Read 3 Done New Done true
19667 6675 Acked 7674 Write 7 Done New Done true
19668 6676 Acked 7675 Read 8 Done New Done true
19669 6677 Acked 7676 Write 7 Done New Done true
19670 6678 Acked 7677 Read 1 Done New Done true
19671 6679 Acked 7678 Write 6 Done New Done true
19672 6680 Acked 7679 Write 1 Done New Done true
19673 6681 Acked 7680 Write 5 Done New Done true
19674 6682 Acked 7681 Flush 0 Done New Done true
19675 6683 Acked 7682 Flush 0 Done New Done true
19676 6684 Acked 7683 Read 8 Done New Done true
19677 6685 Acked 7684 Flush 0 Done New Done true
19678 6686 Acked 7685 Write 2 Done New Done true
19679 6687 Acked 7686 Read 10 Done New Done true
19680 6688 Acked 7687 Read 7 Done New Done true
19681 6689 Acked 7688 Read 8 Done New Done true
19682 6690 Acked 7689 Read 7 Done New Done true
19683 6691 Acked 7690 Read 3 Done New Done true
19684 6692 Acked 7691 Read 4 Done New Done true
19685 6693 Acked 7692 Flush 0 Done New Done true
19686 6694 Acked 7693 Read 6 Done New Done true
19687 6695 Acked 7694 Read 7 Done New Done true
19688 6696 Acked 7695 Write 9 Done New Done true
19689 6697 Acked 7696 Read 5 Done New Done true
19690 6698 Acked 7697 Read 8 Done New Done true
19691 6699 Acked 7698 Read 10 Done New Done true
19692 6700 Acked 7699 Write 10 Done New Done true
19693 6701 Acked 7700 Write 2 Done New Done true
19694 6702 Acked 7701 Write 4 Done New Done true
19695 6703 Acked 7702 Write 5 Done New Done true
19696 6704 Acked 7703 Flush 0 Done New Done true
19697 6705 Acked 7704 Write 8 Done New Done true
19698 6706 Acked 7705 Write 7 Done New Done true
19699 6707 Acked 7706 Write 3 Done New Done true
19700 6708 Acked 7707 Read 10 Done New Done true
19701 6709 Acked 7708 Read 2 Done New Done true
19702 6710 Acked 7709 Read 7 Done New Done true
19703 6711 Acked 7710 Read 8 Done New Done true
19704 6712 Acked 7711 Read 7 Done New Done true
19705 6713 Acked 7712 Write 2 Done New Done true
19706 6714 Acked 7713 Read 8 Done New Done true
19707 6715 Acked 7714 Flush 0 Done New Done true
19708 6716 Acked 7715 Read 9 Done New Done true
19709 6717 Acked 7716 Write 6 Done New Done true
19710 6718 Acked 7717 Read 6 Done New Done true
19711 6719 Acked 7718 Flush 0 Done New Done true
19712 6720 Acked 7719 Write 9 Done New Done true
19713 6721 Acked 7720 Write 3 Done New Done true
19714 6722 Acked 7721 Read 2 Done New Done true
19715 6723 Acked 7722 Write 3 Done New Done true
19716 6724 Acked 7723 Flush 0 Done New Done true
19717 6725 Acked 7724 Read 8 Done New Done true
19718 6726 Acked 7725 Flush 0 Done New Done true
19719 6727 Acked 7726 Read 3 Done New Done true
19720 6728 Acked 7727 Write 8 Done New Done true
19721 6729 Acked 7728 Write 2 Done New Done true
19722 6730 Acked 7729 Read 6 Done New Done true
19723 6731 Acked 7730 Read 6 Done New Done true
19724 6732 Acked 7731 Flush 0 Done New Done true
19725 6733 Acked 7732 Read 10 Done New Done true
19726 6734 Acked 7733 Write 2 Done New Done true
19727 6735 Acked 7734 Flush 0 Done New Done true
19728 6736 Acked 7735 Read 4 Done New Done true
19729 6737 Acked 7736 Write 2 Done New Done true
19730 6738 Acked 7737 Write 8 Done New Done true
19731 6739 Acked 7738 Write 10 Done New Done true
19732 6740 Acked 7739 Read 3 Done New Done true
19733 6741 Acked 7740 Flush 0 Done New Done true
19734 6742 Acked 7741 Write 7 Done New Done true
19735 6743 Acked 7742 Read 9 Done New Done true
19736 6744 Acked 7743 Read 5 Done New Done true
19737 6745 Acked 7744 Read 3 Done New Done true
19738 6746 Acked 7745 Read 2 Done New Done true
19739 6747 Acked 7746 Read 3 Done New Done true
19740 6748 Acked 7747 Read 4 Done New Done true
19741 6749 Acked 7748 Read 2 Done New Done true
19742 6750 Acked 7749 Flush 0 Done New Done true
19743 6751 Acked 7750 Write 5 Done New Done true
19744 6752 Acked 7751 Read 2 Done New Done true
19745 6753 Acked 7752 Write 1 Done New Done true
19746 6754 Acked 7753 Read 9 Done New Done true
19747 6755 Acked 7754 Read 9 Done New Done true
19748 6756 Acked 7755 Write 3 Done New Done true
19749 6757 Acked 7756 Write 8 Done New Done true
19750 6758 Acked 7757 Read 5 Done New Done true
19751 6759 Acked 7758 Read 4 Done New Done true
19752 6760 Acked 7759 Write 1 Done New Done true
19753 6761 Acked 7760 Read 10 Done New Done true
19754 6762 Acked 7761 Write 1 Done New Done true
19755 6763 Acked 7762 Write 1 Done New Done true
19756 6764 Acked 7763 Write 9 Done New Done true
19757 6765 Acked 7764 Flush 0 Done New Done true
19758 6766 Acked 7765 Read 10 Done New Done true
19759 6767 Acked 7766 Read 5 Done New Done true
19760 6768 Acked 7767 Read 10 Done New Done true
19761 6769 Acked 7768 Write 4 Done New Done true
19762 6770 Acked 7769 Read 10 Done New Done true
19763 6771 Acked 7770 Read 5 Done New Done true
19764 6772 Acked 7771 Read 4 Done New Done true
19765 6773 Acked 7772 Read 7 Done New Done true
19766 6774 Acked 7773 Read 2 Done New Done true
19767 6775 Acked 7774 Read 6 Done New Done true
19768 6776 Acked 7775 Read 2 Done New Done true
19769 6777 Acked 7776 Read 9 Done New Done true
19770 6778 Acked 7777 Flush 0 Done New Done true
19771 6779 Acked 7778 Read 7 Done New Done true
19772 6780 Acked 7779 Write 1 Done New Done true
19773 6781 Acked 7780 Read 5 Done New Done true
19774 6782 Acked 7781 Read 5 Done New Done true
19775 6783 Acked 7782 Flush 0 Done New Done true
19776 6784 Acked 7783 Read 3 Done New Done true
19777 6785 Acked 7784 Flush 0 Done New Done true
19778 6786 Acked 7785 Write 10 Done New Done true
19779 6787 Acked 7786 Read 1 Done New Done true
19780 6788 Acked 7787 Write 3 Done New Done true
19781 6789 Acked 7788 Write 9 Done New Done true
19782 6790 Acked 7789 Write 10 Done New Done true
19783 6791 Acked 7790 Flush 0 Done New Done true
19784 6792 Acked 7791 Read 6 Done New Done true
19785 6793 Acked 7792 Write 1 Done New Done true
19786 6794 Acked 7793 Read 7 Done New Done true
19787 6795 Acked 7794 Read 8 Done New Done true
19788 6796 Acked 7795 Flush 0 Done New Done true
19789 6797 Acked 7796 Write 8 Done New Done true
19790 6798 Acked 7797 Read 1 Done New Done true
19791 6799 Acked 7798 Read 3 Done New Done true
19792 6800 Acked 7799 Write 2 Done New Done true
19793 6801 Acked 7800 Read 9 Done New Done true
19794 6802 Acked 7801 Read 2 Done New Done true
19795 6803 Acked 7802 Read 7 Done New Done true
19796 6804 Acked 7803 Write 10 Done New Done true
19797 6805 Acked 7804 Flush 0 Done New Done true
19798 6806 Acked 7805 Read 5 Done New Done true
19799 6807 Acked 7806 Read 5 Done New Done true
19800 6808 Acked 7807 Read 9 Done New Done true
19801 6809 Acked 7808 Write 5 Done New Done true
19802 6810 Acked 7809 Read 1 Done New Done true
19803 6811 Acked 7810 Flush 0 Done New Done true
19804 6812 Acked 7811 Read 10 Done New Done true
19805 6813 Acked 7812 Flush 0 Done New Done true
19806 6814 Acked 7813 Read 1 Done New Done true
19807 6815 Acked 7814 Read 2 Done New Done true
19808 6816 Acked 7815 Read 9 Done New Done true
19809 6817 Acked 7816 Write 3 Done New Done true
19810 6818 Acked 7817 Write 4 Done New Done true
19811 6819 Acked 7818 Write 9 Done New Done true
19812 6820 Acked 7819 Write 4 Done New Done true
19813 6821 Acked 7820 Read 8 Done New Done true
19814 6822 Acked 7821 Write 9 Done New Done true
19815 6823 Acked 7822 Read 3 Done New Done true
19816 6824 Acked 7823 Read 10 Done New Done true
19817 6825 Acked 7824 Write 5 Done New Done true
19818 6826 Acked 7825 Write 6 Done New Done true
19819 6827 Acked 7826 Read 5 Done New Done true
19820 6828 Acked 7827 Write 4 Done New Done true
19821 6829 Acked 7828 Read 8 Done New Done true
19822 6830 Acked 7829 Flush 0 Done New Done true
19823 6831 Acked 7830 Read 7 Done New Done true
19824 6832 Acked 7831 Write 2 Done New Done true
19825 6833 Acked 7832 Write 3 Done New Done true
19826 6834 Acked 7833 Write 2 Done New Done true
19827 6835 Acked 7834 Read 2 Done New Done true
19828 6836 Acked 7835 Read 2 Done New Done true
19829 6837 Acked 7836 Read 2 Done New Done true
19830 6838 Acked 7837 Write 1 Done New Done true
19831 6839 Acked 7838 Read 9 Done New Done true
19832 6840 Acked 7839 Read 2 Done New Done true
19833 6841 Acked 7840 Read 1 Done New Done true
19834 6842 Acked 7841 Write 7 Done New Done true
19835 6843 Acked 7842 Write 2 Done New Done true
19836 6844 Acked 7843 Write 10 Done New Done true
19837 6845 Acked 7844 Write 7 Done New Done true
19838 6846 Acked 7845 Write 6 Done New Done true
19839 6847 Acked 7846 Write 2 Done New Done true
19840 6848 Acked 7847 Read 10 Done New Done true
19841 6849 Acked 7848 Read 2 Done New Done true
19842 6850 Acked 7849 Read 7 Done New Done true
19843 6851 Acked 7850 Flush 0 Done New Done true
19844 6852 Acked 7851 Write 9 Done New Done true
19845 6853 Acked 7852 Read 10 Done New Done true
19846 6854 Acked 7853 Read 10 Done New Done true
19847 6855 Acked 7854 Read 1 Done New Done true
19848 6856 Acked 7855 Read 10 Done New Done true
19849 6857 Acked 7856 Flush 0 Done New Done true
19850 6858 Acked 7857 Read 10 Done New Done true
19851 6859 Acked 7858 Read 9 Done New Done true
19852 6860 Acked 7859 Write 1 Done New Done true
19853 6861 Acked 7860 Write 9 Done New Done true
19854 6862 Acked 7861 Read 6 Done New Done true
19855 6863 Acked 7862 Write 4 Done New Done true
19856 6864 Acked 7863 Flush 0 Done New Done true
19857 6865 Acked 7864 Flush 0 Done New Done true
19858 6866 Acked 7865 Write 5 Done New Done true
19859 6867 Acked 7866 Read 6 Done New Done true
19860 6868 Acked 7867 Write 5 Done New Done true
19861 6869 Acked 7868 Write 9 Done New Done true
19862 6870 Acked 7869 Read 10 Done New Done true
19863 6871 Acked 7870 Write 10 Done New Done true
19864 6872 Acked 7871 Read 8 Done New Done true
19865 6873 Acked 7872 Read 7 Done New Done true
19866 6874 Acked 7873 Read 10 Done New Done true
19867 6875 Acked 7874 Write 10 Done New Done true
19868 6876 Acked 7875 Flush 0 Done New Done true
19869 6877 Acked 7876 Read 3 Done New Done true
19870 6878 Acked 7877 Write 10 Done New Done true
19871 6879 Acked 7878 Read 4 Done New Done true
19872 6880 Acked 7879 Write 9 Done New Done true
19873 6881 Acked 7880 Read 8 Done New Done true
19874 6882 Acked 7881 Write 2 Done New Done true
19875 6883 Acked 7882 Read 5 Done New Done true
19876 6884 Acked 7883 Write 5 Done New Done true
19877 6885 Acked 7884 Read 6 Done New Done true
19878 6886 Acked 7885 Write 3 Done New Done true
19879 6887 Acked 7886 Read 4 Done New Done true
19880 6888 Acked 7887 Write 5 Done New Done true
19881 6889 Acked 7888 Write 8 Done New Done true
19882 6890 Acked 7889 Write 2 Done New Done true
19883 6891 Acked 7890 Write 9 Done New Done true
19884 6892 Acked 7891 Write 2 Done New Done true
19885 6893 Acked 7892 Write 3 Done New Done true
19886 6894 Acked 7893 Read 8 Done New Done true
19887 6895 Acked 7894 Read 7 Done New Done true
19888 6896 Acked 7895 Read 8 Done New Done true
19889 6897 Acked 7896 Write 4 Done New Done true
19890 6898 Acked 7897 Read 9 Done New Done true
19891 6899 Acked 7898 Write 7 Done New Done true
19892 6900 Acked 7899 Flush 0 Done New Done true
19893 6901 Acked 7900 Read 4 Done New Done true
19894 6902 Acked 7901 Write 9 Done New Done true
19895 6903 Acked 7902 Write 10 Done New Done true
19896 6904 Acked 7903 Read 10 Done New Done true
19897 6905 Acked 7904 Write 5 Done New Done true
19898 6906 Acked 7905 Read 10 Done New Done true
19899 6907 Acked 7906 Read 2 Done New Done true
19900 6908 Acked 7907 Write 4 Done New Done true
19901 6909 Acked 7908 Flush 0 Done New Done true
19902 6910 Acked 7909 Flush 0 Done New Done true
19903 6911 Acked 7910 Flush 0 Done New Done true
19904 6912 Acked 7911 Read 2 Done New Done true
19905 6913 Acked 7912 Write 9 Done New Done true
19906 6914 Acked 7913 Write 7 Done New Done true
19907 6915 Acked 7914 Read 7 Done New Done true
19908 6916 Acked 7915 Read 10 Done New Done true
19909 6917 Acked 7916 Read 2 Done New Done true
19910 6918 Acked 7917 Read 7 Done New Done true
19911 6919 Acked 7918 Write 5 Done New Done true
19912 6920 Acked 7919 Read 8 Done New Done true
19913 6921 Acked 7920 Write 8 Done New Done true
19914 6922 Acked 7921 Read 5 Done New Done true
19915 6923 Acked 7922 Read 2 Done New Done true
19916 6924 Acked 7923 Write 9 Done New Done true
19917 6925 Acked 7924 Write 3 Done New Done true
19918 6926 Acked 7925 Read 4 Done New Done true
19919 6927 Acked 7926 Read 9 Done New Done true
19920 6928 Acked 7927 Read 1 Done New Done true
19921 6929 Acked 7928 Write 10 Done New Done true
19922 6930 Acked 7929 Write 6 Done New Done true
19923 6931 Acked 7930 Read 3 Done New Done true
19924 6932 Acked 7931 Read 4 Done New Done true
19925 6933 Acked 7932 Flush 0 Done New Done true
19926 STATES DS:0 DS:1 DS:2 TOTAL
19927 New 0 302 0 302
19928 Sent 0 0 0 0
19929 Done 302 0 302 604
19930 Skipped 0 0 0 0
19931 Error 0 0 0 0
19932 Last Flush: 7932 7630 7932
19933 Downstairs last five completed: 7630 7629 7628 7627 7626
19934 Upstairs last five completed: 6933 6932 6931 6930 6929
19935 CLIENT: Up:0 ds:302 act:2
199362024-02-28T22:17:13.063ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199372024-02-28T22:17:13.064ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199382024-02-28T22:17:13.064ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199392024-02-28T22:17:13.064ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199402024-02-28T22:17:13.064ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199412024-02-28T22:17:13.064ZINFOcrucible: [1] client re-new 302 jobs since flush 7630 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199422024-02-28T22:17:13.064ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
19943 [8820] got start action
19944 [8820] I am going to restart
19945 Starting downstairs at port 8820
19946 Make output file at "/tmp/dsc/downstairs-8820.txt"
19947 [8820][1] reports Starting
19948 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1187)
19949 [8820][1] reports Running
199502024-02-28T22:17:14.065ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199512024-02-28T22:17:14.065ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199522024-02-28T22:17:14.065ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199532024-02-28T22:17:14.065ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199542024-02-28T22:17:14.065ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199552024-02-28T22:17:14.065ZINFOcrucible: [1] client re-new 302 jobs since flush 7630 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199562024-02-28T22:17:14.065ZINFOcrucible: client task is exiting = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199572024-02-28T22:17:15.066ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199582024-02-28T22:17:15.066ZINFOcrucible: ds_connection connected = io task client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199592024-02-28T22:17:15.068ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199602024-02-28T22:17:15.068ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199612024-02-28T22:17:15.068ZINFOcrucible: send last flush ID to this DS: 7630 = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199622024-02-28T22:17:15.068ZINFOcrucible: Replied this last flush ID: 7630 = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199632024-02-28T22:17:15.068ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199642024-02-28T22:17:15.068ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199652024-02-28T22:17:15.068ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199662024-02-28T22:17:15.068ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
19967 ----------------------------------------------------------------
19968 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
19969 Downstairs last five completed: 7932 7931 7930 7929 7928
19970 Upstairs last five completed: 6933 6932 6931 6930 6929
19971 CLIENT: Up:0 ds:0 act:3
19972 Replay: All jobs finished, all DS active.
19973 Main task has work to do, go find it
19974 got dsc Stop(0)
19975 stop 0
19976 [8810] Got stop action so:false kr:false
19977 Feb 28 22:17:16.394 INFO request completed, latency_us: 322, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 00725726-848c-442a-a771-3707a618d195, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
199782024-02-28T22:17:16.395ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199792024-02-28T22:17:16.395ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199802024-02-28T22:17:16.395ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199812024-02-28T22:17:16.395ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199822024-02-28T22:17:16.395ZINFOcrucible: [0] client re-new 0 jobs since flush 7932 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
19983 Feb 28 22:17:16.395 INFO request completed, latency_us: 112, {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:17:16.395825151Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
19984 response_code: 200, uri: /state/cid/0, method: GET, req_id: 872eef87-fba7-48b7-82e6-d876cae8c902, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
19985 [8810] Exited with: ExitStatus(unix_wait_status(9))
19986 [8810][0] reports Exit
199872024-02-28T22:17:17.397ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199882024-02-28T22:17:17.397ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199892024-02-28T22:17:17.397ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199902024-02-28T22:17:17.397ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199912024-02-28T22:17:17.397ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199922024-02-28T22:17:17.397ZINFOcrucible: [0] client re-new 0 jobs since flush 7932 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199932024-02-28T22:17:17.397ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199942024-02-28T22:17:18.399ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199952024-02-28T22:17:18.399ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199962024-02-28T22:17:18.399ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199972024-02-28T22:17:18.399ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199982024-02-28T22:17:18.399ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
199992024-02-28T22:17:18.399ZINFOcrucible: [0] client re-new 0 jobs since flush 7932 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
200002024-02-28T22:17:18.399ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
200012024-02-28T22:17:19.400ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
200022024-02-28T22:17:19.400ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
200032024-02-28T22:17:19.400ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
200042024-02-28T22:17:19.400ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
200052024-02-28T22:17:19.400ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
200062024-02-28T22:17:19.400ZINFOcrucible: [0] client re-new 0 jobs since flush 7932 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
200072024-02-28T22:17:19.400ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
20008 Feb 28 22:17:20.398 INFO request completed, latency_us: 279, response_code: 200, uri: /state/cid/0, method: GET, req_id: d8eee1db-54bc-4c8b-8558-abbf3a09869d, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
20009 001/300 Write block 2341 len 8192 data: 8 6
20010 002/300 Write block 1921 len 24576 data: 4 3 3 4 4 4
20011 003/300 Write block 4373 len 16384 data: 4 6 6 6
20012 004/300 Write block 1526 len 12288 data: 5 3 4
20013 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":005/30030 Read block 4958 len 40960
20014 ,"time":"2024-02-28T22:17:20.401156787Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
200152024-02-28T22:17:20.401ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
200162024-02-28T22:17:20.401ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
200172024-02-28T22:17:20.401ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
200182024-02-28T22:17:20.401ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
200192024-02-28T22:17:20.401ZINFOcrucible: [0] client re-new 5 jobs since flush 7932 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
200202024-02-28T22:17:20.401ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
20021 006/300 Write block 3115 len 40960 data: 5 6 5 6 5 6 6 8 6 6
20022 007/300 Read block 3646 len 8192
20023 008/300 Read block 859 len 36864
20024 009/300 Read block 1754 len 4096
20025 010/300 Write block 2678 len 16384 data: 8 8 7 6
20026 011/300 Read block 4344 len 8192
20027 012/300 Read block 1883 len 36864
20028 013/300 Read block 4680 len 20480
20029 014/300 Write block 750 len 20480 data: 7 7 9 8 8
20030 015/300 Read block 2177 len 4096
20031 016/300 Read block 630 len 28672
20032 017/300 Write block 2643 len 36864 data: 2 3 4 5 4 6 6 7 7
20033 018/300 Read block 4817 len 4096
20034 019/300 Read block 1172 len 16384
20035 020/300 Flush
20036 021/300 Write block 1003 len 36864 data: 5 5 5 4 3 4 3 3 3
20037 022/300 Flush
20038 023/300 Write block 801 len 16384 data: 6 3 4 4
20039 024/300 Write block 4337 len 16384 data: 6 5 5 5
20040 025/300 Read block 2126 len 12288
20041 026/300 Write block 2852 len 36864 data: 3 5 5 5 6 6 5 5 5
20042 027/300 Flush
20043 028/300 Write block 876 len 4096 data: 6
20044 029/300 Write block 650 len 16384 data: 3 4 5 5
20045 030/300 Read block 1116 len 32768
20046 031/300 Read block 658 len 8192
20047 032/300 Write block 4267 len 36864 data: 6 5 5 5 5 5 4 3 3
20048 033/300 Write block 365 len 24576 data: 3 3 3 4 4 4
20049 034/300 Write block 2372 len 12288 data: 4 2 2
20050 035/300 Read block 4902 len 8192
20051 036/300 Write block 962 len 32768 data: 10 9 7 7 6 3 2 2
20052 037/300 Write block 2081 len 4096 data: 5
20053 038/300 Flush
20054 039/300 Read block 3678 len 36864
20055 040/300 Write block 2676 len 40960 data: 5 8 9 9 8 7 7 7 8 8
20056 041/300 Flush
20057 042/300 Flush
20058 043/300 Read block 336 len 28672
20059 044/300 Read block 1709 len 36864
20060 045/300 Read block 4313 len 4096
20061 046/300 Flush
20062 047/300 Write block 910 len 32768 data: 6 5 5 5 6 6 6 6
20063 048/300 Read block 3135 len 28672
20064 049/300 Read block 1366 len 40960
20065 050/300 Write block 1123 len 32768 data: 3 6 8 8 9 9 9 9
20066 051/300 Read block 85 len 28672
20067 052/300 Read block 4622 len 24576
20068 053/300 Flush
20069 054/300 Write block 2070 len 40960 data: 7 7 7 8 6 5 3 2 2 3
20070 055/300 Write block 2083 len 32768 data: 5 5 4 3 4 5 4 5
20071 056/300 Flush
20072 057/300 Write block 578 len 40960 data: 5 3 4 3 4 4 4 3 3 3
20073 058/300 Read block 3229 len 12288
20074 059/300 Read block 3837 len 32768
20075 060/300 Read block 4955 len 36864
20076 061/300 Read block 1734 len 20480
20077 062/300 Read block 1787 len 20480
20078 063/300 Read block 4070 len 36864
20079 064/300 Write block 2176 len 32768 data: 3 4 6 7 7 5 5 5
20080 065/300 Read block 4086 len 12288
20081 066/300 Read block 3800 len 4096
20082 067/300 Read block 3212 len 16384
20083 068/300 Read block 1945 len 28672
20084 069/300 Write block 424 len 8192 data: 7 7
20085 070/300 Flush
20086 071/300 Read block 4382 len 36864
20087 072/300 Write block 2817 len 28672 data: 9 8 8 8 7 8 7
20088 073/300 Read block 402 len 28672
20089 074/300 Read block 3170 len 16384
20090 075/300 Read block 3930 len 8192
20091 076/300 Read block 3779 len 28672
20092 077/300 Flush
20093 078/300 Write block 690 len 4096 data: 6
20094 079/300 Flush
20095 080/300 Read block 1038 len 28672
20096 081/300 Write block 2297 len 20480 data: 6 6 6 6 6
20097 082/300 Flush
20098 083/300 Write block 804 len 36864 data: 5 5 5 4 4 4 5 5 5
20099 084/300 Read block 2106 len 12288
20100 085/300 Read block 2734 len 8192
20101 086/300 Write block 433 len 4096 data: 7
20102 087/300 Read block 4192 len 8192
20103 088/300 Read block 1202 len 36864
20104 089/300 Write block 3254 len 12288 data: 7 5 6
20105 090/300 Read block 312 len 24576
20106 091/300 Read block 4042 len 24576
20107 092/300 Read block 2059 len 20480
20108 093/300 Read block 4954 len 40960
20109 094/300 Read block 4835 len 20480
20110 095/300 Write block 1447 len 40960 data: 3 3 2 3 3 3 4 3 3 4
20111 096/300 Flush
20112 097/300 Read block 1149 len 24576
20113 098/300 Flush
20114 099/300 Write block 1349 len 32768 data: 9 9 8 5 5 4 3 3
20115 100/300 Read block 764 len 28672
20116 101/300 Write block 3561 len 40960 data: 4 5 5 5 5 6 5 6 6 6
20117 102/300 Read block 3523 len 20480
20118 103/300 Write block 4437 len 20480 data: 7 5 4 2 2
20119 104/300 Read block 278 len 20480
20120 105/300 Write block 2171 len 24576 data: 3 4 4 3 3 4
20121 106/300 Write block 985 len 24576 data: 4 3 4 4 4 5
20122 107/300 Read block 3927 len 12288
20123 108/300 Flush
20124 109/300 Read block 1934 len 28672
20125 110/300 Read block 2721 len 8192
20126 111/300 Read block 3122 len 16384
20127 112/300 Read block 1205 len 12288
20128 113/300 Read block 3191 len 32768
20129 114/300 Read block 4339 len 32768
20130 115/300 Read block 4402 len 32768
20131 116/300 Write block 2899 len 8192 data: 5 6
20132 117/300 Read block 1306 len 40960
20133 118/300 Write block 1451 len 16384 data: 4 4 5 4
20134 119/300 Flush
20135 120/300 Write block 4296 len 20480 data: 4 4 4 4 5
20136 121/300 Write block 1386 len 40960 data: 7 8 7 8 7 6 7 7 6 5
20137 122/300 Write block 902 len 40960 data: 3 4 4 4 3 4 5 6 7 6
20138 123/300 Read block 4847 len 4096
20139 124/300 Read block 3007 len 8192
20140 125/300 Read block 1753 len 28672
20141 126/300 Write block 3051 len 24576 data: 6 5 5 5 5 5
20142 127/300 Write block 2143 len 16384 data: 4 5 5 6
20143 128/300 Read block 2405 len 28672
20144 129/300 Read block 1486 len 24576
20145 130/300 Write block 4801 len 40960 data: 8 8 8 8 7 7 6 5 4 3
20146 131/300 Read block 1454 len 36864
20147 132/300 Write block 4812 len 20480 data: 5 4 4 5 4
20148 133/300 Flush
20149 134/300 Read block 881 len 8192
20150 135/300 Read block 4233 len 24576
20151 136/300 Read block 1665 len 36864
20152 137/300 Read block 1717 len 40960
20153 138/300 Read block 3619 len 8192
20154 139/300 Write block 3525 len 36864 data: 4 4 5 5 5 5 5 6 6
20155 140/300 Flush
20156 141/300 Write block 1777 len 20480 data: 5 5 5 6 5
20157 142/300 Read block 4439 len 20480
20158 143/300 Read block 3352 len 20480
20159 144/300 Write block 289 len 16384 data: 2 2 3 3
20160 145/300 Write block 4058 len 16384 data: 5 6 5 5
20161 146/300 Write block 583 len 32768 data: 5 5 4 4 4 4 5 5
20162 147/300 Write block 4000 len 20480 data: 6 5 6 6 5
20163 148/300 Write block 2066 len 40960 data: 9 8 7 7 8 8 8 9 7 6
20164 149/300 Flush
20165 150/300 Read block 871 len 4096
20166 151/300 Read block 4649 len 12288
20167 152/300 Read block 4473 len 12288
20168 153/300 Write block 2456 len 12288 data: 6 6 7
20169 154/300 Write block 877 len 16384 data: 7 6 5 5
20170 155/300 Write block 2729 len 32768 data: 5 5 5 7 5 5 5 3
20171 156/300 Write block 4624 len 4096 data: 4
20172 157/300 Read block 3375 len 28672
20173 158/300 Write block 2044 len 4096 data: 3
20174 159/300 Write block 4696 len 20480 data: 5 5 4 4 5
20175 160/300 Read block 3495 len 20480
20176 161/300 Flush
20177 162/300 Read block 305 len 24576
20178 163/300 Read block 4937 len 36864
20179 164/300 Read block 3824 len 4096
20180 165/300 Read block 165 len 24576
20181 166/300 Read block 701 len 28672
20182 167/300 Read block 3328 len 28672
20183 168/300 Read block 3063 len 40960
20184 169/300 Write block 2042 len 16384 data: 3 3 4 4
20185 170/300 Write block 2369 len 32768 data: 5 5 4 5 3 3 3 2
20186 171/300 Read block 4973 len 24576
20187 172/300 Read block 3373 len 28672
20188 173/300 Read block 2388 len 20480
20189 174/300 Write block 1134 len 8192 data: 8 7
20190 175/300 Write block 3749 len 20480 data: 5 5 5 6 6
20191 176/300 Write block 3883 len 4096 data: 5
20192 177/300 Write block 4187 len 32768 data: 6 7 6 5 6 5 5 4
20193 178/300 Read block 4881 len 20480
20194 179/300 Read block 4297 len 8192
20195 180/300 Read block 208 len 28672
20196 181/300 Read block 265 len 36864
20197 182/300 Read block 1500 len 8192
20198 183/300 Read block 2840 len 20480
20199 184/300 Write block 293 len 12288 data: 5 7 7
20200 185/300 Read block 3880 len 12288
20201 186/300 Read block 1131 len 28672
20202 187/300 Flush
20203 188/300 Write block 455 len 36864 data: 6 6 7 7 7 8 9 7 4
20204 189/300 Read block 3950 len 16384
20205 190/300 Read block 1734 len 32768
20206 191/300 Read block 3820 len 24576
20207 192/300 Read block 4127 len 4096
20208 193/300 Flush
20209 194/300 Read block 339 len 36864
20210 195/300 Read block 863 len 20480
20211 196/300 Write block 4036 len 40960 data: 5 6 5 4 4 4 4 3 2 2
20212 197/300 Read block 4474 len 36864
20213 198/300 Write block 4924 len 16384 data: 5 6 6 6
20214 199/300 Write block 1542 len 28672 data: 5 6 4 4 5 5 5
20215 200/300 Read block 2310 len 28672
20216 201/300 Read block 378 len 12288
20217 202/300 Write block 3703 len 8192 data: 5 7
20218 203/300 Read block 1673 len 4096
20219 204/300 Read block 408 len 20480
20220 205/300 Read block 1261 len 4096
20221 206/300 Write block 2380 len 28672 data: 3 4 5 4 4 4 4
20222 207/300 Read block 4241 len 40960
20223 208/300 Write block 4302 len 4096 data: 5
20224 209/300 Write block 1175 len 4096 data: 4
20225 210/300 Flush
20226 211/300 Write block 3430 len 12288 data: 5 5 7
20227 212/300 Read block 557 len 8192
20228 213/300 Read block 2810 len 36864
20229 214/300 Read block 4246 len 20480
20230 215/300 Read block 784 len 20480
20231 216/300 Read block 980 len 28672
20232 217/300 Read block 2833 len 16384
20233 218/300 Write block 505 len 12288 data: 6 7 7
20234 219/300 Flush
20235 220/300 Read block 2700 len 16384
20236 221/300 Read block 169 len 40960
20237 222/300 Read block 3872 len 40960
20238 223/300 Flush
20239 224/300 Write block 3421 len 32768 data: 2 3 3 4 4 3 3 4
20240 225/300 Read block 527 len 32768
20241 226/300 Write block 2415 len 20480 data: 9 7 8 7 7
20242 227/300 Write block 1165 len 12288 data: 6 6 6
20243 228/300 Read block 4871 len 40960
20244 229/300 Flush
20245 230/300 Write block 1267 len 20480 data: 5 6 5 4 4
20246 231/300 Read block 2194 len 4096
20247 232/300 Write block 1215 len 24576 data: 6 6 6 6 5 6
20248 233/300 Flush
20249 234/300 Write block 3356 len 16384 data: 3 5 7 7
20250 235/300 Read block 797 len 20480
20251 236/300 Read block 3039 len 20480
20252 237/300 Read block 412 len 28672
20253 238/300 Write block 559 len 24576 data: 3 3 5 5 5 4
20254 239/300 Write block 2477 len 40960 data: 3 3 3 3 3 3 2 4 5 5
20255 240/300 Read block 1808 len 8192
20256 241/300 Read block 3551 len 24576
20257 242/300 Write block 661 len 24576 data: 4 4 6 6 4 4
20258 243/300 Write block 3922 len 40960 data: 5 5 6 6 5 5 4 3 2 2
20259 244/300 Read block 3817 len 12288
20260 245/300 Read block 3244 len 12288
20261 246/300 Read block 354 len 24576
20262 247/300 Read block 3641 len 24576
20263 248/300 Read block 3196 len 12288
20264 249/300 Write block 4620 len 36864 data: 8 7 5 5 5 4 7 6 5
20265 250/300 Write block 676 len 28672 data: 7 7 9 6 6 9 7
20266 251/300 Flush
20267 252/300 Read block 3050 len 8192
20268 253/300 Write block 4310 len 36864 data: 5 6 5 5 5 5 4 4 6
20269 254/300 Write block 1198 len 36864 data: 4 4 4 5 6 4 6 6 4
20270 255/300 Read block 185 len 32768
20271 256/300 Read block 4172 len 36864
20272 257/300 Read block 4720 len 24576
20273 258/300 Read block 2343 len 36864
20274 259/300 Write block 1194 len 12288 data: 5 6 6
20275 260/300 Read block 2813 len 24576
20276 261/300 Write block 581 len 40960 data: 4 5 6 6 5 5 5 5 6 6
20277 262/300 Write block 2993 len 32768 data: 4 3 3 2 2 2 3 3
20278 263/300 Read block 2121 len 28672
20279 264/300 Flush
20280 265/300 Write block 525 len 4096 data: 3
20281 266/300 Read block 831 len 36864
20282 267/300 Read block 2297 len 16384
20283 268/300 Read block 444 len 12288
20284 269/300 Read block 3767 len 4096
20285 270/300 Read block 2847 len 4096
20286 271/300 Read block 484 len 32768
20287 272/300 Read block 3112 len 32768
20288 273/300 Write block 4507 len 16384 data: 6 6 5 6
20289 274/300 Read block 4786 len 16384
20290 275/300 Write block 1081 len 16384 data: 4 5 6 5
20291 276/300 Write block 3576 len 12288 data: 7 6 7
20292 277/300 Read block 339 len 8192
20293 278/300 Read block 1296 len 20480
20294 279/300 Write block 2883 len 16384 data: 7 8 7 9
20295 280/300 Write block 1286 len 32768 data: 2 3 3 3 3 3 3 2
20296 281/300 Read block 3030 len 4096
20297 282/300 Read block 2975 len 40960
20298 283/300 Write block 41 len 16384 data: 5 3 3 2
20299 284/300 Write block 1578 len 28672 data: 5 4 7 8 8 8 8
20300 285/300 Read block 1945 len 40960
20301 286/300 Read block 1128 len 4096
20302 287/300 Write block 3720 len 28672 data: 5 6 6 4 5 6 7
20303 288/300 Read block 2010 len 4096
20304 289/300 Read block 2018 len 12288
20305 290/300 Flush
20306 291/300 Read block 3562 len 32768
20307 292/300 Write block 311 len 32768 data: 4 5 5 6 5 5 8 8
20308 293/300 Read block 1570 len 24576
20309 294/300 Read block 4820 len 40960
20310 295/300 Flush
20311 296/300 Write block 291 len 32768 data: 4 4 6 8 8 7 6 6
20312 297/300 Read block 4396 len 24576
20313 298/300 Write block 4011 len 24576 data: 9 11 11 12 10 9
20314 299/300 Write block 1163 len 36864 data: 4 6 7 7 7 6 7 7 6
20315 300/300 Write block 2472 len 16384 data: 7 6 7 3
20316 Main task has work to do, go find it
20317 got dsc Start(0)
20318 start 0
20319 Feb 28 22:17:20.742 INFO request completed, latency_us: 202, response_code: 204, uri: /start/cid/0, method: POST, req_id: 50ee53e1-a356-4314-b01f-62b48d146bb4, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
20320 Replay: started 0, returned:Ok(())
20321 ----------------------------------------------------------------
20322 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
20323 States: Offline Active Active
20324 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
20325 6934 Acked 7933 Write 2 New Done Done true
20326 6935 Acked 7934 Write 6 New Done Done true
20327 6936 Acked 7935 Write 4 New Done Done true
20328 6937 Acked 7936 Write 3 New Done Done true
20329 6938 Acked 7937 Read 10 New Done Done true
20330 6939 Acked 7938 Write 10 New Done Done false
20331 6940 Acked 7939 Read 2 New Done Done false
20332 6941 Acked 7940 Read 9 New Done Done false
20333 6942 Acked 7941 Read 1 New Done Done false
20334 6943 Acked 7942 Write 4 New Done Done false
20335 6944 Acked 7943 Read 2 New Done Done false
20336 6945 Acked 7944 Read 9 New Done Done false
20337 6946 Acked 7945 Read 5 New Done Done false
20338 6947 Acked 7946 Write 5 New Done Done false
20339 6948 Acked 7947 Read 1 New Done Done false
20340 6949 Acked 7948 Read 7 New Done Done false
20341 6950 Acked 7949 Write 9 New Done Done false
20342 6951 Acked 7950 Read 1 New Done Done false
20343 6952 Acked 7951 Read 4 New Done Done false
20344 6953 Acked 7952 Flush 0 New Done Done false
20345 6954 Acked 7953 Write 9 New Done Done false
20346 6955 Acked 7954 Flush 0 New Done Done false
20347 6956 Acked 7955 Write 4 New Done Done false
20348 6957 Acked 7956 Write 4 New Done Done false
20349 6958 Acked 7957 Read 3 New Done Done false
20350 6959 Acked 7958 Write 9 New Done Done false
20351 6960 Acked 7959 Flush 0 New Done Done false
20352 6961 Acked 7960 Write 1 New Done Done false
20353 6962 Acked 7961 Write 4 New Done Done false
20354 6963 Acked 7962 Read 8 New Done Done false
20355 6964 Acked 7963 Read 2 New Done Done false
20356 6965 Acked 7964 Write 9 New Done Done false
20357 6966 Acked 7965 Write 6 New Done Done false
20358 6967 Acked 7966 Write 3 New Done Done false
20359 6968 Acked 7967 Read 2 New Done Done false
20360 6969 Acked 7968 Write 8 New Done Done false
20361 6970 Acked 7969 Write 1 New Done Done false
20362 6971 Acked 7970 Flush 0 New Done Done false
20363 6972 Acked 7971 Read 9 New Done Done false
20364 6973 Acked 7972 Write 10 New Done Done false
20365 6974 Acked 7973 Flush 0 New Done Done false
20366 6975 Acked 7974 Flush 0 New Done Done false
20367 6976 Acked 7975 Read 7 New Done Done false
20368 6977 Acked 7976 Read 9 New Done Done false
20369 6978 Acked 7977 Read 1 New Done Done false
20370 6979 Acked 7978 Flush 0 New Done Done false
20371 6980 Acked 7979 Write 8 New Done Done false
20372 6981 Acked 7980 Read 7 New Done Done false
20373 6982 Acked 7981 Read 10 New Done Done false
20374 6983 Acked 7982 Write 8 New Done Done false
20375 6984 Acked 7983 Read 7 New Done Done false
20376 6985 Acked 7984 Read 6 New Done Done false
20377 6986 Acked 7985 Flush 0 New Done Done false
20378 6987 Acked 7986 Write 10 New Done Done false
20379 6988 Acked 7987 Write 8 New Done Done false
20380 6989 Acked 7988 Flush 0 New Done Done false
20381 6990 Acked 7989 Write 10 New Done Done false
20382 6991 Acked 7990 Read 3 New Done Done false
20383 6992 Acked 7991 Read 8 New Done Done false
20384 6993 Acked 7992 Read 9 New Done Done false
20385 6994 Acked 7993 Read 5 New Done Done false
20386 6995 Acked 7994 Read 5 New Done Done false
20387 6996 Acked 7995 Read 9 New Done Done false
20388 6997 Acked 7996 Write 8 New Done Done false
20389 6998 Acked 7997 Read 3 New Done Done false
20390 6999 Acked 7998 Read 1 New Done Done false
20391 7000 Acked 7999 Read 4 New Done Done false
20392 7001 Acked 8000 Read 7 New Done Done false
20393 7002 Acked 8001 Write 2 New Done Done false
20394 7003 Acked 8002 Flush 0 New Done Done false
20395 7004 Acked 8003 Read 9 New Done Done false
20396 7005 Acked 8004 Write 7 New Done Done false
20397 7006 Acked 8005 Read 7 New Done Done false
20398 7007 Acked 8006 Read 4 New Done Done false
20399 7008 Acked 8007 Read 2 New Done Done false
20400 7009 Acked 8008 Read 7 New Done Done false
20401 7010 Acked 8009 Flush 0 New Done Done false
20402 7011 Acked 8010 Write 1 New Done Done false
20403 7012 Acked 8011 Flush 0 New Done Done false
20404 7013 Acked 8012 Read 7 New Done Done false
20405 7014 Acked 8013 Write 5 New Done Done false
20406 7015 Acked 8014 Flush 0 New Done Done false
20407 7016 Acked 8015 Write 9 New Done Done false
20408 7017 Acked 8016 Read 3 New Done Done false
20409 7018 Acked 8017 Read 2 New Done Done false
20410 7019 Acked 8018 Write 1 New Done Done false
20411 7020 Acked 8019 Read 2 New Done Done false
20412 7021 Acked 8020 Read 9 New Done Done false
20413 7022 Acked 8021 Write 3 New Done Done false
20414 7023 Acked 8022 Read 6 New Done Done false
20415 7024 Acked 8023 Read 6 New Done Done false
20416 7025 Acked 8024 Read 5 New Done Done false
20417 7026 Acked 8025 Read 10 New Done Done false
20418 7027 Acked 8026 Read 5 New Done Done false
20419 7028 Acked 8027 Write 10 New Done Done false
20420 7029 Acked 8028 Flush 0 New Done Done false
20421 7030 Acked 8029 Read 6 New Done Done false
20422 7031 Acked 8030 Flush 0 New Done Done false
20423 7032 Acked 8031 Write 8 New Done Done false
20424 7033 Acked 8032 Read 7 New Done Done false
20425 7034 Acked 8033 Write 10 New Done Done false
20426 7035 Acked 8034 Read 5 New Done Done false
20427 7036 Acked 8035 Write 5 New Done Done false
20428 7037 Acked 8036 Read 5 New Done Done false
20429 7038 Acked 8037 Write 6 New Done Done false
20430 7039 Acked 8038 Write 6 New Done Done false
20431 7040 Acked 8039 Read 3 New Done Done false
20432 7041 Acked 8040 Flush 0 New Done Done false
20433 7042 Acked 8041 Read 7 New Done Done false
20434 7043 Acked 8042 Read 2 New Done Done false
20435 7044 Acked 8043 Read 4 New Done Done false
20436 7045 Acked 8044 Read 3 New Done Done false
20437 7046 Acked 8045 Read 8 New Done Done false
20438 7047 Acked 8046 Read 8 New Done Done false
20439 7048 Acked 8047 Read 8 New Done Done false
20440 7049 Acked 8048 Write 2 New Done Done false
20441 7050 Acked 8049 Read 10 New Done Done false
20442 7051 Acked 8050 Write 4 New Done Done false
20443 7052 Acked 8051 Flush 0 New Done Done false
20444 7053 Acked 8052 Write 5 New Done Done false
20445 7054 Acked 8053 Write 10 New Done Done false
20446 7055 Acked 8054 Write 10 New Done Done false
20447 7056 Acked 8055 Read 1 New Done Done false
20448 7057 Acked 8056 Read 2 New Done Done false
20449 7058 Acked 8057 Read 7 New Done Done false
20450 7059 Acked 8058 Write 6 New Done Done false
20451 7060 Acked 8059 Write 4 New Done Done false
20452 7061 Acked 8060 Read 7 New Done Done false
20453 7062 Acked 8061 Read 6 New Done Done false
20454 7063 Acked 8062 Write 10 New Done Done false
20455 7064 Acked 8063 Read 9 New Done Done false
20456 7065 Acked 8064 Write 5 New Done Done false
20457 7066 Acked 8065 Flush 0 New Done Done false
20458 7067 Acked 8066 Read 2 New Done Done false
20459 7068 Acked 8067 Read 6 New Done Done false
20460 7069 Acked 8068 Read 9 New Done Done false
20461 7070 Acked 8069 Read 10 New Done Done false
20462 7071 Acked 8070 Read 2 New Done Done false
20463 7072 Acked 8071 Write 9 New Done Done false
20464 7073 Acked 8072 Flush 0 New Done Done false
20465 7074 Acked 8073 Write 5 New Done Done false
20466 7075 Acked 8074 Read 5 New Done Done false
20467 7076 Acked 8075 Read 5 New Done Done false
20468 7077 Acked 8076 Write 4 New Done Done false
20469 7078 Acked 8077 Write 4 New Done Done false
20470 7079 Acked 8078 Write 8 New Done Done false
20471 7080 Acked 8079 Write 5 New Done Done false
20472 7081 Acked 8080 Write 10 New Done Done false
20473 7082 Acked 8081 Flush 0 New Done Done false
20474 7083 Acked 8082 Read 1 New Done Done false
20475 7084 Acked 8083 Read 3 New Done Done false
20476 7085 Acked 8084 Read 3 New Done Done false
20477 7086 Acked 8085 Write 3 New Done Done false
20478 7087 Acked 8086 Write 4 New Done Done false
20479 7088 Acked 8087 Write 8 New Done Done false
20480 7089 Acked 8088 Write 1 New Done Done false
20481 7090 Acked 8089 Read 7 New Done Done false
20482 7091 Acked 8090 Write 1 New Done Done false
20483 7092 Acked 8091 Write 5 New Done Done false
20484 7093 Acked 8092 Read 5 New Done Done false
20485 7094 Acked 8093 Flush 0 New Done Done false
20486 7095 Acked 8094 Read 6 New Done Done false
20487 7096 Acked 8095 Read 9 New Done Done false
20488 7097 Acked 8096 Read 1 New Done Done false
20489 7098 Acked 8097 Read 6 New Done Done false
20490 7099 Acked 8098 Read 7 New Done Done false
20491 7100 Acked 8099 Read 7 New Done Done false
20492 7101 Acked 8100 Read 10 New Done Done false
20493 7102 Acked 8101 Write 4 New Done Done false
20494 7103 Acked 8102 Write 8 New Done Done false
20495 7104 Acked 8103 Read 6 New Done Done false
20496 7105 Acked 8104 Read 7 New Done Done false
20497 7106 Acked 8105 Read 5 New Done Done false
20498 7107 Acked 8106 Write 2 New Done Done false
20499 7108 Acked 8107 Write 5 New Done Done false
20500 7109 Acked 8108 Write 1 New Done Done false
20501 7110 Acked 8109 Write 8 New Done Done false
20502 7111 Acked 8110 Read 5 New Done Done false
20503 7112 Acked 8111 Read 2 New Done Done false
20504 7113 Acked 8112 Read 7 New Done Done false
20505 7114 Acked 8113 Read 9 New Done Done false
20506 7115 Acked 8114 Read 2 New Done Done false
20507 7116 Acked 8115 Read 5 New Done Done false
20508 7117 Acked 8116 Write 3 New Done Done false
20509 7118 Acked 8117 Read 3 New Done Done false
20510 7119 Acked 8118 Read 7 New Done Done false
20511 7120 Acked 8119 Flush 0 New Done Done false
20512 7121 Acked 8120 Write 9 New Done Done false
20513 7122 Acked 8121 Read 4 New Done Done false
20514 7123 Acked 8122 Read 8 New Done Done false
20515 7124 Acked 8123 Read 6 New Done Done false
20516 7125 Acked 8124 Read 1 New Done Done false
20517 7126 Acked 8125 Flush 0 New Done Done false
20518 7127 Acked 8126 Read 9 New Done Done false
20519 7128 Acked 8127 Read 5 New Done Done false
20520 7129 Acked 8128 Write 10 New Done Done false
20521 7130 Acked 8129 Read 9 New Done Done false
20522 7131 Acked 8130 Write 4 New Done Done false
20523 7132 Acked 8131 Write 7 New Done Done false
20524 7133 Acked 8132 Read 7 New Done Done false
20525 7134 Acked 8133 Read 3 New Done Done false
20526 7135 Acked 8134 Write 2 New Done Done false
20527 7136 Acked 8135 Read 1 New Done Done false
20528 7137 Acked 8136 Read 5 New Done Done false
20529 7138 Acked 8137 Read 1 New Done Done false
20530 7139 Acked 8138 Write 7 New Done Done false
20531 7140 Acked 8139 Read 10 New Done Done false
20532 7141 Acked 8140 Write 1 New Done Done false
20533 7142 Acked 8141 Write 1 New Done Done false
20534 7143 Acked 8142 Flush 0 New Done Done false
20535 7144 Acked 8143 Write 3 New Done Done false
20536 7145 Acked 8144 Read 2 New Done Done false
20537 7146 Acked 8145 Read 9 New Done Done false
20538 7147 Acked 8146 Read 5 New Done Done false
20539 7148 Acked 8147 Read 5 New Done Done false
20540 7149 Acked 8148 Read 7 New Done Done false
20541 7150 Acked 8149 Read 4 New Done Done false
20542 7151 Acked 8150 Write 3 New Done Done false
20543 7152 Acked 8151 Flush 0 New Done Done false
20544 7153 Acked 8152 Read 4 New Done Done false
20545 7154 Acked 8153 Read 10 New Done Done false
20546 7155 Acked 8154 Read 10 New Done Done false
20547 7156 Acked 8155 Flush 0 New Done Done false
20548 7157 Acked 8156 Write 8 New Done Done false
20549 7158 Acked 8157 Read 8 New Done Done false
20550 7159 Acked 8158 Write 5 New Done Done false
20551 7160 Acked 8159 Write 3 New Done Done false
20552 7161 Acked 8160 Read 10 New Done Done false
20553 7162 Acked 8161 Flush 0 New Done Done false
20554 7163 Acked 8162 Write 5 New Done Done false
20555 7164 Acked 8163 Read 1 New Done Done false
20556 7165 Acked 8164 Write 6 New Done Done false
20557 7166 Acked 8165 Flush 0 New Done Done false
20558 7167 Acked 8166 Write 4 New Done Done false
20559 7168 Acked 8167 Read 5 New Done Done false
20560 7169 Acked 8168 Read 5 New Done Done false
20561 7170 Acked 8169 Read 7 New Done Done false
20562 7171 Acked 8170 Write 6 New Done Done false
20563 7172 Acked 8171 Write 10 New Done Done false
20564 7173 Acked 8172 Read 2 New Done Done false
20565 7174 Acked 8173 Read 6 New Done Done false
20566 7175 Acked 8174 Write 6 New Done Done false
20567 7176 Acked 8175 Write 10 New Done Done false
20568 7177 Acked 8176 Read 3 New Done Done false
20569 7178 Acked 8177 Read 3 New Done Done false
20570 7179 Acked 8178 Read 6 New Done Done false
20571 7180 Acked 8179 Read 6 New Done Done false
20572 7181 Acked 8180 Read 3 New Done Done false
20573 7182 Acked 8181 Write 9 New Done Done false
20574 7183 Acked 8182 Write 7 New Done Done false
20575 7184 Acked 8183 Flush 0 New Done Done false
20576 7185 Acked 8184 Read 2 New Done Done false
20577 7186 Acked 8185 Write 9 New Done Done false
20578 7187 Acked 8186 Write 9 New Done Done false
20579 7188 Acked 8187 Read 8 New Done Done false
20580 7189 Acked 8188 Read 9 New Done Done false
20581 7190 Acked 8189 Read 6 New Done Done false
20582 7191 Acked 8190 Read 9 New Done Done false
20583 7192 Acked 8191 Write 3 New Done Done false
20584 7193 Acked 8192 Read 6 New Done Done false
20585 7194 Acked 8193 Write 10 New Done Done false
20586 7195 Acked 8194 Write 8 New Done Done false
20587 7196 Acked 8195 Read 7 New Done Done false
20588 7197 Acked 8196 Flush 0 New Done Done false
20589 7198 Acked 8197 Write 1 New Done Done false
20590 7199 Acked 8198 Read 9 New Done Done false
20591 7200 Acked 8199 Read 4 New Done Done false
20592 7201 Acked 8200 Read 3 New Done Done false
20593 7202 Acked 8201 Read 1 New Done Done false
20594 7203 Acked 8202 Read 1 New Done Done false
20595 7204 Acked 8203 Read 8 New Done Done false
20596 7205 Acked 8204 Read 8 New Done Done false
20597 7206 Acked 8205 Write 4 New Done Done false
20598 7207 Acked 8206 Read 4 New Done Done false
20599 7208 Acked 8207 Write 4 New Done Done false
20600 7209 Acked 8208 Write 3 New Done Done false
20601 7210 Acked 8209 Read 2 New Done Done false
20602 7211 Acked 8210 Read 5 New Done Done false
20603 7212 Acked 8211 Write 4 New Done Done false
20604 7213 Acked 8212 Write 8 New Done Done false
20605 7214 Acked 8213 Read 1 New Done Done false
20606 7215 Acked 8214 Read 10 New Done Done false
20607 7216 Acked 8215 Write 4 New Done Done false
20608 7217 Acked 8216 Write 7 New Done Done false
20609 7218 Acked 8217 Read 10 New Done Done false
20610 7219 Acked 8218 Read 1 New Done Done false
20611 7220 Acked 8219 Write 7 New Done Done false
20612 7221 Acked 8220 Read 1 New Done Done false
20613 7222 Acked 8221 Read 3 New Done Done false
20614 7223 Acked 8222 Flush 0 New Done Done false
20615 7224 Acked 8223 Read 8 New Done Done false
20616 7225 Acked 8224 Write 8 New Done Done false
20617 7226 Acked 8225 Read 6 New Done Done false
20618 7227 Acked 8226 Read 10 New Done Done false
20619 7228 Acked 8227 Flush 0 New Done Done false
20620 7229 Acked 8228 Write 8 New Done Done false
20621 7230 Acked 8229 Read 6 New Done Done false
20622 7231 Acked 8230 Write 6 New Done Done false
20623 7232 Acked 8231 Write 9 New Done Done false
20624 7233 Acked 8232 Write 4 New Done Done false
20625 STATES DS:0 DS:1 DS:2 TOTAL
20626 New 300 0 0 300
20627 Sent 0 0 0 0
20628 Done 0 300 300 600
20629 Skipped 0 0 0 0
20630 Error 0 0 0 0
20631 Last Flush: 7932 8227 8227
20632 Downstairs last five completed: 7932 7931 7930 7929 7928
20633 Upstairs last five completed: 7233 7232 7231 7230 7229
20634 CLIENT: Up:0 ds:300 act:2
206352024-02-28T22:17:21.402ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206362024-02-28T22:17:21.402ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206372024-02-28T22:17:21.402ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206382024-02-28T22:17:21.402ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206392024-02-28T22:17:21.402ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206402024-02-28T22:17:21.402ZINFOcrucible: [0] client re-new 301 jobs since flush 7932 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206412024-02-28T22:17:21.402ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206422024-02-28T22:17:22.403ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206432024-02-28T22:17:22.403ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206442024-02-28T22:17:22.403ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206452024-02-28T22:17:22.403ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206462024-02-28T22:17:22.403ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206472024-02-28T22:17:22.403ZINFOcrucible: [0] client re-new 301 jobs since flush 7932 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206482024-02-28T22:17:22.403ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206492024-02-28T22:17:23.404ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206502024-02-28T22:17:23.404ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206512024-02-28T22:17:23.404ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206522024-02-28T22:17:23.404ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206532024-02-28T22:17:23.404ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206542024-02-28T22:17:23.404ZINFOcrucible: [0] client re-new 301 jobs since flush 7932 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206552024-02-28T22:17:23.404ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206562024-02-28T22:17:24.405ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206572024-02-28T22:17:24.405ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206582024-02-28T22:17:24.405ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206592024-02-28T22:17:24.405ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206602024-02-28T22:17:24.406ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206612024-02-28T22:17:24.406ZINFOcrucible: [0] client re-new 301 jobs since flush 7932 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
206622024-02-28T22:17:24.406ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
20663 ----------------------------------------------------------------
20664 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20665 States: Offline Active Active
20666 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
20667 6934 Acked 7933 Write 2 New Done Done true
20668 6935 Acked 7934 Write 6 New Done Done true
20669 6936 Acked 7935 Write 4 New Done Done true
20670 6937 Acked 7936 Write 3 New Done Done true
20671 6938 Acked 7937 Read 10 New Done Done true
20672 6939 Acked 7938 Write 10 New Done Done true
20673 6940 Acked 7939 Read 2 New Done Done true
20674 6941 Acked 7940 Read 9 New Done Done true
20675 6942 Acked 7941 Read 1 New Done Done true
20676 6943 Acked 7942 Write 4 New Done Done true
20677 6944 Acked 7943 Read 2 New Done Done true
20678 6945 Acked 7944 Read 9 New Done Done true
20679 6946 Acked 7945 Read 5 New Done Done true
20680 6947 Acked 7946 Write 5 New Done Done true
20681 6948 Acked 7947 Read 1 New Done Done true
20682 6949 Acked 7948 Read 7 New Done Done true
20683 6950 Acked 7949 Write 9 New Done Done true
20684 6951 Acked 7950 Read 1 New Done Done true
20685 6952 Acked 7951 Read 4 New Done Done true
20686 6953 Acked 7952 Flush 0 New Done Done true
20687 6954 Acked 7953 Write 9 New Done Done true
20688 6955 Acked 7954 Flush 0 New Done Done true
20689 6956 Acked 7955 Write 4 New Done Done true
20690 6957 Acked 7956 Write 4 New Done Done true
20691 6958 Acked 7957 Read 3 New Done Done true
20692 6959 Acked 7958 Write 9 New Done Done true
20693 6960 Acked 7959 Flush 0 New Done Done true
20694 6961 Acked 7960 Write 1 New Done Done true
20695 6962 Acked 7961 Write 4 New Done Done true
20696 6963 Acked 7962 Read 8 New Done Done true
20697 6964 Acked 7963 Read 2 New Done Done true
20698 6965 Acked 7964 Write 9 New Done Done true
20699 6966 Acked 7965 Write 6 New Done Done true
20700 6967 Acked 7966 Write 3 New Done Done true
20701 6968 Acked 7967 Read 2 New Done Done true
20702 6969 Acked 7968 Write 8 New Done Done true
20703 6970 Acked 7969 Write 1 New Done Done true
20704 6971 Acked 7970 Flush 0 New Done Done true
20705 6972 Acked 7971 Read 9 New Done Done true
20706 6973 Acked 7972 Write 10 New Done Done true
20707 6974 Acked 7973 Flush 0 New Done Done true
20708 6975 Acked 7974 Flush 0 New Done Done true
20709 6976 Acked 7975 Read 7 New Done Done true
20710 6977 Acked 7976 Read 9 New Done Done true
20711 6978 Acked 7977 Read 1 New Done Done true
20712 6979 Acked 7978 Flush 0 New Done Done true
20713 6980 Acked 7979 Write 8 New Done Done true
20714 6981 Acked 7980 Read 7 New Done Done true
20715 6982 Acked 7981 Read 10 New Done Done true
20716 6983 Acked 7982 Write 8 New Done Done true
20717 6984 Acked 7983 Read 7 New Done Done true
20718 6985 Acked 7984 Read 6 New Done Done true
20719 6986 Acked 7985 Flush 0 New Done Done true
20720 6987 Acked 7986 Write 10 New Done Done true
20721 6988 Acked 7987 Write 8 New Done Done true
20722 6989 Acked 7988 Flush 0 New Done Done true
20723 6990 Acked 7989 Write 10 New Done Done true
20724 6991 Acked 7990 Read 3 New Done Done true
20725 6992 Acked 7991 Read 8 New Done Done true
20726 6993 Acked 7992 Read 9 New Done Done true
20727 6994 Acked 7993 Read 5 New Done Done true
20728 6995 Acked 7994 Read 5 New Done Done true
20729 6996 Acked 7995 Read 9 New Done Done true
20730 6997 Acked 7996 Write 8 New Done Done true
20731 6998 Acked 7997 Read 3 New Done Done true
20732 6999 Acked 7998 Read 1 New Done Done true
20733 7000 Acked 7999 Read 4 New Done Done true
20734 7001 Acked 8000 Read 7 New Done Done true
20735 7002 Acked 8001 Write 2 New Done Done true
20736 7003 Acked 8002 Flush 0 New Done Done true
20737 7004 Acked 8003 Read 9 New Done Done true
20738 7005 Acked 8004 Write 7 New Done Done true
20739 7006 Acked 8005 Read 7 New Done Done true
20740 7007 Acked 8006 Read 4 New Done Done true
20741 7008 Acked 8007 Read 2 New Done Done true
20742 7009 Acked 8008 Read 7 New Done Done true
20743 7010 Acked 8009 Flush 0 New Done Done true
20744 7011 Acked 8010 Write 1 New Done Done true
20745 7012 Acked 8011 Flush 0 New Done Done true
20746 7013 Acked 8012 Read 7 New Done Done true
20747 7014 Acked 8013 Write 5 New Done Done true
20748 7015 Acked 8014 Flush 0 New Done Done true
20749 7016 Acked 8015 Write 9 New Done Done true
20750 7017 Acked 8016 Read 3 New Done Done true
20751 7018 Acked 8017 Read 2 New Done Done true
20752 7019 Acked 8018 Write 1 New Done Done true
20753 7020 Acked 8019 Read 2 New Done Done true
20754 7021 Acked 8020 Read 9 New Done Done true
20755 7022 Acked 8021 Write 3 New Done Done true
20756 7023 Acked 8022 Read 6 New Done Done true
20757 7024 Acked 8023 Read 6 New Done Done true
20758 7025 Acked 8024 Read 5 New Done Done true
20759 7026 Acked 8025 Read 10 New Done Done true
20760 7027 Acked 8026 Read 5 New Done Done true
20761 7028 Acked 8027 Write 10 New Done Done true
20762 7029 Acked 8028 Flush 0 New Done Done true
20763 7030 Acked 8029 Read 6 New Done Done true
20764 7031 Acked 8030 Flush 0 New Done Done true
20765 7032 Acked 8031 Write 8 New Done Done true
20766 7033 Acked 8032 Read 7 New Done Done true
20767 7034 Acked 8033 Write 10 New Done Done true
20768 7035 Acked 8034 Read 5 New Done Done true
20769 7036 Acked 8035 Write 5 New Done Done true
20770 7037 Acked 8036 Read 5 New Done Done true
20771 7038 Acked 8037 Write 6 New Done Done true
20772 7039 Acked 8038 Write 6 New Done Done true
20773 7040 Acked 8039 Read 3 New Done Done true
20774 7041 Acked 8040 Flush 0 New Done Done true
20775 7042 Acked 8041 Read 7 New Done Done true
20776 7043 Acked 8042 Read 2 New Done Done true
20777 7044 Acked 8043 Read 4 New Done Done true
20778 7045 Acked 8044 Read 3 New Done Done true
20779 7046 Acked 8045 Read 8 New Done Done true
20780 7047 Acked 8046 Read 8 New Done Done true
20781 7048 Acked 8047 Read 8 New Done Done true
20782 7049 Acked 8048 Write 2 New Done Done true
20783 7050 Acked 8049 Read 10 New Done Done true
20784 7051 Acked 8050 Write 4 New Done Done true
20785 7052 Acked 8051 Flush 0 New Done Done true
20786 7053 Acked 8052 Write 5 New Done Done true
20787 7054 Acked 8053 Write 10 New Done Done true
20788 7055 Acked 8054 Write 10 New Done Done true
20789 7056 Acked 8055 Read 1 New Done Done true
20790 7057 Acked 8056 Read 2 New Done Done true
20791 7058 Acked 8057 Read 7 New Done Done true
20792 7059 Acked 8058 Write 6 New Done Done true
20793 7060 Acked 8059 Write 4 New Done Done true
20794 7061 Acked 8060 Read 7 New Done Done true
20795 7062 Acked 8061 Read 6 New Done Done true
20796 7063 Acked 8062 Write 10 New Done Done true
20797 7064 Acked 8063 Read 9 New Done Done true
20798 7065 Acked 8064 Write 5 New Done Done true
20799 7066 Acked 8065 Flush 0 New Done Done true
20800 7067 Acked 8066 Read 2 New Done Done true
20801 7068 Acked 8067 Read 6 New Done Done true
20802 7069 Acked 8068 Read 9 New Done Done true
20803 7070 Acked 8069 Read 10 New Done Done true
20804 7071 Acked 8070 Read 2 New Done Done true
20805 7072 Acked 8071 Write 9 New Done Done true
20806 7073 Acked 8072 Flush 0 New Done Done true
20807 7074 Acked 8073 Write 5 New Done Done true
20808 7075 Acked 8074 Read 5 New Done Done true
20809 7076 Acked 8075 Read 5 New Done Done true
20810 7077 Acked 8076 Write 4 New Done Done true
20811 7078 Acked 8077 Write 4 New Done Done true
20812 7079 Acked 8078 Write 8 New Done Done true
20813 7080 Acked 8079 Write 5 New Done Done true
20814 7081 Acked 8080 Write 10 New Done Done true
20815 7082 Acked 8081 Flush 0 New Done Done true
20816 7083 Acked 8082 Read 1 New Done Done true
20817 7084 Acked 8083 Read 3 New Done Done true
20818 7085 Acked 8084 Read 3 New Done Done true
20819 7086 Acked 8085 Write 3 New Done Done true
20820 7087 Acked 8086 Write 4 New Done Done true
20821 7088 Acked 8087 Write 8 New Done Done true
20822 7089 Acked 8088 Write 1 New Done Done true
20823 7090 Acked 8089 Read 7 New Done Done true
20824 7091 Acked 8090 Write 1 New Done Done true
20825 7092 Acked 8091 Write 5 New Done Done true
20826 7093 Acked 8092 Read 5 New Done Done true
20827 7094 Acked 8093 Flush 0 New Done Done true
20828 7095 Acked 8094 Read 6 New Done Done true
20829 7096 Acked 8095 Read 9 New Done Done true
20830 7097 Acked 8096 Read 1 New Done Done true
20831 7098 Acked 8097 Read 6 New Done Done true
20832 7099 Acked 8098 Read 7 New Done Done true
20833 7100 Acked 8099 Read 7 New Done Done true
20834 7101 Acked 8100 Read 10 New Done Done true
20835 7102 Acked 8101 Write 4 New Done Done true
20836 7103 Acked 8102 Write 8 New Done Done true
20837 7104 Acked 8103 Read 6 New Done Done true
20838 7105 Acked 8104 Read 7 New Done Done true
20839 7106 Acked 8105 Read 5 New Done Done true
20840 7107 Acked 8106 Write 2 New Done Done true
20841 7108 Acked 8107 Write 5 New Done Done true
20842 7109 Acked 8108 Write 1 New Done Done true
20843 7110 Acked 8109 Write 8 New Done Done true
20844 7111 Acked 8110 Read 5 New Done Done true
20845 7112 Acked 8111 Read 2 New Done Done true
20846 7113 Acked 8112 Read 7 New Done Done true
20847 7114 Acked 8113 Read 9 New Done Done true
20848 7115 Acked 8114 Read 2 New Done Done true
20849 7116 Acked 8115 Read 5 New Done Done true
20850 7117 Acked 8116 Write 3 New Done Done true
20851 7118 Acked 8117 Read 3 New Done Done true
20852 7119 Acked 8118 Read 7 New Done Done true
20853 7120 Acked 8119 Flush 0 New Done Done true
20854 7121 Acked 8120 Write 9 New Done Done true
20855 7122 Acked 8121 Read 4 New Done Done true
20856 7123 Acked 8122 Read 8 New Done Done true
20857 7124 Acked 8123 Read 6 New Done Done true
20858 7125 Acked 8124 Read 1 New Done Done true
20859 7126 Acked 8125 Flush 0 New Done Done true
20860 7127 Acked 8126 Read 9 New Done Done true
20861 7128 Acked 8127 Read 5 New Done Done true
20862 7129 Acked 8128 Write 10 New Done Done true
20863 7130 Acked 8129 Read 9 New Done Done true
20864 7131 Acked 8130 Write 4 New Done Done true
20865 7132 Acked 8131 Write 7 New Done Done true
20866 7133 Acked 8132 Read 7 New Done Done true
20867 7134 Acked 8133 Read 3 New Done Done true
20868 7135 Acked 8134 Write 2 New Done Done true
20869 7136 Acked 8135 Read 1 New Done Done true
20870 7137 Acked 8136 Read 5 New Done Done true
20871 7138 Acked 8137 Read 1 New Done Done true
20872 7139 Acked 8138 Write 7 New Done Done true
20873 7140 Acked 8139 Read 10 New Done Done true
20874 7141 Acked 8140 Write 1 New Done Done true
20875 7142 Acked 8141 Write 1 New Done Done true
20876 7143 Acked 8142 Flush 0 New Done Done true
20877 7144 Acked 8143 Write 3 New Done Done true
20878 7145 Acked 8144 Read 2 New Done Done true
20879 7146 Acked 8145 Read 9 New Done Done true
20880 7147 Acked 8146 Read 5 New Done Done true
20881 7148 Acked 8147 Read 5 New Done Done true
20882 7149 Acked 8148 Read 7 New Done Done true
20883 7150 Acked 8149 Read 4 New Done Done true
20884 7151 Acked 8150 Write 3 New Done Done true
20885 7152 Acked 8151 Flush 0 New Done Done true
20886 7153 Acked 8152 Read 4 New Done Done true
20887 7154 Acked 8153 Read 10 New Done Done true
20888 7155 Acked 8154 Read 10 New Done Done true
20889 7156 Acked 8155 Flush 0 New Done Done true
20890 7157 Acked 8156 Write 8 New Done Done true
20891 7158 Acked 8157 Read 8 New Done Done true
20892 7159 Acked 8158 Write 5 New Done Done true
20893 7160 Acked 8159 Write 3 New Done Done true
20894 7161 Acked 8160 Read 10 New Done Done true
20895 7162 Acked 8161 Flush 0 New Done Done true
20896 7163 Acked 8162 Write 5 New Done Done true
20897 7164 Acked 8163 Read 1 New Done Done true
20898 7165 Acked 8164 Write 6 New Done Done true
20899 7166 Acked 8165 Flush 0 New Done Done true
20900 7167 Acked 8166 Write 4 New Done Done true
20901 7168 Acked 8167 Read 5 New Done Done true
20902 7169 Acked 8168 Read 5 New Done Done true
20903 7170 Acked 8169 Read 7 New Done Done true
20904 7171 Acked 8170 Write 6 New Done Done true
20905 7172 Acked 8171 Write 10 New Done Done true
20906 7173 Acked 8172 Read 2 New Done Done true
20907 7174 Acked 8173 Read 6 New Done Done true
20908 7175 Acked 8174 Write 6 New Done Done true
20909 7176 Acked 8175 Write 10 New Done Done true
20910 7177 Acked 8176 Read 3 New Done Done true
20911 7178 Acked 8177 Read 3 New Done Done true
20912 7179 Acked 8178 Read 6 New Done Done true
20913 7180 Acked 8179 Read 6 New Done Done true
20914 7181 Acked 8180 Read 3 New Done Done true
20915 7182 Acked 8181 Write 9 New Done Done true
20916 7183 Acked 8182 Write 7 New Done Done true
20917 7184 Acked 8183 Flush 0 New Done Done true
20918 7185 Acked 8184 Read 2 New Done Done true
20919 7186 Acked 8185 Write 9 New Done Done true
20920 7187 Acked 8186 Write 9 New Done Done true
20921 7188 Acked 8187 Read 8 New Done Done true
20922 7189 Acked 8188 Read 9 New Done Done true
20923 7190 Acked 8189 Read 6 New Done Done true
20924 7191 Acked 8190 Read 9 New Done Done true
20925 7192 Acked 8191 Write 3 New Done Done true
20926 7193 Acked 8192 Read 6 New Done Done true
20927 7194 Acked 8193 Write 10 New Done Done true
20928 7195 Acked 8194 Write 8 New Done Done true
20929 7196 Acked 8195 Read 7 New Done Done true
20930 7197 Acked 8196 Flush 0 New Done Done true
20931 7198 Acked 8197 Write 1 New Done Done true
20932 7199 Acked 8198 Read 9 New Done Done true
20933 7200 Acked 8199 Read 4 New Done Done true
20934 7201 Acked 8200 Read 3 New Done Done true
20935 7202 Acked 8201 Read 1 New Done Done true
20936 7203 Acked 8202 Read 1 New Done Done true
20937 7204 Acked 8203 Read 8 New Done Done true
20938 7205 Acked 8204 Read 8 New Done Done true
20939 7206 Acked 8205 Write 4 New Done Done true
20940 7207 Acked 8206 Read 4 New Done Done true
20941 7208 Acked 8207 Write 4 New Done Done true
20942 7209 Acked 8208 Write 3 New Done Done true
20943 7210 Acked 8209 Read 2 New Done Done true
20944 7211 Acked 8210 Read 5 New Done Done true
20945 7212 Acked 8211 Write 4 New Done Done true
20946 7213 Acked 8212 Write 8 New Done Done true
20947 7214 Acked 8213 Read 1 New Done Done true
20948 7215 Acked 8214 Read 10 New Done Done true
20949 7216 Acked 8215 Write 4 New Done Done true
20950 7217 Acked 8216 Write 7 New Done Done true
20951 7218 Acked 8217 Read 10 New Done Done true
20952 7219 Acked 8218 Read 1 New Done Done true
20953 7220 Acked 8219 Write 7 New Done Done true
20954 7221 Acked 8220 Read 1 New Done Done true
20955 7222 Acked 8221 Read 3 New Done Done true
20956 7223 Acked 8222 Flush 0 New Done Done true
20957 7224 Acked 8223 Read 8 New Done Done true
20958 7225 Acked 8224 Write 8 New Done Done true
20959 7226 Acked 8225 Read 6 New Done Done true
20960 7227 Acked 8226 Read 10 New Done Done true
20961 7228 Acked 8227 Flush 0 New Done Done true
20962 7229 Acked 8228 Write 8 New Done Done true
20963 7230 Acked 8229 Read 6 New Done Done true
20964 7231 Acked 8230 Write 6 New Done Done true
20965 7232 Acked 8231 Write 9 New Done Done true
20966 7233 Acked 8232 Write 4 New Done Done true
20967 7234 Acked 8233 Flush 0 New Done Done true
20968 STATES DS:0 DS:1 DS:2 TOTAL
20969 New 301 0 0 301
20970 Sent 0 0 0 0
20971 Done 0 301 301 602
20972 Skipped 0 0 0 0
20973 Error 0 0 0 0
20974 Last Flush: 7932 8233 8233
20975 Downstairs last five completed: 7932 7931 7930 7929 7928
20976 Upstairs last five completed: 7234 7233 7232 7231 7230
20977 CLIENT: Up:0 ds:301 act:2
209782024-02-28T22:17:25.407ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
209792024-02-28T22:17:25.407ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
209802024-02-28T22:17:25.407ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
209812024-02-28T22:17:25.407ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
209822024-02-28T22:17:25.407ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
209832024-02-28T22:17:25.407ZINFOcrucible: [0] client re-new 301 jobs since flush 7932 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
209842024-02-28T22:17:25.407ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
209852024-02-28T22:17:26.408ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
209862024-02-28T22:17:26.408ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
209872024-02-28T22:17:26.408ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
209882024-02-28T22:17:26.408ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
209892024-02-28T22:17:26.408ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
209902024-02-28T22:17:26.408ZINFOcrucible: [0] client re-new 301 jobs since flush 7932 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
209912024-02-28T22:17:26.408ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
209922024-02-28T22:17:27.409ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
209932024-02-28T22:17:27.409ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
209942024-02-28T22:17:27.409ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
209952024-02-28T22:17:27.409ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
209962024-02-28T22:17:27.409ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
209972024-02-28T22:17:27.409ZINFOcrucible: [0] client re-new 301 jobs since flush 7932 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
209982024-02-28T22:17:27.409ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
209992024-02-28T22:17:28.410ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
210002024-02-28T22:17:28.410ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
210012024-02-28T22:17:28.410ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
210022024-02-28T22:17:28.410ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
210032024-02-28T22:17:28.410ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
210042024-02-28T22:17:28.410ZINFOcrucible: [0] client re-new 301 jobs since flush 7932 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
210052024-02-28T22:17:28.410ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
21006 ----------------------------------------------------------------
21007 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21008 States: Offline Active Active
21009 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
21010 6934 Acked 7933 Write 2 New Done Done true
21011 6935 Acked 7934 Write 6 New Done Done true
21012 6936 Acked 7935 Write 4 New Done Done true
21013 6937 Acked 7936 Write 3 New Done Done true
21014 6938 Acked 7937 Read 10 New Done Done true
21015 6939 Acked 7938 Write 10 New Done Done true
21016 6940 Acked 7939 Read 2 New Done Done true
21017 6941 Acked 7940 Read 9 New Done Done true
21018 6942 Acked 7941 Read 1 New Done Done true
21019 6943 Acked 7942 Write 4 New Done Done true
21020 6944 Acked 7943 Read 2 New Done Done true
21021 6945 Acked 7944 Read 9 New Done Done true
21022 6946 Acked 7945 Read 5 New Done Done true
21023 6947 Acked 7946 Write 5 New Done Done true
21024 6948 Acked 7947 Read 1 New Done Done true
21025 6949 Acked 7948 Read 7 New Done Done true
21026 6950 Acked 7949 Write 9 New Done Done true
21027 6951 Acked 7950 Read 1 New Done Done true
21028 6952 Acked 7951 Read 4 New Done Done true
21029 6953 Acked 7952 Flush 0 New Done Done true
21030 6954 Acked 7953 Write 9 New Done Done true
21031 6955 Acked 7954 Flush 0 New Done Done true
21032 6956 Acked 7955 Write 4 New Done Done true
21033 6957 Acked 7956 Write 4 New Done Done true
21034 6958 Acked 7957 Read 3 New Done Done true
21035 6959 Acked 7958 Write 9 New Done Done true
21036 6960 Acked 7959 Flush 0 New Done Done true
21037 6961 Acked 7960 Write 1 New Done Done true
21038 6962 Acked 7961 Write 4 New Done Done true
21039 6963 Acked 7962 Read 8 New Done Done true
21040 6964 Acked 7963 Read 2 New Done Done true
21041 6965 Acked 7964 Write 9 New Done Done true
21042 6966 Acked 7965 Write 6 New Done Done true
21043 6967 Acked 7966 Write 3 New Done Done true
21044 6968 Acked 7967 Read 2 New Done Done true
21045 6969 Acked 7968 Write 8 New Done Done true
21046 6970 Acked 7969 Write 1 New Done Done true
21047 6971 Acked 7970 Flush 0 New Done Done true
21048 6972 Acked 7971 Read 9 New Done Done true
21049 6973 Acked 7972 Write 10 New Done Done true
21050 6974 Acked 7973 Flush 0 New Done Done true
21051 6975 Acked 7974 Flush 0 New Done Done true
21052 6976 Acked 7975 Read 7 New Done Done true
21053 6977 Acked 7976 Read 9 New Done Done true
21054 6978 Acked 7977 Read 1 New Done Done true
21055 6979 Acked 7978 Flush 0 New Done Done true
21056 6980 Acked 7979 Write 8 New Done Done true
21057 6981 Acked 7980 Read 7 New Done Done true
21058 6982 Acked 7981 Read 10 New Done Done true
21059 6983 Acked 7982 Write 8 New Done Done true
21060 6984 Acked 7983 Read 7 New Done Done true
21061 6985 Acked 7984 Read 6 New Done Done true
21062 6986 Acked 7985 Flush 0 New Done Done true
21063 6987 Acked 7986 Write 10 New Done Done true
21064 6988 Acked 7987 Write 8 New Done Done true
21065 6989 Acked 7988 Flush 0 New Done Done true
21066 6990 Acked 7989 Write 10 New Done Done true
21067 6991 Acked 7990 Read 3 New Done Done true
21068 6992 Acked 7991 Read 8 New Done Done true
21069 6993 Acked 7992 Read 9 New Done Done true
21070 6994 Acked 7993 Read 5 New Done Done true
21071 6995 Acked 7994 Read 5 New Done Done true
21072 6996 Acked 7995 Read 9 New Done Done true
21073 6997 Acked 7996 Write 8 New Done Done true
21074 6998 Acked 7997 Read 3 New Done Done true
21075 6999 Acked 7998 Read 1 New Done Done true
21076 7000 Acked 7999 Read 4 New Done Done true
21077 7001 Acked 8000 Read 7 New Done Done true
21078 7002 Acked 8001 Write 2 New Done Done true
21079 7003 Acked 8002 Flush 0 New Done Done true
21080 7004 Acked 8003 Read 9 New Done Done true
21081 7005 Acked 8004 Write 7 New Done Done true
21082 7006 Acked 8005 Read 7 New Done Done true
21083 7007 Acked 8006 Read 4 New Done Done true
21084 7008 Acked 8007 Read 2 New Done Done true
21085 7009 Acked 8008 Read 7 New Done Done true
21086 7010 Acked 8009 Flush 0 New Done Done true
21087 7011 Acked 8010 Write 1 New Done Done true
21088 7012 Acked 8011 Flush 0 New Done Done true
21089 7013 Acked 8012 Read 7 New Done Done true
21090 7014 Acked 8013 Write 5 New Done Done true
21091 7015 Acked 8014 Flush 0 New Done Done true
21092 7016 Acked 8015 Write 9 New Done Done true
21093 7017 Acked 8016 Read 3 New Done Done true
21094 7018 Acked 8017 Read 2 New Done Done true
21095 7019 Acked 8018 Write 1 New Done Done true
21096 7020 Acked 8019 Read 2 New Done Done true
21097 7021 Acked 8020 Read 9 New Done Done true
21098 7022 Acked 8021 Write 3 New Done Done true
21099 7023 Acked 8022 Read 6 New Done Done true
21100 7024 Acked 8023 Read 6 New Done Done true
21101 7025 Acked 8024 Read 5 New Done Done true
21102 7026 Acked 8025 Read 10 New Done Done true
21103 7027 Acked 8026 Read 5 New Done Done true
21104 7028 Acked 8027 Write 10 New Done Done true
21105 7029 Acked 8028 Flush 0 New Done Done true
21106 7030 Acked 8029 Read 6 New Done Done true
21107 7031 Acked 8030 Flush 0 New Done Done true
21108 7032 Acked 8031 Write 8 New Done Done true
21109 7033 Acked 8032 Read 7 New Done Done true
21110 7034 Acked 8033 Write 10 New Done Done true
21111 7035 Acked 8034 Read 5 New Done Done true
21112 7036 Acked 8035 Write 5 New Done Done true
21113 7037 Acked 8036 Read 5 New Done Done true
21114 7038 Acked 8037 Write 6 New Done Done true
21115 7039 Acked 8038 Write 6 New Done Done true
21116 7040 Acked 8039 Read 3 New Done Done true
21117 7041 Acked 8040 Flush 0 New Done Done true
21118 7042 Acked 8041 Read 7 New Done Done true
21119 7043 Acked 8042 Read 2 New Done Done true
21120 7044 Acked 8043 Read 4 New Done Done true
21121 7045 Acked 8044 Read 3 New Done Done true
21122 7046 Acked 8045 Read 8 New Done Done true
21123 7047 Acked 8046 Read 8 New Done Done true
21124 7048 Acked 8047 Read 8 New Done Done true
21125 7049 Acked 8048 Write 2 New Done Done true
21126 7050 Acked 8049 Read 10 New Done Done true
21127 7051 Acked 8050 Write 4 New Done Done true
21128 7052 Acked 8051 Flush 0 New Done Done true
21129 7053 Acked 8052 Write 5 New Done Done true
21130 7054 Acked 8053 Write 10 New Done Done true
21131 7055 Acked 8054 Write 10 New Done Done true
21132 7056 Acked 8055 Read 1 New Done Done true
21133 7057 Acked 8056 Read 2 New Done Done true
21134 7058 Acked 8057 Read 7 New Done Done true
21135 7059 Acked 8058 Write 6 New Done Done true
21136 7060 Acked 8059 Write 4 New Done Done true
21137 7061 Acked 8060 Read 7 New Done Done true
21138 7062 Acked 8061 Read 6 New Done Done true
21139 7063 Acked 8062 Write 10 New Done Done true
21140 7064 Acked 8063 Read 9 New Done Done true
21141 7065 Acked 8064 Write 5 New Done Done true
21142 7066 Acked 8065 Flush 0 New Done Done true
21143 7067 Acked 8066 Read 2 New Done Done true
21144 7068 Acked 8067 Read 6 New Done Done true
21145 7069 Acked 8068 Read 9 New Done Done true
21146 7070 Acked 8069 Read 10 New Done Done true
21147 7071 Acked 8070 Read 2 New Done Done true
21148 7072 Acked 8071 Write 9 New Done Done true
21149 7073 Acked 8072 Flush 0 New Done Done true
21150 7074 Acked 8073 Write 5 New Done Done true
21151 7075 Acked 8074 Read 5 New Done Done true
21152 7076 Acked 8075 Read 5 New Done Done true
21153 7077 Acked 8076 Write 4 New Done Done true
21154 7078 Acked 8077 Write 4 New Done Done true
21155 7079 Acked 8078 Write 8 New Done Done true
21156 7080 Acked 8079 Write 5 New Done Done true
21157 7081 Acked 8080 Write 10 New Done Done true
21158 7082 Acked 8081 Flush 0 New Done Done true
21159 7083 Acked 8082 Read 1 New Done Done true
21160 7084 Acked 8083 Read 3 New Done Done true
21161 7085 Acked 8084 Read 3 New Done Done true
21162 7086 Acked 8085 Write 3 New Done Done true
21163 7087 Acked 8086 Write 4 New Done Done true
21164 7088 Acked 8087 Write 8 New Done Done true
21165 7089 Acked 8088 Write 1 New Done Done true
21166 7090 Acked 8089 Read 7 New Done Done true
21167 7091 Acked 8090 Write 1 New Done Done true
21168 7092 Acked 8091 Write 5 New Done Done true
21169 7093 Acked 8092 Read 5 New Done Done true
21170 7094 Acked 8093 Flush 0 New Done Done true
21171 7095 Acked 8094 Read 6 New Done Done true
21172 7096 Acked 8095 Read 9 New Done Done true
21173 7097 Acked 8096 Read 1 New Done Done true
21174 7098 Acked 8097 Read 6 New Done Done true
21175 7099 Acked 8098 Read 7 New Done Done true
21176 7100 Acked 8099 Read 7 New Done Done true
21177 7101 Acked 8100 Read 10 New Done Done true
21178 7102 Acked 8101 Write 4 New Done Done true
21179 7103 Acked 8102 Write 8 New Done Done true
21180 7104 Acked 8103 Read 6 New Done Done true
21181 7105 Acked 8104 Read 7 New Done Done true
21182 7106 Acked 8105 Read 5 New Done Done true
21183 7107 Acked 8106 Write 2 New Done Done true
21184 7108 Acked 8107 Write 5 New Done Done true
21185 7109 Acked 8108 Write 1 New Done Done true
21186 7110 Acked 8109 Write 8 New Done Done true
21187 7111 Acked 8110 Read 5 New Done Done true
21188 7112 Acked 8111 Read 2 New Done Done true
21189 7113 Acked 8112 Read 7 New Done Done true
21190 7114 Acked 8113 Read 9 New Done Done true
21191 7115 Acked 8114 Read 2 New Done Done true
21192 7116 Acked 8115 Read 5 New Done Done true
21193 7117 Acked 8116 Write 3 New Done Done true
21194 7118 Acked 8117 Read 3 New Done Done true
21195 7119 Acked 8118 Read 7 New Done Done true
21196 7120 Acked 8119 Flush 0 New Done Done true
21197 7121 Acked 8120 Write 9 New Done Done true
21198 7122 Acked 8121 Read 4 New Done Done true
21199 7123 Acked 8122 Read 8 New Done Done true
21200 7124 Acked 8123 Read 6 New Done Done true
21201 7125 Acked 8124 Read 1 New Done Done true
21202 7126 Acked 8125 Flush 0 New Done Done true
21203 7127 Acked 8126 Read 9 New Done Done true
21204 7128 Acked 8127 Read 5 New Done Done true
21205 7129 Acked 8128 Write 10 New Done Done true
21206 7130 Acked 8129 Read 9 New Done Done true
21207 7131 Acked 8130 Write 4 New Done Done true
21208 7132 Acked 8131 Write 7 New Done Done true
21209 7133 Acked 8132 Read 7 New Done Done true
21210 7134 Acked 8133 Read 3 New Done Done true
21211 7135 Acked 8134 Write 2 New Done Done true
21212 7136 Acked 8135 Read 1 New Done Done true
21213 7137 Acked 8136 Read 5 New Done Done true
21214 7138 Acked 8137 Read 1 New Done Done true
21215 7139 Acked 8138 Write 7 New Done Done true
21216 7140 Acked 8139 Read 10 New Done Done true
21217 7141 Acked 8140 Write 1 New Done Done true
21218 7142 Acked 8141 Write 1 New Done Done true
21219 7143 Acked 8142 Flush 0 New Done Done true
21220 7144 Acked 8143 Write 3 New Done Done true
21221 7145 Acked 8144 Read 2 New Done Done true
21222 7146 Acked 8145 Read 9 New Done Done true
21223 7147 Acked 8146 Read 5 New Done Done true
21224 7148 Acked 8147 Read 5 New Done Done true
21225 7149 Acked 8148 Read 7 New Done Done true
21226 7150 Acked 8149 Read 4 New Done Done true
21227 7151 Acked 8150 Write 3 New Done Done true
21228 7152 Acked 8151 Flush 0 New Done Done true
21229 7153 Acked 8152 Read 4 New Done Done true
21230 7154 Acked 8153 Read 10 New Done Done true
21231 7155 Acked 8154 Read 10 New Done Done true
21232 7156 Acked 8155 Flush 0 New Done Done true
21233 7157 Acked 8156 Write 8 New Done Done true
21234 7158 Acked 8157 Read 8 New Done Done true
21235 7159 Acked 8158 Write 5 New Done Done true
21236 7160 Acked 8159 Write 3 New Done Done true
21237 7161 Acked 8160 Read 10 New Done Done true
21238 7162 Acked 8161 Flush 0 New Done Done true
21239 7163 Acked 8162 Write 5 New Done Done true
21240 7164 Acked 8163 Read 1 New Done Done true
21241 7165 Acked 8164 Write 6 New Done Done true
21242 7166 Acked 8165 Flush 0 New Done Done true
21243 7167 Acked 8166 Write 4 New Done Done true
21244 7168 Acked 8167 Read 5 New Done Done true
21245 7169 Acked 8168 Read 5 New Done Done true
21246 7170 Acked 8169 Read 7 New Done Done true
21247 7171 Acked 8170 Write 6 New Done Done true
21248 7172 Acked 8171 Write 10 New Done Done true
21249 7173 Acked 8172 Read 2 New Done Done true
21250 7174 Acked 8173 Read 6 New Done Done true
21251 7175 Acked 8174 Write 6 New Done Done true
21252 7176 Acked 8175 Write 10 New Done Done true
21253 7177 Acked 8176 Read 3 New Done Done true
21254 7178 Acked 8177 Read 3 New Done Done true
21255 7179 Acked 8178 Read 6 New Done Done true
21256 7180 Acked 8179 Read 6 New Done Done true
21257 7181 Acked 8180 Read 3 New Done Done true
21258 7182 Acked 8181 Write 9 New Done Done true
21259 7183 Acked 8182 Write 7 New Done Done true
21260 7184 Acked 8183 Flush 0 New Done Done true
21261 7185 Acked 8184 Read 2 New Done Done true
21262 7186 Acked 8185 Write 9 New Done Done true
21263 7187 Acked 8186 Write 9 New Done Done true
21264 7188 Acked 8187 Read 8 New Done Done true
21265 7189 Acked 8188 Read 9 New Done Done true
21266 7190 Acked 8189 Read 6 New Done Done true
21267 7191 Acked 8190 Read 9 New Done Done true
21268 7192 Acked 8191 Write 3 New Done Done true
21269 7193 Acked 8192 Read 6 New Done Done true
21270 7194 Acked 8193 Write 10 New Done Done true
21271 7195 Acked 8194 Write 8 New Done Done true
21272 7196 Acked 8195 Read 7 New Done Done true
21273 7197 Acked 8196 Flush 0 New Done Done true
21274 7198 Acked 8197 Write 1 New Done Done true
21275 7199 Acked 8198 Read 9 New Done Done true
21276 7200 Acked 8199 Read 4 New Done Done true
21277 7201 Acked 8200 Read 3 New Done Done true
21278 7202 Acked 8201 Read 1 New Done Done true
21279 7203 Acked 8202 Read 1 New Done Done true
21280 7204 Acked 8203 Read 8 New Done Done true
21281 7205 Acked 8204 Read 8 New Done Done true
21282 7206 Acked 8205 Write 4 New Done Done true
21283 7207 Acked 8206 Read 4 New Done Done true
21284 7208 Acked 8207 Write 4 New Done Done true
21285 7209 Acked 8208 Write 3 New Done Done true
21286 7210 Acked 8209 Read 2 New Done Done true
21287 7211 Acked 8210 Read 5 New Done Done true
21288 7212 Acked 8211 Write 4 New Done Done true
21289 7213 Acked 8212 Write 8 New Done Done true
21290 7214 Acked 8213 Read 1 New Done Done true
21291 7215 Acked 8214 Read 10 New Done Done true
21292 7216 Acked 8215 Write 4 New Done Done true
21293 7217 Acked 8216 Write 7 New Done Done true
21294 7218 Acked 8217 Read 10 New Done Done true
21295 7219 Acked 8218 Read 1 New Done Done true
21296 7220 Acked 8219 Write 7 New Done Done true
21297 7221 Acked 8220 Read 1 New Done Done true
21298 7222 Acked 8221 Read 3 New Done Done true
21299 7223 Acked 8222 Flush 0 New Done Done true
21300 7224 Acked 8223 Read 8 New Done Done true
21301 7225 Acked 8224 Write 8 New Done Done true
21302 7226 Acked 8225 Read 6 New Done Done true
21303 7227 Acked 8226 Read 10 New Done Done true
21304 7228 Acked 8227 Flush 0 New Done Done true
21305 7229 Acked 8228 Write 8 New Done Done true
21306 7230 Acked 8229 Read 6 New Done Done true
21307 7231 Acked 8230 Write 6 New Done Done true
21308 7232 Acked 8231 Write 9 New Done Done true
21309 7233 Acked 8232 Write 4 New Done Done true
21310 7234 Acked 8233 Flush 0 New Done Done true
21311 STATES DS:0 DS:1 DS:2 TOTAL
21312 New 301 0 0 301
21313 Sent 0 0 0 0
21314 Done 0 301 301 602
21315 Skipped 0 0 0 0
21316 Error 0 0 0 0
21317 Last Flush: 7932 8233 8233
21318 Downstairs last five completed: 7932 7931 7930 7929 7928
21319 Upstairs last five completed: 7234 7233 7232 7231 7230
21320 CLIENT: Up:0 ds:301 act:2
213212024-02-28T22:17:29.411ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213222024-02-28T22:17:29.411ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213232024-02-28T22:17:29.411ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213242024-02-28T22:17:29.411ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213252024-02-28T22:17:29.411ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213262024-02-28T22:17:29.411ZINFOcrucible: [0] client re-new 301 jobs since flush 7932 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213272024-02-28T22:17:29.412ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
21328 [8810] got start action
21329 [8810] I am going to restart
21330 Starting downstairs at port 8810
21331 Make output file at "/tmp/dsc/downstairs-8810.txt"
21332 [8810][0] reports Starting
21333 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1188)
21334 [8810][0] reports Running
213352024-02-28T22:17:30.412ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213362024-02-28T22:17:30.412ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213372024-02-28T22:17:30.412ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213382024-02-28T22:17:30.412ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213392024-02-28T22:17:30.412ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213402024-02-28T22:17:30.412ZINFOcrucible: [0] client re-new 301 jobs since flush 7932 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213412024-02-28T22:17:30.412ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213422024-02-28T22:17:31.413ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213432024-02-28T22:17:31.413ZINFOcrucible: ds_connection connected = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213442024-02-28T22:17:31.415ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213452024-02-28T22:17:31.415ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213462024-02-28T22:17:31.415ZINFOcrucible: send last flush ID to this DS: 7932 = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213472024-02-28T22:17:31.415ZINFOcrucible: Replied this last flush ID: 7932 = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213482024-02-28T22:17:31.415ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213492024-02-28T22:17:31.415ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213502024-02-28T22:17:31.415ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213512024-02-28T22:17:31.415ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
21352 ----------------------------------------------------------------
21353 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
21354 Downstairs last five completed: 8233 8232 8231 8230 8229
21355 Upstairs last five completed: 7234 7233 7232 7231 7230
21356 CLIENT: Up:0 ds:0 act:3
21357 Replay: All jobs finished, all DS active.
21358 Main task has work to do, go find it
21359 got dsc Stop(2)
21360 stop 2
21361 [8830] Got stop action so:false kr:false
21362 Feb 28 22:17:32.751 INFO request completed, latency_us: 335, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 7df72aa8-97a2-45c5-b600-8807b1bd4363, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
213632024-02-28T22:17:32.751ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213642024-02-28T22:17:32.751ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213652024-02-28T22:17:32.751ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213662024-02-28T22:17:32.751ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213672024-02-28T22:17:32.751ZINFOcrucible: [2] client re-new 0 jobs since flush 8233 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213682024-02-28T22:17:32.751ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
21369 Feb 28 22:17:32.751 INFO request completed, latency_us: 132, response_code: 200, uri: /state/cid/2, method: GET, req_id: 9fcb5b03-d1ea-4860-95bb-13aa5dba9193, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
21370 [8830] Exited with: ExitStatus(unix_wait_status(9))
21371 [8830][2] reports Exit
213722024-02-28T22:17:33.752ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213732024-02-28T22:17:33.753ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213742024-02-28T22:17:33.753ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213752024-02-28T22:17:33.753ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213762024-02-28T22:17:33.753ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213772024-02-28T22:17:33.753ZINFOcrucible: [2] client re-new 0 jobs since flush 8233 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213782024-02-28T22:17:33.753ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213792024-02-28T22:17:34.755ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213802024-02-28T22:17:34.755ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213812024-02-28T22:17:34.755ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213822024-02-28T22:17:34.755ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213832024-02-28T22:17:34.755ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213842024-02-28T22:17:34.755ZINFOcrucible: [2] client re-new 0 jobs since flush 8233 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213852024-02-28T22:17:34.755ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213862024-02-28T22:17:35.757ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213872024-02-28T22:17:35.757ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213882024-02-28T22:17:35.757ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213892024-02-28T22:17:35.757ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213902024-02-28T22:17:35.757ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213912024-02-28T22:17:35.757ZINFOcrucible: [2] client re-new 0 jobs since flush 8233 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213922024-02-28T22:17:35.757ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
21393 Feb 28 22:17:36.754 INFO request completed, latency_us: 279, response_code: 200, uri: /state/cid/2, method: GET, req_id: 0842c679-03de-4130-939c-91fc71bcd55b, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
21394 001/300 Read block 1946 len 12288
21395 002/300 Read block 1912 len 40960
213962024-02-28T22:17:36.758ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
213972024-02-28T22:17:36.758ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
21398 {"msg":"003/300client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \" Write block 4396 len 4096 data:Connection refused\" 4 }))"
21399 ,"v":0,"name":"crucible","level":40,"time":"2024-02-28T22:17:36.758148821Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
214002024-02-28T22:17:36.758ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
214012024-02-28T22:17:36.758ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
214022024-02-28T22:17:36.758ZINFOcrucible: [2] client re-new 2 jobs since flush 8233 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
214032024-02-28T22:17:36.758ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
21404 004/300 Write block 4605 len 12288 data: 5 6 6
21405 005/300 Read block 3499 len 16384
21406 006/300 Write block 2726 len 12288 data: 5 6 6
21407 007/300 Write block 4768 len 4096 data: 6
21408 008/300 Write block 2045 len 32768 data: 5 4 4 4 6 6 8 9
21409 009/300 Write block 2292 len 32768 data: 4 5 5 5 5 7 7 7
21410 010/300 Read block 4793 len 16384
21411 011/300 Read block 3307 len 12288
21412 012/300 Write block 1842 len 4096 data: 4
21413 013/300 Flush
21414 014/300 Flush
21415 015/300 Write block 3854 len 20480 data: 5 8 7 8 6
21416 016/300 Write block 3926 len 8192 data: 6 6
21417 017/300 Write block 723 len 36864 data: 3 3 3 4 4 4 4 5 6
21418 018/300 Write block 445 len 28672 data: 5 6 5 5 5 5 5
21419 019/300 Write block 4408 len 32768 data: 4 3 3 4 5 6 6 6
21420 020/300 Write block 2992 len 8192 data: 4 5
21421 021/300 Read block 1914 len 16384
21422 022/300 Read block 2808 len 4096
21423 023/300 Write block 41 len 28672 data: 6 4 4 3 2 3 3
21424 024/300 Flush
21425 025/300 Write block 3643 len 28672 data: 6 6 6 6 5 7 7
21426 026/300 Read block 4351 len 36864
21427 027/300 Write block 373 len 20480 data: 4 4 4 4 4
21428 028/300 Read block 2516 len 28672
21429 029/300 Read block 4004 len 40960
21430 030/300 Read block 2099 len 24576
21431 031/300 Read block 2451 len 12288
21432 032/300 Read block 3969 len 16384
21433 033/300 Read block 1323 len 8192
21434 034/300 Write block 495 len 32768 data: 7 7 6 7 6 5 4 3
21435 035/300 Read block 2892 len 4096
21436 036/300 Read block 1827 len 40960
21437 037/300 Write block 164 len 4096 data: 4
21438 038/300 Read block 905 len 12288
21439 039/300 Read block 598 len 36864
21440 040/300 Read block 43 len 32768
21441 041/300 Read block 857 len 28672
21442 042/300 Read block 4221 len 4096
21443 043/300 Read block 493 len 24576
21444 044/300 Write block 4967 len 28672 data: 3 3 3 3 3 2 2
21445 045/300 Flush
21446 046/300 Read block 896 len 12288
21447 047/300 Read block 815 len 8192
21448 048/300 Read block 4814 len 40960
21449 049/300 Flush
21450 050/300 Read block 1370 len 16384
21451 051/300 Flush
21452 052/300 Write block 4656 len 40960 data: 7 6 6 5 5 4 4 4 3 4
21453 053/300 Read block 3612 len 8192
21454 054/300 Read block 1896 len 20480
21455 055/300 Write block 4000 len 32768 data: 7 6 7 7 6 6 5 4
21456 056/300 Write block 2347 len 40960 data: 4 5 4 5 4 4 4 3 3 2
21457 057/300 Write block 879 len 12288 data: 6 6 4
21458 058/300 Read block 4315 len 16384
21459 059/300 Write block 4478 len 20480 data: 7 7 7 6 6
21460 060/300 Write block 4152 len 8192 data: 5 6
21461 061/300 Read block 368 len 8192
21462 062/300 Write block 1767 len 16384 data: 4 4 5 7
21463 063/300 Read block 813 len 16384
21464 064/300 Read block 2631 len 40960
21465 065/300 Read block 2137 len 12288
21466 066/300 Write block 794 len 36864 data: 6 6 5 5 5 5 6 7 4
21467 067/300 Read block 2416 len 8192
21468 068/300 Read block 3614 len 32768
21469 069/300 Write block 771 len 36864 data: 4 4 4 5 5 6 7 8 7
21470 070/300 Read block 1793 len 4096
21471 071/300 Flush
21472 072/300 Read block 4574 len 16384
21473 073/300 Write block 1702 len 12288 data: 3 3 4
21474 074/300 Read block 2657 len 8192
21475 075/300 Write block 3060 len 4096 data: 9
21476 076/300 Read block 96 len 16384
21477 077/300 Write block 3521 len 36864 data: 6 4 3 3 5 5 6 6 6
21478 078/300 Read block 695 len 16384
21479 079/300 Write block 1063 len 32768 data: 6 7 6 7 7 6 4 4
21480 080/300 Write block 3068 len 32768 data: 5 6 4 5 5 6 6 6
21481 081/300 Read block 194 len 20480
21482 082/300 Read block 1559 len 40960
21483 083/300 Read block 2671 len 40960
21484 084/300 Read block 1820 len 36864
21485 085/300 Read block 108 len 28672
21486 086/300 Write block 897 len 12288 data: 4 4 2
21487 087/300 Write block 1577 len 8192 data: 3 6
21488 088/300 Write block 187 len 4096 data: 8
21489 089/300 Write block 3261 len 40960 data: 5 5 5 5 4 6 6 5 5 5
21490 090/300 Write block 646 len 12288 data: 3 2 3
21491 091/300 Write block 2699 len 36864 data: 4 4 4 3 3 4 5 5 4
21492 092/300 Flush
21493 093/300 Read block 3429 len 40960
21494 094/300 Write block 105 len 24576 data: 4 5 4 4 4 5
21495 095/300 Read block 3995 len 32768
21496 096/300 Read block 2868 len 12288
21497 097/300 Read block 2420 len 20480
21498 098/300 Write block 468 len 40960 data: 8 8 7 6 6 6 7 8 8 7
21499 099/300 Write block 473 len 8192 data: 7 8
21500 100/300 Write block 111 len 12288 data: 4 5 4
21501 101/300 Read block 4730 len 20480
21502 102/300 Read block 2795 len 8192
21503 103/300 Read block 3418 len 40960
21504 104/300 Write block 1843 len 16384 data: 4 4 5 5
21505 105/300 Read block 2656 len 16384
21506 106/300 Write block 1660 len 4096 data: 3
21507 107/300 Write block 2047 len 16384 data: 5 5 7 7
21508 108/300 Write block 1308 len 12288 data: 4 4 5
21509 109/300 Read block 3107 len 12288
21510 110/300 Read block 3433 len 28672
21511 111/300 Write block 2702 len 36864 data: 4 4 5 6 6 5 3 2 2
21512 112/300 Flush
21513 113/300 Write block 3725 len 24576 data: 7 8 8 8 9 8
21514 114/300 Write block 3534 len 40960 data: 5 6 5 4 4 3 3 4 4 7
21515 115/300 Write block 4145 len 4096 data: 6
21516 116/300 Write block 2 len 28672 data: 4 4 4 5 5 6 5
21517 117/300 Read block 4608 len 8192
21518 118/300 Read block 4726 len 12288
21519 119/300 Read block 2860 len 4096
21520 120/300 Read block 3591 len 8192
21521 121/300 Read block 1655 len 16384
21522 122/300 Flush
21523 123/300 Read block 3037 len 12288
21524 124/300 Read block 1846 len 28672
21525 125/300 Read block 2350 len 4096
21526 126/300 Read block 2750 len 40960
21527 127/300 Read block 2914 len 12288
21528 128/300 Read block 4924 len 8192
21529 129/300 Read block 4792 len 4096
21530 130/300 Read block 4337 len 4096
21531 131/300 Flush
21532 132/300 Flush
21533 133/300 Write block 2765 len 4096 data: 5
21534 134/300 Write block 2890 len 4096 data: 10
21535 135/300 Read block 4205 len 28672
21536 136/300 Write block 1043 len 40960 data: 6 7 7 7 7 6 6 6 5 5
21537 137/300 Flush
21538 138/300 Flush
21539 139/300 Write block 1698 len 40960 data: 2 4 3 4 4 4 5 4 3 2
21540 140/300 Read block 4719 len 36864
21541 141/300 Read block 3721 len 40960
21542 142/300 Read block 3540 len 36864
21543 143/300 Read block 2401 len 40960
21544 144/300 Read block 1598 len 12288
21545 145/300 Write block 4981 len 8192 data: 6 5
21546 146/300 Read block 4655 len 28672
21547 147/300 Write block 4858 len 28672 data: 5 6 7 7 9 9 10
21548 148/300 Flush
21549 149/300 Write block 3481 len 32768 data: 4 5 5 4 4 3 2 3
21550 150/300 Flush
21551 151/300 Write block 4669 len 40960 data: 5 5 5 6 4 4 5 4 4 3
21552 152/300 Write block 3548 len 8192 data: 6 6
21553 153/300 Write block 2670 len 20480 data: 4 6 6 7 6
21554 154/300 Read block 1697 len 12288
21555 155/300 Read block 2279 len 40960
21556 156/300 Read block 2952 len 4096
21557 157/300 Read block 4901 len 12288
21558 158/300 Read block 704 len 24576
21559 159/300 Write block 4598 len 24576 data: 5 5 6 6 5 5
21560 160/300 Write block 611 len 36864 data: 6 6 5 5 5 4 5 6 6
21561 161/300 Write block 134 len 36864 data: 8 6 6 6 5 6 8 8 7
21562 162/300 Write block 4516 len 40960 data: 4 4 3 2 4 3 3 4 4 3
21563 163/300 Read block 1304 len 8192
21564 164/300 Read block 2034 len 8192
21565 165/300 Write block 3706 len 40960 data: 5 4 6 8 6 5 4 4 4 5
21566 166/300 Write block 2753 len 16384 data: 3 3 3 2
21567 167/300 Write block 3562 len 28672 data: 6 6 6 6 7 6 7
21568 168/300 Flush
21569 169/300 Read block 4277 len 16384
21570 170/300 Write block 2181 len 32768 data: 6 6 6 4 5 5 5 5
21571 171/300 Read block 3971 len 20480
21572 172/300 Write block 1865 len 20480 data: 7 7 6 5 4
21573 173/300 Write block 222 len 20480 data: 6 5 4 4 5
21574 174/300 Read block 2161 len 32768
21575 175/300 Write block 4922 len 12288 data: 5 5 6
21576 176/300 Read block 2100 len 24576
21577 177/300 Write block 1307 len 20480 data: 4 5 5 6 6
21578 178/300 Read block 4891 len 4096
21579 179/300 Flush
21580 180/300 Flush
21581 181/300 Read block 1744 len 40960
21582 182/300 Write block 715 len 12288 data: 5 3 5
21583 183/300 Write block 3854 len 16384 data: 6 9 8 9
21584 184/300 Read block 4172 len 24576
21585 185/300 Write block 280 len 28672 data: 6 5 5 5 4 3 3
21586 186/300 Read block 1061 len 16384
21587 187/300 Read block 4764 len 36864
21588 188/300 Flush
21589 189/300 Read block 4687 len 24576
21590 190/300 Read block 225 len 8192
21591 191/300 Write block 2846 len 24576 data: 4 5 3 3 3 4
21592 192/300 Write block 4336 len 8192 data: 9 7
21593 193/300 Read block 3592 len 24576
21594 194/300 Write block 4901 len 8192 data: 4 4
21595 195/300 Read block 4785 len 16384
21596 196/300 Read block 4134 len 20480
21597 197/300 Read block 2036 len 28672
21598 198/300 Read block 842 len 12288
21599 199/300 Read block 2836 len 24576
21600 200/300 Write block 2407 len 24576 data: 5 5 6 6 6 8
21601 201/300 Read block 2975 len 12288
21602 202/300 Write block 1281 len 24576 data: 7 6 5 5 3 3
21603 203/300 Write block 2098 len 8192 data: 6 8
21604 204/300 Read block 2282 len 12288
21605 205/300 Flush
21606 206/300 Write block 3682 len 20480 data: 4 4 3 3 3
21607 207/300 Write block 4238 len 28672 data: 7 7 6 5 3 5 6
21608 208/300 Read block 2897 len 24576
21609 209/300 Write block 2958 len 4096 data: 3
21610 210/300 Write block 2551 len 28672 data: 5 5 5 5 4 4 4
21611 211/300 Write block 1764 len 16384 data: 7 7 5 5
21612 212/300 Write block 4298 len 16384 data: 5 5 6 5
21613 213/300 Read block 3682 len 40960
21614 214/300 Flush
21615 215/300 Read block 780 len 16384
21616 216/300 Read block 4079 len 36864
21617 217/300 Read block 1444 len 24576
21618 218/300 Write block 351 len 36864 data: 4 3 5 5 4 3 3 3 3
21619 219/300 Read block 3612 len 28672
21620 220/300 Flush
21621 221/300 Write block 289 len 28672 data: 3 3 5 5 7 9 9
21622 222/300 Write block 4751 len 32768 data: 6 6 4 4 4 4 4 4
21623 223/300 Read block 655 len 36864
21624 224/300 Write block 1039 len 20480 data: 7 9 9 7 7
21625 225/300 Read block 4093 len 16384
21626 226/300 Read block 4059 len 20480
21627 227/300 Read block 3631 len 36864
21628 228/300 Flush
21629 229/300 Read block 1473 len 12288
21630 230/300 Read block 2579 len 32768
21631 231/300 Write block 633 len 4096 data: 5
21632 232/300 Write block 1563 len 20480 data: 6 5 6 6 3
21633 233/300 Write block 1951 len 24576 data: 5 5 5 5 4 4
21634 234/300 Read block 1041 len 24576
21635 235/300 Read block 4195 len 36864
21636 236/300 Write block 3337 len 40960 data: 2 3 4 2 2 2 3 3 4 3
21637 237/300 Read block 3588 len 12288
21638 238/300 Read block 4830 len 28672
21639 239/300 Read block 4861 len 36864
21640 240/300 Read block 3976 len 4096
21641 241/300 Write block 322 len 16384 data: 6 6 4 4
21642 242/300 Read block 1572 len 24576
21643 243/300 Write block 2536 len 8192 data: 6 6
21644 244/300 Write block 548 len 32768 data: 3 3 3 2 3 2 3 3
21645 245/300 Write block 1204 len 16384 data: 7 7 5 4
21646 246/300 Write block 3913 len 40960 data: 6 5 5 5 5 5 5 4 5 6
21647 247/300 Write block 496 len 24576 data: 8 7 8 7 6 5
21648 248/300 Read block 1155 len 4096
21649 249/300 Read block 4815 len 20480
21650 250/300 Read block 4429 len 16384
21651 251/300 Write block 2906 len 16384 data: 6 5 5 3
21652 252/300 Read block 2689 len 36864
21653 253/300 Read block 1152 len 32768
21654 254/300 Write block 14 len 24576 data: 4 5 4 4 3 3
21655 255/300 Write block 1517 len 20480 data: 8 11 9 9 8
21656 256/300 Flush
21657 257/300 Read block 4587 len 12288
21658 258/300 Write block 846 len 20480 data: 8 7 6 5 5
21659 259/300 Read block 4830 len 8192
21660 260/300 Read block 4194 len 8192
21661 261/300 Write block 3271 len 8192 data: 5 6
21662 262/300 Read block 3783 len 8192
21663 263/300 Read block 4066 len 16384
21664 264/300 Read block 2128 len 32768
21665 265/300 Read block 3347 len 4096
21666 266/300 Write block 4731 len 28672 data: 7 6 4 6 6 7 8
21667 267/300 Read block 2466 len 40960
21668 268/300 Write block 3362 len 20480 data: 6 6 6 6 7
21669 269/300 Write block 613 len 16384 data: 6 6 6 5
21670 270/300 Write block 1088 len 40960 data: 6 6 5 6 5 5 6 4 4 3
21671 271/300 Write block 4127 len 40960 data: 6 6 7 6 4 5 4 4 5 5
21672 272/300 Write block 3528 len 24576 data: 7 7 6 6 7 7
21673 273/300 Flush
21674 274/300 Flush
21675 275/300 Write block 2285 len 24576 data: 5 6 5 4 4 4
21676 276/300 Read block 4635 len 8192
21677 277/300 Write block 3028 len 40960 data: 4 4 4 4 4 5 5 8 7 8
21678 278/300 Flush
21679 279/300 Write block 4165 len 24576 data: 7 6 8 7 7 7
21680 280/300 Flush
21681 281/300 Write block 3042 len 28672 data: 8 7 7 5 5 6 6
21682 282/300 Write block 2500 len 28672 data: 6 6 5 5 4 4 4
21683 283/300 Write block 1547 len 24576 data: 6 6 6 5 6 5
21684 284/300 Read block 4418 len 32768
21685 285/300 Read block 3281 len 32768
21686 286/300 Write block 2390 len 20480 data: 6 7 7 6 6
21687 287/300 Read block 1326 len 36864
21688 288/300 Read block 3013 len 20480
21689 289/300 Write block 2741 len 4096 data: 5
21690 290/300 Flush
21691 291/300 Flush
21692 292/300 Write block 3975 len 8192 data: 4 4
21693 293/300 Flush
21694 294/300 Write block 261 len 4096 data: 4
21695 295/300 Write block 649 len 12288 data: 3 4 5
21696 296/300 Read block 343 len 36864
21697 297/300 Read block 2365 len 20480
21698 298/300 Flush
21699 299/300 Write block 1214 len 28672 data: 5 7 7 7 7 6 7
21700 300/300 Read block 4121 len 20480
21701 Main task has work to do, go find it
21702 got dsc Start(2)
21703 start 2
21704 Feb 28 22:17:37.076 INFO request completed, latency_us: 208, response_code: 204, uri: /start/cid/2, method: POST, req_id: 1e372fe3-7700-4ad0-bc8f-b370ad9109f8, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
21705 Replay: started 2, returned:Ok(())
21706 ----------------------------------------------------------------
21707 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21708 States: Active Active Offline
21709 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
21710 7235 Acked 8234 Read 3 Done Done New true
21711 7236 Acked 8235 Read 10 Done Done New true
21712 7237 Acked 8236 Write 1 Done Done New false
21713 7238 Acked 8237 Write 3 Done Done New false
21714 7239 Acked 8238 Read 4 Done Done New false
21715 7240 Acked 8239 Write 3 Done Done New false
21716 7241 Acked 8240 Write 1 Done Done New false
21717 7242 Acked 8241 Write 8 Done Done New false
21718 7243 Acked 8242 Write 8 Done Done New false
21719 7244 Acked 8243 Read 4 Done Done New false
21720 7245 Acked 8244 Read 3 Done Done New false
21721 7246 Acked 8245 Write 1 Done Done New false
21722 7247 Acked 8246 Flush 0 Done Done New false
21723 7248 Acked 8247 Flush 0 Done Done New false
21724 7249 Acked 8248 Write 5 Done Done New false
21725 7250 Acked 8249 Write 2 Done Done New false
21726 7251 Acked 8250 Write 9 Done Done New false
21727 7252 Acked 8251 Write 7 Done Done New false
21728 7253 Acked 8252 Write 8 Done Done New false
21729 7254 Acked 8253 Write 2 Done Done New false
21730 7255 Acked 8254 Read 4 Done Done New false
21731 7256 Acked 8255 Read 1 Done Done New false
21732 7257 Acked 8256 Write 7 Done Done New false
21733 7258 Acked 8257 Flush 0 Done Done New false
21734 7259 Acked 8258 Write 7 Done Done New false
21735 7260 Acked 8259 Read 9 Done Done New false
21736 7261 Acked 8260 Write 5 Done Done New false
21737 7262 Acked 8261 Read 7 Done Done New false
21738 7263 Acked 8262 Read 10 Done Done New false
21739 7264 Acked 8263 Read 6 Done Done New false
21740 7265 Acked 8264 Read 3 Done Done New false
21741 7266 Acked 8265 Read 4 Done Done New false
21742 7267 Acked 8266 Read 2 Done Done New false
21743 7268 Acked 8267 Write 8 Done Done New false
21744 7269 Acked 8268 Read 1 Done Done New false
21745 7270 Acked 8269 Read 10 Done Done New false
21746 7271 Acked 8270 Write 1 Done Done New false
21747 7272 Acked 8271 Read 3 Done Done New false
21748 7273 Acked 8272 Read 9 Done Done New false
21749 7274 Acked 8273 Read 8 Done Done New false
21750 7275 Acked 8274 Read 7 Done Done New false
21751 7276 Acked 8275 Read 1 Done Done New false
21752 7277 Acked 8276 Read 6 Done Done New false
21753 7278 Acked 8277 Write 7 Done Done New false
21754 7279 Acked 8278 Flush 0 Done Done New false
21755 7280 Acked 8279 Read 3 Done Done New false
21756 7281 Acked 8280 Read 2 Done Done New false
21757 7282 Acked 8281 Read 10 Done Done New false
21758 7283 Acked 8282 Flush 0 Done Done New false
21759 7284 Acked 8283 Read 4 Done Done New false
21760 7285 Acked 8284 Flush 0 Done Done New false
21761 7286 Acked 8285 Write 10 Done Done New false
21762 7287 Acked 8286 Read 2 Done Done New false
21763 7288 Acked 8287 Read 5 Done Done New false
21764 7289 Acked 8288 Write 8 Done Done New false
21765 7290 Acked 8289 Write 10 Done Done New false
21766 7291 Acked 8290 Write 3 Done Done New false
21767 7292 Acked 8291 Read 4 Done Done New false
21768 7293 Acked 8292 Write 5 Done Done New false
21769 7294 Acked 8293 Write 2 Done Done New false
21770 7295 Acked 8294 Read 2 Done Done New false
21771 7296 Acked 8295 Write 4 Done Done New false
21772 7297 Acked 8296 Read 4 Done Done New false
21773 7298 Acked 8297 Read 10 Done Done New false
21774 7299 Acked 8298 Read 3 Done Done New false
21775 7300 Acked 8299 Write 9 Done Done New false
21776 7301 Acked 8300 Read 2 Done Done New false
21777 7302 Acked 8301 Read 8 Done Done New false
21778 7303 Acked 8302 Write 9 Done Done New false
21779 7304 Acked 8303 Read 1 Done Done New false
21780 7305 Acked 8304 Flush 0 Done Done New false
21781 7306 Acked 8305 Read 4 Done Done New false
21782 7307 Acked 8306 Write 3 Done Done New false
21783 7308 Acked 8307 Read 2 Done Done New false
21784 7309 Acked 8308 Write 1 Done Done New false
21785 7310 Acked 8309 Read 4 Done Done New false
21786 7311 Acked 8310 Write 9 Done Done New false
21787 7312 Acked 8311 Read 4 Done Done New false
21788 7313 Acked 8312 Write 8 Done Done New false
21789 7314 Acked 8313 Write 8 Done Done New false
21790 7315 Acked 8314 Read 5 Done Done New false
21791 7316 Acked 8315 Read 10 Done Done New false
21792 7317 Acked 8316 Read 10 Done Done New false
21793 7318 Acked 8317 Read 9 Done Done New false
21794 7319 Acked 8318 Read 7 Done Done New false
21795 7320 Acked 8319 Write 3 Done Done New false
21796 7321 Acked 8320 Write 2 Done Done New false
21797 7322 Acked 8321 Write 1 Done Done New false
21798 7323 Acked 8322 Write 10 Done Done New false
21799 7324 Acked 8323 Write 3 Done Done New false
21800 7325 Acked 8324 Write 9 Done Done New false
21801 7326 Acked 8325 Flush 0 Done Done New false
21802 7327 Acked 8326 Read 10 Done Done New false
21803 7328 Acked 8327 Write 6 Done Done New false
21804 7329 Acked 8328 Read 8 Done Done New false
21805 7330 Acked 8329 Read 3 Done Done New false
21806 7331 Acked 8330 Read 5 Done Done New false
21807 7332 Acked 8331 Flush 0 Done Done New false
21808 7333 Acked 8332 Write 10 Done Done New false
21809 7334 Acked 8333 Write 2 Done Done New false
21810 7335 Acked 8334 Write 3 Done Done New false
21811 7336 Acked 8335 Read 5 Done Done New false
21812 7337 Acked 8336 Read 2 Done Done New false
21813 7338 Acked 8337 Read 10 Done Done New false
21814 7339 Acked 8338 Write 4 Done Done New false
21815 7340 Acked 8339 Read 4 Done Done New false
21816 7341 Acked 8340 Write 1 Done Done New false
21817 7342 Acked 8341 Write 4 Done Done New false
21818 7343 Acked 8342 Write 3 Done Done New false
21819 7344 Acked 8343 Read 3 Done Done New false
21820 7345 Acked 8344 Read 7 Done Done New false
21821 7346 Acked 8345 Write 9 Done Done New false
21822 7347 Acked 8346 Flush 0 Done Done New false
21823 7348 Acked 8347 Write 6 Done Done New false
21824 7349 Acked 8348 Write 10 Done Done New false
21825 7350 Acked 8349 Write 1 Done Done New false
21826 7351 Acked 8350 Write 7 Done Done New false
21827 7352 Acked 8351 Read 2 Done Done New false
21828 7353 Acked 8352 Read 3 Done Done New false
21829 7354 Acked 8353 Read 1 Done Done New false
21830 7355 Acked 8354 Read 2 Done Done New false
21831 7356 Acked 8355 Read 4 Done Done New false
21832 7357 Acked 8356 Flush 0 Done Done New false
21833 7358 Acked 8357 Read 3 Done Done New false
21834 7359 Acked 8358 Read 7 Done Done New false
21835 7360 Acked 8359 Read 1 Done Done New false
21836 7361 Acked 8360 Read 10 Done Done New false
21837 7362 Acked 8361 Read 3 Done Done New false
21838 7363 Acked 8362 Read 2 Done Done New false
21839 7364 Acked 8363 Read 1 Done Done New false
21840 7365 Acked 8364 Read 1 Done Done New false
21841 7366 Acked 8365 Flush 0 Done Done New false
21842 7367 Acked 8366 Flush 0 Done Done New false
21843 7368 Acked 8367 Write 1 Done Done New false
21844 7369 Acked 8368 Write 1 Done Done New false
21845 7370 Acked 8369 Read 7 Done Done New false
21846 7371 Acked 8370 Write 10 Done Done New false
21847 7372 Acked 8371 Flush 0 Done Done New false
21848 7373 Acked 8372 Flush 0 Done Done New false
21849 7374 Acked 8373 Write 10 Done Done New false
21850 7375 Acked 8374 Read 9 Done Done New false
21851 7376 Acked 8375 Read 10 Done Done New false
21852 7377 Acked 8376 Read 9 Done Done New false
21853 7378 Acked 8377 Read 10 Done Done New false
21854 7379 Acked 8378 Read 3 Done Done New false
21855 7380 Acked 8379 Write 2 Done Done New false
21856 7381 Acked 8380 Read 7 Done Done New false
21857 7382 Acked 8381 Write 7 Done Done New false
21858 7383 Acked 8382 Flush 0 Done Done New false
21859 7384 Acked 8383 Write 8 Done Done New false
21860 7385 Acked 8384 Flush 0 Done Done New false
21861 7386 Acked 8385 Write 10 Done Done New false
21862 7387 Acked 8386 Write 2 Done Done New false
21863 7388 Acked 8387 Write 5 Done Done New false
21864 7389 Acked 8388 Read 3 Done Done New false
21865 7390 Acked 8389 Read 10 Done Done New false
21866 7391 Acked 8390 Read 1 Done Done New false
21867 7392 Acked 8391 Read 3 Done Done New false
21868 7393 Acked 8392 Read 6 Done Done New false
21869 7394 Acked 8393 Write 6 Done Done New false
21870 7395 Acked 8394 Write 9 Done Done New false
21871 7396 Acked 8395 Write 9 Done Done New false
21872 7397 Acked 8396 Write 10 Done Done New false
21873 7398 Acked 8397 Read 2 Done Done New false
21874 7399 Acked 8398 Read 2 Done Done New false
21875 7400 Acked 8399 Write 10 Done Done New false
21876 7401 Acked 8400 Write 4 Done Done New false
21877 7402 Acked 8401 Write 7 Done Done New false
21878 7403 Acked 8402 Flush 0 Done Done New false
21879 7404 Acked 8403 Read 4 Done Done New false
21880 7405 Acked 8404 Write 8 Done Done New false
21881 7406 Acked 8405 Read 5 Done Done New false
21882 7407 Acked 8406 Write 5 Done Done New false
21883 7408 Acked 8407 Write 5 Done Done New false
21884 7409 Acked 8408 Read 8 Done Done New false
21885 7410 Acked 8409 Write 3 Done Done New false
21886 7411 Acked 8410 Read 6 Done Done New false
21887 7412 Acked 8411 Write 5 Done Done New false
21888 7413 Acked 8412 Read 1 Done Done New false
21889 7414 Acked 8413 Flush 0 Done Done New false
21890 7415 Acked 8414 Flush 0 Done Done New false
21891 7416 Acked 8415 Read 10 Done Done New false
21892 7417 Acked 8416 Write 3 Done Done New false
21893 7418 Acked 8417 Write 4 Done Done New false
21894 7419 Acked 8418 Read 6 Done Done New false
21895 7420 Acked 8419 Write 7 Done Done New false
21896 7421 Acked 8420 Read 4 Done Done New false
21897 7422 Acked 8421 Read 9 Done Done New false
21898 7423 Acked 8422 Flush 0 Done Done New false
21899 7424 Acked 8423 Read 6 Done Done New false
21900 7425 Acked 8424 Read 2 Done Done New false
21901 7426 Acked 8425 Write 6 Done Done New false
21902 7427 Acked 8426 Write 2 Done Done New false
21903 7428 Acked 8427 Read 6 Done Done New false
21904 7429 Acked 8428 Write 2 Done Done New false
21905 7430 Acked 8429 Read 4 Done Done New false
21906 7431 Acked 8430 Read 5 Done Done New false
21907 7432 Acked 8431 Read 7 Done Done New false
21908 7433 Acked 8432 Read 3 Done Done New false
21909 7434 Acked 8433 Read 6 Done Done New false
21910 7435 Acked 8434 Write 6 Done Done New false
21911 7436 Acked 8435 Read 3 Done Done New false
21912 7437 Acked 8436 Write 6 Done Done New false
21913 7438 Acked 8437 Write 2 Done Done New false
21914 7439 Acked 8438 Read 3 Done Done New false
21915 7440 Acked 8439 Flush 0 Done Done New false
21916 7441 Acked 8440 Write 5 Done Done New false
21917 7442 Acked 8441 Write 7 Done Done New false
21918 7443 Acked 8442 Read 6 Done Done New false
21919 7444 Acked 8443 Write 1 Done Done New false
21920 7445 Acked 8444 Write 7 Done Done New false
21921 7446 Acked 8445 Write 4 Done Done New false
21922 7447 Acked 8446 Write 4 Done Done New false
21923 7448 Acked 8447 Read 10 Done Done New false
21924 7449 Acked 8448 Flush 0 Done Done New false
21925 7450 Acked 8449 Read 4 Done Done New false
21926 7451 Acked 8450 Read 9 Done Done New false
21927 7452 Acked 8451 Read 6 Done Done New false
21928 7453 Acked 8452 Write 9 Done Done New false
21929 7454 Acked 8453 Read 7 Done Done New false
21930 7455 Acked 8454 Flush 0 Done Done New false
21931 7456 Acked 8455 Write 7 Done Done New false
21932 7457 Acked 8456 Write 8 Done Done New false
21933 7458 Acked 8457 Read 9 Done Done New false
21934 7459 Acked 8458 Write 5 Done Done New false
21935 7460 Acked 8459 Read 4 Done Done New false
21936 7461 Acked 8460 Read 5 Done Done New false
21937 7462 Acked 8461 Read 9 Done Done New false
21938 7463 Acked 8462 Flush 0 Done Done New false
21939 7464 Acked 8463 Read 3 Done Done New false
21940 7465 Acked 8464 Read 8 Done Done New false
21941 7466 Acked 8465 Write 1 Done Done New false
21942 7467 Acked 8466 Write 5 Done Done New false
21943 7468 Acked 8467 Write 6 Done Done New false
21944 7469 Acked 8468 Read 6 Done Done New false
21945 7470 Acked 8469 Read 9 Done Done New false
21946 7471 Acked 8470 Write 10 Done Done New false
21947 7472 Acked 8471 Read 3 Done Done New false
21948 7473 Acked 8472 Read 7 Done Done New false
21949 7474 Acked 8473 Read 9 Done Done New false
21950 7475 Acked 8474 Read 1 Done Done New false
21951 7476 Acked 8475 Write 4 Done Done New false
21952 7477 Acked 8476 Read 6 Done Done New false
21953 7478 Acked 8477 Write 2 Done Done New false
21954 7479 Acked 8478 Write 8 Done Done New false
21955 7480 Acked 8479 Write 4 Done Done New false
21956 7481 Acked 8480 Write 10 Done Done New false
21957 7482 Acked 8481 Write 6 Done Done New false
21958 7483 Acked 8482 Read 1 Done Done New false
21959 7484 Acked 8483 Read 5 Done Done New false
21960 7485 Acked 8484 Read 4 Done Done New false
21961 7486 Acked 8485 Write 4 Done Done New false
21962 7487 Acked 8486 Read 9 Done Done New false
21963 7488 Acked 8487 Read 8 Done Done New false
21964 7489 Acked 8488 Write 6 Done Done New false
21965 7490 Acked 8489 Write 5 Done Done New false
21966 7491 Acked 8490 Flush 0 Done Done New false
21967 7492 Acked 8491 Read 3 Done Done New false
21968 7493 Acked 8492 Write 5 Done Done New false
21969 7494 Acked 8493 Read 2 Done Done New false
21970 7495 Acked 8494 Read 2 Done Done New false
21971 7496 Acked 8495 Write 2 Done Done New false
21972 7497 Acked 8496 Read 2 Done Done New false
21973 7498 Acked 8497 Read 4 Done Done New false
21974 7499 Acked 8498 Read 8 Done Done New false
21975 7500 Acked 8499 Read 1 Done Done New false
21976 7501 Acked 8500 Write 7 Done Done New false
21977 7502 Acked 8501 Read 10 Done Done New false
21978 7503 Acked 8502 Write 5 Done Done New false
21979 7504 Acked 8503 Write 4 Done Done New false
21980 7505 Acked 8504 Write 10 Done Done New false
21981 7506 Acked 8505 Write 10 Done Done New false
21982 7507 Acked 8506 Write 6 Done Done New false
21983 7508 Acked 8507 Flush 0 Done Done New false
21984 7509 Acked 8508 Flush 0 Done Done New false
21985 7510 Acked 8509 Write 6 Done Done New false
21986 7511 Acked 8510 Read 2 Done Done New false
21987 7512 Acked 8511 Write 10 Done Done New false
21988 7513 Acked 8512 Flush 0 Done Done New false
21989 7514 Acked 8513 Write 6 Done Done New false
21990 7515 Acked 8514 Flush 0 Done Done New false
21991 7516 Acked 8515 Write 7 Done Done New false
21992 7517 Acked 8516 Write 7 Done Done New false
21993 7518 Acked 8517 Write 6 Done Done New false
21994 7519 Acked 8518 Read 8 Done Done New false
21995 7520 Acked 8519 Read 8 Done Done New false
21996 7521 Acked 8520 Write 5 Done Done New false
21997 7522 Acked 8521 Read 9 Done Done New false
21998 7523 Acked 8522 Read 5 Done Done New false
21999 7524 Acked 8523 Write 1 Done Done New false
22000 7525 Acked 8524 Flush 0 Done Done New false
22001 7526 Acked 8525 Flush 0 Done Done New false
22002 7527 Acked 8526 Write 2 Done Done New false
22003 7528 Acked 8527 Flush 0 Done Done New false
22004 7529 Acked 8528 Write 1 Done Done New false
22005 7530 Acked 8529 Write 3 Done Done New false
22006 7531 Acked 8530 Read 9 Done Done New false
22007 7532 Acked 8531 Read 5 Done Done New false
22008 7533 Acked 8532 Flush 0 Done Done New false
22009 7534 Acked 8533 Write 7 Done Done New false
22010 7535 Acked 8534 Read 5 Done Done New false
22011 STATES DS:0 DS:1 DS:2 TOTAL
22012 New 0 0 301 301
22013 Sent 0 0 0 0
22014 Done 301 301 0 602
22015 Skipped 0 0 0 0
22016 Error 0 0 0 0
22017 Last Flush: 8532 8532 8233
22018 Downstairs last five completed: 8233 8232 8231 8230 8229
22019 Upstairs last five completed: 7535 7534 7533 7532 7531
22020 CLIENT: Up:0 ds:301 act:2
220212024-02-28T22:17:37.759ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220222024-02-28T22:17:37.760ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220232024-02-28T22:17:37.760ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220242024-02-28T22:17:37.760ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220252024-02-28T22:17:37.760ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220262024-02-28T22:17:37.760ZINFOcrucible: [2] client re-new 302 jobs since flush 8233 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220272024-02-28T22:17:37.760ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22028 [8830] got start action
22029 [8830] I am going to restart
22030 Starting downstairs at port 8830
22031 Make output file at "/tmp/dsc/downstairs-8830.txt"
22032 [8830][2] reports Starting
220332024-02-28T22:17:38.761ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220342024-02-28T22:17:38.761ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220352024-02-28T22:17:38.761ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220362024-02-28T22:17:38.761ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220372024-02-28T22:17:38.761ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220382024-02-28T22:17:38.761ZINFOcrucible: [2] client re-new 302 jobs since flush 8233 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220392024-02-28T22:17:38.761ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22040 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1189)
22041 [8830][2] reports Running
220422024-02-28T22:17:39.762ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220432024-02-28T22:17:39.762ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220442024-02-28T22:17:39.764ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220452024-02-28T22:17:39.764ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220462024-02-28T22:17:39.764ZINFOcrucible: send last flush ID to this DS: 8233 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220472024-02-28T22:17:39.764ZINFOcrucible: Replied this last flush ID: 8233 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220482024-02-28T22:17:39.764ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220492024-02-28T22:17:39.764ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220502024-02-28T22:17:39.764ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220512024-02-28T22:17:39.764ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22052 ----------------------------------------------------------------
22053 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22054 Downstairs last five completed: 8535 8534 8533 8532 8531
22055 Upstairs last five completed: 7536 7535 7534 7533 7532
22056 CLIENT: Up:0 ds:0 act:3
22057 Replay: All jobs finished, all DS active.
22058 Main task has work to do, go find it
22059 got dsc Stop(2)
22060 stop 2
22061 Feb 28 22:17:41.079 INFO request completed[8830] Got stop action so:false kr:false
22062 , latency_us: 356, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 5088e611-a8a9-4f3f-9cb4-4b8105f28798, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
220632024-02-28T22:17:41.080ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220642024-02-28T22:17:41.080ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220652024-02-28T22:17:41.080ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220662024-02-28T22:17:41.080ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22067 Feb 28 22:17:41.080 INFO request completed{"msg":"[2] client re-new 0 jobs since flush 8535","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:17:41.080587326Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"downstairs","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
22068 , latency_us: 193, response_code: 200{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:17:41.080626704Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"io task","client":"2","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
22069 , uri: /state/cid/2, method: GET, req_id: f45a6abb-800b-438a-b1ce-93f041fb4269, remote_addr: 127.0.0.1:41656, local_addr: 127.0.0.1:9998
22070 [8830] Exited with: ExitStatus(unix_wait_status(9))
22071 [8830][2] reports Exit
220722024-02-28T22:17:42.081ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220732024-02-28T22:17:42.081ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220742024-02-28T22:17:42.081ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220752024-02-28T22:17:42.081ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220762024-02-28T22:17:42.081ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220772024-02-28T22:17:42.081ZINFOcrucible: [2] client re-new 0 jobs since flush 8535 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220782024-02-28T22:17:42.081ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220792024-02-28T22:17:43.082ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220802024-02-28T22:17:43.082ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220812024-02-28T22:17:43.082ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220822024-02-28T22:17:43.082ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220832024-02-28T22:17:43.082ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220842024-02-28T22:17:43.082ZINFOcrucible: [2] client re-new 0 jobs since flush 8535 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220852024-02-28T22:17:43.082ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220862024-02-28T22:17:44.083ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220872024-02-28T22:17:44.083ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220882024-02-28T22:17:44.083ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220892024-02-28T22:17:44.084ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220902024-02-28T22:17:44.084ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220912024-02-28T22:17:44.084ZINFOcrucible: [2] client re-new 0 jobs since flush 8535 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220922024-02-28T22:17:44.084ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22093 Feb 28 22:17:45.082 INFO request completed, latency_us: 348, response_code: 200, uri: /state/cid/2, method: GET, req_id: 43a0e7be-f681-4c62-9fba-0d6d813103a8, remote_addr: 127.0.0.1:41656, local_addr: 127.0.0.1:9998
22094 001/300 Read block 4129 len 32768
22095 002/300 Read block 4663 len 24576
220962024-02-28T22:17:45.085ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220972024-02-28T22:17:45.085ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220982024-02-28T22:17:45.085ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
220992024-02-28T22:17:45.085ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
221002024-02-28T22:17:45.085ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
221012024-02-28T22:17:45.085ZINFOcrucible: [2] client re-new 2 jobs since flush 8535 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
221022024-02-28T22:17:45.085ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22103 003/300 Read block 4751 len 36864
22104 004/300 Read block 3912 len 28672
22105 005/300 Flush
22106 006/300 Read block 3414 len 36864
22107 007/300 Write block 1697 len 20480 data: 2 3 5 4 5
22108 008/300 Read block 4381 len 20480
22109 009/300 Read block 3105 len 28672
22110 010/300 Read block 3220 len 8192
22111 011/300 Write block 2466 len 32768 data: 6 7 7 8 8 7 8 7
22112 012/300 Flush
22113 013/300 Read block 342 len 40960
22114 014/300 Write block 2275 len 12288 data: 4 4 5
22115 015/300 Read block 782 len 28672
22116 016/300 Write block 2411 len 36864 data: 7 9 9 9 10 8 9 8 8
22117 017/300 Write block 2684 len 8192 data: 9 9
22118 018/300 Write block 2291 len 20480 data: 3 5 6 6 6
22119 019/300 Write block 1333 len 24576 data: 6 7 5 4 3 3
22120 020/300 Write block 572 len 16384 data: 6 6 6 8
22121 021/300 Read block 4404 len 12288
22122 022/300 Read block 3958 len 32768
22123 023/300 Read block 4670 len 32768
22124 024/300 Read block 4763 len 8192
22125 025/300 Read block 1170 len 36864
22126 026/300 Read block 4209 len 40960
22127 027/300 Write block 4823 len 8192 data: 4 4
22128 028/300 Write block 3787 len 8192 data: 3 4
22129 029/300 Write block 530 len 24576 data: 4 5 6 4 7 7
22130 030/300 Write block 2405 len 4096 data: 3
22131 031/300 Read block 4687 len 32768
22132 032/300 Write block 3037 len 12288 data: 9 7 7
22133 033/300 Write block 1476 len 24576 data: 9 8 8 5 5 5
22134 034/300 Read block 3265 len 20480
22135 035/300 Flush
22136 036/300 Write block 1395 len 36864 data: 6 5 5 4 5 5 4 4 7
22137 037/300 Read block 1261 len 24576
22138 038/300 Write block 1464 len 28672 data: 3 4 4 4 4 5 8
22139 039/300 Write block 836 len 24576 data: 4 3 3 4 4 3
22140 040/300 Write block 1180 len 24576 data: 5 6 6 5 5 5
22141 041/300 Read block 3871 len 40960
22142 042/300 Read block 2463 len 12288
22143 043/300 Write block 3381 len 28672 data: 6 7 6 4 2 2 4
22144 044/300 Read block 994 len 12288
22145 045/300 Flush
22146 046/300 Flush
22147 047/300 Write block 774 len 24576 data: 6 6 7 8 9 8
22148 048/300 Write block 3356 len 28672 data: 4 6 8 8 7 7 7
22149 049/300 Write block 3323 len 4096 data: 4
22150 050/300 Write block 4055 len 12288 data: 3 5 5
22151 051/300 Read block 1939 len 4096
22152 052/300 Write block 1449 len 24576 data: 3 4 5 5 6 5
22153 053/300 Write block 3056 len 12288 data: 6 4 6
22154 054/300 Read block 3837 len 36864
22155 055/300 Write block 1289 len 8192 data: 4 4
22156 056/300 Write block 2620 len 32768 data: 7 7 6 5 3 3 2 2
22157 057/300 Read block 2741 len 36864
22158 058/300 Write block 4242 len 32768 data: 4 6 7 6 5 5 5 6
22159 059/300 Write block 2498 len 28672 data: 7 6 7 7 6 6 5
22160 060/300 Read block 1295 len 36864
22161 061/300 Read block 2195 len 40960
22162 062/300 Read block 3437 len 20480
22163 063/300 Read block 1676 len 24576
22164 064/300 Flush
22165 065/300 Write block 2977 len 40960 data: 8 7 5 5 4 4 5 6 6 8
22166 066/300 Read block 2126 len 32768
22167 067/300 Write block 2082 len 20480 data: 5 6 6 5 4
22168 068/300 Read block 428 len 12288
22169 069/300 Read block 2254 len 28672
22170 070/300 Write block 2850 len 36864 data: 4 5 4 6 6 6 7 7 6
22171 071/300 Read block 3811 len 4096
22172 072/300 Read block 4103 len 32768
22173 073/300 Write block 3722 len 20480 data: 7 5 6 8 9
22174 074/300 Read block 1202 len 4096
22175 075/300 Write block 3526 len 32768 data: 6 7 8 8 7 7 8 8
22176 076/300 Read block 91 len 8192
22177 077/300 Write block 1843 len 16384 data: 5 5 6 6
22178 078/300 Read block 586 len 4096
22179 079/300 Read block 4889 len 8192
22180 080/300 Write block 2165 len 36864 data: 5 4 6 6 5 3 4 5 5
22181 081/300 Read block 122 len 28672
22182 082/300 Flush
22183 083/300 Read block 1641 len 8192
22184 084/300 Read block 2506 len 20480
22185 085/300 Read block 2246 len 32768
22186 086/300 Flush
22187 087/300 Read block 3260 len 20480
22188 088/300 Read block 326 len 16384
22189 089/300 Flush
22190 090/300 Read block 176 len 20480
22191 091/300 Write block 1823 len 8192 data: 2 3
22192 092/300 Write block 1896 len 24576 data: 7 7 6 9 10 9
22193 093/300 Read block 1027 len 36864
22194 094/300 Write block 1742 len 12288 data: 4 4 3
22195 095/300 Write block 2719 len 12288 data: 9 9 9
22196 096/300 Write block 331 len 12288 data: 3 3 4
22197 097/300 Read block 3271 len 16384
22198 098/300 Write block 2393 len 40960 data: 7 7 6 5 5 4 4 4 4 5
22199 099/300 Write block 978 len 32768 data: 6 8 7 8 7 4 3 5
22200 100/300 Write block 2743 len 24576 data: 4 4 4 4 3 4
22201 101/300 Read block 966 len 20480
22202 102/300 Read block 1962 len 8192
22203 103/300 Read block 2160 len 24576
22204 104/300 Read block 2001 len 40960
22205 105/300 Read block 1848 len 20480
22206 106/300 Write block 4921 len 40960 data: 4 6 6 7 7 7 7 7 7 6
22207 107/300 Read block 1742 len 24576
22208 108/300 Read block 4457 len 36864
22209 109/300 Read block 1504 len 24576
22210 110/300 Read block 2109 len 12288
22211 111/300 Write block 2013 len 12288 data: 5 5 4
22212 112/300 Flush
22213 113/300 Read block 3963 len 28672
22214 114/300 Flush
22215 115/300 Read block 4581 len 8192
22216 116/300 Write block 3808 len 36864 data: 3 2 2 3 3 4 4 7 5
22217 117/300 Write block 4543 len 28672 data: 8 10 9 8 9 7 7
22218 118/300 Write block 3607 len 36864 data: 4 4 3 3 3 4 5 5 4
22219 119/300 Read block 4698 len 24576
22220 120/300 Read block 3536 len 32768
22221 121/300 Read block 4223 len 16384
22222 122/300 Write block 4234 len 16384 data: 5 5 4 5
22223 123/300 Read block 2093 len 36864
22224 124/300 Write block 2600 len 24576 data: 6 5 7 6 7 7
22225 125/300 Write block 503 len 20480 data: 3 3 7 8 8
22226 126/300 Read block 3763 len 8192
22227 127/300 Read block 2268 len 12288
22228 128/300 Write block 1131 len 8192 data: 8 6
22229 129/300 Read block 2588 len 40960
22230 130/300 Write block 3517 len 24576 data: 7 7 6 6 7 5
22231 131/300 Write block 549 len 4096 data: 4
22232 132/300 Read block 4433 len 16384
22233 133/300 Read block 3071 len 28672
22234 134/300 Read block 2618 len 8192
22235 135/300 Read block 4708 len 40960
22236 136/300 Write block 579 len 8192 data: 4 5
22237 137/300 Write block 2657 len 24576 data: 3 3 3 4 4 3
22238 138/300 Write block 2539 len 36864 data: 7 7 8 8 7 7 6 5 3
22239 139/300 Flush
22240 140/300 Read block 2832 len 20480
22241 141/300 Write block 1494 len 36864 data: 6 5 4 4 4 3 3 3 3
22242 142/300 Write block 3424 len 40960 data: 5 5 4 4 5 4 6 6 8 8
22243 143/300 Read block 359 len 36864
22244 144/300 Read block 3681 len 4096
22245 145/300 Read block 1193 len 40960
22246 146/300 Write block 754 len 28672 data: 9 7 7 10 8 9 7
22247 147/300 Write block 2185 len 12288 data: 6 6 6
22248 148/300 Read block 4156 len 20480
22249 149/300 Read block 3465 len 28672
22250 150/300 Read block 2036 len 4096
22251 151/300 Read block 4066 len 16384
22252 152/300 Write block 3180 len 28672 data: 7 6 6 4 4 4 4
22253 153/300 Write block 2247 len 36864 data: 4 4 3 3 2 3 3 4 3
22254 154/300 Write block 2766 len 36864 data: 4 4 5 5 5 5 6 5 4
22255 155/300 Read block 4166 len 32768
22256 156/300 Write block 4211 len 32768 data: 6 6 6 5 5 5 6 4
22257 157/300 Read block 1574 len 28672
22258 158/300 Flush
22259 159/300 Read block 1090 len 36864
22260 160/300 Write block 1315 len 16384 data: 5 6 6 7
22261 161/300 Read block 3968 len 24576
22262 162/300 Read block 1166 len 8192
22263 163/300 Read block 2235 len 24576
22264 164/300 Read block 4446 len 36864
22265 165/300 Read block 4420 len 16384
22266 166/300 Read block 2287 len 12288
22267 167/300 Read block 4778 len 40960
22268 168/300 Read block 2260 len 16384
22269 169/300 Read block 2206 len 20480
22270 170/300 Read block 3157 len 12288
22271 171/300 Flush
22272 172/300 Write block 1895 len 4096 data: 5
22273 173/300 Write block 4483 len 4096 data: 6
22274 174/300 Write block 3928 len 32768 data: 5 4 3 3 2 2 3 4
22275 175/300 Flush
22276 176/300 Write block 4814 len 16384 data: 5 6 5 4
22277 177/300 Write block 2444 len 24576 data: 5 4 4 4 4 5
22278 178/300 Read block 3687 len 28672
22279 179/300 Read block 801 len 8192
22280 180/300 Flush
22281 181/300 Read block 4823 len 4096
22282 182/300 Read block 1268 len 4096
22283 183/300 Write block 1843 len 36864 data: 6 6 7 7 4 4 3 2 5
22284 184/300 Read block 4264 len 24576
22285 185/300 Read block 330 len 12288
22286 186/300 Write block 4591 len 24576 data: 7 6 4 4 4 5
22287 187/300 Read block 3325 len 32768
22288 188/300 Write block 1564 len 24576 data: 6 7 7 4 2 3
22289 189/300 Write block 905 len 16384 data: 5 4 5 6
22290 190/300 Read block 2530 len 12288
22291 191/300 Write block 2562 len 20480 data: 3 6 4 4 3
22292 192/300 Write block 3571 len 24576 data: 5 7 9 7 7 8
22293 193/300 Write block 2228 len 24576 data: 2 2 4 4 4 3
22294 194/300 Read block 3632 len 40960
22295 195/300 Write block 693 len 40960 data: 6 5 4 4 3 3 5 5 5 4
22296 196/300 Read block 4295 len 8192
22297 197/300 Read block 2745 len 24576
22298 198/300 Write block 1134 len 32768 data: 9 8 7 7 7 7 9 10
22299 199/300 Read block 525 len 16384
22300 200/300 Read block 2276 len 16384
22301 201/300 Write block 4889 len 4096 data: 7
22302 202/300 Write block 2601 len 40960 data: 6 8 7 8 8 7 5 6 6 7
22303 203/300 Flush
22304 204/300 Write block 3267 len 8192 data: 7 6
22305 205/300 Write block 1230 len 28672 data: 7 6 6 6 6 6 5
22306 206/300 Flush
22307 207/300 Read block 3924 len 36864
22308 208/300 Read block 1865 len 16384
22309 209/300 Read block 517 len 16384
22310 210/300 Read block 669 len 24576
22311 211/300 Read block 263 len 16384
22312 212/300 Read block 2910 len 36864
22313 213/300 Write block 2113 len 28672 data: 5 6 5 6 6 5 5
22314 214/300 Read block 1515 len 8192
22315 215/300 Flush
22316 216/300 Write block 2507 len 36864 data: 5 3 3 3 2 3 3 4 6
22317 217/300 Write block 1172 len 12288 data: 6 5 3
22318 218/300 Write block 3684 len 4096 data: 4
22319 219/300 Write block 3131 len 16384 data: 5 5 5 4
22320 220/300 Read block 1265 len 36864
22321 221/300 Write block 1372 len 20480 data: 4 4 3 3 3
22322 222/300 Read block 4372 len 24576
22323 223/300 Read block 848 len 36864
22324 224/300 Read block 3891 len 40960
22325 225/300 Write block 4503 len 12288 data: 7 8 7
22326 226/300 Read block 1407 len 4096
22327 227/300 Read block 2849 len 36864
22328 228/300 Read block 2827 len 16384
22329 229/300 Write block 4246 len 32768 data: 6 6 6 7 6 7 8 8
22330 230/300 Read block 2548 len 8192
22331 231/300 Write block 2227 len 40960 data: 3 3 3 5 5 5 4 4 4 6
22332 232/300 Write block 3277 len 20480 data: 5 5 5 6 6
22333 233/300 Read block 4744 len 32768
22334 234/300 Read block 306 len 8192
22335 235/300 Read block 2834 len 12288
22336 236/300 Read block 3570 len 16384
22337 237/300 Flush
22338 238/300 Read block 3812 len 16384
22339 239/300 Write block 2681 len 28672 data: 8 8 8 10 10 7 8
22340 240/300 Read block 4297 len 16384
22341 241/300 Read block 444 len 40960
22342 242/300 Write block 2671 len 40960 data: 7 7 8 7 5 6 9 10 10 9
22343 243/300 Write block 4238 len 20480 data: 8 8 7 6 5
22344 244/300 Write block 3724 len 32768 data: 7 9 10 9 9 10 9 4
22345 245/300 Read block 4523 len 32768
22346 246/300 Read block 986 len 28672
22347 247/300 Read block 2607 len 4096
22348 248/300 Write block 4016 len 16384 data: 10 6 7 9
22349 249/300 Flush
22350 250/300 Write block 4670 len 4096 data: 6
22351 251/300 Write block 2857 len 36864 data: 8 7 6 6 4 5 6 5 5
22352 252/300 Read block 2678 len 12288
22353 253/300 Write block 4942 len 20480 data: 7 7 6 6 5
22354 254/300 Write block 1997 len 16384 data: 4 4 4 3
22355 255/300 Read block 848 len 12288
22356 256/300 Read block 4708 len 40960
22357 257/300 Read block 4249 len 12288
22358 258/300 Write block 667 len 20480 data: 4 5 5 8 9
22359 259/300 Write block 690 len 40960 data: 7 6 7 7 6 5 5 4 4 6
22360 260/300 Read block 1792 len 4096
22361 261/300 Read block 2484 len 8192
22362 262/300 Write block 3438 len 16384 data: 6 6 8 6
22363 263/300 Read block 3783 len 12288
22364 264/300 Read block 2386 len 12288
22365 265/300 Write block 2544 len 4096 data: 8
22366 266/300 Read block 366 len 36864
22367 267/300 Read block 2392 len 16384
22368 268/300 Write block 3835 len 4096 data: 4
22369 269/300 Write block 3115 len 40960 data: 6 7 6 7 6 7 7 9 7 7
22370 270/300 Write block 3896 len 28672 data: 6 5 5 6 4 5 5
22371 271/300 Read block 4102 len 16384
22372 272/300 Write block 400 len 28672 data: 5 3 3 4 4 5 5
22373 273/300 Read block 4654 len 8192
22374 274/300 Write block 2399 len 28672 data: 5 5 5 6 4 3 4
22375 275/300 Read block 2299 len 20480
22376 276/300 Flush
22377 277/300 Flush
22378 278/300 Read block 4378 len 40960
22379 279/300 Flush
22380 280/300 Read block 2504 len 24576
22381 281/300 Read block 864 len 8192
22382 282/300 Flush
22383 283/300 Read block 217 len 4096
22384 284/300 Read block 208 len 16384
22385 285/300 Read block 435 len 12288
22386 286/300 Flush
22387 287/300 Read block 4370 len 36864
22388 288/300 Read block 501 len 40960
22389 289/300 Read block 4346 len 12288
22390 290/300 Write block 1270 len 36864 data: 5 5 5 7 9 10 7 7 6
22391 291/300 Read block 2760 len 4096
22392 292/300 Write block 2261 len 28672 data: 4 4 4 4 4 4 6
22393 293/300 Flush
22394 294/300 Read block 1944 len 20480
22395 295/300 Read block 1253 len 32768
22396 296/300 Write block 3674 len 32768 data: 3 3 2 3 3 4 4 5
22397 297/300 Flush
22398 298/300 Write block 4105 len 8192 data: 7 8
22399 299/300 Read block 842 len 32768
22400 300/300 Read block 1251 len 40960
22401 Main task has work to do, go find it
22402 got dsc Start(2)
22403 start 2
22404 Feb 28 22:17:45.419 INFO request completed, latency_us: 221, response_code: 204, uri: /start/cid/2, method: POST, req_id: fbcc7f9d-bc99-4839-8144-d535a4c8a842, remote_addr: 127.0.0.1:41656, local_addr: 127.0.0.1:9998
22405 Replay: started 2, returned:Ok(())
22406 ----------------------------------------------------------------
22407 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22408 States: Active Active Offline
22409 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
22410 7537 Acked 8536 Read 8 Done Done New true
22411 7538 Acked 8537 Read 6 Done Done New true
22412 7539 Acked 8538 Read 9 Done Done New false
22413 7540 Acked 8539 Read 7 Done Done New false
22414 7541 Acked 8540 Flush 0 Done Done New false
22415 7542 Acked 8541 Read 9 Done Done New false
22416 7543 Acked 8542 Write 5 Done Done New false
22417 7544 Acked 8543 Read 5 Done Done New false
22418 7545 Acked 8544 Read 7 Done Done New false
22419 7546 Acked 8545 Read 2 Done Done New false
22420 7547 Acked 8546 Write 8 Done Done New false
22421 7548 Acked 8547 Flush 0 Done Done New false
22422 7549 Acked 8548 Read 10 Done Done New false
22423 7550 Acked 8549 Write 3 Done Done New false
22424 7551 Acked 8550 Read 7 Done Done New false
22425 7552 Acked 8551 Write 9 Done Done New false
22426 7553 Acked 8552 Write 2 Done Done New false
22427 7554 Acked 8553 Write 5 Done Done New false
22428 7555 Acked 8554 Write 6 Done Done New false
22429 7556 Acked 8555 Write 4 Done Done New false
22430 7557 Acked 8556 Read 3 Done Done New false
22431 7558 Acked 8557 Read 8 Done Done New false
22432 7559 Acked 8558 Read 8 Done Done New false
22433 7560 Acked 8559 Read 2 Done Done New false
22434 7561 Acked 8560 Read 9 Done Done New false
22435 7562 Acked 8561 Read 10 Done Done New false
22436 7563 Acked 8562 Write 2 Done Done New false
22437 7564 Acked 8563 Write 2 Done Done New false
22438 7565 Acked 8564 Write 6 Done Done New false
22439 7566 Acked 8565 Write 1 Done Done New false
22440 7567 Acked 8566 Read 8 Done Done New false
22441 7568 Acked 8567 Write 3 Done Done New false
22442 7569 Acked 8568 Write 6 Done Done New false
22443 7570 Acked 8569 Read 5 Done Done New false
22444 7571 Acked 8570 Flush 0 Done Done New false
22445 7572 Acked 8571 Write 9 Done Done New false
22446 7573 Acked 8572 Read 6 Done Done New false
22447 7574 Acked 8573 Write 7 Done Done New false
22448 7575 Acked 8574 Write 6 Done Done New false
22449 7576 Acked 8575 Write 6 Done Done New false
22450 7577 Acked 8576 Read 10 Done Done New false
22451 7578 Acked 8577 Read 3 Done Done New false
22452 7579 Acked 8578 Write 7 Done Done New false
22453 7580 Acked 8579 Read 3 Done Done New false
22454 7581 Acked 8580 Flush 0 Done Done New false
22455 7582 Acked 8581 Flush 0 Done Done New false
22456 7583 Acked 8582 Write 6 Done Done New false
22457 7584 Acked 8583 Write 7 Done Done New false
22458 7585 Acked 8584 Write 1 Done Done New false
22459 7586 Acked 8585 Write 3 Done Done New false
22460 7587 Acked 8586 Read 1 Done Done New false
22461 7588 Acked 8587 Write 6 Done Done New false
22462 7589 Acked 8588 Write 3 Done Done New false
22463 7590 Acked 8589 Read 9 Done Done New false
22464 7591 Acked 8590 Write 2 Done Done New false
22465 7592 Acked 8591 Write 8 Done Done New false
22466 7593 Acked 8592 Read 9 Done Done New false
22467 7594 Acked 8593 Write 8 Done Done New false
22468 7595 Acked 8594 Write 7 Done Done New false
22469 7596 Acked 8595 Read 9 Done Done New false
22470 7597 Acked 8596 Read 10 Done Done New false
22471 7598 Acked 8597 Read 5 Done Done New false
22472 7599 Acked 8598 Read 6 Done Done New false
22473 7600 Acked 8599 Flush 0 Done Done New false
22474 7601 Acked 8600 Write 10 Done Done New false
22475 7602 Acked 8601 Read 8 Done Done New false
22476 7603 Acked 8602 Write 5 Done Done New false
22477 7604 Acked 8603 Read 3 Done Done New false
22478 7605 Acked 8604 Read 7 Done Done New false
22479 7606 Acked 8605 Write 9 Done Done New false
22480 7607 Acked 8606 Read 1 Done Done New false
22481 7608 Acked 8607 Read 8 Done Done New false
22482 7609 Acked 8608 Write 5 Done Done New false
22483 7610 Acked 8609 Read 1 Done Done New false
22484 7611 Acked 8610 Write 8 Done Done New false
22485 7612 Acked 8611 Read 2 Done Done New false
22486 7613 Acked 8612 Write 4 Done Done New false
22487 7614 Acked 8613 Read 1 Done Done New false
22488 7615 Acked 8614 Read 2 Done Done New false
22489 7616 Acked 8615 Write 9 Done Done New false
22490 7617 Acked 8616 Read 7 Done Done New false
22491 7618 Acked 8617 Flush 0 Done Done New false
22492 7619 Acked 8618 Read 2 Done Done New false
22493 7620 Acked 8619 Read 5 Done Done New false
22494 7621 Acked 8620 Read 8 Done Done New false
22495 7622 Acked 8621 Flush 0 Done Done New false
22496 7623 Acked 8622 Read 5 Done Done New false
22497 7624 Acked 8623 Read 4 Done Done New false
22498 7625 Acked 8624 Flush 0 Done Done New false
22499 7626 Acked 8625 Read 5 Done Done New false
22500 7627 Acked 8626 Write 2 Done Done New false
22501 7628 Acked 8627 Write 6 Done Done New false
22502 7629 Acked 8628 Read 9 Done Done New false
22503 7630 Acked 8629 Write 3 Done Done New false
22504 7631 Acked 8630 Write 3 Done Done New false
22505 7632 Acked 8631 Write 3 Done Done New false
22506 7633 Acked 8632 Read 4 Done Done New false
22507 7634 Acked 8633 Write 10 Done Done New false
22508 7635 Acked 8634 Write 8 Done Done New false
22509 7636 Acked 8635 Write 6 Done Done New false
22510 7637 Acked 8636 Read 5 Done Done New false
22511 7638 Acked 8637 Read 2 Done Done New false
22512 7639 Acked 8638 Read 6 Done Done New false
22513 7640 Acked 8639 Read 10 Done Done New false
22514 7641 Acked 8640 Read 5 Done Done New false
22515 7642 Acked 8641 Write 10 Done Done New false
22516 7643 Acked 8642 Read 6 Done Done New false
22517 7644 Acked 8643 Read 9 Done Done New false
22518 7645 Acked 8644 Read 6 Done Done New false
22519 7646 Acked 8645 Read 3 Done Done New false
22520 7647 Acked 8646 Write 3 Done Done New false
22521 7648 Acked 8647 Flush 0 Done Done New false
22522 7649 Acked 8648 Read 7 Done Done New false
22523 7650 Acked 8649 Flush 0 Done Done New false
22524 7651 Acked 8650 Read 2 Done Done New false
22525 7652 Acked 8651 Write 9 Done Done New false
22526 7653 Acked 8652 Write 7 Done Done New false
22527 7654 Acked 8653 Write 9 Done Done New false
22528 7655 Acked 8654 Read 6 Done Done New false
22529 7656 Acked 8655 Read 8 Done Done New false
22530 7657 Acked 8656 Read 4 Done Done New false
22531 7658 Acked 8657 Write 4 Done Done New false
22532 7659 Acked 8658 Read 9 Done Done New false
22533 7660 Acked 8659 Write 6 Done Done New false
22534 7661 Acked 8660 Write 5 Done Done New false
22535 7662 Acked 8661 Read 2 Done Done New false
22536 7663 Acked 8662 Read 3 Done Done New false
22537 7664 Acked 8663 Write 2 Done Done New false
22538 7665 Acked 8664 Read 10 Done Done New false
22539 7666 Acked 8665 Write 6 Done Done New false
22540 7667 Acked 8666 Write 1 Done Done New false
22541 7668 Acked 8667 Read 4 Done Done New false
22542 7669 Acked 8668 Read 7 Done Done New false
22543 7670 Acked 8669 Read 2 Done Done New false
22544 7671 Acked 8670 Read 10 Done Done New false
22545 7672 Acked 8671 Write 2 Done Done New false
22546 7673 Acked 8672 Write 6 Done Done New false
22547 7674 Acked 8673 Write 9 Done Done New false
22548 7675 Acked 8674 Flush 0 Done Done New false
22549 7676 Acked 8675 Read 5 Done Done New false
22550 7677 Acked 8676 Write 9 Done Done New false
22551 7678 Acked 8677 Write 10 Done Done New false
22552 7679 Acked 8678 Read 9 Done Done New false
22553 7680 Acked 8679 Read 1 Done Done New false
22554 7681 Acked 8680 Read 10 Done Done New false
22555 7682 Acked 8681 Write 7 Done Done New false
22556 7683 Acked 8682 Write 3 Done Done New false
22557 7684 Acked 8683 Read 5 Done Done New false
22558 7685 Acked 8684 Read 7 Done Done New false
22559 7686 Acked 8685 Read 1 Done Done New false
22560 7687 Acked 8686 Read 4 Done Done New false
22561 7688 Acked 8687 Write 7 Done Done New false
22562 7689 Acked 8688 Write 9 Done Done New false
22563 7690 Acked 8689 Write 9 Done Done New false
22564 7691 Acked 8690 Read 8 Done Done New false
22565 7692 Acked 8691 Write 8 Done Done New false
22566 7693 Acked 8692 Read 7 Done Done New false
22567 7694 Acked 8693 Flush 0 Done Done New false
22568 7695 Acked 8694 Read 9 Done Done New false
22569 7696 Acked 8695 Write 4 Done Done New false
22570 7697 Acked 8696 Read 6 Done Done New false
22571 7698 Acked 8697 Read 2 Done Done New false
22572 7699 Acked 8698 Read 6 Done Done New false
22573 7700 Acked 8699 Read 9 Done Done New false
22574 7701 Acked 8700 Read 4 Done Done New false
22575 7702 Acked 8701 Read 3 Done Done New false
22576 7703 Acked 8702 Read 10 Done Done New false
22577 7704 Acked 8703 Read 4 Done Done New false
22578 7705 Acked 8704 Read 5 Done Done New false
22579 7706 Acked 8705 Read 3 Done Done New false
22580 7707 Acked 8706 Flush 0 Done Done New false
22581 7708 Acked 8707 Write 1 Done Done New false
22582 7709 Acked 8708 Write 1 Done Done New false
22583 7710 Acked 8709 Write 8 Done Done New false
22584 7711 Acked 8710 Flush 0 Done Done New false
22585 7712 Acked 8711 Write 4 Done Done New false
22586 7713 Acked 8712 Write 6 Done Done New false
22587 7714 Acked 8713 Read 7 Done Done New false
22588 7715 Acked 8714 Read 2 Done Done New false
22589 7716 Acked 8715 Flush 0 Done Done New false
22590 7717 Acked 8716 Read 1 Done Done New false
22591 7718 Acked 8717 Read 1 Done Done New false
22592 7719 Acked 8718 Write 9 Done Done New false
22593 7720 Acked 8719 Read 6 Done Done New false
22594 7721 Acked 8720 Read 3 Done Done New false
22595 7722 Acked 8721 Write 6 Done Done New false
22596 7723 Acked 8722 Read 8 Done Done New false
22597 7724 Acked 8723 Write 6 Done Done New false
22598 7725 Acked 8724 Write 4 Done Done New false
22599 7726 Acked 8725 Read 3 Done Done New false
22600 7727 Acked 8726 Write 5 Done Done New false
22601 7728 Acked 8727 Write 6 Done Done New false
22602 7729 Acked 8728 Write 6 Done Done New false
22603 7730 Acked 8729 Read 10 Done Done New false
22604 7731 Acked 8730 Write 10 Done Done New false
22605 7732 Acked 8731 Read 2 Done Done New false
22606 7733 Acked 8732 Read 6 Done Done New false
22607 7734 Acked 8733 Write 8 Done Done New false
22608 7735 Acked 8734 Read 4 Done Done New false
22609 7736 Acked 8735 Read 4 Done Done New false
22610 7737 Acked 8736 Write 1 Done Done New false
22611 7738 Acked 8737 Write 10 Done Done New false
22612 7739 Acked 8738 Flush 0 Done Done New false
22613 7740 Acked 8739 Write 2 Done Done New false
22614 7741 Acked 8740 Write 7 Done Done New false
22615 7742 Acked 8741 Flush 0 Done Done New false
22616 7743 Acked 8742 Read 9 Done Done New false
22617 7744 Acked 8743 Read 4 Done Done New false
22618 7745 Acked 8744 Read 4 Done Done New false
22619 7746 Acked 8745 Read 6 Done Done New false
22620 7747 Acked 8746 Read 4 Done Done New false
22621 7748 Acked 8747 Read 9 Done Done New false
22622 7749 Acked 8748 Write 7 Done Done New false
22623 7750 Acked 8749 Read 2 Done Done New false
22624 7751 Acked 8750 Flush 0 Done Done New false
22625 7752 Acked 8751 Write 9 Done Done New false
22626 7753 Acked 8752 Write 3 Done Done New false
22627 7754 Acked 8753 Write 1 Done Done New false
22628 7755 Acked 8754 Write 4 Done Done New false
22629 7756 Acked 8755 Read 9 Done Done New false
22630 7757 Acked 8756 Write 5 Done Done New false
22631 7758 Acked 8757 Read 6 Done Done New false
22632 7759 Acked 8758 Read 9 Done Done New false
22633 7760 Acked 8759 Read 10 Done Done New false
22634 7761 Acked 8760 Write 3 Done Done New false
22635 7762 Acked 8761 Read 1 Done Done New false
22636 7763 Acked 8762 Read 9 Done Done New false
22637 7764 Acked 8763 Read 4 Done Done New false
22638 7765 Acked 8764 Write 8 Done Done New false
22639 7766 Acked 8765 Read 2 Done Done New false
22640 7767 Acked 8766 Write 10 Done Done New false
22641 7768 Acked 8767 Write 5 Done Done New false
22642 7769 Acked 8768 Read 8 Done Done New false
22643 7770 Acked 8769 Read 2 Done Done New false
22644 7771 Acked 8770 Read 3 Done Done New false
22645 7772 Acked 8771 Read 4 Done Done New false
22646 7773 Acked 8772 Flush 0 Done Done New false
22647 7774 Acked 8773 Read 4 Done Done New false
22648 7775 Acked 8774 Write 7 Done Done New false
22649 7776 Acked 8775 Read 4 Done Done New false
22650 7777 Acked 8776 Read 10 Done Done New false
22651 7778 Acked 8777 Write 10 Done Done New false
22652 7779 Acked 8778 Write 5 Done Done New false
22653 7780 Acked 8779 Write 8 Done Done New false
22654 7781 Acked 8780 Read 8 Done Done New false
22655 7782 Acked 8781 Read 7 Done Done New false
22656 7783 Acked 8782 Read 1 Done Done New false
22657 7784 Acked 8783 Write 4 Done Done New false
22658 7785 Acked 8784 Flush 0 Done Done New false
22659 7786 Acked 8785 Write 1 Done Done New false
22660 7787 Acked 8786 Write 9 Done Done New false
22661 7788 Acked 8787 Read 3 Done Done New false
22662 7789 Acked 8788 Write 5 Done Done New false
22663 7790 Acked 8789 Write 4 Done Done New false
22664 7791 Acked 8790 Read 3 Done Done New false
22665 7792 Acked 8791 Read 10 Done Done New false
22666 7793 Acked 8792 Read 3 Done Done New false
22667 7794 Acked 8793 Write 5 Done Done New false
22668 7795 Acked 8794 Write 10 Done Done New false
22669 7796 Acked 8795 Read 1 Done Done New false
22670 7797 Acked 8796 Read 2 Done Done New false
22671 7798 Acked 8797 Write 4 Done Done New false
22672 7799 Acked 8798 Read 3 Done Done New false
22673 7800 Acked 8799 Read 3 Done Done New false
22674 7801 Acked 8800 Write 1 Done Done New false
22675 7802 Acked 8801 Flush 0 Done Done New false
22676 7803 Acked 8802 Read 9 Done Done New false
22677 7804 Acked 8803 Read 4 Done Done New false
22678 7805 Acked 8804 Write 1 Done Done New false
22679 7806 Acked 8805 Write 10 Done Done New false
22680 7807 Acked 8806 Write 7 Done Done New false
22681 7808 Acked 8807 Read 4 Done Done New false
22682 7809 Acked 8808 Write 7 Done Done New false
22683 7810 Acked 8809 Read 2 Done Done New false
22684 7811 Acked 8810 Write 7 Done Done New false
22685 7812 Acked 8811 Read 5 Done Done New false
22686 7813 Acked 8812 Flush 0 Done Done New false
22687 7814 Acked 8813 Flush 0 Done Done New false
22688 7815 Acked 8814 Read 10 Done Done New false
22689 7816 Acked 8815 Flush 0 Done Done New false
22690 7817 Acked 8816 Read 6 Done Done New false
22691 7818 Acked 8817 Read 2 Done Done New false
22692 7819 Acked 8818 Flush 0 Done Done New false
22693 7820 Acked 8819 Read 1 Done Done New false
22694 7821 Acked 8820 Read 4 Done Done New false
22695 7822 Acked 8821 Read 3 Done Done New false
22696 7823 Acked 8822 Flush 0 Done Done New false
22697 7824 Acked 8823 Read 9 Done Done New false
22698 7825 Acked 8824 Read 10 Done Done New false
22699 7826 Acked 8825 Read 3 Done Done New false
22700 7827 Acked 8826 Write 9 Done Done New false
22701 7828 Acked 8827 Read 1 Done Done New false
22702 7829 Acked 8828 Write 7 Done Done New false
22703 7830 Acked 8829 Flush 0 Done Done New false
22704 7831 Acked 8830 Read 5 Done Done New false
22705 7832 Acked 8831 Read 8 Done Done New false
22706 7833 Acked 8832 Write 8 Done Done New false
22707 7834 Acked 8833 Flush 0 Done Done New false
22708 7835 Acked 8834 Write 2 Done Done New false
22709 7836 Acked 8835 Read 8 Done Done New false
22710 7837 Acked 8836 Read 10 Done Done New false
22711 STATES DS:0 DS:1 DS:2 TOTAL
22712 New 0 0 301 301
22713 Sent 0 0 0 0
22714 Done 301 301 0 602
22715 Skipped 0 0 0 0
22716 Error 0 0 0 0
22717 Last Flush: 8833 8833 8535
22718 Downstairs last five completed: 8535 8534 8533 8532 8531
22719 Upstairs last five completed: 7837 7836 7835 7834 7833
22720 CLIENT: Up:0 ds:301 act:2
227212024-02-28T22:17:46.086ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227222024-02-28T22:17:46.086ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227232024-02-28T22:17:46.086ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227242024-02-28T22:17:46.086ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227252024-02-28T22:17:46.086ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227262024-02-28T22:17:46.086ZINFOcrucible: [2] client re-new 302 jobs since flush 8535 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227272024-02-28T22:17:46.086ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227282024-02-28T22:17:47.087ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227292024-02-28T22:17:47.087ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227302024-02-28T22:17:47.087ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227312024-02-28T22:17:47.087ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227322024-02-28T22:17:47.087ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227332024-02-28T22:17:47.087ZINFOcrucible: [2] client re-new 302 jobs since flush 8535 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227342024-02-28T22:17:47.087ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227352024-02-28T22:17:48.088ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227362024-02-28T22:17:48.088ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227372024-02-28T22:17:48.088ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227382024-02-28T22:17:48.088ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227392024-02-28T22:17:48.088ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227402024-02-28T22:17:48.088ZINFOcrucible: [2] client re-new 302 jobs since flush 8535 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227412024-02-28T22:17:48.088ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22742 [8830] got start action
22743 [8830] I am going to restart
22744 Starting downstairs at port 8830
22745 Make output file at "/tmp/dsc/downstairs-8830.txt"
22746 [8830][2] reports Starting
227472024-02-28T22:17:49.089ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227482024-02-28T22:17:49.089ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227492024-02-28T22:17:49.089ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227502024-02-28T22:17:49.089ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227512024-02-28T22:17:49.090ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227522024-02-28T22:17:49.090ZINFOcrucible: [2] client re-new 302 jobs since flush 8535 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
227532024-02-28T22:17:49.090ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
22754 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1190)
22755 [8830][2] reports Running
22756 ----------------------------------------------------------------
22757 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
22758 States: Active Active Offline
22759 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
22760 7537 Acked 8536 Read 8 Done Done New true
22761 7538 Acked 8537 Read 6 Done Done New true
22762 7539 Acked 8538 Read 9 Done Done New true
22763 7540 Acked 8539 Read 7 Done Done New true
22764 7541 Acked 8540 Flush 0 Done Done New true
22765 7542 Acked 8541 Read 9 Done Done New true
22766 7543 Acked 8542 Write 5 Done Done New true
22767 7544 Acked 8543 Read 5 Done Done New true
22768 7545 Acked 8544 Read 7 Done Done New true
22769 7546 Acked 8545 Read 2 Done Done New true
22770 7547 Acked 8546 Write 8 Done Done New true
22771 7548 Acked 8547 Flush 0 Done Done New true
22772 7549 Acked 8548 Read 10 Done Done New true
22773 7550 Acked 8549 Write 3 Done Done New true
22774 7551 Acked 8550 Read 7 Done Done New true
22775 7552 Acked 8551 Write 9 Done Done New true
22776 7553 Acked 8552 Write 2 Done Done New true
22777 7554 Acked 8553 Write 5 Done Done New true
22778 7555 Acked 8554 Write 6 Done Done New true
22779 7556 Acked 8555 Write 4 Done Done New true
22780 7557 Acked 8556 Read 3 Done Done New true
22781 7558 Acked 8557 Read 8 Done Done New true
22782 7559 Acked 8558 Read 8 Done Done New true
22783 7560 Acked 8559 Read 2 Done Done New true
22784 7561 Acked 8560 Read 9 Done Done New true
22785 7562 Acked 8561 Read 10 Done Done New true
22786 7563 Acked 8562 Write 2 Done Done New true
22787 7564 Acked 8563 Write 2 Done Done New true
22788 7565 Acked 8564 Write 6 Done Done New true
22789 7566 Acked 8565 Write 1 Done Done New true
22790 7567 Acked 8566 Read 8 Done Done New true
22791 7568 Acked 8567 Write 3 Done Done New true
22792 7569 Acked 8568 Write 6 Done Done New true
22793 7570 Acked 8569 Read 5 Done Done New true
22794 7571 Acked 8570 Flush 0 Done Done New true
22795 7572 Acked 8571 Write 9 Done Done New true
22796 7573 Acked 8572 Read 6 Done Done New true
22797 7574 Acked 8573 Write 7 Done Done New true
22798 7575 Acked 8574 Write 6 Done Done New true
22799 7576 Acked 8575 Write 6 Done Done New true
22800 7577 Acked 8576 Read 10 Done Done New true
22801 7578 Acked 8577 Read 3 Done Done New true
22802 7579 Acked 8578 Write 7 Done Done New true
22803 7580 Acked 8579 Read 3 Done Done New true
22804 7581 Acked 8580 Flush 0 Done Done New true
22805 7582 Acked 8581 Flush 0 Done Done New true
22806 7583 Acked 8582 Write 6 Done Done New true
22807 7584 Acked 8583 Write 7 Done Done New true
22808 7585 Acked 8584 Write 1 Done Done New true
22809 7586 Acked 8585 Write 3 Done Done New true
22810 7587 Acked 8586 Read 1 Done Done New true
22811 7588 Acked 8587 Write 6 Done Done New true
22812 7589 Acked 8588 Write 3 Done Done New true
22813 7590 Acked 8589 Read 9 Done Done New true
22814 7591 Acked 8590 Write 2 Done Done New true
22815 7592 Acked 8591 Write 8 Done Done New true
22816 7593 Acked 8592 Read 9 Done Done New true
22817 7594 Acked 8593 Write 8 Done Done New true
22818 7595 Acked 8594 Write 7 Done Done New true
22819 7596 Acked 8595 Read 9 Done Done New true
22820 7597 Acked 8596 Read 10 Done Done New true
22821 7598 Acked 8597 Read 5 Done Done New true
22822 7599 Acked 8598 Read 6 Done Done New true
22823 7600 Acked 8599 Flush 0 Done Done New true
22824 7601 Acked 8600 Write 10 Done Done New true
22825 7602 Acked 8601 Read 8 Done Done New true
22826 7603 Acked 8602 Write 5 Done Done New true
22827 7604 Acked 8603 Read 3 Done Done New true
22828 7605 Acked 8604 Read 7 Done Done New true
22829 7606 Acked 8605 Write 9 Done Done New true
22830 7607 Acked 8606 Read 1 Done Done New true
22831 7608 Acked 8607 Read 8 Done Done New true
22832 7609 Acked 8608 Write 5 Done Done New true
22833 7610 Acked 8609 Read 1 Done Done New true
22834 7611 Acked 8610 Write 8 Done Done New true
22835 7612 Acked 8611 Read 2 Done Done New true
22836 7613 Acked 8612 Write 4 Done Done New true
22837 7614 Acked 8613 Read 1 Done Done New true
22838 7615 Acked 8614 Read 2 Done Done New true
22839 7616 Acked 8615 Write 9 Done Done New true
22840 7617 Acked 8616 Read 7 Done Done New true
22841 7618 Acked 8617 Flush 0 Done Done New true
22842 7619 Acked 8618 Read 2 Done Done New true
22843 7620 Acked 8619 Read 5 Done Done New true
22844 7621 Acked 8620 Read 8 Done Done New true
22845 7622 Acked 8621 Flush 0 Done Done New true
22846 7623 Acked 8622 Read 5 Done Done New true
22847 7624 Acked 8623 Read 4 Done Done New true
22848 7625 Acked 8624 Flush 0 Done Done New true
22849 7626 Acked 8625 Read 5 Done Done New true
22850 7627 Acked 8626 Write 2 Done Done New true
22851 7628 Acked 8627 Write 6 Done Done New true
22852 7629 Acked 8628 Read 9 Done Done New true
22853 7630 Acked 8629 Write 3 Done Done New true
22854 7631 Acked 8630 Write 3 Done Done New true
22855 7632 Acked 8631 Write 3 Done Done New true
22856 7633 Acked 8632 Read 4 Done Done New true
22857 7634 Acked 8633 Write 10 Done Done New true
22858 7635 Acked 8634 Write 8 Done Done New true
22859 7636 Acked 8635 Write 6 Done Done New true
22860 7637 Acked 8636 Read 5 Done Done New true
22861 7638 Acked 8637 Read 2 Done Done New true
22862 7639 Acked 8638 Read 6 Done Done New true
22863 7640 Acked 8639 Read 10 Done Done New true
22864 7641 Acked 8640 Read 5 Done Done New true
22865 7642 Acked 8641 Write 10 Done Done New true
22866 7643 Acked 8642 Read 6 Done Done New true
22867 7644 Acked 8643 Read 9 Done Done New true
22868 7645 Acked 8644 Read 6 Done Done New true
22869 7646 Acked 8645 Read 3 Done Done New true
22870 7647 Acked 8646 Write 3 Done Done New true
22871 7648 Acked 8647 Flush 0 Done Done New true
22872 7649 Acked 8648 Read 7 Done Done New true
22873 7650 Acked 8649 Flush 0 Done Done New true
22874 7651 Acked 8650 Read 2 Done Done New true
22875 7652 Acked 8651 Write 9 Done Done New true
22876 7653 Acked 8652 Write 7 Done Done New true
22877 7654 Acked 8653 Write 9 Done Done New true
22878 7655 Acked 8654 Read 6 Done Done New true
22879 7656 Acked 8655 Read 8 Done Done New true
22880 7657 Acked 8656 Read 4 Done Done New true
22881 7658 Acked 8657 Write 4 Done Done New true
22882 7659 Acked 8658 Read 9 Done Done New true
22883 7660 Acked 8659 Write 6 Done Done New true
22884 7661 Acked 8660 Write 5 Done Done New true
22885 7662 Acked 8661 Read 2 Done Done New true
22886 7663 Acked 8662 Read 3 Done Done New true
22887 7664 Acked 8663 Write 2 Done Done New true
22888 7665 Acked 8664 Read 10 Done Done New true
22889 7666 Acked 8665 Write 6 Done Done New true
22890 7667 Acked 8666 Write 1 Done Done New true
22891 7668 Acked 8667 Read 4 Done Done New true
22892 7669 Acked 8668 Read 7 Done Done New true
22893 7670 Acked 8669 Read 2 Done Done New true
22894 7671 Acked 8670 Read 10 Done Done New true
22895 7672 Acked 8671 Write 2 Done Done New true
22896 7673 Acked 8672 Write 6 Done Done New true
22897 7674 Acked 8673 Write 9 Done Done New true
22898 7675 Acked 8674 Flush 0 Done Done New true
22899 7676 Acked 8675 Read 5 Done Done New true
22900 7677 Acked 8676 Write 9 Done Done New true
22901 7678 Acked 8677 Write 10 Done Done New true
22902 7679 Acked 8678 Read 9 Done Done New true
22903 7680 Acked 8679 Read 1 Done Done New true
22904 7681 Acked 8680 Read 10 Done Done New true
22905 7682 Acked 8681 Write 7 Done Done New true
22906 7683 Acked 8682 Write 3 Done Done New true
22907 7684 Acked 8683 Read 5 Done Done New true
22908 7685 Acked 8684 Read 7 Done Done New true
22909 7686 Acked 8685 Read 1 Done Done New true
22910 7687 Acked 8686 Read 4 Done Done New true
22911 7688 Acked 8687 Write 7 Done Done New true
22912 7689 Acked 8688 Write 9 Done Done New true
22913 7690 Acked 8689 Write 9 Done Done New true
22914 7691 Acked 8690 Read 8 Done Done New true
22915 7692 Acked 8691 Write 8 Done Done New true
22916 7693 Acked 8692 Read 7 Done Done New true
22917 7694 Acked 8693 Flush 0 Done Done New true
22918 7695 Acked 8694 Read 9 Done Done New true
22919 7696 Acked 8695 Write 4 Done Done New true
22920 7697 Acked 8696 Read 6 Done Done New true
22921 7698 Acked 8697 Read 2 Done Done New true
22922 7699 Acked 8698 Read 6 Done Done New true
22923 7700 Acked 8699 Read 9 Done Done New true
22924 7701 Acked 8700 Read 4 Done Done New true
22925 7702 Acked 8701 Read 3 Done Done New true
22926 7703 Acked 8702 Read 10 Done Done New true
22927 7704 Acked 8703 Read 4 Done Done New true
22928 7705 Acked 8704 Read 5 Done Done New true
22929 7706 Acked 8705 Read 3 Done Done New true
22930 7707 Acked 8706 Flush 0 Done Done New true
22931 7708 Acked 8707 Write 1 Done Done New true
22932 7709 Acked 8708 Write 1 Done Done New true
22933 7710 Acked 8709 Write 8 Done Done New true
22934 7711 Acked 8710 Flush 0 Done Done New true
22935 7712 Acked 8711 Write 4 Done Done New true
22936 7713 Acked 8712 Write 6 Done Done New true
22937 7714 Acked 8713 Read 7 Done Done New true
22938 7715 Acked 8714 Read 2 Done Done New true
22939 7716 Acked 8715 Flush 0 Done Done New true
22940 7717 Acked 8716 Read 1 Done Done New true
22941 7718 Acked 8717 Read 1 Done Done New true
22942 7719 Acked 8718 Write 9 Done Done New true
22943 7720 Acked 8719 Read 6 Done Done New true
22944 7721 Acked 8720 Read 3 Done Done New true
22945 7722 Acked 8721 Write 6 Done Done New true
22946 7723 Acked 8722 Read 8 Done Done New true
22947 7724 Acked 8723 Write 6 Done Done New true
22948 7725 Acked 8724 Write 4 Done Done New true
22949 7726 Acked 8725 Read 3 Done Done New true
22950 7727 Acked 8726 Write 5 Done Done New true
22951 7728 Acked 8727 Write 6 Done Done New true
22952 7729 Acked 8728 Write 6 Done Done New true
22953 7730 Acked 8729 Read 10 Done Done New true
22954 7731 Acked 8730 Write 10 Done Done New true
22955 7732 Acked 8731 Read 2 Done Done New true
22956 7733 Acked 8732 Read 6 Done Done New true
22957 7734 Acked 8733 Write 8 Done Done New true
22958 7735 Acked 8734 Read 4 Done Done New true
22959 7736 Acked 8735 Read 4 Done Done New true
22960 7737 Acked 8736 Write 1 Done Done New true
22961 7738 Acked 8737 Write 10 Done Done New true
22962 7739 Acked 8738 Flush 0 Done Done New true
22963 7740 Acked 8739 Write 2 Done Done New true
22964 7741 Acked 8740 Write 7 Done Done New true
22965 7742 Acked 8741 Flush 0 Done Done New true
22966 7743 Acked 8742 Read 9 Done Done New true
22967 7744 Acked 8743 Read 4 Done Done New true
22968 7745 Acked 8744 Read 4 Done Done New true
22969 7746 Acked 8745 Read 6 Done Done New true
22970 7747 Acked 8746 Read 4 Done Done New true
22971 7748 Acked 8747 Read 9 Done Done New true
22972 7749 Acked 8748 Write 7 Done Done New true
22973 7750 Acked 8749 Read 2 Done Done New true
22974 7751 Acked 8750 Flush 0 Done Done New true
22975 7752 Acked 8751 Write 9 Done Done New true
22976 7753 Acked 8752 Write 3 Done Done New true
22977 7754 Acked 8753 Write 1 Done Done New true
22978 7755 Acked 8754 Write 4 Done Done New true
22979 7756 Acked 8755 Read 9 Done Done New true
22980 7757 Acked 8756 Write 5 Done Done New true
22981 7758 Acked 8757 Read 6 Done Done New true
22982 7759 Acked 8758 Read 9 Done Done New true
22983 7760 Acked 8759 Read 10 Done Done New true
22984 7761 Acked 8760 Write 3 Done Done New true
22985 7762 Acked 8761 Read 1 Done Done New true
22986 7763 Acked 8762 Read 9 Done Done New true
22987 7764 Acked 8763 Read 4 Done Done New true
22988 7765 Acked 8764 Write 8 Done Done New true
22989 7766 Acked 8765 Read 2 Done Done New true
22990 7767 Acked 8766 Write 10 Done Done New true
22991 7768 Acked 8767 Write 5 Done Done New true
22992 7769 Acked 8768 Read 8 Done Done New true
22993 7770 Acked 8769 Read 2 Done Done New true
22994 7771 Acked 8770 Read 3 Done Done New true
22995 7772 Acked 8771 Read 4 Done Done New true
22996 7773 Acked 8772 Flush 0 Done Done New true
22997 7774 Acked 8773 Read 4 Done Done New true
22998 7775 Acked 8774 Write 7 Done Done New true
22999 7776 Acked 8775 Read 4 Done Done New true
23000 7777 Acked 8776 Read 10 Done Done New true
23001 7778 Acked 8777 Write 10 Done Done New true
23002 7779 Acked 8778 Write 5 Done Done New true
23003 7780 Acked 8779 Write 8 Done Done New true
23004 7781 Acked 8780 Read 8 Done Done New true
23005 7782 Acked 8781 Read 7 Done Done New true
23006 7783 Acked 8782 Read 1 Done Done New true
23007 7784 Acked 8783 Write 4 Done Done New true
23008 7785 Acked 8784 Flush 0 Done Done New true
23009 7786 Acked 8785 Write 1 Done Done New true
23010 7787 Acked 8786 Write 9 Done Done New true
23011 7788 Acked 8787 Read 3 Done Done New true
23012 7789 Acked 8788 Write 5 Done Done New true
23013 7790 Acked 8789 Write 4 Done Done New true
23014 7791 Acked 8790 Read 3 Done Done New true
23015 7792 Acked 8791 Read 10 Done Done New true
23016 7793 Acked 8792 Read 3 Done Done New true
23017 7794 Acked 8793 Write 5 Done Done New true
23018 7795 Acked 8794 Write 10 Done Done New true
23019 7796 Acked 8795 Read 1 Done Done New true
23020 7797 Acked 8796 Read 2 Done Done New true
23021 7798 Acked 8797 Write 4 Done Done New true
23022 7799 Acked 8798 Read 3 Done Done New true
23023 7800 Acked 8799 Read 3 Done Done New true
23024 7801 Acked 8800 Write 1 Done Done New true
23025 7802 Acked 8801 Flush 0 Done Done New true
23026 7803 Acked 8802 Read 9 Done Done New true
23027 7804 Acked 8803 Read 4 Done Done New true
23028 7805 Acked 8804 Write 1 Done Done New true
23029 7806 Acked 8805 Write 10 Done Done New true
23030 7807 Acked 8806 Write 7 Done Done New true
23031 7808 Acked 8807 Read 4 Done Done New true
23032 7809 Acked 8808 Write 7 Done Done New true
23033 7810 Acked 8809 Read 2 Done Done New true
23034 7811 Acked 8810 Write 7 Done Done New true
23035 7812 Acked 8811 Read 5 Done Done New true
23036 7813 Acked 8812 Flush 0 Done Done New true
23037 7814 Acked 8813 Flush 0 Done Done New true
23038 7815 Acked 8814 Read 10 Done Done New true
23039 7816 Acked 8815 Flush 0 Done Done New true
23040 7817 Acked 8816 Read 6 Done Done New true
23041 7818 Acked 8817 Read 2 Done Done New true
23042 7819 Acked 8818 Flush 0 Done Done New true
23043 7820 Acked 8819 Read 1 Done Done New true
23044 7821 Acked 8820 Read 4 Done Done New true
23045 7822 Acked 8821 Read 3 Done Done New true
23046 7823 Acked 8822 Flush 0 Done Done New true
23047 7824 Acked 8823 Read 9 Done Done New true
23048 7825 Acked 8824 Read 10 Done Done New true
23049 7826 Acked 8825 Read 3 Done Done New true
23050 7827 Acked 8826 Write 9 Done Done New true
23051 7828 Acked 8827 Read 1 Done Done New true
23052 7829 Acked 8828 Write 7 Done Done New true
23053 7830 Acked 8829 Flush 0 Done Done New true
23054 7831 Acked 8830 Read 5 Done Done New true
23055 7832 Acked 8831 Read 8 Done Done New true
23056 7833 Acked 8832 Write 8 Done Done New true
23057 7834 Acked 8833 Flush 0 Done Done New true
23058 7835 Acked 8834 Write 2 Done Done New true
23059 7836 Acked 8835 Read 8 Done Done New true
23060 7837 Acked 8836 Read 10 Done Done New true
23061 7838 Acked 8837 Flush 0 Done Done New true
23062 STATES DS:0 DS:1 DS:2 TOTAL
23063 New 0 0 302 302
23064 Sent 0 0 0 0
23065 Done 302 302 0 604
23066 Skipped 0 0 0 0
23067 Error 0 0 0 0
23068 Last Flush: 8837 8837 8535
23069 Downstairs last five completed: 8535 8534 8533 8532 8531
23070 Upstairs last five completed: 7838 7837 7836 7835 7834
23071 CLIENT: Up:0 ds:302 act:2
230722024-02-28T22:17:50.091ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
230732024-02-28T22:17:50.092ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
230742024-02-28T22:17:50.093ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
230752024-02-28T22:17:50.093ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
230762024-02-28T22:17:50.093ZINFOcrucible: send last flush ID to this DS: 8535 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
230772024-02-28T22:17:50.093ZINFOcrucible: Replied this last flush ID: 8535 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
230782024-02-28T22:17:50.093ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
230792024-02-28T22:17:50.093ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
230802024-02-28T22:17:50.093ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
230812024-02-28T22:17:50.093ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23082 ----------------------------------------------------------------
23083 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23084 Downstairs last five completed: 8837 8836 8835 8834 8833
23085 Upstairs last five completed: 7838 7837 7836 7835 7834
23086 CLIENT: Up:0 ds:0 act:3
23087 Replay: All jobs finished, all DS active.
23088 Main task has work to do, go find it
23089 got dsc Stop(0)
23090 stop 0
23091 Feb 28 22:17:53.425 INFO request completed, [8810] Got stop action so:false kr:false
23092 latency_us: 339, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 4c42f74d-050f-4517-943d-f5870640b2bd, remote_addr: 127.0.0.1:41656, local_addr: 127.0.0.1:9998
230932024-02-28T22:17:53.426ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
230942024-02-28T22:17:53.426ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
230952024-02-28T22:17:53.426ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
230962024-02-28T22:17:53.426ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
230972024-02-28T22:17:53.426ZINFOcrucible: [0] client re-new 0 jobs since flush 8837 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23098 Feb 28 22:17:53.426 INFO request completed, latency_us: {"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:17:53.426373581Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
23099 130, response_code: 200, uri: /state/cid/0, method: GET, req_id: d9bce3be-b853-40c9-806f-13776fc52f54, remote_addr: 127.0.0.1:41656, local_addr: 127.0.0.1:9998
23100 [8810] Exited with: ExitStatus(unix_wait_status(9))
23101 [8810][0] reports Exit
231022024-02-28T22:17:54.427ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231032024-02-28T22:17:54.427ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231042024-02-28T22:17:54.427ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231052024-02-28T22:17:54.427ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231062024-02-28T22:17:54.427ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231072024-02-28T22:17:54.427ZINFOcrucible: [0] client re-new 0 jobs since flush 8837 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231082024-02-28T22:17:54.427ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231092024-02-28T22:17:55.428ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231102024-02-28T22:17:55.428ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231112024-02-28T22:17:55.428ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231122024-02-28T22:17:55.428ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231132024-02-28T22:17:55.428ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231142024-02-28T22:17:55.428ZINFOcrucible: [0] client re-new 0 jobs since flush 8837 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231152024-02-28T22:17:55.428ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231162024-02-28T22:17:56.429ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231172024-02-28T22:17:56.429ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231182024-02-28T22:17:56.429ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231192024-02-28T22:17:56.429ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231202024-02-28T22:17:56.429ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231212024-02-28T22:17:56.429ZINFOcrucible: [0] client re-new 0 jobs since flush 8837 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231222024-02-28T22:17:56.429ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23123 Feb 28 22:17:57.428 INFO request completed, latency_us: 207, response_code: 200, uri: /state/cid/0, method: GET, req_id: 6eea79ed-6e35-4b27-9e51-7ac0469ccc8f, remote_addr: 127.0.0.1:41656, local_addr: 127.0.0.1:9998
23124 001/300 Read block 3529 len 40960
231252024-02-28T22:17:57.430ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231262024-02-28T22:17:57.430ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231272024-02-28T22:17:57.431ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231282024-02-28T22:17:57.431ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231292024-02-28T22:17:57.431ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
231302024-02-28T22:17:57.431ZINFOcrucible: [0] client re-new 1 jobs since flush 8837 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23131 {"msg":"client task is exiting","v":0,"name":"crucible","level":30002/300 Read block 2785 len 24576
23132 ,"time":"2024-02-28T22:17:57.431137757Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1164,"":"io task","client":"0","":"downstairs","session_id":"69c7fbb2-d5b2-4965-9fa4-441c4d60207e"}
23133 003/300 Flush
23134 004/300 Read block 2412 len 28672
23135 005/300 Read block 3474 len 24576
23136 006/300 Read block 3595 len 8192
23137 007/300 Read block 4225 len 16384
23138 008/300 Write block 3655 len 20480 data: 4 4 3 3 3
23139 009/300 Write block 3231 len 36864 data: 6 5 4 4 3 3 5 5 5
23140 010/300 Flush
23141 011/300 Read block 1953 len 8192
23142 012/300 Write block 3503 len 36864 data: 4 5 6 6 5 4 4 4 5
23143 013/300 Read block 2009 len 40960
23144 014/300 Write block 4961 len 8192 data: 5 3
23145 015/300 Read block 4242 len 4096
23146 016/300 Write block 492 len 12288 data: 6 8 7
23147 017/300 Read block 2238 len 40960
23148 018/300 Write block 1753 len 16384 data: 9 9 8 7
23149 019/300 Read block 4097 len 36864
23150 020/300 Write block 2210 len 24576 data: 4 6 6 7 8 5
23151 021/300 Read block 4773 len 32768
23152 022/300 Read block 3825 len 12288
23153 023/300 Read block 1091 len 20480
23154 024/300 Write block 897 len 20480 data: 5 5 3 2 3
23155 025/300 Read block 3030 len 16384
23156 026/300 Write block 661 len 8192 data: 5 5
23157 027/300 Read block 4254 len 24576
23158 028/300 Read block 3814 len 16384
23159 029/300 Read block 3007 len 20480
23160 030/300 Write block 1587 len 20480 data: 6 6 4 4 4
23161 031/300 Read block 2240 len 32768
23162 032/300 Write block 514 len 28672 data: 4 4 5 3 4 5 5
23163 033/300 Read block 3994 len 12288
23164 034/300 Write block 4569 len 24576 data: 7 7 7 5 5 4
23165 035/300 Write block 3295 len 36864 data: 6 6 5 4 2 2 2 4 4
23166 036/300 Read block 938 len 36864
23167 037/300 Read block 1770 len 4096
23168 038/300 Write block 3974 len 40960 data: 4 5 5 3 4 4 3 5 4 5
23169 039/300 Flush
23170 040/300 Write block 3714 len 40960 data: 5 6 4 3 2 4 6 7 8 6
23171 041/300 Write block 2979 len 12288 data: 6 6 5
23172 042/300 Read block 4201 len 24576
23173 043/300 Write block 998 len 28672 data: 6 6 7 7 6 6 6
23174 044/300 Read block 530 len 32768
23175 045/300 Read block 1753 len 28672
23176 046/300 Read block 3289 len 36864
23177 047/300 Read block 4794 len 32768
23178 048/300 Read block 1800 len 4096
23179 049/300 Read block 3898 len 12288
23180 050/300 Write block 4314 len 24576 data: 6 6 5 5 7 5
23181 051/300 Read block 4366 len 40960
23182 052/300 Read block 1662 len 28672
23183 053/300 Read block 148 len 28672
23184 054/300 Read block 4399 len 32768
23185 055/300 Read block 2656 len 32768
23186 056/300 Read block 4116 len 24576
23187 057/300 Read block 690 len 32768
23188 058/300 Read block 4323 len 28672
23189 059/300 Write block 789 len 16384 data: 6 5 4 4
23190 060/300 Write block 3913 len 36864 data: 7 6 6 6 6 6 6 5 6
23191 061/300 Read block 4668 len 8192
23192 062/300 Write block 3362 len 12288 data: 8 7 7
23193 063/300 Write block 1135 len 32768 data: 9 8 8 8 8 10 11 8
23194 064/300 Read block 1867 len 20480
23195 065/300 Read block 2212 len 12288
23196 066/300 Write block 2873 len 16384 data: 7 4 6 5
23197 067/300 Read block 3368 len 36864
23198 068/300 Write block 3458 len 24576 data: 3 4 4 5 6 6
23199 069/300 Write block 3349 len 12288 data: 4 4 4
23200 070/300 Read block 3235 len 20480
23201 071/300 Read block 1264 len 32768
23202 072/300 Write block 4910 len 12288 data: 6 4 5
23203 073/300 Read block 3632 len 32768
23204 074/300 Read block 4340 len 24576
23205 075/300 Write block 604 len 32768 data: 5 3 4 4 4 4 5 7
23206 076/300 Write block 4832 len 40960 data: 4 4 4 5 5 4 3 4 4 3
23207 077/300 Read block 2854 len 32768
23208 078/300 Write block 569 len 24576 data: 3 4 5 7 7 7
23209 079/300 Write block 4252 len 16384 data: 9 9 10 7
23210 080/300 Read block 2969 len 8192
23211 081/300 Read block 4128 len 36864
23212 082/300 Flush
23213 083/300 Write block 4066 len 32768 data: 4 4 5 5 6 7 6 7
23214 084/300 Write block 3115 len 12288 data: 7 8 7
23215 085/300 Write block 844 len 40960 data: 5 5 9 8 7 6 6 6 5 5
23216 086/300 Read block 2307 len 32768
23217 087/300 Write block 2127 len 32768 data: 3 7 6 8 8 8 8 6
23218 088/300 Write block 4745 len 24576 data: 4 5 5 7 6 6
23219 089/300 Write block 2562 len 16384 data: 4 7 5 5
23220 090/300 Read block 2218 len 28672
23221 091/300 Flush
23222 092/300 Write block 1074 len 12288 data: 3 3 3
23223 093/300 Flush
23224 094/300 Flush
23225 095/300 Read block 974 len 12288
23226 096/300 Write block 3202 len 24576 data: 6 8 8 8 8 8
23227 097/300 Read block 4485 len 28672
23228 098/300 Write block 4640 len 16384 data: 3 3 3 4
23229 099/300 Read block 3149 len 28672
23230 100/300 Read block 684 len 12288
23231 101/300 Write block 3433 len 36864 data: 9 8 7 7 5 7 7 9 7
23232 102/300 Write block 4531 len 24576 data: 6 6 5 5 5 6
23233 103/300 Read block 2101 len 8192
23234 104/300 Read block 4373 len 28672
23235 105/300 Write block 3339 len 8192 data: 5 3
23236 106/300 Read block 3122 len 20480
23237 107/300 Write block 1493 len 20480 data: 5 7 6 5 5
23238 108/300 Read block 2511 len 4096
23239 109/300 Read block 3321 len 36864
23240 110/300 Write block 3971 len 24576 data: 9 7 6 5 6 6
23241 111/300 Read block 2131 len 32768
23242 112/300 Write block 509 len 28672 data: 8 6 5 6 5 5 5
23243 113/300 Read block 579 len 32768
23244 114/300 Read block 1882 len 28672
23245 115/300 Read block 692 len 16384
23246 116/300 Read block 4909 len 4096
23247 117/300 Write block 1053 len 8192 data: 6 6
23248 118/300 Read block 1254 len 16384
23249 119/300 Write block 668 len 20480 data: 6 6 9 10 8
23250 120/300 Read block 1194 len 16384
23251 121/300 Flush
23252 122/300 Read block 4299 len 36864
23253 123/300 Write block 1310 len 4096 data: 7
23254 124/300 Write block 3126 len 28672 data: 4 4 4 5 5 6 6
23255 125/300 Read block 1337 len 24576
23256 126/300 Read block 2857 len 40960
23257 127/300 Read block 2773 len 32768
23258 128/300 Read block 3356 len 16384
23259 129/300 Read block 4774 len 20480
23260 130/300 Flush
23261 131/300 Read block 2768 len 28672
23262 132/300 Read block 3661 len 40960
23263 133/300 Read block 800 len 16384
23264 134/300 Read block 1152 len 8192
23265 135/300 Write block 3905 len 32768 data: 6 5 7 6 7 6 5 6
23266 136/300 Read block 1334 len 12288
23267 137/300 Write block 266 len 32768 data: 4 3 5 4 3 4 3 3
23268 138/300 Read block 1828 len 12288
23269 139/300 Read block 852 len 28672
23270 140/300 Write block 3603 len 8192 data: 2 3
23271 141/300 Write block 3854 len 32768 data: 7 10 9 10 7 4 6 6
23272 142/300 Read block 2282 len 24576
23273 143/300 Write block 4442 len 16384 data: 2 2 2 2
23274 144/300 Write block 528 len 16384 data: 3 4 5 6
23275 145/300 Read block 1112 len 40960
23276 146/300 Read block 4102 len 12288
23277 147/300 Read block 2347 len 36864
23278 148/300 Write block 2967 len 28672 data: 3 3 4 4 6 6 7
23279 149/300 Write block 1990 len 36864 data: 9 10 9 6 7 6 5 5 5
23280 150/300 Write block 3881 len 28672 data: 4 5 6 7 6 5 6
23281 151/300 Read block 662 len 28672
23282 152/300 Write block 3805 len 40960 data: 4 4 4 4 3 3 4 4 5 5
23283 153/300 Read block 3413 len 40960
23284 154/300 Write block 2011 len 4096 data: 4
23285 155/300 Read block 340 len 24576
23286 156/300 Read block 417 len 4096
23287 157/300 Read block 1751 len 20480
23288 158/300 Read block 102 len 16384
23289 159/300 Read block 498 len 8192
23290 160/300 Write block 3484 len 16384 data: 5 5 4 3
23291 161/300 Write block 4618 len 28672 data: 6 7 9 8 6 6 6
23292 162/300 Write block 1123 len 20480 data: 4 7 9 9 10
23293 163/300 Read block 3040 len 36864
23294 164/300 Flush
23295 165/300 Flush
23296 166/300 Read block 3010 len 24576
23297 167/300 Write block 671 len 12288 data: 11 9 8
23298 168/300 Flush
23299 169/300 Write block 3114 len 12288 data: 6 8 9
23300 170/300 Write block 79 len 20480 data: 5 5 6 6 5
23301 171/300 Flush
23302 172/300 Read block 871 len 32768
23303 173/300 Read block 208 len 16384
23304 174/300 Flush
23305 175/300 Write block 3291 len 12288 data: 8 8 7
23306 176/300 Write block 1306 len 36864 data: 3 5 6 6 8 7 6 5 5
23307 177/300 Read block 4113 len 40960
23308 178/300 Write block 3534 len 32768 data: 6 7 6 5 5 4 4 5
23309 179/300 Read block 491 len 8192
23310 180/300 Read block 3748 len 8192
23311 181/300 Write block 4206 len 8192 data: 4 4
23312 182/300 Write block 254 len 24576 data: 4 5 4 4 5 5
23313 183/300 Read block 578 len 36864
23314 184/300 Write block 968 len 4096 data: 3
23315 185/300 Read block 4664 len 16384
23316 186/300 Read block 4780 len 4096
23317 187/300 Read block 2651 len 32768
23318 188/300 Read block 3518 len 32768
23319 189/300 Read block 4448 len 32768
23320 190/300 Read block 3600 len 8192
23321 191/300 Read block 108 len 16384
23322 192/300 Flush
23323 193/300 Read block 2557 len 24576
23324 194/300 Read block 4724 len 32768
23325 195/300 Read block 3523 len 28672
23326 196/300 Write block 4631 len 20480 data: 4 4 4 4 4
23327 197/300 Read block 3953 len 24576
23328 198/300 Write block 2782 len 20480 data: 6 7 7 5 5
23329 199/300 Write block 2738 len 40960 data: 3 4 5 6 4 5 5 5 5 4
23330 200/300 Read block 2298 len 36864
23331 201/300 Write block 4923 len 24576 data: 7 8 8 8 8 8
23332 202/300 Read block 58 len 28672
23333 203/300 Read block 550 len 40960
23334 204/300 Write block 4266 len 28672 data: 6 7 6 6 6 6 6
23335 205/300 Read block 3430 len 28672
23336 206/300 Read block 1382 len 40960
23337 207/300 Flush
23338 208/300 Flush
23339 209/300 Read block 1136 len 32768
23340 210/300 Read block 4896 len 24576
23341 211/300 Read block 1517 len 36864
23342 212/300 Read block 631 len 40960
23343 213/300 Flush
23344 214/300 Read block 204 len 32768
23345 215/300 Write block 120 len 20480 data: 5 5 6 5 4
23346 216/300 Write block 391 len 36864 data: 5 6 6 6 5 5 5 5 5
23347 217/300 Write block 4927 len 24576 data: 9 9 8 7 5 4
23348 218/300 Read block 4048 len 8192
23349 219/300 Write block 4554 len 40960 data: 4 5 5 5 4 4 4 4 4 5
23350 220/300 Read block 2158 len 40960
23351 221/300 Read block 4038 len 24576
23352 222/300 Read block 4264 len 8192
23353 223/300 Write block 926 len 28672 data: 5 5 5 5 5 5 6
23354 224/300 Write block 4377 len 40960 data: 6 6 5 6 7 7 6 7 6 6
23355 225/300 Write block 2559 len 16384 data: 4 4 3 5
23356 226/300 Write block 3683 len 36864 data: 5 5 4 4 2 2 3 4 3
23357 227/300 Write block 4171 len 4096 data: 8
23358 228/300 Read block 4208 len 24576
23359 229/300 Write block 2339 len 24576 data: 11 8 9 7 6 4
23360 230/300 Write block 3145 len 36864 data: 4 4 6 8 7 7 8 8 6
23361 231/300 Read block 4220 len 8192
23362 232/300 Write block 4198 len 20480 data: 5 7 6 6 6
23363 233/300 Write block 1600 len 8192 data: 7 7
23364 234/300 Read block 2227 len 20480
23365 235/300 Read block 2600 len 40960
23366 236/300 Read block 1298 len 32768
23367 237/300 Write block 1070 len 8192 data: 5 4
23368 238/300 Read block 244 len 12288
23369 239/300 Write block 1538 len 8192 data: 5 5
23370 240/300 Flush
23371 241/300 Read block 4973 len 8192
23372 242/300 Write block 4547 len 20480 data: 10 8 8 5 4
23373 243/300 Read block 3032 len 32768
23374 244/300 Flush
23375 245/300 Read block 4824 len 24576
23376 246/300 Write block 274 len 28672 data: 3 5 6 6 6 6 7
23377 247/300 Write block 2744 len 28672 data: 6 6 6 5 5 3 3
23378 248/300 Flush
23379 249/300 Write block 1856 len 20480 data: 7 5 7 8 8
23380 250/300 Write block 2656 len 40960 data: 3 4 4 4 5 5 4 2 2 2
23381 251/300 Read block 3032 len 28672
23382 252/300 Read block 3709 len 12288
23383 253/300 Read block 1753 len 40960
23384 254/300 Flush
23385 255/300 Write block 4042 len 8192 data: 5 4
23386 256/300 Read block 1458 len 24576
23387 257/300 Read block 4772 len 24576
23388 258/300 Read block 4440 len 32768
23389 259/300 Write block 1173 len 32768 data: 6 4 5 5 5 5 5 6
23390 260/300 Write block 4193 len 36864 data: 6 5 3 4 5 6 8 7 7
23391 261/300 Read block 4040 len 28672
23392 262/300 Read block 276 len 12288
23393 263/300 Read block 1897 len 32768
23394 264/300 Read block 4827 len 40960
23395 265/300 Write block 972 len 20480 data: 2 3 3 3 4
23396 266/300 Write block 1471 len 40960 data: 8 9 11 10 10 10 9 9 6 6
23397 267/300 Write block 3119 len 32768 data: 7 8 8 10 8 8 5 5
23398 268/300 Read block 3331 len 24576
23399 269/300 Write block 4858 len 36864 data: 6 7 8 8 10 10 11 8 6
23400 270/300 Read block 2713 len 4096
23401 271/300 Read block 3299 len 24576
23402 272/300 Write block 4958 len 36864 data: 3 5 5 6 4 3 3 3 3
23403 273/300 Read block 2047 len 28672
23404 274/300 Write block 902 len 28672 data: 4 5 5 6 5 6 7
23405 275/300 Write block 780 len 32768 data: 8 8 8 8 8 7 7 7
23406 276/300 Read block 4058 len 12288
23407 277/300 Read block 3736 len 12288
23408 278/300 Write block 496 len 40960 data: 9 8 9 8 7 6 4 4 4 8
23409 279/300 Write block 4957 len 8192 data: 3 4
23410 280/300 Write block 922 len 16384 data: 6 4 4 4
23411 281/300 Write block 3117 len 4096 data: 8
23412 282/300 Write block 4770 len 36864 data: 7 6 4 5 5 4 3 3 3
23413 283/300 Write block 2546 len 16384 data: 6 4 3 3
23414 284/300 Write block 3470 len 12288 data: 5 3 4
23415 285/300 Flush
23416 286/300 Flush
23417 287/300 Read block 2840 len 4096
23418 288/300 Read block 194 len 20480
23419 289/300 Write block 3485 len 8192 data: 6 5
23420 290/300 Read block 1873 len 24576
23421 291/300 Read block 4696 len 28672
23422 292/300 Read block 2339 len 4096
23423 293/300 Read block 2113 len 20480
23424 294/300 Read block 4283 len 36864
23425 295/300 Read block 1866 len 40960
23426 296/300 Write block 1092 len 8192 data: 6 6
23427 297/300 Read block 805 len 16384
23428 298/300 Write block 1182 len 16384 data: 7 6 6 6
23429 299/300 Write block 3901 len 36864 data: 6 6 6 6 7 6 8 7 8
23430 300/300 Write block 2252 len 8192 data: 4 4
23431 Main task has work to do, go find it
23432 got dsc Start(0)
23433 start 0
23434 Feb 28 22:17:57.772 INFO request completed, latency_us: 261, response_code: 204, uri: /start/cid/0, method: POST, req_id: 401913c7-c818-4732-bc71-d4f31002972e, remote_addr: 127.0.0.1:41656, local_addr: 127.0.0.1:9998
23435 Replay: started 0, returned:Ok(())
23436 ----------------------------------------------------------------
23437 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
23438 States: Offline Active Active
23439 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
23440 7839 Acked 8838 Read 10 New Done Done true
23441 7840 Acked 8839 Read 6 New Done Done false
23442 7841 Acked 8840 Flush 0 New Done Done false
23443 7842 Acked 8841 Read 7 New Done Done false
23444 7843 Acked 8842 Read 6 New Done Done false
23445 7844 Acked 8843 Read 2 New Done Done false
23446 7845 Acked 8844 Read 4 New Done Done false
23447 7846 Acked 8845 Write 5 New Done Done false
23448 7847 Acked 8846 Write 9 New Done Done false
23449 7848 Acked 8847 Flush 0 New Done Done false
23450 7849 Acked 8848 Read 2 New Done Done false
23451 7850 Acked 8849 Write 9 New Done Done false
23452 7851 Acked 8850 Read 10 New Done Done false
23453 7852 Acked 8851 Write 2 New Done Done false
23454 7853 Acked 8852 Read 1 New Done Done false
23455 7854 Acked 8853 Write 3 New Done Done false
23456 7855 Acked 8854 Read 10 New Done Done false
23457 7856 Acked 8855 Write 4 New Done Done false
23458 7857 Acked 8856 Read 9 New Done Done false
23459 7858 Acked 8857 Write 6 New Done Done false
23460 7859 Acked 8858 Read 8 New Done Done false
23461 7860 Acked 8859 Read 3 New Done Done false
23462 7861 Acked 8860 Read 5 New Done Done false
23463 7862 Acked 8861 Write 5 New Done Done false
23464 7863 Acked 8862 Read 4 New Done Done false
23465 7864 Acked 8863 Write 2 New Done Done false
23466 7865 Acked 8864 Read 6 New Done Done false
23467 7866 Acked 8865 Read 4 New Done Done false
23468 7867 Acked 8866 Read 5 New Done Done false
23469 7868 Acked 8867 Write 5 New Done Done false
23470 7869 Acked 8868 Read 8 New Done Done false
23471 7870 Acked 8869 Write 7 New Done Done false
23472 7871 Acked 8870 Read 3 New Done Done false
23473 7872 Acked 8871 Write 6 New Done Done false
23474 7873 Acked 8872 Write 9 New Done Done false
23475 7874 Acked 8873 Read 9 New Done Done false
23476 7875 Acked 8874 Read 1 New Done Done false
23477 7876 Acked 8875 Write 10 New Done Done false
23478 7877 Acked 8876 Flush 0 New Done Done false
23479 7878 Acked 8877 Write 10 New Done Done false
23480 7879 Acked 8878 Write 3 New Done Done false
23481 7880 Acked 8879 Read 6 New Done Done false
23482 7881 Acked 8880 Write 7 New Done Done false
23483 7882 Acked 8881 Read 8 New Done Done false
23484 7883 Acked 8882 Read 7 New Done Done false
23485 7884 Acked 8883 Read 9 New Done Done false
23486 7885 Acked 8884 Read 8 New Done Done false
23487 7886 Acked 8885 Read 1 New Done Done false
23488 7887 Acked 8886 Read 3 New Done Done false
23489 7888 Acked 8887 Write 6 New Done Done false
23490 7889 Acked 8888 Read 10 New Done Done false
23491 7890 Acked 8889 Read 7 New Done Done false
23492 7891 Acked 8890 Read 7 New Done Done false
23493 7892 Acked 8891 Read 8 New Done Done false
23494 7893 Acked 8892 Read 8 New Done Done false
23495 7894 Acked 8893 Read 6 New Done Done false
23496 7895 Acked 8894 Read 8 New Done Done false
23497 7896 Acked 8895 Read 7 New Done Done false
23498 7897 Acked 8896 Write 4 New Done Done false
23499 7898 Acked 8897 Write 9 New Done Done false
23500 7899 Acked 8898 Read 2 New Done Done false
23501 7900 Acked 8899 Write 3 New Done Done false
23502 7901 Acked 8900 Write 8 New Done Done false
23503 7902 Acked 8901 Read 5 New Done Done false
23504 7903 Acked 8902 Read 3 New Done Done false
23505 7904 Acked 8903 Write 4 New Done Done false
23506 7905 Acked 8904 Read 9 New Done Done false
23507 7906 Acked 8905 Write 6 New Done Done false
23508 7907 Acked 8906 Write 3 New Done Done false
23509 7908 Acked 8907 Read 5 New Done Done false
23510 7909 Acked 8908 Read 8 New Done Done false
23511 7910 Acked 8909 Write 3 New Done Done false
23512 7911 Acked 8910 Read 8 New Done Done false
23513 7912 Acked 8911 Read 6 New Done Done false
23514 7913 Acked 8912 Write 8 New Done Done false
23515 7914 Acked 8913 Write 10 New Done Done false
23516 7915 Acked 8914 Read 8 New Done Done false
23517 7916 Acked 8915 Write 6 New Done Done false
23518 7917 Acked 8916 Write 4 New Done Done false
23519 7918 Acked 8917 Read 2 New Done Done false
23520 7919 Acked 8918 Read 9 New Done Done false
23521 7920 Acked 8919 Flush 0 New Done Done false
23522 7921 Acked 8920 Write 8 New Done Done false
23523 7922 Acked 8921 Write 3 New Done Done false
23524 7923 Acked 8922 Write 10 New Done Done false
23525 7924 Acked 8923 Read 8 New Done Done false
23526 7925 Acked 8924 Write 8 New Done Done false
23527 7926 Acked 8925 Write 6 New Done Done false
23528 7927 Acked 8926 Write 4 New Done Done false
23529 7928 Acked 8927 Read 7 New Done Done false
23530 7929 Acked 8928 Flush 0 New Done Done false
23531 7930 Acked 8929 Write 3 New Done Done false
23532 7931 Acked 8930 Flush 0 New Done Done false
23533 7932 Acked 8931 Flush 0 New Done Done false
23534 7933 Acked 8932 Read 3 New Done Done false
23535 7934 Acked 8933 Write 6 New Done Done false
23536 7935 Acked 8934 Read 7 New Done Done false
23537 7936 Acked 8935 Write 4 New Done Done false
23538 7937 Acked 8936 Read 7 New Done Done false
23539 7938 Acked 8937 Read 3 New Done Done false
23540 7939 Acked 8938 Write 9 New Done Done false
23541 7940 Acked 8939 Write 6 New Done Done false
23542 7941 Acked 8940 Read 2 New Done Done false
23543 7942 Acked 8941 Read 7 New Done Done false
23544 7943 Acked 8942 Write 2 New Done Done false
23545 7944 Acked 8943 Read 5 New Done Done false
23546 7945 Acked 8944 Write 5 New Done Done false
23547 7946 Acked 8945 Read 1 New Done Done false
23548 7947 Acked 8946 Read 9 New Done Done false
23549 7948 Acked 8947 Write 6 New Done Done false
23550 7949 Acked 8948 Read 8 New Done Done false
23551 7950 Acked 8949 Write 7 New Done Done false
23552 7951 Acked 8950 Read 8 New Done Done false
23553 7952 Acked 8951 Read 7 New Done Done false
23554 7953 Acked 8952 Read 4 New Done Done false
23555 7954 Acked 8953 Read 1 New Done Done false
23556 7955 Acked 8954 Write 2 New Done Done false
23557 7956 Acked 8955 Read 4 New Done Done false
23558 7957 Acked 8956 Write 5 New Done Done false
23559 7958 Acked 8957 Read 4 New Done Done false
23560 7959 Acked 8958 Flush 0 New Done Done false
23561 7960 Acked 8959 Read 9 New Done Done false
23562 7961 Acked 8960 Write 1 New Done Done false
23563 7962 Acked 8961 Write 7 New Done Done false
23564 7963 Acked 8962 Read 6 New Done Done false
23565 7964 Acked 8963 Read 10 New Done Done false
23566 7965 Acked 8964 Read 8 New Done Done false
23567 7966 Acked 8965 Read 4 New Done Done false
23568 7967 Acked 8966 Read 5 New Done Done false
23569 7968 Acked 8967 Flush 0 New Done Done false
23570 7969 Acked 8968 Read 7 New Done Done false
23571 7970 Acked 8969 Read 10 New Done Done false
23572 7971 Acked 8970 Read 4 New Done Done false
23573 7972 Acked 8971 Read 2 New Done Done false
23574 7973 Acked 8972 Write 8 New Done Done false
23575 7974 Acked 8973 Read 3 New Done Done false
23576 7975 Acked 8974 Write 8 New Done Done false
23577 7976 Acked 8975 Read 3 New Done Done false
23578 7977 Acked 8976 Read 7 New Done Done false
23579 7978 Acked 8977 Write 2 New Done Done false
23580 7979 Acked 8978 Write 8 New Done Done false
23581 7980 Acked 8979 Read 6 New Done Done false
23582 7981 Acked 8980 Write 4 New Done Done false
23583 7982 Acked 8981 Write 4 New Done Done false
23584 7983 Acked 8982 Read 10 New Done Done false
23585 7984 Acked 8983 Read 3 New Done Done false
23586 7985 Acked 8984 Read 9 New Done Done false
23587 7986 Acked 8985 Write 7 New Done Done false
23588 7987 Acked 8986 Write 9 New Done Done false
23589 7988 Acked 8987 Write 7 New Done Done false
23590 7989 Acked 8988 Read 7 New Done Done false
23591 7990 Acked 8989 Write 10 New Done Done false
23592 7991 Acked 8990 Read 10 New Done Done false
23593 7992 Acked 8991 Write 1 New Done Done false
23594 7993 Acked 8992 Read 6 New Done Done false
23595 7994 Acked 8993 Read 1 New Done Done false
23596 7995 Acked 8994 Read 5 New Done Done false
23597 7996 Acked 8995 Read 4 New Done Done false
23598 7997 Acked 8996 Read 2 New Done Done false
23599 7998 Acked 8997 Write 4 New Done Done false
23600 7999 Acked 8998 Write 7 New Done Done false
23601 8000 Acked 8999 Write 5 New Done Done false
23602 8001 Acked 9000 Read 9 New Done Done false
23603 8002 Acked 9001 Flush 0 New Done Done false
23604 8003 Acked 9002 Flush 0 New Done Done false
23605 8004 Acked 9003 Read 6 New Done Done false
23606 8005 Acked 9004 Write 3 New Done Done false
23607 8006 Acked 9005 Flush 0 New Done Done false
23608 8007 Acked 9006 Write 3 New Done Done false
23609 8008 Acked 9007 Write 5 New Done Done false
23610 8009 Acked 9008 Flush 0 New Done Done false
23611 8010 Acked 9009 Read 8 New Done Done false
23612 8011 Acked 9010 Read 4 New Done Done false
23613 8012 Acked 9011 Flush 0 New Done Done false
23614 8013 Acked 9012 Write 3 New Done Done false
23615 8014 Acked 9013 Write 9 New Done Done false
23616 8015 Acked 9014 Read 10 New Done Done false
23617 8016 Acked 9015 Write 8 New Done Done false
23618 8017 Acked 9016 Read 2 New Done Done false
23619 8018 Acked 9017 Read 2 New Done Done false
23620 8019 Acked 9018 Write 2 New Done Done false
23621 8020 Acked 9019 Write 6 New Done Done false
23622 8021 Acked 9020 Read 9 New Done Done false
23623 8022 Acked 9021 Write 1 New Done Done false
23624 8023 Acked 9022 Read 4 New Done Done false
23625 8024 Acked 9023 Read 1 New Done Done false
23626 8025 Acked 9024 Read 8 New Done Done false
23627 8026 Acked 9025 Read 8 New Done Done false
23628 8027 Acked 9026 Read 8 New Done Done false
23629 8028 Acked 9027 Read 2 New Done Done false
23630 8029 Acked 9028 Read 4 New Done Done false
23631 8030 Acked 9029 Flush 0 New Done Done false
23632 8031 Acked 9030 Read 6 New Done Done false
23633 8032 Acked 9031 Read 8 New Done Done false
23634 8033 Acked 9032 Read 7 New Done Done false
23635 8034 Acked 9033 Write 5 New Done Done false
23636 8035 Acked 9034 Read 6 New Done Done false
23637 8036 Acked 9035 Write 5 New Done Done false
23638 8037 Acked 9036 Write 10 New Done Done false
23639 8038 Acked 9037 Read 9 New Done Done false
23640 8039 Acked 9038 Write 6 New Done Done false
23641 8040 Acked 9039 Read 7 New Done Done false
23642 8041 Acked 9040 Read 10 New Done Done false
23643 8042 Acked 9041 Write 7 New Done Done false
23644 8043 Acked 9042 Read 7 New Done Done false
23645 8044 Acked 9043 Read 10 New Done Done false
23646 8045 Acked 9044 Flush 0 New Done Done false
23647 8046 Acked 9045 Flush 0 New Done Done false
23648 8047 Acked 9046 Read 8 New Done Done false
23649 8048 Acked 9047 Read 6 New Done Done false
23650 8049 Acked 9048 Read 9 New Done Done false
23651 8050 Acked 9049 Read 10 New Done Done false
23652 8051 Acked 9050 Flush 0 New Done Done false
23653 8052 Acked 9051 Read 8 New Done Done false
23654 8053 Acked 9052 Write 5 New Done Done false
23655 8054 Acked 9053 Write 9 New Done Done false
23656 8055 Acked 9054 Write 6 New Done Done false
23657 8056 Acked 9055 Read 2 New Done Done false
23658 8057 Acked 9056 Write 10 New Done Done false
23659 8058 Acked 9057 Read 10 New Done Done false
23660 8059 Acked 9058 Read 6 New Done Done false
23661 8060 Acked 9059 Read 2 New Done Done false
23662 8061 Acked 9060 Write 7 New Done Done false
23663 8062 Acked 9061 Write 10 New Done Done false
23664 8063 Acked 9062 Write 4 New Done Done false
23665 8064 Acked 9063 Write 9 New Done Done false
23666 8065 Acked 9064 Write 1 New Done Done false
23667 8066 Acked 9065 Read 6 New Done Done false
23668 8067 Acked 9066 Write 6 New Done Done false
23669 8068 Acked 9067 Write 9 New Done Done false
23670 8069 Acked 9068 Read 2 New Done Done false
23671 8070 Acked 9069 Write 5 New Done Done false
23672 8071 Acked 9070 Write 2 New Done Done false
23673 8072 Acked 9071 Read 5 New Done Done false
23674 8073 Acked 9072 Read 10 New Done Done false
23675 8074 Acked 9073 Read 8 New Done Done false
23676 8075 Acked 9074 Write 2 New Done Done false
23677 8076 Acked 9075 Read 3 New Done Done false
23678 8077 Acked 9076 Write 2 New Done Done false
23679 8078 Acked 9077 Flush 0 New Done Done false
23680 8079 Acked 9078 Read 2 New Done Done false
23681 8080 Acked 9079 Write 5 New Done Done false
23682 8081 Acked 9080 Read 8 New Done Done false
23683 8082 Acked 9081 Flush 0 New Done Done false
23684 8083 Acked 9082 Read 6 New Done Done false
23685 8084 Acked 9083 Write 7 New Done Done false
23686 8085 Acked 9084 Write 7 New Done Done false
23687 8086 Acked 9085 Flush 0 New Done Done false
23688 8087 Acked 9086 Write 5 New Done Done false
23689 8088 Acked 9087 Write 10 New Done Done false
23690 8089 Acked 9088 Read 7 New Done Done false
23691 8090 Acked 9089 Read 3 New Done Done false
23692 8091 Acked 9090 Read 10 New Done Done false
23693 8092 Acked 9091 Flush 0 New Done Done false
23694 8093 Acked 9092 Write 2 New Done Done false
23695 8094 Acked 9093 Read 6 New Done Done false
23696 8095 Acked 9094 Read 6 New Done Done false
23697 8096 Acked 9095 Read 8 New Done Done false
23698 8097 Acked 9096 Write 8 New Done Done false
23699 8098 Acked 9097 Write 9 New Done Done false
23700 8099 Acked 9098 Read 7 New Done Done false
23701 8100 Acked 9099 Read 3 New Done Done false
23702 8101 Acked 9100 Read 8 New Done Done false
23703 8102 Acked 9101 Read 10 New Done Done false
23704 8103 Acked 9102 Write 5 New Done Done false
23705 8104 Acked 9103 Write 10 New Done Done false
23706 8105 Acked 9104 Write 8 New Done Done false
23707 8106 Acked 9105 Read 6 New Done Done false
23708 8107 Acked 9106 Write 9 New Done Done false
23709 8108 Acked 9107 Read 1 New Done Done false
23710 8109 Acked 9108 Read 6 New Done Done false
23711 8110 Acked 9109 Write 9 New Done Done false
23712 8111 Acked 9110 Read 7 New Done Done false
23713 8112 Acked 9111 Write 7 New Done Done false
23714 8113 Acked 9112 Write 8 New Done Done false
23715 8114 Acked 9113 Read 3 New Done Done false
23716 8115 Acked 9114 Read 3 New Done Done false
23717 8116 Acked 9115 Write 10 New Done Done false
23718 8117 Acked 9116 Write 2 New Done Done false
23719 8118 Acked 9117 Write 4 New Done Done false
23720 8119 Acked 9118 Write 1 New Done Done false
23721 8120 Acked 9119 Write 9 New Done Done false
23722 8121 Acked 9120 Write 4 New Done Done false
23723 8122 Acked 9121 Write 3 New Done Done false
23724 8123 Acked 9122 Flush 0 New Done Done false
23725 8124 Acked 9123 Flush 0 New Done Done false
23726 8125 Acked 9124 Read 1 New Done Done false
23727 8126 Acked 9125 Read 5 New Done Done false
23728 8127 Acked 9126 Write 2 New Done Done false
23729 8128 Acked 9127 Read 6 New Done Done false
23730 8129 Acked 9128 Read 7 New Done Done false
23731 8130 Acked 9129 Read 1 New Done Done false
23732 8131 Acked 9130 Read 5 New Done Done false
23733 8132 Acked 9131 Read 9 New Done Done false
23734 8133 Acked 9132 Read 10 New Done Done false
23735 8134 Acked 9133 Write 2 New Done Done false
23736 8135 Acked 9134 Read 4 New Done Done false
23737 8136 Acked 9135 Write 4 New Done Done false
23738 8137 Acked 9136 Write 9 New Done Done false
23739 8138 Acked 9137 Write 2 New Done Done false
23740 STATES DS:0 DS:1 DS:2 TOTAL
23741 New 300 0 0 300
23742 Sent 0 0 0 0
23743 Done 0 300 300 600
23744 Skipped 0 0 0 0
23745 Error 0 0 0 0
23746 Last Flush: 8837 9123 9123
23747 Downstairs last five completed: 8837 8836 8835 8834 8833
23748 Upstairs last five completed: 8138 8137 8136 8135 8134
23749 CLIENT: Up:0 ds:300 act:2
237502024-02-28T22:17:58.433ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237512024-02-28T22:17:58.433ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237522024-02-28T22:17:58.433ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237532024-02-28T22:17:58.433ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237542024-02-28T22:17:58.433ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237552024-02-28T22:17:58.433ZINFOcrucible: [0] client re-new 301 jobs since flush 8837 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237562024-02-28T22:17:58.433ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237572024-02-28T22:17:59.434ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237582024-02-28T22:17:59.434ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237592024-02-28T22:17:59.434ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237602024-02-28T22:17:59.434ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237612024-02-28T22:17:59.434ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237622024-02-28T22:17:59.434ZINFOcrucible: [0] client re-new 301 jobs since flush 8837 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237632024-02-28T22:17:59.434ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237642024-02-28T22:18:00.435ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237652024-02-28T22:18:00.436ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237662024-02-28T22:18:00.436ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237672024-02-28T22:18:00.436ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237682024-02-28T22:18:00.436ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237692024-02-28T22:18:00.436ZINFOcrucible: [0] client re-new 301 jobs since flush 8837 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237702024-02-28T22:18:00.436ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23771 [8810] got start action
23772 [8810] I am going to restart
23773 Starting downstairs at port 8810
23774 Make output file at "/tmp/dsc/downstairs-8810.txt"
23775 [8810][0] reports Starting
237762024-02-28T22:18:01.437ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237772024-02-28T22:18:01.437ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237782024-02-28T22:18:01.437ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237792024-02-28T22:18:01.437ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237802024-02-28T22:18:01.437ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237812024-02-28T22:18:01.437ZINFOcrucible: [0] client re-new 301 jobs since flush 8837 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
237822024-02-28T22:18:01.437ZINFOcrucible: client task is exiting = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
23783 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1191)
23784 [8810][0] reports Running
23785 ----------------------------------------------------------------
23786 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23787 States: Offline Active Active
23788 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
23789 7839 Acked 8838 Read 10 New Done Done true
23790 7840 Acked 8839 Read 6 New Done Done true
23791 7841 Acked 8840 Flush 0 New Done Done true
23792 7842 Acked 8841 Read 7 New Done Done true
23793 7843 Acked 8842 Read 6 New Done Done true
23794 7844 Acked 8843 Read 2 New Done Done true
23795 7845 Acked 8844 Read 4 New Done Done true
23796 7846 Acked 8845 Write 5 New Done Done true
23797 7847 Acked 8846 Write 9 New Done Done true
23798 7848 Acked 8847 Flush 0 New Done Done true
23799 7849 Acked 8848 Read 2 New Done Done true
23800 7850 Acked 8849 Write 9 New Done Done true
23801 7851 Acked 8850 Read 10 New Done Done true
23802 7852 Acked 8851 Write 2 New Done Done true
23803 7853 Acked 8852 Read 1 New Done Done true
23804 7854 Acked 8853 Write 3 New Done Done true
23805 7855 Acked 8854 Read 10 New Done Done true
23806 7856 Acked 8855 Write 4 New Done Done true
23807 7857 Acked 8856 Read 9 New Done Done true
23808 7858 Acked 8857 Write 6 New Done Done true
23809 7859 Acked 8858 Read 8 New Done Done true
23810 7860 Acked 8859 Read 3 New Done Done true
23811 7861 Acked 8860 Read 5 New Done Done true
23812 7862 Acked 8861 Write 5 New Done Done true
23813 7863 Acked 8862 Read 4 New Done Done true
23814 7864 Acked 8863 Write 2 New Done Done true
23815 7865 Acked 8864 Read 6 New Done Done true
23816 7866 Acked 8865 Read 4 New Done Done true
23817 7867 Acked 8866 Read 5 New Done Done true
23818 7868 Acked 8867 Write 5 New Done Done true
23819 7869 Acked 8868 Read 8 New Done Done true
23820 7870 Acked 8869 Write 7 New Done Done true
23821 7871 Acked 8870 Read 3 New Done Done true
23822 7872 Acked 8871 Write 6 New Done Done true
23823 7873 Acked 8872 Write 9 New Done Done true
23824 7874 Acked 8873 Read 9 New Done Done true
23825 7875 Acked 8874 Read 1 New Done Done true
23826 7876 Acked 8875 Write 10 New Done Done true
23827 7877 Acked 8876 Flush 0 New Done Done true
23828 7878 Acked 8877 Write 10 New Done Done true
23829 7879 Acked 8878 Write 3 New Done Done true
23830 7880 Acked 8879 Read 6 New Done Done true
23831 7881 Acked 8880 Write 7 New Done Done true
23832 7882 Acked 8881 Read 8 New Done Done true
23833 7883 Acked 8882 Read 7 New Done Done true
23834 7884 Acked 8883 Read 9 New Done Done true
23835 7885 Acked 8884 Read 8 New Done Done true
23836 7886 Acked 8885 Read 1 New Done Done true
23837 7887 Acked 8886 Read 3 New Done Done true
23838 7888 Acked 8887 Write 6 New Done Done true
23839 7889 Acked 8888 Read 10 New Done Done true
23840 7890 Acked 8889 Read 7 New Done Done true
23841 7891 Acked 8890 Read 7 New Done Done true
23842 7892 Acked 8891 Read 8 New Done Done true
23843 7893 Acked 8892 Read 8 New Done Done true
23844 7894 Acked 8893 Read 6 New Done Done true
23845 7895 Acked 8894 Read 8 New Done Done true
23846 7896 Acked 8895 Read 7 New Done Done true
23847 7897 Acked 8896 Write 4 New Done Done true
23848 7898 Acked 8897 Write 9 New Done Done true
23849 7899 Acked 8898 Read 2 New Done Done true
23850 7900 Acked 8899 Write 3 New Done Done true
23851 7901 Acked 8900 Write 8 New Done Done true
23852 7902 Acked 8901 Read 5 New Done Done true
23853 7903 Acked 8902 Read 3 New Done Done true
23854 7904 Acked 8903 Write 4 New Done Done true
23855 7905 Acked 8904 Read 9 New Done Done true
23856 7906 Acked 8905 Write 6 New Done Done true
23857 7907 Acked 8906 Write 3 New Done Done true
23858 7908 Acked 8907 Read 5 New Done Done true
23859 7909 Acked 8908 Read 8 New Done Done true
23860 7910 Acked 8909 Write 3 New Done Done true
23861 7911 Acked 8910 Read 8 New Done Done true
23862 7912 Acked 8911 Read 6 New Done Done true
23863 7913 Acked 8912 Write 8 New Done Done true
23864 7914 Acked 8913 Write 10 New Done Done true
23865 7915 Acked 8914 Read 8 New Done Done true
23866 7916 Acked 8915 Write 6 New Done Done true
23867 7917 Acked 8916 Write 4 New Done Done true
23868 7918 Acked 8917 Read 2 New Done Done true
23869 7919 Acked 8918 Read 9 New Done Done true
23870 7920 Acked 8919 Flush 0 New Done Done true
23871 7921 Acked 8920 Write 8 New Done Done true
23872 7922 Acked 8921 Write 3 New Done Done true
23873 7923 Acked 8922 Write 10 New Done Done true
23874 7924 Acked 8923 Read 8 New Done Done true
23875 7925 Acked 8924 Write 8 New Done Done true
23876 7926 Acked 8925 Write 6 New Done Done true
23877 7927 Acked 8926 Write 4 New Done Done true
23878 7928 Acked 8927 Read 7 New Done Done true
23879 7929 Acked 8928 Flush 0 New Done Done true
23880 7930 Acked 8929 Write 3 New Done Done true
23881 7931 Acked 8930 Flush 0 New Done Done true
23882 7932 Acked 8931 Flush 0 New Done Done true
23883 7933 Acked 8932 Read 3 New Done Done true
23884 7934 Acked 8933 Write 6 New Done Done true
23885 7935 Acked 8934 Read 7 New Done Done true
23886 7936 Acked 8935 Write 4 New Done Done true
23887 7937 Acked 8936 Read 7 New Done Done true
23888 7938 Acked 8937 Read 3 New Done Done true
23889 7939 Acked 8938 Write 9 New Done Done true
23890 7940 Acked 8939 Write 6 New Done Done true
23891 7941 Acked 8940 Read 2 New Done Done true
23892 7942 Acked 8941 Read 7 New Done Done true
23893 7943 Acked 8942 Write 2 New Done Done true
23894 7944 Acked 8943 Read 5 New Done Done true
23895 7945 Acked 8944 Write 5 New Done Done true
23896 7946 Acked 8945 Read 1 New Done Done true
23897 7947 Acked 8946 Read 9 New Done Done true
23898 7948 Acked 8947 Write 6 New Done Done true
23899 7949 Acked 8948 Read 8 New Done Done true
23900 7950 Acked 8949 Write 7 New Done Done true
23901 7951 Acked 8950 Read 8 New Done Done true
23902 7952 Acked 8951 Read 7 New Done Done true
23903 7953 Acked 8952 Read 4 New Done Done true
23904 7954 Acked 8953 Read 1 New Done Done true
23905 7955 Acked 8954 Write 2 New Done Done true
23906 7956 Acked 8955 Read 4 New Done Done true
23907 7957 Acked 8956 Write 5 New Done Done true
23908 7958 Acked 8957 Read 4 New Done Done true
23909 7959 Acked 8958 Flush 0 New Done Done true
23910 7960 Acked 8959 Read 9 New Done Done true
23911 7961 Acked 8960 Write 1 New Done Done true
23912 7962 Acked 8961 Write 7 New Done Done true
23913 7963 Acked 8962 Read 6 New Done Done true
23914 7964 Acked 8963 Read 10 New Done Done true
23915 7965 Acked 8964 Read 8 New Done Done true
23916 7966 Acked 8965 Read 4 New Done Done true
23917 7967 Acked 8966 Read 5 New Done Done true
23918 7968 Acked 8967 Flush 0 New Done Done true
23919 7969 Acked 8968 Read 7 New Done Done true
23920 7970 Acked 8969 Read 10 New Done Done true
23921 7971 Acked 8970 Read 4 New Done Done true
23922 7972 Acked 8971 Read 2 New Done Done true
23923 7973 Acked 8972 Write 8 New Done Done true
23924 7974 Acked 8973 Read 3 New Done Done true
23925 7975 Acked 8974 Write 8 New Done Done true
23926 7976 Acked 8975 Read 3 New Done Done true
23927 7977 Acked 8976 Read 7 New Done Done true
23928 7978 Acked 8977 Write 2 New Done Done true
23929 7979 Acked 8978 Write 8 New Done Done true
23930 7980 Acked 8979 Read 6 New Done Done true
23931 7981 Acked 8980 Write 4 New Done Done true
23932 7982 Acked 8981 Write 4 New Done Done true
23933 7983 Acked 8982 Read 10 New Done Done true
23934 7984 Acked 8983 Read 3 New Done Done true
23935 7985 Acked 8984 Read 9 New Done Done true
23936 7986 Acked 8985 Write 7 New Done Done true
23937 7987 Acked 8986 Write 9 New Done Done true
23938 7988 Acked 8987 Write 7 New Done Done true
23939 7989 Acked 8988 Read 7 New Done Done true
23940 7990 Acked 8989 Write 10 New Done Done true
23941 7991 Acked 8990 Read 10 New Done Done true
23942 7992 Acked 8991 Write 1 New Done Done true
23943 7993 Acked 8992 Read 6 New Done Done true
23944 7994 Acked 8993 Read 1 New Done Done true
23945 7995 Acked 8994 Read 5 New Done Done true
23946 7996 Acked 8995 Read 4 New Done Done true
23947 7997 Acked 8996 Read 2 New Done Done true
23948 7998 Acked 8997 Write 4 New Done Done true
23949 7999 Acked 8998 Write 7 New Done Done true
23950 8000 Acked 8999 Write 5 New Done Done true
23951 8001 Acked 9000 Read 9 New Done Done true
23952 8002 Acked 9001 Flush 0 New Done Done true
23953 8003 Acked 9002 Flush 0 New Done Done true
23954 8004 Acked 9003 Read 6 New Done Done true
23955 8005 Acked 9004 Write 3 New Done Done true
23956 8006 Acked 9005 Flush 0 New Done Done true
23957 8007 Acked 9006 Write 3 New Done Done true
23958 8008 Acked 9007 Write 5 New Done Done true
23959 8009 Acked 9008 Flush 0 New Done Done true
23960 8010 Acked 9009 Read 8 New Done Done true
23961 8011 Acked 9010 Read 4 New Done Done true
23962 8012 Acked 9011 Flush 0 New Done Done true
23963 8013 Acked 9012 Write 3 New Done Done true
23964 8014 Acked 9013 Write 9 New Done Done true
23965 8015 Acked 9014 Read 10 New Done Done true
23966 8016 Acked 9015 Write 8 New Done Done true
23967 8017 Acked 9016 Read 2 New Done Done true
23968 8018 Acked 9017 Read 2 New Done Done true
23969 8019 Acked 9018 Write 2 New Done Done true
23970 8020 Acked 9019 Write 6 New Done Done true
23971 8021 Acked 9020 Read 9 New Done Done true
23972 8022 Acked 9021 Write 1 New Done Done true
23973 8023 Acked 9022 Read 4 New Done Done true
23974 8024 Acked 9023 Read 1 New Done Done true
23975 8025 Acked 9024 Read 8 New Done Done true
23976 8026 Acked 9025 Read 8 New Done Done true
23977 8027 Acked 9026 Read 8 New Done Done true
23978 8028 Acked 9027 Read 2 New Done Done true
23979 8029 Acked 9028 Read 4 New Done Done true
23980 8030 Acked 9029 Flush 0 New Done Done true
23981 8031 Acked 9030 Read 6 New Done Done true
23982 8032 Acked 9031 Read 8 New Done Done true
23983 8033 Acked 9032 Read 7 New Done Done true
23984 8034 Acked 9033 Write 5 New Done Done true
23985 8035 Acked 9034 Read 6 New Done Done true
23986 8036 Acked 9035 Write 5 New Done Done true
23987 8037 Acked 9036 Write 10 New Done Done true
23988 8038 Acked 9037 Read 9 New Done Done true
23989 8039 Acked 9038 Write 6 New Done Done true
23990 8040 Acked 9039 Read 7 New Done Done true
23991 8041 Acked 9040 Read 10 New Done Done true
23992 8042 Acked 9041 Write 7 New Done Done true
23993 8043 Acked 9042 Read 7 New Done Done true
23994 8044 Acked 9043 Read 10 New Done Done true
23995 8045 Acked 9044 Flush 0 New Done Done true
23996 8046 Acked 9045 Flush 0 New Done Done true
23997 8047 Acked 9046 Read 8 New Done Done true
23998 8048 Acked 9047 Read 6 New Done Done true
23999 8049 Acked 9048 Read 9 New Done Done true
24000 8050 Acked 9049 Read 10 New Done Done true
24001 8051 Acked 9050 Flush 0 New Done Done true
24002 8052 Acked 9051 Read 8 New Done Done true
24003 8053 Acked 9052 Write 5 New Done Done true
24004 8054 Acked 9053 Write 9 New Done Done true
24005 8055 Acked 9054 Write 6 New Done Done true
24006 8056 Acked 9055 Read 2 New Done Done true
24007 8057 Acked 9056 Write 10 New Done Done true
24008 8058 Acked 9057 Read 10 New Done Done true
24009 8059 Acked 9058 Read 6 New Done Done true
24010 8060 Acked 9059 Read 2 New Done Done true
24011 8061 Acked 9060 Write 7 New Done Done true
24012 8062 Acked 9061 Write 10 New Done Done true
24013 8063 Acked 9062 Write 4 New Done Done true
24014 8064 Acked 9063 Write 9 New Done Done true
24015 8065 Acked 9064 Write 1 New Done Done true
24016 8066 Acked 9065 Read 6 New Done Done true
24017 8067 Acked 9066 Write 6 New Done Done true
24018 8068 Acked 9067 Write 9 New Done Done true
24019 8069 Acked 9068 Read 2 New Done Done true
24020 8070 Acked 9069 Write 5 New Done Done true
24021 8071 Acked 9070 Write 2 New Done Done true
24022 8072 Acked 9071 Read 5 New Done Done true
24023 8073 Acked 9072 Read 10 New Done Done true
24024 8074 Acked 9073 Read 8 New Done Done true
24025 8075 Acked 9074 Write 2 New Done Done true
24026 8076 Acked 9075 Read 3 New Done Done true
24027 8077 Acked 9076 Write 2 New Done Done true
24028 8078 Acked 9077 Flush 0 New Done Done true
24029 8079 Acked 9078 Read 2 New Done Done true
24030 8080 Acked 9079 Write 5 New Done Done true
24031 8081 Acked 9080 Read 8 New Done Done true
24032 8082 Acked 9081 Flush 0 New Done Done true
24033 8083 Acked 9082 Read 6 New Done Done true
24034 8084 Acked 9083 Write 7 New Done Done true
24035 8085 Acked 9084 Write 7 New Done Done true
24036 8086 Acked 9085 Flush 0 New Done Done true
24037 8087 Acked 9086 Write 5 New Done Done true
24038 8088 Acked 9087 Write 10 New Done Done true
24039 8089 Acked 9088 Read 7 New Done Done true
24040 8090 Acked 9089 Read 3 New Done Done true
24041 8091 Acked 9090 Read 10 New Done Done true
24042 8092 Acked 9091 Flush 0 New Done Done true
24043 8093 Acked 9092 Write 2 New Done Done true
24044 8094 Acked 9093 Read 6 New Done Done true
24045 8095 Acked 9094 Read 6 New Done Done true
24046 8096 Acked 9095 Read 8 New Done Done true
24047 8097 Acked 9096 Write 8 New Done Done true
24048 8098 Acked 9097 Write 9 New Done Done true
24049 8099 Acked 9098 Read 7 New Done Done true
24050 8100 Acked 9099 Read 3 New Done Done true
24051 8101 Acked 9100 Read 8 New Done Done true
24052 8102 Acked 9101 Read 10 New Done Done true
24053 8103 Acked 9102 Write 5 New Done Done true
24054 8104 Acked 9103 Write 10 New Done Done true
24055 8105 Acked 9104 Write 8 New Done Done true
24056 8106 Acked 9105 Read 6 New Done Done true
24057 8107 Acked 9106 Write 9 New Done Done true
24058 8108 Acked 9107 Read 1 New Done Done true
24059 8109 Acked 9108 Read 6 New Done Done true
24060 8110 Acked 9109 Write 9 New Done Done true
24061 8111 Acked 9110 Read 7 New Done Done true
24062 8112 Acked 9111 Write 7 New Done Done true
24063 8113 Acked 9112 Write 8 New Done Done true
24064 8114 Acked 9113 Read 3 New Done Done true
24065 8115 Acked 9114 Read 3 New Done Done true
24066 8116 Acked 9115 Write 10 New Done Done true
24067 8117 Acked 9116 Write 2 New Done Done true
24068 8118 Acked 9117 Write 4 New Done Done true
24069 8119 Acked 9118 Write 1 New Done Done true
24070 8120 Acked 9119 Write 9 New Done Done true
24071 8121 Acked 9120 Write 4 New Done Done true
24072 8122 Acked 9121 Write 3 New Done Done true
24073 8123 Acked 9122 Flush 0 New Done Done true
24074 8124 Acked 9123 Flush 0 New Done Done true
24075 8125 Acked 9124 Read 1 New Done Done true
24076 8126 Acked 9125 Read 5 New Done Done true
24077 8127 Acked 9126 Write 2 New Done Done true
24078 8128 Acked 9127 Read 6 New Done Done true
24079 8129 Acked 9128 Read 7 New Done Done true
24080 8130 Acked 9129 Read 1 New Done Done true
24081 8131 Acked 9130 Read 5 New Done Done true
24082 8132 Acked 9131 Read 9 New Done Done true
24083 8133 Acked 9132 Read 10 New Done Done true
24084 8134 Acked 9133 Write 2 New Done Done true
24085 8135 Acked 9134 Read 4 New Done Done true
24086 8136 Acked 9135 Write 4 New Done Done true
24087 8137 Acked 9136 Write 9 New Done Done true
24088 8138 Acked 9137 Write 2 New Done Done true
24089 8139 Acked 9138 Flush 0 New Done Done true
24090 STATES DS:0 DS:1 DS:2 TOTAL
24091 New 301 0 0 301
24092 Sent 0 0 0 0
24093 Done 0 301 301 602
24094 Skipped 0 0 0 0
24095 Error 0 0 0 0
24096 Last Flush: 8837 9138 9138
24097 Downstairs last five completed: 8837 8836 8835 8834 8833
24098 Upstairs last five completed: 8139 8138 8137 8136 8135
24099 CLIENT: Up:0 ds:301 act:2
241002024-02-28T22:18:02.438ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241012024-02-28T22:18:02.438ZINFOcrucible: ds_connection connected = io task client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241022024-02-28T22:18:02.440ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241032024-02-28T22:18:02.440ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241042024-02-28T22:18:02.440ZINFOcrucible: send last flush ID to this DS: 8837 = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241052024-02-28T22:18:02.440ZINFOcrucible: Replied this last flush ID: 8837 = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241062024-02-28T22:18:02.440ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241072024-02-28T22:18:02.440ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241082024-02-28T22:18:02.440ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241092024-02-28T22:18:02.440ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
24110 ----------------------------------------------------------------
24111 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
24112 Downstairs last five completed: 9138 9137 9136 9135 9134
24113 Upstairs last five completed: 8139 8138 8137 8136 8135
24114 CLIENT: Up:0 ds:0 act:3
24115 Replay: All jobs finished, all DS active.
24116 Main task has work to do, go find it
24117 got dsc Stop(2)
24118 stop 2
24119 Feb 28 22:18:05.777 INFO request completed[8830] Got stop action so:false kr:false
24120 , latency_us: 328, response_code: 204, uri: /stop/cid/2, method: POST, req_id: e20688fa-2c13-441a-b507-e732ea9c9b6e, remote_addr: 127.0.0.1:41656, local_addr: 127.0.0.1:9998
241212024-02-28T22:18:05.778ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241222024-02-28T22:18:05.778ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241232024-02-28T22:18:05.778ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241242024-02-28T22:18:05.778ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241252024-02-28T22:18:05.778ZINFOcrucible: [2] client re-new 0 jobs since flush 9138 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241262024-02-28T22:18:05.778ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
24127 Feb 28 22:18:05.778 INFO request completed, latency_us: 162, response_code: 200, uri: /state/cid/2, method: GET, req_id: f508d8ea-4c6a-4215-a19e-c20e10c0d356, remote_addr: 127.0.0.1:41656, local_addr: 127.0.0.1:9998
24128 [8830] Exited with: ExitStatus(unix_wait_status(9))
24129 [8830][2] reports Exit
241302024-02-28T22:18:06.780ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241312024-02-28T22:18:06.780ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241322024-02-28T22:18:06.780ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241332024-02-28T22:18:06.780ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241342024-02-28T22:18:06.780ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241352024-02-28T22:18:06.780ZINFOcrucible: [2] client re-new 0 jobs since flush 9138 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241362024-02-28T22:18:06.780ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241372024-02-28T22:18:07.781ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241382024-02-28T22:18:07.782ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241392024-02-28T22:18:07.782ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241402024-02-28T22:18:07.782ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241412024-02-28T22:18:07.782ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241422024-02-28T22:18:07.782ZINFOcrucible: [2] client re-new 0 jobs since flush 9138 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241432024-02-28T22:18:07.782ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241442024-02-28T22:18:08.783ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241452024-02-28T22:18:08.783ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241462024-02-28T22:18:08.783ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241472024-02-28T22:18:08.783ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241482024-02-28T22:18:08.783ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241492024-02-28T22:18:08.783ZINFOcrucible: [2] client re-new 0 jobs since flush 9138 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241502024-02-28T22:18:08.783ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
24151 Feb 28 22:18:09.780 INFO request completed, latency_us: 204, response_code: 200, uri: /state/cid/2, method: GET, req_id: d182676e-dd5e-4152-bc0a-812cf7c1bd92, remote_addr: 127.0.0.1:41656, local_addr: 127.0.0.1:9998
24152 001/300 Read block 1156 len 20480
24153 002/300 Read block 1058 len 28672
24154 003/300 Read block 3736 len 32768
241552024-02-28T22:18:09.784ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241562024-02-28T22:18:09.784ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241572024-02-28T22:18:09.784ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241582024-02-28T22:18:09.784ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241592024-02-28T22:18:09.784ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241602024-02-28T22:18:09.784ZINFOcrucible: [2] client re-new 3 jobs since flush 9138 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
241612024-02-28T22:18:09.784ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
24162 004/300 Write block 54 len 36864 data: 4 4 6 6 6 7 8 8 8
24163 005/300 Write block 4127 len 32768 data: 7 7 8 7 5 6 5 5
24164 006/300 Write block 385 len 12288 data: 3 3 3
24165 007/300 Read block 3835 len 16384
24166 008/300 Write block 3674 len 4096 data: 4
24167 009/300 Read block 185 len 24576
24168 010/300 Read block 3900 len 28672
24169 011/300 Write block 4091 len 32768 data: 9 8 6 5 5 4 3 3
24170 012/300 Write block 829 len 16384 data: 5 4 5 5
24171 013/300 Flush
24172 014/300 Write block 347 len 16384 data: 8 6 5 5
24173 015/300 Write block 1768 len 24576 data: 5 6 8 9 7 6
24174 016/300 Flush
24175 017/300 Flush
24176 018/300 Read block 2861 len 36864
24177 019/300 Flush
24178 020/300 Flush
24179 021/300 Read block 2529 len 24576
24180 022/300 Read block 4805 len 20480
24181 023/300 Read block 152 len 36864
24182 024/300 Write block 3106 len 36864 data: 2 2 4 4 6 6 9 8 7
24183 025/300 Write block 835 len 40960 data: 4 5 4 4 5 5 4 4 5 6
24184 026/300 Write block 133 len 12288 data: 6 9 7
24185 027/300 Read block 997 len 24576
24186 028/300 Read block 4030 len 32768
24187 029/300 Read block 250 len 16384
24188 030/300 Write block 3328 len 32768 data: 5 5 4 5 5 5 5 6
24189 031/300 Write block 1284 len 36864 data: 6 4 4 4 4 5 5 4 4
24190 032/300 Write block 1433 len 36864 data: 6 5 5 5 4 3 5 6 4
24191 033/300 Read block 208 len 24576
24192 034/300 Read block 3426 len 20480
24193 035/300 Read block 2400 len 16384
24194 036/300 Read block 3495 len 28672
24195 037/300 Write block 829 len 36864 data: 6 5 6 6 5 4 5 6 5
24196 038/300 Read block 936 len 16384
24197 039/300 Write block 1286 len 8192 data: 5 5
24198 040/300 Read block 2844 len 4096
24199 041/300 Read block 3702 len 32768
24200 042/300 Read block 2641 len 40960
24201 043/300 Read block 3167 len 24576
24202 044/300 Flush
24203 045/300 Write block 4824 len 28672 data: 5 2 3 5 5 5 5
24204 046/300 Write block 3636 len 24576 data: 4 5 5 7 7 7
24205 047/300 Write block 3589 len 24576 data: 9 9 8 7 8 5
24206 048/300 Read block 359 len 28672
24207 049/300 Write block 1761 len 20480 data: 6 8 8 8 8
24208 050/300 Read block 3252 len 20480
24209 051/300 Read block 4087 len 20480
24210 052/300 Read block 780 len 28672
24211 053/300 Write block 225 len 12288 data: 5 6 3
24212 054/300 Write block 4084 len 36864 data: 3 4 5 4 7 8 8 10 9
24213 055/300 Write block 4347 len 36864 data: 6 5 6 4 4 5 5 4 4
24214 056/300 Read block 135 len 8192
24215 057/300 Read block 1638 len 28672
24216 058/300 Read block 2029 len 20480
24217 059/300 Read block 1571 len 16384
24218 060/300 Read block 4613 len 16384
24219 061/300 Write block 1213 len 28672 data: 4 6 8 8 8 8 7
24220 062/300 Write block 2703 len 4096 data: 5
24221 063/300 Write block 1649 len 8192 data: 4 3
24222 064/300 Read block 3384 len 32768
24223 065/300 Write block 3414 len 20480 data: 2 2 2 3 3
24224 066/300 Write block 4069 len 40960 data: 6 7 8 7 8 6 6 7 7 6
24225 067/300 Read block 3823 len 28672
24226 068/300 Read block 979 len 28672
24227 069/300 Write block 441 len 24576 data: 7 6 5 5 6 7
24228 070/300 Read block 1365 len 36864
24229 071/300 Write block 1658 len 40960 data: 2 3 4 3 3 5 3 4 4 4
24230 072/300 Read block 209 len 12288
24231 073/300 Read block 2797 len 20480
24232 074/300 Read block 4273 len 20480
24233 075/300 Read block 1832 len 36864
24234 076/300 Write block 2686 len 24576 data: 8 9 6 5 4 4
24235 077/300 Read block 4908 len 32768
24236 078/300 Write block 4585 len 28672 data: 6 3 4 4 5 7 8
24237 079/300 Write block 621 len 16384 data: 4 4 5 5
24238 080/300 Flush
24239 081/300 Read block 2846 len 24576
24240 082/300 Read block 2249 len 28672
24241 083/300 Read block 3595 len 24576
24242 084/300 Read block 2661 len 4096
24243 085/300 Write block 1525 len 32768 data: 6 6 4 5 5 4 5 6
24244 086/300 Read block 794 len 20480
24245 087/300 Flush
24246 088/300 Write block 1443 len 4096 data: 4
24247 089/300 Write block 1464 len 12288 data: 4 5 5
24248 090/300 Read block 3913 len 8192
24249 091/300 Write block 2193 len 16384 data: 6 8 7 5
24250 092/300 Write block 2961 len 40960 data: 5 4 5 5 4 3 4 4 5 5
24251 093/300 Read block 1247 len 24576
24252 094/300 Read block 3433 len 4096
24253 095/300 Write block 3834 len 40960 data: 4 5 4 5 4 5 4 4 5 4
24254 096/300 Write block 1287 len 32768 data: 6 5 6 6 5 5 3 2
24255 097/300 Write block 3292 len 8192 data: 9 8
24256 098/300 Read block 1867 len 4096
24257 099/300 Flush
24258 100/300 Read block 1212 len 36864
24259 101/300 Read block 2557 len 24576
24260 102/300 Write block 2309 len 36864 data: 2 4 4 4 4 4 4 7 6
24261 103/300 Read block 3581 len 16384
24262 104/300 Flush
24263 105/300 Read block 2996 len 24576
24264 106/300 Write block 928 len 36864 data: 6 6 6 6 7 5 5 6 5
24265 107/300 Write block 3772 len 28672 data: 5 7 8 7 7 7 6
24266 108/300 Write block 4923 len 36864 data: 8 9 9 9 10 10 9 8 6
24267 109/300 Write block 825 len 36864 data: 5 4 6 4 7 6 7 7 6
24268 110/300 Read block 923 len 24576
24269 111/300 Read block 3124 len 16384
24270 112/300 Read block 4171 len 16384
24271 113/300 Write block 846 len 24576 data: 10 9 8 7 7 7
24272 114/300 Write block 695 len 28672 data: 6 6 5 5 7 6 6
24273 115/300 Write block 1195 len 12288 data: 7 7 5
24274 116/300 Read block 438 len 12288
24275 117/300 Write block 4617 len 36864 data: 6 7 8 10 9 7 7 7 5
24276 118/300 Read block 4221 len 28672
24277 119/300 Write block 4488 len 24576 data: 3 2 4 5 5 5
24278 120/300 Read block 4100 len 36864
24279 121/300 Write block 560 len 40960 data: 4 6 6 6 5 4 3 3 4 4
24280 122/300 Write block 1182 len 16384 data: 8 7 7 7
24281 123/300 Write block 3090 len 12288 data: 4 4 5
24282 124/300 Flush
24283 125/300 Read block 3872 len 32768
24284 126/300 Write block 574 len 24576 data: 8 9 5 6 6 5
24285 127/300 Write block 881 len 24576 data: 5 3 3 4 4 5
24286 128/300 Write block 580 len 28672 data: 6 5 6 7 7 6 6
24287 129/300 Read block 3797 len 12288
24288 130/300 Write block 3070 len 36864 data: 5 6 6 7 7 7 6 8 7
24289 131/300 Write block 4395 len 40960 data: 3 5 4 4 5 6 7 7 6 5
24290 132/300 Read block 4304 len 40960
24291 133/300 Read block 1334 len 28672
24292 134/300 Read block 2037 len 12288
24293 135/300 Read block 1146 len 24576
24294 136/300 Read block 2866 len 20480
24295 137/300 Read block 1633 len 36864
24296 138/300 Write block 3763 len 20480 data: 4 3 3 4 3
24297 139/300 Write block 1016 len 16384 data: 3 3 3 4
24298 140/300 Write block 3832 len 4096 data: 3
24299 141/300 Flush
24300 142/300 Write block 3578 len 40960 data: 8 5 5 6 5 5 6 8 9 8
24301 143/300 Write block 3286 len 24576 data: 5 5 7 7 7 9
24302 144/300 Write block 4129 len 20480 data: 9 8 6 7 6
24303 145/300 Read block 1452 len 28672
24304 146/300 Write block 4111 len 16384 data: 5 4 3 5
24305 147/300 Read block 3448 len 24576
24306 148/300 Read block 2218 len 32768
24307 149/300 Read block 3513 len 20480
24308 150/300 Read block 1795 len 20480
24309 151/300 Write block 763 len 12288 data: 7 6 4
24310 152/300 Read block 3739 len 24576
24311 153/300 Write block 2792 len 12288 data: 5 4 6
24312 154/300 Write block 2795 len 32768 data: 7 6 6 6 7 7 6 5
24313 155/300 Read block 2225 len 12288
24314 156/300 Read block 4406 len 28672
24315 157/300 Write block 2706 len 40960 data: 7 6 4 3 3 2 3 3 3 4
24316 158/300 Flush
24317 159/300 Write block 1509 len 16384 data: 3 3 3 3
24318 160/300 Read block 3847 len 24576
24319 161/300 Write block 126 len 20480 data: 5 4 3 3 3
24320 162/300 Write block 3911 len 8192 data: 6 7
24321 163/300 Write block 4205 len 24576 data: 6 5 5 3 4 6
24322 164/300 Flush
24323 165/300 Write block 3246 len 32768 data: 7 8 6 8 7 8 8 8
24324 166/300 Read block 1911 len 4096
24325 167/300 Write block 3423 len 32768 data: 4 6 6 5 5 6 5 7
24326 168/300 Read block 3564 len 16384
24327 169/300 Write block 2994 len 24576 data: 4 4 3 3 3 4
24328 170/300 Read block 3853 len 40960
24329 171/300 Read block 4908 len 36864
24330 172/300 Read block 1029 len 20480
24331 173/300 Write block 3254 len 8192 data: 8 6
24332 174/300 Read block 4303 len 24576
24333 175/300 Write block 4710 len 20480 data: 4 4 6 6 8
24334 176/300 Read block 657 len 28672
24335 177/300 Write block 1116 len 40960 data: 5 5 4 5 4 5 3 5 8 10
24336 178/300 Read block 4262 len 32768
24337 179/300 Read block 2098 len 32768
24338 180/300 Read block 1033 len 4096
24339 181/300 Write block 2732 len 24576 data: 8 6 6 6 4 3
24340 182/300 Read block 1339 len 8192
24341 183/300 Read block 4239 len 32768
24342 184/300 Read block 1922 len 36864
24343 185/300 Write block 3400 len 32768 data: 7 7 7 7 7 7 6 9
24344 186/300 Read block 2157 len 8192
24345 187/300 Read block 3963 len 8192
24346 188/300 Read block 1720 len 32768
24347 189/300 Write block 2843 len 12288 data: 5 5 4
24348 190/300 Read block 924 len 8192
24349 191/300 Write block 4876 len 24576 data: 2 6 9 10 9 9
24350 192/300 Read block 3771 len 24576
24351 193/300 Flush
24352 194/300 Write block 697 len 8192 data: 6 6
24353 195/300 Read block 4548 len 4096
24354 196/300 Read block 4881 len 24576
24355 197/300 Read block 2023 len 20480
24356 198/300 Flush
24357 199/300 Write block 4673 len 32768 data: 5 5 6 5 5 4 4 3
24358 200/300 Write block 4100 len 12288 data: 3 3 4
24359 201/300 Write block 699 len 36864 data: 8 7 7 5 4 3 3 5 5
24360 202/300 Read block 3879 len 24576
24361 203/300 Write block 2686 len 8192 data: 9 10
24362 204/300 Read block 1255 len 12288
24363 205/300 Write block 4793 len 36864 data: 4 5 5 6 6 7 6 5 9
24364 206/300 Read block 1390 len 20480
24365 207/300 Write block 1609 len 16384 data: 6 6 7 7
24366 208/300 Write block 1515 len 36864 data: 5 8 9 12 10 10 9 9 8
24367 209/300 Write block 2353 len 24576 data: 5 4 4 3 4 3
24368 210/300 Write block 1884 len 20480 data: 6 7 7 5 5
24369 211/300 Read block 4202 len 16384
24370 212/300 Read block 4453 len 4096
24371 213/300 Read block 29 len 8192
24372 214/300 Flush
24373 215/300 Write block 1345 len 16384 data: 5 6 7 7
24374 216/300 Write block 1718 len 12288 data: 9 6 7
24375 217/300 Flush
24376 218/300 Write block 3390 len 20480 data: 6 7 8 8 9
24377 219/300 Flush
24378 220/300 Write block 1648 len 4096 data: 3
24379 221/300 Read block 4900 len 8192
24380 222/300 Write block 3123 len 20480 data: 9 9 6 6 5
24381 223/300 Flush
24382 224/300 Read block 4869 len 36864
24383 225/300 Read block 4686 len 28672
24384 226/300 Read block 4274 len 40960
24385 227/300 Read block 4498 len 28672
24386 228/300 Read block 3314 len 20480
24387 229/300 Read block 3676 len 24576
24388 230/300 Read block 1207 len 40960
24389 231/300 Read block 837 len 40960
24390 232/300 Write block 2546 len 36864 data: 7 5 4 4 3 6 6 6 6
24391 233/300 Read block 3983 len 16384
24392 234/300 Write block 3345 len 24576 data: 5 4 4 4 5 5
24393 235/300 Write block 2152 len 12288 data: 3 4 4
24394 236/300 Read block 1097 len 36864
24395 237/300 Write block 2484 len 24576 data: 5 6 6 5 5 5
24396 238/300 Read block 4674 len 28672
24397 239/300 Read block 2227 len 16384
24398 240/300 Write block 2803 len 4096 data: 5
24399 241/300 Read block 985 len 12288
24400 242/300 Write block 1359 len 20480 data: 3 4 3 4 4
24401 243/300 Write block 3802 len 20480 data: 3 3 4 5 5
24402 244/300 Read block 1316 len 36864
24403 245/300 Write block 4215 len 28672 data: 6 6 7 5 4 4 5
24404 246/300 Flush
24405 247/300 Write block 2582 len 8192 data: 7 5
24406 248/300 Write block 3726 len 20480 data: 11 10 10 11 10
24407 249/300 Flush
24408 250/300 Write block 4453 len 24576 data: 5 5 5 5 4 5
24409 251/300 Read block 1083 len 16384
24410 252/300 Read block 4567 len 24576
24411 253/300 Read block 861 len 4096
24412 254/300 Read block 945 len 24576
24413 255/300 Flush
24414 256/300 Write block 843 len 24576 data: 6 7 6 11 10 9
24415 257/300 Read block 4711 len 12288
24416 258/300 Write block 2540 len 16384 data: 8 9 9 8
24417 259/300 Read block 4855 len 4096
24418 260/300 Write block 2206 len 8192 data: 6 5
24419 261/300 Read block 1912 len 12288
24420 262/300 Read block 1167 len 20480
24421 263/300 Write block 2476 len 40960 data: 3 4 4 4 4 4 4 3 6 7
24422 264/300 Read block 2618 len 8192
24423 265/300 Read block 1892 len 24576
24424 266/300 Write block 3191 len 32768 data: 6 6 4 4 4 4 6 6
24425 267/300 Write block 2349 len 28672 data: 5 6 5 5 6 5 5
24426 268/300 Read block 68 len 24576
24427 269/300 Write block 161 len 28672 data: 7 5 5 5 4 3 3
24428 270/300 Write block 2133 len 8192 data: 9 7
24429 271/300 Read block 4284 len 16384
24430 272/300 Read block 2902 len 4096
24431 273/300 Write block 3304 len 8192 data: 4 5
24432 274/300 Flush
24433 275/300 Flush
24434 276/300 Write block 4332 len 16384 data: 9 7 7 10
24435 277/300 Write block 2323 len 16384 data: 6 5 6 3
24436 278/300 Write block 568 len 36864 data: 5 5 5 6 8 8 9 10 6
24437 279/300 Read block 3499 len 4096
24438 280/300 Read block 936 len 20480
24439 281/300 Write block 516 len 24576 data: 6 4 5 6 6 5
24440 282/300 Write block 1744 len 4096 data: 4
24441 283/300 Write block 4801 len 8192 data: 10 9
24442 284/300 Write block 4884 len 28672 data: 10 10 7 7 7 8 5
24443 285/300 Write block 3835 len 16384 data: 6 5 6 5
24444 286/300 Read block 4687 len 32768
24445 287/300 Read block 4647 len 40960
24446 288/300 Write block 748 len 20480 data: 5 6 8 8 10
24447 289/300 Read block 1492 len 12288
24448 290/300 Write block 971 len 20480 data: 2 3 4 4 4
24449 291/300 Write block 4013 len 40960 data: 12 13 11 11 7 8 10 9 5 4
24450 292/300 Write block 3896 len 8192 data: 7 6
24451 293/300 Read block 2073 len 28672
24452 294/300 Write block 3656 len 32768 data: 5 4 4 4 4 4 5 4
24453 295/300 Read block 182 len 40960
24454 296/300 Read block 4348 len 8192
24455 297/300 Read block 4622 len 4096
24456 298/300 Write block 2674 len 36864 data: 8 6 7 10 11 11 10 9 9
24457 299/300 Write block 3376 len 40960 data: 4 4 3 3 4 7 8 7 5 3
24458 300/300 Read block 2929 len 20480
24459 Main task has work to do, go find it
24460 got dsc Start(2)
24461 start 2
24462 Feb 28 22:18:10.117 INFO request completed, latency_us: 239, response_code: 204, uri: /start/cid/2, method: POST, req_id: ebc88989-b888-4cad-9e3b-9eeea9c05dc7, remote_addr: 127.0.0.1:41656, local_addr: 127.0.0.1:9998
24463 Replay: started 2, returned:Ok(())
24464 ----------------------------------------------------------------
24465 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24466 States: Active Active Offline
24467 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
24468 8140 Acked 9139 Read 5 Done Done New true
24469 8141 Acked 9140 Read 7 Done Done New true
24470 8142 Acked 9141 Read 8 Done Done New true
24471 8143 Acked 9142 Write 9 Done Done New false
24472 8144 Acked 9143 Write 8 Done Done New false
24473 8145 Acked 9144 Write 3 Done Done New false
24474 8146 Acked 9145 Read 4 Done Done New false
24475 8147 Acked 9146 Write 1 Done Done New false
24476 8148 Acked 9147 Read 6 Done Done New false
24477 8149 Acked 9148 Read 7 Done Done New false
24478 8150 Acked 9149 Write 8 Done Done New false
24479 8151 Acked 9150 Write 4 Done Done New false
24480 8152 Acked 9151 Flush 0 Done Done New false
24481 8153 Acked 9152 Write 4 Done Done New false
24482 8154 Acked 9153 Write 6 Done Done New false
24483 8155 Acked 9154 Flush 0 Done Done New false
24484 8156 Acked 9155 Flush 0 Done Done New false
24485 8157 Acked 9156 Read 9 Done Done New false
24486 8158 Acked 9157 Flush 0 Done Done New false
24487 8159 Acked 9158 Flush 0 Done Done New false
24488 8160 Acked 9159 Read 6 Done Done New false
24489 8161 Acked 9160 Read 5 Done Done New false
24490 8162 Acked 9161 Read 9 Done Done New false
24491 8163 Acked 9162 Write 9 Done Done New false
24492 8164 Acked 9163 Write 10 Done Done New false
24493 8165 Acked 9164 Write 3 Done Done New false
24494 8166 Acked 9165 Read 6 Done Done New false
24495 8167 Acked 9166 Read 8 Done Done New false
24496 8168 Acked 9167 Read 4 Done Done New false
24497 8169 Acked 9168 Write 8 Done Done New false
24498 8170 Acked 9169 Write 9 Done Done New false
24499 8171 Acked 9170 Write 9 Done Done New false
24500 8172 Acked 9171 Read 6 Done Done New false
24501 8173 Acked 9172 Read 5 Done Done New false
24502 8174 Acked 9173 Read 4 Done Done New false
24503 8175 Acked 9174 Read 7 Done Done New false
24504 8176 Acked 9175 Write 9 Done Done New false
24505 8177 Acked 9176 Read 4 Done Done New false
24506 8178 Acked 9177 Write 2 Done Done New false
24507 8179 Acked 9178 Read 1 Done Done New false
24508 8180 Acked 9179 Read 8 Done Done New false
24509 8181 Acked 9180 Read 10 Done Done New false
24510 8182 Acked 9181 Read 6 Done Done New false
24511 8183 Acked 9182 Flush 0 Done Done New false
24512 8184 Acked 9183 Write 7 Done Done New false
24513 8185 Acked 9184 Write 6 Done Done New false
24514 8186 Acked 9185 Write 6 Done Done New false
24515 8187 Acked 9186 Read 7 Done Done New false
24516 8188 Acked 9187 Write 5 Done Done New false
24517 8189 Acked 9188 Read 5 Done Done New false
24518 8190 Acked 9189 Read 5 Done Done New false
24519 8191 Acked 9190 Read 7 Done Done New false
24520 8192 Acked 9191 Write 3 Done Done New false
24521 8193 Acked 9192 Write 9 Done Done New false
24522 8194 Acked 9193 Write 9 Done Done New false
24523 8195 Acked 9194 Read 2 Done Done New false
24524 8196 Acked 9195 Read 7 Done Done New false
24525 8197 Acked 9196 Read 5 Done Done New false
24526 8198 Acked 9197 Read 4 Done Done New false
24527 8199 Acked 9198 Read 4 Done Done New false
24528 8200 Acked 9199 Write 7 Done Done New false
24529 8201 Acked 9200 Write 1 Done Done New false
24530 8202 Acked 9201 Write 2 Done Done New false
24531 8203 Acked 9202 Read 8 Done Done New false
24532 8204 Acked 9203 Write 5 Done Done New false
24533 8205 Acked 9204 Write 10 Done Done New false
24534 8206 Acked 9205 Read 7 Done Done New false
24535 8207 Acked 9206 Read 7 Done Done New false
24536 8208 Acked 9207 Write 6 Done Done New false
24537 8209 Acked 9208 Read 9 Done Done New false
24538 8210 Acked 9209 Write 10 Done Done New false
24539 8211 Acked 9210 Read 3 Done Done New false
24540 8212 Acked 9211 Read 5 Done Done New false
24541 8213 Acked 9212 Read 5 Done Done New false
24542 8214 Acked 9213 Read 9 Done Done New false
24543 8215 Acked 9214 Write 6 Done Done New false
24544 8216 Acked 9215 Read 8 Done Done New false
24545 8217 Acked 9216 Write 7 Done Done New false
24546 8218 Acked 9217 Write 4 Done Done New false
24547 8219 Acked 9218 Flush 0 Done Done New false
24548 8220 Acked 9219 Read 6 Done Done New false
24549 8221 Acked 9220 Read 7 Done Done New false
24550 8222 Acked 9221 Read 6 Done Done New false
24551 8223 Acked 9222 Read 1 Done Done New false
24552 8224 Acked 9223 Write 8 Done Done New false
24553 8225 Acked 9224 Read 5 Done Done New false
24554 8226 Acked 9225 Flush 0 Done Done New false
24555 8227 Acked 9226 Write 1 Done Done New false
24556 8228 Acked 9227 Write 3 Done Done New false
24557 8229 Acked 9228 Read 2 Done Done New false
24558 8230 Acked 9229 Write 4 Done Done New false
24559 8231 Acked 9230 Write 10 Done Done New false
24560 8232 Acked 9231 Read 6 Done Done New false
24561 8233 Acked 9232 Read 1 Done Done New false
24562 8234 Acked 9233 Write 10 Done Done New false
24563 8235 Acked 9234 Write 8 Done Done New false
24564 8236 Acked 9235 Write 2 Done Done New false
24565 8237 Acked 9236 Read 1 Done Done New false
24566 8238 Acked 9237 Flush 0 Done Done New false
24567 8239 Acked 9238 Read 9 Done Done New false
24568 8240 Acked 9239 Read 6 Done Done New false
24569 8241 Acked 9240 Write 9 Done Done New false
24570 8242 Acked 9241 Read 4 Done Done New false
24571 8243 Acked 9242 Flush 0 Done Done New false
24572 8244 Acked 9243 Read 6 Done Done New false
24573 8245 Acked 9244 Write 9 Done Done New false
24574 8246 Acked 9245 Write 7 Done Done New false
24575 8247 Acked 9246 Write 9 Done Done New false
24576 8248 Acked 9247 Write 9 Done Done New false
24577 8249 Acked 9248 Read 6 Done Done New false
24578 8250 Acked 9249 Read 4 Done Done New false
24579 8251 Acked 9250 Read 4 Done Done New false
24580 8252 Acked 9251 Write 6 Done Done New false
24581 8253 Acked 9252 Write 7 Done Done New false
24582 8254 Acked 9253 Write 3 Done Done New false
24583 8255 Acked 9254 Read 3 Done Done New false
24584 8256 Acked 9255 Write 9 Done Done New false
24585 8257 Acked 9256 Read 7 Done Done New false
24586 8258 Acked 9257 Write 6 Done Done New false
24587 8259 Acked 9258 Read 9 Done Done New false
24588 8260 Acked 9259 Write 10 Done Done New false
24589 8261 Acked 9260 Write 4 Done Done New false
24590 8262 Acked 9261 Write 3 Done Done New false
24591 8263 Acked 9262 Flush 0 Done Done New false
24592 8264 Acked 9263 Read 8 Done Done New false
24593 8265 Acked 9264 Write 6 Done Done New false
24594 8266 Acked 9265 Write 6 Done Done New false
24595 8267 Acked 9266 Write 7 Done Done New false
24596 8268 Acked 9267 Read 3 Done Done New false
24597 8269 Acked 9268 Write 9 Done Done New false
24598 8270 Acked 9269 Write 10 Done Done New false
24599 8271 Acked 9270 Read 10 Done Done New false
24600 8272 Acked 9271 Read 7 Done Done New false
24601 8273 Acked 9272 Read 3 Done Done New false
24602 8274 Acked 9273 Flush 0 Done Done New false
24603 8275 Acked 9274 Read 6 Done Done New false
24604 8276 Acked 9275 Read 5 Done Done New false
24605 8277 Acked 9276 Read 9 Done Done New false
24606 8278 Acked 9277 Write 5 Done Done New false
24607 8279 Acked 9278 Write 4 Done Done New false
24608 8280 Acked 9279 Write 1 Done Done New false
24609 8281 Acked 9280 Flush 0 Done Done New false
24610 8282 Acked 9281 Write 10 Done Done New false
24611 8283 Acked 9282 Write 6 Done Done New false
24612 8284 Acked 9283 Write 5 Done Done New false
24613 8285 Acked 9284 Read 7 Done Done New false
24614 8286 Acked 9285 Write 4 Done Done New false
24615 8287 Acked 9286 Read 6 Done Done New false
24616 8288 Acked 9287 Read 8 Done Done New false
24617 8289 Acked 9288 Read 5 Done Done New false
24618 8290 Acked 9289 Read 5 Done Done New false
24619 8291 Acked 9290 Write 3 Done Done New false
24620 8292 Acked 9291 Read 6 Done Done New false
24621 8293 Acked 9292 Write 3 Done Done New false
24622 8294 Acked 9293 Write 8 Done Done New false
24623 8295 Acked 9294 Read 3 Done Done New false
24624 8296 Acked 9295 Read 7 Done Done New false
24625 8297 Acked 9296 Write 10 Done Done New false
24626 8298 Acked 9297 Flush 0 Done Done New false
24627 8299 Acked 9298 Write 4 Done Done New false
24628 8300 Acked 9299 Read 6 Done Done New false
24629 8301 Acked 9300 Write 5 Done Done New false
24630 8302 Acked 9301 Write 2 Done Done New false
24631 8303 Acked 9302 Write 6 Done Done New false
24632 8304 Acked 9303 Flush 0 Done Done New false
24633 8305 Acked 9304 Write 8 Done Done New false
24634 8306 Acked 9305 Read 1 Done Done New false
24635 8307 Acked 9306 Write 8 Done Done New false
24636 8308 Acked 9307 Read 4 Done Done New false
24637 8309 Acked 9308 Write 6 Done Done New false
24638 8310 Acked 9309 Read 10 Done Done New false
24639 8311 Acked 9310 Read 9 Done Done New false
24640 8312 Acked 9311 Read 5 Done Done New false
24641 8313 Acked 9312 Write 2 Done Done New false
24642 8314 Acked 9313 Read 6 Done Done New false
24643 8315 Acked 9314 Write 5 Done Done New false
24644 8316 Acked 9315 Read 7 Done Done New false
24645 8317 Acked 9316 Write 10 Done Done New false
24646 8318 Acked 9317 Read 8 Done Done New false
24647 8319 Acked 9318 Read 8 Done Done New false
24648 8320 Acked 9319 Read 1 Done Done New false
24649 8321 Acked 9320 Write 6 Done Done New false
24650 8322 Acked 9321 Read 2 Done Done New false
24651 8323 Acked 9322 Read 8 Done Done New false
24652 8324 Acked 9323 Read 9 Done Done New false
24653 8325 Acked 9324 Write 8 Done Done New false
24654 8326 Acked 9325 Read 2 Done Done New false
24655 8327 Acked 9326 Read 2 Done Done New false
24656 8328 Acked 9327 Read 8 Done Done New false
24657 8329 Acked 9328 Write 3 Done Done New false
24658 8330 Acked 9329 Read 2 Done Done New false
24659 8331 Acked 9330 Write 6 Done Done New false
24660 8332 Acked 9331 Read 6 Done Done New false
24661 8333 Acked 9332 Flush 0 Done Done New false
24662 8334 Acked 9333 Write 2 Done Done New false
24663 8335 Acked 9334 Read 1 Done Done New false
24664 8336 Acked 9335 Read 6 Done Done New false
24665 8337 Acked 9336 Read 5 Done Done New false
24666 8338 Acked 9337 Flush 0 Done Done New false
24667 8339 Acked 9338 Write 8 Done Done New false
24668 8340 Acked 9339 Write 3 Done Done New false
24669 8341 Acked 9340 Write 9 Done Done New false
24670 8342 Acked 9341 Read 6 Done Done New false
24671 8343 Acked 9342 Write 2 Done Done New false
24672 8344 Acked 9343 Read 3 Done Done New false
24673 8345 Acked 9344 Write 9 Done Done New false
24674 8346 Acked 9345 Read 5 Done Done New false
24675 8347 Acked 9346 Write 4 Done Done New false
24676 8348 Acked 9347 Write 9 Done Done New false
24677 8349 Acked 9348 Write 6 Done Done New false
24678 8350 Acked 9349 Write 5 Done Done New false
24679 8351 Acked 9350 Read 4 Done Done New false
24680 8352 Acked 9351 Read 1 Done Done New false
24681 8353 Acked 9352 Read 2 Done Done New false
24682 8354 Acked 9353 Flush 0 Done Done New false
24683 8355 Acked 9354 Write 4 Done Done New false
24684 8356 Acked 9355 Write 3 Done Done New false
24685 8357 Acked 9356 Flush 0 Done Done New false
24686 8358 Acked 9357 Write 5 Done Done New false
24687 8359 Acked 9358 Flush 0 Done Done New false
24688 8360 Acked 9359 Write 1 Done Done New false
24689 8361 Acked 9360 Read 2 Done Done New false
24690 8362 Acked 9361 Write 5 Done Done New false
24691 8363 Acked 9362 Flush 0 Done Done New false
24692 8364 Acked 9363 Read 9 Done Done New false
24693 8365 Acked 9364 Read 7 Done Done New false
24694 8366 Acked 9365 Read 10 Done Done New false
24695 8367 Acked 9366 Read 7 Done Done New false
24696 8368 Acked 9367 Read 5 Done Done New false
24697 8369 Acked 9368 Read 6 Done Done New false
24698 8370 Acked 9369 Read 10 Done Done New false
24699 8371 Acked 9370 Read 10 Done Done New false
24700 8372 Acked 9371 Write 9 Done Done New false
24701 8373 Acked 9372 Read 4 Done Done New false
24702 8374 Acked 9373 Write 6 Done Done New false
24703 8375 Acked 9374 Write 3 Done Done New false
24704 8376 Acked 9375 Read 9 Done Done New false
24705 8377 Acked 9376 Write 6 Done Done New false
24706 8378 Acked 9377 Read 7 Done Done New false
24707 8379 Acked 9378 Read 4 Done Done New false
24708 8380 Acked 9379 Write 1 Done Done New false
24709 8381 Acked 9380 Read 3 Done Done New false
24710 8382 Acked 9381 Write 5 Done Done New false
24711 8383 Acked 9382 Write 5 Done Done New false
24712 8384 Acked 9383 Read 9 Done Done New false
24713 8385 Acked 9384 Write 7 Done Done New false
24714 8386 Acked 9385 Flush 0 Done Done New false
24715 8387 Acked 9386 Write 2 Done Done New false
24716 8388 Acked 9387 Write 5 Done Done New false
24717 8389 Acked 9388 Flush 0 Done Done New false
24718 8390 Acked 9389 Write 6 Done Done New false
24719 8391 Acked 9390 Read 4 Done Done New false
24720 8392 Acked 9391 Read 6 Done Done New false
24721 8393 Acked 9392 Read 1 Done Done New false
24722 8394 Acked 9393 Read 6 Done Done New false
24723 8395 Acked 9394 Flush 0 Done Done New false
24724 8396 Acked 9395 Write 6 Done Done New false
24725 8397 Acked 9396 Read 3 Done Done New false
24726 8398 Acked 9397 Write 4 Done Done New false
24727 8399 Acked 9398 Read 1 Done Done New false
24728 8400 Acked 9399 Write 2 Done Done New false
24729 8401 Acked 9400 Read 3 Done Done New false
24730 8402 Acked 9401 Read 5 Done Done New false
24731 8403 Acked 9402 Write 10 Done Done New false
24732 8404 Acked 9403 Read 2 Done Done New false
24733 8405 Acked 9404 Read 6 Done Done New false
24734 8406 Acked 9405 Write 8 Done Done New false
24735 8407 Acked 9406 Write 7 Done Done New false
24736 8408 Acked 9407 Read 6 Done Done New false
24737 8409 Acked 9408 Write 7 Done Done New false
24738 8410 Acked 9409 Write 2 Done Done New false
24739 8411 Acked 9410 Read 4 Done Done New false
24740 8412 Acked 9411 Read 1 Done Done New false
24741 8413 Acked 9412 Write 2 Done Done New false
24742 8414 Acked 9413 Flush 0 Done Done New false
24743 8415 Acked 9414 Flush 0 Done Done New false
24744 8416 Acked 9415 Write 4 Done Done New false
24745 8417 Acked 9416 Write 4 Done Done New false
24746 8418 Acked 9417 Write 9 Done Done New false
24747 8419 Acked 9418 Read 1 Done Done New false
24748 8420 Acked 9419 Read 5 Done Done New false
24749 8421 Acked 9420 Write 6 Done Done New false
24750 8422 Acked 9421 Write 1 Done Done New false
24751 8423 Acked 9422 Write 2 Done Done New false
24752 8424 Acked 9423 Write 7 Done Done New false
24753 8425 Acked 9424 Write 4 Done Done New false
24754 8426 Acked 9425 Read 8 Done Done New false
24755 8427 Acked 9426 Read 10 Done Done New false
24756 8428 Acked 9427 Write 5 Done Done New false
24757 8429 Acked 9428 Read 3 Done Done New false
24758 8430 Acked 9429 Write 5 Done Done New false
24759 8431 Acked 9430 Write 10 Done Done New false
24760 8432 Acked 9431 Write 2 Done Done New false
24761 8433 Acked 9432 Read 7 Done Done New false
24762 8434 Acked 9433 Write 8 Done Done New false
24763 8435 Acked 9434 Read 10 Done Done New false
24764 8436 Acked 9435 Read 2 Done Done New false
24765 8437 Acked 9436 Read 1 Done Done New false
24766 8438 Acked 9437 Write 9 Done Done New false
24767 8439 Acked 9438 Write 10 Done Done New false
24768 8440 Acked 9439 Read 5 Done Done New false
24769 STATES DS:0 DS:1 DS:2 TOTAL
24770 New 0 0 301 301
24771 Sent 0 0 0 0
24772 Done 301 301 0 602
24773 Skipped 0 0 0 0
24774 Error 0 0 0 0
24775 Last Flush: 9414 9414 9138
24776 Downstairs last five completed: 9138 9137 9136 9135 9134
24777 Upstairs last five completed: 8440 8439 8438 8437 8436
24778 CLIENT: Up:0 ds:301 act:2
247792024-02-28T22:18:10.785ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
247802024-02-28T22:18:10.785ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
247812024-02-28T22:18:10.785ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
247822024-02-28T22:18:10.785ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
247832024-02-28T22:18:10.785ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
247842024-02-28T22:18:10.785ZINFOcrucible: [2] client re-new 302 jobs since flush 9138 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
247852024-02-28T22:18:10.785ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
24786 [8830] got start action
24787 [8830] I am going to restart
24788 Starting downstairs at port 8830
24789 Make output file at "/tmp/dsc/downstairs-8830.txt"
24790 [8830][2] reports Starting
247912024-02-28T22:18:11.786ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
247922024-02-28T22:18:11.786ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
247932024-02-28T22:18:11.786ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
247942024-02-28T22:18:11.786ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
247952024-02-28T22:18:11.786ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
247962024-02-28T22:18:11.786ZINFOcrucible: [2] client re-new 302 jobs since flush 9138 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
247972024-02-28T22:18:11.786ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
24798 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1192)
24799 [8830][2] reports Running
248002024-02-28T22:18:12.787ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248012024-02-28T22:18:12.787ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248022024-02-28T22:18:12.788ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248032024-02-28T22:18:12.788ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248042024-02-28T22:18:12.788ZINFOcrucible: send last flush ID to this DS: 9138 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248052024-02-28T22:18:12.789ZINFOcrucible: Replied this last flush ID: 9138 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248062024-02-28T22:18:12.789ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248072024-02-28T22:18:12.789ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248082024-02-28T22:18:12.789ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248092024-02-28T22:18:12.789ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
24810 ----------------------------------------------------------------
24811 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
24812 Downstairs last five completed: 9440 9439 9438 9437 9436
24813 Upstairs last five completed: 8441 8440 8439 8438 8437
24814 CLIENT: Up:0 ds:0 act:3
24815 Replay: All jobs finished, all DS active.
24816 Main task has work to do, go find it
24817 got dsc Stop(2)
24818 stop 2
24819 Feb 28 22:18:14.120 INFO request completed, latency_us: 339, response_code:[8830] Got stop action so:false kr:false
24820 204, uri: /stop/cid/2, method: POST, req_id: 0cf77dca-1a40-497c-96c3-584345b9db2c, remote_addr: 127.0.0.1:41656, local_addr: 127.0.0.1:9998
248212024-02-28T22:18:14.121ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248222024-02-28T22:18:14.121ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248232024-02-28T22:18:14.121ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248242024-02-28T22:18:14.121ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248252024-02-28T22:18:14.121ZINFOcrucible: [2] client re-new 0 jobs since flush 9440 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248262024-02-28T22:18:14.121ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
24827 Feb 28 22:18:14.121 INFO request completed, latency_us: 145, response_code: 200, uri: /state/cid/2, method: GET, req_id: f84c93cb-acf1-429b-b788-c49b400a220a, remote_addr: 127.0.0.1:41656, local_addr: 127.0.0.1:9998
24828 [8830] Exited with: ExitStatus(unix_wait_status(9))
24829 [8830][2] reports Exit
248302024-02-28T22:18:15.122ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248312024-02-28T22:18:15.122ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248322024-02-28T22:18:15.122ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248332024-02-28T22:18:15.122ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248342024-02-28T22:18:15.122ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248352024-02-28T22:18:15.122ZINFOcrucible: [2] client re-new 0 jobs since flush 9440 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248362024-02-28T22:18:15.122ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248372024-02-28T22:18:16.123ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248382024-02-28T22:18:16.123ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248392024-02-28T22:18:16.123ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248402024-02-28T22:18:16.123ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248412024-02-28T22:18:16.123ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248422024-02-28T22:18:16.123ZINFOcrucible: [2] client re-new 0 jobs since flush 9440 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248432024-02-28T22:18:16.123ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248442024-02-28T22:18:17.125ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248452024-02-28T22:18:17.125ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248462024-02-28T22:18:17.125ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248472024-02-28T22:18:17.125ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248482024-02-28T22:18:17.125ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248492024-02-28T22:18:17.125ZINFOcrucible: [2] client re-new 0 jobs since flush 9440 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248502024-02-28T22:18:17.125ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
24851 Feb 28 22:18:18.124 INFO request completed, latency_us: 250, response_code: 200, uri: /state/cid/2, method: GET, req_id: 21a5cba9-4251-4082-9e70-a3bb15ab7f8d, remote_addr: 127.0.0.1:41656, local_addr: 127.0.0.1:9998
24852 001/300 Read block 1844 len 20480
24853 002/300 Write block 4019 len 16384 data: 11 10 6 5
24854 003/300 Write block 2957 len 36864 data: 4 4 3 4 6 5 6 6 5
248552024-02-28T22:18:18.127ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248562024-02-28T22:18:18.127ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248572024-02-28T22:18:18.127ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248582024-02-28T22:18:18.127ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248592024-02-28T22:18:18.127ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248602024-02-28T22:18:18.127ZINFOcrucible: [2] client re-new 2 jobs since flush 9440 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
248612024-02-28T22:18:18.127ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
24862 004/300 Write block 1601 len 16384 data: 8 7 7 5
24863 005/300 Read block 270 len 40960
24864 006/300 Write block 586 len 28672 data: 7 6 6 7 7 4 4
24865 007/300 Read block 1720 len 4096
24866 008/300 Write block 2242 len 40960 data: 7 7 7 6 5 5 5 4 4 3
24867 009/300 Read block 2647 len 20480
24868 010/300 Write block 123 len 12288 data: 6 5 5
24869 011/300 Flush
24870 012/300 Read block 4287 len 20480
24871 013/300 Write block 2763 len 4096 data: 4
24872 014/300 Read block 1075 len 12288
24873 015/300 Read block 4689 len 28672
24874 016/300 Read block 247 len 20480
24875 017/300 Write block 3600 len 12288 data: 5 4 5
24876 018/300 Flush
24877 019/300 Read block 2249 len 12288
24878 020/300 Write block 1552 len 32768 data: 6 5 5 6 6 6 4 5
24879 021/300 Read block 1287 len 36864
24880 022/300 Write block 1279 len 40960 data: 6 7 8 7 6 7 5 6 7 6
24881 023/300 Read block 4264 len 40960
24882 024/300 Read block 3304 len 12288
24883 025/300 Read block 4809 len 8192
24884 026/300 Flush
24885 027/300 Read block 3532 len 24576
24886 028/300 Read block 3929 len 8192
24887 029/300 Read block 3098 len 24576
24888 030/300 Write block 4312 len 12288 data: 6 6 7
24889 031/300 Read block 1951 len 28672
24890 032/300 Write block 2926 len 24576 data: 5 5 5 7 6 6
24891 033/300 Flush
24892 034/300 Flush
24893 035/300 Read block 3359 len 12288
24894 036/300 Read block 1106 len 28672
24895 037/300 Read block 4794 len 36864
24896 038/300 Read block 187 len 28672
24897 039/300 Read block 3197 len 28672
24898 040/300 Read block 2864 len 32768
24899 041/300 Write block 1721 len 24576 data: 5 8 7 5 4 4
24900 042/300 Read block 4225 len 8192
24901 043/300 Write block 945 len 20480 data: 4 5 5 5 5
24902 044/300 Write block 4650 len 12288 data: 3 4 4
24903 045/300 Flush
24904 046/300 Read block 4026 len 20480
24905 047/300 Read block 4491 len 36864
24906 048/300 Read block 4205 len 4096
24907 049/300 Write block 397 len 16384 data: 6 6 6 6
24908 050/300 Write block 3072 len 4096 data: 7
24909 051/300 Write block 2662 len 4096 data: 5
24910 052/300 Write block 2584 len 12288 data: 5 6 3
24911 053/300 Write block 3664 len 16384 data: 3 2 2 2
24912 054/300 Write block 4592 len 20480 data: 7 5 5 5 6
24913 055/300 Read block 2368 len 36864
24914 056/300 Read block 4291 len 36864
24915 057/300 Write block 3809 len 16384 data: 4 4 5 5
24916 058/300 Read block 2981 len 36864
24917 059/300 Read block 4356 len 40960
24918 060/300 Read block 4494 len 8192
24919 061/300 Write block 3244 len 32768 data: 5 6 8 9 7 9 8 9
24920 062/300 Write block 4507 len 28672 data: 7 7 6 7 4 3 4
24921 063/300 Read block 4096 len 20480
24922 064/300 Read block 1074 len 20480
24923 065/300 Write block 645 len 40960 data: 3 4 3 4 4 5 6 6 6 6
24924 066/300 Read block 3642 len 4096
24925 067/300 Flush
24926 068/300 Read block 1704 len 40960
24927 069/300 Write block 4689 len 24576 data: 5 5 6 7 7 6
24928 070/300 Read block 2594 len 4096
24929 071/300 Write block 1041 len 4096 data: 10
24930 072/300 Flush
24931 073/300 Read block 4533 len 32768
24932 074/300 Read block 1513 len 16384
24933 075/300 Read block 4331 len 16384
24934 076/300 Read block 2630 len 28672
24935 077/300 Read block 4752 len 40960
24936 078/300 Read block 4849 len 20480
24937 079/300 Read block 292 len 36864
24938 080/300 Write block 1418 len 32768 data: 8 9 7 6 6 7 7 5
24939 081/300 Read block 2484 len 4096
24940 082/300 Write block 2723 len 4096 data: 8
24941 083/300 Read block 2973 len 8192
24942 084/300 Write block 1169 len 24576 data: 8 8 7 7 7 5
24943 085/300 Write block 3296 len 32768 data: 7 6 5 3 3 3 5 5
24944 086/300 Write block 4843 len 12288 data: 4 4 4
24945 087/300 Flush
24946 088/300 Read block 2845 len 8192
24947 089/300 Write block 3066 len 20480 data: 6 5 6 7 6
24948 090/300 Read block 3699 len 28672
24949 091/300 Write block 105 len 36864 data: 5 6 5 5 5 6 5 6 5
24950 092/300 Read block 1307 len 4096
24951 093/300 Write block 1893 len 4096 data: 6
24952 094/300 Write block 19 len 24576 data: 4 3 6 5 7 8
24953 095/300 Read block 840 len 12288
24954 096/300 Write block 4283 len 20480 data: 6 6 6 5 5
24955 097/300 Write block 2878 len 8192 data: 5 4
24956 098/300 Write block 3790 len 32768 data: 5 6 4 4 4 4 4 3
24957 099/300 Write block 4598 len 32768 data: 6 6 7 7 6 6 5 6
24958 100/300 Write block 2827 len 12288 data: 3 3 2
24959 101/300 Write block 4331 len 12288 data: 8 10 8
24960 102/300 Read block 1454 len 20480
24961 103/300 Write block 3563 len 16384 data: 7 7 7 8
24962 104/300 Write block 1366 len 16384 data: 2 2 3 3
24963 105/300 Write block 4233 len 12288 data: 5 6 6
24964 106/300 Write block 358 len 8192 data: 4 4
24965 107/300 Read block 4397 len 36864
24966 108/300 Read block 618 len 40960
24967 109/300 Read block 4632 len 8192
24968 110/300 Write block 3702 len 20480 data: 6 6 8 6 6
24969 111/300 Read block 355 len 16384
24970 112/300 Read block 1444 len 16384
24971 113/300 Write block 1733 len 28672 data: 4 4 4 4 3 4 4
24972 114/300 Read block 1732 len 36864
24973 115/300 Flush
24974 116/300 Read block 836 len 28672
24975 117/300 Read block 1993 len 24576
24976 118/300 Read block 2934 len 8192
24977 119/300 Write block 1322 len 8192 data: 6 5
24978 120/300 Flush
24979 121/300 Read block 70 len 28672
24980 122/300 Write block 995 len 8192 data: 4 4
24981 123/300 Read block 3429 len 32768
24982 124/300 Read block 766 len 12288
24983 125/300 Flush
24984 126/300 Read block 2994 len 12288
24985 127/300 Read block 2768 len 36864
24986 128/300 Flush
24987 129/300 Write block 4804 len 4096 data: 9
24988 130/300 Read block 4075 len 8192
24989 131/300 Flush
24990 132/300 Read block 2548 len 12288
24991 133/300 Flush
24992 134/300 Flush
24993 135/300 Read block 1203 len 32768
24994 136/300 Read block 2883 len 24576
24995 137/300 Read block 4494 len 8192
24996 138/300 Write block 4351 len 8192 data: 5 6
24997 139/300 Write block 2534 len 16384 data: 5 5 7 7
24998 140/300 Read block 1858 len 16384
24999 141/300 Read block 1577 len 20480
25000 142/300 Read block 947 len 12288
25001 143/300 Read block 1085 len 12288
25002 144/300 Read block 635 len 40960
25003 145/300 Write block 1851 len 16384 data: 6 5 4 4
25004 146/300 Write block 3900 len 12288 data: 5 7 7
25005 147/300 Write block 413 len 32768 data: 6 6 8 8 9 8 8 9
25006 148/300 Read block 4176 len 28672
25007 149/300 Read block 4488 len 36864
25008 150/300 Flush
25009 151/300 Read block 1103 len 36864
25010 152/300 Write block 4713 len 40960 data: 7 9 8 8 8 9 9 6 5 6
25011 153/300 Write block 1409 len 24576 data: 5 3 6 6 6 5
25012 154/300 Write block 3612 len 16384 data: 5 6 6 5
25013 155/300 Read block 1328 len 40960
25014 156/300 Write block 4662 len 4096 data: 5
25015 157/300 Write block 4947 len 40960 data: 4 4 6 6 4 3 4 3 3 2
25016 158/300 Flush
25017 159/300 Read block 2562 len 32768
25018 160/300 Read block 1178 len 36864
25019 161/300 Read block 1799 len 36864
25020 162/300 Write block 2072 len 20480 data: 9 10 8 7 4
25021 163/300 Write block 4422 len 12288 data: 3 4 4
25022 164/300 Flush
25023 165/300 Read block 1978 len 36864
25024 166/300 Write block 2420 len 28672 data: 6 6 5 5 5 5 5
25025 167/300 Write block 3547 len 28672 data: 7 7 7 7 7 6 3
25026 168/300 Write block 4949 len 12288 data: 7 7 5
25027 169/300 Write block 1093 len 12288 data: 7 7 5
25028 170/300 Flush
25029 171/300 Read block 2664 len 24576
25030 172/300 Read block 3599 len 8192
25031 173/300 Write block 3030 len 8192 data: 5 5
25032 174/300 Read block 4739 len 12288
25033 175/300 Read block 2114 len 16384
25034 176/300 Read block 3062 len 20480
25035 177/300 Flush
25036 178/300 Read block 3103 len 36864
25037 179/300 Flush
25038 180/300 Read block 4764 len 28672
25039 181/300 Write block 288 len 20480 data: 3 4 4 6 6
25040 182/300 Write block 2296 len 16384 data: 6 8 8 8
25041 183/300 Read block 4219 len 4096
25042 184/300 Read block 2884 len 40960
25043 185/300 Read block 549 len 4096
25044 186/300 Write block 1814 len 40960 data: 3 2 3 3 3 4 4 3 3 3
25045 187/300 Write block 193 len 40960 data: 5 6 5 6 4 4 3 3 2 5
25046 188/300 Read block 2916 len 8192
25047 189/300 Write block 2516 len 16384 data: 8 8 9 6
25048 190/300 Read block 521 len 12288
25049 191/300 Write block 2751 len 16384 data: 3 3 4 4
25050 192/300 Write block 2456 len 24576 data: 7 7 8 7 5 4
25051 193/300 Read block 2546 len 28672
25052 194/300 Read block 3278 len 8192
25053 195/300 Write block 303 len 40960 data: 5 4 5 5 5 5 4 5 5 6
25054 196/300 Write block 4729 len 8192 data: 6 7
25055 197/300 Flush
25056 198/300 Read block 3359 len 16384
25057 199/300 Read block 1971 len 28672
25058 200/300 Read block 3593 len 36864
25059 201/300 Write block 1215 len 12288 data: 9 9 9
25060 202/300 Read block 4786 len 24576
25061 203/300 Write block 3096 len 40960 data: 6 5 5 5 5 5 4 2 2 2
25062 204/300 Read block 4847 len 36864
25063 205/300 Read block 4589 len 36864
25064 206/300 Read block 160 len 16384
25065 207/300 Flush
25066 208/300 Read block 4267 len 20480
25067 209/300 Read block 3634 len 4096
25068 210/300 Write block 1348 len 24576 data: 8 10 10 9 6 6
25069 211/300 Read block 3633 len 16384
25070 212/300 Write block 3081 len 24576 data: 5 4 6 3 2 2
25071 213/300 Read block 4256 len 16384
25072 214/300 Write block 2650 len 4096 data: 8
25073 215/300 Write block 2694 len 36864 data: 3 4 4 4 4 5 5 5 5
25074 216/300 Read block 1302 len 40960
25075 217/300 Read block 848 len 32768
25076 218/300 Read block 812 len 20480
25077 219/300 Write block 1041 len 4096 data: 11
25078 220/300 Write block 117 len 28672 data: 7 7 7 6 6 7 7
25079 221/300 Read block 1155 len 16384
25080 222/300 Write block 95 len 8192 data: 6 5
25081 223/300 Read block 489 len 12288
25082 224/300 Write block 2328 len 8192 data: 3 3
25083 225/300 Read block 3437 len 20480
25084 226/300 Read block 1673 len 20480
25085 227/300 Write block 4484 len 4096 data: 6
25086 228/300 Write block 1359 len 36864 data: 4 5 4 5 5 2 2 3 3
25087 229/300 Read block 1267 len 16384
25088 230/300 Flush
25089 231/300 Read block 4093 len 20480
25090 232/300 Read block 3070 len 4096
25091 233/300 Read block 1958 len 24576
25092 234/300 Read block 564 len 24576
25093 235/300 Read block 4356 len 20480
25094 236/300 Write block 3581 len 36864 data: 7 6 6 7 9 10 9 8 10
25095 237/300 Read block 1522 len 8192
25096 238/300 Write block 406 len 40960 data: 6 5 5 4 4 5 6 7 7 9
25097 239/300 Flush
25098 240/300 Read block 925 len 28672
25099 241/300 Flush
25100 242/300 Read block 842 len 32768
25101 243/300 Read block 101 len 28672
25102 244/300 Write block 3756 len 28672 data: 5 5 6 4 4 4 4
25103 245/300 Write block 666 len 36864 data: 5 5 7 7 10 12 10 9 7
25104 246/300 Read block 3089 len 4096
25105 247/300 Flush
25106 248/300 Flush
25107 249/300 Write block 4664 len 24576 data: 4 5 4 4 5 6
25108 250/300 Read block 4455 len 8192
25109 251/300 Write block 4706 len 16384 data: 4 3 3 3
25110 252/300 Write block 2060 len 24576 data: 3 3 5 5 4 6
25111 253/300 Write block 2052 len 24576 data: 10 8 7 7 6 4
25112 254/300 Write block 3871 len 20480 data: 5 5 6 5 5
25113 255/300 Read block 1614 len 40960
25114 256/300 Write block 4066 len 40960 data: 5 5 6 7 8 9 8 9 7 7
25115 257/300 Read block 1296 len 28672
25116 258/300 Write block 1416 len 28672 data: 7 8 9 10 8 7 7
25117 259/300 Flush
25118 260/300 Flush
25119 261/300 Write block 4545 len 16384 data: 10 9 11 9
25120 262/300 Write block 1975 len 28672 data: 6 5 6 7 9 6 7
25121 263/300 Read block 4507 len 40960
25122 264/300 Read block 4159 len 12288
25123 265/300 Read block 2913 len 32768
25124 266/300 Read block 4404 len 28672
25125 267/300 Flush
25126 268/300 Write block 4890 len 20480 data: 6 4 4 4 4
25127 269/300 Read block 3123 len 36864
25128 270/300 Read block 582 len 8192
25129 271/300 Read block 4468 len 36864
25130 272/300 Read block 1837 len 36864
25131 273/300 Write block 934 len 40960 data: 6 7 6 5 4 5 4 4 4 4
25132 274/300 Read block 1923 len 8192
25133 275/300 Write block 4657 len 40960 data: 7 7 6 6 5 6 5 5 6 5
25134 276/300 Read block 4191 len 8192
25135 277/300 Write block 4567 len 36864 data: 7 7 8 8 8 6 6 5 4
25136 278/300 Write block 1795 len 36864 data: 5 6 7 8 8 8 8 8 8
25137 279/300 Read block 2859 len 28672
25138 280/300 Read block 1957 len 32768
25139 281/300 Write block 3145 len 28672 data: 5 5 7 9 8 8 9
25140 282/300 Read block 995 len 8192
25141 283/300 Write block 3343 len 12288 data: 4 4 6
25142 284/300 Write block 558 len 12288 data: 2 4 5
25143 285/300 Read block 2010 len 8192
25144 286/300 Flush
25145 287/300 Read block 1275 len 24576
25146 288/300 Write block 2248 len 16384 data: 6 5 5 4
25147 289/300 Write block 2514 len 24576 data: 5 7 9 9 10 7
25148 290/300 Write block 1169 len 8192 data: 9 9
25149 291/300 Write block 770 len 40960 data: 5 5 5 5 7 7 8 9 10 9
25150 292/300 Write block 3499 len 36864 data: 5 5 4 5 5 6 7 7 6
25151 293/300 Write block 1776 len 4096 data: 5
25152 294/300 Read block 383 len 8192
25153 295/300 Write block 4276 len 20480 data: 3 5 5 4 4
25154 296/300 Write block 3973 len 32768 data: 7 6 7 7 4 5 5 4
25155 297/300 Write block 2871 len 36864 data: 6 8 8 5 7 6 5 6 5
25156 298/300 Read block 2221 len 12288
25157 299/300 Read block 2198 len 36864
25158 300/300 Read block 4452 len 16384
25159 Main task has work to do, go find it
25160 got dsc Start(2)
25161 start 2
25162 Feb 28 22:18:18.460 INFO request completed, latency_us: 201, response_code: 204, uri: /start/cid/2, method: POST, req_id: 5b4ac67c-615a-476e-8a7e-3fd5af8a8e29, remote_addr: 127.0.0.1:41656, local_addr: 127.0.0.1:9998
25163 Replay: started 2, returned:Ok(())
25164 ----------------------------------------------------------------
25165 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25166 States: Active Active Offline
25167 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
25168 8442 Acked 9441 Read 5 Done Done New true
25169 8443 Acked 9442 Write 4 Done Done New true
25170 8444 Acked 9443 Write 9 Done Done New false
25171 8445 Acked 9444 Write 4 Done Done New false
25172 8446 Acked 9445 Read 10 Done Done New false
25173 8447 Acked 9446 Write 7 Done Done New false
25174 8448 Acked 9447 Read 1 Done Done New false
25175 8449 Acked 9448 Write 10 Done Done New false
25176 8450 Acked 9449 Read 5 Done Done New false
25177 8451 Acked 9450 Write 3 Done Done New false
25178 8452 Acked 9451 Flush 0 Done Done New false
25179 8453 Acked 9452 Read 5 Done Done New false
25180 8454 Acked 9453 Write 1 Done Done New false
25181 8455 Acked 9454 Read 3 Done Done New false
25182 8456 Acked 9455 Read 7 Done Done New false
25183 8457 Acked 9456 Read 5 Done Done New false
25184 8458 Acked 9457 Write 3 Done Done New false
25185 8459 Acked 9458 Flush 0 Done Done New false
25186 8460 Acked 9459 Read 3 Done Done New false
25187 8461 Acked 9460 Write 8 Done Done New false
25188 8462 Acked 9461 Read 9 Done Done New false
25189 8463 Acked 9462 Write 10 Done Done New false
25190 8464 Acked 9463 Read 10 Done Done New false
25191 8465 Acked 9464 Read 3 Done Done New false
25192 8466 Acked 9465 Read 2 Done Done New false
25193 8467 Acked 9466 Flush 0 Done Done New false
25194 8468 Acked 9467 Read 6 Done Done New false
25195 8469 Acked 9468 Read 2 Done Done New false
25196 8470 Acked 9469 Read 6 Done Done New false
25197 8471 Acked 9470 Write 3 Done Done New false
25198 8472 Acked 9471 Read 7 Done Done New false
25199 8473 Acked 9472 Write 6 Done Done New false
25200 8474 Acked 9473 Flush 0 Done Done New false
25201 8475 Acked 9474 Flush 0 Done Done New false
25202 8476 Acked 9475 Read 3 Done Done New false
25203 8477 Acked 9476 Read 7 Done Done New false
25204 8478 Acked 9477 Read 9 Done Done New false
25205 8479 Acked 9478 Read 7 Done Done New false
25206 8480 Acked 9479 Read 7 Done Done New false
25207 8481 Acked 9480 Read 8 Done Done New false
25208 8482 Acked 9481 Write 6 Done Done New false
25209 8483 Acked 9482 Read 2 Done Done New false
25210 8484 Acked 9483 Write 5 Done Done New false
25211 8485 Acked 9484 Write 3 Done Done New false
25212 8486 Acked 9485 Flush 0 Done Done New false
25213 8487 Acked 9486 Read 5 Done Done New false
25214 8488 Acked 9487 Read 9 Done Done New false
25215 8489 Acked 9488 Read 1 Done Done New false
25216 8490 Acked 9489 Write 4 Done Done New false
25217 8491 Acked 9490 Write 1 Done Done New false
25218 8492 Acked 9491 Write 1 Done Done New false
25219 8493 Acked 9492 Write 3 Done Done New false
25220 8494 Acked 9493 Write 4 Done Done New false
25221 8495 Acked 9494 Write 5 Done Done New false
25222 8496 Acked 9495 Read 9 Done Done New false
25223 8497 Acked 9496 Read 9 Done Done New false
25224 8498 Acked 9497 Write 4 Done Done New false
25225 8499 Acked 9498 Read 9 Done Done New false
25226 8500 Acked 9499 Read 10 Done Done New false
25227 8501 Acked 9500 Read 2 Done Done New false
25228 8502 Acked 9501 Write 8 Done Done New false
25229 8503 Acked 9502 Write 7 Done Done New false
25230 8504 Acked 9503 Read 5 Done Done New false
25231 8505 Acked 9504 Read 5 Done Done New false
25232 8506 Acked 9505 Write 10 Done Done New false
25233 8507 Acked 9506 Read 1 Done Done New false
25234 8508 Acked 9507 Flush 0 Done Done New false
25235 8509 Acked 9508 Read 10 Done Done New false
25236 8510 Acked 9509 Write 6 Done Done New false
25237 8511 Acked 9510 Read 1 Done Done New false
25238 8512 Acked 9511 Write 1 Done Done New false
25239 8513 Acked 9512 Flush 0 Done Done New false
25240 8514 Acked 9513 Read 8 Done Done New false
25241 8515 Acked 9514 Read 4 Done Done New false
25242 8516 Acked 9515 Read 4 Done Done New false
25243 8517 Acked 9516 Read 7 Done Done New false
25244 8518 Acked 9517 Read 10 Done Done New false
25245 8519 Acked 9518 Read 5 Done Done New false
25246 8520 Acked 9519 Read 9 Done Done New false
25247 8521 Acked 9520 Write 8 Done Done New false
25248 8522 Acked 9521 Read 1 Done Done New false
25249 8523 Acked 9522 Write 1 Done Done New false
25250 8524 Acked 9523 Read 2 Done Done New false
25251 8525 Acked 9524 Write 6 Done Done New false
25252 8526 Acked 9525 Write 8 Done Done New false
25253 8527 Acked 9526 Write 3 Done Done New false
25254 8528 Acked 9527 Flush 0 Done Done New false
25255 8529 Acked 9528 Read 2 Done Done New false
25256 8530 Acked 9529 Write 5 Done Done New false
25257 8531 Acked 9530 Read 7 Done Done New false
25258 8532 Acked 9531 Write 9 Done Done New false
25259 8533 Acked 9532 Read 1 Done Done New false
25260 8534 Acked 9533 Write 1 Done Done New false
25261 8535 Acked 9534 Write 6 Done Done New false
25262 8536 Acked 9535 Read 3 Done Done New false
25263 8537 Acked 9536 Write 5 Done Done New false
25264 8538 Acked 9537 Write 2 Done Done New false
25265 8539 Acked 9538 Write 8 Done Done New false
25266 8540 Acked 9539 Write 8 Done Done New false
25267 8541 Acked 9540 Write 3 Done Done New false
25268 8542 Acked 9541 Write 3 Done Done New false
25269 8543 Acked 9542 Read 5 Done Done New false
25270 8544 Acked 9543 Write 4 Done Done New false
25271 8545 Acked 9544 Write 4 Done Done New false
25272 8546 Acked 9545 Write 3 Done Done New false
25273 8547 Acked 9546 Write 2 Done Done New false
25274 8548 Acked 9547 Read 9 Done Done New false
25275 8549 Acked 9548 Read 10 Done Done New false
25276 8550 Acked 9549 Read 2 Done Done New false
25277 8551 Acked 9550 Write 5 Done Done New false
25278 8552 Acked 9551 Read 4 Done Done New false
25279 8553 Acked 9552 Read 4 Done Done New false
25280 8554 Acked 9553 Write 7 Done Done New false
25281 8555 Acked 9554 Read 9 Done Done New false
25282 8556 Acked 9555 Flush 0 Done Done New false
25283 8557 Acked 9556 Read 7 Done Done New false
25284 8558 Acked 9557 Read 6 Done Done New false
25285 8559 Acked 9558 Read 2 Done Done New false
25286 8560 Acked 9559 Write 2 Done Done New false
25287 8561 Acked 9560 Flush 0 Done Done New false
25288 8562 Acked 9561 Read 7 Done Done New false
25289 8563 Acked 9562 Write 2 Done Done New false
25290 8564 Acked 9563 Read 8 Done Done New false
25291 8565 Acked 9564 Read 3 Done Done New false
25292 8566 Acked 9565 Flush 0 Done Done New false
25293 8567 Acked 9566 Read 3 Done Done New false
25294 8568 Acked 9567 Read 9 Done Done New false
25295 8569 Acked 9568 Flush 0 Done Done New false
25296 8570 Acked 9569 Write 1 Done Done New false
25297 8571 Acked 9570 Read 2 Done Done New false
25298 8572 Acked 9571 Flush 0 Done Done New false
25299 8573 Acked 9572 Read 3 Done Done New false
25300 8574 Acked 9573 Flush 0 Done Done New false
25301 8575 Acked 9574 Flush 0 Done Done New false
25302 8576 Acked 9575 Read 8 Done Done New false
25303 8577 Acked 9576 Read 6 Done Done New false
25304 8578 Acked 9577 Read 2 Done Done New false
25305 8579 Acked 9578 Write 2 Done Done New false
25306 8580 Acked 9579 Write 4 Done Done New false
25307 8581 Acked 9580 Read 4 Done Done New false
25308 8582 Acked 9581 Read 5 Done Done New false
25309 8583 Acked 9582 Read 3 Done Done New false
25310 8584 Acked 9583 Read 3 Done Done New false
25311 8585 Acked 9584 Read 10 Done Done New false
25312 8586 Acked 9585 Write 4 Done Done New false
25313 8587 Acked 9586 Write 3 Done Done New false
25314 8588 Acked 9587 Write 8 Done Done New false
25315 8589 Acked 9588 Read 7 Done Done New false
25316 8590 Acked 9589 Read 9 Done Done New false
25317 8591 Acked 9590 Flush 0 Done Done New false
25318 8592 Acked 9591 Read 9 Done Done New false
25319 8593 Acked 9592 Write 10 Done Done New false
25320 8594 Acked 9593 Write 6 Done Done New false
25321 8595 Acked 9594 Write 4 Done Done New false
25322 8596 Acked 9595 Read 10 Done Done New false
25323 8597 Acked 9596 Write 1 Done Done New false
25324 8598 Acked 9597 Write 10 Done Done New false
25325 8599 Acked 9598 Flush 0 Done Done New false
25326 8600 Acked 9599 Read 8 Done Done New false
25327 8601 Acked 9600 Read 9 Done Done New false
25328 8602 Acked 9601 Read 9 Done Done New false
25329 8603 Acked 9602 Write 5 Done Done New false
25330 8604 Acked 9603 Write 3 Done Done New false
25331 8605 Acked 9604 Flush 0 Done Done New false
25332 8606 Acked 9605 Read 9 Done Done New false
25333 8607 Acked 9606 Write 7 Done Done New false
25334 8608 Acked 9607 Write 7 Done Done New false
25335 8609 Acked 9608 Write 3 Done Done New false
25336 8610 Acked 9609 Write 3 Done Done New false
25337 8611 Acked 9610 Flush 0 Done Done New false
25338 8612 Acked 9611 Read 6 Done Done New false
25339 8613 Acked 9612 Read 2 Done Done New false
25340 8614 Acked 9613 Write 2 Done Done New false
25341 8615 Acked 9614 Read 3 Done Done New false
25342 8616 Acked 9615 Read 4 Done Done New false
25343 8617 Acked 9616 Read 5 Done Done New false
25344 8618 Acked 9617 Flush 0 Done Done New false
25345 8619 Acked 9618 Read 9 Done Done New false
25346 8620 Acked 9619 Flush 0 Done Done New false
25347 8621 Acked 9620 Read 7 Done Done New false
25348 8622 Acked 9621 Write 5 Done Done New false
25349 8623 Acked 9622 Write 4 Done Done New false
25350 8624 Acked 9623 Read 1 Done Done New false
25351 8625 Acked 9624 Read 10 Done Done New false
25352 8626 Acked 9625 Read 1 Done Done New false
25353 8627 Acked 9626 Write 10 Done Done New false
25354 8628 Acked 9627 Write 10 Done Done New false
25355 8629 Acked 9628 Read 2 Done Done New false
25356 8630 Acked 9629 Write 4 Done Done New false
25357 8631 Acked 9630 Read 3 Done Done New false
25358 8632 Acked 9631 Write 4 Done Done New false
25359 8633 Acked 9632 Write 6 Done Done New false
25360 8634 Acked 9633 Read 7 Done Done New false
25361 8635 Acked 9634 Read 2 Done Done New false
25362 8636 Acked 9635 Write 10 Done Done New false
25363 8637 Acked 9636 Write 2 Done Done New false
25364 8638 Acked 9637 Flush 0 Done Done New false
25365 8639 Acked 9638 Read 4 Done Done New false
25366 8640 Acked 9639 Read 7 Done Done New false
25367 8641 Acked 9640 Read 9 Done Done New false
25368 8642 Acked 9641 Write 3 Done Done New false
25369 8643 Acked 9642 Read 6 Done Done New false
25370 8644 Acked 9643 Write 10 Done Done New false
25371 8645 Acked 9644 Read 9 Done Done New false
25372 8646 Acked 9645 Read 9 Done Done New false
25373 8647 Acked 9646 Read 4 Done Done New false
25374 8648 Acked 9647 Flush 0 Done Done New false
25375 8649 Acked 9648 Read 5 Done Done New false
25376 8650 Acked 9649 Read 1 Done Done New false
25377 8651 Acked 9650 Write 6 Done Done New false
25378 8652 Acked 9651 Read 4 Done Done New false
25379 8653 Acked 9652 Write 6 Done Done New false
25380 8654 Acked 9653 Read 4 Done Done New false
25381 8655 Acked 9654 Write 1 Done Done New false
25382 8656 Acked 9655 Write 9 Done Done New false
25383 8657 Acked 9656 Read 10 Done Done New false
25384 8658 Acked 9657 Read 8 Done Done New false
25385 8659 Acked 9658 Read 5 Done Done New false
25386 8660 Acked 9659 Write 1 Done Done New false
25387 8661 Acked 9660 Write 7 Done Done New false
25388 8662 Acked 9661 Read 4 Done Done New false
25389 8663 Acked 9662 Write 2 Done Done New false
25390 8664 Acked 9663 Read 3 Done Done New false
25391 8665 Acked 9664 Write 2 Done Done New false
25392 8666 Acked 9665 Read 5 Done Done New false
25393 8667 Acked 9666 Read 5 Done Done New false
25394 8668 Acked 9667 Write 1 Done Done New false
25395 8669 Acked 9668 Write 9 Done Done New false
25396 8670 Acked 9669 Read 4 Done Done New false
25397 8671 Acked 9670 Flush 0 Done Done New false
25398 8672 Acked 9671 Read 5 Done Done New false
25399 8673 Acked 9672 Read 1 Done Done New false
25400 8674 Acked 9673 Read 6 Done Done New false
25401 8675 Acked 9674 Read 6 Done Done New false
25402 8676 Acked 9675 Read 5 Done Done New false
25403 8677 Acked 9676 Write 9 Done Done New false
25404 8678 Acked 9677 Read 2 Done Done New false
25405 8679 Acked 9678 Write 10 Done Done New false
25406 8680 Acked 9679 Flush 0 Done Done New false
25407 8681 Acked 9680 Read 7 Done Done New false
25408 8682 Acked 9681 Flush 0 Done Done New false
25409 8683 Acked 9682 Read 8 Done Done New false
25410 8684 Acked 9683 Read 7 Done Done New false
25411 8685 Acked 9684 Write 7 Done Done New false
25412 8686 Acked 9685 Write 9 Done Done New false
25413 8687 Acked 9686 Read 1 Done Done New false
25414 8688 Acked 9687 Flush 0 Done Done New false
25415 8689 Acked 9688 Flush 0 Done Done New false
25416 8690 Acked 9689 Write 6 Done Done New false
25417 8691 Acked 9690 Read 2 Done Done New false
25418 8692 Acked 9691 Write 4 Done Done New false
25419 8693 Acked 9692 Write 6 Done Done New false
25420 8694 Acked 9693 Write 6 Done Done New false
25421 8695 Acked 9694 Write 5 Done Done New false
25422 8696 Acked 9695 Read 10 Done Done New false
25423 8697 Acked 9696 Write 10 Done Done New false
25424 8698 Acked 9697 Read 7 Done Done New false
25425 8699 Acked 9698 Write 7 Done Done New false
25426 8700 Acked 9699 Flush 0 Done Done New false
25427 8701 Acked 9700 Flush 0 Done Done New false
25428 8702 Acked 9701 Write 4 Done Done New false
25429 8703 Acked 9702 Write 7 Done Done New false
25430 8704 Acked 9703 Read 10 Done Done New false
25431 8705 Acked 9704 Read 3 Done Done New false
25432 8706 Acked 9705 Read 8 Done Done New false
25433 8707 Acked 9706 Read 7 Done Done New false
25434 8708 Acked 9707 Flush 0 Done Done New false
25435 8709 Acked 9708 Write 5 Done Done New false
25436 8710 Acked 9709 Read 9 Done Done New false
25437 8711 Acked 9710 Read 2 Done Done New false
25438 8712 Acked 9711 Read 9 Done Done New false
25439 8713 Acked 9712 Read 9 Done Done New false
25440 8714 Acked 9713 Write 10 Done Done New false
25441 8715 Acked 9714 Read 2 Done Done New false
25442 8716 Acked 9715 Write 10 Done Done New false
25443 8717 Acked 9716 Read 2 Done Done New false
25444 8718 Acked 9717 Write 9 Done Done New false
25445 8719 Acked 9718 Write 9 Done Done New false
25446 8720 Acked 9719 Read 7 Done Done New false
25447 8721 Acked 9720 Read 8 Done Done New false
25448 8722 Acked 9721 Write 7 Done Done New false
25449 8723 Acked 9722 Read 2 Done Done New false
25450 8724 Acked 9723 Write 3 Done Done New false
25451 8725 Acked 9724 Write 3 Done Done New false
25452 8726 Acked 9725 Read 2 Done Done New false
25453 8727 Acked 9726 Flush 0 Done Done New false
25454 8728 Acked 9727 Read 6 Done Done New false
25455 8729 Acked 9728 Write 4 Done Done New false
25456 8730 Acked 9729 Flush 0 Done Done New false
25457 8731 Acked 9730 Write 6 Done Done New false
25458 8732 Acked 9731 Write 2 Done Done New false
25459 8733 Acked 9732 Write 10 Done Done New false
25460 8734 Acked 9733 Write 9 Done Done New false
25461 8735 Acked 9734 Write 1 Done Done New false
25462 8736 Acked 9735 Read 2 Done Done New false
25463 8737 Acked 9736 Write 5 Done Done New false
25464 8738 Acked 9737 Write 8 Done Done New false
25465 8739 Acked 9738 Write 9 Done Done New false
25466 8740 Acked 9739 Read 3 Done Done New false
25467 8741 Acked 9740 Read 9 Done Done New false
25468 8742 Acked 9741 Read 4 Done Done New false
25469 STATES DS:0 DS:1 DS:2 TOTAL
25470 New 0 0 301 301
25471 Sent 0 0 0 0
25472 Done 301 301 0 602
25473 Skipped 0 0 0 0
25474 Error 0 0 0 0
25475 Last Flush: 9729 9729 9440
25476 Downstairs last five completed: 9440 9439 9438 9437 9436
25477 Upstairs last five completed: 8742 8741 8740 8739 8738
25478 CLIENT: Up:0 ds:301 act:2
254792024-02-28T22:18:19.128ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
254802024-02-28T22:18:19.128ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
254812024-02-28T22:18:19.128ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
254822024-02-28T22:18:19.128ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
254832024-02-28T22:18:19.129ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
254842024-02-28T22:18:19.129ZINFOcrucible: [2] client re-new 302 jobs since flush 9440 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
254852024-02-28T22:18:19.129ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
254862024-02-28T22:18:20.130ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
254872024-02-28T22:18:20.130ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
254882024-02-28T22:18:20.130ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
254892024-02-28T22:18:20.130ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
254902024-02-28T22:18:20.130ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
254912024-02-28T22:18:20.130ZINFOcrucible: [2] client re-new 302 jobs since flush 9440 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
254922024-02-28T22:18:20.130ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
25493 [8830] got start action
25494 [8830] I am going to restart
25495 Starting downstairs at port 8830
25496 Make output file at "/tmp/dsc/downstairs-8830.txt"
25497 [8830][2] reports Starting
25498 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1193)
25499 [8830][2] reports Running
255002024-02-28T22:18:21.131ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255012024-02-28T22:18:21.131ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255022024-02-28T22:18:21.133ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255032024-02-28T22:18:21.133ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255042024-02-28T22:18:21.133ZINFOcrucible: send last flush ID to this DS: 9440 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255052024-02-28T22:18:21.133ZINFOcrucible: Replied this last flush ID: 9440 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255062024-02-28T22:18:21.133ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255072024-02-28T22:18:21.133ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255082024-02-28T22:18:21.133ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255092024-02-28T22:18:21.133ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
25510 ----------------------------------------------------------------
25511 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25512 Downstairs last five completed: 9742 9741 9740 9739 9738
25513 Upstairs last five completed: 8743 8742 8741 8740 8739
25514 CLIENT: Up:0 ds:0 act:3
25515 Replay: All jobs finished, all DS active.
25516 Main task has work to do, go find it
25517 got dsc Stop(2)
25518 stop 2
25519 [8830] Got stop action so:false kr:false
25520 Feb 28 22:18:22.464 INFO request completed, latency_us: 350, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 49d4c6e5-8a19-40dc-b918-5479e05eb9ca, remote_addr: 127.0.0.1:41656, local_addr: 127.0.0.1:9998
25521 Feb 28 22:18:22.464 INFO request completed, latency_us: 108, response_code: 200, uri: /state/cid/2, method: GET, req_id: 8540fec6-fea1-4ccb-ac0d-fb240a47c239, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
255222024-02-28T22:18:22.465ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255232024-02-28T22:18:22.465ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255242024-02-28T22:18:22.465ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255252024-02-28T22:18:22.465ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255262024-02-28T22:18:22.465ZINFOcrucible: [2] client re-new 0 jobs since flush 9742 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255272024-02-28T22:18:22.465ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
25528 [8830] Exited with: ExitStatus(unix_wait_status(9))
25529 [8830][2] reports Exit
255302024-02-28T22:18:23.467ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255312024-02-28T22:18:23.467ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255322024-02-28T22:18:23.467ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255332024-02-28T22:18:23.467ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255342024-02-28T22:18:23.467ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255352024-02-28T22:18:23.467ZINFOcrucible: [2] client re-new 0 jobs since flush 9742 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255362024-02-28T22:18:23.467ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255372024-02-28T22:18:24.468ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255382024-02-28T22:18:24.469ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255392024-02-28T22:18:24.469ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255402024-02-28T22:18:24.469ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255412024-02-28T22:18:24.469ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255422024-02-28T22:18:24.469ZINFOcrucible: [2] client re-new 0 jobs since flush 9742 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255432024-02-28T22:18:24.469ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255442024-02-28T22:18:25.470ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255452024-02-28T22:18:25.470ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255462024-02-28T22:18:25.470ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255472024-02-28T22:18:25.470ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255482024-02-28T22:18:25.470ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255492024-02-28T22:18:25.470ZINFOcrucible: [2] client re-new 0 jobs since flush 9742 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255502024-02-28T22:18:25.470ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
25551 Feb 28 22:18:26.467 INFO request completed, latency_us: 213, response_code: 200, uri: /state/cid/2, method: GET, req_id: b1c342a2-1bad-4366-b8ff-f1e48f5c35f5, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
25552 001/300 Read block 1850 len 32768
25553 002/300 Write block 188 len 16384 data: 8 8 6 5
25554 003/300 Write block 1150 len 20480 data: 6 6 6 6 3
25555 004/300 Flush
255562024-02-28T22:18:26.471ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255572024-02-28T22:18:26.471ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255582024-02-28T22:18:26.471ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255592024-02-28T22:18:26.471ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255602024-02-28T22:18:26.471ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255612024-02-28T22:18:26.471ZINFOcrucible: [2] client re-new 4 jobs since flush 9742 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
255622024-02-28T22:18:26.471ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
25563 005/300 Flush
25564 006/300 Read block 4764 len 28672
25565 007/300 Write block 636 len 16384 data: 4 5 5 5
25566 008/300 Write block 3760 len 40960 data: 5 5 5 5 4 4 5 4 2 3
25567 009/300 Write block 2697 len 20480 data: 5 5 6 6 6
25568 010/300 Read block 1795 len 4096
25569 011/300 Write block 2188 len 36864 data: 6 6 7 8 8 7 9 8 6
25570 012/300 Read block 196 len 36864
25571 013/300 Flush
25572 014/300 Read block 697 len 28672
25573 015/300 Flush
25574 016/300 Write block 1534 len 28672 data: 7 5 6 5 6 6 4
25575 017/300 Read block 555 len 32768
25576 018/300 Write block 2503 len 32768 data: 7 6 5 5 6 4 4 4
25577 019/300 Read block 593 len 32768
25578 020/300 Write block 1257 len 4096 data: 5
25579 021/300 Read block 884 len 40960
25580 022/300 Read block 1353 len 20480
25581 023/300 Read block 3665 len 16384
25582 024/300 Read block 4447 len 32768
25583 025/300 Write block 2300 len 20480 data: 7 7 5 3 3
25584 026/300 Read block 2823 len 40960
25585 027/300 Read block 3190 len 4096
25586 028/300 Read block 3075 len 28672
25587 029/300 Read block 4961 len 16384
25588 030/300 Read block 2896 len 8192
25589 031/300 Read block 2913 len 36864
25590 032/300 Write block 750 len 28672 data: 9 9 11 9 10 8 8
25591 033/300 Flush
25592 034/300 Read block 2291 len 12288
25593 035/300 Write block 4571 len 36864 data: 9 7 7 6 5 5 5 6 6
25594 036/300 Read block 2735 len 12288
25595 037/300 Read block 1055 len 20480
25596 038/300 Read block 4017 len 4096
25597 039/300 Write block 4761 len 24576 data: 4 5 6 5 4 5
25598 040/300 Write block 3336 len 16384 data: 3 3 4 6
25599 041/300 Write block 839 len 20480 data: 6 6 5 5 7
25600 042/300 Write block 2301 len 40960 data: 8 6 4 4 3 4 3 3 3 5
25601 043/300 Read block 1616 len 16384
25602 044/300 Read block 3879 len 4096
25603 045/300 Write block 1949 len 36864 data: 4 4 6 6 6 6 5 5 4
25604 046/300 Flush
25605 047/300 Read block 1623 len 16384
25606 048/300 Read block 730 len 16384
25607 049/300 Read block 4397 len 40960
25608 050/300 Read block 3682 len 4096
25609 051/300 Read block 3425 len 40960
25610 052/300 Flush
25611 053/300 Read block 2936 len 32768
25612 054/300 Read block 505 len 40960
25613 055/300 Write block 988 len 36864 data: 5 5 6 6 6 4 3 5 5
25614 056/300 Read block 3330 len 36864
25615 057/300 Write block 1128 len 4096 data: 10
25616 058/300 Flush
25617 059/300 Read block 28 len 32768
25618 060/300 Read block 1103 len 8192
25619 061/300 Read block 4249 len 28672
25620 062/300 Write block 547 len 40960 data: 4 4 5 4 3 4 3 4 4 2
25621 063/300 Read block 72 len 32768
25622 064/300 Write block 1431 len 32768 data: 4 5 7 6 6 6 5 4
25623 065/300 Write block 2610 len 40960 data: 8 7 5 4 4 5 4 6 6 6
25624 066/300 Write block 4938 len 36864 data: 4 4 4 6 8 8 7 7 6
25625 067/300 Write block 40 len 16384 data: 4 7 5 5
25626 068/300 Write block 1897 len 16384 data: 8 7 10 11
25627 069/300 Read block 2563 len 36864
25628 070/300 Read block 857 len 12288
25629 071/300 Read block 424 len 32768
25630 072/300 Read block 721 len 40960
25631 073/300 Flush
25632 074/300 Write block 2735 len 40960 data: 7 5 4 4 5 6 7 5 6 7
25633 075/300 Read block 4183 len 40960
25634 076/300 Read block 3378 len 12288
25635 077/300 Write block 4899 len 24576 data: 4 4 5 5 4 4
25636 078/300 Flush
25637 079/300 Write block 320 len 8192 data: 7 6
25638 080/300 Flush
25639 081/300 Write block 4824 len 24576 data: 6 3 4 6 6 6
25640 082/300 Read block 1836 len 8192
25641 083/300 Flush
25642 084/300 Read block 1426 len 32768
25643 085/300 Read block 1264 len 20480
25644 086/300 Write block 1237 len 40960 data: 5 5 5 5 4 4 4 4 3 2
25645 087/300 Read block 4506 len 12288
25646 088/300 Write block 1260 len 36864 data: 8 6 6 5 4 4 4 6 7
25647 089/300 Write block 3838 len 40960 data: 6 6 5 5 6 5 3 4 4 4
25648 090/300 Read block 4279 len 12288
25649 091/300 Read block 1368 len 32768
25650 092/300 Read block 543 len 12288
25651 093/300 Write block 3556 len 24576 data: 6 5 4 4 4 5
25652 094/300 Write block 1474 len 32768 data: 11 11 11 10 10 7 7 6
25653 095/300 Write block 1924 len 40960 data: 5 5 5 5 4 3 2 4 4 4
25654 096/300 Read block 4429 len 16384
25655 097/300 Read block 735 len 20480
25656 098/300 Write block 3160 len 28672 data: 4 5 5 7 6 5 5
25657 099/300 Write block 1819 len 40960 data: 5 5 4 4 4 4 3 4 4 5
25658 100/300 Read block 4455 len 40960
25659 101/300 Write block 3652 len 28672 data: 3 3 4 5 6 5 5
25660 102/300 Write block 1043 len 12288 data: 8 8 8
25661 103/300 Read block 3761 len 12288
25662 104/300 Write block 1760 len 12288 data: 6 7 9
25663 105/300 Read block 4359 len 20480
25664 106/300 Write block 4570 len 40960 data: 9 10 8 8 7 6 6 6 7 7
25665 107/300 Write block 4679 len 12288 data: 5 4 5
25666 108/300 Flush
25667 109/300 Read block 2124 len 28672
25668 110/300 Read block 2515 len 4096
25669 111/300 Read block 3065 len 24576
25670 112/300 Write block 3196 len 8192 data: 5 7
25671 113/300 Write block 2172 len 28672 data: 6 6 4 4 5 5 7
25672 114/300 Read block 4294 len 16384
25673 115/300 Read block 4953 len 4096
25674 116/300 Read block 4798 len 16384
25675 117/300 Read block 3646 len 20480
25676 118/300 Read block 2857 len 40960
25677 119/300 Write block 1913 len 40960 data: 3 5 6 5 5 5 5 6 5 4
25678 120/300 Read block 2873 len 28672
25679 121/300 Read block 3160 len 12288
25680 122/300 Write block 820 len 4096 data: 3
25681 123/300 Read block 1886 len 28672
25682 124/300 Read block 3062 len 8192
25683 125/300 Read block 2062 len 40960
25684 126/300 Read block 3512 len 12288
25685 127/300 Flush
25686 128/300 Read block 899 len 8192
25687 129/300 Write block 1660 len 40960 data: 5 4 4 6 4 5 5 5 3 4
25688 130/300 Write block 3286 len 8192 data: 6 6
25689 131/300 Flush
25690 132/300 Read block 1194 len 12288
25691 133/300 Write block 3348 len 40960 data: 5 6 6 5 4 3 2 3 5 7
25692 134/300 Flush
25693 135/300 Read block 1073 len 40960
25694 136/300 Read block 3445 len 28672
25695 137/300 Write block 3557 len 24576 data: 6 5 5 5 6 7
25696 138/300 Write block 977 len 24576 data: 5 7 9 8 9 8
25697 139/300 Read block 2775 len 32768
25698 140/300 Read block 34 len 16384
25699 141/300 Read block 1023 len 8192
25700 142/300 Read block 1776 len 4096
25701 143/300 Read block 4957 len 28672
25702 144/300 Write block 82 len 28672 data: 7 6 7 5 5 4 4
25703 145/300 Read block 2201 len 32768
25704 146/300 Write block 2734 len 16384 data: 7 8 6 5
25705 147/300 Write block 2956 len 8192 data: 5 5
25706 148/300 Write block 4775 len 36864 data: 5 4 4 4 3 4 5 5 5
25707 149/300 Read block 3916 len 20480
25708 150/300 Write block 1023 len 8192 data: 4 3
25709 151/300 Read block 1747 len 20480
25710 152/300 Flush
25711 153/300 Write block 2737 len 24576 data: 6 5 6 7 8 6
25712 154/300 Write block 1327 len 28672 data: 5 5 7 6 7 7 7
25713 155/300 Write block 4755 len 8192 data: 5 5
25714 156/300 Read block 2821 len 24576
25715 157/300 Write block 3430 len 16384 data: 8 7 9 10
25716 158/300 Write block 4078 len 32768 data: 7 7 6 4 3 2 4 5
25717 159/300 Read block 1834 len 24576
25718 160/300 Read block 3369 len 20480
25719 161/300 Write block 1261 len 40960 data: 7 7 6 5 5 5 7 8 6 6
25720 162/300 Flush
25721 163/300 Write block 1350 len 36864 data: 11 10 7 7 5 4 4 3 3
25722 164/300 Write block 1679 len 20480 data: 4 4 3 3 3
25723 165/300 Read block 591 len 20480
25724 166/300 Write block 3124 len 8192 data: 10 7
25725 167/300 Read block 3565 len 8192
25726 168/300 Write block 1532 len 4096 data: 7
25727 169/300 Read block 1566 len 8192
25728 170/300 Write block 4292 len 12288 data: 5 4 4
25729 171/300 Read block 1710 len 16384
25730 172/300 Write block 286 len 28672 data: 4 3 4 5 5 7 7
25731 173/300 Read block 3382 len 8192
25732 174/300 Write block 498 len 36864 data: 10 9 8 7 5 5 5 9 9
25733 175/300 Read block 1649 len 40960
25734 176/300 Read block 2792 len 8192
25735 177/300 Read block 4666 len 12288
25736 178/300 Write block 1362 len 12288 data: 6 6 3
25737 179/300 Write block 4386 len 28672 data: 7 6 3 2 2 3 3
25738 180/300 Read block 2924 len 4096
25739 181/300 Read block 12 len 36864
25740 182/300 Flush
25741 183/300 Read block 3527 len 36864
25742 184/300 Write block 382 len 28672 data: 3 3 4 4 4 4 3
25743 185/300 Write block 534 len 4096 data: 8
25744 186/300 Read block 1375 len 16384
25745 187/300 Read block 11 len 28672
25746 188/300 Write block 4008 len 36864 data: 5 7 8 10 12 13 14 12 12
25747 189/300 Read block 3023 len 28672
25748 190/300 Read block 1161 len 16384
25749 191/300 Read block 3356 len 4096
25750 192/300 Write block 585 len 20480 data: 7 8 7 7 8
25751 193/300 Write block 169 len 40960 data: 6 5 5 5 5 5 4 4 3 4
25752 194/300 Write block 1923 len 36864 data: 4 6 6 6 6 5 4 3 5
25753 195/300 Flush
25754 196/300 Write block 3682 len 32768 data: 5 6 6 5 5 3 3 4
25755 197/300 Flush
25756 198/300 Read block 720 len 4096
25757 199/300 Flush
25758 200/300 Read block 4517 len 12288
25759 201/300 Write block 302 len 40960 data: 5 6 5 6 6 6 6 5 6 6
25760 202/300 Write block 4558 len 24576 data: 5 5 5 5 5 6
25761 203/300 Write block 4780 len 20480 data: 5 6 6 6 5
25762 204/300 Read block 2728 len 28672
25763 205/300 Read block 3065 len 40960
25764 206/300 Flush
25765 207/300 Read block 4213 len 8192
25766 208/300 Flush
25767 209/300 Read block 1071 len 8192
25768 210/300 Write block 4096 len 24576 data: 5 4 4 3 4 4
25769 211/300 Read block 4338 len 24576
25770 212/300 Read block 3653 len 32768
25771 213/300 Write block 74 len 4096 data: 4
25772 214/300 Read block 1108 len 32768
25773 215/300 Read block 4113 len 36864
25774 216/300 Read block 3049 len 36864
25775 217/300 Write block 4177 len 12288 data: 6 4 5
25776 218/300 Write block 2970 len 32768 data: 6 7 7 8 7 9 8 9
25777 219/300 Read block 3239 len 40960
25778 220/300 Read block 4981 len 28672
25779 221/300 Write block 1291 len 4096 data: 6
25780 222/300 Read block 4059 len 32768
25781 223/300 Read block 2411 len 32768
25782 224/300 Write block 3996 len 36864 data: 4 5 6 6 8 7 8 8 7
25783 225/300 Read block 1261 len 8192
25784 226/300 Write block 4366 len 16384 data: 8 6 3 4
25785 227/300 Read block 1985 len 40960
25786 228/300 Flush
25787 229/300 Read block 2531 len 40960
25788 230/300 Write block 1340 len 4096 data: 2
25789 231/300 Write block 2334 len 32768 data: 4 6 9 11 11 12 9 10
25790 232/300 Write block 575 len 40960 data: 11 7 7 7 6 7 6 7 8 8
25791 233/300 Read block 4248 len 8192
25792 234/300 Read block 2186 len 8192
25793 235/300 Read block 2702 len 36864
25794 236/300 Read block 55 len 8192
25795 237/300 Read block 2687 len 24576
25796 238/300 Read block 18 len 32768
25797 239/300 Write block 3344 len 4096 data: 5
25798 240/300 Read block 2287 len 4096
25799 241/300 Write block 1037 len 4096 data: 6
25800 242/300 Read block 4858 len 40960
25801 243/300 Read block 4394 len 12288
25802 244/300 Read block 822 len 12288
25803 245/300 Read block 597 len 4096
25804 246/300 Read block 299 len 36864
25805 247/300 Write block 4799 len 36864 data: 7 6 11 10 9 10 8 8 7
25806 248/300 Read block 932 len 8192
25807 249/300 Write block 3203 len 36864 data: 9 9 9 9 9 7 6 5 6
25808 250/300 Read block 4512 len 20480
25809 251/300 Read block 2776 len 12288
25810 252/300 Flush
25811 253/300 Read block 1885 len 8192
25812 254/300 Read block 4864 len 28672
25813 255/300 Write block 1767 len 20480 data: 6 6 7 9 10
25814 256/300 Flush
25815 257/300 Write block 4244 len 28672 data: 8 7 7 7 7 8 7
25816 258/300 Read block 2210 len 36864
25817 259/300 Read block 1805 len 40960
25818 260/300 Read block 1898 len 16384
25819 261/300 Read block 3431 len 12288
25820 262/300 Write block 1146 len 16384 data: 4 3 4 4
25821 263/300 Write block 3074 len 32768 data: 8 8 7 9 8 7 6 6
25822 264/300 Read block 2160 len 36864
25823 265/300 Read block 3893 len 40960
25824 266/300 Read block 4331 len 8192
25825 267/300 Write block 4324 len 16384 data: 5 4 4 5
25826 268/300 Write block 3969 len 36864 data: 4 7 10 8 8 7 8 8 5
25827 269/300 Flush
25828 270/300 Write block 4831 len 20480 data: 5 5 5 5 6
25829 271/300 Read block 3115 len 40960
25830 272/300 Read block 3505 len 20480
25831 273/300 Read block 935 len 28672
25832 274/300 Read block 3421 len 24576
25833 275/300 Read block 2419 len 24576
25834 276/300 Read block 1897 len 24576
25835 277/300 Write block 966 len 4096 data: 7
25836 278/300 Flush
25837 279/300 Flush
25838 280/300 Write block 4187 len 16384 data: 7 8 7 6
25839 281/300 Read block 4984 len 36864
25840 282/300 Write block 3207 len 32768 data: 10 8 7 6 7 7 6 7
25841 283/300 Write block 4287 len 28672 data: 6 5 5 4 5 6 5
25842 284/300 Read block 4556 len 32768
25843 285/300 Write block 113 len 16384 data: 6 5 6 6
25844 286/300 Read block 4226 len 16384
25845 287/300 Flush
25846 288/300 Write block 2178 len 4096 data: 8
25847 289/300 Write block 2135 len 20480 data: 5 6 5 4 5
25848 290/300 Flush
25849 291/300 Read block 160 len 20480
25850 292/300 Read block 732 len 40960
25851 293/300 Write block 3478 len 24576 data: 7 8 7 5 6 6
25852 294/300 Read block 480 len 16384
25853 295/300 Flush
25854 296/300 Read block 7 len 36864
25855 297/300 Write block 2113 len 16384 data: 6 7 6 7
25856 298/300 Write block 2626 len 28672 data: 3 3 2 2 2 2 2
25857 299/300 Read block 3794 len 8192
25858 300/300 Write block 4758 len 8192 data: 5 5
25859 Main task has work to do, go find it
25860 got dsc Start(2)
25861 start 2
25862 Feb 28 22:18:26.815 INFO request completed, latency_us: 210, response_code: 204, uri: /start/cid/2, method: POST, req_id: a909ce63-bd28-4476-8706-54e9e48181e7, remote_addr: 127.0.0.1:36617, local_addr: 127.0.0.1:9998
25863 Replay: started 2, returned:Ok(())
25864 ----------------------------------------------------------------
25865 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
25866 States: Active Active Offline
25867 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
25868 8744 Acked 9743 Read 8 Done Done New true
25869 8745 Acked 9744 Write 4 Done Done New true
25870 8746 Acked 9745 Write 5 Done Done New true
25871 8747 Acked 9746 Flush 0 Done Done New true
25872 8748 Acked 9747 Flush 0 Done Done New false
25873 8749 Acked 9748 Read 7 Done Done New false
25874 8750 Acked 9749 Write 4 Done Done New false
25875 8751 Acked 9750 Write 10 Done Done New false
25876 8752 Acked 9751 Write 5 Done Done New false
25877 8753 Acked 9752 Read 1 Done Done New false
25878 8754 Acked 9753 Write 9 Done Done New false
25879 8755 Acked 9754 Read 9 Done Done New false
25880 8756 Acked 9755 Flush 0 Done Done New false
25881 8757 Acked 9756 Read 7 Done Done New false
25882 8758 Acked 9757 Flush 0 Done Done New false
25883 8759 Acked 9758 Write 7 Done Done New false
25884 8760 Acked 9759 Read 8 Done Done New false
25885 8761 Acked 9760 Write 8 Done Done New false
25886 8762 Acked 9761 Read 8 Done Done New false
25887 8763 Acked 9762 Write 1 Done Done New false
25888 8764 Acked 9763 Read 10 Done Done New false
25889 8765 Acked 9764 Read 5 Done Done New false
25890 8766 Acked 9765 Read 4 Done Done New false
25891 8767 Acked 9766 Read 8 Done Done New false
25892 8768 Acked 9767 Write 5 Done Done New false
25893 8769 Acked 9768 Read 10 Done Done New false
25894 8770 Acked 9769 Read 1 Done Done New false
25895 8771 Acked 9770 Read 7 Done Done New false
25896 8772 Acked 9771 Read 4 Done Done New false
25897 8773 Acked 9772 Read 2 Done Done New false
25898 8774 Acked 9773 Read 9 Done Done New false
25899 8775 Acked 9774 Write 7 Done Done New false
25900 8776 Acked 9775 Flush 0 Done Done New false
25901 8777 Acked 9776 Read 3 Done Done New false
25902 8778 Acked 9777 Write 9 Done Done New false
25903 8779 Acked 9778 Read 3 Done Done New false
25904 8780 Acked 9779 Read 5 Done Done New false
25905 8781 Acked 9780 Read 1 Done Done New false
25906 8782 Acked 9781 Write 6 Done Done New false
25907 8783 Acked 9782 Write 4 Done Done New false
25908 8784 Acked 9783 Write 5 Done Done New false
25909 8785 Acked 9784 Write 10 Done Done New false
25910 8786 Acked 9785 Read 4 Done Done New false
25911 8787 Acked 9786 Read 1 Done Done New false
25912 8788 Acked 9787 Write 9 Done Done New false
25913 8789 Acked 9788 Flush 0 Done Done New false
25914 8790 Acked 9789 Read 4 Done Done New false
25915 8791 Acked 9790 Read 4 Done Done New false
25916 8792 Acked 9791 Read 10 Done Done New false
25917 8793 Acked 9792 Read 1 Done Done New false
25918 8794 Acked 9793 Read 10 Done Done New false
25919 8795 Acked 9794 Flush 0 Done Done New false
25920 8796 Acked 9795 Read 8 Done Done New false
25921 8797 Acked 9796 Read 10 Done Done New false
25922 8798 Acked 9797 Write 9 Done Done New false
25923 8799 Acked 9798 Read 9 Done Done New false
25924 8800 Acked 9799 Write 1 Done Done New false
25925 8801 Acked 9800 Flush 0 Done Done New false
25926 8802 Acked 9801 Read 8 Done Done New false
25927 8803 Acked 9802 Read 2 Done Done New false
25928 8804 Acked 9803 Read 7 Done Done New false
25929 8805 Acked 9804 Write 10 Done Done New false
25930 8806 Acked 9805 Read 8 Done Done New false
25931 8807 Acked 9806 Write 8 Done Done New false
25932 8808 Acked 9807 Write 10 Done Done New false
25933 8809 Acked 9808 Write 9 Done Done New false
25934 8810 Acked 9809 Write 4 Done Done New false
25935 8811 Acked 9810 Write 4 Done Done New false
25936 8812 Acked 9811 Read 9 Done Done New false
25937 8813 Acked 9812 Read 3 Done Done New false
25938 8814 Acked 9813 Read 8 Done Done New false
25939 8815 Acked 9814 Read 10 Done Done New false
25940 8816 Acked 9815 Flush 0 Done Done New false
25941 8817 Acked 9816 Write 10 Done Done New false
25942 8818 Acked 9817 Read 10 Done Done New false
25943 8819 Acked 9818 Read 3 Done Done New false
25944 8820 Acked 9819 Write 6 Done Done New false
25945 8821 Acked 9820 Flush 0 Done Done New false
25946 8822 Acked 9821 Write 2 Done Done New false
25947 8823 Acked 9822 Flush 0 Done Done New false
25948 8824 Acked 9823 Write 6 Done Done New false
25949 8825 Acked 9824 Read 2 Done Done New false
25950 8826 Acked 9825 Flush 0 Done Done New false
25951 8827 Acked 9826 Read 8 Done Done New false
25952 8828 Acked 9827 Read 5 Done Done New false
25953 8829 Acked 9828 Write 10 Done Done New false
25954 8830 Acked 9829 Read 3 Done Done New false
25955 8831 Acked 9830 Write 9 Done Done New false
25956 8832 Acked 9831 Write 10 Done Done New false
25957 8833 Acked 9832 Read 3 Done Done New false
25958 8834 Acked 9833 Read 8 Done Done New false
25959 8835 Acked 9834 Read 3 Done Done New false
25960 8836 Acked 9835 Write 6 Done Done New false
25961 8837 Acked 9836 Write 8 Done Done New false
25962 8838 Acked 9837 Write 10 Done Done New false
25963 8839 Acked 9838 Read 4 Done Done New false
25964 8840 Acked 9839 Read 5 Done Done New false
25965 8841 Acked 9840 Write 7 Done Done New false
25966 8842 Acked 9841 Write 10 Done Done New false
25967 8843 Acked 9842 Read 10 Done Done New false
25968 8844 Acked 9843 Write 7 Done Done New false
25969 8845 Acked 9844 Write 3 Done Done New false
25970 8846 Acked 9845 Read 3 Done Done New false
25971 8847 Acked 9846 Write 3 Done Done New false
25972 8848 Acked 9847 Read 5 Done Done New false
25973 8849 Acked 9848 Write 10 Done Done New false
25974 8850 Acked 9849 Write 3 Done Done New false
25975 8851 Acked 9850 Flush 0 Done Done New false
25976 8852 Acked 9851 Read 7 Done Done New false
25977 8853 Acked 9852 Read 1 Done Done New false
25978 8854 Acked 9853 Read 6 Done Done New false
25979 8855 Acked 9854 Write 2 Done Done New false
25980 8856 Acked 9855 Write 7 Done Done New false
25981 8857 Acked 9856 Read 4 Done Done New false
25982 8858 Acked 9857 Read 1 Done Done New false
25983 8859 Acked 9858 Read 4 Done Done New false
25984 8860 Acked 9859 Read 5 Done Done New false
25985 8861 Acked 9860 Read 10 Done Done New false
25986 8862 Acked 9861 Write 10 Done Done New false
25987 8863 Acked 9862 Read 7 Done Done New false
25988 8864 Acked 9863 Read 3 Done Done New false
25989 8865 Acked 9864 Write 1 Done Done New false
25990 8866 Acked 9865 Read 7 Done Done New false
25991 8867 Acked 9866 Read 2 Done Done New false
25992 8868 Acked 9867 Read 10 Done Done New false
25993 8869 Acked 9868 Read 3 Done Done New false
25994 8870 Acked 9869 Flush 0 Done Done New false
25995 8871 Acked 9870 Read 2 Done Done New false
25996 8872 Acked 9871 Write 10 Done Done New false
25997 8873 Acked 9872 Write 2 Done Done New false
25998 8874 Acked 9873 Flush 0 Done Done New false
25999 8875 Acked 9874 Read 3 Done Done New false
26000 8876 Acked 9875 Write 10 Done Done New false
26001 8877 Acked 9876 Flush 0 Done Done New false
26002 8878 Acked 9877 Read 10 Done Done New false
26003 8879 Acked 9878 Read 7 Done Done New false
26004 8880 Acked 9879 Write 6 Done Done New false
26005 8881 Acked 9880 Write 6 Done Done New false
26006 8882 Acked 9881 Read 8 Done Done New false
26007 8883 Acked 9882 Read 4 Done Done New false
26008 8884 Acked 9883 Read 2 Done Done New false
26009 8885 Acked 9884 Read 1 Done Done New false
26010 8886 Acked 9885 Read 7 Done Done New false
26011 8887 Acked 9886 Write 7 Done Done New false
26012 8888 Acked 9887 Read 8 Done Done New false
26013 8889 Acked 9888 Write 4 Done Done New false
26014 8890 Acked 9889 Write 2 Done Done New false
26015 8891 Acked 9890 Write 9 Done Done New false
26016 8892 Acked 9891 Read 5 Done Done New false
26017 8893 Acked 9892 Write 2 Done Done New false
26018 8894 Acked 9893 Read 5 Done Done New false
26019 8895 Acked 9894 Flush 0 Done Done New false
26020 8896 Acked 9895 Write 6 Done Done New false
26021 8897 Acked 9896 Write 7 Done Done New false
26022 8898 Acked 9897 Write 2 Done Done New false
26023 8899 Acked 9898 Read 6 Done Done New false
26024 8900 Acked 9899 Write 4 Done Done New false
26025 8901 Acked 9900 Write 8 Done Done New false
26026 8902 Acked 9901 Read 6 Done Done New false
26027 8903 Acked 9902 Read 5 Done Done New false
26028 8904 Acked 9903 Write 10 Done Done New false
26029 8905 Acked 9904 Flush 0 Done Done New false
26030 8906 Acked 9905 Write 9 Done Done New false
26031 8907 Acked 9906 Write 5 Done Done New false
26032 8908 Acked 9907 Read 5 Done Done New false
26033 8909 Acked 9908 Write 2 Done Done New false
26034 8910 Acked 9909 Read 2 Done Done New false
26035 8911 Acked 9910 Write 1 Done Done New false
26036 8912 Acked 9911 Read 2 Done Done New false
26037 8913 Acked 9912 Write 3 Done Done New false
26038 8914 Acked 9913 Read 4 Done Done New false
26039 8915 Acked 9914 Write 7 Done Done New false
26040 8916 Acked 9915 Read 2 Done Done New false
26041 8917 Acked 9916 Write 9 Done Done New false
26042 8918 Acked 9917 Read 10 Done Done New false
26043 8919 Acked 9918 Read 2 Done Done New false
26044 8920 Acked 9919 Read 3 Done Done New false
26045 8921 Acked 9920 Write 3 Done Done New false
26046 8922 Acked 9921 Write 7 Done Done New false
26047 8923 Acked 9922 Read 1 Done Done New false
26048 8924 Acked 9923 Read 9 Done Done New false
26049 8925 Acked 9924 Flush 0 Done Done New false
26050 8926 Acked 9925 Read 9 Done Done New false
26051 8927 Acked 9926 Write 7 Done Done New false
26052 8928 Acked 9927 Write 1 Done Done New false
26053 8929 Acked 9928 Read 4 Done Done New false
26054 8930 Acked 9929 Read 7 Done Done New false
26055 8931 Acked 9930 Write 9 Done Done New false
26056 8932 Acked 9931 Read 7 Done Done New false
26057 8933 Acked 9932 Read 4 Done Done New false
26058 8934 Acked 9933 Read 1 Done Done New false
26059 8935 Acked 9934 Write 5 Done Done New false
26060 8936 Acked 9935 Write 10 Done Done New false
26061 8937 Acked 9936 Write 9 Done Done New false
26062 8938 Acked 9937 Flush 0 Done Done New false
26063 8939 Acked 9938 Write 8 Done Done New false
26064 8940 Acked 9939 Flush 0 Done Done New false
26065 8941 Acked 9940 Read 1 Done Done New false
26066 8942 Acked 9941 Flush 0 Done Done New false
26067 8943 Acked 9942 Read 3 Done Done New false
26068 8944 Acked 9943 Write 10 Done Done New false
26069 8945 Acked 9944 Write 6 Done Done New false
26070 8946 Acked 9945 Write 5 Done Done New false
26071 8947 Acked 9946 Read 7 Done Done New false
26072 8948 Acked 9947 Read 10 Done Done New false
26073 8949 Acked 9948 Flush 0 Done Done New false
26074 8950 Acked 9949 Read 2 Done Done New false
26075 8951 Acked 9950 Flush 0 Done Done New false
26076 8952 Acked 9951 Read 2 Done Done New false
26077 8953 Acked 9952 Write 6 Done Done New false
26078 8954 Acked 9953 Read 6 Done Done New false
26079 8955 Acked 9954 Read 8 Done Done New false
26080 8956 Acked 9955 Write 1 Done Done New false
26081 8957 Acked 9956 Read 8 Done Done New false
26082 8958 Acked 9957 Read 9 Done Done New false
26083 8959 Acked 9958 Read 9 Done Done New false
26084 8960 Acked 9959 Write 3 Done Done New false
26085 8961 Acked 9960 Write 8 Done Done New false
26086 8962 Acked 9961 Read 10 Done Done New false
26087 8963 Acked 9962 Read 7 Done Done New false
26088 8964 Acked 9963 Write 1 Done Done New false
26089 8965 Acked 9964 Read 8 Done Done New false
26090 8966 Acked 9965 Read 8 Done Done New false
26091 8967 Acked 9966 Write 9 Done Done New false
26092 8968 Acked 9967 Read 2 Done Done New false
26093 8969 Acked 9968 Write 4 Done Done New false
26094 8970 Acked 9969 Read 10 Done Done New false
26095 8971 Acked 9970 Flush 0 Done Done New false
26096 8972 Acked 9971 Read 10 Done Done New false
26097 8973 Acked 9972 Write 1 Done Done New false
26098 8974 Acked 9973 Write 8 Done Done New false
26099 8975 Acked 9974 Write 10 Done Done New false
26100 8976 Acked 9975 Read 2 Done Done New false
26101 8977 Acked 9976 Read 2 Done Done New false
26102 8978 Acked 9977 Read 9 Done Done New false
26103 8979 Acked 9978 Read 2 Done Done New false
26104 8980 Acked 9979 Read 6 Done Done New false
26105 8981 Acked 9980 Read 8 Done Done New false
26106 8982 Acked 9981 Write 1 Done Done New false
26107 8983 Acked 9982 Read 1 Done Done New false
26108 8984 Acked 9983 Write 1 Done Done New false
26109 8985 Acked 9984 Read 10 Done Done New false
26110 8986 Acked 9985 Read 3 Done Done New false
26111 8987 Acked 9986 Read 3 Done Done New false
26112 8988 Acked 9987 Read 1 Done Done New false
26113 8989 Acked 9988 Read 9 Done Done New false
26114 8990 Acked 9989 Write 9 Done Done New false
26115 8991 Acked 9990 Read 2 Done Done New false
26116 8992 Acked 9991 Write 9 Done Done New false
26117 8993 Acked 9992 Read 5 Done Done New false
26118 8994 Acked 9993 Read 3 Done Done New false
26119 8995 Acked 9994 Flush 0 Done Done New false
26120 8996 Acked 9995 Read 2 Done Done New false
26121 8997 Acked 9996 Read 7 Done Done New false
26122 8998 Acked 9997 Write 5 Done Done New false
26123 8999 Acked 9998 Flush 0 Done Done New false
26124 9000 Acked 9999 Write 7 Done Done New false
26125 9001 Acked 10000 Read 9 Done Done New false
26126 9002 Acked 10001 Read 10 Done Done New false
26127 9003 Acked 10002 Read 4 Done Done New false
26128 9004 Acked 10003 Read 3 Done Done New false
26129 9005 Acked 10004 Write 4 Done Done New false
26130 9006 Acked 10005 Write 8 Done Done New false
26131 9007 Acked 10006 Read 9 Done Done New false
26132 9008 Acked 10007 Read 10 Done Done New false
26133 9009 Acked 10008 Read 2 Done Done New false
26134 9010 Acked 10009 Write 4 Done Done New false
26135 9011 Acked 10010 Write 9 Done Done New false
26136 9012 Acked 10011 Flush 0 Done Done New false
26137 9013 Acked 10012 Write 5 Done Done New false
26138 9014 Acked 10013 Read 10 Done Done New false
26139 9015 Acked 10014 Read 5 Done Done New false
26140 9016 Acked 10015 Read 7 Done Done New false
26141 9017 Acked 10016 Read 6 Done Done New false
26142 9018 Acked 10017 Read 6 Done Done New false
26143 9019 Acked 10018 Read 6 Done Done New false
26144 9020 Acked 10019 Write 1 Done Done New false
26145 9021 Acked 10020 Flush 0 Done Done New false
26146 9022 Acked 10021 Flush 0 Done Done New false
26147 9023 Acked 10022 Write 4 Done Done New false
26148 9024 Acked 10023 Read 9 Done Done New false
26149 9025 Acked 10024 Write 8 Done Done New false
26150 9026 Acked 10025 Write 7 Done Done New false
26151 9027 Acked 10026 Read 8 Done Done New false
26152 9028 Acked 10027 Write 4 Done Done New false
26153 9029 Acked 10028 Read 4 Done Done New false
26154 9030 Acked 10029 Flush 0 Done Done New false
26155 9031 Acked 10030 Write 1 Done Done New false
26156 9032 Acked 10031 Write 5 Done Done New false
26157 9033 Acked 10032 Flush 0 Done Done New false
26158 9034 Acked 10033 Read 5 Done Done New false
26159 9035 Acked 10034 Read 10 Done Done New false
26160 9036 Acked 10035 Write 6 Done Done New false
26161 9037 Acked 10036 Read 4 Done Done New false
26162 9038 Acked 10037 Flush 0 Done Done New false
26163 9039 Acked 10038 Read 9 Done Done New false
26164 9040 Acked 10039 Write 4 Done Done New false
26165 9041 Acked 10040 Write 7 Done Done New false
26166 9042 Acked 10041 Read 2 Done Done New false
26167 9043 Acked 10042 Write 2 Done Done New false
26168 STATES DS:0 DS:1 DS:2 TOTAL
26169 New 0 0 300 300
26170 Sent 0 0 0 0
26171 Done 300 300 0 600
26172 Skipped 0 0 0 0
26173 Error 0 0 0 0
26174 Last Flush: 10037 10037 9742
26175 Downstairs last five completed: 9742 9741 9740 9739 9738
26176 Upstairs last five completed: 9043 9042 9041 9040 9039
26177 CLIENT: Up:0 ds:300 act:2
261782024-02-28T22:18:27.472ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
261792024-02-28T22:18:27.472ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
261802024-02-28T22:18:27.472ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
261812024-02-28T22:18:27.472ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
261822024-02-28T22:18:27.472ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
261832024-02-28T22:18:27.472ZINFOcrucible: [2] client re-new 301 jobs since flush 9742 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
261842024-02-28T22:18:27.473ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
26185 [8830] got start action
26186 [8830] I am going to restart
26187 Starting downstairs at port 8830
26188 Make output file at "/tmp/dsc/downstairs-8830.txt"
26189 [8830][2] reports Starting
261902024-02-28T22:18:28.473ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
261912024-02-28T22:18:28.473ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
261922024-02-28T22:18:28.473ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
261932024-02-28T22:18:28.474ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
261942024-02-28T22:18:28.474ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
261952024-02-28T22:18:28.474ZINFOcrucible: [2] client re-new 301 jobs since flush 9742 = downstairs session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
261962024-02-28T22:18:28.474ZINFOcrucible: client task is exiting = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
26197 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1194)
26198 [8830][2] reports Running
261992024-02-28T22:18:29.476ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
262002024-02-28T22:18:29.476ZINFOcrucible: ds_connection connected = io task client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
262012024-02-28T22:18:29.477ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
262022024-02-28T22:18:29.477ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
262032024-02-28T22:18:29.477ZINFOcrucible: send last flush ID to this DS: 9742 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
262042024-02-28T22:18:29.477ZINFOcrucible: Replied this last flush ID: 9742 = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
262052024-02-28T22:18:29.478ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
262062024-02-28T22:18:29.478ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
262072024-02-28T22:18:29.478ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
262082024-02-28T22:18:29.478ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = 69c7fbb2-d5b2-4965-9fa4-441c4d60207e
26209 ----------------------------------------------------------------
26210 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
26211 Downstairs last five completed: 10043 10042 10041 10040 10039
26212 Upstairs last five completed: 9044 9043 9042 9041 9040
26213 CLIENT: Up:0 ds:0 act:3
26214 Replay: All jobs finished, all DS active.
26215 Test replay has completed
26216 Wrote out file "alan"
26217 CLIENT: Tests done. All submitted work has been ACK'd
26218 ----------------------------------------------------------------
26219 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
26220 Downstairs last five completed: 10043 10042 10041 10040 10039
26221 Upstairs last five completed: 9044 9043 9042 9041 9040
26222 CLIENT: Up:0 ds:0 act:3
26223 CLIENT: All jobs finished, all DS active.
26224 Replays:30 time: 5:17
26225 Do final verify
262262024-02-28T22:18:30.849ZINFOcrucible: Upstairs starts
26227 {"msg":"Crucible runtime is spawned
26228 Crucible Version: BuildInfo {\n version: \"0.0.1\",\n git_sha: \"80b6f734175c2bc8e222c8d6f7875e53cbd5c9ca\",\n git_commit_timestamp: \"2024-02-28T21:51:33.000000000Z\",\n git_branch: \"main\",\n rustc_semver: \"1.75.0\",\n rustc_channel: \"stable\",\n rustc_host_triple: \"x86_64-unknown-illumos\",\n rustc_commit_sha: \"82e1608dfa6e0b5569232559e3d385fea5a93112\",\n cargo_triple: \"x86_64-unknown-illumos\",\n debug: true,\n opt_level: 0,\n}","v":0,"name":"crucible","level":30,"time":"2024-02-28T22:18:30.850216891Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1199}
262292024-02-28T22:18:30.850ZINFOcrucible: Upstairs <-> Downstairs Message Version: 5
262302024-02-28T22:18:30.850ZINFOcrucible: Crucible 72df7840-e60c-4c5b-a3ab-777e13f32496 has session id: db98faaa-8c79-4ef0-8d67-80b76617f989 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262312024-02-28T22:18:30.850ZINFOcrucible: Upstairs opts: Upstairs UUID: 72df7840-e60c-4c5b-a3ab-777e13f32496, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262322024-02-28T22:18:30.850ZINFOcrucible: Crucible stats registered with UUID: 72df7840-e60c-4c5b-a3ab-777e13f32496 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262332024-02-28T22:18:30.850ZINFOcrucible: The guest has requested activation with gen:3
262342024-02-28T22:18:30.850ZINFOcrucible: 72df7840-e60c-4c5b-a3ab-777e13f32496 active request set session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262352024-02-28T22:18:30.850ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262362024-02-28T22:18:30.850ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 0 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262372024-02-28T22:18:30.850ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262382024-02-28T22:18:30.850ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 1 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262392024-02-28T22:18:30.850ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262402024-02-28T22:18:30.850ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 2 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262412024-02-28T22:18:30.850ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262422024-02-28T22:18:30.850ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262432024-02-28T22:18:30.850ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262442024-02-28T22:18:30.850ZINFOcrucible: ds_connection connected = io task client = 0 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262452024-02-28T22:18:30.850ZINFOcrucible: ds_connection connected = io task client = 2 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262462024-02-28T22:18:30.850ZINFOcrucible: ds_connection connected = io task client = 1 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262472024-02-28T22:18:30.851ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 1 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262482024-02-28T22:18:30.851ZINFOcrucible: [1] Transition from New to WaitActive = downstairs client = 1 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262492024-02-28T22:18:30.851ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 2 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262502024-02-28T22:18:30.851ZINFOcrucible: [2] Transition from New to WaitActive = downstairs client = 2 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262512024-02-28T22:18:30.851ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 0 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262522024-02-28T22:18:30.851ZINFOcrucible: [0] Transition from New to WaitActive = downstairs client = 0 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262532024-02-28T22:18:30.852ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262542024-02-28T22:18:30.852ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262552024-02-28T22:18:30.852ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262562024-02-28T22:18:30.852ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262572024-02-28T22:18:30.852ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262582024-02-28T22:18:30.852ZINFOcrucible: Waiting for true more clients to be ready session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262592024-02-28T22:18:30.852ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262602024-02-28T22:18:30.853ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262612024-02-28T22:18:30.853ZINFOcrucible: Waiting for true more clients to be ready session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262622024-02-28T22:18:30.853ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262632024-02-28T22:18:30.853ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262642024-02-28T22:18:30.853ZINFOcrucible: [0]R flush_numbers[0..12]: [928, 934, 922, 926, 922, 929, 906, 908, 917, 932, 929, 931] = downstairs session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262652024-02-28T22:18:30.853ZINFOcrucible: [0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262662024-02-28T22:18:30.853ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
26267 {"msg":"[1]R flush_numbers[0..12]: [928, 934, 922, 926, 922, 929, 906, 908, 917, 932, 929, 931]","v":0,"name":"crucible","level":30,"time"Wait for a query_work_queue command to finish before sending IO
26268 :"2024-02-28T22:18:30.853180296Z","hostname":"ip-10-150-1-9.us-west-2.compute.internal","pid":1199,"":"downstairs","":"downstairs","session_id":"db98faaa-8c79-4ef0-8d67-80b76617f989"}
262692024-02-28T22:18:30.853ZINFOcrucible: [1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262702024-02-28T22:18:30.853ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262712024-02-28T22:18:30.853ZINFOcrucible: [2]R flush_numbers[0..12]: [928, 934, 922, 926, 922, 929, 906, 908, 917, 932, 929, 931] = downstairs session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262722024-02-28T22:18:30.853ZINFOcrucible: [2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262732024-02-28T22:18:30.853ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262742024-02-28T22:18:30.853ZINFOcrucible: Max found gen is 3 = downstairs session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262752024-02-28T22:18:30.853ZINFOcrucible: Generation requested: 3 >= found:3 = downstairs session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262762024-02-28T22:18:30.853ZINFOcrucible: Next flush: 938 = downstairs session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262772024-02-28T22:18:30.853ZINFOcrucible: All extents match = downstairs session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262782024-02-28T22:18:30.853ZINFOcrucible: No downstairs reconciliation required session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262792024-02-28T22:18:30.853ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262802024-02-28T22:18:30.853ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262812024-02-28T22:18:30.853ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
26282 Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
262832024-02-28T22:18:30.853ZINFOcrucible: All required reconciliation work is completed session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262842024-02-28T22:18:30.853ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262852024-02-28T22:18:30.853ZINFOcrucible: 72df7840-e60c-4c5b-a3ab-777e13f32496 is now active with session: db98faaa-8c79-4ef0-8d67-80b76617f989 session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262862024-02-28T22:18:30.853ZINFOcrucible: Set Active after no reconciliation session_id = db98faaa-8c79-4ef0-8d67-80b76617f989
262872024-02-28T22:18:30.853ZINFOcrucible: The guest has finished waiting for activation with:3
26288 Loading write count information from file "alan"
26289 Read and Verify all blocks (0..5000 range:false)
26290 Wrote out file "alan"
26291 Verify test completed
26292 Wrote out file "alan"
26293 CLIENT: Tests done. All submitted work has been ACK'd
26294 ----------------------------------------------------------------
26295 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:4
26296 States: Active Active Active
26297 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
26298 48 Acked 1047 Read 100 Done Done Done false
26299 49 Acked 1048 Read 100 Done Done Done false
26300 50 Acked 1049 Read 100 Done Done Done false
26301 51 Acked 1050 Read 100 Done Done Done false
26302 STATES DS:0 DS:1 DS:2 TOTAL
26303 New 0 0 0 0
26304 Sent 0 0 0 0
26305 Done 4 4 4 12
26306 Skipped 0 0 0 0
26307 Error 0 0 0 0
26308 Last Flush: 1046 1046 1046
26309 Downstairs last five completed: 1046 1045 1044 1043 1042
26310 Upstairs last five completed: 51 50 49 48 47
26311 CLIENT: Up:0 ds:4 act:3
26312 ----------------------------------------------------------------
26313 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
26314 Downstairs last five completed: 1051 1050 1049 1048 1047
26315 Upstairs last five completed: 52 51 50 49 48
26316 CLIENT: Up:0 ds:0 act:3
26317 CLIENT: All crucible jobs finished, exiting program
26318 Feb 28 22:18:35.467 INFO accepted connection, remote_addr: 127.0.0.1:39231, local_addr: 127.0.0.1:9998
26319 Main task has work to do, go find it
26320 got dsc Shutdown
26321 Shutdown
26322 Shut it down
26323 Feb 28 22:18:35.468 INFO request completed, latency_us: 163, response_code: 204, uri: /shutdown[8820] Disable keep_running
26324 , [8820] Got stop action so:false kr:false
26325 method: POST, req_id: dca8c279-c46e-443f-aad6-fff7dbdc8ccf, remote_addr: 127.[8830] Disable keep_running
26326 0.0[8810] Disable keep_running
26327 .[8810] Got stop action so:false kr:false
26328 1:39231, local_addr: 127.0.0.1:9998
26329 [8830] Got stop action so:false kr:false
26330 [8830] Exited with: ExitStatus(unix_wait_status(9))
26331 [8810] Exited with: ExitStatus(unix_wait_status(9))
26332 [8820] Exited with: ExitStatus(unix_wait_status(9))
26333 [8830][2] reports Exit
26334 [8810][0] reports Exit
26335 [8820][1] reports Exit
26336 February 28, 2024 at 10:18:40 PM UTC Test ends with 0