Max level shown:
1 starting February 27, 2024 at 09:44:31 PM UTC
2 Creating downstairs regions
3 Creating region directory at: /var/tmp/dsc/region
4 Creating dsc directory at: /tmp/dsc
5 Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.2845883
6 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.06717852
7 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.06437937
8 Region set with 3 regions was created
9 Starting downstairs
10 Using existing output directory "/tmp/dsc"
11 start ds: 8810
12 start access at:127.0.0.1:9998
13 start ds: 8820
14 start ds: 8830
15 Starting downstairs at port 8810
16 Starting downstairs at port 8820
17 Make output file at "/tmp/dsc/downstairs-8810.txt"
18 Starting downstairs at port 8830
19 Make output file at "/tmp/dsc/downstairs-8830.txt"
20 Make output file at "/tmp/dsc/downstairs-8820.txt"
21 [8810][0] reports Starting
22 [8820][1] reports Starting
23 [8830][2] reports Starting
24 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1157)
25 [8810][0] reports Running
26 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1158)
27 [8820][1] reports Running
28 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1159)
29 [8830][2] reports Running
30 Feb 27 21:44:31.598 INFO listening, local_addr: 127.0.0.1:9998
31 Control access at:127.0.0.1:9998
32 Running initial fill
332024-02-27T21:44:36.615ZINFOcrucible: Upstairs starts
342024-02-27T21:44:36.616ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "f6c40444ff68361c835e88ee91c409e8deb48d7d", git_commit_timestamp: "2024-02-27T21:23:06.000000000Z", git_branch: "main", rustc_semver: "1.75.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "82e1608dfa6e0b5569232559e3d385fea5a93112", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
352024-02-27T21:44:36.616ZINFOcrucible: Upstairs <-> Downstairs Message Version: 5
362024-02-27T21:44:36.616ZINFOcrucible: Crucible ef2b48cb-019e-44e3-9b38-aa1793a88801 has session id: ae3a9757-cd3e-4be1-ab60-acc44f80befc session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
372024-02-27T21:44:36.616ZINFOcrucible: Upstairs opts: Upstairs UUID: ef2b48cb-019e-44e3-9b38-aa1793a88801, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
382024-02-27T21:44:36.616ZINFOcrucible: Crucible stats registered with UUID: ef2b48cb-019e-44e3-9b38-aa1793a88801 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
39 Crucible runtime is spawned
402024-02-27T21:44:36.616ZINFOcrucible: The guest has requested activation with gen:1
412024-02-27T21:44:36.616ZINFOcrucible: ef2b48cb-019e-44e3-9b38-aa1793a88801 active request set session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
422024-02-27T21:44:36.616ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
432024-02-27T21:44:36.616ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 0 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
442024-02-27T21:44:36.616ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
452024-02-27T21:44:36.616ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 1 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
462024-02-27T21:44:36.616ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
472024-02-27T21:44:36.616ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 2 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
482024-02-27T21:44:36.616ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
492024-02-27T21:44:36.616ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
502024-02-27T21:44:36.616ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
512024-02-27T21:44:36.617ZINFOcrucible: ds_connection connected = io task client = 1 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
522024-02-27T21:44:36.617ZINFOcrucible: ds_connection connected = io task client = 0 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
532024-02-27T21:44:36.617ZINFOcrucible: ds_connection connected = io task client = 2 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
542024-02-27T21:44:36.619ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 1 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
552024-02-27T21:44:36.619ZINFOcrucible: [1] Transition from New to WaitActive = downstairs client = 1 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
562024-02-27T21:44:36.619ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 2 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
572024-02-27T21:44:36.619ZINFOcrucible: [2] Transition from New to WaitActive = downstairs client = 2 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
582024-02-27T21:44:36.619ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 0 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
592024-02-27T21:44:36.619ZINFOcrucible: [0] Transition from New to WaitActive = downstairs client = 0 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
602024-02-27T21:44:36.620ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
612024-02-27T21:44:36.620ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
622024-02-27T21:44:36.620ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
632024-02-27T21:44:36.621ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
642024-02-27T21:44:36.621ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
652024-02-27T21:44:36.622ZINFOcrucible: Waiting for true more clients to be ready session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
662024-02-27T21:44:36.622ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
672024-02-27T21:44:36.622ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
682024-02-27T21:44:36.622ZINFOcrucible: Waiting for true more clients to be ready session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
692024-02-27T21:44:36.622ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
702024-02-27T21:44:36.622ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
712024-02-27T21:44:36.622ZINFOcrucible: [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
722024-02-27T21:44:36.622ZINFOcrucible: [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
732024-02-27T21:44:36.622ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
742024-02-27T21:44:36.622ZINFOcrucible: [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
752024-02-27T21:44:36.622ZINFOcrucible: [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
762024-02-27T21:44:36.622ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
772024-02-27T21:44:36.622ZINFOcrucible: [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
782024-02-27T21:44:36.622ZINFOcrucible: [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] = downstairs session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
792024-02-27T21:44:36.622ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
802024-02-27T21:44:36.622ZINFOcrucible: Max found gen is 1 = downstairs session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
812024-02-27T21:44:36.622ZINFOcrucible: Generation requested: 1 >= found:1 = downstairs session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
822024-02-27T21:44:36.622ZINFOcrucible: Next flush: 1 = downstairs session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
832024-02-27T21:44:36.622ZINFOcrucible: All extents match = downstairs session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
842024-02-27T21:44:36.622ZINFOcrucible: No downstairs reconciliation required session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
852024-02-27T21:44:36.622ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
86 {"msg":"Transition from WaitQuorum to Active","v":0,"name":"crucible","level":30Activate successful
87 ,Wait for a query_work_queue command to finish before sending IO
88 "time":"2024-02-27T21:44:36.622696876Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1163,"client":"1","":"downstairs","session_id":"ae3a9757-cd3e-4be1-ab60-acc44f80befc"}
892024-02-27T21:44:36.622ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
902024-02-27T21:44:36.622ZINFOcrucible: All required reconciliation work is completed session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
912024-02-27T21:44:36.622ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
922024-02-27T21:44:36.622ZINFOcrucible: ef2b48cb-019e-44e3-9b38-aa1793a88801 is now active with session: ae3a9757-cd3e-4be1-ab60-acc44f80befc session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
932024-02-27T21:44:36.622ZINFOcrucible: Set Active after no reconciliation session_id = ae3a9757-cd3e-4be1-ab60-acc44f80befc
942024-02-27T21:44:36.622ZINFOcrucible: The guest has finished waiting for activation with:1
95 Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
96 Fill test
97 Read and Verify all blocks (0..5000 range:false)
98 Wrote out file "alan"
99 CLIENT: Tests done. All submitted work has been ACK'd
100 ----------------------------------------------------------------
101 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:25
102 States: Active Active Active
103 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
104 78 Acked 1077 Read 100 Done Done Done false
105 79 Acked 1078 Read 100 Done Done Done false
106 80 Acked 1079 Read 100 Done Done Done false
107 81 Acked 1080 Read 100 Done Done Done false
108 82 Acked 1081 Read 100 Done Done Done false
109 83 Acked 1082 Read 100 Done Done Done false
110 84 Acked 1083 Read 100 Done Done Done false
111 85 Acked 1084 Read 100 Done Done Done false
112 86 Acked 1085 Read 100 Done Done Done false
113 87 Acked 1086 Read 100 Done Done Done false
114 88 Acked 1087 Read 100 Done Done Done false
115 89 Acked 1088 Read 100 Done Done Done false
116 90 Acked 1089 Read 100 Done Done Done false
117 91 Acked 1090 Read 100 Done Done Done false
118 92 Acked 1091 Read 100 Done Done Done false
119 93 Acked 1092 Read 100 Done Done Done false
120 94 Acked 1093 Read 100 Done Done Done false
121 95 Acked 1094 Read 100 Done Done Done false
122 96 Acked 1095 Read 100 Done Done Done false
123 97 Acked 1096 Read 100 Done Done Done false
124 98 Acked 1097 Read 100 Done Done Done false
125 99 Acked 1098 Read 100 Done Done Done false
126 100 Acked 1099 Read 100 Done Done Done false
127 101 Acked 1100 Read 100 Done Done Done false
128 102 Acked 1101 Read 100 Done Done Done false
129 STATES DS:0 DS:1 DS:2 TOTAL
130 New 0 0 0 0
131 Sent 0 0 0 0
132 Done 25 25 25 75
133 Skipped 0 0 0 0
134 Error 0 0 0 0
135 Last Flush: 1076 1076 1076
136 Downstairs last five completed: 1076 1075 1074 1073 1072
137 Upstairs last five completed: 102 101 100 99 98
138 CLIENT: Up:0 ds:25 act:3
139 ----------------------------------------------------------------
140 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
141 Downstairs last five completed: 1102 1101 1100 1099 1098
142 Upstairs last five completed: 103 102 101 100 99
143 CLIENT: Up:0 ds:0 act:3
144 CLIENT: All crucible jobs finished, exiting program
1452024-02-27T21:44:41.896ZWARNcrucible: Guest handle has been dropped
146 Replay loop starts now February 27, 2024 at 09:44:41 PM UTC
1472024-02-27T21:44:41.922ZINFOcrucible: Upstairs starts
1482024-02-27T21:44:41.922ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "f6c40444ff68361c835e88ee91c409e8deb48d7d", git_commit_timestamp: "2024-02-27T21:23:06.000000000Z", git_branch: "main", rustc_semver: "1.75.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "82e1608dfa6e0b5569232559e3d385fea5a93112", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
149 {"msg":"Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30Crucible runtime is spawned
150 ,"time":"2024-02-27T21:44:41.922573196Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167}
1512024-02-27T21:44:41.922ZINFOcrucible: Crucible b419cc70-54d3-4dc3-b588-7e9fe56cb23f has session id: e6a5a974-925a-494f-8845-facc441c2690 session_id = e6a5a974-925a-494f-8845-facc441c2690
1522024-02-27T21:44:41.922ZINFOcrucible: Upstairs opts: Upstairs UUID: b419cc70-54d3-4dc3-b588-7e9fe56cb23f, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = e6a5a974-925a-494f-8845-facc441c2690
1532024-02-27T21:44:41.922ZINFOcrucible: Crucible stats registered with UUID: b419cc70-54d3-4dc3-b588-7e9fe56cb23f session_id = e6a5a974-925a-494f-8845-facc441c2690
1542024-02-27T21:44:41.922ZINFOcrucible: The guest has requested activation with gen:2
1552024-02-27T21:44:41.922ZINFOcrucible: b419cc70-54d3-4dc3-b588-7e9fe56cb23f active request set session_id = e6a5a974-925a-494f-8845-facc441c2690
1562024-02-27T21:44:41.922ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
1572024-02-27T21:44:41.922ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
1582024-02-27T21:44:41.922ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
1592024-02-27T21:44:41.922ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
1602024-02-27T21:44:41.922ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
1612024-02-27T21:44:41.922ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
1622024-02-27T21:44:41.922ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
1632024-02-27T21:44:41.922ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
1642024-02-27T21:44:41.922ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
1652024-02-27T21:44:41.923ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
1662024-02-27T21:44:41.923ZINFOcrucible: ds_connection connected = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
1672024-02-27T21:44:41.923ZINFOcrucible: ds_connection connected = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
1682024-02-27T21:44:41.924ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
1692024-02-27T21:44:41.924ZINFOcrucible: [2] Transition from New to WaitActive = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
1702024-02-27T21:44:41.924ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
1712024-02-27T21:44:41.924ZINFOcrucible: [0] Transition from New to WaitActive = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
1722024-02-27T21:44:41.924ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
1732024-02-27T21:44:41.924ZINFOcrucible: [1] Transition from New to WaitActive = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
1742024-02-27T21:44:41.924ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
1752024-02-27T21:44:41.924ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
1762024-02-27T21:44:41.924ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
1772024-02-27T21:44:41.925ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
1782024-02-27T21:44:41.925ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
1792024-02-27T21:44:41.925ZINFOcrucible: Waiting for true more clients to be ready session_id = e6a5a974-925a-494f-8845-facc441c2690
1802024-02-27T21:44:41.925ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
1812024-02-27T21:44:41.925ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
1822024-02-27T21:44:41.925ZINFOcrucible: Waiting for true more clients to be ready session_id = e6a5a974-925a-494f-8845-facc441c2690
1832024-02-27T21:44:41.925ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
1842024-02-27T21:44:41.925ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
1852024-02-27T21:44:41.925ZINFOcrucible: [0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
1862024-02-27T21:44:41.925ZINFOcrucible: [0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
1872024-02-27T21:44:41.925ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
1882024-02-27T21:44:41.925ZINFOcrucible: [1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
189 {"Activate successful
190 msgWait for a query_work_queue command to finish before sending IO
191 ":"[1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:44:41.925698917Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"downstairs","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
1922024-02-27T21:44:41.925ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
1932024-02-27T21:44:41.925ZINFOcrucible: [2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
1942024-02-27T21:44:41.925ZINFOcrucible: [2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
1952024-02-27T21:44:41.925ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
1962024-02-27T21:44:41.925ZINFOcrucible: Max found gen is 2 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
1972024-02-27T21:44:41.925ZINFOcrucible: Generation requested: 2 >= found:2 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
1982024-02-27T21:44:41.925ZINFOcrucible: Next flush: 2 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
1992024-02-27T21:44:41.925ZINFOcrucible: All extents match = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
2002024-02-27T21:44:41.925ZINFOcrucible: No downstairs reconciliation required session_id = e6a5a974-925a-494f-8845-facc441c2690
2012024-02-27T21:44:41.925ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
2022024-02-27T21:44:41.925ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
2032024-02-27T21:44:41.925ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2042024-02-27T21:44:41.925ZINFOcrucible: All required reconciliation work is completed session_id = e6a5a974-925a-494f-8845-facc441c2690
2052024-02-27T21:44:41.925ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = e6a5a974-925a-494f-8845-facc441c2690
2062024-02-27T21:44:41.925ZINFOcrucible: b419cc70-54d3-4dc3-b588-7e9fe56cb23f is now active with session: e6a5a974-925a-494f-8845-facc441c2690 session_id = e6a5a974-925a-494f-8845-facc441c2690
2072024-02-27T21:44:41.925ZINFOcrucible: Set Active after no reconciliation session_id = e6a5a974-925a-494f-8845-facc441c2690
2082024-02-27T21:44:41.925ZINFOcrucible: The guest has finished waiting for activation with:2
209 Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
210 Loading write count information from file "alan"
211 Feb 27 21:44:41.976 INFO accepted connection, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
212 Main task has work to do, go find it
213 got dsc Stop(2)
214 stop 2
215 Feb 27 21:44:41.978 INFO request completed, latency_us: 530, response_code: 204, uri: /stop/cid/2, method: POST, req_id: c5e09152-6f76-4047-b9dd-6e50e6aa3c0d, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
216 [8830] Got stop action so:false kr:false
2172024-02-27T21:44:41.978ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2182024-02-27T21:44:41.979ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2192024-02-27T21:44:41.979ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = e6a5a974-925a-494f-8845-facc441c2690
2202024-02-27T21:44:41.979ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2212024-02-27T21:44:41.979ZINFOcrucible: [2] client re-new 0 jobs since flush 0 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
2222024-02-27T21:44:41.979ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
223 Feb 27 21:44:41.979 INFO request completed, latency_us: 223, response_code: 200, uri: /state/cid/2, method: GET, req_id: 7ecd8385-495d-4810-8328-fff7ad44c74e, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
224 [8830] Exited with: ExitStatus(unix_wait_status(9))
225 [8830][2] reports Exit
2262024-02-27T21:44:42.980ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2272024-02-27T21:44:42.981ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2282024-02-27T21:44:42.981ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2292024-02-27T21:44:42.981ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
2302024-02-27T21:44:42.981ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2312024-02-27T21:44:42.981ZINFOcrucible: [2] client re-new 0 jobs since flush 0 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
2322024-02-27T21:44:42.981ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2332024-02-27T21:44:43.982ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2342024-02-27T21:44:43.983ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2352024-02-27T21:44:43.983ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2362024-02-27T21:44:43.983ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
2372024-02-27T21:44:43.983ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2382024-02-27T21:44:43.983ZINFOcrucible: [2] client re-new 0 jobs since flush 0 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
2392024-02-27T21:44:43.983ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2402024-02-27T21:44:44.984ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2412024-02-27T21:44:44.984ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2422024-02-27T21:44:44.984ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2432024-02-27T21:44:44.984ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
2442024-02-27T21:44:44.985ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2452024-02-27T21:44:44.985ZINFOcrucible: [2] client re-new 0 jobs since flush 0 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
2462024-02-27T21:44:44.985ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
247 Feb 27 21:44:45.982 INFO request completed, latency_us: 242, response_code: 200, uri: /state/cid/2, method: GET, req_id: a1fde9ae-a244-4018-8a35-ec88f29fba3e, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
248 001/300 Read block 1664 len 8192
249 002/300 Read block 4838 len 36864
2502024-02-27T21:44:45.987ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2512024-02-27T21:44:45.987ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2522024-02-27T21:44:45.987ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2532024-02-27T21:44:45.987ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
2542024-02-27T21:44:45.987ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2552024-02-27T21:44:45.987ZINFOcrucible: [2] client re-new 2 jobs since flush 0 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
2562024-02-27T21:44:45.987ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
257 003/300 Write block 547 len 24576 data: 2 2 2 2 2 2
258 004/300 Read block 833 len 28672
259 005/300 Read block 3606 len 16384
260 006/300 Write block 371 len 16384 data: 2 2 2 2
261 007/300 Write block 2864 len 4096 data: 2
262 008/300 Read block 2809 len 16384
263 009/300 Write block 3457 len 16384 data: 2 2 2 2
264 010/300 Write block 1407 len 20480 data: 2 2 2 2 2
265 011/300 Flush
266 012/300 Write block 1910 len 20480 data: 2 2 2 2 2
267 013/300 Read block 4497 len 8192
268 014/300 Read block 431 len 24576
269 015/300 Read block 1700 len 40960
270 016/300 Read block 3232 len 36864
271 017/300 Read block 1974 len 12288
272 018/300 Write block 480 len 4096 data: 2
273 019/300 Flush
274 020/300 Write block 1947 len 8192 data: 2 2
275 021/300 Read block 1385 len 12288
276 022/300 Flush
277 023/300 Write block 766 len 8192 data: 2 2
278 024/300 Read block 4304 len 16384
279 025/300 Read block 2873 len 8192
280 026/300 Read block 2317 len 28672
281 027/300 Flush
282 028/300 Read block 4511 len 40960
283 029/300 Read block 2497 len 20480
284 030/300 Write block 3296 len 8192 data: 2 2
285 031/300 Read block 3702 len 36864
286 032/300 Read block 2355 len 4096
287 033/300 Write block 2186 len 4096 data: 2
288 034/300 Read block 1415 len 36864
289 035/300 Write block 4566 len 28672 data: 2 2 2 2 2 2 2
290 036/300 Write block 3555 len 8192 data: 2 2
291 037/300 Write block 3181 len 12288 data: 2 2 2
292 038/300 Read block 1748 len 4096
293 039/300 Write block 4636 len 32768 data: 2 2 2 2 2 2 2 2
294 040/300 Read block 4279 len 4096
295 041/300 Read block 2613 len 32768
296 042/300 Read block 3501 len 16384
297 043/300 Write block 2256 len 28672 data: 2 2 2 2 2 2 2
298 044/300 Read block 4909 len 32768
299 045/300 Read block 2063 len 40960
300 046/300 Read block 1912 len 12288
301 047/300 Write block 3753 len 4096 data: 2
302 048/300 Write block 3115 len 20480 data: 2 2 2 2 2
303 049/300 Read block 2281 len 8192
304 050/300 Read block 491 len 32768
305 051/300 Read block 4044 len 4096
306 052/300 Read block 107 len 32768
307 053/300 Read block 3551 len 28672
308 054/300 Read block 3468 len 4096
309 055/300 Flush
310 056/300 Write block 4021 len 20480 data: 2 2 2 2 2
311 057/300 Write block 3427 len 8192 data: 2 2
312 058/300 Write block 413 len 8192 data: 2 2
313 059/300 Write block 4361 len 40960 data: 2 2 2 2 2 2 2 2 2 2
314 060/300 Read block 4520 len 4096
315 061/300 Read block 1494 len 24576
316 062/300 Write block 3675 len 24576 data: 2 2 2 2 2 2
317 063/300 Read block 3795 len 12288
318 064/300 Write block 198 len 8192 data: 2 2
319 065/300 Read block 1552 len 24576
320 066/300 Read block 1898 len 32768
321 067/300 Write block 575 len 24576 data: 2 2 2 2 2 2
322 068/300 Read block 1017 len 24576
323 069/300 Write block 2302 len 16384 data: 2 2 2 2
324 070/300 Write block 2191 len 16384 data: 2 2 2 2
325 071/300 Write block 2779 len 16384 data: 2 2 2 2
326 072/300 Write block 4626 len 32768 data: 2 2 2 2 2 2 2 2
327 073/300 Write block 4051 len 36864 data: 2 2 2 2 2 2 2 2 2
328 074/300 Read block 806 len 8192
329 075/300 Write block 2446 len 12288 data: 2 2 2
330 076/300 Read block 1104 len 8192
331 077/300 Read block 720 len 32768
332 078/300 Read block 843 len 28672
333 079/300 Flush
334 080/300 Flush
335 081/300 Flush
336 082/300 Flush
337 083/300 Write block 3272 len 16384 data: 2 2 2 2
338 084/300 Read block 3525 len 12288
339 085/300 Read block 1058 len 36864
340 086/300 Read block 4350 len 12288
341 087/300 Read block 3443 len 12288
342 088/300 Write block 4745 len 20480 data: 2 2 2 2 2
343 089/300 Read block 2511 len 8192
344 090/300 Read block 4845 len 12288
345 091/300 Read block 3725 len 24576
346 092/300 Flush
347 093/300 Read block 0 len 4096
348 094/300 Read block 3884 len 16384
349 095/300 Read block 4448 len 12288
350 096/300 Write block 4564 len 40960 data: 2 2 3 3 3 3 3 3 3 2
351 097/300 Read block 530 len 36864
352 098/300 Read block 689 len 40960
353 099/300 Flush
354 100/300 Read block 4376 len 28672
355 101/300 Read block 834 len 4096
356 102/300 Read block 625 len 24576
357 103/300 Read block 4100 len 20480
358 104/300 Write block 2761 len 40960 data: 2 2 2 2 2 2 2 2 2 2
359 105/300 Read block 1483 len 8192
360 106/300 Read block 973 len 4096
361 107/300 Read block 155 len 20480
362 108/300 Write block 1920 len 8192 data: 2 2
363 109/300 Write block 2284 len 40960 data: 2 2 2 2 2 2 2 2 2 2
364 110/300 Read block 413 len 4096
365 111/300 Read block 3507 len 4096
366 112/300 Write block 3345 len 12288 data: 2 2 2
367 113/300 Read block 3925 len 32768
368 114/300 Read block 1051 len 36864
369 115/300 Write block 4763 len 12288 data: 2 2 2
370 116/300 Read block 2905 len 12288
371 117/300 Write block 2636 len 32768 data: 2 2 2 2 2 2 2 2
372 118/300 Read block 4317 len 28672
373 119/300 Read block 676 len 36864
374 120/300 Read block 3622 len 36864
375 121/300 Write block 1872 len 40960 data: 2 2 2 2 2 2 2 2 2 2
376 122/300 Read block 2310 len 4096
377 123/300 Read block 3947 len 16384
378 124/300 Write block 4367 len 36864 data: 3 3 3 3 2 2 2 2 2
379 125/300 Write block 1132 len 4096 data: 2
380 126/300 Read block 3742 len 40960
381 127/300 Write block 857 len 28672 data: 2 2 2 2 2 2 2
382 128/300 Write block 4024 len 36864 data: 3 3 2 2 2 2 2 2 2
383 129/300 Write block 357 len 28672 data: 2 2 2 2 2 2 2
384 130/300 Read block 132 len 24576
385 131/300 Write block 3491 len 28672 data: 2 2 2 2 2 2 2
386 132/300 Read block 4456 len 36864
387 133/300 Write block 3297 len 28672 data: 3 2 2 2 2 2 2
388 134/300 Write block 3482 len 4096 data: 2
389 135/300 Read block 3167 len 40960
390 136/300 Write block 1207 len 12288 data: 2 2 2
391 137/300 Read block 3512 len 24576
392 138/300 Write block 4133 len 24576 data: 2 2 2 2 2 2
393 139/300 Read block 1297 len 12288
394 140/300 Write block 1073 len 24576 data: 2 2 2 2 2 2
395 141/300 Read block 1762 len 20480
396 142/300 Flush
397 143/300 Read block 3623 len 20480
398 144/300 Flush
399 145/300 Read block 1932 len 20480
400 146/300 Write block 2241 len 24576 data: 2 2 2 2 2 2
401 147/300 Flush
402 148/300 Read block 2778 len 40960
403 149/300 Write block 876 len 8192 data: 2 2
404 150/300 Read block 2675 len 28672
405 151/300 Flush
406 152/300 Flush
407 153/300 Write block 4273 len 16384 data: 2 2 2 2
408 154/300 Write block 2037 len 12288 data: 2 2 2
409 155/300 Read block 2008 len 28672
410 156/300 Read block 4846 len 36864
411 157/300 Read block 1121 len 28672
412 158/300 Read block 1289 len 32768
413 159/300 Write block 994 len 4096 data: 2
414 160/300 Write block 3632 len 24576 data: 2 2 2 2 2 2
415 161/300 Read block 1217 len 36864
416 162/300 Read block 2201 len 12288
417 163/300 Write block 3370 len 28672 data: 2 2 2 2 2 2 2
418 164/300 Flush
419 165/300 Write block 3971 len 36864 data: 2 2 2 2 2 2 2 2 2
420 166/300 Write block 182 len 20480 data: 2 2 2 2 2
421 167/300 Read block 1500 len 8192
422 168/300 Write block 69 len 4096 data: 2
423 169/300 Read block 1752 len 16384
424 170/300 Read block 4386 len 4096
425 171/300 Read block 4333 len 16384
426 172/300 Write block 2053 len 20480 data: 2 2 2 2 2
427 173/300 Write block 91 len 32768 data: 2 2 2 2 2 2 2 2
428 174/300 Flush
429 175/300 Read block 2804 len 36864
430 176/300 Write block 1069 len 28672 data: 2 2 2 2 3 3 3
431 177/300 Flush
432 178/300 Read block 1687 len 20480
433 179/300 Read block 1760 len 24576
434 180/300 Flush
435 181/300 Write block 2211 len 28672 data: 2 2 2 2 2 2 2
436 182/300 Read block 4183 len 24576
437 183/300 Write block 3274 len 32768 data: 3 3 2 2 2 2 2 2
438 184/300 Write block 2139 len 28672 data: 2 2 2 2 2 2 2
439 185/300 Read block 4254 len 12288
440 186/300 Write block 4019 len 4096 data: 2
441 187/300 Read block 909 len 28672
442 188/300 Write block 517 len 28672 data: 2 2 2 2 2 2 2
443 189/300 Write block 4813 len 12288 data: 2 2 2
444 190/300 Read block 2209 len 4096
445 191/300 Read block 4202 len 36864
446 192/300 Read block 2766 len 28672
447 193/300 Write block 4486 len 28672 data: 2 2 2 2 2 2 2
448 194/300 Write block 192 len 12288 data: 2 2 2
449 195/300 Read block 1102 len 40960
450 196/300 Read block 2424 len 8192
451 197/300 Write block 3399 len 20480 data: 2 2 2 2 2
452 198/300 Write block 4716 len 24576 data: 2 2 2 2 2 2
453 199/300 Read block 2425 len 20480
454 200/300 Read block 3364 len 24576
455 201/300 Read block 97 len 32768
456 202/300 Write block 4705 len 8192 data: 2 2
457 203/300 Write block 308 len 36864 data: 2 2 2 2 2 2 2 2 2
458 204/300 Read block 1738 len 36864
459 205/300 Write block 497 len 16384 data: 2 2 2 2
460 206/300 Read block 3109 len 32768
461 207/300 Flush
462 208/300 Read block 10 len 16384
463 209/300 Read block 4336 len 40960
464 210/300 Read block 844 len 20480
465 211/300 Read block 4115 len 20480
466 212/300 Write block 3609 len 36864 data: 2 2 2 2 2 2 2 2 2
467 213/300 Write block 3016 len 36864 data: 2 2 2 2 2 2 2 2 2
468 214/300 Read block 3563 len 24576
469 215/300 Write block 2286 len 8192 data: 3 3
470 216/300 Write block 1011 len 4096 data: 2
471 217/300 Read block 1601 len 24576
472 218/300 Read block 4296 len 40960
473 219/300 Flush
474 220/300 Write block 4712 len 20480 data: 2 2 2 2 3
475 221/300 Read block 2511 len 24576
476 222/300 Write block 1712 len 32768 data: 2 2 2 2 2 2 2 2
477 223/300 Flush
478 224/300 Read block 765 len 20480
479 225/300 Read block 3768 len 28672
480 226/300 Write block 3175 len 28672 data: 2 2 2 2 2 2 3
481 227/300 Read block 520 len 20480
482 228/300 Read block 2607 len 24576
483 229/300 Flush
484 230/300 Read block 4587 len 40960
485 231/300 Write block 2348 len 40960 data: 2 2 2 2 2 2 2 2 2 2
486 232/300 Read block 19 len 12288
487 233/300 Read block 3524 len 28672
488 234/300 Write block 48 len 20480 data: 2 2 2 2 2
489 235/300 Write block 4749 len 28672 data: 3 2 2 2 2 2 2
490 236/300 Write block 4406 len 40960 data: 2 2 2 2 2 2 2 2 2 2
491 237/300 Flush
492 238/300 Read block 1475 len 4096
493 239/300 Read block 3254 len 4096
494 240/300 Read block 4282 len 32768
495 241/300 Read block 2264 len 8192
496 242/300 Write block 3811 len 40960 data: 2 2 2 2 2 2 2 2 2 2
497 243/300 Read block 2083 len 16384
498 244/300 Read block 2748 len 12288
499 245/300 Read block 4167 len 40960
500 246/300 Write block 4483 len 4096 data: 2
501 247/300 Write block 114 len 36864 data: 2 2 2 2 2 2 2 2 2
502 248/300 Write block 432 len 24576 data: 2 2 2 2 2 2
503 249/300 Read block 223 len 20480
504 250/300 Write block 1509 len 16384 data: 2 2 2 2
505 251/300 Read block 2923 len 20480
506 252/300 Write block 2167 len 12288 data: 2 2 2
507 253/300 Read block 4625 len 4096
508 254/300 Read block 2256 len 12288
509 255/300 Write block 4226 len 12288 data: 2 2 2
510 256/300 Read block 3398 len 28672
511 257/300 Read block 637 len 20480
512 258/300 Write block 4921 len 36864 data: 2 2 2 2 2 2 2 2 2
513 259/300 Write block 2273 len 32768 data: 2 2 2 2 2 2 2 2
514 260/300 Write block 1273 len 8192 data: 2 2
515 261/300 Read block 2433 len 40960
516 262/300 Write block 3235 len 40960 data: 2 2 2 2 2 2 2 2 2 2
517 263/300 Read block 2066 len 40960
518 264/300 Read block 2996 len 12288
519 265/300 Write block 2503 len 28672 data: 2 2 2 2 2 2 2
520 266/300 Read block 3820 len 32768
521 267/300 Flush
522 268/300 Flush
523 269/300 Flush
524 270/300 Read block 576 len 4096
525 271/300 Write block 908 len 40960 data: 2 2 2 2 2 2 2 2 2 2
526 272/300 Write block 2057 len 24576 data: 3 2 2 2 2 2
527 273/300 Write block 4929 len 8192 data: 3 2
528 274/300 Write block 3331 len 24576 data: 2 2 2 2 2 2
529 275/300 Write block 2908 len 16384 data: 2 2 2 2
530 276/300 Read block 897 len 28672
531 277/300 Read block 3190 len 20480
532 278/300 Read block 4335 len 12288
533 279/300 Flush
534 280/300 Write block 2711 len 32768 data: 2 2 2 2 2 2 2 2
535 281/300 Write block 2262 len 32768 data: 3 2 2 2 2 2 2 2
536 282/300 Read block 813 len 32768
537 283/300 Read block 1416 len 40960
538 284/300 Write block 1263 len 32768 data: 2 2 2 2 2 2 2 2
539 285/300 Read block 110 len 24576
540 286/300 Flush
541 287/300 Read block 4941 len 40960
542 288/300 Read block 1227 len 16384
543 289/300 Write block 172 len 4096 data: 2
544 290/300 Write block 2122 len 32768 data: 2 2 2 2 2 2 2 2
545 291/300 Write block 414 len 32768 data: 3 2 2 2 2 2 2 2
546 292/300 Write block 294 len 8192 data: 2 2
547 293/300 Read block 1067 len 8192
548 294/300 Read block 3741 len 24576
549 295/300 Read block 1285 len 8192
550 296/300 Read block 517 len 28672
551 297/300 Flush
552 298/300 Write block 2221 len 28672 data: 2 2 2 2 2 2 2
553 299/300 Write block 2970 len 40960 data: 2 2 2 2 2 2 2 2 2 2
554 300/300 Read block 1151 len 4096
555 Main task has work to do, go find it
556 got dsc Start(2)
557 start 2
558 Feb 27 21:44:46.325 INFO request completed, latency_us: 235, response_code: 204, uri: /start/cid/2, method: POST, req_id: 0d203032-225a-44d0-a06a-c70bf2b541d0, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
559 Replay: started 2, returned:Ok(())
560 ----------------------------------------------------------------
561 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
562 States: Active Active Offline
563 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
564 1 Acked 1000 Read 2 Done Done New true
565 2 Acked 1001 Read 9 Done Done New true
566 3 Acked 1002 Write 6 Done Done New false
567 4 Acked 1003 Read 7 Done Done New false
568 5 Acked 1004 Read 4 Done Done New false
569 6 Acked 1005 Write 4 Done Done New false
570 7 Acked 1006 Write 1 Done Done New false
571 8 Acked 1007 Read 4 Done Done New false
572 9 Acked 1008 Write 4 Done Done New false
573 10 Acked 1009 Write 5 Done Done New false
574 11 Acked 1010 Flush 0 Done Done New false
575 12 Acked 1011 Write 5 Done Done New false
576 13 Acked 1012 Read 2 Done Done New false
577 14 Acked 1013 Read 6 Done Done New false
578 15 Acked 1014 Read 10 Done Done New false
579 16 Acked 1015 Read 9 Done Done New false
580 17 Acked 1016 Read 3 Done Done New false
581 18 Acked 1017 Write 1 Done Done New false
582 19 Acked 1018 Flush 0 Done Done New false
583 20 Acked 1019 Write 2 Done Done New false
584 21 Acked 1020 Read 3 Done Done New false
585 22 Acked 1021 Flush 0 Done Done New false
586 23 Acked 1022 Write 2 Done Done New false
587 24 Acked 1023 Read 4 Done Done New false
588 25 Acked 1024 Read 2 Done Done New false
589 26 Acked 1025 Read 7 Done Done New false
590 27 Acked 1026 Flush 0 Done Done New false
591 28 Acked 1027 Read 10 Done Done New false
592 29 Acked 1028 Read 5 Done Done New false
593 30 Acked 1029 Write 2 Done Done New false
594 31 Acked 1030 Read 9 Done Done New false
595 32 Acked 1031 Read 1 Done Done New false
596 33 Acked 1032 Write 1 Done Done New false
597 34 Acked 1033 Read 9 Done Done New false
598 35 Acked 1034 Write 7 Done Done New false
599 36 Acked 1035 Write 2 Done Done New false
600 37 Acked 1036 Write 3 Done Done New false
601 38 Acked 1037 Read 1 Done Done New false
602 39 Acked 1038 Write 8 Done Done New false
603 40 Acked 1039 Read 1 Done Done New false
604 41 Acked 1040 Read 8 Done Done New false
605 42 Acked 1041 Read 4 Done Done New false
606 43 Acked 1042 Write 7 Done Done New false
607 44 Acked 1043 Read 8 Done Done New false
608 45 Acked 1044 Read 10 Done Done New false
609 46 Acked 1045 Read 3 Done Done New false
610 47 Acked 1046 Write 1 Done Done New false
611 48 Acked 1047 Write 5 Done Done New false
612 49 Acked 1048 Read 2 Done Done New false
613 50 Acked 1049 Read 8 Done Done New false
614 51 Acked 1050 Read 1 Done Done New false
615 52 Acked 1051 Read 8 Done Done New false
616 53 Acked 1052 Read 7 Done Done New false
617 54 Acked 1053 Read 1 Done Done New false
618 55 Acked 1054 Flush 0 Done Done New false
619 56 Acked 1055 Write 5 Done Done New false
620 57 Acked 1056 Write 2 Done Done New false
621 58 Acked 1057 Write 2 Done Done New false
622 59 Acked 1058 Write 10 Done Done New false
623 60 Acked 1059 Read 1 Done Done New false
624 61 Acked 1060 Read 6 Done Done New false
625 62 Acked 1061 Write 6 Done Done New false
626 63 Acked 1062 Read 3 Done Done New false
627 64 Acked 1063 Write 2 Done Done New false
628 65 Acked 1064 Read 6 Done Done New false
629 66 Acked 1065 Read 8 Done Done New false
630 67 Acked 1066 Write 6 Done Done New false
631 68 Acked 1067 Read 6 Done Done New false
632 69 Acked 1068 Write 4 Done Done New false
633 70 Acked 1069 Write 4 Done Done New false
634 71 Acked 1070 Write 4 Done Done New false
635 72 Acked 1071 Write 8 Done Done New false
636 73 Acked 1072 Write 9 Done Done New false
637 74 Acked 1073 Read 2 Done Done New false
638 75 Acked 1074 Write 3 Done Done New false
639 76 Acked 1075 Read 2 Done Done New false
640 77 Acked 1076 Read 8 Done Done New false
641 78 Acked 1077 Read 7 Done Done New false
642 79 Acked 1078 Flush 0 Done Done New false
643 80 Acked 1079 Flush 0 Done Done New false
644 81 Acked 1080 Flush 0 Done Done New false
645 82 Acked 1081 Flush 0 Done Done New false
646 83 Acked 1082 Write 4 Done Done New false
647 84 Acked 1083 Read 3 Done Done New false
648 85 Acked 1084 Read 9 Done Done New false
649 86 Acked 1085 Read 3 Done Done New false
650 87 Acked 1086 Read 3 Done Done New false
651 88 Acked 1087 Write 5 Done Done New false
652 89 Acked 1088 Read 2 Done Done New false
653 90 Acked 1089 Read 3 Done Done New false
654 91 Acked 1090 Read 6 Done Done New false
655 92 Acked 1091 Flush 0 Done Done New false
656 93 Acked 1092 Read 1 Done Done New false
657 94 Acked 1093 Read 4 Done Done New false
658 95 Acked 1094 Read 3 Done Done New false
659 96 Acked 1095 Write 10 Done Done New false
660 97 Acked 1096 Read 9 Done Done New false
661 98 Acked 1097 Read 10 Done Done New false
662 99 Acked 1098 Flush 0 Done Done New false
663 100 Acked 1099 Read 7 Done Done New false
664 101 Acked 1100 Read 1 Done Done New false
665 102 Acked 1101 Read 6 Done Done New false
666 103 Acked 1102 Read 5 Done Done New false
667 104 Acked 1103 Write 10 Done Done New false
668 105 Acked 1104 Read 2 Done Done New false
669 106 Acked 1105 Read 1 Done Done New false
670 107 Acked 1106 Read 5 Done Done New false
671 108 Acked 1107 Write 2 Done Done New false
672 109 Acked 1108 Write 10 Done Done New false
673 110 Acked 1109 Read 1 Done Done New false
674 111 Acked 1110 Read 1 Done Done New false
675 112 Acked 1111 Write 3 Done Done New false
676 113 Acked 1112 Read 8 Done Done New false
677 114 Acked 1113 Read 9 Done Done New false
678 115 Acked 1114 Write 3 Done Done New false
679 116 Acked 1115 Read 3 Done Done New false
680 117 Acked 1116 Write 8 Done Done New false
681 118 Acked 1117 Read 7 Done Done New false
682 119 Acked 1118 Read 9 Done Done New false
683 120 Acked 1119 Read 9 Done Done New false
684 121 Acked 1120 Write 10 Done Done New false
685 122 Acked 1121 Read 1 Done Done New false
686 123 Acked 1122 Read 4 Done Done New false
687 124 Acked 1123 Write 9 Done Done New false
688 125 Acked 1124 Write 1 Done Done New false
689 126 Acked 1125 Read 10 Done Done New false
690 127 Acked 1126 Write 7 Done Done New false
691 128 Acked 1127 Write 9 Done Done New false
692 129 Acked 1128 Write 7 Done Done New false
693 130 Acked 1129 Read 6 Done Done New false
694 131 Acked 1130 Write 7 Done Done New false
695 132 Acked 1131 Read 9 Done Done New false
696 133 Acked 1132 Write 7 Done Done New false
697 134 Acked 1133 Write 1 Done Done New false
698 135 Acked 1134 Read 10 Done Done New false
699 136 Acked 1135 Write 3 Done Done New false
700 137 Acked 1136 Read 6 Done Done New false
701 138 Acked 1137 Write 6 Done Done New false
702 139 Acked 1138 Read 3 Done Done New false
703 140 Acked 1139 Write 6 Done Done New false
704 141 Acked 1140 Read 5 Done Done New false
705 142 Acked 1141 Flush 0 Done Done New false
706 143 Acked 1142 Read 5 Done Done New false
707 144 Acked 1143 Flush 0 Done Done New false
708 145 Acked 1144 Read 5 Done Done New false
709 146 Acked 1145 Write 6 Done Done New false
710 147 Acked 1146 Flush 0 Done Done New false
711 148 Acked 1147 Read 10 Done Done New false
712 149 Acked 1148 Write 2 Done Done New false
713 150 Acked 1149 Read 7 Done Done New false
714 151 Acked 1150 Flush 0 Done Done New false
715 152 Acked 1151 Flush 0 Done Done New false
716 153 Acked 1152 Write 4 Done Done New false
717 154 Acked 1153 Write 3 Done Done New false
718 155 Acked 1154 Read 7 Done Done New false
719 156 Acked 1155 Read 9 Done Done New false
720 157 Acked 1156 Read 7 Done Done New false
721 158 Acked 1157 Read 8 Done Done New false
722 159 Acked 1158 Write 1 Done Done New false
723 160 Acked 1159 Write 6 Done Done New false
724 161 Acked 1160 Read 9 Done Done New false
725 162 Acked 1161 Read 3 Done Done New false
726 163 Acked 1162 Write 7 Done Done New false
727 164 Acked 1163 Flush 0 Done Done New false
728 165 Acked 1164 Write 9 Done Done New false
729 166 Acked 1165 Write 5 Done Done New false
730 167 Acked 1166 Read 2 Done Done New false
731 168 Acked 1167 Write 1 Done Done New false
732 169 Acked 1168 Read 4 Done Done New false
733 170 Acked 1169 Read 1 Done Done New false
734 171 Acked 1170 Read 4 Done Done New false
735 172 Acked 1171 Write 5 Done Done New false
736 173 Acked 1172 Write 8 Done Done New false
737 174 Acked 1173 Flush 0 Done Done New false
738 175 Acked 1174 Read 9 Done Done New false
739 176 Acked 1175 Write 7 Done Done New false
740 177 Acked 1176 Flush 0 Done Done New false
741 178 Acked 1177 Read 5 Done Done New false
742 179 Acked 1178 Read 6 Done Done New false
743 180 Acked 1179 Flush 0 Done Done New false
744 181 Acked 1180 Write 7 Done Done New false
745 182 Acked 1181 Read 6 Done Done New false
746 183 Acked 1182 Write 8 Done Done New false
747 184 Acked 1183 Write 7 Done Done New false
748 185 Acked 1184 Read 3 Done Done New false
749 186 Acked 1185 Write 1 Done Done New false
750 187 Acked 1186 Read 7 Done Done New false
751 188 Acked 1187 Write 7 Done Done New false
752 189 Acked 1188 Write 3 Done Done New false
753 190 Acked 1189 Read 1 Done Done New false
754 191 Acked 1190 Read 9 Done Done New false
755 192 Acked 1191 Read 7 Done Done New false
756 193 Acked 1192 Write 7 Done Done New false
757 194 Acked 1193 Write 3 Done Done New false
758 195 Acked 1194 Read 10 Done Done New false
759 196 Acked 1195 Read 2 Done Done New false
760 197 Acked 1196 Write 5 Done Done New false
761 198 Acked 1197 Write 6 Done Done New false
762 199 Acked 1198 Read 5 Done Done New false
763 200 Acked 1199 Read 6 Done Done New false
764 201 Acked 1200 Read 8 Done Done New false
765 202 Acked 1201 Write 2 Done Done New false
766 203 Acked 1202 Write 9 Done Done New false
767 204 Acked 1203 Read 9 Done Done New false
768 205 Acked 1204 Write 4 Done Done New false
769 206 Acked 1205 Read 8 Done Done New false
770 207 Acked 1206 Flush 0 Done Done New false
771 208 Acked 1207 Read 4 Done Done New false
772 209 Acked 1208 Read 10 Done Done New false
773 210 Acked 1209 Read 5 Done Done New false
774 211 Acked 1210 Read 5 Done Done New false
775 212 Acked 1211 Write 9 Done Done New false
776 213 Acked 1212 Write 9 Done Done New false
777 214 Acked 1213 Read 6 Done Done New false
778 215 Acked 1214 Write 2 Done Done New false
779 216 Acked 1215 Write 1 Done Done New false
780 217 Acked 1216 Read 6 Done Done New false
781 218 Acked 1217 Read 10 Done Done New false
782 219 Acked 1218 Flush 0 Done Done New false
783 220 Acked 1219 Write 5 Done Done New false
784 221 Acked 1220 Read 6 Done Done New false
785 222 Acked 1221 Write 8 Done Done New false
786 223 Acked 1222 Flush 0 Done Done New false
787 224 Acked 1223 Read 5 Done Done New false
788 225 Acked 1224 Read 7 Done Done New false
789 226 Acked 1225 Write 7 Done Done New false
790 227 Acked 1226 Read 5 Done Done New false
791 228 Acked 1227 Read 6 Done Done New false
792 229 Acked 1228 Flush 0 Done Done New false
793 230 Acked 1229 Read 10 Done Done New false
794 231 Acked 1230 Write 10 Done Done New false
795 232 Acked 1231 Read 3 Done Done New false
796 233 Acked 1232 Read 7 Done Done New false
797 234 Acked 1233 Write 5 Done Done New false
798 235 Acked 1234 Write 7 Done Done New false
799 236 Acked 1235 Write 10 Done Done New false
800 237 Acked 1236 Flush 0 Done Done New false
801 238 Acked 1237 Read 1 Done Done New false
802 239 Acked 1238 Read 1 Done Done New false
803 240 Acked 1239 Read 8 Done Done New false
804 241 Acked 1240 Read 2 Done Done New false
805 242 Acked 1241 Write 10 Done Done New false
806 243 Acked 1242 Read 4 Done Done New false
807 244 Acked 1243 Read 3 Done Done New false
808 245 Acked 1244 Read 10 Done Done New false
809 246 Acked 1245 Write 1 Done Done New false
810 247 Acked 1246 Write 9 Done Done New false
811 248 Acked 1247 Write 6 Done Done New false
812 249 Acked 1248 Read 5 Done Done New false
813 250 Acked 1249 Write 4 Done Done New false
814 251 Acked 1250 Read 5 Done Done New false
815 252 Acked 1251 Write 3 Done Done New false
816 253 Acked 1252 Read 1 Done Done New false
817 254 Acked 1253 Read 3 Done Done New false
818 255 Acked 1254 Write 3 Done Done New false
819 256 Acked 1255 Read 7 Done Done New false
820 257 Acked 1256 Read 5 Done Done New false
821 258 Acked 1257 Write 9 Done Done New false
822 259 Acked 1258 Write 8 Done Done New false
823 260 Acked 1259 Write 2 Done Done New false
824 261 Acked 1260 Read 10 Done Done New false
825 262 Acked 1261 Write 10 Done Done New false
826 263 Acked 1262 Read 10 Done Done New false
827 264 Acked 1263 Read 3 Done Done New false
828 265 Acked 1264 Write 7 Done Done New false
829 266 Acked 1265 Read 8 Done Done New false
830 267 Acked 1266 Flush 0 Done Done New false
831 268 Acked 1267 Flush 0 Done Done New false
832 269 Acked 1268 Flush 0 Done Done New false
833 270 Acked 1269 Read 1 Done Done New false
834 271 Acked 1270 Write 10 Done Done New false
835 272 Acked 1271 Write 6 Done Done New false
836 273 Acked 1272 Write 2 Done Done New false
837 274 Acked 1273 Write 6 Done Done New false
838 275 Acked 1274 Write 4 Done Done New false
839 276 Acked 1275 Read 7 Done Done New false
840 277 Acked 1276 Read 5 Done Done New false
841 278 Acked 1277 Read 3 Done Done New false
842 279 Acked 1278 Flush 0 Done Done New false
843 280 Acked 1279 Write 8 Done Done New false
844 281 Acked 1280 Write 8 Done Done New false
845 282 Acked 1281 Read 8 Done Done New false
846 283 Acked 1282 Read 10 Done Done New false
847 284 Acked 1283 Write 8 Done Done New false
848 285 Acked 1284 Read 6 Done Done New false
849 286 Acked 1285 Flush 0 Done Done New false
850 287 Acked 1286 Read 10 Done Done New false
851 288 Acked 1287 Read 4 Done Done New false
852 289 Acked 1288 Write 1 Done Done New false
853 290 Acked 1289 Write 8 Done Done New false
854 291 Acked 1290 Write 8 Done Done New false
855 292 Acked 1291 Write 2 Done Done New false
856 293 Acked 1292 Read 2 Done Done New false
857 294 Acked 1293 Read 6 Done Done New false
858 295 Acked 1294 Read 2 Done Done New false
859 296 Acked 1295 Read 7 Done Done New false
860 297 Acked 1296 Flush 0 Done Done New false
861 298 Acked 1297 Write 7 Done Done New false
862 299 Acked 1298 Write 10 Done Done New false
863 300 Acked 1299 Read 1 Done Done New false
864 STATES DS:0 DS:1 DS:2 TOTAL
865 New 0 0 300 300
866 Sent 0 0 0 0
867 Done 300 300 0 600
868 Skipped 0 0 0 0
869 Error 0 0 0 0
870 Last Flush: 1296 1296 0
871 Downstairs last five completed:
872 Upstairs last five completed: 300 299 298 297 296
873 CLIENT: Up:0 ds:300 act:2
8742024-02-27T21:44:46.989ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
8752024-02-27T21:44:46.989ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
8762024-02-27T21:44:46.989ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
8772024-02-27T21:44:46.989ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
8782024-02-27T21:44:46.989ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
8792024-02-27T21:44:46.989ZINFOcrucible: [2] client re-new 301 jobs since flush 0 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
8802024-02-27T21:44:46.989ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
881 [8830] got start action
882 [8830] I am going to restart
883 Starting downstairs at port 8830
884 Make output file at "/tmp/dsc/downstairs-8830.txt"
885 [8830][2] reports Starting
886 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1168)
887 [8830][2] reports Running
8882024-02-27T21:44:47.991ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
8892024-02-27T21:44:47.991ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
8902024-02-27T21:44:47.991ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
8912024-02-27T21:44:47.991ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
8922024-02-27T21:44:47.991ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
8932024-02-27T21:44:47.991ZINFOcrucible: [2] client re-new 301 jobs since flush 0 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
8942024-02-27T21:44:47.991ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
8952024-02-27T21:44:48.992ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
8962024-02-27T21:44:48.992ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
8972024-02-27T21:44:48.993ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
8982024-02-27T21:44:48.993ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
8992024-02-27T21:44:48.993ZINFOcrucible: send last flush ID to this DS: 0 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9002024-02-27T21:44:48.994ZINFOcrucible: Replied this last flush ID: 0 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9012024-02-27T21:44:48.994ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9022024-02-27T21:44:48.994ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9032024-02-27T21:44:48.994ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9042024-02-27T21:44:48.994ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
905 ----------------------------------------------------------------
906 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
907 Downstairs last five completed: 1300 1299 1298 1297 1296
908 Upstairs last five completed: 301 300 299 298 297
909 CLIENT: Up:0 ds:0 act:3
910 Replay: All jobs finished, all DS active.
911 Main task has work to do, go find it
912 got dsc Stop(2)
913 stop 2
914 Feb 27 21:44:50.328 INFO request completed, [8830] Got stop action so:false kr:false
915 latency_us: 346, response_code: 204, uri: /stop/cid/2, method: POST, req_id: c811b92a-8ff7-4978-a14f-974abecf3a35, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
916 Feb 27 21:44:50.329 INFO request completed, latency_us: 140, response_code: 200, uri: /state/cid/2, method: GET, req_id: a8861504-3706-4d9d-a7ea-04cb698d9e00, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
9172024-02-27T21:44:50.329ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9182024-02-27T21:44:50.329ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9192024-02-27T21:44:50.329ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = e6a5a974-925a-494f-8845-facc441c2690
9202024-02-27T21:44:50.329ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9212024-02-27T21:44:50.329ZINFOcrucible: [2] client re-new 0 jobs since flush 1300 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
9222024-02-27T21:44:50.329ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
923 [8830] Exited with: ExitStatus(unix_wait_status(9))
924 [8830][2] reports Exit
9252024-02-27T21:44:51.330ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9262024-02-27T21:44:51.330ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9272024-02-27T21:44:51.330ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9282024-02-27T21:44:51.330ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
9292024-02-27T21:44:51.330ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9302024-02-27T21:44:51.330ZINFOcrucible: [2] client re-new 0 jobs since flush 1300 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
9312024-02-27T21:44:51.331ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9322024-02-27T21:44:52.333ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9332024-02-27T21:44:52.333ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9342024-02-27T21:44:52.333ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9352024-02-27T21:44:52.333ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
9362024-02-27T21:44:52.333ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9372024-02-27T21:44:52.333ZINFOcrucible: [2] client re-new 0 jobs since flush 1300 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
9382024-02-27T21:44:52.333ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9392024-02-27T21:44:53.334ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9402024-02-27T21:44:53.334ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9412024-02-27T21:44:53.334ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9422024-02-27T21:44:53.334ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
9432024-02-27T21:44:53.334ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9442024-02-27T21:44:53.334ZINFOcrucible: [2] client re-new 0 jobs since flush 1300 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
9452024-02-27T21:44:53.334ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
946 Feb 27 21:44:54.332 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/2, method: GET, req_id: 71d6f2ce-ebfc-4791-a3a4-d7cc3ebc6281, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
947 001/300 Write block 2542 len 36864 data: 2 2 2 2 2 2 2 2 2
948 002/300 Read block 4570 len 40960
9492024-02-27T21:44:54.335ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9502024-02-27T21:44:54.335ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9512024-02-27T21:44:54.335ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9522024-02-27T21:44:54.335ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
9532024-02-27T21:44:54.335ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
9542024-02-27T21:44:54.335ZINFOcrucible: [2] client re-new 2 jobs since flush 1300 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
9552024-02-27T21:44:54.335ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
956 003/300 Read block 877 len 32768
957 004/300 Write block 280 len 20480 data: 2 2 2 2 2
958 005/300 Write block 1477 len 24576 data: 2 2 2 2 2 2
959 006/300 Flush
960 007/300 Read block 3653 len 8192
961 008/300 Read block 4654 len 36864
962 009/300 Write block 1757 len 36864 data: 2 2 2 2 2 2 2 2 2
963 010/300 Read block 4838 len 36864
964 011/300 Flush
965 012/300 Write block 2476 len 28672 data: 2 2 2 2 2 2 2
966 013/300 Read block 1580 len 40960
967 014/300 Read block 3653 len 32768
968 015/300 Read block 4640 len 28672
969 016/300 Write block 1792 len 16384 data: 2 2 2 2
970 017/300 Write block 2206 len 8192 data: 2 2
971 018/300 Read block 3750 len 8192
972 019/300 Read block 1377 len 12288
973 020/300 Write block 3203 len 12288 data: 2 2 2
974 021/300 Read block 4694 len 12288
975 022/300 Read block 375 len 24576
976 023/300 Write block 1215 len 28672 data: 2 2 2 2 2 2 2
977 024/300 Write block 2646 len 16384 data: 2 2 2 2
978 025/300 Write block 939 len 24576 data: 2 2 2 2 2 2
979 026/300 Write block 537 len 8192 data: 2 2
980 027/300 Write block 2791 len 16384 data: 2 2 2 2
981 028/300 Write block 2570 len 32768 data: 2 2 2 2 2 2 2 2
982 029/300 Write block 665 len 36864 data: 2 2 2 2 2 2 2 2 2
983 030/300 Read block 387 len 24576
984 031/300 Write block 4045 len 20480 data: 2 2 2 2 2
985 032/300 Write block 957 len 20480 data: 2 2 2 2 2
986 033/300 Write block 4588 len 4096 data: 2
987 034/300 Write block 316 len 28672 data: 3 2 2 2 2 2 2
988 035/300 Read block 4098 len 12288
989 036/300 Read block 2511 len 28672
990 037/300 Read block 2186 len 12288
991 038/300 Read block 3631 len 24576
992 039/300 Write block 202 len 28672 data: 2 2 2 2 2 2 2
993 040/300 Write block 117 len 32768 data: 3 3 3 3 3 3 2 2
994 041/300 Write block 3238 len 36864 data: 3 3 3 3 3 3 3 2 2
995 042/300 Read block 138 len 4096
996 043/300 Read block 3173 len 4096
997 044/300 Flush
998 045/300 Read block 4315 len 4096
999 046/300 Write block 2796 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1000 047/300 Read block 1370 len 12288
1001 048/300 Read block 999 len 12288
1002 049/300 Read block 172 len 28672
1003 050/300 Write block 2940 len 24576 data: 2 2 2 2 2 2
1004 051/300 Read block 2535 len 32768
1005 052/300 Write block 4191 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1006 053/300 Write block 2024 len 4096 data: 2
1007 054/300 Read block 2088 len 32768
1008 055/300 Write block 4823 len 28672 data: 2 2 2 2 2 2 2
1009 056/300 Read block 123 len 20480
1010 057/300 Read block 4023 len 40960
1011 058/300 Write block 1017 len 12288 data: 2 2 2
1012 059/300 Read block 1075 len 36864
1013 060/300 Read block 4862 len 8192
1014 061/300 Read block 4796 len 12288
1015 062/300 Write block 2613 len 4096 data: 2
1016 063/300 Read block 952 len 24576
1017 064/300 Read block 2413 len 40960
1018 065/300 Write block 4484 len 4096 data: 2
1019 066/300 Read block 1096 len 12288
1020 067/300 Read block 3666 len 12288
1021 068/300 Read block 2259 len 20480
1022 069/300 Read block 4771 len 16384
1023 070/300 Read block 1768 len 32768
1024 071/300 Read block 4722 len 24576
1025 072/300 Read block 1612 len 4096
1026 073/300 Write block 1439 len 28672 data: 2 2 2 2 2 2 2
1027 074/300 Read block 2357 len 28672
1028 075/300 Read block 4911 len 16384
1029 076/300 Read block 688 len 36864
1030 077/300 Read block 2192 len 8192
1031 078/300 Read block 777 len 40960
1032 079/300 Write block 195 len 20480 data: 2 2 2 3 3
1033 080/300 Write block 3023 len 20480 data: 3 3 2 2 2
1034 081/300 Write block 4385 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1035 082/300 Read block 2867 len 4096
1036 083/300 Read block 4976 len 24576
1037 084/300 Read block 36 len 32768
1038 085/300 Read block 3969 len 28672
1039 086/300 Read block 4950 len 12288
1040 087/300 Read block 441 len 4096
1041 088/300 Read block 2214 len 40960
1042 089/300 Flush
1043 090/300 Write block 1649 len 20480 data: 2 2 2 2 2
1044 091/300 Write block 3798 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1045 092/300 Read block 4883 len 4096
1046 093/300 Write block 3943 len 20480 data: 2 2 2 2 2
1047 094/300 Write block 226 len 8192 data: 2 2
1048 095/300 Write block 1864 len 12288 data: 2 2 2
1049 096/300 Flush
1050 097/300 Read block 1593 len 28672
1051 098/300 Write block 4359 len 40960 data: 2 2 3 3 3 3 3 3 4 4
1052 099/300 Write block 1443 len 28672 data: 3 3 3 2 2 2 2
1053 100/300 Read block 3387 len 8192
1054 101/300 Read block 863 len 28672
1055 102/300 Read block 2826 len 28672
1056 103/300 Read block 1302 len 12288
1057 104/300 Read block 1822 len 28672
1058 105/300 Read block 2407 len 16384
1059 106/300 Write block 574 len 16384 data: 2 3 3 3
1060 107/300 Read block 1515 len 24576
1061 108/300 Write block 1224 len 32768 data: 2 2 2 2 2 2 2 2
1062 109/300 Write block 2449 len 36864 data: 2 2 2 2 2 2 2 2 2
1063 110/300 Read block 193 len 8192
1064 111/300 Read block 2231 len 36864
1065 112/300 Write block 819 len 4096 data: 2
1066 113/300 Write block 941 len 24576 data: 3 3 3 3 2 2
1067 114/300 Flush
1068 115/300 Read block 715 len 12288
1069 116/300 Read block 250 len 40960
1070 117/300 Read block 707 len 16384
1071 118/300 Write block 3015 len 32768 data: 2 3 3 3 3 3 3 3
1072 119/300 Read block 4787 len 28672
1073 120/300 Write block 2231 len 36864 data: 2 2 2 2 2 2 2 2 2
1074 121/300 Read block 749 len 40960
1075 122/300 Flush
1076 123/300 Read block 1899 len 4096
1077 124/300 Flush
1078 125/300 Write block 4505 len 32768 data: 2 2 2 2 2 2 2 2
1079 126/300 Read block 3964 len 4096
1080 127/300 Read block 2773 len 40960
1081 128/300 Flush
1082 129/300 Read block 3368 len 20480
1083 130/300 Write block 1051 len 8192 data: 2 2
1084 131/300 Read block 3170 len 12288
1085 132/300 Read block 3197 len 4096
1086 133/300 Write block 4945 len 32768 data: 2 2 2 2 2 2 2 2
1087 134/300 Read block 3383 len 12288
1088 135/300 Read block 3821 len 16384
1089 136/300 Write block 1323 len 12288 data: 2 2 2
1090 137/300 Write block 1667 len 36864 data: 2 2 2 2 2 2 2 2 2
1091 138/300 Read block 304 len 4096
1092 139/300 Write block 4640 len 24576 data: 3 3 3 3 2 2
1093 140/300 Flush
1094 141/300 Write block 469 len 24576 data: 2 2 2 2 2 2
1095 142/300 Write block 1091 len 20480 data: 2 2 2 2 2
1096 143/300 Write block 1554 len 32768 data: 2 2 2 2 2 2 2 2
1097 144/300 Read block 2304 len 24576
1098 145/300 Flush
1099 146/300 Flush
1100 147/300 Read block 2811 len 12288
1101 148/300 Read block 396 len 36864
1102 149/300 Write block 4823 len 12288 data: 3 3 3
1103 150/300 Write block 4970 len 24576 data: 2 2 2 2 2 2
1104 151/300 Read block 1043 len 8192
1105 152/300 Write block 3000 len 4096 data: 2
1106 153/300 Write block 793 len 32768 data: 2 2 2 2 2 2 2 2
1107 154/300 Flush
1108 155/300 Read block 2613 len 40960
1109 156/300 Read block 610 len 8192
1110 157/300 Read block 2582 len 8192
1111 158/300 Write block 3008 len 8192 data: 2 2
1112 159/300 Read block 2311 len 16384
1113 160/300 Read block 3205 len 36864
1114 161/300 Write block 1591 len 24576 data: 2 2 2 2 2 2
1115 162/300 Read block 3796 len 40960
1116 163/300 Read block 3293 len 32768
1117 164/300 Write block 1964 len 4096 data: 2
1118 165/300 Read block 1956 len 28672
1119 166/300 Read block 1019 len 28672
1120 167/300 Read block 3696 len 16384
1121 168/300 Write block 3595 len 36864 data: 2 2 2 2 2 2 2 2 2
1122 169/300 Write block 3706 len 16384 data: 2 2 2 2
1123 170/300 Read block 3687 len 28672
1124 171/300 Read block 1600 len 32768
1125 172/300 Read block 592 len 28672
1126 173/300 Read block 1181 len 32768
1127 174/300 Read block 802 len 36864
1128 175/300 Write block 11 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1129 176/300 Read block 1629 len 32768
1130 177/300 Read block 4407 len 36864
1131 178/300 Write block 696 len 20480 data: 2 2 2 2 2
1132 179/300 Write block 3353 len 4096 data: 2
1133 180/300 Read block 1951 len 4096
1134 181/300 Read block 4519 len 4096
1135 182/300 Write block 18 len 32768 data: 3 3 3 2 2 2 2 2
1136 183/300 Write block 4981 len 20480 data: 2 2 2 2 2
1137 184/300 Write block 3055 len 24576 data: 2 2 2 2 2 2
1138 185/300 Read block 3861 len 40960
1139 186/300 Read block 2858 len 28672
1140 187/300 Read block 3311 len 16384
1141 188/300 Read block 932 len 24576
1142 189/300 Read block 452 len 4096
1143 190/300 Read block 1620 len 8192
1144 191/300 Write block 4655 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1145 192/300 Read block 2563 len 8192
1146 193/300 Read block 2221 len 36864
1147 194/300 Read block 2602 len 12288
1148 195/300 Read block 2472 len 20480
1149 196/300 Write block 3686 len 16384 data: 2 2 2 2
1150 197/300 Read block 626 len 36864
1151 198/300 Read block 3836 len 8192
1152 199/300 Read block 29 len 28672
1153 200/300 Write block 3249 len 28672 data: 2 2 2 2 2 2 2
1154 201/300 Write block 1643 len 28672 data: 2 2 2 2 2 2 3
1155 202/300 Read block 3571 len 28672
1156 203/300 Write block 2349 len 32768 data: 3 3 3 3 3 3 3 3
1157 204/300 Read block 1441 len 28672
1158 205/300 Write block 4991 len 8192 data: 2 2
1159 206/300 Read block 4411 len 40960
1160 207/300 Write block 1657 len 16384 data: 2 2 2 2
1161 208/300 Read block 4473 len 4096
1162 209/300 Write block 3682 len 24576 data: 2 2 2 2 3 3
1163 210/300 Read block 4649 len 24576
1164 211/300 Read block 2843 len 32768
1165 212/300 Write block 2022 len 40960 data: 2 2 3 2 2 2 2 2 2 2
1166 213/300 Read block 3925 len 40960
1167 214/300 Write block 3562 len 8192 data: 2 2
1168 215/300 Read block 1859 len 32768
1169 216/300 Write block 1335 len 28672 data: 2 2 2 2 2 2 2
1170 217/300 Read block 2964 len 12288
1171 218/300 Write block 4145 len 24576 data: 2 2 2 2 2 2
1172 219/300 Write block 4748 len 12288 data: 3 4 3
1173 220/300 Read block 1918 len 40960
1174 221/300 Flush
1175 222/300 Read block 1477 len 12288
1176 223/300 Read block 1534 len 24576
1177 224/300 Write block 1368 len 20480 data: 2 2 2 2 2
1178 225/300 Write block 1077 len 36864 data: 3 3 2 2 2 2 2 2 2
1179 226/300 Write block 2343 len 32768 data: 2 2 2 2 2 3 4 4
1180 227/300 Read block 896 len 12288
1181 228/300 Read block 4043 len 20480
1182 229/300 Read block 1880 len 4096
1183 230/300 Read block 2393 len 24576
1184 231/300 Read block 4444 len 4096
1185 232/300 Write block 1090 len 20480 data: 2 3 3 3 3
1186 233/300 Read block 3067 len 16384
1187 234/300 Write block 4259 len 36864 data: 2 2 2 2 2 2 2 2 2
1188 235/300 Write block 1287 len 28672 data: 2 2 2 2 2 2 2
1189 236/300 Flush
1190 237/300 Write block 4430 len 28672 data: 2 2 2 2 2 2 2
1191 238/300 Read block 4261 len 28672
1192 239/300 Write block 407 len 40960 data: 2 2 2 2 2 2 3 4 3 3
1193 240/300 Read block 3436 len 24576
1194 241/300 Write block 2515 len 32768 data: 2 2 2 2 2 2 2 2
1195 242/300 Read block 4008 len 40960
1196 243/300 Read block 705 len 16384
1197 244/300 Write block 4813 len 36864 data: 3 3 3 2 2 2 2 2 2
1198 245/300 Write block 4112 len 8192 data: 2 2
1199 246/300 Read block 3054 len 28672
1200 247/300 Write block 1825 len 16384 data: 2 2 2 2
1201 248/300 Read block 4957 len 36864
1202 249/300 Flush
1203 250/300 Read block 4499 len 40960
1204 251/300 Read block 4141 len 4096
1205 252/300 Write block 2487 len 36864 data: 2 2 2 2 2 2 2 2 2
1206 253/300 Write block 2027 len 16384 data: 3 3 3 3
1207 254/300 Write block 2507 len 8192 data: 3 3
1208 255/300 Read block 4697 len 4096
1209 256/300 Write block 433 len 36864 data: 3 3 3 3 3 2 2 2 2
1210 257/300 Read block 3164 len 28672
1211 258/300 Flush
1212 259/300 Write block 1664 len 4096 data: 2
1213 260/300 Write block 4883 len 28672 data: 2 2 2 2 2 2 2
1214 261/300 Read block 2886 len 40960
1215 262/300 Read block 2367 len 4096
1216 263/300 Read block 3251 len 20480
1217 264/300 Flush
1218 265/300 Read block 4909 len 28672
1219 266/300 Read block 3978 len 28672
1220 267/300 Write block 4092 len 36864 data: 2 2 2 2 2 2 2 2 2
1221 268/300 Read block 870 len 36864
1222 269/300 Read block 2902 len 28672
1223 270/300 Write block 2479 len 32768 data: 3 3 3 3 2 2 2 2
1224 271/300 Read block 3443 len 40960
1225 272/300 Read block 2999 len 24576
1226 273/300 Write block 1449 len 40960 data: 3 2 2 2 2 2 2 2 2 2
1227 274/300 Read block 3042 len 4096
1228 275/300 Flush
1229 276/300 Read block 4341 len 32768
1230 277/300 Read block 1445 len 12288
1231 278/300 Write block 1028 len 12288 data: 2 2 2
1232 279/300 Write block 3787 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1233 280/300 Read block 1277 len 20480
1234 281/300 Flush
1235 282/300 Write block 4951 len 12288 data: 3 3 2
1236 283/300 Read block 2193 len 40960
1237 284/300 Write block 3716 len 36864 data: 2 2 2 2 2 2 2 2 2
1238 285/300 Flush
1239 286/300 Write block 3704 len 24576 data: 2 2 3 3 3 3
1240 287/300 Read block 3426 len 40960
1241 288/300 Write block 316 len 4096 data: 4
1242 289/300 Read block 379 len 12288
1243 290/300 Read block 4914 len 40960
1244 291/300 Write block 2461 len 16384 data: 2 2 2 2
1245 292/300 Read block 3255 len 36864
1246 293/300 Write block 1378 len 24576 data: 2 2 2 2 2 2
1247 294/300 Read block 4264 len 20480
1248 295/300 Read block 1975 len 4096
1249 296/300 Read block 4761 len 36864
1250 297/300 Read block 3007 len 24576
1251 298/300 Read block 3012 len 4096
1252 299/300 Write block 4180 len 4096 data: 2
1253 300/300 Flush
1254 Main task has work to do, go find it
1255 got dsc Start(2)
1256 start 2
1257 Feb 27 21:44:54.687 INFO request completed, latency_us: 251, response_code: 204, uri: /start/cid/2, method: POST, req_id: 807b1a9d-76d9-4ced-8d4b-e7d04de29777, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
1258 Replay: started 2, returned:Ok(())
1259 ----------------------------------------------------------------
1260 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1261 States: Active Active Offline
1262 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
1263 302 Acked 1301 Write 9 Done Done New true
1264 303 Acked 1302 Read 10 Done Done New true
1265 304 Acked 1303 Read 8 Done Done New false
1266 305 Acked 1304 Write 5 Done Done New false
1267 306 Acked 1305 Write 6 Done Done New false
1268 307 Acked 1306 Flush 0 Done Done New false
1269 308 Acked 1307 Read 2 Done Done New false
1270 309 Acked 1308 Read 9 Done Done New false
1271 310 Acked 1309 Write 9 Done Done New false
1272 311 Acked 1310 Read 9 Done Done New false
1273 312 Acked 1311 Flush 0 Done Done New false
1274 313 Acked 1312 Write 7 Done Done New false
1275 314 Acked 1313 Read 10 Done Done New false
1276 315 Acked 1314 Read 8 Done Done New false
1277 316 Acked 1315 Read 7 Done Done New false
1278 317 Acked 1316 Write 4 Done Done New false
1279 318 Acked 1317 Write 2 Done Done New false
1280 319 Acked 1318 Read 2 Done Done New false
1281 320 Acked 1319 Read 3 Done Done New false
1282 321 Acked 1320 Write 3 Done Done New false
1283 322 Acked 1321 Read 3 Done Done New false
1284 323 Acked 1322 Read 6 Done Done New false
1285 324 Acked 1323 Write 7 Done Done New false
1286 325 Acked 1324 Write 4 Done Done New false
1287 326 Acked 1325 Write 6 Done Done New false
1288 327 Acked 1326 Write 2 Done Done New false
1289 328 Acked 1327 Write 4 Done Done New false
1290 329 Acked 1328 Write 8 Done Done New false
1291 330 Acked 1329 Write 9 Done Done New false
1292 331 Acked 1330 Read 6 Done Done New false
1293 332 Acked 1331 Write 5 Done Done New false
1294 333 Acked 1332 Write 5 Done Done New false
1295 334 Acked 1333 Write 1 Done Done New false
1296 335 Acked 1334 Write 7 Done Done New false
1297 336 Acked 1335 Read 3 Done Done New false
1298 337 Acked 1336 Read 7 Done Done New false
1299 338 Acked 1337 Read 3 Done Done New false
1300 339 Acked 1338 Read 6 Done Done New false
1301 340 Acked 1339 Write 7 Done Done New false
1302 341 Acked 1340 Write 8 Done Done New false
1303 342 Acked 1341 Write 9 Done Done New false
1304 343 Acked 1342 Read 1 Done Done New false
1305 344 Acked 1343 Read 1 Done Done New false
1306 345 Acked 1344 Flush 0 Done Done New false
1307 346 Acked 1345 Read 1 Done Done New false
1308 347 Acked 1346 Write 10 Done Done New false
1309 348 Acked 1347 Read 3 Done Done New false
1310 349 Acked 1348 Read 3 Done Done New false
1311 350 Acked 1349 Read 7 Done Done New false
1312 351 Acked 1350 Write 6 Done Done New false
1313 352 Acked 1351 Read 8 Done Done New false
1314 353 Acked 1352 Write 10 Done Done New false
1315 354 Acked 1353 Write 1 Done Done New false
1316 355 Acked 1354 Read 8 Done Done New false
1317 356 Acked 1355 Write 7 Done Done New false
1318 357 Acked 1356 Read 5 Done Done New false
1319 358 Acked 1357 Read 10 Done Done New false
1320 359 Acked 1358 Write 3 Done Done New false
1321 360 Acked 1359 Read 9 Done Done New false
1322 361 Acked 1360 Read 2 Done Done New false
1323 362 Acked 1361 Read 3 Done Done New false
1324 363 Acked 1362 Write 1 Done Done New false
1325 364 Acked 1363 Read 6 Done Done New false
1326 365 Acked 1364 Read 10 Done Done New false
1327 366 Acked 1365 Write 1 Done Done New false
1328 367 Acked 1366 Read 3 Done Done New false
1329 368 Acked 1367 Read 3 Done Done New false
1330 369 Acked 1368 Read 5 Done Done New false
1331 370 Acked 1369 Read 4 Done Done New false
1332 371 Acked 1370 Read 8 Done Done New false
1333 372 Acked 1371 Read 6 Done Done New false
1334 373 Acked 1372 Read 1 Done Done New false
1335 374 Acked 1373 Write 7 Done Done New false
1336 375 Acked 1374 Read 7 Done Done New false
1337 376 Acked 1375 Read 4 Done Done New false
1338 377 Acked 1376 Read 9 Done Done New false
1339 378 Acked 1377 Read 2 Done Done New false
1340 379 Acked 1378 Read 10 Done Done New false
1341 380 Acked 1379 Write 5 Done Done New false
1342 381 Acked 1380 Write 5 Done Done New false
1343 382 Acked 1381 Write 10 Done Done New false
1344 383 Acked 1382 Read 1 Done Done New false
1345 384 Acked 1383 Read 6 Done Done New false
1346 385 Acked 1384 Read 8 Done Done New false
1347 386 Acked 1385 Read 7 Done Done New false
1348 387 Acked 1386 Read 3 Done Done New false
1349 388 Acked 1387 Read 1 Done Done New false
1350 389 Acked 1388 Read 10 Done Done New false
1351 390 Acked 1389 Flush 0 Done Done New false
1352 391 Acked 1390 Write 5 Done Done New false
1353 392 Acked 1391 Write 10 Done Done New false
1354 393 Acked 1392 Read 1 Done Done New false
1355 394 Acked 1393 Write 5 Done Done New false
1356 395 Acked 1394 Write 2 Done Done New false
1357 396 Acked 1395 Write 3 Done Done New false
1358 397 Acked 1396 Flush 0 Done Done New false
1359 398 Acked 1397 Read 7 Done Done New false
1360 399 Acked 1398 Write 10 Done Done New false
1361 400 Acked 1399 Write 7 Done Done New false
1362 401 Acked 1400 Read 2 Done Done New false
1363 402 Acked 1401 Flush 0 Done Done New false
1364 403 Acked 1402 Read 7 Done Done New false
1365 404 Acked 1403 Read 7 Done Done New false
1366 405 Acked 1404 Read 3 Done Done New false
1367 406 Acked 1405 Read 7 Done Done New false
1368 407 Acked 1406 Read 4 Done Done New false
1369 408 Acked 1407 Write 4 Done Done New false
1370 409 Acked 1408 Read 6 Done Done New false
1371 410 Acked 1409 Write 8 Done Done New false
1372 411 Acked 1410 Write 9 Done Done New false
1373 412 Acked 1411 Read 2 Done Done New false
1374 413 Acked 1412 Read 9 Done Done New false
1375 414 Acked 1413 Write 1 Done Done New false
1376 415 Acked 1414 Write 6 Done Done New false
1377 416 Acked 1415 Flush 0 Done Done New false
1378 417 Acked 1416 Read 3 Done Done New false
1379 418 Acked 1417 Read 10 Done Done New false
1380 419 Acked 1418 Read 4 Done Done New false
1381 420 Acked 1419 Write 8 Done Done New false
1382 421 Acked 1420 Read 7 Done Done New false
1383 422 Acked 1421 Write 9 Done Done New false
1384 423 Acked 1422 Read 10 Done Done New false
1385 424 Acked 1423 Flush 0 Done Done New false
1386 425 Acked 1424 Read 1 Done Done New false
1387 426 Acked 1425 Flush 0 Done Done New false
1388 427 Acked 1426 Write 8 Done Done New false
1389 428 Acked 1427 Read 1 Done Done New false
1390 429 Acked 1428 Read 10 Done Done New false
1391 430 Acked 1429 Flush 0 Done Done New false
1392 431 Acked 1430 Read 5 Done Done New false
1393 432 Acked 1431 Write 2 Done Done New false
1394 433 Acked 1432 Read 3 Done Done New false
1395 434 Acked 1433 Read 1 Done Done New false
1396 435 Acked 1434 Write 8 Done Done New false
1397 436 Acked 1435 Read 3 Done Done New false
1398 437 Acked 1436 Read 4 Done Done New false
1399 438 Acked 1437 Write 3 Done Done New false
1400 439 Acked 1438 Write 9 Done Done New false
1401 440 Acked 1439 Read 1 Done Done New false
1402 441 Acked 1440 Write 6 Done Done New false
1403 442 Acked 1441 Flush 0 Done Done New false
1404 443 Acked 1442 Write 6 Done Done New false
1405 444 Acked 1443 Write 5 Done Done New false
1406 445 Acked 1444 Write 8 Done Done New false
1407 446 Acked 1445 Read 6 Done Done New false
1408 447 Acked 1446 Flush 0 Done Done New false
1409 448 Acked 1447 Flush 0 Done Done New false
1410 449 Acked 1448 Read 3 Done Done New false
1411 450 Acked 1449 Read 9 Done Done New false
1412 451 Acked 1450 Write 3 Done Done New false
1413 452 Acked 1451 Write 6 Done Done New false
1414 453 Acked 1452 Read 2 Done Done New false
1415 454 Acked 1453 Write 1 Done Done New false
1416 455 Acked 1454 Write 8 Done Done New false
1417 456 Acked 1455 Flush 0 Done Done New false
1418 457 Acked 1456 Read 10 Done Done New false
1419 458 Acked 1457 Read 2 Done Done New false
1420 459 Acked 1458 Read 2 Done Done New false
1421 460 Acked 1459 Write 2 Done Done New false
1422 461 Acked 1460 Read 4 Done Done New false
1423 462 Acked 1461 Read 9 Done Done New false
1424 463 Acked 1462 Write 6 Done Done New false
1425 464 Acked 1463 Read 10 Done Done New false
1426 465 Acked 1464 Read 8 Done Done New false
1427 466 Acked 1465 Write 1 Done Done New false
1428 467 Acked 1466 Read 7 Done Done New false
1429 468 Acked 1467 Read 7 Done Done New false
1430 469 Acked 1468 Read 4 Done Done New false
1431 470 Acked 1469 Write 9 Done Done New false
1432 471 Acked 1470 Write 4 Done Done New false
1433 472 Acked 1471 Read 7 Done Done New false
1434 473 Acked 1472 Read 8 Done Done New false
1435 474 Acked 1473 Read 7 Done Done New false
1436 475 Acked 1474 Read 8 Done Done New false
1437 476 Acked 1475 Read 9 Done Done New false
1438 477 Acked 1476 Write 10 Done Done New false
1439 478 Acked 1477 Read 8 Done Done New false
1440 479 Acked 1478 Read 9 Done Done New false
1441 480 Acked 1479 Write 5 Done Done New false
1442 481 Acked 1480 Write 1 Done Done New false
1443 482 Acked 1481 Read 1 Done Done New false
1444 483 Acked 1482 Read 1 Done Done New false
1445 484 Acked 1483 Write 8 Done Done New false
1446 485 Acked 1484 Write 5 Done Done New false
1447 486 Acked 1485 Write 6 Done Done New false
1448 487 Acked 1486 Read 10 Done Done New false
1449 488 Acked 1487 Read 7 Done Done New false
1450 489 Acked 1488 Read 4 Done Done New false
1451 490 Acked 1489 Read 6 Done Done New false
1452 491 Acked 1490 Read 1 Done Done New false
1453 492 Acked 1491 Read 2 Done Done New false
1454 493 Acked 1492 Write 10 Done Done New false
1455 494 Acked 1493 Read 2 Done Done New false
1456 495 Acked 1494 Read 9 Done Done New false
1457 496 Acked 1495 Read 3 Done Done New false
1458 497 Acked 1496 Read 5 Done Done New false
1459 498 Acked 1497 Write 4 Done Done New false
1460 499 Acked 1498 Read 9 Done Done New false
1461 500 Acked 1499 Read 2 Done Done New false
1462 501 Acked 1500 Read 7 Done Done New false
1463 502 Acked 1501 Write 7 Done Done New false
1464 503 Acked 1502 Write 7 Done Done New false
1465 504 Acked 1503 Read 7 Done Done New false
1466 505 Acked 1504 Write 8 Done Done New false
1467 506 Acked 1505 Read 7 Done Done New false
1468 507 Acked 1506 Write 2 Done Done New false
1469 508 Acked 1507 Read 10 Done Done New false
1470 509 Acked 1508 Write 4 Done Done New false
1471 510 Acked 1509 Read 1 Done Done New false
1472 511 Acked 1510 Write 6 Done Done New false
1473 512 Acked 1511 Read 6 Done Done New false
1474 513 Acked 1512 Read 8 Done Done New false
1475 514 Acked 1513 Write 10 Done Done New false
1476 515 Acked 1514 Read 10 Done Done New false
1477 516 Acked 1515 Write 2 Done Done New false
1478 517 Acked 1516 Read 8 Done Done New false
1479 518 Acked 1517 Write 7 Done Done New false
1480 519 Acked 1518 Read 3 Done Done New false
1481 520 Acked 1519 Write 6 Done Done New false
1482 521 Acked 1520 Write 3 Done Done New false
1483 522 Acked 1521 Read 10 Done Done New false
1484 523 Acked 1522 Flush 0 Done Done New false
1485 524 Acked 1523 Read 3 Done Done New false
1486 525 Acked 1524 Read 6 Done Done New false
1487 526 Acked 1525 Write 5 Done Done New false
1488 527 Acked 1526 Write 9 Done Done New false
1489 528 Acked 1527 Write 8 Done Done New false
1490 529 Acked 1528 Read 3 Done Done New false
1491 530 Acked 1529 Read 5 Done Done New false
1492 531 Acked 1530 Read 1 Done Done New false
1493 532 Acked 1531 Read 6 Done Done New false
1494 533 Acked 1532 Read 1 Done Done New false
1495 534 Acked 1533 Write 5 Done Done New false
1496 535 Acked 1534 Read 4 Done Done New false
1497 536 Acked 1535 Write 9 Done Done New false
1498 537 Acked 1536 Write 7 Done Done New false
1499 538 Acked 1537 Flush 0 Done Done New false
1500 539 Acked 1538 Write 7 Done Done New false
1501 540 Acked 1539 Read 7 Done Done New false
1502 541 Acked 1540 Write 10 Done Done New false
1503 542 Acked 1541 Read 6 Done Done New false
1504 543 Acked 1542 Write 8 Done Done New false
1505 544 Acked 1543 Read 10 Done Done New false
1506 545 Acked 1544 Read 4 Done Done New false
1507 546 Acked 1545 Write 9 Done Done New false
1508 547 Acked 1546 Write 2 Done Done New false
1509 548 Acked 1547 Read 7 Done Done New false
1510 549 Acked 1548 Write 4 Done Done New false
1511 550 Acked 1549 Read 9 Done Done New false
1512 551 Acked 1550 Flush 0 Done Done New false
1513 552 Acked 1551 Read 10 Done Done New false
1514 553 Acked 1552 Read 1 Done Done New false
1515 554 Acked 1553 Write 9 Done Done New false
1516 555 Acked 1554 Write 4 Done Done New false
1517 556 Acked 1555 Write 2 Done Done New false
1518 557 Acked 1556 Read 1 Done Done New false
1519 558 Acked 1557 Write 9 Done Done New false
1520 559 Acked 1558 Read 7 Done Done New false
1521 560 Acked 1559 Flush 0 Done Done New false
1522 561 Acked 1560 Write 1 Done Done New false
1523 562 Acked 1561 Write 7 Done Done New false
1524 563 Acked 1562 Read 10 Done Done New false
1525 564 Acked 1563 Read 1 Done Done New false
1526 565 Acked 1564 Read 5 Done Done New false
1527 566 Acked 1565 Flush 0 Done Done New false
1528 567 Acked 1566 Read 7 Done Done New false
1529 568 Acked 1567 Read 7 Done Done New false
1530 569 Acked 1568 Write 9 Done Done New false
1531 570 Acked 1569 Read 9 Done Done New false
1532 571 Acked 1570 Read 7 Done Done New false
1533 572 Acked 1571 Write 8 Done Done New false
1534 573 Acked 1572 Read 10 Done Done New false
1535 574 Acked 1573 Read 6 Done Done New false
1536 575 Acked 1574 Write 10 Done Done New false
1537 576 Acked 1575 Read 1 Done Done New false
1538 577 Acked 1576 Flush 0 Done Done New false
1539 578 Acked 1577 Read 8 Done Done New false
1540 579 Acked 1578 Read 3 Done Done New false
1541 580 Acked 1579 Write 3 Done Done New false
1542 581 Acked 1580 Write 10 Done Done New false
1543 582 Acked 1581 Read 5 Done Done New false
1544 583 Acked 1582 Flush 0 Done Done New false
1545 584 Acked 1583 Write 3 Done Done New false
1546 585 Acked 1584 Read 10 Done Done New false
1547 586 Acked 1585 Write 9 Done Done New false
1548 587 Acked 1586 Flush 0 Done Done New false
1549 588 Acked 1587 Write 6 Done Done New false
1550 589 Acked 1588 Read 10 Done Done New false
1551 590 Acked 1589 Write 1 Done Done New false
1552 591 Acked 1590 Read 3 Done Done New false
1553 592 Acked 1591 Read 10 Done Done New false
1554 593 Acked 1592 Write 4 Done Done New false
1555 594 Acked 1593 Read 9 Done Done New false
1556 595 Acked 1594 Write 6 Done Done New false
1557 596 Acked 1595 Read 5 Done Done New false
1558 597 Acked 1596 Read 1 Done Done New false
1559 598 Acked 1597 Read 9 Done Done New false
1560 599 Acked 1598 Read 6 Done Done New false
1561 600 Acked 1599 Read 1 Done Done New false
1562 601 Acked 1600 Write 1 Done Done New false
1563 602 Acked 1601 Flush 0 Done Done New false
1564 STATES DS:0 DS:1 DS:2 TOTAL
1565 New 0 0 301 301
1566 Sent 0 0 0 0
1567 Done 301 301 0 602
1568 Skipped 0 0 0 0
1569 Error 0 0 0 0
1570 Last Flush: 1601 1601 1300
1571 Downstairs last five completed: 1300 1299 1298 1297 1296
1572 Upstairs last five completed: 602 601 600 599 598
1573 CLIENT: Up:0 ds:301 act:2
15742024-02-27T21:44:55.336ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
15752024-02-27T21:44:55.336ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
15762024-02-27T21:44:55.336ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
15772024-02-27T21:44:55.336ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
15782024-02-27T21:44:55.336ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
15792024-02-27T21:44:55.336ZINFOcrucible: [2] client re-new 301 jobs since flush 1300 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
15802024-02-27T21:44:55.336ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
15812024-02-27T21:44:56.337ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
15822024-02-27T21:44:56.337ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
15832024-02-27T21:44:56.337ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
15842024-02-27T21:44:56.337ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
15852024-02-27T21:44:56.337ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
15862024-02-27T21:44:56.337ZINFOcrucible: [2] client re-new 301 jobs since flush 1300 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
15872024-02-27T21:44:56.338ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
15882024-02-27T21:44:57.338ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
15892024-02-27T21:44:57.338ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
15902024-02-27T21:44:57.338ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
15912024-02-27T21:44:57.339ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
15922024-02-27T21:44:57.339ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
15932024-02-27T21:44:57.339ZINFOcrucible: [2] client re-new 301 jobs since flush 1300 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
15942024-02-27T21:44:57.339ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
15952024-02-27T21:44:58.341ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
15962024-02-27T21:44:58.341ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
15972024-02-27T21:44:58.341ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
15982024-02-27T21:44:58.341ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
15992024-02-27T21:44:58.341ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
16002024-02-27T21:44:58.341ZINFOcrucible: [2] client re-new 301 jobs since flush 1300 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
16012024-02-27T21:44:58.341ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
1602 ----------------------------------------------------------------
1603 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1604 States: Active Active Offline
1605 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
1606 302 Acked 1301 Write 9 Done Done New true
1607 303 Acked 1302 Read 10 Done Done New true
1608 304 Acked 1303 Read 8 Done Done New true
1609 305 Acked 1304 Write 5 Done Done New true
1610 306 Acked 1305 Write 6 Done Done New true
1611 307 Acked 1306 Flush 0 Done Done New true
1612 308 Acked 1307 Read 2 Done Done New true
1613 309 Acked 1308 Read 9 Done Done New true
1614 310 Acked 1309 Write 9 Done Done New true
1615 311 Acked 1310 Read 9 Done Done New true
1616 312 Acked 1311 Flush 0 Done Done New true
1617 313 Acked 1312 Write 7 Done Done New true
1618 314 Acked 1313 Read 10 Done Done New true
1619 315 Acked 1314 Read 8 Done Done New true
1620 316 Acked 1315 Read 7 Done Done New true
1621 317 Acked 1316 Write 4 Done Done New true
1622 318 Acked 1317 Write 2 Done Done New true
1623 319 Acked 1318 Read 2 Done Done New true
1624 320 Acked 1319 Read 3 Done Done New true
1625 321 Acked 1320 Write 3 Done Done New true
1626 322 Acked 1321 Read 3 Done Done New true
1627 323 Acked 1322 Read 6 Done Done New true
1628 324 Acked 1323 Write 7 Done Done New true
1629 325 Acked 1324 Write 4 Done Done New true
1630 326 Acked 1325 Write 6 Done Done New true
1631 327 Acked 1326 Write 2 Done Done New true
1632 328 Acked 1327 Write 4 Done Done New true
1633 329 Acked 1328 Write 8 Done Done New true
1634 330 Acked 1329 Write 9 Done Done New true
1635 331 Acked 1330 Read 6 Done Done New true
1636 332 Acked 1331 Write 5 Done Done New true
1637 333 Acked 1332 Write 5 Done Done New true
1638 334 Acked 1333 Write 1 Done Done New true
1639 335 Acked 1334 Write 7 Done Done New true
1640 336 Acked 1335 Read 3 Done Done New true
1641 337 Acked 1336 Read 7 Done Done New true
1642 338 Acked 1337 Read 3 Done Done New true
1643 339 Acked 1338 Read 6 Done Done New true
1644 340 Acked 1339 Write 7 Done Done New true
1645 341 Acked 1340 Write 8 Done Done New true
1646 342 Acked 1341 Write 9 Done Done New true
1647 343 Acked 1342 Read 1 Done Done New true
1648 344 Acked 1343 Read 1 Done Done New true
1649 345 Acked 1344 Flush 0 Done Done New true
1650 346 Acked 1345 Read 1 Done Done New true
1651 347 Acked 1346 Write 10 Done Done New true
1652 348 Acked 1347 Read 3 Done Done New true
1653 349 Acked 1348 Read 3 Done Done New true
1654 350 Acked 1349 Read 7 Done Done New true
1655 351 Acked 1350 Write 6 Done Done New true
1656 352 Acked 1351 Read 8 Done Done New true
1657 353 Acked 1352 Write 10 Done Done New true
1658 354 Acked 1353 Write 1 Done Done New true
1659 355 Acked 1354 Read 8 Done Done New true
1660 356 Acked 1355 Write 7 Done Done New true
1661 357 Acked 1356 Read 5 Done Done New true
1662 358 Acked 1357 Read 10 Done Done New true
1663 359 Acked 1358 Write 3 Done Done New true
1664 360 Acked 1359 Read 9 Done Done New true
1665 361 Acked 1360 Read 2 Done Done New true
1666 362 Acked 1361 Read 3 Done Done New true
1667 363 Acked 1362 Write 1 Done Done New true
1668 364 Acked 1363 Read 6 Done Done New true
1669 365 Acked 1364 Read 10 Done Done New true
1670 366 Acked 1365 Write 1 Done Done New true
1671 367 Acked 1366 Read 3 Done Done New true
1672 368 Acked 1367 Read 3 Done Done New true
1673 369 Acked 1368 Read 5 Done Done New true
1674 370 Acked 1369 Read 4 Done Done New true
1675 371 Acked 1370 Read 8 Done Done New true
1676 372 Acked 1371 Read 6 Done Done New true
1677 373 Acked 1372 Read 1 Done Done New true
1678 374 Acked 1373 Write 7 Done Done New true
1679 375 Acked 1374 Read 7 Done Done New true
1680 376 Acked 1375 Read 4 Done Done New true
1681 377 Acked 1376 Read 9 Done Done New true
1682 378 Acked 1377 Read 2 Done Done New true
1683 379 Acked 1378 Read 10 Done Done New true
1684 380 Acked 1379 Write 5 Done Done New true
1685 381 Acked 1380 Write 5 Done Done New true
1686 382 Acked 1381 Write 10 Done Done New true
1687 383 Acked 1382 Read 1 Done Done New true
1688 384 Acked 1383 Read 6 Done Done New true
1689 385 Acked 1384 Read 8 Done Done New true
1690 386 Acked 1385 Read 7 Done Done New true
1691 387 Acked 1386 Read 3 Done Done New true
1692 388 Acked 1387 Read 1 Done Done New true
1693 389 Acked 1388 Read 10 Done Done New true
1694 390 Acked 1389 Flush 0 Done Done New true
1695 391 Acked 1390 Write 5 Done Done New true
1696 392 Acked 1391 Write 10 Done Done New true
1697 393 Acked 1392 Read 1 Done Done New true
1698 394 Acked 1393 Write 5 Done Done New true
1699 395 Acked 1394 Write 2 Done Done New true
1700 396 Acked 1395 Write 3 Done Done New true
1701 397 Acked 1396 Flush 0 Done Done New true
1702 398 Acked 1397 Read 7 Done Done New true
1703 399 Acked 1398 Write 10 Done Done New true
1704 400 Acked 1399 Write 7 Done Done New true
1705 401 Acked 1400 Read 2 Done Done New true
1706 402 Acked 1401 Flush 0 Done Done New true
1707 403 Acked 1402 Read 7 Done Done New true
1708 404 Acked 1403 Read 7 Done Done New true
1709 405 Acked 1404 Read 3 Done Done New true
1710 406 Acked 1405 Read 7 Done Done New true
1711 407 Acked 1406 Read 4 Done Done New true
1712 408 Acked 1407 Write 4 Done Done New true
1713 409 Acked 1408 Read 6 Done Done New true
1714 410 Acked 1409 Write 8 Done Done New true
1715 411 Acked 1410 Write 9 Done Done New true
1716 412 Acked 1411 Read 2 Done Done New true
1717 413 Acked 1412 Read 9 Done Done New true
1718 414 Acked 1413 Write 1 Done Done New true
1719 415 Acked 1414 Write 6 Done Done New true
1720 416 Acked 1415 Flush 0 Done Done New true
1721 417 Acked 1416 Read 3 Done Done New true
1722 418 Acked 1417 Read 10 Done Done New true
1723 419 Acked 1418 Read 4 Done Done New true
1724 420 Acked 1419 Write 8 Done Done New true
1725 421 Acked 1420 Read 7 Done Done New true
1726 422 Acked 1421 Write 9 Done Done New true
1727 423 Acked 1422 Read 10 Done Done New true
1728 424 Acked 1423 Flush 0 Done Done New true
1729 425 Acked 1424 Read 1 Done Done New true
1730 426 Acked 1425 Flush 0 Done Done New true
1731 427 Acked 1426 Write 8 Done Done New true
1732 428 Acked 1427 Read 1 Done Done New true
1733 429 Acked 1428 Read 10 Done Done New true
1734 430 Acked 1429 Flush 0 Done Done New true
1735 431 Acked 1430 Read 5 Done Done New true
1736 432 Acked 1431 Write 2 Done Done New true
1737 433 Acked 1432 Read 3 Done Done New true
1738 434 Acked 1433 Read 1 Done Done New true
1739 435 Acked 1434 Write 8 Done Done New true
1740 436 Acked 1435 Read 3 Done Done New true
1741 437 Acked 1436 Read 4 Done Done New true
1742 438 Acked 1437 Write 3 Done Done New true
1743 439 Acked 1438 Write 9 Done Done New true
1744 440 Acked 1439 Read 1 Done Done New true
1745 441 Acked 1440 Write 6 Done Done New true
1746 442 Acked 1441 Flush 0 Done Done New true
1747 443 Acked 1442 Write 6 Done Done New true
1748 444 Acked 1443 Write 5 Done Done New true
1749 445 Acked 1444 Write 8 Done Done New true
1750 446 Acked 1445 Read 6 Done Done New true
1751 447 Acked 1446 Flush 0 Done Done New true
1752 448 Acked 1447 Flush 0 Done Done New true
1753 449 Acked 1448 Read 3 Done Done New true
1754 450 Acked 1449 Read 9 Done Done New true
1755 451 Acked 1450 Write 3 Done Done New true
1756 452 Acked 1451 Write 6 Done Done New true
1757 453 Acked 1452 Read 2 Done Done New true
1758 454 Acked 1453 Write 1 Done Done New true
1759 455 Acked 1454 Write 8 Done Done New true
1760 456 Acked 1455 Flush 0 Done Done New true
1761 457 Acked 1456 Read 10 Done Done New true
1762 458 Acked 1457 Read 2 Done Done New true
1763 459 Acked 1458 Read 2 Done Done New true
1764 460 Acked 1459 Write 2 Done Done New true
1765 461 Acked 1460 Read 4 Done Done New true
1766 462 Acked 1461 Read 9 Done Done New true
1767 463 Acked 1462 Write 6 Done Done New true
1768 464 Acked 1463 Read 10 Done Done New true
1769 465 Acked 1464 Read 8 Done Done New true
1770 466 Acked 1465 Write 1 Done Done New true
1771 467 Acked 1466 Read 7 Done Done New true
1772 468 Acked 1467 Read 7 Done Done New true
1773 469 Acked 1468 Read 4 Done Done New true
1774 470 Acked 1469 Write 9 Done Done New true
1775 471 Acked 1470 Write 4 Done Done New true
1776 472 Acked 1471 Read 7 Done Done New true
1777 473 Acked 1472 Read 8 Done Done New true
1778 474 Acked 1473 Read 7 Done Done New true
1779 475 Acked 1474 Read 8 Done Done New true
1780 476 Acked 1475 Read 9 Done Done New true
1781 477 Acked 1476 Write 10 Done Done New true
1782 478 Acked 1477 Read 8 Done Done New true
1783 479 Acked 1478 Read 9 Done Done New true
1784 480 Acked 1479 Write 5 Done Done New true
1785 481 Acked 1480 Write 1 Done Done New true
1786 482 Acked 1481 Read 1 Done Done New true
1787 483 Acked 1482 Read 1 Done Done New true
1788 484 Acked 1483 Write 8 Done Done New true
1789 485 Acked 1484 Write 5 Done Done New true
1790 486 Acked 1485 Write 6 Done Done New true
1791 487 Acked 1486 Read 10 Done Done New true
1792 488 Acked 1487 Read 7 Done Done New true
1793 489 Acked 1488 Read 4 Done Done New true
1794 490 Acked 1489 Read 6 Done Done New true
1795 491 Acked 1490 Read 1 Done Done New true
1796 492 Acked 1491 Read 2 Done Done New true
1797 493 Acked 1492 Write 10 Done Done New true
1798 494 Acked 1493 Read 2 Done Done New true
1799 495 Acked 1494 Read 9 Done Done New true
1800 496 Acked 1495 Read 3 Done Done New true
1801 497 Acked 1496 Read 5 Done Done New true
1802 498 Acked 1497 Write 4 Done Done New true
1803 499 Acked 1498 Read 9 Done Done New true
1804 500 Acked 1499 Read 2 Done Done New true
1805 501 Acked 1500 Read 7 Done Done New true
1806 502 Acked 1501 Write 7 Done Done New true
1807 503 Acked 1502 Write 7 Done Done New true
1808 504 Acked 1503 Read 7 Done Done New true
1809 505 Acked 1504 Write 8 Done Done New true
1810 506 Acked 1505 Read 7 Done Done New true
1811 507 Acked 1506 Write 2 Done Done New true
1812 508 Acked 1507 Read 10 Done Done New true
1813 509 Acked 1508 Write 4 Done Done New true
1814 510 Acked 1509 Read 1 Done Done New true
1815 511 Acked 1510 Write 6 Done Done New true
1816 512 Acked 1511 Read 6 Done Done New true
1817 513 Acked 1512 Read 8 Done Done New true
1818 514 Acked 1513 Write 10 Done Done New true
1819 515 Acked 1514 Read 10 Done Done New true
1820 516 Acked 1515 Write 2 Done Done New true
1821 517 Acked 1516 Read 8 Done Done New true
1822 518 Acked 1517 Write 7 Done Done New true
1823 519 Acked 1518 Read 3 Done Done New true
1824 520 Acked 1519 Write 6 Done Done New true
1825 521 Acked 1520 Write 3 Done Done New true
1826 522 Acked 1521 Read 10 Done Done New true
1827 523 Acked 1522 Flush 0 Done Done New true
1828 524 Acked 1523 Read 3 Done Done New true
1829 525 Acked 1524 Read 6 Done Done New true
1830 526 Acked 1525 Write 5 Done Done New true
1831 527 Acked 1526 Write 9 Done Done New true
1832 528 Acked 1527 Write 8 Done Done New true
1833 529 Acked 1528 Read 3 Done Done New true
1834 530 Acked 1529 Read 5 Done Done New true
1835 531 Acked 1530 Read 1 Done Done New true
1836 532 Acked 1531 Read 6 Done Done New true
1837 533 Acked 1532 Read 1 Done Done New true
1838 534 Acked 1533 Write 5 Done Done New true
1839 535 Acked 1534 Read 4 Done Done New true
1840 536 Acked 1535 Write 9 Done Done New true
1841 537 Acked 1536 Write 7 Done Done New true
1842 538 Acked 1537 Flush 0 Done Done New true
1843 539 Acked 1538 Write 7 Done Done New true
1844 540 Acked 1539 Read 7 Done Done New true
1845 541 Acked 1540 Write 10 Done Done New true
1846 542 Acked 1541 Read 6 Done Done New true
1847 543 Acked 1542 Write 8 Done Done New true
1848 544 Acked 1543 Read 10 Done Done New true
1849 545 Acked 1544 Read 4 Done Done New true
1850 546 Acked 1545 Write 9 Done Done New true
1851 547 Acked 1546 Write 2 Done Done New true
1852 548 Acked 1547 Read 7 Done Done New true
1853 549 Acked 1548 Write 4 Done Done New true
1854 550 Acked 1549 Read 9 Done Done New true
1855 551 Acked 1550 Flush 0 Done Done New true
1856 552 Acked 1551 Read 10 Done Done New true
1857 553 Acked 1552 Read 1 Done Done New true
1858 554 Acked 1553 Write 9 Done Done New true
1859 555 Acked 1554 Write 4 Done Done New true
1860 556 Acked 1555 Write 2 Done Done New true
1861 557 Acked 1556 Read 1 Done Done New true
1862 558 Acked 1557 Write 9 Done Done New true
1863 559 Acked 1558 Read 7 Done Done New true
1864 560 Acked 1559 Flush 0 Done Done New true
1865 561 Acked 1560 Write 1 Done Done New true
1866 562 Acked 1561 Write 7 Done Done New true
1867 563 Acked 1562 Read 10 Done Done New true
1868 564 Acked 1563 Read 1 Done Done New true
1869 565 Acked 1564 Read 5 Done Done New true
1870 566 Acked 1565 Flush 0 Done Done New true
1871 567 Acked 1566 Read 7 Done Done New true
1872 568 Acked 1567 Read 7 Done Done New true
1873 569 Acked 1568 Write 9 Done Done New true
1874 570 Acked 1569 Read 9 Done Done New true
1875 571 Acked 1570 Read 7 Done Done New true
1876 572 Acked 1571 Write 8 Done Done New true
1877 573 Acked 1572 Read 10 Done Done New true
1878 574 Acked 1573 Read 6 Done Done New true
1879 575 Acked 1574 Write 10 Done Done New true
1880 576 Acked 1575 Read 1 Done Done New true
1881 577 Acked 1576 Flush 0 Done Done New true
1882 578 Acked 1577 Read 8 Done Done New true
1883 579 Acked 1578 Read 3 Done Done New true
1884 580 Acked 1579 Write 3 Done Done New true
1885 581 Acked 1580 Write 10 Done Done New true
1886 582 Acked 1581 Read 5 Done Done New true
1887 583 Acked 1582 Flush 0 Done Done New true
1888 584 Acked 1583 Write 3 Done Done New true
1889 585 Acked 1584 Read 10 Done Done New true
1890 586 Acked 1585 Write 9 Done Done New true
1891 587 Acked 1586 Flush 0 Done Done New true
1892 588 Acked 1587 Write 6 Done Done New true
1893 589 Acked 1588 Read 10 Done Done New true
1894 590 Acked 1589 Write 1 Done Done New true
1895 591 Acked 1590 Read 3 Done Done New true
1896 592 Acked 1591 Read 10 Done Done New true
1897 593 Acked 1592 Write 4 Done Done New true
1898 594 Acked 1593 Read 9 Done Done New true
1899 595 Acked 1594 Write 6 Done Done New true
1900 596 Acked 1595 Read 5 Done Done New true
1901 597 Acked 1596 Read 1 Done Done New true
1902 598 Acked 1597 Read 9 Done Done New true
1903 599 Acked 1598 Read 6 Done Done New true
1904 600 Acked 1599 Read 1 Done Done New true
1905 601 Acked 1600 Write 1 Done Done New true
1906 602 Acked 1601 Flush 0 Done Done New true
1907 STATES DS:0 DS:1 DS:2 TOTAL
1908 New 0 0 301 301
1909 Sent 0 0 0 0
1910 Done 301 301 0 602
1911 Skipped 0 0 0 0
1912 Error 0 0 0 0
1913 Last Flush: 1601 1601 1300
1914 Downstairs last five completed: 1300 1299 1298 1297 1296
1915 Upstairs last five completed: 602 601 600 599 598
1916 CLIENT: Up:0 ds:301 act:2
19172024-02-27T21:44:59.342ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19182024-02-27T21:44:59.342ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19192024-02-27T21:44:59.342ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19202024-02-27T21:44:59.342ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
19212024-02-27T21:44:59.342ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19222024-02-27T21:44:59.342ZINFOcrucible: [2] client re-new 301 jobs since flush 1300 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
19232024-02-27T21:44:59.342ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
1924 [8830] got start action
1925 [8830] I am going to restart
1926 Starting downstairs at port 8830
1927 Make output file at "/tmp/dsc/downstairs-8830.txt"
1928 [8830][2] reports Starting
1929 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1169)
1930 [8830][2] reports Running
19312024-02-27T21:45:00.343ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19322024-02-27T21:45:00.344ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19332024-02-27T21:45:00.344ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19342024-02-27T21:45:00.344ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
19352024-02-27T21:45:00.344ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19362024-02-27T21:45:00.344ZINFOcrucible: [2] client re-new 301 jobs since flush 1300 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
19372024-02-27T21:45:00.344ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19382024-02-27T21:45:01.346ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19392024-02-27T21:45:01.346ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19402024-02-27T21:45:01.347ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19412024-02-27T21:45:01.347ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19422024-02-27T21:45:01.347ZINFOcrucible: send last flush ID to this DS: 1300 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19432024-02-27T21:45:01.347ZINFOcrucible: Replied this last flush ID: 1300 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19442024-02-27T21:45:01.347ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19452024-02-27T21:45:01.347ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19462024-02-27T21:45:01.348ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19472024-02-27T21:45:01.348ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
1948 ----------------------------------------------------------------
1949 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
1950 Downstairs last five completed: 1601 1600 1599 1598 1597
1951 Upstairs last five completed: 602 601 600 599 598
1952 CLIENT: Up:0 ds:0 act:3
1953 Replay: All jobs finished, all DS active.
1954 Main task has work to do, go find it
1955 got dsc Stop(2)
1956 stop 2
1957 Feb 27 21:45:02.692 INFO request completed, latency_us: 363, response_code: 204, uri:[8830] Got stop action so:false kr:false
1958 /stop/cid/2, method: POST, req_id: 97619b6a-8f70-4243-be23-46639e3fe378, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
19592024-02-27T21:45:02.693ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
1960 Feb 27 21:45:02.693 INFO request completed, latency_us: 126, response_code: 200, uri: /state/cid/2, method: GET, req_id{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-27T21:45:02.693319799Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"io task","client":"2","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
1961 : 2de97362-bf0d-4adf-810c-798080366971, remote_addr: 127.0.0.1:44347, {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-27T21:45:02.693352085Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
1962 local_addr: 127.0.0.1:9998
19632024-02-27T21:45:02.693ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19642024-02-27T21:45:02.693ZINFOcrucible: [2] client re-new 0 jobs since flush 1601 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
19652024-02-27T21:45:02.693ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
1966 [8830] Exited with: ExitStatus(unix_wait_status(9))
1967 [8830][2] reports Exit
19682024-02-27T21:45:03.694ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19692024-02-27T21:45:03.694ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19702024-02-27T21:45:03.694ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19712024-02-27T21:45:03.694ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
19722024-02-27T21:45:03.694ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19732024-02-27T21:45:03.694ZINFOcrucible: [2] client re-new 0 jobs since flush 1601 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
19742024-02-27T21:45:03.694ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19752024-02-27T21:45:04.695ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19762024-02-27T21:45:04.695ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19772024-02-27T21:45:04.695ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19782024-02-27T21:45:04.695ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
19792024-02-27T21:45:04.695ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19802024-02-27T21:45:04.695ZINFOcrucible: [2] client re-new 0 jobs since flush 1601 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
19812024-02-27T21:45:04.695ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19822024-02-27T21:45:05.696ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19832024-02-27T21:45:05.696ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19842024-02-27T21:45:05.696ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19852024-02-27T21:45:05.696ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
19862024-02-27T21:45:05.696ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19872024-02-27T21:45:05.696ZINFOcrucible: [2] client re-new 0 jobs since flush 1601 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
19882024-02-27T21:45:05.696ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
1989 Feb 27 21:45:06.695 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/2, method: GET, req_id: 7f85ca52-b506-412b-a93c-5be67c63f21f, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
1990 001/300 Read block 3509 len 20480
1991 002/300 Read block 3705 len 36864
19922024-02-27T21:45:06.697ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19932024-02-27T21:45:06.697ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19942024-02-27T21:45:06.697ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19952024-02-27T21:45:06.697ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
19962024-02-27T21:45:06.697ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
19972024-02-27T21:45:06.698ZINFOcrucible: [2] client re-new 2 jobs since flush 1601 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
19982024-02-27T21:45:06.698ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
1999 003/300 Flush
2000 004/300 Read block 4891 len 12288
2001 005/300 Read block 372 len 4096
2002 006/300 Read block 4765 len 28672
2003 007/300 Flush
2004 008/300 Read block 3930 len 36864
2005 009/300 Flush
2006 010/300 Read block 1965 len 16384
2007 011/300 Read block 266 len 12288
2008 012/300 Read block 184 len 24576
2009 013/300 Read block 3309 len 20480
2010 014/300 Write block 882 len 8192 data: 2 2
2011 015/300 Read block 2297 len 16384
2012 016/300 Write block 785 len 40960 data: 2 2 2 2 2 2 2 2 3 3
2013 017/300 Read block 783 len 24576
2014 018/300 Write block 2075 len 16384 data: 2 2 2 2
2015 019/300 Read block 4000 len 36864
2016 020/300 Read block 492 len 12288
2017 021/300 Write block 2878 len 20480 data: 2 2 2 2 2
2018 022/300 Write block 4576 len 36864 data: 2 2 2 2 2 2 2 2 2
2019 023/300 Read block 3914 len 8192
2020 024/300 Write block 1217 len 8192 data: 3 3
2021 025/300 Read block 629 len 28672
2022 026/300 Read block 4082 len 4096
2023 027/300 Read block 3264 len 24576
2024 028/300 Read block 246 len 36864
2025 029/300 Write block 2366 len 32768 data: 2 2 2 2 2 2 2 2
2026 030/300 Write block 2358 len 28672 data: 2 2 2 2 2 2 2
2027 031/300 Write block 1482 len 24576 data: 3 2 2 2 2 2
2028 032/300 Flush
2029 033/300 Write block 4646 len 36864 data: 2 2 2 2 2 2 2 2 2
2030 034/300 Read block 1234 len 20480
2031 035/300 Read block 3019 len 24576
2032 036/300 Read block 2337 len 16384
2033 037/300 Write block 2314 len 24576 data: 2 2 2 2 2 2
2034 038/300 Flush
2035 039/300 Flush
2036 040/300 Flush
2037 041/300 Read block 4192 len 28672
2038 042/300 Write block 186 len 16384 data: 3 2 2 2
2039 043/300 Write block 2898 len 16384 data: 2 2 2 2
2040 044/300 Write block 1975 len 12288 data: 2 2 2
2041 045/300 Read block 1614 len 8192
2042 046/300 Read block 81 len 40960
2043 047/300 Write block 595 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2044 048/300 Read block 65 len 20480
2045 049/300 Write block 3327 len 36864 data: 2 2 2 2 3 3 3 3 3
2046 050/300 Read block 1293 len 32768
2047 051/300 Read block 2630 len 4096
2048 052/300 Write block 42 len 20480 data: 2 2 2 2 2
2049 053/300 Flush
2050 054/300 Read block 2562 len 32768
2051 055/300 Read block 4538 len 4096
2052 056/300 Write block 4948 len 28672 data: 3 3 3 4 4 3 2
2053 057/300 Read block 4756 len 36864
2054 058/300 Write block 3633 len 28672 data: 3 3 3 3 3 2 2
2055 059/300 Read block 3325 len 28672
2056 060/300 Read block 4279 len 12288
2057 061/300 Write block 3486 len 8192 data: 2 2
2058 062/300 Write block 3797 len 20480 data: 2 3 3 3 3
2059 063/300 Read block 1584 len 32768
2060 064/300 Read block 595 len 8192
2061 065/300 Flush
2062 066/300 Read block 1424 len 20480
2063 067/300 Write block 3105 len 16384 data: 2 2 2 2
2064 068/300 Read block 4732 len 4096
2065 069/300 Write block 494 len 40960 data: 2 2 2 3 3 3 3 2 2 2
2066 070/300 Write block 2021 len 28672 data: 2 3 3 4 3 3 4
2067 071/300 Write block 3964 len 8192 data: 2 2
2068 072/300 Write block 4941 len 36864 data: 2 2 2 2 3 3 3 4 4
2069 073/300 Read block 3660 len 40960
2070 074/300 Write block 4487 len 28672 data: 3 3 3 3 3 3 2
2071 075/300 Write block 3311 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2072 076/300 Read block 2329 len 8192
2073 077/300 Read block 3580 len 8192
2074 078/300 Read block 4853 len 40960
2075 079/300 Read block 1671 len 32768
2076 080/300 Flush
2077 081/300 Read block 561 len 20480
2078 082/300 Write block 1569 len 16384 data: 2 2 2 2
2079 083/300 Read block 100 len 40960
2080 084/300 Read block 2676 len 16384
2081 085/300 Flush
2082 086/300 Read block 3933 len 40960
2083 087/300 Write block 988 len 24576 data: 2 2 2 2 2 2
2084 088/300 Read block 2556 len 40960
2085 089/300 Read block 561 len 32768
2086 090/300 Write block 528 len 12288 data: 2 2 2
2087 091/300 Write block 3468 len 28672 data: 2 2 2 2 2 2 2
2088 092/300 Write block 4056 len 28672 data: 3 3 3 3 2 2 2
2089 093/300 Read block 1033 len 40960
2090 094/300 Read block 14 len 20480
2091 095/300 Read block 2120 len 28672
2092 096/300 Read block 245 len 32768
2093 097/300 Read block 1230 len 4096
2094 098/300 Write block 3525 len 12288 data: 2 2 2
2095 099/300 Read block 2499 len 28672
2096 100/300 Read block 4434 len 32768
2097 101/300 Read block 2092 len 28672
2098 102/300 Read block 1808 len 20480
2099 103/300 Read block 2959 len 8192
2100 104/300 Read block 2259 len 40960
2101 105/300 Read block 3438 len 24576
2102 106/300 Read block 4212 len 28672
2103 107/300 Write block 3114 len 36864 data: 2 3 3 3 3 3 2 2 2
2104 108/300 Read block 4587 len 24576
2105 109/300 Read block 4221 len 16384
2106 110/300 Write block 613 len 28672 data: 2 2 2 2 2 2 2
2107 111/300 Write block 4249 len 12288 data: 2 2 2
2108 112/300 Read block 3591 len 20480
2109 113/300 Read block 2773 len 32768
2110 114/300 Write block 942 len 40960 data: 4 4 4 3 3 2 2 2 2 2
2111 115/300 Write block 4645 len 12288 data: 3 3 3
2112 116/300 Write block 578 len 24576 data: 3 3 3 2 2 2
2113 117/300 Read block 3330 len 16384
2114 118/300 Read block 449 len 24576
2115 119/300 Write block 2615 len 20480 data: 2 2 2 2 2
2116 120/300 Read block 705 len 24576
2117 121/300 Write block 505 len 36864 data: 2 2 2 2 2 2 2 2 2
2118 122/300 Read block 3540 len 4096
2119 123/300 Write block 1314 len 4096 data: 2
2120 124/300 Read block 4647 len 8192
2121 125/300 Read block 3559 len 40960
2122 126/300 Read block 886 len 36864
2123 127/300 Write block 1850 len 8192 data: 2 2
2124 128/300 Read block 1434 len 12288
2125 129/300 Flush
2126 130/300 Write block 930 len 28672 data: 2 2 2 2 2 2 2
2127 131/300 Write block 2327 len 4096 data: 2
2128 132/300 Write block 96 len 36864 data: 3 3 3 2 2 2 2 2 2
2129 133/300 Write block 107 len 12288 data: 2 2 2
2130 134/300 Write block 2255 len 36864 data: 2 3 3 3 3 3 3 4 3
2131 135/300 Read block 3844 len 36864
2132 136/300 Write block 2460 len 32768 data: 2 3 3 3 3 2 2 2
2133 137/300 Write block 1059 len 36864 data: 2 2 2 2 2 2 2 2 2
2134 138/300 Write block 4703 len 32768 data: 2 2 3 3 2 2 2 2
2135 139/300 Write block 2665 len 4096 data: 2
2136 140/300 Read block 957 len 4096
2137 141/300 Read block 4600 len 32768
2138 142/300 Write block 3665 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2139 143/300 Write block 523 len 16384 data: 3 2 2 2
2140 144/300 Write block 1098 len 12288 data: 2 2 2
2141 145/300 Write block 687 len 4096 data: 2
2142 146/300 Read block 4497 len 40960
2143 147/300 Read block 131 len 16384
2144 148/300 Write block 1783 len 20480 data: 2 2 2 2 2
2145 149/300 Write block 2481 len 4096 data: 4
2146 150/300 Flush
2147 151/300 Write block 2548 len 16384 data: 3 3 3 2
2148 152/300 Read block 1360 len 24576
2149 153/300 Read block 3992 len 28672
2150 154/300 Read block 4640 len 4096
2151 155/300 Read block 1137 len 40960
2152 156/300 Write block 2357 len 8192 data: 3 3
2153 157/300 Read block 1014 len 20480
2154 158/300 Write block 1655 len 40960 data: 2 2 3 3 3 3 2 2 2 3
2155 159/300 Write block 4906 len 32768 data: 2 2 2 2 2 2 2 2
2156 160/300 Write block 3481 len 32768 data: 2 3 2 2 2 3 3 2
2157 161/300 Read block 5 len 20480
2158 162/300 Read block 2753 len 12288
2159 163/300 Write block 929 len 4096 data: 2
2160 164/300 Read block 4458 len 24576
2161 165/300 Read block 1881 len 8192
2162 166/300 Write block 4691 len 28672 data: 2 2 2 2 2 2 2
2163 167/300 Read block 4475 len 24576
2164 168/300 Read block 2701 len 8192
2165 169/300 Flush
2166 170/300 Flush
2167 171/300 Read block 490 len 12288
2168 172/300 Read block 271 len 40960
2169 173/300 Write block 178 len 36864 data: 2 2 2 2 3 3 3 3 4
2170 174/300 Read block 554 len 28672
2171 175/300 Read block 3780 len 4096
2172 176/300 Read block 2775 len 4096
2173 177/300 Flush
2174 178/300 Write block 502 len 8192 data: 3 3
2175 179/300 Write block 266 len 32768 data: 2 2 2 2 2 2 2 2
2176 180/300 Read block 2923 len 32768
2177 181/300 Write block 3571 len 32768 data: 2 2 2 2 2 2 2 2
2178 182/300 Flush
2179 183/300 Write block 2843 len 32768 data: 2 2 2 2 2 2 2 2
2180 184/300 Write block 2310 len 36864 data: 2 2 2 2 3 3 3 3 3
2181 185/300 Write block 4207 len 4096 data: 2
2182 186/300 Flush
2183 187/300 Read block 3759 len 16384
2184 188/300 Read block 2648 len 32768
2185 189/300 Write block 2615 len 4096 data: 3
2186 190/300 Read block 4646 len 4096
2187 191/300 Write block 1217 len 20480 data: 4 4 3 3 3
2188 192/300 Write block 4444 len 16384 data: 2 2 2 2
2189 193/300 Read block 3175 len 24576
2190 194/300 Write block 4892 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2191 195/300 Read block 187 len 4096
2192 196/300 Write block 3410 len 32768 data: 2 2 2 2 2 2 2 2
2193 197/300 Read block 2901 len 16384
2194 198/300 Flush
2195 199/300 Write block 726 len 36864 data: 2 2 2 2 2 2 2 2 2
2196 200/300 Write block 2202 len 4096 data: 2
2197 201/300 Read block 3518 len 8192
2198 202/300 Read block 4827 len 8192
2199 203/300 Read block 359 len 12288
2200 204/300 Write block 1603 len 4096 data: 2
2201 205/300 Read block 3781 len 16384
2202 206/300 Read block 254 len 8192
2203 207/300 Flush
2204 208/300 Read block 375 len 8192
2205 209/300 Write block 2546 len 12288 data: 3 3 4
2206 210/300 Read block 1208 len 12288
2207 211/300 Write block 601 len 12288 data: 3 3 3
2208 212/300 Write block 4320 len 28672 data: 2 2 2 2 2 2 2
2209 213/300 Read block 2275 len 32768
2210 214/300 Write block 3409 len 8192 data: 2 3
2211 215/300 Write block 4086 len 32768 data: 2 2 2 2 2 2 3 3
2212 216/300 Read block 162 len 32768
2213 217/300 Read block 767 len 40960
2214 218/300 Write block 3977 len 12288 data: 3 3 3
2215 219/300 Read block 1274 len 24576
2216 220/300 Read block 3497 len 32768
2217 221/300 Write block 2683 len 16384 data: 2 2 2 2
2218 222/300 Read block 4951 len 36864
2219 223/300 Flush
2220 224/300 Write block 3850 len 28672 data: 2 2 2 2 2 2 2
2221 225/300 Read block 2763 len 32768
2222 226/300 Write block 2469 len 32768 data: 2 2 2 2 2 2 2 3
2223 227/300 Write block 3568 len 12288 data: 2 2 2
2224 228/300 Write block 1143 len 24576 data: 2 2 2 2 2 2
2225 229/300 Flush
2226 230/300 Write block 3265 len 8192 data: 2 2
2227 231/300 Read block 4196 len 24576
2228 232/300 Write block 3386 len 12288 data: 2 2 2
2229 233/300 Read block 177 len 4096
2230 234/300 Read block 4174 len 36864
2231 235/300 Read block 1335 len 16384
2232 236/300 Flush
2233 237/300 Write block 338 len 16384 data: 2 2 2 2
2234 238/300 Write block 3932 len 32768 data: 2 2 2 2 2 2 2 2
2235 239/300 Read block 723 len 4096
2236 240/300 Read block 1713 len 12288
2237 241/300 Write block 3286 len 20480 data: 2 2 2 2 2
2238 242/300 Read block 3965 len 32768
2239 243/300 Write block 2711 len 40960 data: 3 3 3 3 3 3 3 3 2 2
2240 244/300 Write block 3900 len 28672 data: 2 2 2 2 2 2 2
2241 245/300 Write block 784 len 28672 data: 2 3 3 3 3 3 3
2242 246/300 Write block 4766 len 36864 data: 2 2 2 2 2 2 2 2 2
2243 247/300 Read block 4675 len 32768
2244 248/300 Flush
2245 249/300 Write block 3689 len 8192 data: 3 2
2246 250/300 Read block 1098 len 36864
2247 251/300 Write block 1307 len 40960 data: 2 2 2 2 2 2 2 3 2 2
2248 252/300 Read block 3259 len 4096
2249 253/300 Write block 1406 len 12288 data: 2 3 3
2250 254/300 Write block 2905 len 36864 data: 2 2 2 3 3 3 3 2 2
2251 255/300 Write block 2858 len 8192 data: 2 2
2252 256/300 Write block 2048 len 32768 data: 2 2 2 2 2 3 3 3
2253 257/300 Flush
2254 258/300 Read block 1824 len 12288
2255 259/300 Read block 2015 len 4096
2256 260/300 Write block 2375 len 8192 data: 2 2
2257 261/300 Write block 4486 len 4096 data: 3
2258 262/300 Read block 3958 len 16384
2259 263/300 Flush
2260 264/300 Write block 2334 len 16384 data: 2 2 2 2
2261 265/300 Write block 255 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2262 266/300 Write block 4744 len 28672 data: 2 3 3 3 4 5 4
2263 267/300 Write block 4622 len 12288 data: 2 2 2
2264 268/300 Write block 4428 len 4096 data: 2
2265 269/300 Read block 2882 len 16384
2266 270/300 Write block 3952 len 32768 data: 2 2 2 2 2 2 2 2
2267 271/300 Read block 4225 len 28672
2268 272/300 Read block 3578 len 8192
2269 273/300 Write block 3796 len 32768 data: 3 3 4 4 4 4 3 3
2270 274/300 Write block 3759 len 12288 data: 2 2 2
2271 275/300 Read block 1223 len 8192
2272 276/300 Read block 4702 len 12288
2273 277/300 Read block 575 len 36864
2274 278/300 Read block 4225 len 40960
2275 279/300 Read block 3005 len 24576
2276 280/300 Read block 2300 len 36864
2277 281/300 Read block 3366 len 36864
2278 282/300 Flush
2279 283/300 Write block 3679 len 4096 data: 3
2280 284/300 Read block 2659 len 8192
2281 285/300 Flush
2282 286/300 Read block 3804 len 40960
2283 287/300 Write block 16 len 8192 data: 3 3
2284 288/300 Write block 1118 len 24576 data: 2 2 2 2 2 2
2285 289/300 Write block 3489 len 40960 data: 2 2 3 3 3 3 3 3 3 2
2286 290/300 Write block 678 len 4096 data: 2
2287 291/300 Read block 4946 len 28672
2288 292/300 Write block 3678 len 24576 data: 3 4 3 2 3 3
2289 293/300 Write block 2293 len 20480 data: 3 2 2 2 2
2290 294/300 Read block 225 len 20480
2291 295/300 Write block 3943 len 24576 data: 3 3 3 3 3 2
2292 296/300 Write block 1361 len 32768 data: 2 2 2 2 2 2 2 3
2293 297/300 Write block 4677 len 32768 data: 2 2 2 2 2 2 2 2
2294 298/300 Flush
2295 299/300 Write block 765 len 36864 data: 2 3 3 2 2 2 2 2 2
2296 300/300 Write block 200 len 12288 data: 2 2 3
2297 Main task has work to do, go find it
2298 got dsc Start(2)
2299 start 2
2300 Feb 27 21:45:07.029 INFO request completed, latency_us: 221, response_code: 204, uri: /start/cid/2, method: POST, req_id: 8603b42d-2a72-46d5-b4f6-475162a58c8e, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
2301 Replay: started 2, returned:Ok(())
2302 ----------------------------------------------------------------
2303 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
2304 States: Active Active Offline
2305 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
2306 603 Acked 1602 Read 5 Done Done New true
2307 604 Acked 1603 Read 9 Done Done New true
2308 605 Acked 1604 Flush 0 Done Done New false
2309 606 Acked 1605 Read 3 Done Done New false
2310 607 Acked 1606 Read 1 Done Done New false
2311 608 Acked 1607 Read 7 Done Done New false
2312 609 Acked 1608 Flush 0 Done Done New false
2313 610 Acked 1609 Read 9 Done Done New false
2314 611 Acked 1610 Flush 0 Done Done New false
2315 612 Acked 1611 Read 4 Done Done New false
2316 613 Acked 1612 Read 3 Done Done New false
2317 614 Acked 1613 Read 6 Done Done New false
2318 615 Acked 1614 Read 5 Done Done New false
2319 616 Acked 1615 Write 2 Done Done New false
2320 617 Acked 1616 Read 4 Done Done New false
2321 618 Acked 1617 Write 10 Done Done New false
2322 619 Acked 1618 Read 6 Done Done New false
2323 620 Acked 1619 Write 4 Done Done New false
2324 621 Acked 1620 Read 9 Done Done New false
2325 622 Acked 1621 Read 3 Done Done New false
2326 623 Acked 1622 Write 5 Done Done New false
2327 624 Acked 1623 Write 9 Done Done New false
2328 625 Acked 1624 Read 2 Done Done New false
2329 626 Acked 1625 Write 2 Done Done New false
2330 627 Acked 1626 Read 7 Done Done New false
2331 628 Acked 1627 Read 1 Done Done New false
2332 629 Acked 1628 Read 6 Done Done New false
2333 630 Acked 1629 Read 9 Done Done New false
2334 631 Acked 1630 Write 8 Done Done New false
2335 632 Acked 1631 Write 7 Done Done New false
2336 633 Acked 1632 Write 6 Done Done New false
2337 634 Acked 1633 Flush 0 Done Done New false
2338 635 Acked 1634 Write 9 Done Done New false
2339 636 Acked 1635 Read 5 Done Done New false
2340 637 Acked 1636 Read 6 Done Done New false
2341 638 Acked 1637 Read 4 Done Done New false
2342 639 Acked 1638 Write 6 Done Done New false
2343 640 Acked 1639 Flush 0 Done Done New false
2344 641 Acked 1640 Flush 0 Done Done New false
2345 642 Acked 1641 Flush 0 Done Done New false
2346 643 Acked 1642 Read 7 Done Done New false
2347 644 Acked 1643 Write 4 Done Done New false
2348 645 Acked 1644 Write 4 Done Done New false
2349 646 Acked 1645 Write 3 Done Done New false
2350 647 Acked 1646 Read 2 Done Done New false
2351 648 Acked 1647 Read 10 Done Done New false
2352 649 Acked 1648 Write 10 Done Done New false
2353 650 Acked 1649 Read 5 Done Done New false
2354 651 Acked 1650 Write 9 Done Done New false
2355 652 Acked 1651 Read 8 Done Done New false
2356 653 Acked 1652 Read 1 Done Done New false
2357 654 Acked 1653 Write 5 Done Done New false
2358 655 Acked 1654 Flush 0 Done Done New false
2359 656 Acked 1655 Read 8 Done Done New false
2360 657 Acked 1656 Read 1 Done Done New false
2361 658 Acked 1657 Write 7 Done Done New false
2362 659 Acked 1658 Read 9 Done Done New false
2363 660 Acked 1659 Write 7 Done Done New false
2364 661 Acked 1660 Read 7 Done Done New false
2365 662 Acked 1661 Read 3 Done Done New false
2366 663 Acked 1662 Write 2 Done Done New false
2367 664 Acked 1663 Write 5 Done Done New false
2368 665 Acked 1664 Read 8 Done Done New false
2369 666 Acked 1665 Read 2 Done Done New false
2370 667 Acked 1666 Flush 0 Done Done New false
2371 668 Acked 1667 Read 5 Done Done New false
2372 669 Acked 1668 Write 4 Done Done New false
2373 670 Acked 1669 Read 1 Done Done New false
2374 671 Acked 1670 Write 10 Done Done New false
2375 672 Acked 1671 Write 7 Done Done New false
2376 673 Acked 1672 Write 2 Done Done New false
2377 674 Acked 1673 Write 9 Done Done New false
2378 675 Acked 1674 Read 10 Done Done New false
2379 676 Acked 1675 Write 7 Done Done New false
2380 677 Acked 1676 Write 10 Done Done New false
2381 678 Acked 1677 Read 2 Done Done New false
2382 679 Acked 1678 Read 2 Done Done New false
2383 680 Acked 1679 Read 10 Done Done New false
2384 681 Acked 1680 Read 8 Done Done New false
2385 682 Acked 1681 Flush 0 Done Done New false
2386 683 Acked 1682 Read 5 Done Done New false
2387 684 Acked 1683 Write 4 Done Done New false
2388 685 Acked 1684 Read 10 Done Done New false
2389 686 Acked 1685 Read 4 Done Done New false
2390 687 Acked 1686 Flush 0 Done Done New false
2391 688 Acked 1687 Read 10 Done Done New false
2392 689 Acked 1688 Write 6 Done Done New false
2393 690 Acked 1689 Read 10 Done Done New false
2394 691 Acked 1690 Read 8 Done Done New false
2395 692 Acked 1691 Write 3 Done Done New false
2396 693 Acked 1692 Write 7 Done Done New false
2397 694 Acked 1693 Write 7 Done Done New false
2398 695 Acked 1694 Read 10 Done Done New false
2399 696 Acked 1695 Read 5 Done Done New false
2400 697 Acked 1696 Read 7 Done Done New false
2401 698 Acked 1697 Read 8 Done Done New false
2402 699 Acked 1698 Read 1 Done Done New false
2403 700 Acked 1699 Write 3 Done Done New false
2404 701 Acked 1700 Read 7 Done Done New false
2405 702 Acked 1701 Read 8 Done Done New false
2406 703 Acked 1702 Read 7 Done Done New false
2407 704 Acked 1703 Read 5 Done Done New false
2408 705 Acked 1704 Read 2 Done Done New false
2409 706 Acked 1705 Read 10 Done Done New false
2410 707 Acked 1706 Read 6 Done Done New false
2411 708 Acked 1707 Read 7 Done Done New false
2412 709 Acked 1708 Write 9 Done Done New false
2413 710 Acked 1709 Read 6 Done Done New false
2414 711 Acked 1710 Read 4 Done Done New false
2415 712 Acked 1711 Write 7 Done Done New false
2416 713 Acked 1712 Write 3 Done Done New false
2417 714 Acked 1713 Read 5 Done Done New false
2418 715 Acked 1714 Read 8 Done Done New false
2419 716 Acked 1715 Write 10 Done Done New false
2420 717 Acked 1716 Write 3 Done Done New false
2421 718 Acked 1717 Write 6 Done Done New false
2422 719 Acked 1718 Read 4 Done Done New false
2423 720 Acked 1719 Read 6 Done Done New false
2424 721 Acked 1720 Write 5 Done Done New false
2425 722 Acked 1721 Read 6 Done Done New false
2426 723 Acked 1722 Write 9 Done Done New false
2427 724 Acked 1723 Read 1 Done Done New false
2428 725 Acked 1724 Write 1 Done Done New false
2429 726 Acked 1725 Read 2 Done Done New false
2430 727 Acked 1726 Read 10 Done Done New false
2431 728 Acked 1727 Read 9 Done Done New false
2432 729 Acked 1728 Write 2 Done Done New false
2433 730 Acked 1729 Read 3 Done Done New false
2434 731 Acked 1730 Flush 0 Done Done New false
2435 732 Acked 1731 Write 7 Done Done New false
2436 733 Acked 1732 Write 1 Done Done New false
2437 734 Acked 1733 Write 9 Done Done New false
2438 735 Acked 1734 Write 3 Done Done New false
2439 736 Acked 1735 Write 9 Done Done New false
2440 737 Acked 1736 Read 9 Done Done New false
2441 738 Acked 1737 Write 8 Done Done New false
2442 739 Acked 1738 Write 9 Done Done New false
2443 740 Acked 1739 Write 8 Done Done New false
2444 741 Acked 1740 Write 1 Done Done New false
2445 742 Acked 1741 Read 1 Done Done New false
2446 743 Acked 1742 Read 8 Done Done New false
2447 744 Acked 1743 Write 10 Done Done New false
2448 745 Acked 1744 Write 4 Done Done New false
2449 746 Acked 1745 Write 3 Done Done New false
2450 747 Acked 1746 Write 1 Done Done New false
2451 748 Acked 1747 Read 10 Done Done New false
2452 749 Acked 1748 Read 4 Done Done New false
2453 750 Acked 1749 Write 5 Done Done New false
2454 751 Acked 1750 Write 1 Done Done New false
2455 752 Acked 1751 Flush 0 Done Done New false
2456 753 Acked 1752 Write 4 Done Done New false
2457 754 Acked 1753 Read 6 Done Done New false
2458 755 Acked 1754 Read 7 Done Done New false
2459 756 Acked 1755 Read 1 Done Done New false
2460 757 Acked 1756 Read 10 Done Done New false
2461 758 Acked 1757 Write 2 Done Done New false
2462 759 Acked 1758 Read 5 Done Done New false
2463 760 Acked 1759 Write 10 Done Done New false
2464 761 Acked 1760 Write 8 Done Done New false
2465 762 Acked 1761 Write 8 Done Done New false
2466 763 Acked 1762 Read 5 Done Done New false
2467 764 Acked 1763 Read 3 Done Done New false
2468 765 Acked 1764 Write 1 Done Done New false
2469 766 Acked 1765 Read 6 Done Done New false
2470 767 Acked 1766 Read 2 Done Done New false
2471 768 Acked 1767 Write 7 Done Done New false
2472 769 Acked 1768 Read 6 Done Done New false
2473 770 Acked 1769 Read 2 Done Done New false
2474 771 Acked 1770 Flush 0 Done Done New false
2475 772 Acked 1771 Flush 0 Done Done New false
2476 773 Acked 1772 Read 3 Done Done New false
2477 774 Acked 1773 Read 10 Done Done New false
2478 775 Acked 1774 Write 9 Done Done New false
2479 776 Acked 1775 Read 7 Done Done New false
2480 777 Acked 1776 Read 1 Done Done New false
2481 778 Acked 1777 Read 1 Done Done New false
2482 779 Acked 1778 Flush 0 Done Done New false
2483 780 Acked 1779 Write 2 Done Done New false
2484 781 Acked 1780 Write 8 Done Done New false
2485 782 Acked 1781 Read 8 Done Done New false
2486 783 Acked 1782 Write 8 Done Done New false
2487 784 Acked 1783 Flush 0 Done Done New false
2488 785 Acked 1784 Write 8 Done Done New false
2489 786 Acked 1785 Write 9 Done Done New false
2490 787 Acked 1786 Write 1 Done Done New false
2491 788 Acked 1787 Flush 0 Done Done New false
2492 789 Acked 1788 Read 4 Done Done New false
2493 790 Acked 1789 Read 8 Done Done New false
2494 791 Acked 1790 Write 1 Done Done New false
2495 792 Acked 1791 Read 1 Done Done New false
2496 793 Acked 1792 Write 5 Done Done New false
2497 794 Acked 1793 Write 4 Done Done New false
2498 795 Acked 1794 Read 6 Done Done New false
2499 796 Acked 1795 Write 10 Done Done New false
2500 797 Acked 1796 Read 1 Done Done New false
2501 798 Acked 1797 Write 8 Done Done New false
2502 799 Acked 1798 Read 4 Done Done New false
2503 800 Acked 1799 Flush 0 Done Done New false
2504 801 Acked 1800 Write 9 Done Done New false
2505 802 Acked 1801 Write 1 Done Done New false
2506 803 Acked 1802 Read 2 Done Done New false
2507 804 Acked 1803 Read 2 Done Done New false
2508 805 Acked 1804 Read 3 Done Done New false
2509 806 Acked 1805 Write 1 Done Done New false
2510 807 Acked 1806 Read 4 Done Done New false
2511 808 Acked 1807 Read 2 Done Done New false
2512 809 Acked 1808 Flush 0 Done Done New false
2513 810 Acked 1809 Read 2 Done Done New false
2514 811 Acked 1810 Write 3 Done Done New false
2515 812 Acked 1811 Read 3 Done Done New false
2516 813 Acked 1812 Write 3 Done Done New false
2517 814 Acked 1813 Write 7 Done Done New false
2518 815 Acked 1814 Read 8 Done Done New false
2519 816 Acked 1815 Write 2 Done Done New false
2520 817 Acked 1816 Write 8 Done Done New false
2521 818 Acked 1817 Read 8 Done Done New false
2522 819 Acked 1818 Read 10 Done Done New false
2523 820 Acked 1819 Write 3 Done Done New false
2524 821 Acked 1820 Read 6 Done Done New false
2525 822 Acked 1821 Read 8 Done Done New false
2526 823 Acked 1822 Write 4 Done Done New false
2527 824 Acked 1823 Read 9 Done Done New false
2528 825 Acked 1824 Flush 0 Done Done New false
2529 826 Acked 1825 Write 7 Done Done New false
2530 827 Acked 1826 Read 8 Done Done New false
2531 828 Acked 1827 Write 8 Done Done New false
2532 829 Acked 1828 Write 3 Done Done New false
2533 830 Acked 1829 Write 6 Done Done New false
2534 831 Acked 1830 Flush 0 Done Done New false
2535 832 Acked 1831 Write 2 Done Done New false
2536 833 Acked 1832 Read 6 Done Done New false
2537 834 Acked 1833 Write 3 Done Done New false
2538 835 Acked 1834 Read 1 Done Done New false
2539 836 Acked 1835 Read 9 Done Done New false
2540 837 Acked 1836 Read 4 Done Done New false
2541 838 Acked 1837 Flush 0 Done Done New false
2542 839 Acked 1838 Write 4 Done Done New false
2543 840 Acked 1839 Write 8 Done Done New false
2544 841 Acked 1840 Read 1 Done Done New false
2545 842 Acked 1841 Read 3 Done Done New false
2546 843 Acked 1842 Write 5 Done Done New false
2547 844 Acked 1843 Read 8 Done Done New false
2548 845 Acked 1844 Write 10 Done Done New false
2549 846 Acked 1845 Write 7 Done Done New false
2550 847 Acked 1846 Write 7 Done Done New false
2551 848 Acked 1847 Write 9 Done Done New false
2552 849 Acked 1848 Read 8 Done Done New false
2553 850 Acked 1849 Flush 0 Done Done New false
2554 851 Acked 1850 Write 2 Done Done New false
2555 852 Acked 1851 Read 9 Done Done New false
2556 853 Acked 1852 Write 10 Done Done New false
2557 854 Acked 1853 Read 1 Done Done New false
2558 855 Acked 1854 Write 3 Done Done New false
2559 856 Acked 1855 Write 9 Done Done New false
2560 857 Acked 1856 Write 2 Done Done New false
2561 858 Acked 1857 Write 8 Done Done New false
2562 859 Acked 1858 Flush 0 Done Done New false
2563 860 Acked 1859 Read 3 Done Done New false
2564 861 Acked 1860 Read 1 Done Done New false
2565 862 Acked 1861 Write 2 Done Done New false
2566 863 Acked 1862 Write 1 Done Done New false
2567 864 Acked 1863 Read 4 Done Done New false
2568 865 Acked 1864 Flush 0 Done Done New false
2569 866 Acked 1865 Write 4 Done Done New false
2570 867 Acked 1866 Write 10 Done Done New false
2571 868 Acked 1867 Write 7 Done Done New false
2572 869 Acked 1868 Write 3 Done Done New false
2573 870 Acked 1869 Write 1 Done Done New false
2574 871 Acked 1870 Read 4 Done Done New false
2575 872 Acked 1871 Write 8 Done Done New false
2576 873 Acked 1872 Read 7 Done Done New false
2577 874 Acked 1873 Read 2 Done Done New false
2578 875 Acked 1874 Write 8 Done Done New false
2579 876 Acked 1875 Write 3 Done Done New false
2580 877 Acked 1876 Read 2 Done Done New false
2581 878 Acked 1877 Read 3 Done Done New false
2582 879 Acked 1878 Read 9 Done Done New false
2583 880 Acked 1879 Read 10 Done Done New false
2584 881 Acked 1880 Read 6 Done Done New false
2585 882 Acked 1881 Read 9 Done Done New false
2586 883 Acked 1882 Read 9 Done Done New false
2587 884 Acked 1883 Flush 0 Done Done New false
2588 885 Acked 1884 Write 1 Done Done New false
2589 886 Acked 1885 Read 2 Done Done New false
2590 887 Acked 1886 Flush 0 Done Done New false
2591 888 Acked 1887 Read 10 Done Done New false
2592 889 Acked 1888 Write 2 Done Done New false
2593 890 Acked 1889 Write 6 Done Done New false
2594 891 Acked 1890 Write 10 Done Done New false
2595 892 Acked 1891 Write 1 Done Done New false
2596 893 Acked 1892 Read 7 Done Done New false
2597 894 Acked 1893 Write 6 Done Done New false
2598 895 Acked 1894 Write 5 Done Done New false
2599 896 Acked 1895 Read 5 Done Done New false
2600 897 Acked 1896 Write 6 Done Done New false
2601 898 Acked 1897 Write 8 Done Done New false
2602 899 Acked 1898 Write 8 Done Done New false
2603 900 Acked 1899 Flush 0 Done Done New false
2604 901 Acked 1900 Write 9 Done Done New false
2605 902 Acked 1901 Write 3 Done Done New false
2606 STATES DS:0 DS:1 DS:2 TOTAL
2607 New 0 0 300 300
2608 Sent 0 0 0 0
2609 Done 300 300 0 600
2610 Skipped 0 0 0 0
2611 Error 0 0 0 0
2612 Last Flush: 1899 1899 1601
2613 Downstairs last five completed: 1601 1600 1599 1598 1597
2614 Upstairs last five completed: 902 901 900 899 898
2615 CLIENT: Up:0 ds:300 act:2
26162024-02-27T21:45:07.699ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
26172024-02-27T21:45:07.699ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
26182024-02-27T21:45:07.699ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
26192024-02-27T21:45:07.699ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
26202024-02-27T21:45:07.699ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
26212024-02-27T21:45:07.699ZINFOcrucible: [2] client re-new 301 jobs since flush 1601 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
26222024-02-27T21:45:07.700ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
26232024-02-27T21:45:08.700ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
26242024-02-27T21:45:08.700ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
26252024-02-27T21:45:08.700ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
26262024-02-27T21:45:08.700ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
26272024-02-27T21:45:08.700ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
26282024-02-27T21:45:08.700ZINFOcrucible: [2] client re-new 301 jobs since flush 1601 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
26292024-02-27T21:45:08.700ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
26302024-02-27T21:45:09.701ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
26312024-02-27T21:45:09.701ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
26322024-02-27T21:45:09.701ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
26332024-02-27T21:45:09.701ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
26342024-02-27T21:45:09.701ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
26352024-02-27T21:45:09.701ZINFOcrucible: [2] client re-new 301 jobs since flush 1601 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
26362024-02-27T21:45:09.702ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2637 [8830] got start action
2638 [8830] I am going to restart
2639 Starting downstairs at port 8830
2640 Make output file at "/tmp/dsc/downstairs-8830.txt"
2641 [8830][2] reports Starting
26422024-02-27T21:45:10.703ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
26432024-02-27T21:45:10.703ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
26442024-02-27T21:45:10.703ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
26452024-02-27T21:45:10.703ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
26462024-02-27T21:45:10.703ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
26472024-02-27T21:45:10.703ZINFOcrucible: [2] client re-new 301 jobs since flush 1601 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
26482024-02-27T21:45:10.703ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2649 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1170)
2650 [8830][2] reports Running
2651 ----------------------------------------------------------------
2652 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
2653 States: Active Active Offline
2654 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
2655 603 Acked 1602 Read 5 Done Done New true
2656 604 Acked 1603 Read 9 Done Done New true
2657 605 Acked 1604 Flush 0 Done Done New true
2658 606 Acked 1605 Read 3 Done Done New true
2659 607 Acked 1606 Read 1 Done Done New true
2660 608 Acked 1607 Read 7 Done Done New true
2661 609 Acked 1608 Flush 0 Done Done New true
2662 610 Acked 1609 Read 9 Done Done New true
2663 611 Acked 1610 Flush 0 Done Done New true
2664 612 Acked 1611 Read 4 Done Done New true
2665 613 Acked 1612 Read 3 Done Done New true
2666 614 Acked 1613 Read 6 Done Done New true
2667 615 Acked 1614 Read 5 Done Done New true
2668 616 Acked 1615 Write 2 Done Done New true
2669 617 Acked 1616 Read 4 Done Done New true
2670 618 Acked 1617 Write 10 Done Done New true
2671 619 Acked 1618 Read 6 Done Done New true
2672 620 Acked 1619 Write 4 Done Done New true
2673 621 Acked 1620 Read 9 Done Done New true
2674 622 Acked 1621 Read 3 Done Done New true
2675 623 Acked 1622 Write 5 Done Done New true
2676 624 Acked 1623 Write 9 Done Done New true
2677 625 Acked 1624 Read 2 Done Done New true
2678 626 Acked 1625 Write 2 Done Done New true
2679 627 Acked 1626 Read 7 Done Done New true
2680 628 Acked 1627 Read 1 Done Done New true
2681 629 Acked 1628 Read 6 Done Done New true
2682 630 Acked 1629 Read 9 Done Done New true
2683 631 Acked 1630 Write 8 Done Done New true
2684 632 Acked 1631 Write 7 Done Done New true
2685 633 Acked 1632 Write 6 Done Done New true
2686 634 Acked 1633 Flush 0 Done Done New true
2687 635 Acked 1634 Write 9 Done Done New true
2688 636 Acked 1635 Read 5 Done Done New true
2689 637 Acked 1636 Read 6 Done Done New true
2690 638 Acked 1637 Read 4 Done Done New true
2691 639 Acked 1638 Write 6 Done Done New true
2692 640 Acked 1639 Flush 0 Done Done New true
2693 641 Acked 1640 Flush 0 Done Done New true
2694 642 Acked 1641 Flush 0 Done Done New true
2695 643 Acked 1642 Read 7 Done Done New true
2696 644 Acked 1643 Write 4 Done Done New true
2697 645 Acked 1644 Write 4 Done Done New true
2698 646 Acked 1645 Write 3 Done Done New true
2699 647 Acked 1646 Read 2 Done Done New true
2700 648 Acked 1647 Read 10 Done Done New true
2701 649 Acked 1648 Write 10 Done Done New true
2702 650 Acked 1649 Read 5 Done Done New true
2703 651 Acked 1650 Write 9 Done Done New true
2704 652 Acked 1651 Read 8 Done Done New true
2705 653 Acked 1652 Read 1 Done Done New true
2706 654 Acked 1653 Write 5 Done Done New true
2707 655 Acked 1654 Flush 0 Done Done New true
2708 656 Acked 1655 Read 8 Done Done New true
2709 657 Acked 1656 Read 1 Done Done New true
2710 658 Acked 1657 Write 7 Done Done New true
2711 659 Acked 1658 Read 9 Done Done New true
2712 660 Acked 1659 Write 7 Done Done New true
2713 661 Acked 1660 Read 7 Done Done New true
2714 662 Acked 1661 Read 3 Done Done New true
2715 663 Acked 1662 Write 2 Done Done New true
2716 664 Acked 1663 Write 5 Done Done New true
2717 665 Acked 1664 Read 8 Done Done New true
2718 666 Acked 1665 Read 2 Done Done New true
2719 667 Acked 1666 Flush 0 Done Done New true
2720 668 Acked 1667 Read 5 Done Done New true
2721 669 Acked 1668 Write 4 Done Done New true
2722 670 Acked 1669 Read 1 Done Done New true
2723 671 Acked 1670 Write 10 Done Done New true
2724 672 Acked 1671 Write 7 Done Done New true
2725 673 Acked 1672 Write 2 Done Done New true
2726 674 Acked 1673 Write 9 Done Done New true
2727 675 Acked 1674 Read 10 Done Done New true
2728 676 Acked 1675 Write 7 Done Done New true
2729 677 Acked 1676 Write 10 Done Done New true
2730 678 Acked 1677 Read 2 Done Done New true
2731 679 Acked 1678 Read 2 Done Done New true
2732 680 Acked 1679 Read 10 Done Done New true
2733 681 Acked 1680 Read 8 Done Done New true
2734 682 Acked 1681 Flush 0 Done Done New true
2735 683 Acked 1682 Read 5 Done Done New true
2736 684 Acked 1683 Write 4 Done Done New true
2737 685 Acked 1684 Read 10 Done Done New true
2738 686 Acked 1685 Read 4 Done Done New true
2739 687 Acked 1686 Flush 0 Done Done New true
2740 688 Acked 1687 Read 10 Done Done New true
2741 689 Acked 1688 Write 6 Done Done New true
2742 690 Acked 1689 Read 10 Done Done New true
2743 691 Acked 1690 Read 8 Done Done New true
2744 692 Acked 1691 Write 3 Done Done New true
2745 693 Acked 1692 Write 7 Done Done New true
2746 694 Acked 1693 Write 7 Done Done New true
2747 695 Acked 1694 Read 10 Done Done New true
2748 696 Acked 1695 Read 5 Done Done New true
2749 697 Acked 1696 Read 7 Done Done New true
2750 698 Acked 1697 Read 8 Done Done New true
2751 699 Acked 1698 Read 1 Done Done New true
2752 700 Acked 1699 Write 3 Done Done New true
2753 701 Acked 1700 Read 7 Done Done New true
2754 702 Acked 1701 Read 8 Done Done New true
2755 703 Acked 1702 Read 7 Done Done New true
2756 704 Acked 1703 Read 5 Done Done New true
2757 705 Acked 1704 Read 2 Done Done New true
2758 706 Acked 1705 Read 10 Done Done New true
2759 707 Acked 1706 Read 6 Done Done New true
2760 708 Acked 1707 Read 7 Done Done New true
2761 709 Acked 1708 Write 9 Done Done New true
2762 710 Acked 1709 Read 6 Done Done New true
2763 711 Acked 1710 Read 4 Done Done New true
2764 712 Acked 1711 Write 7 Done Done New true
2765 713 Acked 1712 Write 3 Done Done New true
2766 714 Acked 1713 Read 5 Done Done New true
2767 715 Acked 1714 Read 8 Done Done New true
2768 716 Acked 1715 Write 10 Done Done New true
2769 717 Acked 1716 Write 3 Done Done New true
2770 718 Acked 1717 Write 6 Done Done New true
2771 719 Acked 1718 Read 4 Done Done New true
2772 720 Acked 1719 Read 6 Done Done New true
2773 721 Acked 1720 Write 5 Done Done New true
2774 722 Acked 1721 Read 6 Done Done New true
2775 723 Acked 1722 Write 9 Done Done New true
2776 724 Acked 1723 Read 1 Done Done New true
2777 725 Acked 1724 Write 1 Done Done New true
2778 726 Acked 1725 Read 2 Done Done New true
2779 727 Acked 1726 Read 10 Done Done New true
2780 728 Acked 1727 Read 9 Done Done New true
2781 729 Acked 1728 Write 2 Done Done New true
2782 730 Acked 1729 Read 3 Done Done New true
2783 731 Acked 1730 Flush 0 Done Done New true
2784 732 Acked 1731 Write 7 Done Done New true
2785 733 Acked 1732 Write 1 Done Done New true
2786 734 Acked 1733 Write 9 Done Done New true
2787 735 Acked 1734 Write 3 Done Done New true
2788 736 Acked 1735 Write 9 Done Done New true
2789 737 Acked 1736 Read 9 Done Done New true
2790 738 Acked 1737 Write 8 Done Done New true
2791 739 Acked 1738 Write 9 Done Done New true
2792 740 Acked 1739 Write 8 Done Done New true
2793 741 Acked 1740 Write 1 Done Done New true
2794 742 Acked 1741 Read 1 Done Done New true
2795 743 Acked 1742 Read 8 Done Done New true
2796 744 Acked 1743 Write 10 Done Done New true
2797 745 Acked 1744 Write 4 Done Done New true
2798 746 Acked 1745 Write 3 Done Done New true
2799 747 Acked 1746 Write 1 Done Done New true
2800 748 Acked 1747 Read 10 Done Done New true
2801 749 Acked 1748 Read 4 Done Done New true
2802 750 Acked 1749 Write 5 Done Done New true
2803 751 Acked 1750 Write 1 Done Done New true
2804 752 Acked 1751 Flush 0 Done Done New true
2805 753 Acked 1752 Write 4 Done Done New true
2806 754 Acked 1753 Read 6 Done Done New true
2807 755 Acked 1754 Read 7 Done Done New true
2808 756 Acked 1755 Read 1 Done Done New true
2809 757 Acked 1756 Read 10 Done Done New true
2810 758 Acked 1757 Write 2 Done Done New true
2811 759 Acked 1758 Read 5 Done Done New true
2812 760 Acked 1759 Write 10 Done Done New true
2813 761 Acked 1760 Write 8 Done Done New true
2814 762 Acked 1761 Write 8 Done Done New true
2815 763 Acked 1762 Read 5 Done Done New true
2816 764 Acked 1763 Read 3 Done Done New true
2817 765 Acked 1764 Write 1 Done Done New true
2818 766 Acked 1765 Read 6 Done Done New true
2819 767 Acked 1766 Read 2 Done Done New true
2820 768 Acked 1767 Write 7 Done Done New true
2821 769 Acked 1768 Read 6 Done Done New true
2822 770 Acked 1769 Read 2 Done Done New true
2823 771 Acked 1770 Flush 0 Done Done New true
2824 772 Acked 1771 Flush 0 Done Done New true
2825 773 Acked 1772 Read 3 Done Done New true
2826 774 Acked 1773 Read 10 Done Done New true
2827 775 Acked 1774 Write 9 Done Done New true
2828 776 Acked 1775 Read 7 Done Done New true
2829 777 Acked 1776 Read 1 Done Done New true
2830 778 Acked 1777 Read 1 Done Done New true
2831 779 Acked 1778 Flush 0 Done Done New true
2832 780 Acked 1779 Write 2 Done Done New true
2833 781 Acked 1780 Write 8 Done Done New true
2834 782 Acked 1781 Read 8 Done Done New true
2835 783 Acked 1782 Write 8 Done Done New true
2836 784 Acked 1783 Flush 0 Done Done New true
2837 785 Acked 1784 Write 8 Done Done New true
2838 786 Acked 1785 Write 9 Done Done New true
2839 787 Acked 1786 Write 1 Done Done New true
2840 788 Acked 1787 Flush 0 Done Done New true
2841 789 Acked 1788 Read 4 Done Done New true
2842 790 Acked 1789 Read 8 Done Done New true
2843 791 Acked 1790 Write 1 Done Done New true
2844 792 Acked 1791 Read 1 Done Done New true
2845 793 Acked 1792 Write 5 Done Done New true
2846 794 Acked 1793 Write 4 Done Done New true
2847 795 Acked 1794 Read 6 Done Done New true
2848 796 Acked 1795 Write 10 Done Done New true
2849 797 Acked 1796 Read 1 Done Done New true
2850 798 Acked 1797 Write 8 Done Done New true
2851 799 Acked 1798 Read 4 Done Done New true
2852 800 Acked 1799 Flush 0 Done Done New true
2853 801 Acked 1800 Write 9 Done Done New true
2854 802 Acked 1801 Write 1 Done Done New true
2855 803 Acked 1802 Read 2 Done Done New true
2856 804 Acked 1803 Read 2 Done Done New true
2857 805 Acked 1804 Read 3 Done Done New true
2858 806 Acked 1805 Write 1 Done Done New true
2859 807 Acked 1806 Read 4 Done Done New true
2860 808 Acked 1807 Read 2 Done Done New true
2861 809 Acked 1808 Flush 0 Done Done New true
2862 810 Acked 1809 Read 2 Done Done New true
2863 811 Acked 1810 Write 3 Done Done New true
2864 812 Acked 1811 Read 3 Done Done New true
2865 813 Acked 1812 Write 3 Done Done New true
2866 814 Acked 1813 Write 7 Done Done New true
2867 815 Acked 1814 Read 8 Done Done New true
2868 816 Acked 1815 Write 2 Done Done New true
2869 817 Acked 1816 Write 8 Done Done New true
2870 818 Acked 1817 Read 8 Done Done New true
2871 819 Acked 1818 Read 10 Done Done New true
2872 820 Acked 1819 Write 3 Done Done New true
2873 821 Acked 1820 Read 6 Done Done New true
2874 822 Acked 1821 Read 8 Done Done New true
2875 823 Acked 1822 Write 4 Done Done New true
2876 824 Acked 1823 Read 9 Done Done New true
2877 825 Acked 1824 Flush 0 Done Done New true
2878 826 Acked 1825 Write 7 Done Done New true
2879 827 Acked 1826 Read 8 Done Done New true
2880 828 Acked 1827 Write 8 Done Done New true
2881 829 Acked 1828 Write 3 Done Done New true
2882 830 Acked 1829 Write 6 Done Done New true
2883 831 Acked 1830 Flush 0 Done Done New true
2884 832 Acked 1831 Write 2 Done Done New true
2885 833 Acked 1832 Read 6 Done Done New true
2886 834 Acked 1833 Write 3 Done Done New true
2887 835 Acked 1834 Read 1 Done Done New true
2888 836 Acked 1835 Read 9 Done Done New true
2889 837 Acked 1836 Read 4 Done Done New true
2890 838 Acked 1837 Flush 0 Done Done New true
2891 839 Acked 1838 Write 4 Done Done New true
2892 840 Acked 1839 Write 8 Done Done New true
2893 841 Acked 1840 Read 1 Done Done New true
2894 842 Acked 1841 Read 3 Done Done New true
2895 843 Acked 1842 Write 5 Done Done New true
2896 844 Acked 1843 Read 8 Done Done New true
2897 845 Acked 1844 Write 10 Done Done New true
2898 846 Acked 1845 Write 7 Done Done New true
2899 847 Acked 1846 Write 7 Done Done New true
2900 848 Acked 1847 Write 9 Done Done New true
2901 849 Acked 1848 Read 8 Done Done New true
2902 850 Acked 1849 Flush 0 Done Done New true
2903 851 Acked 1850 Write 2 Done Done New true
2904 852 Acked 1851 Read 9 Done Done New true
2905 853 Acked 1852 Write 10 Done Done New true
2906 854 Acked 1853 Read 1 Done Done New true
2907 855 Acked 1854 Write 3 Done Done New true
2908 856 Acked 1855 Write 9 Done Done New true
2909 857 Acked 1856 Write 2 Done Done New true
2910 858 Acked 1857 Write 8 Done Done New true
2911 859 Acked 1858 Flush 0 Done Done New true
2912 860 Acked 1859 Read 3 Done Done New true
2913 861 Acked 1860 Read 1 Done Done New true
2914 862 Acked 1861 Write 2 Done Done New true
2915 863 Acked 1862 Write 1 Done Done New true
2916 864 Acked 1863 Read 4 Done Done New true
2917 865 Acked 1864 Flush 0 Done Done New true
2918 866 Acked 1865 Write 4 Done Done New true
2919 867 Acked 1866 Write 10 Done Done New true
2920 868 Acked 1867 Write 7 Done Done New true
2921 869 Acked 1868 Write 3 Done Done New true
2922 870 Acked 1869 Write 1 Done Done New true
2923 871 Acked 1870 Read 4 Done Done New true
2924 872 Acked 1871 Write 8 Done Done New true
2925 873 Acked 1872 Read 7 Done Done New true
2926 874 Acked 1873 Read 2 Done Done New true
2927 875 Acked 1874 Write 8 Done Done New true
2928 876 Acked 1875 Write 3 Done Done New true
2929 877 Acked 1876 Read 2 Done Done New true
2930 878 Acked 1877 Read 3 Done Done New true
2931 879 Acked 1878 Read 9 Done Done New true
2932 880 Acked 1879 Read 10 Done Done New true
2933 881 Acked 1880 Read 6 Done Done New true
2934 882 Acked 1881 Read 9 Done Done New true
2935 883 Acked 1882 Read 9 Done Done New true
2936 884 Acked 1883 Flush 0 Done Done New true
2937 885 Acked 1884 Write 1 Done Done New true
2938 886 Acked 1885 Read 2 Done Done New true
2939 887 Acked 1886 Flush 0 Done Done New true
2940 888 Acked 1887 Read 10 Done Done New true
2941 889 Acked 1888 Write 2 Done Done New true
2942 890 Acked 1889 Write 6 Done Done New true
2943 891 Acked 1890 Write 10 Done Done New true
2944 892 Acked 1891 Write 1 Done Done New true
2945 893 Acked 1892 Read 7 Done Done New true
2946 894 Acked 1893 Write 6 Done Done New true
2947 895 Acked 1894 Write 5 Done Done New true
2948 896 Acked 1895 Read 5 Done Done New true
2949 897 Acked 1896 Write 6 Done Done New true
2950 898 Acked 1897 Write 8 Done Done New true
2951 899 Acked 1898 Write 8 Done Done New true
2952 900 Acked 1899 Flush 0 Done Done New true
2953 901 Acked 1900 Write 9 Done Done New true
2954 902 Acked 1901 Write 3 Done Done New true
2955 903 Acked 1902 Flush 0 Done Done New true
2956 STATES DS:0 DS:1 DS:2 TOTAL
2957 New 0 0 301 301
2958 Sent 0 0 0 0
2959 Done 301 301 0 602
2960 Skipped 0 0 0 0
2961 Error 0 0 0 0
2962 Last Flush: 1902 1902 1601
2963 Downstairs last five completed: 1601 1600 1599 1598 1597
2964 Upstairs last five completed: 903 902 901 900 899
2965 CLIENT: Up:0 ds:301 act:2
29662024-02-27T21:45:11.704ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
29672024-02-27T21:45:11.704ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
29682024-02-27T21:45:11.706ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
29692024-02-27T21:45:11.706ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
29702024-02-27T21:45:11.706ZINFOcrucible: send last flush ID to this DS: 1601 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
29712024-02-27T21:45:11.706ZINFOcrucible: Replied this last flush ID: 1601 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
29722024-02-27T21:45:11.706ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
29732024-02-27T21:45:11.706ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
29742024-02-27T21:45:11.706ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
29752024-02-27T21:45:11.706ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
2976 ----------------------------------------------------------------
2977 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
2978 Downstairs last five completed: 1902 1901 1900 1899 1898
2979 Upstairs last five completed: 903 902 901 900 899
2980 CLIENT: Up:0 ds:0 act:3
2981 Replay: All jobs finished, all DS active.
2982 Main task has work to do, go find it
2983 got dsc Stop(1)
2984 stop 1
2985 Feb 27 21:45:15.035 INFO request completed, latency_us: 364, [8820] Got stop action so:false kr:false
2986 response_code: 204, uri: /stop/cid/1, method: POST, req_id: 6d2c9713-e749-4c36-99ea-f35e4270af80, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
29872024-02-27T21:45:15.036ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
29882024-02-27T21:45:15.036ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
29892024-02-27T21:45:15.036ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = e6a5a974-925a-494f-8845-facc441c2690
2990 Feb 27 21:45:15.036 {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:15.036404079Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"client":"1","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
2991 INFO request completed, latency_us: 141, response_code: 200, uri: /state/cid/1, method: GET, req_id: {"msg":"[1] client re-new 0 jobs since flush 1902","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:15.036434325Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"downstairs","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
2992 99714ef9-aca2-4ac0-9f8c-30cb08c5b24a, remote_addr: 127.0.0.1:44347, local_addr{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:15.036460042Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"io task","client":"1","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
2993 : 127.0.0.1:9998
2994 [8820] Exited with: ExitStatus(unix_wait_status(9))
2995 [8820][1] reports Exit
29962024-02-27T21:45:16.037ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
29972024-02-27T21:45:16.037ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
29982024-02-27T21:45:16.037ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
29992024-02-27T21:45:16.037ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
30002024-02-27T21:45:16.037ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
30012024-02-27T21:45:16.037ZINFOcrucible: [1] client re-new 0 jobs since flush 1902 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
30022024-02-27T21:45:16.037ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
30032024-02-27T21:45:17.039ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
30042024-02-27T21:45:17.039ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
30052024-02-27T21:45:17.039ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
30062024-02-27T21:45:17.039ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
30072024-02-27T21:45:17.039ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
30082024-02-27T21:45:17.039ZINFOcrucible: [1] client re-new 0 jobs since flush 1902 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
30092024-02-27T21:45:17.039ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
30102024-02-27T21:45:18.041ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
30112024-02-27T21:45:18.041ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
30122024-02-27T21:45:18.041ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
30132024-02-27T21:45:18.041ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
30142024-02-27T21:45:18.041ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
30152024-02-27T21:45:18.041ZINFOcrucible: [1] client re-new 0 jobs since flush 1902 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
30162024-02-27T21:45:18.041ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
3017 Feb 27 21:45:19.037 INFO request completed, latency_us: 206, response_code: 200, uri: /state/cid/1, method: GET, req_id: b5422d0d-9258-4c4d-8699-6a0239264a62, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
3018 001/300 Write block 1876 len 4096 data: 3
3019 002/300 Read block 4235 len 40960
3020 003/300 Read block 760 len 4096
3021 004/300 Flush
30222024-02-27T21:45:19.042ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
30232024-02-27T21:45:19.042ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
30242024-02-27T21:45:19.042ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
30252024-02-27T21:45:19.042ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
30262024-02-27T21:45:19.042ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
30272024-02-27T21:45:19.042ZINFOcrucible: [1] client re-new 4 jobs since flush 1902 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
30282024-02-27T21:45:19.042ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
3029 005/300 Read block 2075 len 32768
3030 006/300 Read block 651 len 36864
3031 007/300 Write block 3957 len 12288 data: 3 3 3
3032 008/300 Read block 3947 len 40960
3033 009/300 Read block 3424 len 40960
3034 010/300 Flush
3035 011/300 Read block 3943 len 24576
3036 012/300 Write block 2223 len 4096 data: 3
3037 013/300 Write block 4738 len 40960 data: 2 2 2 2 2 2 3 4 4 4
3038 014/300 Write block 4116 len 36864 data: 2 2 2 2 2 2 2 2 2
3039 015/300 Write block 1662 len 36864 data: 3 3 4 2 2 3 3 3 3
3040 016/300 Read block 1074 len 24576
3041 017/300 Write block 407 len 36864 data: 3 3 3 3 3 3 4 5 4
3042 018/300 Write block 3010 len 36864 data: 2 2 2 2 2 3 4 4 4
3043 019/300 Read block 1312 len 20480
3044 020/300 Read block 4358 len 24576
3045 021/300 Write block 907 len 36864 data: 2 3 3 3 3 3 3 3 3
3046 022/300 Write block 1943 len 8192 data: 2 2
3047 023/300 Read block 2378 len 4096
3048 024/300 Write block 3833 len 24576 data: 2 2 2 2 2 2
3049 025/300 Read block 4125 len 8192
3050 026/300 Read block 4431 len 24576
3051 027/300 Write block 243 len 12288 data: 2 2 2
3052 028/300 Read block 3142 len 20480
3053 029/300 Read block 3050 len 20480
3054 030/300 Write block 3345 len 24576 data: 3 3 3 2 2 2
3055 031/300 Read block 3598 len 28672
3056 032/300 Read block 4329 len 32768
3057 033/300 Write block 1189 len 4096 data: 2
3058 034/300 Read block 4210 len 40960
3059 035/300 Write block 1615 len 32768 data: 2 2 2 2 2 2 2 2
3060 036/300 Read block 4148 len 36864
3061 037/300 Read block 2364 len 40960
3062 038/300 Read block 4984 len 24576
3063 039/300 Read block 3223 len 16384
3064 040/300 Write block 4619 len 24576 data: 2 2 2 3 3 3
3065 041/300 Write block 1911 len 28672 data: 3 3 3 3 2 2 2
3066 042/300 Read block 1850 len 16384
3067 043/300 Write block 3933 len 4096 data: 3
3068 044/300 Read block 4265 len 28672
3069 045/300 Write block 3819 len 8192 data: 3 3
3070 046/300 Flush
3071 047/300 Read block 3097 len 8192
3072 048/300 Read block 13 len 40960
3073 049/300 Read block 3881 len 4096
3074 050/300 Read block 2098 len 16384
3075 051/300 Flush
3076 052/300 Write block 3963 len 36864 data: 2 3 3 2 2 2 2 2 3
3077 053/300 Write block 3873 len 32768 data: 2 2 2 2 2 2 2 2
3078 054/300 Write block 2452 len 12288 data: 3 3 3
3079 055/300 Write block 4665 len 32768 data: 2 2 2 2 2 2 2 2
3080 056/300 Read block 4499 len 28672
3081 057/300 Write block 121 len 8192 data: 4 4
3082 058/300 Read block 1195 len 28672
3083 059/300 Read block 1356 len 28672
3084 060/300 Write block 4018 len 16384 data: 2 3 2 3
3085 061/300 Read block 4809 len 16384
3086 062/300 Write block 2045 len 28672 data: 2 2 2 3 3 3 3
3087 063/300 Read block 2962 len 12288
3088 064/300 Read block 3258 len 20480
3089 065/300 Read block 2611 len 36864
3090 066/300 Read block 4929 len 12288
3091 067/300 Write block 3692 len 28672 data: 2 2 2 2 2 2 2
3092 068/300 Read block 3818 len 20480
3093 069/300 Read block 3323 len 32768
3094 070/300 Read block 1989 len 12288
3095 071/300 Write block 4802 len 28672 data: 2 2 2 2 2 2 2
3096 072/300 Read block 2877 len 4096
3097 073/300 Read block 3744 len 16384
3098 074/300 Read block 4955 len 16384
3099 075/300 Flush
3100 076/300 Write block 2485 len 32768 data: 3 3 3 3 3 3 3 3
3101 077/300 Flush
3102 078/300 Read block 1230 len 20480
3103 079/300 Flush
3104 080/300 Write block 2818 len 8192 data: 2 2
3105 081/300 Read block 1563 len 36864
3106 082/300 Read block 4718 len 24576
3107 083/300 Read block 3924 len 4096
3108 084/300 Write block 3652 len 16384 data: 2 2 2 2
3109 085/300 Read block 2565 len 16384
3110 086/300 Write block 4744 len 28672 data: 4 5 5 5 5 6 5
3111 087/300 Read block 4907 len 20480
3112 088/300 Read block 3594 len 8192
3113 089/300 Write block 950 len 28672 data: 3 3 2 2 2 2 2
3114 090/300 Read block 4220 len 16384
3115 091/300 Write block 1049 len 36864 data: 2 2 3 3 2 2 2 2 2
3116 092/300 Read block 1352 len 36864
3117 093/300 Read block 1349 len 36864
3118 094/300 Write block 674 len 20480 data: 2 2 2 2 3
3119 095/300 Write block 2334 len 24576 data: 3 3 3 3 2 2
3120 096/300 Write block 170 len 16384 data: 2 2 3 2
3121 097/300 Read block 3179 len 20480
3122 098/300 Write block 1975 len 32768 data: 3 3 3 2 2 2 2 2
3123 099/300 Read block 1523 len 16384
3124 100/300 Write block 1926 len 24576 data: 2 2 2 2 2 2
3125 101/300 Write block 1599 len 8192 data: 2 2
3126 102/300 Read block 1908 len 28672
3127 103/300 Read block 2865 len 24576
3128 104/300 Read block 3774 len 32768
3129 105/300 Read block 3425 len 4096
3130 106/300 Flush
3131 107/300 Read block 4550 len 8192
3132 108/300 Write block 108 len 32768 data: 3 3 2 2 2 2 3 3
3133 109/300 Read block 1219 len 36864
3134 110/300 Write block 3162 len 16384 data: 2 2 2 2
3135 111/300 Flush
3136 112/300 Write block 4127 len 28672 data: 2 2 2 2 2 2 3
3137 113/300 Read block 4871 len 12288
3138 114/300 Read block 2598 len 32768
3139 115/300 Read block 4812 len 8192
3140 116/300 Flush
3141 117/300 Read block 3324 len 24576
3142 118/300 Flush
3143 119/300 Write block 4769 len 20480 data: 3 3 3 3 3
3144 120/300 Read block 3620 len 16384
3145 121/300 Write block 1792 len 20480 data: 3 3 3 3 2
3146 122/300 Read block 3097 len 20480
3147 123/300 Read block 916 len 12288
3148 124/300 Flush
3149 125/300 Read block 498 len 12288
3150 126/300 Read block 4034 len 40960
3151 127/300 Write block 449 len 16384 data: 2 2 2 2
3152 128/300 Read block 1701 len 4096
3153 129/300 Write block 3839 len 36864 data: 2 2 2 2 2 2 2 2 2
3154 130/300 Write block 4549 len 16384 data: 2 2 2 2
3155 131/300 Write block 4120 len 24576 data: 3 3 3 3 3 2
3156 132/300 Write block 4733 len 24576 data: 2 2 2 2 2 3
3157 133/300 Read block 4001 len 8192
3158 134/300 Read block 3035 len 16384
3159 135/300 Read block 3730 len 20480
3160 136/300 Flush
3161 137/300 Write block 4682 len 32768 data: 3 3 3 2 2 2 2 2
3162 138/300 Flush
3163 139/300 Read block 1746 len 36864
3164 140/300 Write block 4196 len 20480 data: 3 3 3 3 3
3165 141/300 Write block 716 len 4096 data: 2
3166 142/300 Read block 4379 len 16384
3167 143/300 Read block 1957 len 4096
3168 144/300 Read block 4017 len 40960
3169 145/300 Read block 3467 len 8192
3170 146/300 Read block 4758 len 4096
3171 147/300 Read block 1952 len 32768
3172 148/300 Read block 3294 len 16384
3173 149/300 Write block 4396 len 28672 data: 2 2 2 2 2 2 2
3174 150/300 Read block 3762 len 20480
3175 151/300 Write block 3028 len 12288 data: 2 2 2
3176 152/300 Write block 2193 len 4096 data: 3
3177 153/300 Read block 3173 len 32768
3178 154/300 Read block 4259 len 12288
3179 155/300 Read block 1944 len 16384
3180 156/300 Read block 2248 len 32768
3181 157/300 Read block 2362 len 20480
3182 158/300 Read block 1849 len 12288
3183 159/300 Flush
3184 160/300 Write block 2692 len 24576 data: 2 2 2 2 2 2
3185 161/300 Read block 4895 len 8192
3186 162/300 Write block 2724 len 12288 data: 2 2 2
3187 163/300 Flush
3188 164/300 Read block 2898 len 12288
3189 165/300 Read block 521 len 20480
3190 166/300 Write block 3813 len 16384 data: 3 3 3 3
3191 167/300 Read block 4854 len 40960
3192 168/300 Write block 3043 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3193 169/300 Read block 2710 len 20480
3194 170/300 Write block 1615 len 32768 data: 3 3 3 3 3 3 3 3
3195 171/300 Read block 1904 len 4096
3196 172/300 Read block 4714 len 36864
3197 173/300 Read block 2784 len 4096
3198 174/300 Read block 3540 len 28672
3199 175/300 Read block 3310 len 36864
3200 176/300 Write block 869 len 40960 data: 2 2 2 2 2 2 2 3 3 2
3201 177/300 Flush
3202 178/300 Write block 1302 len 40960 data: 2 2 2 2 2 3 3 3 3 3
3203 179/300 Write block 555 len 16384 data: 2 2 2 2
3204 180/300 Read block 2416 len 20480
3205 181/300 Read block 2205 len 8192
3206 182/300 Read block 1212 len 12288
3207 183/300 Write block 685 len 20480 data: 2 2 3 2 2
3208 184/300 Read block 4974 len 8192
3209 185/300 Write block 1157 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3210 186/300 Read block 387 len 16384
3211 187/300 Read block 3052 len 16384
3212 188/300 Read block 2826 len 12288
3213 189/300 Write block 4941 len 20480 data: 3 3 3 3 4
3214 190/300 Write block 1398 len 20480 data: 2 2 2 2 2
3215 191/300 Read block 3684 len 8192
3216 192/300 Write block 1789 len 24576 data: 2 2 2 4 4 4
3217 193/300 Read block 4758 len 12288
3218 194/300 Write block 1307 len 20480 data: 4 4 4 4 4
3219 195/300 Read block 4740 len 4096
3220 196/300 Write block 2072 len 20480 data: 2 2 2 3 3
3221 197/300 Write block 1802 len 16384 data: 2 2 2 2
3222 198/300 Flush
3223 199/300 Write block 3825 len 4096 data: 2
3224 200/300 Read block 2653 len 40960
3225 201/300 Write block 1613 len 20480 data: 2 2 4 4 4
3226 202/300 Flush
3227 203/300 Read block 1466 len 32768
3228 204/300 Write block 4284 len 4096 data: 2
3229 205/300 Read block 1032 len 24576
3230 206/300 Write block 3036 len 40960 data: 2 2 2 2 2 2 2 3 3 3
3231 207/300 Read block 3924 len 36864
3232 208/300 Read block 2625 len 28672
3233 209/300 Read block 837 len 28672
3234 210/300 Read block 1128 len 32768
3235 211/300 Flush
3236 212/300 Read block 585 len 16384
3237 213/300 Write block 1167 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3238 214/300 Write block 1083 len 40960 data: 3 3 3 2 2 2 2 3 4 4
3239 215/300 Write block 3153 len 28672 data: 2 2 2 2 2 2 2
3240 216/300 Read block 4603 len 24576
3241 217/300 Write block 4262 len 40960 data: 3 3 3 3 3 3 2 2 2 2
3242 218/300 Read block 708 len 40960
3243 219/300 Read block 190 len 16384
3244 220/300 Read block 2777 len 12288
3245 221/300 Read block 1075 len 40960
3246 222/300 Write block 1433 len 4096 data: 2
3247 223/300 Write block 3555 len 36864 data: 3 3 2 2 2 2 2 3 3
3248 224/300 Read block 1644 len 40960
3249 225/300 Write block 4931 len 16384 data: 2 2 2 2
3250 226/300 Write block 1200 len 20480 data: 2 2 2 2 2
3251 227/300 Read block 2309 len 16384
3252 228/300 Read block 1778 len 16384
3253 229/300 Write block 731 len 36864 data: 3 3 3 3 2 2 2 2 2
3254 230/300 Read block 4600 len 20480
3255 231/300 Read block 992 len 8192
3256 232/300 Write block 841 len 24576 data: 2 2 2 2 2 2
3257 233/300 Read block 4708 len 24576
3258 234/300 Read block 4904 len 24576
3259 235/300 Read block 2769 len 8192
3260 236/300 Read block 1213 len 20480
3261 237/300 Write block 1989 len 36864 data: 2 2 2 2 2 2 2 2 2
3262 238/300 Write block 4824 len 28672 data: 4 4 3 3 3 3 2
3263 239/300 Read block 743 len 40960
3264 240/300 Write block 1767 len 16384 data: 2 2 2 2
3265 241/300 Flush
3266 242/300 Read block 2190 len 28672
3267 243/300 Write block 2250 len 4096 data: 2
3268 244/300 Write block 3980 len 32768 data: 2 2 2 2 2 2 2 2
3269 245/300 Write block 3145 len 16384 data: 2 2 2 2
3270 246/300 Write block 4010 len 20480 data: 2 2 2 2 2
3271 247/300 Read block 2065 len 28672
3272 248/300 Flush
3273 249/300 Write block 1770 len 12288 data: 3 2 2
3274 250/300 Flush
3275 251/300 Read block 4927 len 20480
3276 252/300 Read block 102 len 24576
3277 253/300 Write block 2668 len 20480 data: 2 2 2 2 2
3278 254/300 Read block 2697 len 8192
3279 255/300 Read block 3089 len 20480
3280 256/300 Read block 3454 len 16384
3281 257/300 Read block 3718 len 16384
3282 258/300 Read block 4441 len 16384
3283 259/300 Write block 1945 len 8192 data: 2 2
3284 260/300 Read block 834 len 32768
3285 261/300 Read block 2193 len 36864
3286 262/300 Read block 2220 len 4096
3287 263/300 Read block 2446 len 36864
3288 264/300 Read block 922 len 28672
3289 265/300 Read block 743 len 20480
3290 266/300 Write block 4197 len 8192 data: 4 4
3291 267/300 Read block 2066 len 4096
3292 268/300 Write block 3316 len 12288 data: 3 3 3
3293 269/300 Write block 4490 len 4096 data: 4
3294 270/300 Flush
3295 271/300 Read block 926 len 36864
3296 272/300 Read block 3996 len 12288
3297 273/300 Read block 2792 len 8192
3298 274/300 Read block 1923 len 32768
3299 275/300 Read block 4900 len 20480
3300 276/300 Write block 3263 len 16384 data: 2 2 3 3
3301 277/300 Write block 1376 len 16384 data: 2 2 3 3
3302 278/300 Flush
3303 279/300 Read block 1647 len 16384
3304 280/300 Write block 2504 len 4096 data: 3
3305 281/300 Write block 644 len 12288 data: 2 2 2
3306 282/300 Flush
3307 283/300 Read block 1336 len 32768
3308 284/300 Read block 1942 len 20480
3309 285/300 Read block 886 len 40960
3310 286/300 Flush
3311 287/300 Read block 2834 len 24576
3312 288/300 Write block 4596 len 16384 data: 2 2 2 2
3313 289/300 Write block 1653 len 32768 data: 3 2 3 3 4 4 4 4
3314 290/300 Write block 4822 len 24576 data: 2 4 5 5 4 4
3315 291/300 Write block 2644 len 32768 data: 2 2 3 3 3 3 2 2
3316 292/300 Read block 658 len 8192
3317 293/300 Write block 3448 len 40960 data: 2 2 2 2 2 2 2 2 2 3
3318 294/300 Read block 47 len 20480
3319 295/300 Read block 3480 len 40960
3320 296/300 Write block 2440 len 24576 data: 2 2 2 2 2 2
3321 297/300 Read block 3746 len 36864
3322 298/300 Write block 2518 len 24576 data: 3 3 3 3 3 2
3323 299/300 Read block 3266 len 20480
3324 300/300 Read block 1974 len 36864
3325 Main task has work to do, go find it
3326 got dsc Start(1)
3327 start 1
3328 Feb 27 21:45:19.382 INFO request completed, latency_us: 214, response_code: 204, uri: /start/cid/1, method: POST, req_id: 90ed1348-4f7f-4257-84f6-6d34f4bc694b, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
3329 Replay: started 1, returned:Ok(())
3330 ----------------------------------------------------------------
3331 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
3332 States: Active Offline Active
3333 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
3334 904 Acked 1903 Write 1 Done New Done true
3335 905 Acked 1904 Read 10 Done New Done true
3336 906 Acked 1905 Read 1 Done New Done true
3337 907 Acked 1906 Flush 0 Done New Done true
3338 908 Acked 1907 Read 8 Done New Done false
3339 909 Acked 1908 Read 9 Done New Done false
3340 910 Acked 1909 Write 3 Done New Done false
3341 911 Acked 1910 Read 10 Done New Done false
3342 912 Acked 1911 Read 10 Done New Done false
3343 913 Acked 1912 Flush 0 Done New Done false
3344 914 Acked 1913 Read 6 Done New Done false
3345 915 Acked 1914 Write 1 Done New Done false
3346 916 Acked 1915 Write 10 Done New Done false
3347 917 Acked 1916 Write 9 Done New Done false
3348 918 Acked 1917 Write 9 Done New Done false
3349 919 Acked 1918 Read 6 Done New Done false
3350 920 Acked 1919 Write 9 Done New Done false
3351 921 Acked 1920 Write 9 Done New Done false
3352 922 Acked 1921 Read 5 Done New Done false
3353 923 Acked 1922 Read 6 Done New Done false
3354 924 Acked 1923 Write 9 Done New Done false
3355 925 Acked 1924 Write 2 Done New Done false
3356 926 Acked 1925 Read 1 Done New Done false
3357 927 Acked 1926 Write 6 Done New Done false
3358 928 Acked 1927 Read 2 Done New Done false
3359 929 Acked 1928 Read 6 Done New Done false
3360 930 Acked 1929 Write 3 Done New Done false
3361 931 Acked 1930 Read 5 Done New Done false
3362 932 Acked 1931 Read 5 Done New Done false
3363 933 Acked 1932 Write 6 Done New Done false
3364 934 Acked 1933 Read 7 Done New Done false
3365 935 Acked 1934 Read 8 Done New Done false
3366 936 Acked 1935 Write 1 Done New Done false
3367 937 Acked 1936 Read 10 Done New Done false
3368 938 Acked 1937 Write 8 Done New Done false
3369 939 Acked 1938 Read 9 Done New Done false
3370 940 Acked 1939 Read 10 Done New Done false
3371 941 Acked 1940 Read 6 Done New Done false
3372 942 Acked 1941 Read 4 Done New Done false
3373 943 Acked 1942 Write 6 Done New Done false
3374 944 Acked 1943 Write 7 Done New Done false
3375 945 Acked 1944 Read 4 Done New Done false
3376 946 Acked 1945 Write 1 Done New Done false
3377 947 Acked 1946 Read 7 Done New Done false
3378 948 Acked 1947 Write 2 Done New Done false
3379 949 Acked 1948 Flush 0 Done New Done false
3380 950 Acked 1949 Read 2 Done New Done false
3381 951 Acked 1950 Read 10 Done New Done false
3382 952 Acked 1951 Read 1 Done New Done false
3383 953 Acked 1952 Read 4 Done New Done false
3384 954 Acked 1953 Flush 0 Done New Done false
3385 955 Acked 1954 Write 9 Done New Done false
3386 956 Acked 1955 Write 8 Done New Done false
3387 957 Acked 1956 Write 3 Done New Done false
3388 958 Acked 1957 Write 8 Done New Done false
3389 959 Acked 1958 Read 7 Done New Done false
3390 960 Acked 1959 Write 2 Done New Done false
3391 961 Acked 1960 Read 7 Done New Done false
3392 962 Acked 1961 Read 7 Done New Done false
3393 963 Acked 1962 Write 4 Done New Done false
3394 964 Acked 1963 Read 4 Done New Done false
3395 965 Acked 1964 Write 7 Done New Done false
3396 966 Acked 1965 Read 3 Done New Done false
3397 967 Acked 1966 Read 5 Done New Done false
3398 968 Acked 1967 Read 9 Done New Done false
3399 969 Acked 1968 Read 3 Done New Done false
3400 970 Acked 1969 Write 7 Done New Done false
3401 971 Acked 1970 Read 5 Done New Done false
3402 972 Acked 1971 Read 8 Done New Done false
3403 973 Acked 1972 Read 3 Done New Done false
3404 974 Acked 1973 Write 7 Done New Done false
3405 975 Acked 1974 Read 1 Done New Done false
3406 976 Acked 1975 Read 4 Done New Done false
3407 977 Acked 1976 Read 4 Done New Done false
3408 978 Acked 1977 Flush 0 Done New Done false
3409 979 Acked 1978 Write 8 Done New Done false
3410 980 Acked 1979 Flush 0 Done New Done false
3411 981 Acked 1980 Read 5 Done New Done false
3412 982 Acked 1981 Flush 0 Done New Done false
3413 983 Acked 1982 Write 2 Done New Done false
3414 984 Acked 1983 Read 9 Done New Done false
3415 985 Acked 1984 Read 6 Done New Done false
3416 986 Acked 1985 Read 1 Done New Done false
3417 987 Acked 1986 Write 4 Done New Done false
3418 988 Acked 1987 Read 4 Done New Done false
3419 989 Acked 1988 Write 7 Done New Done false
3420 990 Acked 1989 Read 5 Done New Done false
3421 991 Acked 1990 Read 2 Done New Done false
3422 992 Acked 1991 Write 7 Done New Done false
3423 993 Acked 1992 Read 4 Done New Done false
3424 994 Acked 1993 Write 9 Done New Done false
3425 995 Acked 1994 Read 9 Done New Done false
3426 996 Acked 1995 Read 9 Done New Done false
3427 997 Acked 1996 Write 5 Done New Done false
3428 998 Acked 1997 Write 6 Done New Done false
3429 999 Acked 1998 Write 4 Done New Done false
3430 1000 Acked 1999 Read 5 Done New Done false
3431 1001 Acked 2000 Write 8 Done New Done false
3432 1002 Acked 2001 Read 4 Done New Done false
3433 1003 Acked 2002 Write 6 Done New Done false
3434 1004 Acked 2003 Write 2 Done New Done false
3435 1005 Acked 2004 Read 7 Done New Done false
3436 1006 Acked 2005 Read 6 Done New Done false
3437 1007 Acked 2006 Read 8 Done New Done false
3438 1008 Acked 2007 Read 1 Done New Done false
3439 1009 Acked 2008 Flush 0 Done New Done false
3440 1010 Acked 2009 Read 2 Done New Done false
3441 1011 Acked 2010 Write 8 Done New Done false
3442 1012 Acked 2011 Read 9 Done New Done false
3443 1013 Acked 2012 Write 4 Done New Done false
3444 1014 Acked 2013 Flush 0 Done New Done false
3445 1015 Acked 2014 Write 7 Done New Done false
3446 1016 Acked 2015 Read 3 Done New Done false
3447 1017 Acked 2016 Read 8 Done New Done false
3448 1018 Acked 2017 Read 2 Done New Done false
3449 1019 Acked 2018 Flush 0 Done New Done false
3450 1020 Acked 2019 Read 6 Done New Done false
3451 1021 Acked 2020 Flush 0 Done New Done false
3452 1022 Acked 2021 Write 5 Done New Done false
3453 1023 Acked 2022 Read 4 Done New Done false
3454 1024 Acked 2023 Write 5 Done New Done false
3455 1025 Acked 2024 Read 5 Done New Done false
3456 1026 Acked 2025 Read 3 Done New Done false
3457 1027 Acked 2026 Flush 0 Done New Done false
3458 1028 Acked 2027 Read 3 Done New Done false
3459 1029 Acked 2028 Read 10 Done New Done false
3460 1030 Acked 2029 Write 4 Done New Done false
3461 1031 Acked 2030 Read 1 Done New Done false
3462 1032 Acked 2031 Write 9 Done New Done false
3463 1033 Acked 2032 Write 4 Done New Done false
3464 1034 Acked 2033 Write 6 Done New Done false
3465 1035 Acked 2034 Write 6 Done New Done false
3466 1036 Acked 2035 Read 2 Done New Done false
3467 1037 Acked 2036 Read 4 Done New Done false
3468 1038 Acked 2037 Read 5 Done New Done false
3469 1039 Acked 2038 Flush 0 Done New Done false
3470 1040 Acked 2039 Write 8 Done New Done false
3471 1041 Acked 2040 Flush 0 Done New Done false
3472 1042 Acked 2041 Read 9 Done New Done false
3473 1043 Acked 2042 Write 5 Done New Done false
3474 1044 Acked 2043 Write 1 Done New Done false
3475 1045 Acked 2044 Read 4 Done New Done false
3476 1046 Acked 2045 Read 1 Done New Done false
3477 1047 Acked 2046 Read 10 Done New Done false
3478 1048 Acked 2047 Read 2 Done New Done false
3479 1049 Acked 2048 Read 1 Done New Done false
3480 1050 Acked 2049 Read 8 Done New Done false
3481 1051 Acked 2050 Read 4 Done New Done false
3482 1052 Acked 2051 Write 7 Done New Done false
3483 1053 Acked 2052 Read 5 Done New Done false
3484 1054 Acked 2053 Write 3 Done New Done false
3485 1055 Acked 2054 Write 1 Done New Done false
3486 1056 Acked 2055 Read 8 Done New Done false
3487 1057 Acked 2056 Read 3 Done New Done false
3488 1058 Acked 2057 Read 4 Done New Done false
3489 1059 Acked 2058 Read 8 Done New Done false
3490 1060 Acked 2059 Read 5 Done New Done false
3491 1061 Acked 2060 Read 3 Done New Done false
3492 1062 Acked 2061 Flush 0 Done New Done false
3493 1063 Acked 2062 Write 6 Done New Done false
3494 1064 Acked 2063 Read 2 Done New Done false
3495 1065 Acked 2064 Write 3 Done New Done false
3496 1066 Acked 2065 Flush 0 Done New Done false
3497 1067 Acked 2066 Read 3 Done New Done false
3498 1068 Acked 2067 Read 5 Done New Done false
3499 1069 Acked 2068 Write 4 Done New Done false
3500 1070 Acked 2069 Read 10 Done New Done false
3501 1071 Acked 2070 Write 10 Done New Done false
3502 1072 Acked 2071 Read 5 Done New Done false
3503 1073 Acked 2072 Write 8 Done New Done false
3504 1074 Acked 2073 Read 1 Done New Done false
3505 1075 Acked 2074 Read 9 Done New Done false
3506 1076 Acked 2075 Read 1 Done New Done false
3507 1077 Acked 2076 Read 7 Done New Done false
3508 1078 Acked 2077 Read 9 Done New Done false
3509 1079 Acked 2078 Write 10 Done New Done false
3510 1080 Acked 2079 Flush 0 Done New Done false
3511 1081 Acked 2080 Write 10 Done New Done false
3512 1082 Acked 2081 Write 4 Done New Done false
3513 1083 Acked 2082 Read 5 Done New Done false
3514 1084 Acked 2083 Read 2 Done New Done false
3515 1085 Acked 2084 Read 3 Done New Done false
3516 1086 Acked 2085 Write 5 Done New Done false
3517 1087 Acked 2086 Read 2 Done New Done false
3518 1088 Acked 2087 Write 10 Done New Done false
3519 1089 Acked 2088 Read 4 Done New Done false
3520 1090 Acked 2089 Read 4 Done New Done false
3521 1091 Acked 2090 Read 3 Done New Done false
3522 1092 Acked 2091 Write 5 Done New Done false
3523 1093 Acked 2092 Write 5 Done New Done false
3524 1094 Acked 2093 Read 2 Done New Done false
3525 1095 Acked 2094 Write 6 Done New Done false
3526 1096 Acked 2095 Read 3 Done New Done false
3527 1097 Acked 2096 Write 5 Done New Done false
3528 1098 Acked 2097 Read 1 Done New Done false
3529 1099 Acked 2098 Write 5 Done New Done false
3530 1100 Acked 2099 Write 4 Done New Done false
3531 1101 Acked 2100 Flush 0 Done New Done false
3532 1102 Acked 2101 Write 1 Done New Done false
3533 1103 Acked 2102 Read 10 Done New Done false
3534 1104 Acked 2103 Write 5 Done New Done false
3535 1105 Acked 2104 Flush 0 Done New Done false
3536 1106 Acked 2105 Read 8 Done New Done false
3537 1107 Acked 2106 Write 1 Done New Done false
3538 1108 Acked 2107 Read 6 Done New Done false
3539 1109 Acked 2108 Write 10 Done New Done false
3540 1110 Acked 2109 Read 9 Done New Done false
3541 1111 Acked 2110 Read 7 Done New Done false
3542 1112 Acked 2111 Read 7 Done New Done false
3543 1113 Acked 2112 Read 8 Done New Done false
3544 1114 Acked 2113 Flush 0 Done New Done false
3545 1115 Acked 2114 Read 4 Done New Done false
3546 1116 Acked 2115 Write 10 Done New Done false
3547 1117 Acked 2116 Write 10 Done New Done false
3548 1118 Acked 2117 Write 7 Done New Done false
3549 1119 Acked 2118 Read 6 Done New Done false
3550 1120 Acked 2119 Write 10 Done New Done false
3551 1121 Acked 2120 Read 10 Done New Done false
3552 1122 Acked 2121 Read 4 Done New Done false
3553 1123 Acked 2122 Read 3 Done New Done false
3554 1124 Acked 2123 Read 10 Done New Done false
3555 1125 Acked 2124 Write 1 Done New Done false
3556 1126 Acked 2125 Write 9 Done New Done false
3557 1127 Acked 2126 Read 10 Done New Done false
3558 1128 Acked 2127 Write 4 Done New Done false
3559 1129 Acked 2128 Write 5 Done New Done false
3560 1130 Acked 2129 Read 4 Done New Done false
3561 1131 Acked 2130 Read 4 Done New Done false
3562 1132 Acked 2131 Write 9 Done New Done false
3563 1133 Acked 2132 Read 5 Done New Done false
3564 1134 Acked 2133 Read 2 Done New Done false
3565 1135 Acked 2134 Write 6 Done New Done false
3566 1136 Acked 2135 Read 6 Done New Done false
3567 1137 Acked 2136 Read 6 Done New Done false
3568 1138 Acked 2137 Read 2 Done New Done false
3569 1139 Acked 2138 Read 5 Done New Done false
3570 1140 Acked 2139 Write 9 Done New Done false
3571 1141 Acked 2140 Write 7 Done New Done false
3572 1142 Acked 2141 Read 10 Done New Done false
3573 1143 Acked 2142 Write 4 Done New Done false
3574 1144 Acked 2143 Flush 0 Done New Done false
3575 1145 Acked 2144 Read 7 Done New Done false
3576 1146 Acked 2145 Write 1 Done New Done false
3577 1147 Acked 2146 Write 8 Done New Done false
3578 1148 Acked 2147 Write 4 Done New Done false
3579 1149 Acked 2148 Write 5 Done New Done false
3580 1150 Acked 2149 Read 7 Done New Done false
3581 1151 Acked 2150 Flush 0 Done New Done false
3582 1152 Acked 2151 Write 3 Done New Done false
3583 1153 Acked 2152 Flush 0 Done New Done false
3584 1154 Acked 2153 Read 5 Done New Done false
3585 1155 Acked 2154 Read 6 Done New Done false
3586 1156 Acked 2155 Write 5 Done New Done false
3587 1157 Acked 2156 Read 2 Done New Done false
3588 1158 Acked 2157 Read 5 Done New Done false
3589 1159 Acked 2158 Read 4 Done New Done false
3590 1160 Acked 2159 Read 4 Done New Done false
3591 1161 Acked 2160 Read 4 Done New Done false
3592 1162 Acked 2161 Write 2 Done New Done false
3593 1163 Acked 2162 Read 8 Done New Done false
3594 1164 Acked 2163 Read 9 Done New Done false
3595 1165 Acked 2164 Read 1 Done New Done false
3596 1166 Acked 2165 Read 9 Done New Done false
3597 1167 Acked 2166 Read 7 Done New Done false
3598 1168 Acked 2167 Read 5 Done New Done false
3599 1169 Acked 2168 Write 2 Done New Done false
3600 1170 Acked 2169 Read 1 Done New Done false
3601 1171 Acked 2170 Write 3 Done New Done false
3602 1172 Acked 2171 Write 1 Done New Done false
3603 1173 Acked 2172 Flush 0 Done New Done false
3604 1174 Acked 2173 Read 9 Done New Done false
3605 1175 Acked 2174 Read 3 Done New Done false
3606 1176 Acked 2175 Read 2 Done New Done false
3607 1177 Acked 2176 Read 8 Done New Done false
3608 1178 Acked 2177 Read 5 Done New Done false
3609 1179 Acked 2178 Write 4 Done New Done false
3610 1180 Acked 2179 Write 4 Done New Done false
3611 1181 Acked 2180 Flush 0 Done New Done false
3612 1182 Acked 2181 Read 4 Done New Done false
3613 1183 Acked 2182 Write 1 Done New Done false
3614 1184 Acked 2183 Write 3 Done New Done false
3615 1185 Acked 2184 Flush 0 Done New Done false
3616 1186 Acked 2185 Read 8 Done New Done false
3617 1187 Acked 2186 Read 5 Done New Done false
3618 1188 Acked 2187 Read 10 Done New Done false
3619 1189 Acked 2188 Flush 0 Done New Done false
3620 1190 Acked 2189 Read 6 Done New Done false
3621 1191 Acked 2190 Write 4 Done New Done false
3622 1192 Acked 2191 Write 8 Done New Done false
3623 1193 Acked 2192 Write 6 Done New Done false
3624 1194 Acked 2193 Write 8 Done New Done false
3625 1195 Acked 2194 Read 2 Done New Done false
3626 1196 Acked 2195 Write 10 Done New Done false
3627 1197 Acked 2196 Read 5 Done New Done false
3628 1198 Acked 2197 Read 10 Done New Done false
3629 1199 Acked 2198 Write 6 Done New Done false
3630 1200 Acked 2199 Read 9 Done New Done false
3631 1201 Acked 2200 Write 6 Done New Done false
3632 1202 Acked 2201 Read 5 Done New Done false
3633 1203 Acked 2202 Read 9 Done New Done false
3634 STATES DS:0 DS:1 DS:2 TOTAL
3635 New 0 300 0 300
3636 Sent 0 0 0 0
3637 Done 300 0 300 600
3638 Skipped 0 0 0 0
3639 Error 0 0 0 0
3640 Last Flush: 2188 1902 2188
3641 Downstairs last five completed: 1902 1901 1900 1899 1898
3642 Upstairs last five completed: 1203 1202 1201 1200 1199
3643 CLIENT: Up:0 ds:300 act:2
36442024-02-27T21:45:20.043ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
36452024-02-27T21:45:20.043ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
36462024-02-27T21:45:20.043ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
36472024-02-27T21:45:20.043ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
36482024-02-27T21:45:20.043ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
36492024-02-27T21:45:20.043ZINFOcrucible: [1] client re-new 301 jobs since flush 1902 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
36502024-02-27T21:45:20.044ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
36512024-02-27T21:45:21.044ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
36522024-02-27T21:45:21.044ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
36532024-02-27T21:45:21.044ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
36542024-02-27T21:45:21.044ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
36552024-02-27T21:45:21.044ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
36562024-02-27T21:45:21.045ZINFOcrucible: [1] client re-new 301 jobs since flush 1902 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
36572024-02-27T21:45:21.045ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
36582024-02-27T21:45:22.046ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
36592024-02-27T21:45:22.047ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
36602024-02-27T21:45:22.047ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
36612024-02-27T21:45:22.047ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
36622024-02-27T21:45:22.047ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
36632024-02-27T21:45:22.047ZINFOcrucible: [1] client re-new 301 jobs since flush 1902 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
36642024-02-27T21:45:22.047ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
36652024-02-27T21:45:23.049ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
36662024-02-27T21:45:23.049ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
36672024-02-27T21:45:23.049ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
36682024-02-27T21:45:23.049ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
36692024-02-27T21:45:23.049ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
36702024-02-27T21:45:23.049ZINFOcrucible: [1] client re-new 301 jobs since flush 1902 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
36712024-02-27T21:45:23.049ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
3672 ----------------------------------------------------------------
3673 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
3674 States: Active Offline Active
3675 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
3676 904 Acked 1903 Write 1 Done New Done true
3677 905 Acked 1904 Read 10 Done New Done true
3678 906 Acked 1905 Read 1 Done New Done true
3679 907 Acked 1906 Flush 0 Done New Done true
3680 908 Acked 1907 Read 8 Done New Done true
3681 909 Acked 1908 Read 9 Done New Done true
3682 910 Acked 1909 Write 3 Done New Done true
3683 911 Acked 1910 Read 10 Done New Done true
3684 912 Acked 1911 Read 10 Done New Done true
3685 913 Acked 1912 Flush 0 Done New Done true
3686 914 Acked 1913 Read 6 Done New Done true
3687 915 Acked 1914 Write 1 Done New Done true
3688 916 Acked 1915 Write 10 Done New Done true
3689 917 Acked 1916 Write 9 Done New Done true
3690 918 Acked 1917 Write 9 Done New Done true
3691 919 Acked 1918 Read 6 Done New Done true
3692 920 Acked 1919 Write 9 Done New Done true
3693 921 Acked 1920 Write 9 Done New Done true
3694 922 Acked 1921 Read 5 Done New Done true
3695 923 Acked 1922 Read 6 Done New Done true
3696 924 Acked 1923 Write 9 Done New Done true
3697 925 Acked 1924 Write 2 Done New Done true
3698 926 Acked 1925 Read 1 Done New Done true
3699 927 Acked 1926 Write 6 Done New Done true
3700 928 Acked 1927 Read 2 Done New Done true
3701 929 Acked 1928 Read 6 Done New Done true
3702 930 Acked 1929 Write 3 Done New Done true
3703 931 Acked 1930 Read 5 Done New Done true
3704 932 Acked 1931 Read 5 Done New Done true
3705 933 Acked 1932 Write 6 Done New Done true
3706 934 Acked 1933 Read 7 Done New Done true
3707 935 Acked 1934 Read 8 Done New Done true
3708 936 Acked 1935 Write 1 Done New Done true
3709 937 Acked 1936 Read 10 Done New Done true
3710 938 Acked 1937 Write 8 Done New Done true
3711 939 Acked 1938 Read 9 Done New Done true
3712 940 Acked 1939 Read 10 Done New Done true
3713 941 Acked 1940 Read 6 Done New Done true
3714 942 Acked 1941 Read 4 Done New Done true
3715 943 Acked 1942 Write 6 Done New Done true
3716 944 Acked 1943 Write 7 Done New Done true
3717 945 Acked 1944 Read 4 Done New Done true
3718 946 Acked 1945 Write 1 Done New Done true
3719 947 Acked 1946 Read 7 Done New Done true
3720 948 Acked 1947 Write 2 Done New Done true
3721 949 Acked 1948 Flush 0 Done New Done true
3722 950 Acked 1949 Read 2 Done New Done true
3723 951 Acked 1950 Read 10 Done New Done true
3724 952 Acked 1951 Read 1 Done New Done true
3725 953 Acked 1952 Read 4 Done New Done true
3726 954 Acked 1953 Flush 0 Done New Done true
3727 955 Acked 1954 Write 9 Done New Done true
3728 956 Acked 1955 Write 8 Done New Done true
3729 957 Acked 1956 Write 3 Done New Done true
3730 958 Acked 1957 Write 8 Done New Done true
3731 959 Acked 1958 Read 7 Done New Done true
3732 960 Acked 1959 Write 2 Done New Done true
3733 961 Acked 1960 Read 7 Done New Done true
3734 962 Acked 1961 Read 7 Done New Done true
3735 963 Acked 1962 Write 4 Done New Done true
3736 964 Acked 1963 Read 4 Done New Done true
3737 965 Acked 1964 Write 7 Done New Done true
3738 966 Acked 1965 Read 3 Done New Done true
3739 967 Acked 1966 Read 5 Done New Done true
3740 968 Acked 1967 Read 9 Done New Done true
3741 969 Acked 1968 Read 3 Done New Done true
3742 970 Acked 1969 Write 7 Done New Done true
3743 971 Acked 1970 Read 5 Done New Done true
3744 972 Acked 1971 Read 8 Done New Done true
3745 973 Acked 1972 Read 3 Done New Done true
3746 974 Acked 1973 Write 7 Done New Done true
3747 975 Acked 1974 Read 1 Done New Done true
3748 976 Acked 1975 Read 4 Done New Done true
3749 977 Acked 1976 Read 4 Done New Done true
3750 978 Acked 1977 Flush 0 Done New Done true
3751 979 Acked 1978 Write 8 Done New Done true
3752 980 Acked 1979 Flush 0 Done New Done true
3753 981 Acked 1980 Read 5 Done New Done true
3754 982 Acked 1981 Flush 0 Done New Done true
3755 983 Acked 1982 Write 2 Done New Done true
3756 984 Acked 1983 Read 9 Done New Done true
3757 985 Acked 1984 Read 6 Done New Done true
3758 986 Acked 1985 Read 1 Done New Done true
3759 987 Acked 1986 Write 4 Done New Done true
3760 988 Acked 1987 Read 4 Done New Done true
3761 989 Acked 1988 Write 7 Done New Done true
3762 990 Acked 1989 Read 5 Done New Done true
3763 991 Acked 1990 Read 2 Done New Done true
3764 992 Acked 1991 Write 7 Done New Done true
3765 993 Acked 1992 Read 4 Done New Done true
3766 994 Acked 1993 Write 9 Done New Done true
3767 995 Acked 1994 Read 9 Done New Done true
3768 996 Acked 1995 Read 9 Done New Done true
3769 997 Acked 1996 Write 5 Done New Done true
3770 998 Acked 1997 Write 6 Done New Done true
3771 999 Acked 1998 Write 4 Done New Done true
3772 1000 Acked 1999 Read 5 Done New Done true
3773 1001 Acked 2000 Write 8 Done New Done true
3774 1002 Acked 2001 Read 4 Done New Done true
3775 1003 Acked 2002 Write 6 Done New Done true
3776 1004 Acked 2003 Write 2 Done New Done true
3777 1005 Acked 2004 Read 7 Done New Done true
3778 1006 Acked 2005 Read 6 Done New Done true
3779 1007 Acked 2006 Read 8 Done New Done true
3780 1008 Acked 2007 Read 1 Done New Done true
3781 1009 Acked 2008 Flush 0 Done New Done true
3782 1010 Acked 2009 Read 2 Done New Done true
3783 1011 Acked 2010 Write 8 Done New Done true
3784 1012 Acked 2011 Read 9 Done New Done true
3785 1013 Acked 2012 Write 4 Done New Done true
3786 1014 Acked 2013 Flush 0 Done New Done true
3787 1015 Acked 2014 Write 7 Done New Done true
3788 1016 Acked 2015 Read 3 Done New Done true
3789 1017 Acked 2016 Read 8 Done New Done true
3790 1018 Acked 2017 Read 2 Done New Done true
3791 1019 Acked 2018 Flush 0 Done New Done true
3792 1020 Acked 2019 Read 6 Done New Done true
3793 1021 Acked 2020 Flush 0 Done New Done true
3794 1022 Acked 2021 Write 5 Done New Done true
3795 1023 Acked 2022 Read 4 Done New Done true
3796 1024 Acked 2023 Write 5 Done New Done true
3797 1025 Acked 2024 Read 5 Done New Done true
3798 1026 Acked 2025 Read 3 Done New Done true
3799 1027 Acked 2026 Flush 0 Done New Done true
3800 1028 Acked 2027 Read 3 Done New Done true
3801 1029 Acked 2028 Read 10 Done New Done true
3802 1030 Acked 2029 Write 4 Done New Done true
3803 1031 Acked 2030 Read 1 Done New Done true
3804 1032 Acked 2031 Write 9 Done New Done true
3805 1033 Acked 2032 Write 4 Done New Done true
3806 1034 Acked 2033 Write 6 Done New Done true
3807 1035 Acked 2034 Write 6 Done New Done true
3808 1036 Acked 2035 Read 2 Done New Done true
3809 1037 Acked 2036 Read 4 Done New Done true
3810 1038 Acked 2037 Read 5 Done New Done true
3811 1039 Acked 2038 Flush 0 Done New Done true
3812 1040 Acked 2039 Write 8 Done New Done true
3813 1041 Acked 2040 Flush 0 Done New Done true
3814 1042 Acked 2041 Read 9 Done New Done true
3815 1043 Acked 2042 Write 5 Done New Done true
3816 1044 Acked 2043 Write 1 Done New Done true
3817 1045 Acked 2044 Read 4 Done New Done true
3818 1046 Acked 2045 Read 1 Done New Done true
3819 1047 Acked 2046 Read 10 Done New Done true
3820 1048 Acked 2047 Read 2 Done New Done true
3821 1049 Acked 2048 Read 1 Done New Done true
3822 1050 Acked 2049 Read 8 Done New Done true
3823 1051 Acked 2050 Read 4 Done New Done true
3824 1052 Acked 2051 Write 7 Done New Done true
3825 1053 Acked 2052 Read 5 Done New Done true
3826 1054 Acked 2053 Write 3 Done New Done true
3827 1055 Acked 2054 Write 1 Done New Done true
3828 1056 Acked 2055 Read 8 Done New Done true
3829 1057 Acked 2056 Read 3 Done New Done true
3830 1058 Acked 2057 Read 4 Done New Done true
3831 1059 Acked 2058 Read 8 Done New Done true
3832 1060 Acked 2059 Read 5 Done New Done true
3833 1061 Acked 2060 Read 3 Done New Done true
3834 1062 Acked 2061 Flush 0 Done New Done true
3835 1063 Acked 2062 Write 6 Done New Done true
3836 1064 Acked 2063 Read 2 Done New Done true
3837 1065 Acked 2064 Write 3 Done New Done true
3838 1066 Acked 2065 Flush 0 Done New Done true
3839 1067 Acked 2066 Read 3 Done New Done true
3840 1068 Acked 2067 Read 5 Done New Done true
3841 1069 Acked 2068 Write 4 Done New Done true
3842 1070 Acked 2069 Read 10 Done New Done true
3843 1071 Acked 2070 Write 10 Done New Done true
3844 1072 Acked 2071 Read 5 Done New Done true
3845 1073 Acked 2072 Write 8 Done New Done true
3846 1074 Acked 2073 Read 1 Done New Done true
3847 1075 Acked 2074 Read 9 Done New Done true
3848 1076 Acked 2075 Read 1 Done New Done true
3849 1077 Acked 2076 Read 7 Done New Done true
3850 1078 Acked 2077 Read 9 Done New Done true
3851 1079 Acked 2078 Write 10 Done New Done true
3852 1080 Acked 2079 Flush 0 Done New Done true
3853 1081 Acked 2080 Write 10 Done New Done true
3854 1082 Acked 2081 Write 4 Done New Done true
3855 1083 Acked 2082 Read 5 Done New Done true
3856 1084 Acked 2083 Read 2 Done New Done true
3857 1085 Acked 2084 Read 3 Done New Done true
3858 1086 Acked 2085 Write 5 Done New Done true
3859 1087 Acked 2086 Read 2 Done New Done true
3860 1088 Acked 2087 Write 10 Done New Done true
3861 1089 Acked 2088 Read 4 Done New Done true
3862 1090 Acked 2089 Read 4 Done New Done true
3863 1091 Acked 2090 Read 3 Done New Done true
3864 1092 Acked 2091 Write 5 Done New Done true
3865 1093 Acked 2092 Write 5 Done New Done true
3866 1094 Acked 2093 Read 2 Done New Done true
3867 1095 Acked 2094 Write 6 Done New Done true
3868 1096 Acked 2095 Read 3 Done New Done true
3869 1097 Acked 2096 Write 5 Done New Done true
3870 1098 Acked 2097 Read 1 Done New Done true
3871 1099 Acked 2098 Write 5 Done New Done true
3872 1100 Acked 2099 Write 4 Done New Done true
3873 1101 Acked 2100 Flush 0 Done New Done true
3874 1102 Acked 2101 Write 1 Done New Done true
3875 1103 Acked 2102 Read 10 Done New Done true
3876 1104 Acked 2103 Write 5 Done New Done true
3877 1105 Acked 2104 Flush 0 Done New Done true
3878 1106 Acked 2105 Read 8 Done New Done true
3879 1107 Acked 2106 Write 1 Done New Done true
3880 1108 Acked 2107 Read 6 Done New Done true
3881 1109 Acked 2108 Write 10 Done New Done true
3882 1110 Acked 2109 Read 9 Done New Done true
3883 1111 Acked 2110 Read 7 Done New Done true
3884 1112 Acked 2111 Read 7 Done New Done true
3885 1113 Acked 2112 Read 8 Done New Done true
3886 1114 Acked 2113 Flush 0 Done New Done true
3887 1115 Acked 2114 Read 4 Done New Done true
3888 1116 Acked 2115 Write 10 Done New Done true
3889 1117 Acked 2116 Write 10 Done New Done true
3890 1118 Acked 2117 Write 7 Done New Done true
3891 1119 Acked 2118 Read 6 Done New Done true
3892 1120 Acked 2119 Write 10 Done New Done true
3893 1121 Acked 2120 Read 10 Done New Done true
3894 1122 Acked 2121 Read 4 Done New Done true
3895 1123 Acked 2122 Read 3 Done New Done true
3896 1124 Acked 2123 Read 10 Done New Done true
3897 1125 Acked 2124 Write 1 Done New Done true
3898 1126 Acked 2125 Write 9 Done New Done true
3899 1127 Acked 2126 Read 10 Done New Done true
3900 1128 Acked 2127 Write 4 Done New Done true
3901 1129 Acked 2128 Write 5 Done New Done true
3902 1130 Acked 2129 Read 4 Done New Done true
3903 1131 Acked 2130 Read 4 Done New Done true
3904 1132 Acked 2131 Write 9 Done New Done true
3905 1133 Acked 2132 Read 5 Done New Done true
3906 1134 Acked 2133 Read 2 Done New Done true
3907 1135 Acked 2134 Write 6 Done New Done true
3908 1136 Acked 2135 Read 6 Done New Done true
3909 1137 Acked 2136 Read 6 Done New Done true
3910 1138 Acked 2137 Read 2 Done New Done true
3911 1139 Acked 2138 Read 5 Done New Done true
3912 1140 Acked 2139 Write 9 Done New Done true
3913 1141 Acked 2140 Write 7 Done New Done true
3914 1142 Acked 2141 Read 10 Done New Done true
3915 1143 Acked 2142 Write 4 Done New Done true
3916 1144 Acked 2143 Flush 0 Done New Done true
3917 1145 Acked 2144 Read 7 Done New Done true
3918 1146 Acked 2145 Write 1 Done New Done true
3919 1147 Acked 2146 Write 8 Done New Done true
3920 1148 Acked 2147 Write 4 Done New Done true
3921 1149 Acked 2148 Write 5 Done New Done true
3922 1150 Acked 2149 Read 7 Done New Done true
3923 1151 Acked 2150 Flush 0 Done New Done true
3924 1152 Acked 2151 Write 3 Done New Done true
3925 1153 Acked 2152 Flush 0 Done New Done true
3926 1154 Acked 2153 Read 5 Done New Done true
3927 1155 Acked 2154 Read 6 Done New Done true
3928 1156 Acked 2155 Write 5 Done New Done true
3929 1157 Acked 2156 Read 2 Done New Done true
3930 1158 Acked 2157 Read 5 Done New Done true
3931 1159 Acked 2158 Read 4 Done New Done true
3932 1160 Acked 2159 Read 4 Done New Done true
3933 1161 Acked 2160 Read 4 Done New Done true
3934 1162 Acked 2161 Write 2 Done New Done true
3935 1163 Acked 2162 Read 8 Done New Done true
3936 1164 Acked 2163 Read 9 Done New Done true
3937 1165 Acked 2164 Read 1 Done New Done true
3938 1166 Acked 2165 Read 9 Done New Done true
3939 1167 Acked 2166 Read 7 Done New Done true
3940 1168 Acked 2167 Read 5 Done New Done true
3941 1169 Acked 2168 Write 2 Done New Done true
3942 1170 Acked 2169 Read 1 Done New Done true
3943 1171 Acked 2170 Write 3 Done New Done true
3944 1172 Acked 2171 Write 1 Done New Done true
3945 1173 Acked 2172 Flush 0 Done New Done true
3946 1174 Acked 2173 Read 9 Done New Done true
3947 1175 Acked 2174 Read 3 Done New Done true
3948 1176 Acked 2175 Read 2 Done New Done true
3949 1177 Acked 2176 Read 8 Done New Done true
3950 1178 Acked 2177 Read 5 Done New Done true
3951 1179 Acked 2178 Write 4 Done New Done true
3952 1180 Acked 2179 Write 4 Done New Done true
3953 1181 Acked 2180 Flush 0 Done New Done true
3954 1182 Acked 2181 Read 4 Done New Done true
3955 1183 Acked 2182 Write 1 Done New Done true
3956 1184 Acked 2183 Write 3 Done New Done true
3957 1185 Acked 2184 Flush 0 Done New Done true
3958 1186 Acked 2185 Read 8 Done New Done true
3959 1187 Acked 2186 Read 5 Done New Done true
3960 1188 Acked 2187 Read 10 Done New Done true
3961 1189 Acked 2188 Flush 0 Done New Done true
3962 1190 Acked 2189 Read 6 Done New Done true
3963 1191 Acked 2190 Write 4 Done New Done true
3964 1192 Acked 2191 Write 8 Done New Done true
3965 1193 Acked 2192 Write 6 Done New Done true
3966 1194 Acked 2193 Write 8 Done New Done true
3967 1195 Acked 2194 Read 2 Done New Done true
3968 1196 Acked 2195 Write 10 Done New Done true
3969 1197 Acked 2196 Read 5 Done New Done true
3970 1198 Acked 2197 Read 10 Done New Done true
3971 1199 Acked 2198 Write 6 Done New Done true
3972 1200 Acked 2199 Read 9 Done New Done true
3973 1201 Acked 2200 Write 6 Done New Done true
3974 1202 Acked 2201 Read 5 Done New Done true
3975 1203 Acked 2202 Read 9 Done New Done true
3976 1204 Acked 2203 Flush 0 Done New Done true
3977 STATES DS:0 DS:1 DS:2 TOTAL
3978 New 0 301 0 301
3979 Sent 0 0 0 0
3980 Done 301 0 301 602
3981 Skipped 0 0 0 0
3982 Error 0 0 0 0
3983 Last Flush: 2203 1902 2203
3984 Downstairs last five completed: 1902 1901 1900 1899 1898
3985 Upstairs last five completed: 1204 1203 1202 1201 1200
3986 CLIENT: Up:0 ds:301 act:2
39872024-02-27T21:45:24.050ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
39882024-02-27T21:45:24.050ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
39892024-02-27T21:45:24.050ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
39902024-02-27T21:45:24.050ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
39912024-02-27T21:45:24.050ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
39922024-02-27T21:45:24.050ZINFOcrucible: [1] client re-new 301 jobs since flush 1902 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
39932024-02-27T21:45:24.050ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
39942024-02-27T21:45:25.051ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
39952024-02-27T21:45:25.051ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
39962024-02-27T21:45:25.051ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
39972024-02-27T21:45:25.051ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
39982024-02-27T21:45:25.051ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
39992024-02-27T21:45:25.051ZINFOcrucible: [1] client re-new 301 jobs since flush 1902 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
40002024-02-27T21:45:25.051ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
40012024-02-27T21:45:26.052ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
40022024-02-27T21:45:26.052ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
40032024-02-27T21:45:26.052ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
40042024-02-27T21:45:26.052ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
40052024-02-27T21:45:26.052ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
40062024-02-27T21:45:26.052ZINFOcrucible: [1] client re-new 301 jobs since flush 1902 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
40072024-02-27T21:45:26.053ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
4008 [8820] got start action
4009 [8820] I am going to restart
4010 Starting downstairs at port 8820
4011 Make output file at "/tmp/dsc/downstairs-8820.txt"
4012 [8820][1] reports Starting
4013 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1171)
4014 [8820][1] reports Running
40152024-02-27T21:45:27.054ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
40162024-02-27T21:45:27.054ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
40172024-02-27T21:45:27.054ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
40182024-02-27T21:45:27.054ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
40192024-02-27T21:45:27.054ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
40202024-02-27T21:45:27.054ZINFOcrucible: [1] client re-new 301 jobs since flush 1902 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
40212024-02-27T21:45:27.054ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
4022 ----------------------------------------------------------------
4023 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4024 States: Active Offline Active
4025 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
4026 904 Acked 1903 Write 1 Done New Done true
4027 905 Acked 1904 Read 10 Done New Done true
4028 906 Acked 1905 Read 1 Done New Done true
4029 907 Acked 1906 Flush 0 Done New Done true
4030 908 Acked 1907 Read 8 Done New Done true
4031 909 Acked 1908 Read 9 Done New Done true
4032 910 Acked 1909 Write 3 Done New Done true
4033 911 Acked 1910 Read 10 Done New Done true
4034 912 Acked 1911 Read 10 Done New Done true
4035 913 Acked 1912 Flush 0 Done New Done true
4036 914 Acked 1913 Read 6 Done New Done true
4037 915 Acked 1914 Write 1 Done New Done true
4038 916 Acked 1915 Write 10 Done New Done true
4039 917 Acked 1916 Write 9 Done New Done true
4040 918 Acked 1917 Write 9 Done New Done true
4041 919 Acked 1918 Read 6 Done New Done true
4042 920 Acked 1919 Write 9 Done New Done true
4043 921 Acked 1920 Write 9 Done New Done true
4044 922 Acked 1921 Read 5 Done New Done true
4045 923 Acked 1922 Read 6 Done New Done true
4046 924 Acked 1923 Write 9 Done New Done true
4047 925 Acked 1924 Write 2 Done New Done true
4048 926 Acked 1925 Read 1 Done New Done true
4049 927 Acked 1926 Write 6 Done New Done true
4050 928 Acked 1927 Read 2 Done New Done true
4051 929 Acked 1928 Read 6 Done New Done true
4052 930 Acked 1929 Write 3 Done New Done true
4053 931 Acked 1930 Read 5 Done New Done true
4054 932 Acked 1931 Read 5 Done New Done true
4055 933 Acked 1932 Write 6 Done New Done true
4056 934 Acked 1933 Read 7 Done New Done true
4057 935 Acked 1934 Read 8 Done New Done true
4058 936 Acked 1935 Write 1 Done New Done true
4059 937 Acked 1936 Read 10 Done New Done true
4060 938 Acked 1937 Write 8 Done New Done true
4061 939 Acked 1938 Read 9 Done New Done true
4062 940 Acked 1939 Read 10 Done New Done true
4063 941 Acked 1940 Read 6 Done New Done true
4064 942 Acked 1941 Read 4 Done New Done true
4065 943 Acked 1942 Write 6 Done New Done true
4066 944 Acked 1943 Write 7 Done New Done true
4067 945 Acked 1944 Read 4 Done New Done true
4068 946 Acked 1945 Write 1 Done New Done true
4069 947 Acked 1946 Read 7 Done New Done true
4070 948 Acked 1947 Write 2 Done New Done true
4071 949 Acked 1948 Flush 0 Done New Done true
4072 950 Acked 1949 Read 2 Done New Done true
4073 951 Acked 1950 Read 10 Done New Done true
4074 952 Acked 1951 Read 1 Done New Done true
4075 953 Acked 1952 Read 4 Done New Done true
4076 954 Acked 1953 Flush 0 Done New Done true
4077 955 Acked 1954 Write 9 Done New Done true
4078 956 Acked 1955 Write 8 Done New Done true
4079 957 Acked 1956 Write 3 Done New Done true
4080 958 Acked 1957 Write 8 Done New Done true
4081 959 Acked 1958 Read 7 Done New Done true
4082 960 Acked 1959 Write 2 Done New Done true
4083 961 Acked 1960 Read 7 Done New Done true
4084 962 Acked 1961 Read 7 Done New Done true
4085 963 Acked 1962 Write 4 Done New Done true
4086 964 Acked 1963 Read 4 Done New Done true
4087 965 Acked 1964 Write 7 Done New Done true
4088 966 Acked 1965 Read 3 Done New Done true
4089 967 Acked 1966 Read 5 Done New Done true
4090 968 Acked 1967 Read 9 Done New Done true
4091 969 Acked 1968 Read 3 Done New Done true
4092 970 Acked 1969 Write 7 Done New Done true
4093 971 Acked 1970 Read 5 Done New Done true
4094 972 Acked 1971 Read 8 Done New Done true
4095 973 Acked 1972 Read 3 Done New Done true
4096 974 Acked 1973 Write 7 Done New Done true
4097 975 Acked 1974 Read 1 Done New Done true
4098 976 Acked 1975 Read 4 Done New Done true
4099 977 Acked 1976 Read 4 Done New Done true
4100 978 Acked 1977 Flush 0 Done New Done true
4101 979 Acked 1978 Write 8 Done New Done true
4102 980 Acked 1979 Flush 0 Done New Done true
4103 981 Acked 1980 Read 5 Done New Done true
4104 982 Acked 1981 Flush 0 Done New Done true
4105 983 Acked 1982 Write 2 Done New Done true
4106 984 Acked 1983 Read 9 Done New Done true
4107 985 Acked 1984 Read 6 Done New Done true
4108 986 Acked 1985 Read 1 Done New Done true
4109 987 Acked 1986 Write 4 Done New Done true
4110 988 Acked 1987 Read 4 Done New Done true
4111 989 Acked 1988 Write 7 Done New Done true
4112 990 Acked 1989 Read 5 Done New Done true
4113 991 Acked 1990 Read 2 Done New Done true
4114 992 Acked 1991 Write 7 Done New Done true
4115 993 Acked 1992 Read 4 Done New Done true
4116 994 Acked 1993 Write 9 Done New Done true
4117 995 Acked 1994 Read 9 Done New Done true
4118 996 Acked 1995 Read 9 Done New Done true
4119 997 Acked 1996 Write 5 Done New Done true
4120 998 Acked 1997 Write 6 Done New Done true
4121 999 Acked 1998 Write 4 Done New Done true
4122 1000 Acked 1999 Read 5 Done New Done true
4123 1001 Acked 2000 Write 8 Done New Done true
4124 1002 Acked 2001 Read 4 Done New Done true
4125 1003 Acked 2002 Write 6 Done New Done true
4126 1004 Acked 2003 Write 2 Done New Done true
4127 1005 Acked 2004 Read 7 Done New Done true
4128 1006 Acked 2005 Read 6 Done New Done true
4129 1007 Acked 2006 Read 8 Done New Done true
4130 1008 Acked 2007 Read 1 Done New Done true
4131 1009 Acked 2008 Flush 0 Done New Done true
4132 1010 Acked 2009 Read 2 Done New Done true
4133 1011 Acked 2010 Write 8 Done New Done true
4134 1012 Acked 2011 Read 9 Done New Done true
4135 1013 Acked 2012 Write 4 Done New Done true
4136 1014 Acked 2013 Flush 0 Done New Done true
4137 1015 Acked 2014 Write 7 Done New Done true
4138 1016 Acked 2015 Read 3 Done New Done true
4139 1017 Acked 2016 Read 8 Done New Done true
4140 1018 Acked 2017 Read 2 Done New Done true
4141 1019 Acked 2018 Flush 0 Done New Done true
4142 1020 Acked 2019 Read 6 Done New Done true
4143 1021 Acked 2020 Flush 0 Done New Done true
4144 1022 Acked 2021 Write 5 Done New Done true
4145 1023 Acked 2022 Read 4 Done New Done true
4146 1024 Acked 2023 Write 5 Done New Done true
4147 1025 Acked 2024 Read 5 Done New Done true
4148 1026 Acked 2025 Read 3 Done New Done true
4149 1027 Acked 2026 Flush 0 Done New Done true
4150 1028 Acked 2027 Read 3 Done New Done true
4151 1029 Acked 2028 Read 10 Done New Done true
4152 1030 Acked 2029 Write 4 Done New Done true
4153 1031 Acked 2030 Read 1 Done New Done true
4154 1032 Acked 2031 Write 9 Done New Done true
4155 1033 Acked 2032 Write 4 Done New Done true
4156 1034 Acked 2033 Write 6 Done New Done true
4157 1035 Acked 2034 Write 6 Done New Done true
4158 1036 Acked 2035 Read 2 Done New Done true
4159 1037 Acked 2036 Read 4 Done New Done true
4160 1038 Acked 2037 Read 5 Done New Done true
4161 1039 Acked 2038 Flush 0 Done New Done true
4162 1040 Acked 2039 Write 8 Done New Done true
4163 1041 Acked 2040 Flush 0 Done New Done true
4164 1042 Acked 2041 Read 9 Done New Done true
4165 1043 Acked 2042 Write 5 Done New Done true
4166 1044 Acked 2043 Write 1 Done New Done true
4167 1045 Acked 2044 Read 4 Done New Done true
4168 1046 Acked 2045 Read 1 Done New Done true
4169 1047 Acked 2046 Read 10 Done New Done true
4170 1048 Acked 2047 Read 2 Done New Done true
4171 1049 Acked 2048 Read 1 Done New Done true
4172 1050 Acked 2049 Read 8 Done New Done true
4173 1051 Acked 2050 Read 4 Done New Done true
4174 1052 Acked 2051 Write 7 Done New Done true
4175 1053 Acked 2052 Read 5 Done New Done true
4176 1054 Acked 2053 Write 3 Done New Done true
4177 1055 Acked 2054 Write 1 Done New Done true
4178 1056 Acked 2055 Read 8 Done New Done true
4179 1057 Acked 2056 Read 3 Done New Done true
4180 1058 Acked 2057 Read 4 Done New Done true
4181 1059 Acked 2058 Read 8 Done New Done true
4182 1060 Acked 2059 Read 5 Done New Done true
4183 1061 Acked 2060 Read 3 Done New Done true
4184 1062 Acked 2061 Flush 0 Done New Done true
4185 1063 Acked 2062 Write 6 Done New Done true
4186 1064 Acked 2063 Read 2 Done New Done true
4187 1065 Acked 2064 Write 3 Done New Done true
4188 1066 Acked 2065 Flush 0 Done New Done true
4189 1067 Acked 2066 Read 3 Done New Done true
4190 1068 Acked 2067 Read 5 Done New Done true
4191 1069 Acked 2068 Write 4 Done New Done true
4192 1070 Acked 2069 Read 10 Done New Done true
4193 1071 Acked 2070 Write 10 Done New Done true
4194 1072 Acked 2071 Read 5 Done New Done true
4195 1073 Acked 2072 Write 8 Done New Done true
4196 1074 Acked 2073 Read 1 Done New Done true
4197 1075 Acked 2074 Read 9 Done New Done true
4198 1076 Acked 2075 Read 1 Done New Done true
4199 1077 Acked 2076 Read 7 Done New Done true
4200 1078 Acked 2077 Read 9 Done New Done true
4201 1079 Acked 2078 Write 10 Done New Done true
4202 1080 Acked 2079 Flush 0 Done New Done true
4203 1081 Acked 2080 Write 10 Done New Done true
4204 1082 Acked 2081 Write 4 Done New Done true
4205 1083 Acked 2082 Read 5 Done New Done true
4206 1084 Acked 2083 Read 2 Done New Done true
4207 1085 Acked 2084 Read 3 Done New Done true
4208 1086 Acked 2085 Write 5 Done New Done true
4209 1087 Acked 2086 Read 2 Done New Done true
4210 1088 Acked 2087 Write 10 Done New Done true
4211 1089 Acked 2088 Read 4 Done New Done true
4212 1090 Acked 2089 Read 4 Done New Done true
4213 1091 Acked 2090 Read 3 Done New Done true
4214 1092 Acked 2091 Write 5 Done New Done true
4215 1093 Acked 2092 Write 5 Done New Done true
4216 1094 Acked 2093 Read 2 Done New Done true
4217 1095 Acked 2094 Write 6 Done New Done true
4218 1096 Acked 2095 Read 3 Done New Done true
4219 1097 Acked 2096 Write 5 Done New Done true
4220 1098 Acked 2097 Read 1 Done New Done true
4221 1099 Acked 2098 Write 5 Done New Done true
4222 1100 Acked 2099 Write 4 Done New Done true
4223 1101 Acked 2100 Flush 0 Done New Done true
4224 1102 Acked 2101 Write 1 Done New Done true
4225 1103 Acked 2102 Read 10 Done New Done true
4226 1104 Acked 2103 Write 5 Done New Done true
4227 1105 Acked 2104 Flush 0 Done New Done true
4228 1106 Acked 2105 Read 8 Done New Done true
4229 1107 Acked 2106 Write 1 Done New Done true
4230 1108 Acked 2107 Read 6 Done New Done true
4231 1109 Acked 2108 Write 10 Done New Done true
4232 1110 Acked 2109 Read 9 Done New Done true
4233 1111 Acked 2110 Read 7 Done New Done true
4234 1112 Acked 2111 Read 7 Done New Done true
4235 1113 Acked 2112 Read 8 Done New Done true
4236 1114 Acked 2113 Flush 0 Done New Done true
4237 1115 Acked 2114 Read 4 Done New Done true
4238 1116 Acked 2115 Write 10 Done New Done true
4239 1117 Acked 2116 Write 10 Done New Done true
4240 1118 Acked 2117 Write 7 Done New Done true
4241 1119 Acked 2118 Read 6 Done New Done true
4242 1120 Acked 2119 Write 10 Done New Done true
4243 1121 Acked 2120 Read 10 Done New Done true
4244 1122 Acked 2121 Read 4 Done New Done true
4245 1123 Acked 2122 Read 3 Done New Done true
4246 1124 Acked 2123 Read 10 Done New Done true
4247 1125 Acked 2124 Write 1 Done New Done true
4248 1126 Acked 2125 Write 9 Done New Done true
4249 1127 Acked 2126 Read 10 Done New Done true
4250 1128 Acked 2127 Write 4 Done New Done true
4251 1129 Acked 2128 Write 5 Done New Done true
4252 1130 Acked 2129 Read 4 Done New Done true
4253 1131 Acked 2130 Read 4 Done New Done true
4254 1132 Acked 2131 Write 9 Done New Done true
4255 1133 Acked 2132 Read 5 Done New Done true
4256 1134 Acked 2133 Read 2 Done New Done true
4257 1135 Acked 2134 Write 6 Done New Done true
4258 1136 Acked 2135 Read 6 Done New Done true
4259 1137 Acked 2136 Read 6 Done New Done true
4260 1138 Acked 2137 Read 2 Done New Done true
4261 1139 Acked 2138 Read 5 Done New Done true
4262 1140 Acked 2139 Write 9 Done New Done true
4263 1141 Acked 2140 Write 7 Done New Done true
4264 1142 Acked 2141 Read 10 Done New Done true
4265 1143 Acked 2142 Write 4 Done New Done true
4266 1144 Acked 2143 Flush 0 Done New Done true
4267 1145 Acked 2144 Read 7 Done New Done true
4268 1146 Acked 2145 Write 1 Done New Done true
4269 1147 Acked 2146 Write 8 Done New Done true
4270 1148 Acked 2147 Write 4 Done New Done true
4271 1149 Acked 2148 Write 5 Done New Done true
4272 1150 Acked 2149 Read 7 Done New Done true
4273 1151 Acked 2150 Flush 0 Done New Done true
4274 1152 Acked 2151 Write 3 Done New Done true
4275 1153 Acked 2152 Flush 0 Done New Done true
4276 1154 Acked 2153 Read 5 Done New Done true
4277 1155 Acked 2154 Read 6 Done New Done true
4278 1156 Acked 2155 Write 5 Done New Done true
4279 1157 Acked 2156 Read 2 Done New Done true
4280 1158 Acked 2157 Read 5 Done New Done true
4281 1159 Acked 2158 Read 4 Done New Done true
4282 1160 Acked 2159 Read 4 Done New Done true
4283 1161 Acked 2160 Read 4 Done New Done true
4284 1162 Acked 2161 Write 2 Done New Done true
4285 1163 Acked 2162 Read 8 Done New Done true
4286 1164 Acked 2163 Read 9 Done New Done true
4287 1165 Acked 2164 Read 1 Done New Done true
4288 1166 Acked 2165 Read 9 Done New Done true
4289 1167 Acked 2166 Read 7 Done New Done true
4290 1168 Acked 2167 Read 5 Done New Done true
4291 1169 Acked 2168 Write 2 Done New Done true
4292 1170 Acked 2169 Read 1 Done New Done true
4293 1171 Acked 2170 Write 3 Done New Done true
4294 1172 Acked 2171 Write 1 Done New Done true
4295 1173 Acked 2172 Flush 0 Done New Done true
4296 1174 Acked 2173 Read 9 Done New Done true
4297 1175 Acked 2174 Read 3 Done New Done true
4298 1176 Acked 2175 Read 2 Done New Done true
4299 1177 Acked 2176 Read 8 Done New Done true
4300 1178 Acked 2177 Read 5 Done New Done true
4301 1179 Acked 2178 Write 4 Done New Done true
4302 1180 Acked 2179 Write 4 Done New Done true
4303 1181 Acked 2180 Flush 0 Done New Done true
4304 1182 Acked 2181 Read 4 Done New Done true
4305 1183 Acked 2182 Write 1 Done New Done true
4306 1184 Acked 2183 Write 3 Done New Done true
4307 1185 Acked 2184 Flush 0 Done New Done true
4308 1186 Acked 2185 Read 8 Done New Done true
4309 1187 Acked 2186 Read 5 Done New Done true
4310 1188 Acked 2187 Read 10 Done New Done true
4311 1189 Acked 2188 Flush 0 Done New Done true
4312 1190 Acked 2189 Read 6 Done New Done true
4313 1191 Acked 2190 Write 4 Done New Done true
4314 1192 Acked 2191 Write 8 Done New Done true
4315 1193 Acked 2192 Write 6 Done New Done true
4316 1194 Acked 2193 Write 8 Done New Done true
4317 1195 Acked 2194 Read 2 Done New Done true
4318 1196 Acked 2195 Write 10 Done New Done true
4319 1197 Acked 2196 Read 5 Done New Done true
4320 1198 Acked 2197 Read 10 Done New Done true
4321 1199 Acked 2198 Write 6 Done New Done true
4322 1200 Acked 2199 Read 9 Done New Done true
4323 1201 Acked 2200 Write 6 Done New Done true
4324 1202 Acked 2201 Read 5 Done New Done true
4325 1203 Acked 2202 Read 9 Done New Done true
4326 1204 Acked 2203 Flush 0 Done New Done true
4327 STATES DS:0 DS:1 DS:2 TOTAL
4328 New 0 301 0 301
4329 Sent 0 0 0 0
4330 Done 301 0 301 602
4331 Skipped 0 0 0 0
4332 Error 0 0 0 0
4333 Last Flush: 2203 1902 2203
4334 Downstairs last five completed: 1902 1901 1900 1899 1898
4335 Upstairs last five completed: 1204 1203 1202 1201 1200
4336 CLIENT: Up:0 ds:301 act:2
43372024-02-27T21:45:28.055ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
43382024-02-27T21:45:28.055ZINFOcrucible: ds_connection connected = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
43392024-02-27T21:45:28.057ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
43402024-02-27T21:45:28.057ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
43412024-02-27T21:45:28.057ZINFOcrucible: send last flush ID to this DS: 1902 = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
43422024-02-27T21:45:28.057ZINFOcrucible: Replied this last flush ID: 1902 = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
43432024-02-27T21:45:28.057ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
43442024-02-27T21:45:28.057ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
43452024-02-27T21:45:28.057ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
43462024-02-27T21:45:28.057ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
4347 ----------------------------------------------------------------
4348 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
4349 Downstairs last five completed: 2203 2202 2201 2200 2199
4350 Upstairs last five completed: 1204 1203 1202 1201 1200
4351 CLIENT: Up:0 ds:0 act:3
4352 Replay: All jobs finished, all DS active.
4353 Main task has work to do, go find it
4354 got dsc Stop(2)
4355 stop 2
4356 Feb 27 21:45:31.390 INFO request completed, latency_us: 332, response_code: 204, uri: /stop/cid/2, method[8830] Got stop action so:false kr:false
4357 : POST, req_id: bdc4efae-eb19-4d14-8014-7e3cd1e2c014, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
43582024-02-27T21:45:31.391ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
43592024-02-27T21:45:31.391ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
43602024-02-27T21:45:31.391ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = e6a5a974-925a-494f-8845-facc441c2690
43612024-02-27T21:45:31.391ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
4362 Feb 27 21:45:31.391 {"msg":"[2] client re-new 0 jobs since flush 2203","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:31.39154487Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"downstairs","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
4363 INFO request completed, latency_us: 140, response_code: 200, uri: /state/cid/2, method: GET, req_id: 653d7d76-cf79-4aaf-b24c-fe6b9fd6bced, remote_addr: 127.0.0.1:44347, local_addr:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:31.391611181Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"io task","client":"2","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
4364 127.0.0.1:9998
4365 [8830] Exited with: ExitStatus(unix_wait_status(9))
4366 [8830][2] reports Exit
43672024-02-27T21:45:32.392ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
43682024-02-27T21:45:32.392ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
43692024-02-27T21:45:32.392ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
43702024-02-27T21:45:32.393ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
43712024-02-27T21:45:32.393ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
43722024-02-27T21:45:32.393ZINFOcrucible: [2] client re-new 0 jobs since flush 2203 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
43732024-02-27T21:45:32.393ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
43742024-02-27T21:45:33.394ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
43752024-02-27T21:45:33.394ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
43762024-02-27T21:45:33.394ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
43772024-02-27T21:45:33.394ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
43782024-02-27T21:45:33.394ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
43792024-02-27T21:45:33.394ZINFOcrucible: [2] client re-new 0 jobs since flush 2203 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
43802024-02-27T21:45:33.394ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
43812024-02-27T21:45:34.395ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
43822024-02-27T21:45:34.395ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
43832024-02-27T21:45:34.395ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
43842024-02-27T21:45:34.395ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
43852024-02-27T21:45:34.395ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
43862024-02-27T21:45:34.395ZINFOcrucible: [2] client re-new 0 jobs since flush 2203 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
43872024-02-27T21:45:34.395ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
4388 Feb 27 21:45:35.393 INFO request completed, latency_us: 278, response_code: 200, uri: /state/cid/2, method: GET, req_id: 765a2fea-efcb-40b1-9d56-90a3b7994337, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
4389 001/300 Write block 3660 len 28672 data: 2 2 2 2 2 3 3
4390 002/300 Read block 2352 len 20480
43912024-02-27T21:45:35.396ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
43922024-02-27T21:45:35.396ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
43932024-02-27T21:45:35.396ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
43942024-02-27T21:45:35.396ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
43952024-02-27T21:45:35.396ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
43962024-02-27T21:45:35.396ZINFOcrucible: [2] client re-new 2 jobs since flush 2203 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
43972024-02-27T21:45:35.396ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
4398 003/300 Write block 3976 len 32768 data: 3 4 4 4 3 3 3 3
4399 004/300 Flush
4400 005/300 Write block 1678 len 8192 data: 2 2
4401 006/300 Write block 3980 len 32768 data: 4 4 4 4 3 3 3 3
4402 007/300 Read block 4027 len 4096
4403 008/300 Write block 1254 len 28672 data: 2 2 2 2 2 2 2
4404 009/300 Read block 845 len 8192
4405 010/300 Write block 2870 len 12288 data: 2 2 2
4406 011/300 Read block 2059 len 12288
4407 012/300 Write block 4867 len 8192 data: 2 2
4408 013/300 Read block 1613 len 28672
4409 014/300 Write block 957 len 16384 data: 3 3 3 3
4410 015/300 Write block 2958 len 4096 data: 2
4411 016/300 Write block 3255 len 20480 data: 3 2 2 2 2
4412 017/300 Read block 4436 len 24576
4413 018/300 Write block 1045 len 8192 data: 2 2
4414 019/300 Read block 3705 len 12288
4415 020/300 Flush
4416 021/300 Write block 1529 len 4096 data: 2
4417 022/300 Read block 3806 len 16384
4418 023/300 Write block 2537 len 40960 data: 2 2 2 2 2 3 3 3 3 4
4419 024/300 Write block 3301 len 40960 data: 3 3 3 2 2 2 2 2 2 2
4420 025/300 Write block 614 len 4096 data: 3
4421 026/300 Flush
4422 027/300 Read block 4683 len 36864
4423 028/300 Read block 3873 len 24576
4424 029/300 Read block 2501 len 32768
4425 030/300 Write block 2598 len 20480 data: 2 2 2 2 2
4426 031/300 Write block 2277 len 24576 data: 3 3 3 3 2 2
4427 032/300 Write block 4873 len 24576 data: 2 2 2 2 2 2
4428 033/300 Read block 3243 len 8192
4429 034/300 Read block 3888 len 20480
4430 035/300 Write block 4437 len 20480 data: 2 2 2 2 2
4431 036/300 Write block 4035 len 8192 data: 2 2
4432 037/300 Write block 4775 len 20480 data: 2 2 2 2 2
4433 038/300 Read block 4266 len 32768
4434 039/300 Read block 935 len 24576
4435 040/300 Read block 1979 len 16384
4436 041/300 Read block 2145 len 24576
4437 042/300 Flush
4438 043/300 Read block 4811 len 40960
4439 044/300 Read block 2865 len 16384
4440 045/300 Read block 3650 len 24576
4441 046/300 Write block 3592 len 28672 data: 2 2 2 3 3 3 3
4442 047/300 Read block 2695 len 28672
4443 048/300 Read block 2039 len 4096
4444 049/300 Write block 3054 len 20480 data: 2 3 3 3 3
4445 050/300 Flush
4446 051/300 Write block 3662 len 8192 data: 3 3
4447 052/300 Read block 4661 len 16384
4448 053/300 Write block 2647 len 4096 data: 4
4449 054/300 Write block 674 len 16384 data: 3 3 3 3
4450 055/300 Write block 1441 len 16384 data: 3 3 4 4
4451 056/300 Write block 4885 len 12288 data: 3 3 3
4452 057/300 Write block 2002 len 4096 data: 2
4453 058/300 Read block 3057 len 24576
4454 059/300 Read block 2877 len 20480
4455 060/300 Read block 1786 len 36864
4456 061/300 Write block 837 len 8192 data: 2 2
4457 062/300 Read block 347 len 28672
4458 063/300 Read block 3918 len 28672
4459 064/300 Read block 3539 len 32768
4460 065/300 Write block 4550 len 40960 data: 3 3 3 2 2 2 2 2 2 2
4461 066/300 Write block 3193 len 32768 data: 2 2 2 2 2 2 2 2
4462 067/300 Read block 1193 len 36864
4463 068/300 Write block 2523 len 12288 data: 3 2 2
4464 069/300 Read block 682 len 36864
4465 070/300 Read block 4056 len 8192
4466 071/300 Read block 2823 len 40960
4467 072/300 Write block 392 len 24576 data: 2 2 2 2 2 2
4468 073/300 Write block 1785 len 8192 data: 3 3
4469 074/300 Read block 3678 len 4096
4470 075/300 Write block 2466 len 36864 data: 3 3 2 3 3 3 3 3 3
4471 076/300 Write block 1473 len 12288 data: 2 2 2
4472 077/300 Read block 3467 len 32768
4473 078/300 Read block 218 len 36864
4474 079/300 Write block 2318 len 28672 data: 4 3 2 2 2 2 2
4475 080/300 Read block 564 len 28672
4476 081/300 Write block 1361 len 28672 data: 3 3 3 3 3 3 3
4477 082/300 Write block 1975 len 40960 data: 4 4 4 3 3 3 3 3 2 2
4478 083/300 Read block 2858 len 16384
4479 084/300 Read block 4577 len 32768
4480 085/300 Write block 4352 len 8192 data: 2 2
4481 086/300 Read block 4821 len 16384
4482 087/300 Flush
4483 088/300 Write block 838 len 32768 data: 3 2 2 3 3 3 3 3
4484 089/300 Write block 1004 len 40960 data: 2 2 2 2 2 2 2 3 2 2
4485 090/300 Write block 2665 len 36864 data: 3 2 2 3 3 3 3 3 2
4486 091/300 Read block 3744 len 4096
4487 092/300 Read block 2006 len 32768
4488 093/300 Write block 2622 len 24576 data: 2 2 2 2 2 2
4489 094/300 Read block 847 len 40960
4490 095/300 Flush
4491 096/300 Write block 4313 len 36864 data: 2 2 2 2 2 2 2 3 3
4492 097/300 Flush
4493 098/300 Read block 2056 len 20480
4494 099/300 Write block 2254 len 8192 data: 2 3
4495 100/300 Flush
4496 101/300 Flush
4497 102/300 Write block 3830 len 12288 data: 2 2 2
4498 103/300 Read block 1153 len 20480
4499 104/300 Read block 3612 len 8192
4500 105/300 Read block 2583 len 24576
4501 106/300 Read block 4263 len 40960
4502 107/300 Read block 2673 len 24576
4503 108/300 Write block 2693 len 4096 data: 3
4504 109/300 Write block 1132 len 12288 data: 3 2 2
4505 110/300 Flush
4506 111/300 Read block 456 len 20480
4507 112/300 Read block 3582 len 16384
4508 113/300 Write block 3558 len 24576 data: 3 3 3 3 4 4
4509 114/300 Write block 3340 len 24576 data: 2 2 2 2 2 4
4510 115/300 Read block 2608 len 20480
4511 116/300 Read block 3944 len 20480
4512 117/300 Write block 3637 len 24576 data: 4 3 3 2 2 2
4513 118/300 Write block 4650 len 36864 data: 3 3 3 3 3 3 3 3 3
4514 119/300 Read block 2106 len 16384
4515 120/300 Write block 4103 len 12288 data: 2 2 2
4516 121/300 Flush
4517 122/300 Write block 1128 len 24576 data: 2 2 2 2 4 3
4518 123/300 Write block 1176 len 16384 data: 3 2 2 2
4519 124/300 Write block 1310 len 24576 data: 5 5 3 3 4 3
4520 125/300 Write block 4810 len 40960 data: 2 2 2 4 4 4 3 3 3 3
4521 126/300 Write block 2815 len 16384 data: 2 2 2 3
4522 127/300 Write block 3059 len 32768 data: 3 3 2 2 2 2 2 2
4523 128/300 Write block 3333 len 24576 data: 4 4 4 3 2 2
4524 129/300 Read block 1981 len 24576
4525 130/300 Write block 3407 len 28672 data: 2 2 3 4 3 3 3
4526 131/300 Write block 3783 len 32768 data: 2 2 2 2 3 3 3 3
4527 132/300 Write block 4628 len 32768 data: 3 3 3 3 3 3 2 2
4528 133/300 Read block 1976 len 4096
4529 134/300 Write block 2296 len 16384 data: 3 3 2 2
4530 135/300 Write block 1471 len 16384 data: 2 2 3 3
4531 136/300 Flush
4532 137/300 Read block 2080 len 32768
4533 138/300 Write block 90 len 20480 data: 2 3 3 3 3
4534 139/300 Write block 4686 len 8192 data: 3 3
4535 140/300 Read block 2734 len 24576
4536 141/300 Flush
4537 142/300 Flush
4538 143/300 Read block 2467 len 20480
4539 144/300 Write block 2998 len 24576 data: 2 2 3 2 2 2
4540 145/300 Write block 4075 len 4096 data: 2
4541 146/300 Read block 4657 len 12288
4542 147/300 Read block 551 len 12288
4543 148/300 Flush
4544 149/300 Read block 4807 len 12288
4545 150/300 Write block 4801 len 32768 data: 2 3 3 3 3 3 3 3
4546 151/300 Write block 3771 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4547 152/300 Read block 2012 len 4096
4548 153/300 Read block 2210 len 32768
4549 154/300 Read block 3061 len 16384
4550 155/300 Write block 1465 len 36864 data: 2 2 2 2 2 2 3 3 4
4551 156/300 Write block 2457 len 24576 data: 3 2 2 3 4 4
4552 157/300 Read block 1392 len 40960
4553 158/300 Read block 2288 len 20480
4554 159/300 Write block 4659 len 32768 data: 3 3 3 3 3 3 3 3
4555 160/300 Flush
4556 161/300 Read block 3659 len 28672
4557 162/300 Read block 4053 len 36864
4558 163/300 Read block 3660 len 40960
4559 164/300 Read block 2862 len 4096
4560 165/300 Read block 753 len 28672
4561 166/300 Read block 88 len 28672
4562 167/300 Write block 3272 len 12288 data: 3 3 4
4563 168/300 Write block 1713 len 28672 data: 3 3 3 3 3 3 3
4564 169/300 Write block 3641 len 16384 data: 3 3 2 2
4565 170/300 Read block 2513 len 36864
4566 171/300 Read block 3736 len 28672
4567 172/300 Write block 4025 len 28672 data: 4 3 3 3 3 3 3
4568 173/300 Read block 2032 len 40960
4569 174/300 Read block 4351 len 16384
4570 175/300 Read block 2908 len 12288
4571 176/300 Read block 2942 len 8192
4572 177/300 Write block 4 len 8192 data: 2 2
4573 178/300 Read block 1449 len 24576
4574 179/300 Read block 1138 len 20480
4575 180/300 Write block 3446 len 28672 data: 2 2 3 3 3 3 3
4576 181/300 Read block 1224 len 16384
4577 182/300 Read block 3690 len 40960
4578 183/300 Read block 3145 len 4096
4579 184/300 Read block 4025 len 8192
4580 185/300 Flush
4581 186/300 Read block 1134 len 16384
4582 187/300 Read block 2161 len 36864
4583 188/300 Write block 1226 len 40960 data: 3 3 3 3 3 3 2 2 2 2
4584 189/300 Read block 2759 len 36864
4585 190/300 Write block 1318 len 40960 data: 2 2 2 2 2 3 3 3 2 2
4586 191/300 Write block 4208 len 12288 data: 2 2 2
4587 192/300 Read block 156 len 28672
4588 193/300 Read block 970 len 16384
4589 194/300 Flush
4590 195/300 Write block 3539 len 24576 data: 2 2 2 2 2 2
4591 196/300 Read block 495 len 24576
4592 197/300 Read block 1696 len 28672
4593 198/300 Read block 3049 len 28672
4594 199/300 Read block 2433 len 24576
4595 200/300 Write block 3396 len 32768 data: 2 2 2 3 3 3 3 3
4596 201/300 Write block 2082 len 32768 data: 2 2 2 2 2 2 2 2
4597 202/300 Read block 4624 len 12288
4598 203/300 Flush
4599 204/300 Flush
4600 205/300 Write block 1658 len 40960 data: 5 5 5 3 4 4 5 3 3 4
4601 206/300 Read block 2164 len 4096
4602 207/300 Read block 2087 len 4096
4603 208/300 Read block 3984 len 32768
4604 209/300 Write block 2565 len 28672 data: 2 2 2 2 2 3 3
4605 210/300 Write block 2637 len 16384 data: 3 3 3 3
4606 211/300 Read block 2409 len 40960
4607 212/300 Read block 3584 len 32768
4608 213/300 Read block 2697 len 16384
4609 214/300 Write block 431 len 8192 data: 2 3
4610 215/300 Read block 2574 len 4096
4611 216/300 Read block 566 len 24576
4612 217/300 Flush
4613 218/300 Flush
4614 219/300 Write block 2572 len 32768 data: 3 3 3 3 3 3 2 2
4615 220/300 Write block 571 len 12288 data: 2 2 2
4616 221/300 Flush
4617 222/300 Write block 3510 len 8192 data: 2 2
4618 223/300 Write block 2191 len 40960 data: 3 3 4 3 2 2 2 2 2 2
4619 224/300 Write block 1806 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4620 225/300 Read block 761 len 32768
4621 226/300 Read block 4924 len 20480
4622 227/300 Flush
4623 228/300 Write block 906 len 8192 data: 2 3
4624 229/300 Read block 4576 len 32768
4625 230/300 Read block 2227 len 16384
4626 231/300 Read block 316 len 36864
4627 232/300 Read block 1403 len 28672
4628 233/300 Write block 1216 len 8192 data: 3 5
4629 234/300 Read block 2660 len 4096
4630 235/300 Read block 1076 len 12288
4631 236/300 Read block 3503 len 28672
4632 237/300 Read block 4838 len 8192
4633 238/300 Read block 4069 len 28672
4634 239/300 Write block 3355 len 8192 data: 2 2
4635 240/300 Write block 275 len 12288 data: 2 2 2
4636 241/300 Flush
4637 242/300 Read block 3547 len 28672
4638 243/300 Write block 394 len 36864 data: 3 3 3 3 2 2 2 2 2
4639 244/300 Write block 4148 len 28672 data: 3 3 3 2 2 2 2
4640 245/300 Flush
4641 246/300 Read block 390 len 36864
4642 247/300 Read block 1998 len 28672
4643 248/300 Read block 2084 len 8192
4644 249/300 Write block 2405 len 36864 data: 2 2 2 2 2 2 2 2 2
4645 250/300 Read block 2972 len 40960
4646 251/300 Read block 1097 len 40960
4647 252/300 Read block 3656 len 28672
4648 253/300 Write block 1288 len 36864 data: 3 3 3 3 3 3 2 2 2
4649 254/300 Write block 4365 len 12288 data: 4 4 5
4650 255/300 Write block 4397 len 16384 data: 3 3 3 3
4651 256/300 Write block 1945 len 32768 data: 3 3 3 3 2 2 2 2
4652 257/300 Write block 1089 len 12288 data: 3 4 5
4653 258/300 Read block 2521 len 16384
4654 259/300 Read block 2103 len 28672
4655 260/300 Read block 4570 len 16384
4656 261/300 Read block 2806 len 4096
4657 262/300 Read block 2189 len 40960
4658 263/300 Write block 65 len 40960 data: 2 2 2 2 3 2 2 2 2 2
4659 264/300 Write block 4379 len 12288 data: 2 2 2
4660 265/300 Write block 2554 len 12288 data: 2 2 2
4661 266/300 Read block 4916 len 24576
4662 267/300 Read block 1732 len 8192
4663 268/300 Write block 505 len 32768 data: 3 3 3 3 3 3 3 3
4664 269/300 Flush
4665 270/300 Write block 3809 len 8192 data: 2 2
4666 271/300 Read block 565 len 28672
4667 272/300 Flush
4668 273/300 Read block 2664 len 20480
4669 274/300 Read block 3008 len 36864
4670 275/300 Read block 2043 len 32768
4671 276/300 Read block 208 len 28672
4672 277/300 Write block 1928 len 32768 data: 3 3 3 3 2 2 2 2
4673 278/300 Read block 2020 len 12288
4674 279/300 Write block 2613 len 36864 data: 3 2 4 3 3 3 3 2 2
4675 280/300 Read block 3715 len 28672
4676 281/300 Read block 585 len 28672
4677 282/300 Read block 3191 len 12288
4678 283/300 Write block 1848 len 24576 data: 2 2 3 3 2 2
4679 284/300 Write block 4141 len 12288 data: 2 2 2
4680 285/300 Read block 2710 len 40960
4681 286/300 Write block 2587 len 28672 data: 2 2 2 2 2 2 2
4682 287/300 Write block 1400 len 16384 data: 3 3 3 2
4683 288/300 Read block 3193 len 8192
4684 289/300 Read block 1196 len 4096
4685 290/300 Read block 1360 len 40960
4686 291/300 Write block 3032 len 16384 data: 2 2 2 2
4687 292/300 Read block 2556 len 12288
4688 293/300 Write block 2692 len 8192 data: 3 4
4689 294/300 Read block 509 len 8192
4690 295/300 Write block 4978 len 36864 data: 2 2 2 3 3 3 3 3 2
4691 296/300 Read block 56 len 8192
4692 297/300 Read block 4207 len 24576
4693 298/300 Write block 3023 len 12288 data: 4 4 3
4694 299/300 Read block 3868 len 28672
4695 300/300 Write block 4934 len 12288 data: 3 2 2
4696 Main task has work to do, go find it
4697 got dsc Start(2)
4698 start 2
4699 Feb 27 21:45:35.740 INFO request completed, latency_us: 233, response_code: 204, uri: /start/cid/2, method: POST, req_id: 1651c32b-9166-45fd-9d6f-91d936c71691, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
4700 Replay: started 2, returned:Ok(())
4701 ----------------------------------------------------------------
4702 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
4703 States: Active Active Offline
4704 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
4705 1205 Acked 2204 Write 7 Done Done New true
4706 1206 Acked 2205 Read 5 Done Done New true
4707 1207 Acked 2206 Write 8 Done Done New false
4708 1208 Acked 2207 Flush 0 Done Done New false
4709 1209 Acked 2208 Write 2 Done Done New false
4710 1210 Acked 2209 Write 8 Done Done New false
4711 1211 Acked 2210 Read 1 Done Done New false
4712 1212 Acked 2211 Write 7 Done Done New false
4713 1213 Acked 2212 Read 2 Done Done New false
4714 1214 Acked 2213 Write 3 Done Done New false
4715 1215 Acked 2214 Read 3 Done Done New false
4716 1216 Acked 2215 Write 2 Done Done New false
4717 1217 Acked 2216 Read 7 Done Done New false
4718 1218 Acked 2217 Write 4 Done Done New false
4719 1219 Acked 2218 Write 1 Done Done New false
4720 1220 Acked 2219 Write 5 Done Done New false
4721 1221 Acked 2220 Read 6 Done Done New false
4722 1222 Acked 2221 Write 2 Done Done New false
4723 1223 Acked 2222 Read 3 Done Done New false
4724 1224 Acked 2223 Flush 0 Done Done New false
4725 1225 Acked 2224 Write 1 Done Done New false
4726 1226 Acked 2225 Read 4 Done Done New false
4727 1227 Acked 2226 Write 10 Done Done New false
4728 1228 Acked 2227 Write 10 Done Done New false
4729 1229 Acked 2228 Write 1 Done Done New false
4730 1230 Acked 2229 Flush 0 Done Done New false
4731 1231 Acked 2230 Read 9 Done Done New false
4732 1232 Acked 2231 Read 6 Done Done New false
4733 1233 Acked 2232 Read 8 Done Done New false
4734 1234 Acked 2233 Write 5 Done Done New false
4735 1235 Acked 2234 Write 6 Done Done New false
4736 1236 Acked 2235 Write 6 Done Done New false
4737 1237 Acked 2236 Read 2 Done Done New false
4738 1238 Acked 2237 Read 5 Done Done New false
4739 1239 Acked 2238 Write 5 Done Done New false
4740 1240 Acked 2239 Write 2 Done Done New false
4741 1241 Acked 2240 Write 5 Done Done New false
4742 1242 Acked 2241 Read 8 Done Done New false
4743 1243 Acked 2242 Read 6 Done Done New false
4744 1244 Acked 2243 Read 4 Done Done New false
4745 1245 Acked 2244 Read 6 Done Done New false
4746 1246 Acked 2245 Flush 0 Done Done New false
4747 1247 Acked 2246 Read 10 Done Done New false
4748 1248 Acked 2247 Read 4 Done Done New false
4749 1249 Acked 2248 Read 6 Done Done New false
4750 1250 Acked 2249 Write 7 Done Done New false
4751 1251 Acked 2250 Read 7 Done Done New false
4752 1252 Acked 2251 Read 1 Done Done New false
4753 1253 Acked 2252 Write 5 Done Done New false
4754 1254 Acked 2253 Flush 0 Done Done New false
4755 1255 Acked 2254 Write 2 Done Done New false
4756 1256 Acked 2255 Read 4 Done Done New false
4757 1257 Acked 2256 Write 1 Done Done New false
4758 1258 Acked 2257 Write 4 Done Done New false
4759 1259 Acked 2258 Write 4 Done Done New false
4760 1260 Acked 2259 Write 3 Done Done New false
4761 1261 Acked 2260 Write 1 Done Done New false
4762 1262 Acked 2261 Read 6 Done Done New false
4763 1263 Acked 2262 Read 5 Done Done New false
4764 1264 Acked 2263 Read 9 Done Done New false
4765 1265 Acked 2264 Write 2 Done Done New false
4766 1266 Acked 2265 Read 7 Done Done New false
4767 1267 Acked 2266 Read 7 Done Done New false
4768 1268 Acked 2267 Read 8 Done Done New false
4769 1269 Acked 2268 Write 10 Done Done New false
4770 1270 Acked 2269 Write 8 Done Done New false
4771 1271 Acked 2270 Read 9 Done Done New false
4772 1272 Acked 2271 Write 3 Done Done New false
4773 1273 Acked 2272 Read 9 Done Done New false
4774 1274 Acked 2273 Read 2 Done Done New false
4775 1275 Acked 2274 Read 10 Done Done New false
4776 1276 Acked 2275 Write 6 Done Done New false
4777 1277 Acked 2276 Write 2 Done Done New false
4778 1278 Acked 2277 Read 1 Done Done New false
4779 1279 Acked 2278 Write 9 Done Done New false
4780 1280 Acked 2279 Write 3 Done Done New false
4781 1281 Acked 2280 Read 8 Done Done New false
4782 1282 Acked 2281 Read 9 Done Done New false
4783 1283 Acked 2282 Write 7 Done Done New false
4784 1284 Acked 2283 Read 7 Done Done New false
4785 1285 Acked 2284 Write 7 Done Done New false
4786 1286 Acked 2285 Write 10 Done Done New false
4787 1287 Acked 2286 Read 4 Done Done New false
4788 1288 Acked 2287 Read 8 Done Done New false
4789 1289 Acked 2288 Write 2 Done Done New false
4790 1290 Acked 2289 Read 4 Done Done New false
4791 1291 Acked 2290 Flush 0 Done Done New false
4792 1292 Acked 2291 Write 8 Done Done New false
4793 1293 Acked 2292 Write 10 Done Done New false
4794 1294 Acked 2293 Write 9 Done Done New false
4795 1295 Acked 2294 Read 1 Done Done New false
4796 1296 Acked 2295 Read 8 Done Done New false
4797 1297 Acked 2296 Write 6 Done Done New false
4798 1298 Acked 2297 Read 10 Done Done New false
4799 1299 Acked 2298 Flush 0 Done Done New false
4800 1300 Acked 2299 Write 9 Done Done New false
4801 1301 Acked 2300 Flush 0 Done Done New false
4802 1302 Acked 2301 Read 5 Done Done New false
4803 1303 Acked 2302 Write 2 Done Done New false
4804 1304 Acked 2303 Flush 0 Done Done New false
4805 1305 Acked 2304 Flush 0 Done Done New false
4806 1306 Acked 2305 Write 3 Done Done New false
4807 1307 Acked 2306 Read 5 Done Done New false
4808 1308 Acked 2307 Read 2 Done Done New false
4809 1309 Acked 2308 Read 6 Done Done New false
4810 1310 Acked 2309 Read 10 Done Done New false
4811 1311 Acked 2310 Read 6 Done Done New false
4812 1312 Acked 2311 Write 1 Done Done New false
4813 1313 Acked 2312 Write 3 Done Done New false
4814 1314 Acked 2313 Flush 0 Done Done New false
4815 1315 Acked 2314 Read 5 Done Done New false
4816 1316 Acked 2315 Read 4 Done Done New false
4817 1317 Acked 2316 Write 6 Done Done New false
4818 1318 Acked 2317 Write 6 Done Done New false
4819 1319 Acked 2318 Read 5 Done Done New false
4820 1320 Acked 2319 Read 5 Done Done New false
4821 1321 Acked 2320 Write 6 Done Done New false
4822 1322 Acked 2321 Write 9 Done Done New false
4823 1323 Acked 2322 Read 4 Done Done New false
4824 1324 Acked 2323 Write 3 Done Done New false
4825 1325 Acked 2324 Flush 0 Done Done New false
4826 1326 Acked 2325 Write 6 Done Done New false
4827 1327 Acked 2326 Flush 0 Done Done New false
4828 1328 Acked 2327 Write 4 Done Done New false
4829 1329 Acked 2328 Write 6 Done Done New false
4830 1330 Acked 2329 Write 10 Done Done New false
4831 1331 Acked 2330 Write 4 Done Done New false
4832 1332 Acked 2331 Write 8 Done Done New false
4833 1333 Acked 2332 Write 6 Done Done New false
4834 1334 Acked 2333 Read 6 Done Done New false
4835 1335 Acked 2334 Write 7 Done Done New false
4836 1336 Acked 2335 Write 8 Done Done New false
4837 1337 Acked 2336 Write 8 Done Done New false
4838 1338 Acked 2337 Read 1 Done Done New false
4839 1339 Acked 2338 Write 4 Done Done New false
4840 1340 Acked 2339 Write 4 Done Done New false
4841 1341 Acked 2340 Flush 0 Done Done New false
4842 1342 Acked 2341 Read 8 Done Done New false
4843 1343 Acked 2342 Write 5 Done Done New false
4844 1344 Acked 2343 Write 2 Done Done New false
4845 1345 Acked 2344 Read 6 Done Done New false
4846 1346 Acked 2345 Flush 0 Done Done New false
4847 1347 Acked 2346 Flush 0 Done Done New false
4848 1348 Acked 2347 Read 5 Done Done New false
4849 1349 Acked 2348 Write 6 Done Done New false
4850 1350 Acked 2349 Write 1 Done Done New false
4851 1351 Acked 2350 Read 3 Done Done New false
4852 1352 Acked 2351 Read 3 Done Done New false
4853 1353 Acked 2352 Flush 0 Done Done New false
4854 1354 Acked 2353 Read 3 Done Done New false
4855 1355 Acked 2354 Write 8 Done Done New false
4856 1356 Acked 2355 Write 10 Done Done New false
4857 1357 Acked 2356 Read 1 Done Done New false
4858 1358 Acked 2357 Read 8 Done Done New false
4859 1359 Acked 2358 Read 4 Done Done New false
4860 1360 Acked 2359 Write 9 Done Done New false
4861 1361 Acked 2360 Write 6 Done Done New false
4862 1362 Acked 2361 Read 10 Done Done New false
4863 1363 Acked 2362 Read 5 Done Done New false
4864 1364 Acked 2363 Write 8 Done Done New false
4865 1365 Acked 2364 Flush 0 Done Done New false
4866 1366 Acked 2365 Read 7 Done Done New false
4867 1367 Acked 2366 Read 9 Done Done New false
4868 1368 Acked 2367 Read 10 Done Done New false
4869 1369 Acked 2368 Read 1 Done Done New false
4870 1370 Acked 2369 Read 7 Done Done New false
4871 1371 Acked 2370 Read 7 Done Done New false
4872 1372 Acked 2371 Write 3 Done Done New false
4873 1373 Acked 2372 Write 7 Done Done New false
4874 1374 Acked 2373 Write 4 Done Done New false
4875 1375 Acked 2374 Read 9 Done Done New false
4876 1376 Acked 2375 Read 7 Done Done New false
4877 1377 Acked 2376 Write 7 Done Done New false
4878 1378 Acked 2377 Read 10 Done Done New false
4879 1379 Acked 2378 Read 4 Done Done New false
4880 1380 Acked 2379 Read 3 Done Done New false
4881 1381 Acked 2380 Read 2 Done Done New false
4882 1382 Acked 2381 Write 2 Done Done New false
4883 1383 Acked 2382 Read 6 Done Done New false
4884 1384 Acked 2383 Read 5 Done Done New false
4885 1385 Acked 2384 Write 7 Done Done New false
4886 1386 Acked 2385 Read 4 Done Done New false
4887 1387 Acked 2386 Read 10 Done Done New false
4888 1388 Acked 2387 Read 1 Done Done New false
4889 1389 Acked 2388 Read 2 Done Done New false
4890 1390 Acked 2389 Flush 0 Done Done New false
4891 1391 Acked 2390 Read 4 Done Done New false
4892 1392 Acked 2391 Read 9 Done Done New false
4893 1393 Acked 2392 Write 10 Done Done New false
4894 1394 Acked 2393 Read 9 Done Done New false
4895 1395 Acked 2394 Write 10 Done Done New false
4896 1396 Acked 2395 Write 3 Done Done New false
4897 1397 Acked 2396 Read 7 Done Done New false
4898 1398 Acked 2397 Read 4 Done Done New false
4899 1399 Acked 2398 Flush 0 Done Done New false
4900 1400 Acked 2399 Write 6 Done Done New false
4901 1401 Acked 2400 Read 6 Done Done New false
4902 1402 Acked 2401 Read 7 Done Done New false
4903 1403 Acked 2402 Read 7 Done Done New false
4904 1404 Acked 2403 Read 6 Done Done New false
4905 1405 Acked 2404 Write 8 Done Done New false
4906 1406 Acked 2405 Write 8 Done Done New false
4907 1407 Acked 2406 Read 3 Done Done New false
4908 1408 Acked 2407 Flush 0 Done Done New false
4909 1409 Acked 2408 Flush 0 Done Done New false
4910 1410 Acked 2409 Write 10 Done Done New false
4911 1411 Acked 2410 Read 1 Done Done New false
4912 1412 Acked 2411 Read 1 Done Done New false
4913 1413 Acked 2412 Read 8 Done Done New false
4914 1414 Acked 2413 Write 7 Done Done New false
4915 1415 Acked 2414 Write 4 Done Done New false
4916 1416 Acked 2415 Read 10 Done Done New false
4917 1417 Acked 2416 Read 8 Done Done New false
4918 1418 Acked 2417 Read 4 Done Done New false
4919 1419 Acked 2418 Write 2 Done Done New false
4920 1420 Acked 2419 Read 1 Done Done New false
4921 1421 Acked 2420 Read 6 Done Done New false
4922 1422 Acked 2421 Flush 0 Done Done New false
4923 1423 Acked 2422 Flush 0 Done Done New false
4924 1424 Acked 2423 Write 8 Done Done New false
4925 1425 Acked 2424 Write 3 Done Done New false
4926 1426 Acked 2425 Flush 0 Done Done New false
4927 1427 Acked 2426 Write 2 Done Done New false
4928 1428 Acked 2427 Write 10 Done Done New false
4929 1429 Acked 2428 Write 10 Done Done New false
4930 1430 Acked 2429 Read 8 Done Done New false
4931 1431 Acked 2430 Read 5 Done Done New false
4932 1432 Acked 2431 Flush 0 Done Done New false
4933 1433 Acked 2432 Write 2 Done Done New false
4934 1434 Acked 2433 Read 8 Done Done New false
4935 1435 Acked 2434 Read 4 Done Done New false
4936 1436 Acked 2435 Read 9 Done Done New false
4937 1437 Acked 2436 Read 7 Done Done New false
4938 1438 Acked 2437 Write 2 Done Done New false
4939 1439 Acked 2438 Read 1 Done Done New false
4940 1440 Acked 2439 Read 3 Done Done New false
4941 1441 Acked 2440 Read 7 Done Done New false
4942 1442 Acked 2441 Read 2 Done Done New false
4943 1443 Acked 2442 Read 7 Done Done New false
4944 1444 Acked 2443 Write 2 Done Done New false
4945 1445 Acked 2444 Write 3 Done Done New false
4946 1446 Acked 2445 Flush 0 Done Done New false
4947 1447 Acked 2446 Read 7 Done Done New false
4948 1448 Acked 2447 Write 9 Done Done New false
4949 1449 Acked 2448 Write 7 Done Done New false
4950 1450 Acked 2449 Flush 0 Done Done New false
4951 1451 Acked 2450 Read 9 Done Done New false
4952 1452 Acked 2451 Read 7 Done Done New false
4953 1453 Acked 2452 Read 2 Done Done New false
4954 1454 Acked 2453 Write 9 Done Done New false
4955 1455 Acked 2454 Read 10 Done Done New false
4956 1456 Acked 2455 Read 10 Done Done New false
4957 1457 Acked 2456 Read 7 Done Done New false
4958 1458 Acked 2457 Write 9 Done Done New false
4959 1459 Acked 2458 Write 3 Done Done New false
4960 1460 Acked 2459 Write 4 Done Done New false
4961 1461 Acked 2460 Write 8 Done Done New false
4962 1462 Acked 2461 Write 3 Done Done New false
4963 1463 Acked 2462 Read 4 Done Done New false
4964 1464 Acked 2463 Read 7 Done Done New false
4965 1465 Acked 2464 Read 4 Done Done New false
4966 1466 Acked 2465 Read 1 Done Done New false
4967 1467 Acked 2466 Read 10 Done Done New false
4968 1468 Acked 2467 Write 10 Done Done New false
4969 1469 Acked 2468 Write 3 Done Done New false
4970 1470 Acked 2469 Write 3 Done Done New false
4971 1471 Acked 2470 Read 6 Done Done New false
4972 1472 Acked 2471 Read 2 Done Done New false
4973 1473 Acked 2472 Write 8 Done Done New false
4974 1474 Acked 2473 Flush 0 Done Done New false
4975 1475 Acked 2474 Write 2 Done Done New false
4976 1476 Acked 2475 Read 7 Done Done New false
4977 1477 Acked 2476 Flush 0 Done Done New false
4978 1478 Acked 2477 Read 5 Done Done New false
4979 1479 Acked 2478 Read 9 Done Done New false
4980 1480 Acked 2479 Read 8 Done Done New false
4981 1481 Acked 2480 Read 7 Done Done New false
4982 1482 Acked 2481 Write 8 Done Done New false
4983 1483 Acked 2482 Read 3 Done Done New false
4984 1484 Acked 2483 Write 9 Done Done New false
4985 1485 Acked 2484 Read 7 Done Done New false
4986 1486 Acked 2485 Read 7 Done Done New false
4987 1487 Acked 2486 Read 3 Done Done New false
4988 1488 Acked 2487 Write 6 Done Done New false
4989 1489 Acked 2488 Write 3 Done Done New false
4990 1490 Acked 2489 Read 10 Done Done New false
4991 1491 Acked 2490 Write 7 Done Done New false
4992 1492 Acked 2491 Write 4 Done Done New false
4993 1493 Acked 2492 Read 2 Done Done New false
4994 1494 Acked 2493 Read 1 Done Done New false
4995 1495 Acked 2494 Read 10 Done Done New false
4996 1496 Acked 2495 Write 4 Done Done New false
4997 1497 Acked 2496 Read 3 Done Done New false
4998 1498 Acked 2497 Write 2 Done Done New false
4999 1499 Acked 2498 Read 2 Done Done New false
5000 1500 Acked 2499 Write 9 Done Done New false
5001 1501 Acked 2500 Read 2 Done Done New false
5002 1502 Acked 2501 Read 6 Done Done New false
5003 1503 Acked 2502 Write 3 Done Done New false
5004 1504 Acked 2503 Read 7 Done Done New false
5005 1505 Acked 2504 Write 3 Done Done New false
5006 STATES DS:0 DS:1 DS:2 TOTAL
5007 New 0 0 301 301
5008 Sent 0 0 0 0
5009 Done 301 301 0 602
5010 Skipped 0 0 0 0
5011 Error 0 0 0 0
5012 Last Flush: 2476 2476 2203
5013 Downstairs last five completed: 2203 2202 2201 2200 2199
5014 Upstairs last five completed: 1505 1504 1503 1502 1501
5015 CLIENT: Up:0 ds:301 act:2
50162024-02-27T21:45:36.397ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
50172024-02-27T21:45:36.397ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
50182024-02-27T21:45:36.397ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
50192024-02-27T21:45:36.397ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
50202024-02-27T21:45:36.397ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
50212024-02-27T21:45:36.397ZINFOcrucible: [2] client re-new 302 jobs since flush 2203 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
50222024-02-27T21:45:36.397ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
50232024-02-27T21:45:37.399ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
50242024-02-27T21:45:37.399ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
50252024-02-27T21:45:37.399ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
50262024-02-27T21:45:37.399ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
50272024-02-27T21:45:37.399ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
50282024-02-27T21:45:37.399ZINFOcrucible: [2] client re-new 302 jobs since flush 2203 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
50292024-02-27T21:45:37.399ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
50302024-02-27T21:45:38.401ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
50312024-02-27T21:45:38.401ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
50322024-02-27T21:45:38.401ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
50332024-02-27T21:45:38.401ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
50342024-02-27T21:45:38.401ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
50352024-02-27T21:45:38.401ZINFOcrucible: [2] client re-new 302 jobs since flush 2203 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
50362024-02-27T21:45:38.401ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
50372024-02-27T21:45:39.402ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
50382024-02-27T21:45:39.402ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
50392024-02-27T21:45:39.402ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
50402024-02-27T21:45:39.402ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
50412024-02-27T21:45:39.402ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
50422024-02-27T21:45:39.402ZINFOcrucible: [2] client re-new 302 jobs since flush 2203 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
50432024-02-27T21:45:39.402ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
5044 ----------------------------------------------------------------
5045 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
5046 States: Active Active Offline
5047 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
5048 1205 Acked 2204 Write 7 Done Done New true
5049 1206 Acked 2205 Read 5 Done Done New true
5050 1207 Acked 2206 Write 8 Done Done New true
5051 1208 Acked 2207 Flush 0 Done Done New true
5052 1209 Acked 2208 Write 2 Done Done New true
5053 1210 Acked 2209 Write 8 Done Done New true
5054 1211 Acked 2210 Read 1 Done Done New true
5055 1212 Acked 2211 Write 7 Done Done New true
5056 1213 Acked 2212 Read 2 Done Done New true
5057 1214 Acked 2213 Write 3 Done Done New true
5058 1215 Acked 2214 Read 3 Done Done New true
5059 1216 Acked 2215 Write 2 Done Done New true
5060 1217 Acked 2216 Read 7 Done Done New true
5061 1218 Acked 2217 Write 4 Done Done New true
5062 1219 Acked 2218 Write 1 Done Done New true
5063 1220 Acked 2219 Write 5 Done Done New true
5064 1221 Acked 2220 Read 6 Done Done New true
5065 1222 Acked 2221 Write 2 Done Done New true
5066 1223 Acked 2222 Read 3 Done Done New true
5067 1224 Acked 2223 Flush 0 Done Done New true
5068 1225 Acked 2224 Write 1 Done Done New true
5069 1226 Acked 2225 Read 4 Done Done New true
5070 1227 Acked 2226 Write 10 Done Done New true
5071 1228 Acked 2227 Write 10 Done Done New true
5072 1229 Acked 2228 Write 1 Done Done New true
5073 1230 Acked 2229 Flush 0 Done Done New true
5074 1231 Acked 2230 Read 9 Done Done New true
5075 1232 Acked 2231 Read 6 Done Done New true
5076 1233 Acked 2232 Read 8 Done Done New true
5077 1234 Acked 2233 Write 5 Done Done New true
5078 1235 Acked 2234 Write 6 Done Done New true
5079 1236 Acked 2235 Write 6 Done Done New true
5080 1237 Acked 2236 Read 2 Done Done New true
5081 1238 Acked 2237 Read 5 Done Done New true
5082 1239 Acked 2238 Write 5 Done Done New true
5083 1240 Acked 2239 Write 2 Done Done New true
5084 1241 Acked 2240 Write 5 Done Done New true
5085 1242 Acked 2241 Read 8 Done Done New true
5086 1243 Acked 2242 Read 6 Done Done New true
5087 1244 Acked 2243 Read 4 Done Done New true
5088 1245 Acked 2244 Read 6 Done Done New true
5089 1246 Acked 2245 Flush 0 Done Done New true
5090 1247 Acked 2246 Read 10 Done Done New true
5091 1248 Acked 2247 Read 4 Done Done New true
5092 1249 Acked 2248 Read 6 Done Done New true
5093 1250 Acked 2249 Write 7 Done Done New true
5094 1251 Acked 2250 Read 7 Done Done New true
5095 1252 Acked 2251 Read 1 Done Done New true
5096 1253 Acked 2252 Write 5 Done Done New true
5097 1254 Acked 2253 Flush 0 Done Done New true
5098 1255 Acked 2254 Write 2 Done Done New true
5099 1256 Acked 2255 Read 4 Done Done New true
5100 1257 Acked 2256 Write 1 Done Done New true
5101 1258 Acked 2257 Write 4 Done Done New true
5102 1259 Acked 2258 Write 4 Done Done New true
5103 1260 Acked 2259 Write 3 Done Done New true
5104 1261 Acked 2260 Write 1 Done Done New true
5105 1262 Acked 2261 Read 6 Done Done New true
5106 1263 Acked 2262 Read 5 Done Done New true
5107 1264 Acked 2263 Read 9 Done Done New true
5108 1265 Acked 2264 Write 2 Done Done New true
5109 1266 Acked 2265 Read 7 Done Done New true
5110 1267 Acked 2266 Read 7 Done Done New true
5111 1268 Acked 2267 Read 8 Done Done New true
5112 1269 Acked 2268 Write 10 Done Done New true
5113 1270 Acked 2269 Write 8 Done Done New true
5114 1271 Acked 2270 Read 9 Done Done New true
5115 1272 Acked 2271 Write 3 Done Done New true
5116 1273 Acked 2272 Read 9 Done Done New true
5117 1274 Acked 2273 Read 2 Done Done New true
5118 1275 Acked 2274 Read 10 Done Done New true
5119 1276 Acked 2275 Write 6 Done Done New true
5120 1277 Acked 2276 Write 2 Done Done New true
5121 1278 Acked 2277 Read 1 Done Done New true
5122 1279 Acked 2278 Write 9 Done Done New true
5123 1280 Acked 2279 Write 3 Done Done New true
5124 1281 Acked 2280 Read 8 Done Done New true
5125 1282 Acked 2281 Read 9 Done Done New true
5126 1283 Acked 2282 Write 7 Done Done New true
5127 1284 Acked 2283 Read 7 Done Done New true
5128 1285 Acked 2284 Write 7 Done Done New true
5129 1286 Acked 2285 Write 10 Done Done New true
5130 1287 Acked 2286 Read 4 Done Done New true
5131 1288 Acked 2287 Read 8 Done Done New true
5132 1289 Acked 2288 Write 2 Done Done New true
5133 1290 Acked 2289 Read 4 Done Done New true
5134 1291 Acked 2290 Flush 0 Done Done New true
5135 1292 Acked 2291 Write 8 Done Done New true
5136 1293 Acked 2292 Write 10 Done Done New true
5137 1294 Acked 2293 Write 9 Done Done New true
5138 1295 Acked 2294 Read 1 Done Done New true
5139 1296 Acked 2295 Read 8 Done Done New true
5140 1297 Acked 2296 Write 6 Done Done New true
5141 1298 Acked 2297 Read 10 Done Done New true
5142 1299 Acked 2298 Flush 0 Done Done New true
5143 1300 Acked 2299 Write 9 Done Done New true
5144 1301 Acked 2300 Flush 0 Done Done New true
5145 1302 Acked 2301 Read 5 Done Done New true
5146 1303 Acked 2302 Write 2 Done Done New true
5147 1304 Acked 2303 Flush 0 Done Done New true
5148 1305 Acked 2304 Flush 0 Done Done New true
5149 1306 Acked 2305 Write 3 Done Done New true
5150 1307 Acked 2306 Read 5 Done Done New true
5151 1308 Acked 2307 Read 2 Done Done New true
5152 1309 Acked 2308 Read 6 Done Done New true
5153 1310 Acked 2309 Read 10 Done Done New true
5154 1311 Acked 2310 Read 6 Done Done New true
5155 1312 Acked 2311 Write 1 Done Done New true
5156 1313 Acked 2312 Write 3 Done Done New true
5157 1314 Acked 2313 Flush 0 Done Done New true
5158 1315 Acked 2314 Read 5 Done Done New true
5159 1316 Acked 2315 Read 4 Done Done New true
5160 1317 Acked 2316 Write 6 Done Done New true
5161 1318 Acked 2317 Write 6 Done Done New true
5162 1319 Acked 2318 Read 5 Done Done New true
5163 1320 Acked 2319 Read 5 Done Done New true
5164 1321 Acked 2320 Write 6 Done Done New true
5165 1322 Acked 2321 Write 9 Done Done New true
5166 1323 Acked 2322 Read 4 Done Done New true
5167 1324 Acked 2323 Write 3 Done Done New true
5168 1325 Acked 2324 Flush 0 Done Done New true
5169 1326 Acked 2325 Write 6 Done Done New true
5170 1327 Acked 2326 Flush 0 Done Done New true
5171 1328 Acked 2327 Write 4 Done Done New true
5172 1329 Acked 2328 Write 6 Done Done New true
5173 1330 Acked 2329 Write 10 Done Done New true
5174 1331 Acked 2330 Write 4 Done Done New true
5175 1332 Acked 2331 Write 8 Done Done New true
5176 1333 Acked 2332 Write 6 Done Done New true
5177 1334 Acked 2333 Read 6 Done Done New true
5178 1335 Acked 2334 Write 7 Done Done New true
5179 1336 Acked 2335 Write 8 Done Done New true
5180 1337 Acked 2336 Write 8 Done Done New true
5181 1338 Acked 2337 Read 1 Done Done New true
5182 1339 Acked 2338 Write 4 Done Done New true
5183 1340 Acked 2339 Write 4 Done Done New true
5184 1341 Acked 2340 Flush 0 Done Done New true
5185 1342 Acked 2341 Read 8 Done Done New true
5186 1343 Acked 2342 Write 5 Done Done New true
5187 1344 Acked 2343 Write 2 Done Done New true
5188 1345 Acked 2344 Read 6 Done Done New true
5189 1346 Acked 2345 Flush 0 Done Done New true
5190 1347 Acked 2346 Flush 0 Done Done New true
5191 1348 Acked 2347 Read 5 Done Done New true
5192 1349 Acked 2348 Write 6 Done Done New true
5193 1350 Acked 2349 Write 1 Done Done New true
5194 1351 Acked 2350 Read 3 Done Done New true
5195 1352 Acked 2351 Read 3 Done Done New true
5196 1353 Acked 2352 Flush 0 Done Done New true
5197 1354 Acked 2353 Read 3 Done Done New true
5198 1355 Acked 2354 Write 8 Done Done New true
5199 1356 Acked 2355 Write 10 Done Done New true
5200 1357 Acked 2356 Read 1 Done Done New true
5201 1358 Acked 2357 Read 8 Done Done New true
5202 1359 Acked 2358 Read 4 Done Done New true
5203 1360 Acked 2359 Write 9 Done Done New true
5204 1361 Acked 2360 Write 6 Done Done New true
5205 1362 Acked 2361 Read 10 Done Done New true
5206 1363 Acked 2362 Read 5 Done Done New true
5207 1364 Acked 2363 Write 8 Done Done New true
5208 1365 Acked 2364 Flush 0 Done Done New true
5209 1366 Acked 2365 Read 7 Done Done New true
5210 1367 Acked 2366 Read 9 Done Done New true
5211 1368 Acked 2367 Read 10 Done Done New true
5212 1369 Acked 2368 Read 1 Done Done New true
5213 1370 Acked 2369 Read 7 Done Done New true
5214 1371 Acked 2370 Read 7 Done Done New true
5215 1372 Acked 2371 Write 3 Done Done New true
5216 1373 Acked 2372 Write 7 Done Done New true
5217 1374 Acked 2373 Write 4 Done Done New true
5218 1375 Acked 2374 Read 9 Done Done New true
5219 1376 Acked 2375 Read 7 Done Done New true
5220 1377 Acked 2376 Write 7 Done Done New true
5221 1378 Acked 2377 Read 10 Done Done New true
5222 1379 Acked 2378 Read 4 Done Done New true
5223 1380 Acked 2379 Read 3 Done Done New true
5224 1381 Acked 2380 Read 2 Done Done New true
5225 1382 Acked 2381 Write 2 Done Done New true
5226 1383 Acked 2382 Read 6 Done Done New true
5227 1384 Acked 2383 Read 5 Done Done New true
5228 1385 Acked 2384 Write 7 Done Done New true
5229 1386 Acked 2385 Read 4 Done Done New true
5230 1387 Acked 2386 Read 10 Done Done New true
5231 1388 Acked 2387 Read 1 Done Done New true
5232 1389 Acked 2388 Read 2 Done Done New true
5233 1390 Acked 2389 Flush 0 Done Done New true
5234 1391 Acked 2390 Read 4 Done Done New true
5235 1392 Acked 2391 Read 9 Done Done New true
5236 1393 Acked 2392 Write 10 Done Done New true
5237 1394 Acked 2393 Read 9 Done Done New true
5238 1395 Acked 2394 Write 10 Done Done New true
5239 1396 Acked 2395 Write 3 Done Done New true
5240 1397 Acked 2396 Read 7 Done Done New true
5241 1398 Acked 2397 Read 4 Done Done New true
5242 1399 Acked 2398 Flush 0 Done Done New true
5243 1400 Acked 2399 Write 6 Done Done New true
5244 1401 Acked 2400 Read 6 Done Done New true
5245 1402 Acked 2401 Read 7 Done Done New true
5246 1403 Acked 2402 Read 7 Done Done New true
5247 1404 Acked 2403 Read 6 Done Done New true
5248 1405 Acked 2404 Write 8 Done Done New true
5249 1406 Acked 2405 Write 8 Done Done New true
5250 1407 Acked 2406 Read 3 Done Done New true
5251 1408 Acked 2407 Flush 0 Done Done New true
5252 1409 Acked 2408 Flush 0 Done Done New true
5253 1410 Acked 2409 Write 10 Done Done New true
5254 1411 Acked 2410 Read 1 Done Done New true
5255 1412 Acked 2411 Read 1 Done Done New true
5256 1413 Acked 2412 Read 8 Done Done New true
5257 1414 Acked 2413 Write 7 Done Done New true
5258 1415 Acked 2414 Write 4 Done Done New true
5259 1416 Acked 2415 Read 10 Done Done New true
5260 1417 Acked 2416 Read 8 Done Done New true
5261 1418 Acked 2417 Read 4 Done Done New true
5262 1419 Acked 2418 Write 2 Done Done New true
5263 1420 Acked 2419 Read 1 Done Done New true
5264 1421 Acked 2420 Read 6 Done Done New true
5265 1422 Acked 2421 Flush 0 Done Done New true
5266 1423 Acked 2422 Flush 0 Done Done New true
5267 1424 Acked 2423 Write 8 Done Done New true
5268 1425 Acked 2424 Write 3 Done Done New true
5269 1426 Acked 2425 Flush 0 Done Done New true
5270 1427 Acked 2426 Write 2 Done Done New true
5271 1428 Acked 2427 Write 10 Done Done New true
5272 1429 Acked 2428 Write 10 Done Done New true
5273 1430 Acked 2429 Read 8 Done Done New true
5274 1431 Acked 2430 Read 5 Done Done New true
5275 1432 Acked 2431 Flush 0 Done Done New true
5276 1433 Acked 2432 Write 2 Done Done New true
5277 1434 Acked 2433 Read 8 Done Done New true
5278 1435 Acked 2434 Read 4 Done Done New true
5279 1436 Acked 2435 Read 9 Done Done New true
5280 1437 Acked 2436 Read 7 Done Done New true
5281 1438 Acked 2437 Write 2 Done Done New true
5282 1439 Acked 2438 Read 1 Done Done New true
5283 1440 Acked 2439 Read 3 Done Done New true
5284 1441 Acked 2440 Read 7 Done Done New true
5285 1442 Acked 2441 Read 2 Done Done New true
5286 1443 Acked 2442 Read 7 Done Done New true
5287 1444 Acked 2443 Write 2 Done Done New true
5288 1445 Acked 2444 Write 3 Done Done New true
5289 1446 Acked 2445 Flush 0 Done Done New true
5290 1447 Acked 2446 Read 7 Done Done New true
5291 1448 Acked 2447 Write 9 Done Done New true
5292 1449 Acked 2448 Write 7 Done Done New true
5293 1450 Acked 2449 Flush 0 Done Done New true
5294 1451 Acked 2450 Read 9 Done Done New true
5295 1452 Acked 2451 Read 7 Done Done New true
5296 1453 Acked 2452 Read 2 Done Done New true
5297 1454 Acked 2453 Write 9 Done Done New true
5298 1455 Acked 2454 Read 10 Done Done New true
5299 1456 Acked 2455 Read 10 Done Done New true
5300 1457 Acked 2456 Read 7 Done Done New true
5301 1458 Acked 2457 Write 9 Done Done New true
5302 1459 Acked 2458 Write 3 Done Done New true
5303 1460 Acked 2459 Write 4 Done Done New true
5304 1461 Acked 2460 Write 8 Done Done New true
5305 1462 Acked 2461 Write 3 Done Done New true
5306 1463 Acked 2462 Read 4 Done Done New true
5307 1464 Acked 2463 Read 7 Done Done New true
5308 1465 Acked 2464 Read 4 Done Done New true
5309 1466 Acked 2465 Read 1 Done Done New true
5310 1467 Acked 2466 Read 10 Done Done New true
5311 1468 Acked 2467 Write 10 Done Done New true
5312 1469 Acked 2468 Write 3 Done Done New true
5313 1470 Acked 2469 Write 3 Done Done New true
5314 1471 Acked 2470 Read 6 Done Done New true
5315 1472 Acked 2471 Read 2 Done Done New true
5316 1473 Acked 2472 Write 8 Done Done New true
5317 1474 Acked 2473 Flush 0 Done Done New true
5318 1475 Acked 2474 Write 2 Done Done New true
5319 1476 Acked 2475 Read 7 Done Done New true
5320 1477 Acked 2476 Flush 0 Done Done New true
5321 1478 Acked 2477 Read 5 Done Done New true
5322 1479 Acked 2478 Read 9 Done Done New true
5323 1480 Acked 2479 Read 8 Done Done New true
5324 1481 Acked 2480 Read 7 Done Done New true
5325 1482 Acked 2481 Write 8 Done Done New true
5326 1483 Acked 2482 Read 3 Done Done New true
5327 1484 Acked 2483 Write 9 Done Done New true
5328 1485 Acked 2484 Read 7 Done Done New true
5329 1486 Acked 2485 Read 7 Done Done New true
5330 1487 Acked 2486 Read 3 Done Done New true
5331 1488 Acked 2487 Write 6 Done Done New true
5332 1489 Acked 2488 Write 3 Done Done New true
5333 1490 Acked 2489 Read 10 Done Done New true
5334 1491 Acked 2490 Write 7 Done Done New true
5335 1492 Acked 2491 Write 4 Done Done New true
5336 1493 Acked 2492 Read 2 Done Done New true
5337 1494 Acked 2493 Read 1 Done Done New true
5338 1495 Acked 2494 Read 10 Done Done New true
5339 1496 Acked 2495 Write 4 Done Done New true
5340 1497 Acked 2496 Read 3 Done Done New true
5341 1498 Acked 2497 Write 2 Done Done New true
5342 1499 Acked 2498 Read 2 Done Done New true
5343 1500 Acked 2499 Write 9 Done Done New true
5344 1501 Acked 2500 Read 2 Done Done New true
5345 1502 Acked 2501 Read 6 Done Done New true
5346 1503 Acked 2502 Write 3 Done Done New true
5347 1504 Acked 2503 Read 7 Done Done New true
5348 1505 Acked 2504 Write 3 Done Done New true
5349 1506 Acked 2505 Flush 0 Done Done New true
5350 STATES DS:0 DS:1 DS:2 TOTAL
5351 New 0 0 302 302
5352 Sent 0 0 0 0
5353 Done 302 302 0 604
5354 Skipped 0 0 0 0
5355 Error 0 0 0 0
5356 Last Flush: 2505 2505 2203
5357 Downstairs last five completed: 2203 2202 2201 2200 2199
5358 Upstairs last five completed: 1506 1505 1504 1503 1502
5359 CLIENT: Up:0 ds:302 act:2
53602024-02-27T21:45:40.403ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
53612024-02-27T21:45:40.403ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
53622024-02-27T21:45:40.403ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
53632024-02-27T21:45:40.403ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
53642024-02-27T21:45:40.403ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
53652024-02-27T21:45:40.403ZINFOcrucible: [2] client re-new 302 jobs since flush 2203 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
53662024-02-27T21:45:40.403ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
5367 [8830] got start action
5368 [8830] I am going to restart
5369 Starting downstairs at port 8830
5370 Make output file at "/tmp/dsc/downstairs-8830.txt"
5371 [8830][2] reports Starting
53722024-02-27T21:45:41.404ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
53732024-02-27T21:45:41.404ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
53742024-02-27T21:45:41.404ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
53752024-02-27T21:45:41.404ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
53762024-02-27T21:45:41.404ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
53772024-02-27T21:45:41.404ZINFOcrucible: [2] client re-new 302 jobs since flush 2203 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
53782024-02-27T21:45:41.404ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
5379 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1172)
5380 [8830][2] reports Running
53812024-02-27T21:45:42.406ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
53822024-02-27T21:45:42.406ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
53832024-02-27T21:45:42.407ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
53842024-02-27T21:45:42.407ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
53852024-02-27T21:45:42.407ZINFOcrucible: send last flush ID to this DS: 2203 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
53862024-02-27T21:45:42.407ZINFOcrucible: Replied this last flush ID: 2203 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
53872024-02-27T21:45:42.407ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
53882024-02-27T21:45:42.408ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
53892024-02-27T21:45:42.408ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
53902024-02-27T21:45:42.408ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
5391 ----------------------------------------------------------------
5392 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
5393 Downstairs last five completed: 2505 2504 2503 2502 2501
5394 Upstairs last five completed: 1506 1505 1504 1503 1502
5395 CLIENT: Up:0 ds:0 act:3
5396 Replay: All jobs finished, all DS active.
5397 Main task has work to do, go find it
5398 got dsc Stop(1)
5399 stop 1
5400 Feb 27 21:45:43.745 INFO request completed, latency_us: 384, response_code: 204, uri: /stop/cid/1, method:[8820] Got stop action so:false kr:false
5401 POST, req_id: fc481267-3501-4d8d-bad4-b437e08bce75, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
54022024-02-27T21:45:43.746ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
54032024-02-27T21:45:43.746ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
54042024-02-27T21:45:43.746ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = e6a5a974-925a-494f-8845-facc441c2690
5405 Feb 27 21:45:43.746 INFO request completed, latency_us: 129, response_code: 200{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:43.746636059Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"client":"1","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
5406 , uri: /state/cid/1, method: GET, req_id: a9b2007d-71a9-48a7-9f21-cbbfa077e687, remote_addr: 127.0.0.{"msg":"[1] client re-new 0 jobs since flush 2505","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:45:43.746662246Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"downstairs","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
5407 1:44347, local_addr: 127.0.0.1:9998
54082024-02-27T21:45:43.746ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
5409 [8820] Exited with: ExitStatus(unix_wait_status(9))
5410 [8820][1] reports Exit
54112024-02-27T21:45:44.747ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
54122024-02-27T21:45:44.747ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
54132024-02-27T21:45:44.747ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
54142024-02-27T21:45:44.747ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
54152024-02-27T21:45:44.747ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
54162024-02-27T21:45:44.747ZINFOcrucible: [1] client re-new 0 jobs since flush 2505 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
54172024-02-27T21:45:44.747ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
54182024-02-27T21:45:45.748ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
54192024-02-27T21:45:45.748ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
54202024-02-27T21:45:45.748ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
54212024-02-27T21:45:45.749ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
54222024-02-27T21:45:45.749ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
54232024-02-27T21:45:45.749ZINFOcrucible: [1] client re-new 0 jobs since flush 2505 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
54242024-02-27T21:45:45.749ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
54252024-02-27T21:45:46.751ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
54262024-02-27T21:45:46.751ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
54272024-02-27T21:45:46.751ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
54282024-02-27T21:45:46.751ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
54292024-02-27T21:45:46.751ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
54302024-02-27T21:45:46.751ZINFOcrucible: [1] client re-new 0 jobs since flush 2505 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
54312024-02-27T21:45:46.751ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
5432 Feb 27 21:45:47.747 INFO request completed, latency_us: 200, response_code: 200, uri: /state/cid/1, method: GET, req_id: d5cbd3b4-fe45-4db6-8706-80625d6aedb1, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
5433 001/300 Write block 4829 len 24576 data: 4 3 2 2 2 2
5434 002/300 Read block 812 len 40960
5435 003/300 Read block 496 len 24576
54362024-02-27T21:45:47.752ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
54372024-02-27T21:45:47.752ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
54382024-02-27T21:45:47.752ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
54392024-02-27T21:45:47.752ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
54402024-02-27T21:45:47.752ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
54412024-02-27T21:45:47.752ZINFOcrucible: [1] client re-new 3 jobs since flush 2505 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
54422024-02-27T21:45:47.752ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
5443 004/300 Read block 4777 len 28672
5444 005/300 Read block 1304 len 20480
5445 006/300 Read block 3450 len 12288
5446 007/300 Read block 569 len 16384
5447 008/300 Flush
5448 009/300 Read block 3889 len 32768
5449 010/300 Read block 3282 len 28672
5450 011/300 Write block 4783 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5451 012/300 Write block 1100 len 8192 data: 3 2
5452 013/300 Read block 2070 len 28672
5453 014/300 Read block 2043 len 28672
5454 015/300 Read block 4846 len 28672
5455 016/300 Read block 1056 len 32768
5456 017/300 Read block 524 len 16384
5457 018/300 Write block 1246 len 16384 data: 2 2 2 2
5458 019/300 Read block 561 len 12288
5459 020/300 Write block 487 len 24576 data: 2 2 2 2 2 2
5460 021/300 Read block 1628 len 8192
5461 022/300 Read block 3347 len 20480
5462 023/300 Read block 805 len 8192
5463 024/300 Read block 3436 len 8192
5464 025/300 Write block 3655 len 24576 data: 3 2 2 2 2 3
5465 026/300 Read block 4166 len 40960
5466 027/300 Write block 4563 len 36864 data: 2 3 3 4 4 4 4 4 4
5467 028/300 Read block 4823 len 40960
5468 029/300 Read block 4837 len 16384
5469 030/300 Read block 2926 len 12288
5470 031/300 Read block 2325 len 16384
5471 032/300 Write block 4711 len 8192 data: 2 3
5472 033/300 Read block 4208 len 12288
5473 034/300 Read block 1282 len 20480
5474 035/300 Write block 3026 len 28672 data: 3 3 3 3 3 2 3
5475 036/300 Read block 129 len 40960
5476 037/300 Write block 1217 len 40960 data: 6 5 4 4 4 2 2 3 3 4
5477 038/300 Read block 2728 len 36864
5478 039/300 Read block 3762 len 24576
5479 040/300 Flush
5480 041/300 Read block 1432 len 40960
5481 042/300 Read block 3756 len 12288
5482 043/300 Read block 4250 len 8192
5483 044/300 Write block 1236 len 12288 data: 2 2 2
5484 045/300 Read block 3596 len 24576
5485 046/300 Read block 6 len 12288
5486 047/300 Write block 3606 len 24576 data: 2 2 2 3 3 3
5487 048/300 Read block 634 len 12288
5488 049/300 Write block 1473 len 8192 data: 5 4
5489 050/300 Write block 3924 len 16384 data: 2 2 2 2
5490 051/300 Write block 4536 len 8192 data: 2 2
5491 052/300 Read block 1360 len 12288
5492 053/300 Read block 2325 len 16384
5493 054/300 Read block 4378 len 16384
5494 055/300 Read block 1242 len 40960
5495 056/300 Read block 1356 len 40960
5496 057/300 Read block 96 len 28672
5497 058/300 Read block 2472 len 4096
5498 059/300 Read block 1629 len 28672
5499 060/300 Read block 2429 len 4096
5500 061/300 Write block 3486 len 20480 data: 4 4 3 3 3
5501 062/300 Read block 2374 len 40960
5502 063/300 Write block 4572 len 36864 data: 4 3 2 2 3 3 3 3 3
5503 064/300 Read block 2991 len 24576
5504 065/300 Write block 4295 len 36864 data: 2 2 2 2 2 2 2 2 2
5505 066/300 Write block 3181 len 20480 data: 4 3 3 2 2
5506 067/300 Read block 379 len 8192
5507 068/300 Write block 1646 len 8192 data: 3 3
5508 069/300 Read block 4523 len 36864
5509 070/300 Write block 1641 len 40960 data: 2 2 3 3 3 4 4 3 4 3
5510 071/300 Read block 861 len 4096
5511 072/300 Read block 4205 len 12288
5512 073/300 Read block 4019 len 36864
5513 074/300 Read block 4533 len 32768
5514 075/300 Read block 1796 len 12288
5515 076/300 Write block 2936 len 16384 data: 2 2 2 2
5516 077/300 Write block 4131 len 32768 data: 3 3 4 3 3 3 3 3
5517 078/300 Write block 147 len 4096 data: 2
5518 079/300 Read block 1482 len 8192
5519 080/300 Write block 4327 len 12288 data: 2 2 2
5520 081/300 Read block 4516 len 24576
5521 082/300 Read block 2762 len 28672
5522 083/300 Read block 2571 len 20480
5523 084/300 Write block 2994 len 36864 data: 2 2 2 2 3 3 4 3 3
5524 085/300 Write block 446 len 40960 data: 2 2 2 3 3 3 3 2 2 2
5525 086/300 Write block 3928 len 12288 data: 2 2 2
5526 087/300 Write block 1833 len 36864 data: 2 2 2 2 2 2 2 2 2
5527 088/300 Read block 2043 len 20480
5528 089/300 Write block 1353 len 4096 data: 2
5529 090/300 Read block 4682 len 4096
5530 091/300 Write block 4547 len 28672 data: 2 2 3 4 4 4 3
5531 092/300 Read block 1379 len 24576
5532 093/300 Write block 542 len 16384 data: 2 2 2 2
5533 094/300 Read block 2767 len 28672
5534 095/300 Write block 3257 len 8192 data: 3 3
5535 096/300 Read block 2575 len 32768
5536 097/300 Write block 3443 len 24576 data: 2 2 2 3 3 4
5537 098/300 Write block 4265 len 32768 data: 4 4 4 3 3 3 3 2
5538 099/300 Write block 1135 len 4096 data: 2
5539 100/300 Read block 242 len 28672
5540 101/300 Write block 1638 len 24576 data: 2 2 2 3 3 4
5541 102/300 Read block 803 len 24576
5542 103/300 Read block 2293 len 24576
5543 104/300 Read block 3120 len 28672
5544 105/300 Write block 2499 len 40960 data: 2 2 2 2 3 4 3 3 4 4
5545 106/300 Read block 891 len 40960
5546 107/300 Write block 630 len 8192 data: 2 2
5547 108/300 Write block 3373 len 20480 data: 3 3 3 3 2
5548 109/300 Read block 2074 len 12288
5549 110/300 Read block 4429 len 16384
5550 111/300 Write block 2339 len 36864 data: 3 2 2 2 3 3 3 3 3
5551 112/300 Write block 4035 len 12288 data: 3 3 2
5552 113/300 Read block 1618 len 20480
5553 114/300 Read block 4212 len 20480
5554 115/300 Read block 4295 len 40960
5555 116/300 Read block 4435 len 20480
5556 117/300 Flush
5557 118/300 Read block 878 len 32768
5558 119/300 Write block 2592 len 16384 data: 3 3 2 2
5559 120/300 Read block 1605 len 8192
5560 121/300 Read block 1027 len 12288
5561 122/300 Write block 2647 len 16384 data: 5 4 4 3
5562 123/300 Read block 108 len 16384
5563 124/300 Read block 109 len 8192
5564 125/300 Read block 1628 len 28672
5565 126/300 Read block 1764 len 16384
5566 127/300 Read block 1289 len 24576
5567 128/300 Read block 4515 len 12288
5568 129/300 Read block 2775 len 12288
5569 130/300 Write block 1111 len 32768 data: 2 2 2 2 2 2 2 3
5570 131/300 Read block 3491 len 4096
5571 132/300 Write block 4490 len 24576 data: 5 4 4 3 2 2
5572 133/300 Read block 3938 len 16384
5573 134/300 Read block 4544 len 36864
5574 135/300 Read block 3413 len 28672
5575 136/300 Read block 4050 len 12288
5576 137/300 Read block 2484 len 24576
5577 138/300 Flush
5578 139/300 Write block 4435 len 20480 data: 3 3 3 3 3
5579 140/300 Write block 3374 len 8192 data: 4 4
5580 141/300 Write block 23 len 12288 data: 3 3 3
5581 142/300 Write block 42 len 36864 data: 3 3 3 3 3 2 3 3 3
5582 143/300 Read block 4518 len 20480
5583 144/300 Write block 1515 len 12288 data: 2 2 2
5584 145/300 Flush
5585 146/300 Read block 4141 len 32768
5586 147/300 Read block 2191 len 40960
5587 148/300 Read block 4543 len 8192
5588 149/300 Write block 2318 len 36864 data: 5 4 3 3 3 3 3 2 2
5589 150/300 Write block 3570 len 28672 data: 3 3 3 3 3 3 3
5590 151/300 Read block 1117 len 16384
5591 152/300 Read block 3029 len 32768
5592 153/300 Write block 676 len 28672 data: 4 4 4 2 2 2 2
5593 154/300 Flush
5594 155/300 Write block 2460 len 20480 data: 4 5 5 4 4
5595 156/300 Read block 534 len 36864
5596 157/300 Flush
5597 158/300 Read block 2228 len 28672
5598 159/300 Flush
5599 160/300 Write block 4683 len 32768 data: 4 4 3 4 4 3 3 2
5600 161/300 Flush
5601 162/300 Write block 4692 len 16384 data: 3 3 3 3
5602 163/300 Read block 702 len 32768
5603 164/300 Read block 1169 len 12288
5604 165/300 Read block 3044 len 8192
5605 166/300 Read block 4310 len 16384
5606 167/300 Read block 305 len 40960
5607 168/300 Write block 1682 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5608 169/300 Read block 3101 len 40960
5609 170/300 Read block 350 len 16384
5610 171/300 Flush
5611 172/300 Read block 2051 len 32768
5612 173/300 Write block 4250 len 24576 data: 3 3 2 2 2 2
5613 174/300 Write block 4264 len 20480 data: 4 5 5 5 4
5614 175/300 Read block 2320 len 28672
5615 176/300 Read block 3062 len 16384
5616 177/300 Flush
5617 178/300 Read block 4778 len 32768
5618 179/300 Read block 502 len 12288
5619 180/300 Write block 3811 len 20480 data: 3 3 4 4 4
5620 181/300 Read block 2694 len 8192
5621 182/300 Write block 1720 len 24576 data: 2 2 2 2 2 2
5622 183/300 Write block 2709 len 4096 data: 2
5623 184/300 Write block 3768 len 16384 data: 2 2 2 3
5624 185/300 Flush
5625 186/300 Read block 1731 len 40960
5626 187/300 Flush
5627 188/300 Read block 160 len 4096
5628 189/300 Write block 4901 len 40960 data: 3 2 2 2 2 3 3 3 3 3
5629 190/300 Read block 3631 len 36864
5630 191/300 Read block 8 len 24576
5631 192/300 Read block 1664 len 4096
5632 193/300 Read block 1031 len 32768
5633 194/300 Write block 4327 len 20480 data: 3 3 3 2 2
5634 195/300 Read block 421 len 8192
5635 196/300 Write block 847 len 12288 data: 2 2 2
5636 197/300 Read block 856 len 12288
5637 198/300 Read block 1984 len 4096
5638 199/300 Read block 4950 len 36864
5639 200/300 Write block 4151 len 20480 data: 3 3 3 3 2
5640 201/300 Read block 4055 len 40960
5641 202/300 Write block 2264 len 32768 data: 3 3 3 3 3 3 2 2
5642 203/300 Read block 2183 len 24576
5643 204/300 Read block 3381 len 20480
5644 205/300 Read block 3059 len 32768
5645 206/300 Read block 2795 len 40960
5646 207/300 Read block 4031 len 28672
5647 208/300 Read block 825 len 32768
5648 209/300 Write block 2096 len 24576 data: 2 2 2 2 2 2
5649 210/300 Write block 1660 len 24576 data: 6 4 5 5 6 4
5650 211/300 Read block 2165 len 12288
5651 212/300 Read block 1838 len 8192
5652 213/300 Flush
5653 214/300 Read block 3845 len 36864
5654 215/300 Read block 3977 len 28672
5655 216/300 Read block 3045 len 20480
5656 217/300 Read block 202 len 12288
5657 218/300 Write block 3794 len 12288 data: 3 3 4
5658 219/300 Write block 4484 len 28672 data: 3 2 4 4 4 4 6
5659 220/300 Flush
5660 221/300 Write block 763 len 12288 data: 2 2 3
5661 222/300 Flush
5662 223/300 Write block 1346 len 28672 data: 2 2 2 2 2 2 2
5663 224/300 Read block 2101 len 32768
5664 225/300 Write block 418 len 32768 data: 3 3 3 3 2 2 2 2
5665 226/300 Write block 780 len 40960 data: 2 2 2 2 3 4 4 4 4 4
5666 227/300 Write block 1315 len 24576 data: 4 3 2 3 3 3
5667 228/300 Write block 1559 len 32768 data: 3 3 3 2 2 2 2 2
5668 229/300 Write block 2569 len 16384 data: 3 4 4 4
5669 230/300 Write block 3417 len 28672 data: 3 2 2 2 2 2 2
5670 231/300 Write block 4911 len 36864 data: 3 3 3 2 2 2 2 2 2
5671 232/300 Read block 269 len 16384
5672 233/300 Read block 4573 len 20480
5673 234/300 Read block 2550 len 28672
5674 235/300 Write block 4459 len 36864 data: 2 2 2 2 2 2 2 2 2
5675 236/300 Read block 3784 len 20480
5676 237/300 Write block 3138 len 40960 data: 2 2 2 2 2 2 2 3 3 3
5677 238/300 Write block 1432 len 4096 data: 2
5678 239/300 Read block 3344 len 4096
5679 240/300 Write block 3102 len 40960 data: 2 2 2 3 3 3 3 2 2 2
5680 241/300 Flush
5681 242/300 Read block 760 len 32768
5682 243/300 Read block 386 len 36864
5683 244/300 Write block 1394 len 20480 data: 2 2 2 2 3
5684 245/300 Write block 3367 len 8192 data: 2 2
5685 246/300 Read block 3403 len 20480
5686 247/300 Read block 3854 len 36864
5687 248/300 Write block 1758 len 32768 data: 3 3 3 3 3 3 3 3
5688 249/300 Read block 2975 len 16384
5689 250/300 Read block 2432 len 40960
5690 251/300 Read block 1588 len 4096
5691 252/300 Read block 578 len 28672
5692 253/300 Write block 4263 len 16384 data: 4 5 6 6
5693 254/300 Write block 985 len 24576 data: 2 2 2 3 3 3
5694 255/300 Write block 3961 len 12288 data: 2 2 3
5695 256/300 Write block 594 len 20480 data: 2 3 3 3 3
5696 257/300 Read block 3650 len 36864
5697 258/300 Read block 3639 len 16384
5698 259/300 Read block 3544 len 16384
5699 260/300 Read block 2817 len 24576
5700 261/300 Write block 2731 len 28672 data: 2 2 2 2 2 2 2
5701 262/300 Read block 691 len 12288
5702 263/300 Read block 2002 len 20480
5703 264/300 Write block 3476 len 12288 data: 2 2 2
5704 265/300 Read block 1356 len 40960
5705 266/300 Write block 4451 len 40960 data: 2 2 2 2 2 2 2 2 3 3
5706 267/300 Flush
5707 268/300 Write block 385 len 20480 data: 2 2 2 2 2
5708 269/300 Read block 657 len 28672
5709 270/300 Read block 447 len 28672
5710 271/300 Flush
5711 272/300 Write block 1317 len 16384 data: 3 4 4 4
5712 273/300 Read block 1922 len 20480
5713 274/300 Write block 3842 len 16384 data: 3 3 3 3
5714 275/300 Write block 536 len 8192 data: 2 3
5715 276/300 Read block 2854 len 24576
5716 277/300 Write block 2624 len 28672 data: 3 3 3 3 2 2 2
5717 278/300 Read block 2731 len 4096
5718 279/300 Read block 3781 len 36864
5719 280/300 Write block 2323 len 36864 data: 4 4 3 3 3 2 2 2 2
5720 281/300 Read block 3031 len 8192
5721 282/300 Write block 499 len 20480 data: 4 4 3 4 4
5722 283/300 Read block 546 len 4096
5723 284/300 Write block 1404 len 12288 data: 2 2 3
5724 285/300 Write block 508 len 40960 data: 4 4 4 4 4 3 2 2 2 3
5725 286/300 Write block 2500 len 32768 data: 3 3 3 4 5 4 4 5
5726 287/300 Flush
5727 288/300 Read block 4711 len 24576
5728 289/300 Write block 1315 len 4096 data: 5
5729 290/300 Read block 4421 len 40960
5730 291/300 Write block 2361 len 4096 data: 3
5731 292/300 Flush
5732 293/300 Read block 4146 len 20480
5733 294/300 Write block 3531 len 32768 data: 2 2 2 2 2 2 2 2
5734 295/300 Write block 3442 len 12288 data: 2 3 3
5735 296/300 Write block 1882 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5736 297/300 Write block 946 len 28672 data: 4 3 3 3 4 4 3
5737 298/300 Read block 4402 len 36864
5738 299/300 Write block 257 len 20480 data: 3 3 3 3 3
5739 300/300 Write block 958 len 8192 data: 4 4
5740 Main task has work to do, go find it
5741 got dsc Start(1)
5742 start 1
5743 Feb 27 21:45:48.100 INFO request completed, latency_us: 215, response_code: 204, uri: /start/cid/1, method: POST, req_id: 005b0db6-ee1c-4206-b5a9-777917426b92, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
5744 Replay: started 1, returned:Ok(())
5745 ----------------------------------------------------------------
5746 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5747 States: Active Offline Active
5748 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
5749 1507 Acked 2506 Write 6 Done New Done true
5750 1508 Acked 2507 Read 10 Done New Done true
5751 1509 Acked 2508 Read 6 Done New Done true
5752 1510 Acked 2509 Read 7 Done New Done false
5753 1511 Acked 2510 Read 5 Done New Done false
5754 1512 Acked 2511 Read 3 Done New Done false
5755 1513 Acked 2512 Read 4 Done New Done false
5756 1514 Acked 2513 Flush 0 Done New Done false
5757 1515 Acked 2514 Read 8 Done New Done false
5758 1516 Acked 2515 Read 7 Done New Done false
5759 1517 Acked 2516 Write 10 Done New Done false
5760 1518 Acked 2517 Write 2 Done New Done false
5761 1519 Acked 2518 Read 7 Done New Done false
5762 1520 Acked 2519 Read 7 Done New Done false
5763 1521 Acked 2520 Read 7 Done New Done false
5764 1522 Acked 2521 Read 8 Done New Done false
5765 1523 Acked 2522 Read 4 Done New Done false
5766 1524 Acked 2523 Write 4 Done New Done false
5767 1525 Acked 2524 Read 3 Done New Done false
5768 1526 Acked 2525 Write 6 Done New Done false
5769 1527 Acked 2526 Read 2 Done New Done false
5770 1528 Acked 2527 Read 5 Done New Done false
5771 1529 Acked 2528 Read 2 Done New Done false
5772 1530 Acked 2529 Read 2 Done New Done false
5773 1531 Acked 2530 Write 6 Done New Done false
5774 1532 Acked 2531 Read 10 Done New Done false
5775 1533 Acked 2532 Write 9 Done New Done false
5776 1534 Acked 2533 Read 10 Done New Done false
5777 1535 Acked 2534 Read 4 Done New Done false
5778 1536 Acked 2535 Read 3 Done New Done false
5779 1537 Acked 2536 Read 4 Done New Done false
5780 1538 Acked 2537 Write 2 Done New Done false
5781 1539 Acked 2538 Read 3 Done New Done false
5782 1540 Acked 2539 Read 5 Done New Done false
5783 1541 Acked 2540 Write 7 Done New Done false
5784 1542 Acked 2541 Read 10 Done New Done false
5785 1543 Acked 2542 Write 10 Done New Done false
5786 1544 Acked 2543 Read 9 Done New Done false
5787 1545 Acked 2544 Read 6 Done New Done false
5788 1546 Acked 2545 Flush 0 Done New Done false
5789 1547 Acked 2546 Read 10 Done New Done false
5790 1548 Acked 2547 Read 3 Done New Done false
5791 1549 Acked 2548 Read 2 Done New Done false
5792 1550 Acked 2549 Write 3 Done New Done false
5793 1551 Acked 2550 Read 6 Done New Done false
5794 1552 Acked 2551 Read 3 Done New Done false
5795 1553 Acked 2552 Write 6 Done New Done false
5796 1554 Acked 2553 Read 3 Done New Done false
5797 1555 Acked 2554 Write 2 Done New Done false
5798 1556 Acked 2555 Write 4 Done New Done false
5799 1557 Acked 2556 Write 2 Done New Done false
5800 1558 Acked 2557 Read 3 Done New Done false
5801 1559 Acked 2558 Read 4 Done New Done false
5802 1560 Acked 2559 Read 4 Done New Done false
5803 1561 Acked 2560 Read 10 Done New Done false
5804 1562 Acked 2561 Read 10 Done New Done false
5805 1563 Acked 2562 Read 7 Done New Done false
5806 1564 Acked 2563 Read 1 Done New Done false
5807 1565 Acked 2564 Read 7 Done New Done false
5808 1566 Acked 2565 Read 1 Done New Done false
5809 1567 Acked 2566 Write 5 Done New Done false
5810 1568 Acked 2567 Read 10 Done New Done false
5811 1569 Acked 2568 Write 9 Done New Done false
5812 1570 Acked 2569 Read 6 Done New Done false
5813 1571 Acked 2570 Write 9 Done New Done false
5814 1572 Acked 2571 Write 5 Done New Done false
5815 1573 Acked 2572 Read 2 Done New Done false
5816 1574 Acked 2573 Write 2 Done New Done false
5817 1575 Acked 2574 Read 9 Done New Done false
5818 1576 Acked 2575 Write 10 Done New Done false
5819 1577 Acked 2576 Read 1 Done New Done false
5820 1578 Acked 2577 Read 3 Done New Done false
5821 1579 Acked 2578 Read 9 Done New Done false
5822 1580 Acked 2579 Read 8 Done New Done false
5823 1581 Acked 2580 Read 3 Done New Done false
5824 1582 Acked 2581 Write 4 Done New Done false
5825 1583 Acked 2582 Write 8 Done New Done false
5826 1584 Acked 2583 Write 1 Done New Done false
5827 1585 Acked 2584 Read 2 Done New Done false
5828 1586 Acked 2585 Write 3 Done New Done false
5829 1587 Acked 2586 Read 6 Done New Done false
5830 1588 Acked 2587 Read 7 Done New Done false
5831 1589 Acked 2588 Read 5 Done New Done false
5832 1590 Acked 2589 Write 9 Done New Done false
5833 1591 Acked 2590 Write 10 Done New Done false
5834 1592 Acked 2591 Write 3 Done New Done false
5835 1593 Acked 2592 Write 9 Done New Done false
5836 1594 Acked 2593 Read 5 Done New Done false
5837 1595 Acked 2594 Write 1 Done New Done false
5838 1596 Acked 2595 Read 1 Done New Done false
5839 1597 Acked 2596 Write 7 Done New Done false
5840 1598 Acked 2597 Read 6 Done New Done false
5841 1599 Acked 2598 Write 4 Done New Done false
5842 1600 Acked 2599 Read 7 Done New Done false
5843 1601 Acked 2600 Write 2 Done New Done false
5844 1602 Acked 2601 Read 8 Done New Done false
5845 1603 Acked 2602 Write 6 Done New Done false
5846 1604 Acked 2603 Write 8 Done New Done false
5847 1605 Acked 2604 Write 1 Done New Done false
5848 1606 Acked 2605 Read 7 Done New Done false
5849 1607 Acked 2606 Write 6 Done New Done false
5850 1608 Acked 2607 Read 6 Done New Done false
5851 1609 Acked 2608 Read 6 Done New Done false
5852 1610 Acked 2609 Read 7 Done New Done false
5853 1611 Acked 2610 Write 10 Done New Done false
5854 1612 Acked 2611 Read 10 Done New Done false
5855 1613 Acked 2612 Write 2 Done New Done false
5856 1614 Acked 2613 Write 5 Done New Done false
5857 1615 Acked 2614 Read 3 Done New Done false
5858 1616 Acked 2615 Read 4 Done New Done false
5859 1617 Acked 2616 Write 9 Done New Done false
5860 1618 Acked 2617 Write 3 Done New Done false
5861 1619 Acked 2618 Read 5 Done New Done false
5862 1620 Acked 2619 Read 5 Done New Done false
5863 1621 Acked 2620 Read 10 Done New Done false
5864 1622 Acked 2621 Read 5 Done New Done false
5865 1623 Acked 2622 Flush 0 Done New Done false
5866 1624 Acked 2623 Read 8 Done New Done false
5867 1625 Acked 2624 Write 4 Done New Done false
5868 1626 Acked 2625 Read 2 Done New Done false
5869 1627 Acked 2626 Read 3 Done New Done false
5870 1628 Acked 2627 Write 4 Done New Done false
5871 1629 Acked 2628 Read 4 Done New Done false
5872 1630 Acked 2629 Read 2 Done New Done false
5873 1631 Acked 2630 Read 7 Done New Done false
5874 1632 Acked 2631 Read 4 Done New Done false
5875 1633 Acked 2632 Read 6 Done New Done false
5876 1634 Acked 2633 Read 3 Done New Done false
5877 1635 Acked 2634 Read 3 Done New Done false
5878 1636 Acked 2635 Write 8 Done New Done false
5879 1637 Acked 2636 Read 1 Done New Done false
5880 1638 Acked 2637 Write 6 Done New Done false
5881 1639 Acked 2638 Read 4 Done New Done false
5882 1640 Acked 2639 Read 9 Done New Done false
5883 1641 Acked 2640 Read 7 Done New Done false
5884 1642 Acked 2641 Read 3 Done New Done false
5885 1643 Acked 2642 Read 6 Done New Done false
5886 1644 Acked 2643 Flush 0 Done New Done false
5887 1645 Acked 2644 Write 5 Done New Done false
5888 1646 Acked 2645 Write 2 Done New Done false
5889 1647 Acked 2646 Write 3 Done New Done false
5890 1648 Acked 2647 Write 9 Done New Done false
5891 1649 Acked 2648 Read 5 Done New Done false
5892 1650 Acked 2649 Write 3 Done New Done false
5893 1651 Acked 2650 Flush 0 Done New Done false
5894 1652 Acked 2651 Read 8 Done New Done false
5895 1653 Acked 2652 Read 10 Done New Done false
5896 1654 Acked 2653 Read 2 Done New Done false
5897 1655 Acked 2654 Write 9 Done New Done false
5898 1656 Acked 2655 Write 7 Done New Done false
5899 1657 Acked 2656 Read 4 Done New Done false
5900 1658 Acked 2657 Read 8 Done New Done false
5901 1659 Acked 2658 Write 7 Done New Done false
5902 1660 Acked 2659 Flush 0 Done New Done false
5903 1661 Acked 2660 Write 5 Done New Done false
5904 1662 Acked 2661 Read 9 Done New Done false
5905 1663 Acked 2662 Flush 0 Done New Done false
5906 1664 Acked 2663 Read 7 Done New Done false
5907 1665 Acked 2664 Flush 0 Done New Done false
5908 1666 Acked 2665 Write 8 Done New Done false
5909 1667 Acked 2666 Flush 0 Done New Done false
5910 1668 Acked 2667 Write 4 Done New Done false
5911 1669 Acked 2668 Read 8 Done New Done false
5912 1670 Acked 2669 Read 3 Done New Done false
5913 1671 Acked 2670 Read 2 Done New Done false
5914 1672 Acked 2671 Read 4 Done New Done false
5915 1673 Acked 2672 Read 10 Done New Done false
5916 1674 Acked 2673 Write 10 Done New Done false
5917 1675 Acked 2674 Read 10 Done New Done false
5918 1676 Acked 2675 Read 4 Done New Done false
5919 1677 Acked 2676 Flush 0 Done New Done false
5920 1678 Acked 2677 Read 8 Done New Done false
5921 1679 Acked 2678 Write 6 Done New Done false
5922 1680 Acked 2679 Write 5 Done New Done false
5923 1681 Acked 2680 Read 7 Done New Done false
5924 1682 Acked 2681 Read 4 Done New Done false
5925 1683 Acked 2682 Flush 0 Done New Done false
5926 1684 Acked 2683 Read 8 Done New Done false
5927 1685 Acked 2684 Read 3 Done New Done false
5928 1686 Acked 2685 Write 5 Done New Done false
5929 1687 Acked 2686 Read 2 Done New Done false
5930 1688 Acked 2687 Write 6 Done New Done false
5931 1689 Acked 2688 Write 1 Done New Done false
5932 1690 Acked 2689 Write 4 Done New Done false
5933 1691 Acked 2690 Flush 0 Done New Done false
5934 1692 Acked 2691 Read 10 Done New Done false
5935 1693 Acked 2692 Flush 0 Done New Done false
5936 1694 Acked 2693 Read 1 Done New Done false
5937 1695 Acked 2694 Write 10 Done New Done false
5938 1696 Acked 2695 Read 9 Done New Done false
5939 1697 Acked 2696 Read 6 Done New Done false
5940 1698 Acked 2697 Read 1 Done New Done false
5941 1699 Acked 2698 Read 8 Done New Done false
5942 1700 Acked 2699 Write 5 Done New Done false
5943 1701 Acked 2700 Read 2 Done New Done false
5944 1702 Acked 2701 Write 3 Done New Done false
5945 1703 Acked 2702 Read 3 Done New Done false
5946 1704 Acked 2703 Read 1 Done New Done false
5947 1705 Acked 2704 Read 9 Done New Done false
5948 1706 Acked 2705 Write 5 Done New Done false
5949 1707 Acked 2706 Read 10 Done New Done false
5950 1708 Acked 2707 Write 8 Done New Done false
5951 1709 Acked 2708 Read 6 Done New Done false
5952 1710 Acked 2709 Read 5 Done New Done false
5953 1711 Acked 2710 Read 8 Done New Done false
5954 1712 Acked 2711 Read 10 Done New Done false
5955 1713 Acked 2712 Read 7 Done New Done false
5956 1714 Acked 2713 Read 8 Done New Done false
5957 1715 Acked 2714 Write 6 Done New Done false
5958 1716 Acked 2715 Write 6 Done New Done false
5959 1717 Acked 2716 Read 3 Done New Done false
5960 1718 Acked 2717 Read 2 Done New Done false
5961 1719 Acked 2718 Flush 0 Done New Done false
5962 1720 Acked 2719 Read 9 Done New Done false
5963 1721 Acked 2720 Read 7 Done New Done false
5964 1722 Acked 2721 Read 5 Done New Done false
5965 1723 Acked 2722 Read 3 Done New Done false
5966 1724 Acked 2723 Write 3 Done New Done false
5967 1725 Acked 2724 Write 7 Done New Done false
5968 1726 Acked 2725 Flush 0 Done New Done false
5969 1727 Acked 2726 Write 3 Done New Done false
5970 1728 Acked 2727 Flush 0 Done New Done false
5971 1729 Acked 2728 Write 7 Done New Done false
5972 1730 Acked 2729 Read 8 Done New Done false
5973 1731 Acked 2730 Write 8 Done New Done false
5974 1732 Acked 2731 Write 10 Done New Done false
5975 1733 Acked 2732 Write 6 Done New Done false
5976 1734 Acked 2733 Write 8 Done New Done false
5977 1735 Acked 2734 Write 4 Done New Done false
5978 1736 Acked 2735 Write 7 Done New Done false
5979 1737 Acked 2736 Write 9 Done New Done false
5980 1738 Acked 2737 Read 4 Done New Done false
5981 1739 Acked 2738 Read 5 Done New Done false
5982 1740 Acked 2739 Read 7 Done New Done false
5983 1741 Acked 2740 Write 9 Done New Done false
5984 1742 Acked 2741 Read 5 Done New Done false
5985 1743 Acked 2742 Write 10 Done New Done false
5986 1744 Acked 2743 Write 1 Done New Done false
5987 1745 Acked 2744 Read 1 Done New Done false
5988 1746 Acked 2745 Write 10 Done New Done false
5989 1747 Acked 2746 Flush 0 Done New Done false
5990 1748 Acked 2747 Read 8 Done New Done false
5991 1749 Acked 2748 Read 9 Done New Done false
5992 1750 Acked 2749 Write 5 Done New Done false
5993 1751 Acked 2750 Write 2 Done New Done false
5994 1752 Acked 2751 Read 5 Done New Done false
5995 1753 Acked 2752 Read 9 Done New Done false
5996 1754 Acked 2753 Write 8 Done New Done false
5997 1755 Acked 2754 Read 4 Done New Done false
5998 1756 Acked 2755 Read 10 Done New Done false
5999 1757 Acked 2756 Read 1 Done New Done false
6000 1758 Acked 2757 Read 7 Done New Done false
6001 1759 Acked 2758 Write 4 Done New Done false
6002 1760 Acked 2759 Write 6 Done New Done false
6003 1761 Acked 2760 Write 3 Done New Done false
6004 1762 Acked 2761 Write 5 Done New Done false
6005 1763 Acked 2762 Read 9 Done New Done false
6006 1764 Acked 2763 Read 4 Done New Done false
6007 1765 Acked 2764 Read 4 Done New Done false
6008 1766 Acked 2765 Read 6 Done New Done false
6009 1767 Acked 2766 Write 7 Done New Done false
6010 1768 Acked 2767 Read 3 Done New Done false
6011 1769 Acked 2768 Read 5 Done New Done false
6012 1770 Acked 2769 Write 3 Done New Done false
6013 1771 Acked 2770 Read 10 Done New Done false
6014 1772 Acked 2771 Flush 0 Done New Done false
6015 1773 Acked 2772 Write 10 Done New Done false
6016 1774 Acked 2773 Flush 0 Done New Done false
6017 1775 Acked 2774 Write 5 Done New Done false
6018 1776 Acked 2775 Read 7 Done New Done false
6019 1777 Acked 2776 Read 7 Done New Done false
6020 1778 Acked 2777 Flush 0 Done New Done false
6021 1779 Acked 2778 Write 4 Done New Done false
6022 1780 Acked 2779 Read 5 Done New Done false
6023 1781 Acked 2780 Write 4 Done New Done false
6024 1782 Acked 2781 Write 2 Done New Done false
6025 1783 Acked 2782 Read 6 Done New Done false
6026 1784 Acked 2783 Write 7 Done New Done false
6027 1785 Acked 2784 Read 1 Done New Done false
6028 1786 Acked 2785 Read 9 Done New Done false
6029 1787 Acked 2786 Write 9 Done New Done false
6030 1788 Acked 2787 Read 2 Done New Done false
6031 1789 Acked 2788 Write 5 Done New Done false
6032 1790 Acked 2789 Read 1 Done New Done false
6033 1791 Acked 2790 Write 3 Done New Done false
6034 1792 Acked 2791 Write 10 Done New Done false
6035 1793 Acked 2792 Write 8 Done New Done false
6036 1794 Acked 2793 Flush 0 Done New Done false
6037 1795 Acked 2794 Read 6 Done New Done false
6038 1796 Acked 2795 Write 1 Done New Done false
6039 1797 Acked 2796 Read 10 Done New Done false
6040 1798 Acked 2797 Write 1 Done New Done false
6041 1799 Acked 2798 Flush 0 Done New Done false
6042 1800 Acked 2799 Read 5 Done New Done false
6043 1801 Acked 2800 Write 8 Done New Done false
6044 1802 Acked 2801 Write 3 Done New Done false
6045 1803 Acked 2802 Write 10 Done New Done false
6046 1804 Acked 2803 Write 7 Done New Done false
6047 1805 Acked 2804 Read 9 Done New Done false
6048 1806 Acked 2805 Write 5 Done New Done false
6049 1807 Acked 2806 Write 2 Done New Done false
6050 STATES DS:0 DS:1 DS:2 TOTAL
6051 New 0 301 0 301
6052 Sent 0 0 0 0
6053 Done 301 0 301 602
6054 Skipped 0 0 0 0
6055 Error 0 0 0 0
6056 Last Flush: 2798 2505 2798
6057 Downstairs last five completed: 2505 2504 2503 2502 2501
6058 Upstairs last five completed: 1807 1806 1805 1804 1803
6059 CLIENT: Up:0 ds:301 act:2
60602024-02-27T21:45:48.754ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
60612024-02-27T21:45:48.754ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
60622024-02-27T21:45:48.754ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
60632024-02-27T21:45:48.754ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
60642024-02-27T21:45:48.754ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
60652024-02-27T21:45:48.754ZINFOcrucible: [1] client re-new 302 jobs since flush 2505 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
60662024-02-27T21:45:48.754ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
60672024-02-27T21:45:49.755ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
60682024-02-27T21:45:49.755ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
60692024-02-27T21:45:49.755ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
60702024-02-27T21:45:49.755ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
60712024-02-27T21:45:49.755ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
60722024-02-27T21:45:49.755ZINFOcrucible: [1] client re-new 302 jobs since flush 2505 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
60732024-02-27T21:45:49.756ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
60742024-02-27T21:45:50.756ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
60752024-02-27T21:45:50.757ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
60762024-02-27T21:45:50.757ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
60772024-02-27T21:45:50.757ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
60782024-02-27T21:45:50.757ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
60792024-02-27T21:45:50.757ZINFOcrucible: [1] client re-new 302 jobs since flush 2505 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
60802024-02-27T21:45:50.757ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
60812024-02-27T21:45:51.759ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
60822024-02-27T21:45:51.759ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
60832024-02-27T21:45:51.759ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
60842024-02-27T21:45:51.759ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
60852024-02-27T21:45:51.759ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
60862024-02-27T21:45:51.759ZINFOcrucible: [1] client re-new 302 jobs since flush 2505 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
60872024-02-27T21:45:51.759ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
6088 ----------------------------------------------------------------
6089 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
6090 States: Active Offline Active
6091 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
6092 1507 Acked 2506 Write 6 Done New Done true
6093 1508 Acked 2507 Read 10 Done New Done true
6094 1509 Acked 2508 Read 6 Done New Done true
6095 1510 Acked 2509 Read 7 Done New Done true
6096 1511 Acked 2510 Read 5 Done New Done true
6097 1512 Acked 2511 Read 3 Done New Done true
6098 1513 Acked 2512 Read 4 Done New Done true
6099 1514 Acked 2513 Flush 0 Done New Done true
6100 1515 Acked 2514 Read 8 Done New Done true
6101 1516 Acked 2515 Read 7 Done New Done true
6102 1517 Acked 2516 Write 10 Done New Done true
6103 1518 Acked 2517 Write 2 Done New Done true
6104 1519 Acked 2518 Read 7 Done New Done true
6105 1520 Acked 2519 Read 7 Done New Done true
6106 1521 Acked 2520 Read 7 Done New Done true
6107 1522 Acked 2521 Read 8 Done New Done true
6108 1523 Acked 2522 Read 4 Done New Done true
6109 1524 Acked 2523 Write 4 Done New Done true
6110 1525 Acked 2524 Read 3 Done New Done true
6111 1526 Acked 2525 Write 6 Done New Done true
6112 1527 Acked 2526 Read 2 Done New Done true
6113 1528 Acked 2527 Read 5 Done New Done true
6114 1529 Acked 2528 Read 2 Done New Done true
6115 1530 Acked 2529 Read 2 Done New Done true
6116 1531 Acked 2530 Write 6 Done New Done true
6117 1532 Acked 2531 Read 10 Done New Done true
6118 1533 Acked 2532 Write 9 Done New Done true
6119 1534 Acked 2533 Read 10 Done New Done true
6120 1535 Acked 2534 Read 4 Done New Done true
6121 1536 Acked 2535 Read 3 Done New Done true
6122 1537 Acked 2536 Read 4 Done New Done true
6123 1538 Acked 2537 Write 2 Done New Done true
6124 1539 Acked 2538 Read 3 Done New Done true
6125 1540 Acked 2539 Read 5 Done New Done true
6126 1541 Acked 2540 Write 7 Done New Done true
6127 1542 Acked 2541 Read 10 Done New Done true
6128 1543 Acked 2542 Write 10 Done New Done true
6129 1544 Acked 2543 Read 9 Done New Done true
6130 1545 Acked 2544 Read 6 Done New Done true
6131 1546 Acked 2545 Flush 0 Done New Done true
6132 1547 Acked 2546 Read 10 Done New Done true
6133 1548 Acked 2547 Read 3 Done New Done true
6134 1549 Acked 2548 Read 2 Done New Done true
6135 1550 Acked 2549 Write 3 Done New Done true
6136 1551 Acked 2550 Read 6 Done New Done true
6137 1552 Acked 2551 Read 3 Done New Done true
6138 1553 Acked 2552 Write 6 Done New Done true
6139 1554 Acked 2553 Read 3 Done New Done true
6140 1555 Acked 2554 Write 2 Done New Done true
6141 1556 Acked 2555 Write 4 Done New Done true
6142 1557 Acked 2556 Write 2 Done New Done true
6143 1558 Acked 2557 Read 3 Done New Done true
6144 1559 Acked 2558 Read 4 Done New Done true
6145 1560 Acked 2559 Read 4 Done New Done true
6146 1561 Acked 2560 Read 10 Done New Done true
6147 1562 Acked 2561 Read 10 Done New Done true
6148 1563 Acked 2562 Read 7 Done New Done true
6149 1564 Acked 2563 Read 1 Done New Done true
6150 1565 Acked 2564 Read 7 Done New Done true
6151 1566 Acked 2565 Read 1 Done New Done true
6152 1567 Acked 2566 Write 5 Done New Done true
6153 1568 Acked 2567 Read 10 Done New Done true
6154 1569 Acked 2568 Write 9 Done New Done true
6155 1570 Acked 2569 Read 6 Done New Done true
6156 1571 Acked 2570 Write 9 Done New Done true
6157 1572 Acked 2571 Write 5 Done New Done true
6158 1573 Acked 2572 Read 2 Done New Done true
6159 1574 Acked 2573 Write 2 Done New Done true
6160 1575 Acked 2574 Read 9 Done New Done true
6161 1576 Acked 2575 Write 10 Done New Done true
6162 1577 Acked 2576 Read 1 Done New Done true
6163 1578 Acked 2577 Read 3 Done New Done true
6164 1579 Acked 2578 Read 9 Done New Done true
6165 1580 Acked 2579 Read 8 Done New Done true
6166 1581 Acked 2580 Read 3 Done New Done true
6167 1582 Acked 2581 Write 4 Done New Done true
6168 1583 Acked 2582 Write 8 Done New Done true
6169 1584 Acked 2583 Write 1 Done New Done true
6170 1585 Acked 2584 Read 2 Done New Done true
6171 1586 Acked 2585 Write 3 Done New Done true
6172 1587 Acked 2586 Read 6 Done New Done true
6173 1588 Acked 2587 Read 7 Done New Done true
6174 1589 Acked 2588 Read 5 Done New Done true
6175 1590 Acked 2589 Write 9 Done New Done true
6176 1591 Acked 2590 Write 10 Done New Done true
6177 1592 Acked 2591 Write 3 Done New Done true
6178 1593 Acked 2592 Write 9 Done New Done true
6179 1594 Acked 2593 Read 5 Done New Done true
6180 1595 Acked 2594 Write 1 Done New Done true
6181 1596 Acked 2595 Read 1 Done New Done true
6182 1597 Acked 2596 Write 7 Done New Done true
6183 1598 Acked 2597 Read 6 Done New Done true
6184 1599 Acked 2598 Write 4 Done New Done true
6185 1600 Acked 2599 Read 7 Done New Done true
6186 1601 Acked 2600 Write 2 Done New Done true
6187 1602 Acked 2601 Read 8 Done New Done true
6188 1603 Acked 2602 Write 6 Done New Done true
6189 1604 Acked 2603 Write 8 Done New Done true
6190 1605 Acked 2604 Write 1 Done New Done true
6191 1606 Acked 2605 Read 7 Done New Done true
6192 1607 Acked 2606 Write 6 Done New Done true
6193 1608 Acked 2607 Read 6 Done New Done true
6194 1609 Acked 2608 Read 6 Done New Done true
6195 1610 Acked 2609 Read 7 Done New Done true
6196 1611 Acked 2610 Write 10 Done New Done true
6197 1612 Acked 2611 Read 10 Done New Done true
6198 1613 Acked 2612 Write 2 Done New Done true
6199 1614 Acked 2613 Write 5 Done New Done true
6200 1615 Acked 2614 Read 3 Done New Done true
6201 1616 Acked 2615 Read 4 Done New Done true
6202 1617 Acked 2616 Write 9 Done New Done true
6203 1618 Acked 2617 Write 3 Done New Done true
6204 1619 Acked 2618 Read 5 Done New Done true
6205 1620 Acked 2619 Read 5 Done New Done true
6206 1621 Acked 2620 Read 10 Done New Done true
6207 1622 Acked 2621 Read 5 Done New Done true
6208 1623 Acked 2622 Flush 0 Done New Done true
6209 1624 Acked 2623 Read 8 Done New Done true
6210 1625 Acked 2624 Write 4 Done New Done true
6211 1626 Acked 2625 Read 2 Done New Done true
6212 1627 Acked 2626 Read 3 Done New Done true
6213 1628 Acked 2627 Write 4 Done New Done true
6214 1629 Acked 2628 Read 4 Done New Done true
6215 1630 Acked 2629 Read 2 Done New Done true
6216 1631 Acked 2630 Read 7 Done New Done true
6217 1632 Acked 2631 Read 4 Done New Done true
6218 1633 Acked 2632 Read 6 Done New Done true
6219 1634 Acked 2633 Read 3 Done New Done true
6220 1635 Acked 2634 Read 3 Done New Done true
6221 1636 Acked 2635 Write 8 Done New Done true
6222 1637 Acked 2636 Read 1 Done New Done true
6223 1638 Acked 2637 Write 6 Done New Done true
6224 1639 Acked 2638 Read 4 Done New Done true
6225 1640 Acked 2639 Read 9 Done New Done true
6226 1641 Acked 2640 Read 7 Done New Done true
6227 1642 Acked 2641 Read 3 Done New Done true
6228 1643 Acked 2642 Read 6 Done New Done true
6229 1644 Acked 2643 Flush 0 Done New Done true
6230 1645 Acked 2644 Write 5 Done New Done true
6231 1646 Acked 2645 Write 2 Done New Done true
6232 1647 Acked 2646 Write 3 Done New Done true
6233 1648 Acked 2647 Write 9 Done New Done true
6234 1649 Acked 2648 Read 5 Done New Done true
6235 1650 Acked 2649 Write 3 Done New Done true
6236 1651 Acked 2650 Flush 0 Done New Done true
6237 1652 Acked 2651 Read 8 Done New Done true
6238 1653 Acked 2652 Read 10 Done New Done true
6239 1654 Acked 2653 Read 2 Done New Done true
6240 1655 Acked 2654 Write 9 Done New Done true
6241 1656 Acked 2655 Write 7 Done New Done true
6242 1657 Acked 2656 Read 4 Done New Done true
6243 1658 Acked 2657 Read 8 Done New Done true
6244 1659 Acked 2658 Write 7 Done New Done true
6245 1660 Acked 2659 Flush 0 Done New Done true
6246 1661 Acked 2660 Write 5 Done New Done true
6247 1662 Acked 2661 Read 9 Done New Done true
6248 1663 Acked 2662 Flush 0 Done New Done true
6249 1664 Acked 2663 Read 7 Done New Done true
6250 1665 Acked 2664 Flush 0 Done New Done true
6251 1666 Acked 2665 Write 8 Done New Done true
6252 1667 Acked 2666 Flush 0 Done New Done true
6253 1668 Acked 2667 Write 4 Done New Done true
6254 1669 Acked 2668 Read 8 Done New Done true
6255 1670 Acked 2669 Read 3 Done New Done true
6256 1671 Acked 2670 Read 2 Done New Done true
6257 1672 Acked 2671 Read 4 Done New Done true
6258 1673 Acked 2672 Read 10 Done New Done true
6259 1674 Acked 2673 Write 10 Done New Done true
6260 1675 Acked 2674 Read 10 Done New Done true
6261 1676 Acked 2675 Read 4 Done New Done true
6262 1677 Acked 2676 Flush 0 Done New Done true
6263 1678 Acked 2677 Read 8 Done New Done true
6264 1679 Acked 2678 Write 6 Done New Done true
6265 1680 Acked 2679 Write 5 Done New Done true
6266 1681 Acked 2680 Read 7 Done New Done true
6267 1682 Acked 2681 Read 4 Done New Done true
6268 1683 Acked 2682 Flush 0 Done New Done true
6269 1684 Acked 2683 Read 8 Done New Done true
6270 1685 Acked 2684 Read 3 Done New Done true
6271 1686 Acked 2685 Write 5 Done New Done true
6272 1687 Acked 2686 Read 2 Done New Done true
6273 1688 Acked 2687 Write 6 Done New Done true
6274 1689 Acked 2688 Write 1 Done New Done true
6275 1690 Acked 2689 Write 4 Done New Done true
6276 1691 Acked 2690 Flush 0 Done New Done true
6277 1692 Acked 2691 Read 10 Done New Done true
6278 1693 Acked 2692 Flush 0 Done New Done true
6279 1694 Acked 2693 Read 1 Done New Done true
6280 1695 Acked 2694 Write 10 Done New Done true
6281 1696 Acked 2695 Read 9 Done New Done true
6282 1697 Acked 2696 Read 6 Done New Done true
6283 1698 Acked 2697 Read 1 Done New Done true
6284 1699 Acked 2698 Read 8 Done New Done true
6285 1700 Acked 2699 Write 5 Done New Done true
6286 1701 Acked 2700 Read 2 Done New Done true
6287 1702 Acked 2701 Write 3 Done New Done true
6288 1703 Acked 2702 Read 3 Done New Done true
6289 1704 Acked 2703 Read 1 Done New Done true
6290 1705 Acked 2704 Read 9 Done New Done true
6291 1706 Acked 2705 Write 5 Done New Done true
6292 1707 Acked 2706 Read 10 Done New Done true
6293 1708 Acked 2707 Write 8 Done New Done true
6294 1709 Acked 2708 Read 6 Done New Done true
6295 1710 Acked 2709 Read 5 Done New Done true
6296 1711 Acked 2710 Read 8 Done New Done true
6297 1712 Acked 2711 Read 10 Done New Done true
6298 1713 Acked 2712 Read 7 Done New Done true
6299 1714 Acked 2713 Read 8 Done New Done true
6300 1715 Acked 2714 Write 6 Done New Done true
6301 1716 Acked 2715 Write 6 Done New Done true
6302 1717 Acked 2716 Read 3 Done New Done true
6303 1718 Acked 2717 Read 2 Done New Done true
6304 1719 Acked 2718 Flush 0 Done New Done true
6305 1720 Acked 2719 Read 9 Done New Done true
6306 1721 Acked 2720 Read 7 Done New Done true
6307 1722 Acked 2721 Read 5 Done New Done true
6308 1723 Acked 2722 Read 3 Done New Done true
6309 1724 Acked 2723 Write 3 Done New Done true
6310 1725 Acked 2724 Write 7 Done New Done true
6311 1726 Acked 2725 Flush 0 Done New Done true
6312 1727 Acked 2726 Write 3 Done New Done true
6313 1728 Acked 2727 Flush 0 Done New Done true
6314 1729 Acked 2728 Write 7 Done New Done true
6315 1730 Acked 2729 Read 8 Done New Done true
6316 1731 Acked 2730 Write 8 Done New Done true
6317 1732 Acked 2731 Write 10 Done New Done true
6318 1733 Acked 2732 Write 6 Done New Done true
6319 1734 Acked 2733 Write 8 Done New Done true
6320 1735 Acked 2734 Write 4 Done New Done true
6321 1736 Acked 2735 Write 7 Done New Done true
6322 1737 Acked 2736 Write 9 Done New Done true
6323 1738 Acked 2737 Read 4 Done New Done true
6324 1739 Acked 2738 Read 5 Done New Done true
6325 1740 Acked 2739 Read 7 Done New Done true
6326 1741 Acked 2740 Write 9 Done New Done true
6327 1742 Acked 2741 Read 5 Done New Done true
6328 1743 Acked 2742 Write 10 Done New Done true
6329 1744 Acked 2743 Write 1 Done New Done true
6330 1745 Acked 2744 Read 1 Done New Done true
6331 1746 Acked 2745 Write 10 Done New Done true
6332 1747 Acked 2746 Flush 0 Done New Done true
6333 1748 Acked 2747 Read 8 Done New Done true
6334 1749 Acked 2748 Read 9 Done New Done true
6335 1750 Acked 2749 Write 5 Done New Done true
6336 1751 Acked 2750 Write 2 Done New Done true
6337 1752 Acked 2751 Read 5 Done New Done true
6338 1753 Acked 2752 Read 9 Done New Done true
6339 1754 Acked 2753 Write 8 Done New Done true
6340 1755 Acked 2754 Read 4 Done New Done true
6341 1756 Acked 2755 Read 10 Done New Done true
6342 1757 Acked 2756 Read 1 Done New Done true
6343 1758 Acked 2757 Read 7 Done New Done true
6344 1759 Acked 2758 Write 4 Done New Done true
6345 1760 Acked 2759 Write 6 Done New Done true
6346 1761 Acked 2760 Write 3 Done New Done true
6347 1762 Acked 2761 Write 5 Done New Done true
6348 1763 Acked 2762 Read 9 Done New Done true
6349 1764 Acked 2763 Read 4 Done New Done true
6350 1765 Acked 2764 Read 4 Done New Done true
6351 1766 Acked 2765 Read 6 Done New Done true
6352 1767 Acked 2766 Write 7 Done New Done true
6353 1768 Acked 2767 Read 3 Done New Done true
6354 1769 Acked 2768 Read 5 Done New Done true
6355 1770 Acked 2769 Write 3 Done New Done true
6356 1771 Acked 2770 Read 10 Done New Done true
6357 1772 Acked 2771 Flush 0 Done New Done true
6358 1773 Acked 2772 Write 10 Done New Done true
6359 1774 Acked 2773 Flush 0 Done New Done true
6360 1775 Acked 2774 Write 5 Done New Done true
6361 1776 Acked 2775 Read 7 Done New Done true
6362 1777 Acked 2776 Read 7 Done New Done true
6363 1778 Acked 2777 Flush 0 Done New Done true
6364 1779 Acked 2778 Write 4 Done New Done true
6365 1780 Acked 2779 Read 5 Done New Done true
6366 1781 Acked 2780 Write 4 Done New Done true
6367 1782 Acked 2781 Write 2 Done New Done true
6368 1783 Acked 2782 Read 6 Done New Done true
6369 1784 Acked 2783 Write 7 Done New Done true
6370 1785 Acked 2784 Read 1 Done New Done true
6371 1786 Acked 2785 Read 9 Done New Done true
6372 1787 Acked 2786 Write 9 Done New Done true
6373 1788 Acked 2787 Read 2 Done New Done true
6374 1789 Acked 2788 Write 5 Done New Done true
6375 1790 Acked 2789 Read 1 Done New Done true
6376 1791 Acked 2790 Write 3 Done New Done true
6377 1792 Acked 2791 Write 10 Done New Done true
6378 1793 Acked 2792 Write 8 Done New Done true
6379 1794 Acked 2793 Flush 0 Done New Done true
6380 1795 Acked 2794 Read 6 Done New Done true
6381 1796 Acked 2795 Write 1 Done New Done true
6382 1797 Acked 2796 Read 10 Done New Done true
6383 1798 Acked 2797 Write 1 Done New Done true
6384 1799 Acked 2798 Flush 0 Done New Done true
6385 1800 Acked 2799 Read 5 Done New Done true
6386 1801 Acked 2800 Write 8 Done New Done true
6387 1802 Acked 2801 Write 3 Done New Done true
6388 1803 Acked 2802 Write 10 Done New Done true
6389 1804 Acked 2803 Write 7 Done New Done true
6390 1805 Acked 2804 Read 9 Done New Done true
6391 1806 Acked 2805 Write 5 Done New Done true
6392 1807 Acked 2806 Write 2 Done New Done true
6393 1808 Acked 2807 Flush 0 Done New Done true
6394 STATES DS:0 DS:1 DS:2 TOTAL
6395 New 0 302 0 302
6396 Sent 0 0 0 0
6397 Done 302 0 302 604
6398 Skipped 0 0 0 0
6399 Error 0 0 0 0
6400 Last Flush: 2807 2505 2807
6401 Downstairs last five completed: 2505 2504 2503 2502 2501
6402 Upstairs last five completed: 1808 1807 1806 1805 1804
6403 CLIENT: Up:0 ds:302 act:2
64042024-02-27T21:45:52.760ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
64052024-02-27T21:45:52.760ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
64062024-02-27T21:45:52.760ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
64072024-02-27T21:45:52.760ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
64082024-02-27T21:45:52.760ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
64092024-02-27T21:45:52.760ZINFOcrucible: [1] client re-new 302 jobs since flush 2505 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
64102024-02-27T21:45:52.760ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
64112024-02-27T21:45:53.761ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
64122024-02-27T21:45:53.761ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
64132024-02-27T21:45:53.761ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
64142024-02-27T21:45:53.761ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
64152024-02-27T21:45:53.761ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
64162024-02-27T21:45:53.761ZINFOcrucible: [1] client re-new 302 jobs since flush 2505 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
64172024-02-27T21:45:53.762ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
64182024-02-27T21:45:54.762ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
64192024-02-27T21:45:54.762ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
64202024-02-27T21:45:54.762ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
64212024-02-27T21:45:54.762ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
64222024-02-27T21:45:54.762ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
64232024-02-27T21:45:54.763ZINFOcrucible: [1] client re-new 302 jobs since flush 2505 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
64242024-02-27T21:45:54.763ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
6425 [8820] got start action
6426 [8820] I am going to restart
6427 Starting downstairs at port 8820
6428 Make output file at "/tmp/dsc/downstairs-8820.txt"
6429 [8820][1] reports Starting
6430 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1173)
6431 [8820][1] reports Running
64322024-02-27T21:45:55.765ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
64332024-02-27T21:45:55.765ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
64342024-02-27T21:45:55.765ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
64352024-02-27T21:45:55.765ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
64362024-02-27T21:45:55.765ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
64372024-02-27T21:45:55.765ZINFOcrucible: [1] client re-new 302 jobs since flush 2505 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
64382024-02-27T21:45:55.765ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
6439 ----------------------------------------------------------------
6440 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
6441 States: Active Offline Active
6442 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
6443 1507 Acked 2506 Write 6 Done New Done true
6444 1508 Acked 2507 Read 10 Done New Done true
6445 1509 Acked 2508 Read 6 Done New Done true
6446 1510 Acked 2509 Read 7 Done New Done true
6447 1511 Acked 2510 Read 5 Done New Done true
6448 1512 Acked 2511 Read 3 Done New Done true
6449 1513 Acked 2512 Read 4 Done New Done true
6450 1514 Acked 2513 Flush 0 Done New Done true
6451 1515 Acked 2514 Read 8 Done New Done true
6452 1516 Acked 2515 Read 7 Done New Done true
6453 1517 Acked 2516 Write 10 Done New Done true
6454 1518 Acked 2517 Write 2 Done New Done true
6455 1519 Acked 2518 Read 7 Done New Done true
6456 1520 Acked 2519 Read 7 Done New Done true
6457 1521 Acked 2520 Read 7 Done New Done true
6458 1522 Acked 2521 Read 8 Done New Done true
6459 1523 Acked 2522 Read 4 Done New Done true
6460 1524 Acked 2523 Write 4 Done New Done true
6461 1525 Acked 2524 Read 3 Done New Done true
6462 1526 Acked 2525 Write 6 Done New Done true
6463 1527 Acked 2526 Read 2 Done New Done true
6464 1528 Acked 2527 Read 5 Done New Done true
6465 1529 Acked 2528 Read 2 Done New Done true
6466 1530 Acked 2529 Read 2 Done New Done true
6467 1531 Acked 2530 Write 6 Done New Done true
6468 1532 Acked 2531 Read 10 Done New Done true
6469 1533 Acked 2532 Write 9 Done New Done true
6470 1534 Acked 2533 Read 10 Done New Done true
6471 1535 Acked 2534 Read 4 Done New Done true
6472 1536 Acked 2535 Read 3 Done New Done true
6473 1537 Acked 2536 Read 4 Done New Done true
6474 1538 Acked 2537 Write 2 Done New Done true
6475 1539 Acked 2538 Read 3 Done New Done true
6476 1540 Acked 2539 Read 5 Done New Done true
6477 1541 Acked 2540 Write 7 Done New Done true
6478 1542 Acked 2541 Read 10 Done New Done true
6479 1543 Acked 2542 Write 10 Done New Done true
6480 1544 Acked 2543 Read 9 Done New Done true
6481 1545 Acked 2544 Read 6 Done New Done true
6482 1546 Acked 2545 Flush 0 Done New Done true
6483 1547 Acked 2546 Read 10 Done New Done true
6484 1548 Acked 2547 Read 3 Done New Done true
6485 1549 Acked 2548 Read 2 Done New Done true
6486 1550 Acked 2549 Write 3 Done New Done true
6487 1551 Acked 2550 Read 6 Done New Done true
6488 1552 Acked 2551 Read 3 Done New Done true
6489 1553 Acked 2552 Write 6 Done New Done true
6490 1554 Acked 2553 Read 3 Done New Done true
6491 1555 Acked 2554 Write 2 Done New Done true
6492 1556 Acked 2555 Write 4 Done New Done true
6493 1557 Acked 2556 Write 2 Done New Done true
6494 1558 Acked 2557 Read 3 Done New Done true
6495 1559 Acked 2558 Read 4 Done New Done true
6496 1560 Acked 2559 Read 4 Done New Done true
6497 1561 Acked 2560 Read 10 Done New Done true
6498 1562 Acked 2561 Read 10 Done New Done true
6499 1563 Acked 2562 Read 7 Done New Done true
6500 1564 Acked 2563 Read 1 Done New Done true
6501 1565 Acked 2564 Read 7 Done New Done true
6502 1566 Acked 2565 Read 1 Done New Done true
6503 1567 Acked 2566 Write 5 Done New Done true
6504 1568 Acked 2567 Read 10 Done New Done true
6505 1569 Acked 2568 Write 9 Done New Done true
6506 1570 Acked 2569 Read 6 Done New Done true
6507 1571 Acked 2570 Write 9 Done New Done true
6508 1572 Acked 2571 Write 5 Done New Done true
6509 1573 Acked 2572 Read 2 Done New Done true
6510 1574 Acked 2573 Write 2 Done New Done true
6511 1575 Acked 2574 Read 9 Done New Done true
6512 1576 Acked 2575 Write 10 Done New Done true
6513 1577 Acked 2576 Read 1 Done New Done true
6514 1578 Acked 2577 Read 3 Done New Done true
6515 1579 Acked 2578 Read 9 Done New Done true
6516 1580 Acked 2579 Read 8 Done New Done true
6517 1581 Acked 2580 Read 3 Done New Done true
6518 1582 Acked 2581 Write 4 Done New Done true
6519 1583 Acked 2582 Write 8 Done New Done true
6520 1584 Acked 2583 Write 1 Done New Done true
6521 1585 Acked 2584 Read 2 Done New Done true
6522 1586 Acked 2585 Write 3 Done New Done true
6523 1587 Acked 2586 Read 6 Done New Done true
6524 1588 Acked 2587 Read 7 Done New Done true
6525 1589 Acked 2588 Read 5 Done New Done true
6526 1590 Acked 2589 Write 9 Done New Done true
6527 1591 Acked 2590 Write 10 Done New Done true
6528 1592 Acked 2591 Write 3 Done New Done true
6529 1593 Acked 2592 Write 9 Done New Done true
6530 1594 Acked 2593 Read 5 Done New Done true
6531 1595 Acked 2594 Write 1 Done New Done true
6532 1596 Acked 2595 Read 1 Done New Done true
6533 1597 Acked 2596 Write 7 Done New Done true
6534 1598 Acked 2597 Read 6 Done New Done true
6535 1599 Acked 2598 Write 4 Done New Done true
6536 1600 Acked 2599 Read 7 Done New Done true
6537 1601 Acked 2600 Write 2 Done New Done true
6538 1602 Acked 2601 Read 8 Done New Done true
6539 1603 Acked 2602 Write 6 Done New Done true
6540 1604 Acked 2603 Write 8 Done New Done true
6541 1605 Acked 2604 Write 1 Done New Done true
6542 1606 Acked 2605 Read 7 Done New Done true
6543 1607 Acked 2606 Write 6 Done New Done true
6544 1608 Acked 2607 Read 6 Done New Done true
6545 1609 Acked 2608 Read 6 Done New Done true
6546 1610 Acked 2609 Read 7 Done New Done true
6547 1611 Acked 2610 Write 10 Done New Done true
6548 1612 Acked 2611 Read 10 Done New Done true
6549 1613 Acked 2612 Write 2 Done New Done true
6550 1614 Acked 2613 Write 5 Done New Done true
6551 1615 Acked 2614 Read 3 Done New Done true
6552 1616 Acked 2615 Read 4 Done New Done true
6553 1617 Acked 2616 Write 9 Done New Done true
6554 1618 Acked 2617 Write 3 Done New Done true
6555 1619 Acked 2618 Read 5 Done New Done true
6556 1620 Acked 2619 Read 5 Done New Done true
6557 1621 Acked 2620 Read 10 Done New Done true
6558 1622 Acked 2621 Read 5 Done New Done true
6559 1623 Acked 2622 Flush 0 Done New Done true
6560 1624 Acked 2623 Read 8 Done New Done true
6561 1625 Acked 2624 Write 4 Done New Done true
6562 1626 Acked 2625 Read 2 Done New Done true
6563 1627 Acked 2626 Read 3 Done New Done true
6564 1628 Acked 2627 Write 4 Done New Done true
6565 1629 Acked 2628 Read 4 Done New Done true
6566 1630 Acked 2629 Read 2 Done New Done true
6567 1631 Acked 2630 Read 7 Done New Done true
6568 1632 Acked 2631 Read 4 Done New Done true
6569 1633 Acked 2632 Read 6 Done New Done true
6570 1634 Acked 2633 Read 3 Done New Done true
6571 1635 Acked 2634 Read 3 Done New Done true
6572 1636 Acked 2635 Write 8 Done New Done true
6573 1637 Acked 2636 Read 1 Done New Done true
6574 1638 Acked 2637 Write 6 Done New Done true
6575 1639 Acked 2638 Read 4 Done New Done true
6576 1640 Acked 2639 Read 9 Done New Done true
6577 1641 Acked 2640 Read 7 Done New Done true
6578 1642 Acked 2641 Read 3 Done New Done true
6579 1643 Acked 2642 Read 6 Done New Done true
6580 1644 Acked 2643 Flush 0 Done New Done true
6581 1645 Acked 2644 Write 5 Done New Done true
6582 1646 Acked 2645 Write 2 Done New Done true
6583 1647 Acked 2646 Write 3 Done New Done true
6584 1648 Acked 2647 Write 9 Done New Done true
6585 1649 Acked 2648 Read 5 Done New Done true
6586 1650 Acked 2649 Write 3 Done New Done true
6587 1651 Acked 2650 Flush 0 Done New Done true
6588 1652 Acked 2651 Read 8 Done New Done true
6589 1653 Acked 2652 Read 10 Done New Done true
6590 1654 Acked 2653 Read 2 Done New Done true
6591 1655 Acked 2654 Write 9 Done New Done true
6592 1656 Acked 2655 Write 7 Done New Done true
6593 1657 Acked 2656 Read 4 Done New Done true
6594 1658 Acked 2657 Read 8 Done New Done true
6595 1659 Acked 2658 Write 7 Done New Done true
6596 1660 Acked 2659 Flush 0 Done New Done true
6597 1661 Acked 2660 Write 5 Done New Done true
6598 1662 Acked 2661 Read 9 Done New Done true
6599 1663 Acked 2662 Flush 0 Done New Done true
6600 1664 Acked 2663 Read 7 Done New Done true
6601 1665 Acked 2664 Flush 0 Done New Done true
6602 1666 Acked 2665 Write 8 Done New Done true
6603 1667 Acked 2666 Flush 0 Done New Done true
6604 1668 Acked 2667 Write 4 Done New Done true
6605 1669 Acked 2668 Read 8 Done New Done true
6606 1670 Acked 2669 Read 3 Done New Done true
6607 1671 Acked 2670 Read 2 Done New Done true
6608 1672 Acked 2671 Read 4 Done New Done true
6609 1673 Acked 2672 Read 10 Done New Done true
6610 1674 Acked 2673 Write 10 Done New Done true
6611 1675 Acked 2674 Read 10 Done New Done true
6612 1676 Acked 2675 Read 4 Done New Done true
6613 1677 Acked 2676 Flush 0 Done New Done true
6614 1678 Acked 2677 Read 8 Done New Done true
6615 1679 Acked 2678 Write 6 Done New Done true
6616 1680 Acked 2679 Write 5 Done New Done true
6617 1681 Acked 2680 Read 7 Done New Done true
6618 1682 Acked 2681 Read 4 Done New Done true
6619 1683 Acked 2682 Flush 0 Done New Done true
6620 1684 Acked 2683 Read 8 Done New Done true
6621 1685 Acked 2684 Read 3 Done New Done true
6622 1686 Acked 2685 Write 5 Done New Done true
6623 1687 Acked 2686 Read 2 Done New Done true
6624 1688 Acked 2687 Write 6 Done New Done true
6625 1689 Acked 2688 Write 1 Done New Done true
6626 1690 Acked 2689 Write 4 Done New Done true
6627 1691 Acked 2690 Flush 0 Done New Done true
6628 1692 Acked 2691 Read 10 Done New Done true
6629 1693 Acked 2692 Flush 0 Done New Done true
6630 1694 Acked 2693 Read 1 Done New Done true
6631 1695 Acked 2694 Write 10 Done New Done true
6632 1696 Acked 2695 Read 9 Done New Done true
6633 1697 Acked 2696 Read 6 Done New Done true
6634 1698 Acked 2697 Read 1 Done New Done true
6635 1699 Acked 2698 Read 8 Done New Done true
6636 1700 Acked 2699 Write 5 Done New Done true
6637 1701 Acked 2700 Read 2 Done New Done true
6638 1702 Acked 2701 Write 3 Done New Done true
6639 1703 Acked 2702 Read 3 Done New Done true
6640 1704 Acked 2703 Read 1 Done New Done true
6641 1705 Acked 2704 Read 9 Done New Done true
6642 1706 Acked 2705 Write 5 Done New Done true
6643 1707 Acked 2706 Read 10 Done New Done true
6644 1708 Acked 2707 Write 8 Done New Done true
6645 1709 Acked 2708 Read 6 Done New Done true
6646 1710 Acked 2709 Read 5 Done New Done true
6647 1711 Acked 2710 Read 8 Done New Done true
6648 1712 Acked 2711 Read 10 Done New Done true
6649 1713 Acked 2712 Read 7 Done New Done true
6650 1714 Acked 2713 Read 8 Done New Done true
6651 1715 Acked 2714 Write 6 Done New Done true
6652 1716 Acked 2715 Write 6 Done New Done true
6653 1717 Acked 2716 Read 3 Done New Done true
6654 1718 Acked 2717 Read 2 Done New Done true
6655 1719 Acked 2718 Flush 0 Done New Done true
6656 1720 Acked 2719 Read 9 Done New Done true
6657 1721 Acked 2720 Read 7 Done New Done true
6658 1722 Acked 2721 Read 5 Done New Done true
6659 1723 Acked 2722 Read 3 Done New Done true
6660 1724 Acked 2723 Write 3 Done New Done true
6661 1725 Acked 2724 Write 7 Done New Done true
6662 1726 Acked 2725 Flush 0 Done New Done true
6663 1727 Acked 2726 Write 3 Done New Done true
6664 1728 Acked 2727 Flush 0 Done New Done true
6665 1729 Acked 2728 Write 7 Done New Done true
6666 1730 Acked 2729 Read 8 Done New Done true
6667 1731 Acked 2730 Write 8 Done New Done true
6668 1732 Acked 2731 Write 10 Done New Done true
6669 1733 Acked 2732 Write 6 Done New Done true
6670 1734 Acked 2733 Write 8 Done New Done true
6671 1735 Acked 2734 Write 4 Done New Done true
6672 1736 Acked 2735 Write 7 Done New Done true
6673 1737 Acked 2736 Write 9 Done New Done true
6674 1738 Acked 2737 Read 4 Done New Done true
6675 1739 Acked 2738 Read 5 Done New Done true
6676 1740 Acked 2739 Read 7 Done New Done true
6677 1741 Acked 2740 Write 9 Done New Done true
6678 1742 Acked 2741 Read 5 Done New Done true
6679 1743 Acked 2742 Write 10 Done New Done true
6680 1744 Acked 2743 Write 1 Done New Done true
6681 1745 Acked 2744 Read 1 Done New Done true
6682 1746 Acked 2745 Write 10 Done New Done true
6683 1747 Acked 2746 Flush 0 Done New Done true
6684 1748 Acked 2747 Read 8 Done New Done true
6685 1749 Acked 2748 Read 9 Done New Done true
6686 1750 Acked 2749 Write 5 Done New Done true
6687 1751 Acked 2750 Write 2 Done New Done true
6688 1752 Acked 2751 Read 5 Done New Done true
6689 1753 Acked 2752 Read 9 Done New Done true
6690 1754 Acked 2753 Write 8 Done New Done true
6691 1755 Acked 2754 Read 4 Done New Done true
6692 1756 Acked 2755 Read 10 Done New Done true
6693 1757 Acked 2756 Read 1 Done New Done true
6694 1758 Acked 2757 Read 7 Done New Done true
6695 1759 Acked 2758 Write 4 Done New Done true
6696 1760 Acked 2759 Write 6 Done New Done true
6697 1761 Acked 2760 Write 3 Done New Done true
6698 1762 Acked 2761 Write 5 Done New Done true
6699 1763 Acked 2762 Read 9 Done New Done true
6700 1764 Acked 2763 Read 4 Done New Done true
6701 1765 Acked 2764 Read 4 Done New Done true
6702 1766 Acked 2765 Read 6 Done New Done true
6703 1767 Acked 2766 Write 7 Done New Done true
6704 1768 Acked 2767 Read 3 Done New Done true
6705 1769 Acked 2768 Read 5 Done New Done true
6706 1770 Acked 2769 Write 3 Done New Done true
6707 1771 Acked 2770 Read 10 Done New Done true
6708 1772 Acked 2771 Flush 0 Done New Done true
6709 1773 Acked 2772 Write 10 Done New Done true
6710 1774 Acked 2773 Flush 0 Done New Done true
6711 1775 Acked 2774 Write 5 Done New Done true
6712 1776 Acked 2775 Read 7 Done New Done true
6713 1777 Acked 2776 Read 7 Done New Done true
6714 1778 Acked 2777 Flush 0 Done New Done true
6715 1779 Acked 2778 Write 4 Done New Done true
6716 1780 Acked 2779 Read 5 Done New Done true
6717 1781 Acked 2780 Write 4 Done New Done true
6718 1782 Acked 2781 Write 2 Done New Done true
6719 1783 Acked 2782 Read 6 Done New Done true
6720 1784 Acked 2783 Write 7 Done New Done true
6721 1785 Acked 2784 Read 1 Done New Done true
6722 1786 Acked 2785 Read 9 Done New Done true
6723 1787 Acked 2786 Write 9 Done New Done true
6724 1788 Acked 2787 Read 2 Done New Done true
6725 1789 Acked 2788 Write 5 Done New Done true
6726 1790 Acked 2789 Read 1 Done New Done true
6727 1791 Acked 2790 Write 3 Done New Done true
6728 1792 Acked 2791 Write 10 Done New Done true
6729 1793 Acked 2792 Write 8 Done New Done true
6730 1794 Acked 2793 Flush 0 Done New Done true
6731 1795 Acked 2794 Read 6 Done New Done true
6732 1796 Acked 2795 Write 1 Done New Done true
6733 1797 Acked 2796 Read 10 Done New Done true
6734 1798 Acked 2797 Write 1 Done New Done true
6735 1799 Acked 2798 Flush 0 Done New Done true
6736 1800 Acked 2799 Read 5 Done New Done true
6737 1801 Acked 2800 Write 8 Done New Done true
6738 1802 Acked 2801 Write 3 Done New Done true
6739 1803 Acked 2802 Write 10 Done New Done true
6740 1804 Acked 2803 Write 7 Done New Done true
6741 1805 Acked 2804 Read 9 Done New Done true
6742 1806 Acked 2805 Write 5 Done New Done true
6743 1807 Acked 2806 Write 2 Done New Done true
6744 1808 Acked 2807 Flush 0 Done New Done true
6745 STATES DS:0 DS:1 DS:2 TOTAL
6746 New 0 302 0 302
6747 Sent 0 0 0 0
6748 Done 302 0 302 604
6749 Skipped 0 0 0 0
6750 Error 0 0 0 0
6751 Last Flush: 2807 2505 2807
6752 Downstairs last five completed: 2505 2504 2503 2502 2501
6753 Upstairs last five completed: 1808 1807 1806 1805 1804
6754 CLIENT: Up:0 ds:302 act:2
67552024-02-27T21:45:56.766ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
67562024-02-27T21:45:56.766ZINFOcrucible: ds_connection connected = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
67572024-02-27T21:45:56.768ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
67582024-02-27T21:45:56.768ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
67592024-02-27T21:45:56.768ZINFOcrucible: send last flush ID to this DS: 2505 = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
67602024-02-27T21:45:56.768ZINFOcrucible: Replied this last flush ID: 2505 = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
67612024-02-27T21:45:56.768ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
67622024-02-27T21:45:56.768ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
67632024-02-27T21:45:56.768ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
67642024-02-27T21:45:56.768ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
6765 ----------------------------------------------------------------
6766 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
6767 Downstairs last five completed: 2807 2806 2805 2804 2803
6768 Upstairs last five completed: 1808 1807 1806 1805 1804
6769 CLIENT: Up:0 ds:0 act:3
6770 Replay: All jobs finished, all DS active.
6771 Main task has work to do, go find it
6772 got dsc Stop(2)
6773 stop 2
6774 Feb 27 21:46:00.107 INFO request completed, latency_us: 345, response_code: 204, uri: /stop/cid/2, method[8830] Got stop action so:false kr:false
6775 : POST, req_id: 7cc2dab7-7c1e-423a-80e5-f076f12db0ec, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
67762024-02-27T21:46:00.108ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
67772024-02-27T21:46:00.108ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
67782024-02-27T21:46:00.108ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = e6a5a974-925a-494f-8845-facc441c2690
67792024-02-27T21:46:00.108ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
67802024-02-27T21:46:00.108ZINFOcrucible: [2] client re-new 0 jobs since flush 2807 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
67812024-02-27T21:46:00.108ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
6782 Feb 27 21:46:00.108 INFO request completed, latency_us: 194, response_code: 200, uri: /state/cid/2, method: GET, req_id: 3571da39-783a-4a76-a3ee-1957eec1f8ab, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
6783 [8830] Exited with: ExitStatus(unix_wait_status(9))
6784 [8830][2] reports Exit
67852024-02-27T21:46:01.109ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
67862024-02-27T21:46:01.109ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
67872024-02-27T21:46:01.110ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
67882024-02-27T21:46:01.110ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
67892024-02-27T21:46:01.110ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
67902024-02-27T21:46:01.110ZINFOcrucible: [2] client re-new 0 jobs since flush 2807 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
67912024-02-27T21:46:01.110ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
67922024-02-27T21:46:02.111ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
67932024-02-27T21:46:02.111ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
67942024-02-27T21:46:02.111ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
67952024-02-27T21:46:02.111ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
67962024-02-27T21:46:02.111ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
67972024-02-27T21:46:02.112ZINFOcrucible: [2] client re-new 0 jobs since flush 2807 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
67982024-02-27T21:46:02.112ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
67992024-02-27T21:46:03.113ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
68002024-02-27T21:46:03.113ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
68012024-02-27T21:46:03.113ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
68022024-02-27T21:46:03.113ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
68032024-02-27T21:46:03.113ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
68042024-02-27T21:46:03.113ZINFOcrucible: [2] client re-new 0 jobs since flush 2807 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
68052024-02-27T21:46:03.113ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
6806 Feb 27 21:46:04.111 INFO request completed, latency_us: 237, response_code: 200, uri: /state/cid/2, method: GET, req_id: 7e8096bb-4037-42eb-8029-599fb1711530, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
6807 001/300 Read block 55 len 36864
6808 002/300 Write block 561 len 32768 data: 2 2 2 2 2 2 2 2
68092024-02-27T21:46:04.115ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
68102024-02-27T21:46:04.115ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
68112024-02-27T21:46:04.115ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
68122024-02-27T21:46:04.115ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
68132024-02-27T21:46:04.115ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
68142024-02-27T21:46:04.115ZINFOcrucible: [2] client re-new 2 jobs since flush 2807 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
68152024-02-27T21:46:04.115ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
6816 003/300 Write block 1705 len 36864 data: 2 2 2 2 2 2 2 3 4
6817 004/300 Read block 2505 len 40960
6818 005/300 Read block 4310 len 28672
6819 006/300 Write block 2555 len 36864 data: 3 3 2 2 2 2 2 2 2
6820 007/300 Flush
6821 008/300 Write block 4877 len 36864 data: 3 3 2 2 2 2 3 3 4
6822 009/300 Write block 3309 len 36864 data: 3 3 3 3 3 3 3 4 4
6823 010/300 Read block 4372 len 16384
6824 011/300 Read block 4209 len 20480
6825 012/300 Read block 1850 len 16384
6826 013/300 Write block 63 len 4096 data: 2
6827 014/300 Read block 3412 len 16384
6828 015/300 Flush
6829 016/300 Read block 3698 len 12288
6830 017/300 Read block 1535 len 28672
6831 018/300 Write block 4629 len 8192 data: 4 4
6832 019/300 Write block 1703 len 40960 data: 2 2 3 3 3 3 3 3 3 4
6833 020/300 Write block 2349 len 4096 data: 5
6834 021/300 Write block 3816 len 12288 data: 4 3 3
6835 022/300 Read block 3966 len 24576
6836 023/300 Flush
6837 024/300 Write block 2178 len 28672 data: 2 2 2 2 2 2 2
6838 025/300 Read block 1823 len 8192
6839 026/300 Read block 4983 len 4096
6840 027/300 Read block 3610 len 28672
6841 028/300 Write block 3063 len 32768 data: 3 3 3 3 2 2 2 2
6842 029/300 Read block 4455 len 4096
6843 030/300 Read block 1039 len 16384
6844 031/300 Read block 734 len 32768
6845 032/300 Write block 2335 len 32768 data: 4 4 4 3 4 3 3 3
6846 033/300 Write block 1329 len 8192 data: 2 2
6847 034/300 Read block 3903 len 40960
6848 035/300 Read block 90 len 16384
6849 036/300 Read block 2745 len 40960
6850 037/300 Write block 1842 len 28672 data: 2 2 2 2 2 2 3
6851 038/300 Read block 3781 len 24576
6852 039/300 Write block 3754 len 24576 data: 2 2 2 2 2 3
6853 040/300 Write block 2173 len 8192 data: 2 2
6854 041/300 Write block 4498 len 12288 data: 2 2 2
6855 042/300 Read block 4590 len 40960
6856 043/300 Write block 566 len 8192 data: 3 3
6857 044/300 Read block 1158 len 12288
6858 045/300 Write block 1186 len 12288 data: 2 2 2
6859 046/300 Read block 734 len 20480
6860 047/300 Read block 4748 len 8192
6861 048/300 Read block 574 len 36864
6862 049/300 Flush
6863 050/300 Read block 2086 len 16384
6864 051/300 Flush
6865 052/300 Write block 2998 len 20480 data: 4 4 5 4 4
6866 053/300 Read block 3479 len 8192
6867 054/300 Read block 2540 len 40960
6868 055/300 Write block 1520 len 8192 data: 2 2
6869 056/300 Write block 3023 len 32768 data: 5 5 4 4 4 4 4 4
6870 057/300 Write block 4956 len 20480 data: 2 2 2 2 2
6871 058/300 Read block 3389 len 16384
6872 059/300 Write block 206 len 16384 data: 3 3 3 2
6873 060/300 Read block 2900 len 16384
6874 061/300 Read block 1897 len 32768
6875 062/300 Read block 2013 len 32768
6876 063/300 Read block 1120 len 20480
6877 064/300 Write block 1738 len 16384 data: 2 2 2 2
6878 065/300 Read block 1462 len 28672
6879 066/300 Read block 3200 len 32768
6880 067/300 Read block 3071 len 36864
6881 068/300 Write block 4496 len 32768 data: 2 2 3 3 3 2 2 2
6882 069/300 Read block 4136 len 16384
6883 070/300 Write block 1505 len 36864 data: 2 2 2 2 3 3 3 3 2
6884 071/300 Read block 1743 len 24576
6885 072/300 Read block 4464 len 20480
6886 073/300 Read block 1208 len 20480
6887 074/300 Read block 3289 len 12288
6888 075/300 Read block 554 len 36864
6889 076/300 Write block 4395 len 32768 data: 2 3 4 4 4 4 3 3
6890 077/300 Flush
6891 078/300 Read block 128 len 16384
6892 079/300 Write block 1578 len 36864 data: 2 2 2 2 2 2 2 2 2
6893 080/300 Write block 2819 len 28672 data: 3 2 2 2 2 2 2
6894 081/300 Read block 1051 len 36864
6895 082/300 Read block 1451 len 12288
6896 083/300 Read block 571 len 12288
6897 084/300 Read block 1037 len 28672
6898 085/300 Read block 2133 len 12288
6899 086/300 Read block 1737 len 32768
6900 087/300 Write block 3239 len 12288 data: 4 4 4
6901 088/300 Write block 584 len 32768 data: 2 2 2 2 2 2 2 2
6902 089/300 Write block 2421 len 24576 data: 2 2 2 2 2 2
6903 090/300 Read block 1818 len 40960
6904 091/300 Read block 2285 len 28672
6905 092/300 Write block 2062 len 4096 data: 3
6906 093/300 Read block 2604 len 16384
6907 094/300 Read block 991 len 12288
6908 095/300 Write block 3514 len 32768 data: 2 2 2 2 2 2 2 2
6909 096/300 Read block 2976 len 32768
6910 097/300 Read block 2950 len 12288
6911 098/300 Read block 2933 len 20480
6912 099/300 Flush
6913 100/300 Read block 4114 len 36864
6914 101/300 Write block 2169 len 8192 data: 3 2
6915 102/300 Write block 2623 len 36864 data: 3 4 4 4 4 3 3 3 2
6916 103/300 Read block 118 len 4096
6917 104/300 Write block 1178 len 12288 data: 3 3 2
6918 105/300 Write block 3284 len 4096 data: 2
6919 106/300 Write block 1474 len 36864 data: 5 3 2 3 3 3 3 3 4
6920 107/300 Write block 993 len 4096 data: 3
6921 108/300 Read block 4716 len 24576
6922 109/300 Write block 3523 len 40960 data: 2 2 3 3 3 2 2 2 3 3
6923 110/300 Read block 1490 len 32768
6924 111/300 Write block 2988 len 24576 data: 2 2 2 2 2 2
6925 112/300 Write block 4690 len 16384 data: 3 3 4 4
6926 113/300 Write block 4832 len 32768 data: 3 3 3 2 2 2 2 2
6927 114/300 Read block 4897 len 28672
6928 115/300 Write block 836 len 28672 data: 2 3 4 3 3 4 4
6929 116/300 Write block 3569 len 20480 data: 3 4 4 4 4
6930 117/300 Write block 4237 len 20480 data: 2 2 2 2 2
6931 118/300 Read block 1835 len 40960
6932 119/300 Read block 4168 len 36864
6933 120/300 Flush
6934 121/300 Write block 3522 len 24576 data: 2 3 3 4 4 4
6935 122/300 Read block 1442 len 24576
6936 123/300 Read block 1879 len 36864
6937 124/300 Flush
6938 125/300 Flush
6939 126/300 Read block 3274 len 36864
6940 127/300 Flush
6941 128/300 Write block 922 len 40960 data: 2 2 2 2 2 2 2 3 3 3
6942 129/300 Flush
6943 130/300 Flush
6944 131/300 Read block 2777 len 40960
6945 132/300 Flush
6946 133/300 Write block 444 len 32768 data: 2 2 3 3 3 4 4 4
6947 134/300 Write block 3020 len 20480 data: 4 4 4 6 6
6948 135/300 Write block 4572 len 28672 data: 5 4 3 3 4 4 4
6949 136/300 Write block 1267 len 12288 data: 3 3 3
6950 137/300 Read block 2510 len 20480
6951 138/300 Read block 938 len 4096
6952 139/300 Write block 3392 len 40960 data: 2 2 2 2 3 3 3 4 4 4
6953 140/300 Read block 2101 len 36864
6954 141/300 Read block 899 len 40960
6955 142/300 Read block 3724 len 40960
6956 143/300 Write block 2830 len 32768 data: 2 2 2 2 2 2 2 2
6957 144/300 Write block 4732 len 40960 data: 2 3 3 3 3 3 4 3 3 3
6958 145/300 Read block 2099 len 16384
6959 146/300 Read block 254 len 32768
6960 147/300 Read block 178 len 8192
6961 148/300 Write block 4735 len 40960 data: 4 4 4 5 4 4 4 3 3 5
6962 149/300 Read block 4149 len 8192
6963 150/300 Write block 4625 len 4096 data: 2
6964 151/300 Flush
6965 152/300 Flush
6966 153/300 Read block 437 len 8192
6967 154/300 Read block 3087 len 24576
6968 155/300 Write block 4283 len 16384 data: 2 3 2 2
6969 156/300 Write block 551 len 32768 data: 3 3 2 2 3 3 3 3
6970 157/300 Read block 1702 len 36864
6971 158/300 Write block 3359 len 12288 data: 2 2 2
6972 159/300 Write block 4221 len 12288 data: 2 2 2
6973 160/300 Read block 78 len 40960
6974 161/300 Write block 3982 len 4096 data: 5
6975 162/300 Read block 2532 len 8192
6976 163/300 Read block 4019 len 24576
6977 164/300 Read block 432 len 24576
6978 165/300 Write block 1612 len 24576 data: 2 3 3 5 5 5
6979 166/300 Read block 752 len 12288
6980 167/300 Read block 344 len 4096
6981 168/300 Read block 1035 len 12288
6982 169/300 Read block 4415 len 16384
6983 170/300 Write block 4621 len 20480 data: 3 4 4 4 3
6984 171/300 Write block 2156 len 32768 data: 2 2 2 2 2 2 2 2
6985 172/300 Write block 559 len 4096 data: 2
6986 173/300 Read block 3683 len 36864
6987 174/300 Read block 3061 len 36864
6988 175/300 Read block 695 len 4096
6989 176/300 Read block 3130 len 20480
6990 177/300 Read block 3976 len 8192
6991 178/300 Read block 3917 len 12288
6992 179/300 Write block 2168 len 24576 data: 3 4 3 2 2 3
6993 180/300 Read block 3261 len 12288
6994 181/300 Read block 3105 len 28672
6995 182/300 Write block 4345 len 40960 data: 2 2 2 2 2 2 2 3 3 2
6996 183/300 Write block 3852 len 24576 data: 3 3 3 3 3 2
6997 184/300 Read block 1356 len 28672
6998 185/300 Write block 3587 len 28672 data: 2 2 2 2 2 3 3
6999 186/300 Read block 4496 len 12288
7000 187/300 Write block 2593 len 8192 data: 4 3
7001 188/300 Read block 3236 len 40960
7002 189/300 Flush
7003 190/300 Write block 4969 len 28672 data: 2 3 3 3 3 3 3
7004 191/300 Write block 1546 len 36864 data: 2 2 2 2 2 2 2 2 3
7005 192/300 Write block 2780 len 12288 data: 3 3 3
7006 193/300 Read block 2438 len 20480
7007 194/300 Flush
7008 195/300 Write block 2235 len 20480 data: 3 3 3 3 3
7009 196/300 Flush
7010 197/300 Read block 2076 len 16384
7011 198/300 Write block 1458 len 28672 data: 3 2 2 2 2 2 2
7012 199/300 Read block 1961 len 20480
7013 200/300 Write block 860 len 12288 data: 3 3 3
7014 201/300 Read block 2720 len 28672
7015 202/300 Write block 717 len 20480 data: 2 2 2 2 2
7016 203/300 Write block 2823 len 16384 data: 3 3 3 2
7017 204/300 Flush
7018 205/300 Write block 1712 len 8192 data: 5 5
7019 206/300 Flush
7020 207/300 Read block 2923 len 28672
7021 208/300 Write block 16 len 40960 data: 4 4 4 4 4 3 3 4 4 4
7022 209/300 Read block 553 len 4096
7023 210/300 Write block 72 len 16384 data: 3 3 3 2
7024 211/300 Write block 2613 len 12288 data: 4 3 5
7025 212/300 Write block 4375 len 36864 data: 3 2 2 2 3 3 3 2 2
7026 213/300 Write block 246 len 28672 data: 2 2 2 2 2 2 2
7027 214/300 Flush
7028 215/300 Read block 4376 len 36864
7029 216/300 Read block 3554 len 16384
7030 217/300 Read block 4530 len 24576
7031 218/300 Read block 2510 len 16384
7032 219/300 Flush
7033 220/300 Write block 580 len 28672 data: 4 3 3 3 3 3 3
7034 221/300 Write block 1316 len 36864 data: 4 4 5 5 5 3 3 4 4
7035 222/300 Write block 965 len 36864 data: 2 2 2 2 2 2 2 2 2
7036 223/300 Read block 764 len 24576
7037 224/300 Write block 4656 len 4096 data: 4
7038 225/300 Read block 1964 len 8192
7039 226/300 Read block 2684 len 8192
7040 227/300 Write block 1137 len 28672 data: 2 2 2 2 2 2 3
7041 228/300 Read block 1606 len 12288
7042 229/300 Read block 2315 len 16384
7043 230/300 Write block 4303 len 16384 data: 3 2 2 2
7044 231/300 Write block 3700 len 28672 data: 2 2 2 2 3 3 4
7045 232/300 Read block 2248 len 40960
7046 233/300 Write block 2560 len 8192 data: 3 3
7047 234/300 Flush
7048 235/300 Write block 2153 len 40960 data: 2 2 2 3 3 3 3 3 3 3
7049 236/300 Flush
7050 237/300 Read block 3860 len 28672
7051 238/300 Write block 4608 len 36864 data: 2 2 2 2 2 2 2 2 2
7052 239/300 Read block 1625 len 20480
7053 240/300 Read block 1633 len 40960
7054 241/300 Read block 2591 len 28672
7055 242/300 Read block 2191 len 16384
7056 243/300 Flush
7057 244/300 Read block 1594 len 8192
7058 245/300 Flush
7059 246/300 Write block 1013 len 16384 data: 3 2 2 2
7060 247/300 Write block 1984 len 4096 data: 3
7061 248/300 Write block 3175 len 12288 data: 3 3 3
7062 249/300 Flush
7063 250/300 Read block 3390 len 24576
7064 251/300 Write block 4773 len 28672 data: 4 3 3 3 3 3 3
7065 252/300 Flush
7066 253/300 Read block 4578 len 20480
7067 254/300 Read block 99 len 32768
7068 255/300 Write block 874 len 16384 data: 3 3 4 4
7069 256/300 Flush
7070 257/300 Read block 3516 len 8192
7071 258/300 Write block 2955 len 36864 data: 2 2 2 3 2 2 2 2 2
7072 259/300 Read block 2196 len 16384
7073 260/300 Write block 3678 len 8192 data: 4 5
7074 261/300 Read block 3668 len 36864
7075 262/300 Flush
7076 263/300 Read block 4794 len 4096
7077 264/300 Flush
7078 265/300 Write block 1924 len 28672 data: 2 2 3 3 4 4 4
7079 266/300 Write block 1218 len 8192 data: 6 5
7080 267/300 Write block 1732 len 36864 data: 2 2 2 2 2 2 3 3 3
7081 268/300 Write block 4078 len 12288 data: 2 2 2
7082 269/300 Write block 1080 len 24576 data: 3 3 3 4 4 4
7083 270/300 Flush
7084 271/300 Read block 3086 len 32768
7085 272/300 Write block 4649 len 4096 data: 3
7086 273/300 Read block 1352 len 36864
7087 274/300 Read block 4062 len 20480
7088 275/300 Read block 3659 len 40960
7089 276/300 Write block 982 len 20480 data: 2 2 2 3 3
7090 277/300 Read block 1678 len 4096
7091 278/300 Write block 2324 len 16384 data: 5 4 4 4
7092 279/300 Write block 541 len 28672 data: 2 3 3 3 3 2 3
7093 280/300 Read block 4069 len 36864
7094 281/300 Write block 3895 len 8192 data: 2 2
7095 282/300 Read block 873 len 28672
7096 283/300 Read block 725 len 12288
7097 284/300 Read block 276 len 20480
7098 285/300 Read block 2673 len 12288
7099 286/300 Read block 4764 len 12288
7100 287/300 Read block 1977 len 16384
7101 288/300 Write block 3717 len 40960 data: 3 3 3 3 3 3 3 3 2 2
7102 289/300 Read block 2366 len 40960
7103 290/300 Flush
7104 291/300 Read block 3516 len 36864
7105 292/300 Write block 264 len 8192 data: 3 2
7106 293/300 Write block 2828 len 36864 data: 2 2 3 3 3 3 3 3 3
7107 294/300 Write block 3808 len 16384 data: 2 3 3 4
7108 295/300 Flush
7109 296/300 Flush
7110 297/300 Read block 1400 len 28672
7111 298/300 Write block 4059 len 36864 data: 4 3 3 3 2 2 2 2 2
7112 299/300 Write block 4774 len 20480 data: 4 4 4 4 4
7113 300/300 Read block 1327 len 16384
7114 Main task has work to do, go find it
7115 got dsc Start(2)
7116 start 2
7117 Feb 27 21:46:04.461 INFO request completed, latency_us: 219, response_code: 204, uri: /start/cid/2, method: POST, req_id: bfa7a799-abe6-4030-9b6e-b17ba0bcfce5, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
7118 Replay: started 2, returned:Ok(())
7119 ----------------------------------------------------------------
7120 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
7121 States: Active Active Offline
7122 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
7123 1809 Acked 2808 Read 9 Done Done New true
7124 1810 Acked 2809 Write 8 Done Done New true
7125 1811 Acked 2810 Write 9 Done Done New false
7126 1812 Acked 2811 Read 10 Done Done New false
7127 1813 Acked 2812 Read 7 Done Done New false
7128 1814 Acked 2813 Write 9 Done Done New false
7129 1815 Acked 2814 Flush 0 Done Done New false
7130 1816 Acked 2815 Write 9 Done Done New false
7131 1817 Acked 2816 Write 9 Done Done New false
7132 1818 Acked 2817 Read 4 Done Done New false
7133 1819 Acked 2818 Read 5 Done Done New false
7134 1820 Acked 2819 Read 4 Done Done New false
7135 1821 Acked 2820 Write 1 Done Done New false
7136 1822 Acked 2821 Read 4 Done Done New false
7137 1823 Acked 2822 Flush 0 Done Done New false
7138 1824 Acked 2823 Read 3 Done Done New false
7139 1825 Acked 2824 Read 7 Done Done New false
7140 1826 Acked 2825 Write 2 Done Done New false
7141 1827 Acked 2826 Write 10 Done Done New false
7142 1828 Acked 2827 Write 1 Done Done New false
7143 1829 Acked 2828 Write 3 Done Done New false
7144 1830 Acked 2829 Read 6 Done Done New false
7145 1831 Acked 2830 Flush 0 Done Done New false
7146 1832 Acked 2831 Write 7 Done Done New false
7147 1833 Acked 2832 Read 2 Done Done New false
7148 1834 Acked 2833 Read 1 Done Done New false
7149 1835 Acked 2834 Read 7 Done Done New false
7150 1836 Acked 2835 Write 8 Done Done New false
7151 1837 Acked 2836 Read 1 Done Done New false
7152 1838 Acked 2837 Read 4 Done Done New false
7153 1839 Acked 2838 Read 8 Done Done New false
7154 1840 Acked 2839 Write 8 Done Done New false
7155 1841 Acked 2840 Write 2 Done Done New false
7156 1842 Acked 2841 Read 10 Done Done New false
7157 1843 Acked 2842 Read 4 Done Done New false
7158 1844 Acked 2843 Read 10 Done Done New false
7159 1845 Acked 2844 Write 7 Done Done New false
7160 1846 Acked 2845 Read 6 Done Done New false
7161 1847 Acked 2846 Write 6 Done Done New false
7162 1848 Acked 2847 Write 2 Done Done New false
7163 1849 Acked 2848 Write 3 Done Done New false
7164 1850 Acked 2849 Read 10 Done Done New false
7165 1851 Acked 2850 Write 2 Done Done New false
7166 1852 Acked 2851 Read 3 Done Done New false
7167 1853 Acked 2852 Write 3 Done Done New false
7168 1854 Acked 2853 Read 5 Done Done New false
7169 1855 Acked 2854 Read 2 Done Done New false
7170 1856 Acked 2855 Read 9 Done Done New false
7171 1857 Acked 2856 Flush 0 Done Done New false
7172 1858 Acked 2857 Read 4 Done Done New false
7173 1859 Acked 2858 Flush 0 Done Done New false
7174 1860 Acked 2859 Write 5 Done Done New false
7175 1861 Acked 2860 Read 2 Done Done New false
7176 1862 Acked 2861 Read 10 Done Done New false
7177 1863 Acked 2862 Write 2 Done Done New false
7178 1864 Acked 2863 Write 8 Done Done New false
7179 1865 Acked 2864 Write 5 Done Done New false
7180 1866 Acked 2865 Read 4 Done Done New false
7181 1867 Acked 2866 Write 4 Done Done New false
7182 1868 Acked 2867 Read 4 Done Done New false
7183 1869 Acked 2868 Read 8 Done Done New false
7184 1870 Acked 2869 Read 8 Done Done New false
7185 1871 Acked 2870 Read 5 Done Done New false
7186 1872 Acked 2871 Write 4 Done Done New false
7187 1873 Acked 2872 Read 7 Done Done New false
7188 1874 Acked 2873 Read 8 Done Done New false
7189 1875 Acked 2874 Read 9 Done Done New false
7190 1876 Acked 2875 Write 8 Done Done New false
7191 1877 Acked 2876 Read 4 Done Done New false
7192 1878 Acked 2877 Write 9 Done Done New false
7193 1879 Acked 2878 Read 6 Done Done New false
7194 1880 Acked 2879 Read 5 Done Done New false
7195 1881 Acked 2880 Read 5 Done Done New false
7196 1882 Acked 2881 Read 3 Done Done New false
7197 1883 Acked 2882 Read 9 Done Done New false
7198 1884 Acked 2883 Write 8 Done Done New false
7199 1885 Acked 2884 Flush 0 Done Done New false
7200 1886 Acked 2885 Read 4 Done Done New false
7201 1887 Acked 2886 Write 9 Done Done New false
7202 1888 Acked 2887 Write 7 Done Done New false
7203 1889 Acked 2888 Read 9 Done Done New false
7204 1890 Acked 2889 Read 3 Done Done New false
7205 1891 Acked 2890 Read 3 Done Done New false
7206 1892 Acked 2891 Read 7 Done Done New false
7207 1893 Acked 2892 Read 3 Done Done New false
7208 1894 Acked 2893 Read 8 Done Done New false
7209 1895 Acked 2894 Write 3 Done Done New false
7210 1896 Acked 2895 Write 8 Done Done New false
7211 1897 Acked 2896 Write 6 Done Done New false
7212 1898 Acked 2897 Read 10 Done Done New false
7213 1899 Acked 2898 Read 7 Done Done New false
7214 1900 Acked 2899 Write 1 Done Done New false
7215 1901 Acked 2900 Read 4 Done Done New false
7216 1902 Acked 2901 Read 3 Done Done New false
7217 1903 Acked 2902 Write 8 Done Done New false
7218 1904 Acked 2903 Read 8 Done Done New false
7219 1905 Acked 2904 Read 3 Done Done New false
7220 1906 Acked 2905 Read 5 Done Done New false
7221 1907 Acked 2906 Flush 0 Done Done New false
7222 1908 Acked 2907 Read 9 Done Done New false
7223 1909 Acked 2908 Write 2 Done Done New false
7224 1910 Acked 2909 Write 9 Done Done New false
7225 1911 Acked 2910 Read 1 Done Done New false
7226 1912 Acked 2911 Write 3 Done Done New false
7227 1913 Acked 2912 Write 1 Done Done New false
7228 1914 Acked 2913 Write 9 Done Done New false
7229 1915 Acked 2914 Write 1 Done Done New false
7230 1916 Acked 2915 Read 6 Done Done New false
7231 1917 Acked 2916 Write 10 Done Done New false
7232 1918 Acked 2917 Read 8 Done Done New false
7233 1919 Acked 2918 Write 6 Done Done New false
7234 1920 Acked 2919 Write 4 Done Done New false
7235 1921 Acked 2920 Write 8 Done Done New false
7236 1922 Acked 2921 Read 7 Done Done New false
7237 1923 Acked 2922 Write 7 Done Done New false
7238 1924 Acked 2923 Write 5 Done Done New false
7239 1925 Acked 2924 Write 5 Done Done New false
7240 1926 Acked 2925 Read 10 Done Done New false
7241 1927 Acked 2926 Read 9 Done Done New false
7242 1928 Acked 2927 Flush 0 Done Done New false
7243 1929 Acked 2928 Write 6 Done Done New false
7244 1930 Acked 2929 Read 6 Done Done New false
7245 1931 Acked 2930 Read 9 Done Done New false
7246 1932 Acked 2931 Flush 0 Done Done New false
7247 1933 Acked 2932 Flush 0 Done Done New false
7248 1934 Acked 2933 Read 9 Done Done New false
7249 1935 Acked 2934 Flush 0 Done Done New false
7250 1936 Acked 2935 Write 10 Done Done New false
7251 1937 Acked 2936 Flush 0 Done Done New false
7252 1938 Acked 2937 Flush 0 Done Done New false
7253 1939 Acked 2938 Read 10 Done Done New false
7254 1940 Acked 2939 Flush 0 Done Done New false
7255 1941 Acked 2940 Write 8 Done Done New false
7256 1942 Acked 2941 Write 5 Done Done New false
7257 1943 Acked 2942 Write 7 Done Done New false
7258 1944 Acked 2943 Write 3 Done Done New false
7259 1945 Acked 2944 Read 5 Done Done New false
7260 1946 Acked 2945 Read 1 Done Done New false
7261 1947 Acked 2946 Write 10 Done Done New false
7262 1948 Acked 2947 Read 9 Done Done New false
7263 1949 Acked 2948 Read 10 Done Done New false
7264 1950 Acked 2949 Read 10 Done Done New false
7265 1951 Acked 2950 Write 8 Done Done New false
7266 1952 Acked 2951 Write 10 Done Done New false
7267 1953 Acked 2952 Read 4 Done Done New false
7268 1954 Acked 2953 Read 8 Done Done New false
7269 1955 Acked 2954 Read 2 Done Done New false
7270 1956 Acked 2955 Write 10 Done Done New false
7271 1957 Acked 2956 Read 2 Done Done New false
7272 1958 Acked 2957 Write 1 Done Done New false
7273 1959 Acked 2958 Flush 0 Done Done New false
7274 1960 Acked 2959 Flush 0 Done Done New false
7275 1961 Acked 2960 Read 2 Done Done New false
7276 1962 Acked 2961 Read 6 Done Done New false
7277 1963 Acked 2962 Write 4 Done Done New false
7278 1964 Acked 2963 Write 8 Done Done New false
7279 1965 Acked 2964 Read 9 Done Done New false
7280 1966 Acked 2965 Write 3 Done Done New false
7281 1967 Acked 2966 Write 3 Done Done New false
7282 1968 Acked 2967 Read 10 Done Done New false
7283 1969 Acked 2968 Write 1 Done Done New false
7284 1970 Acked 2969 Read 2 Done Done New false
7285 1971 Acked 2970 Read 6 Done Done New false
7286 1972 Acked 2971 Read 6 Done Done New false
7287 1973 Acked 2972 Write 6 Done Done New false
7288 1974 Acked 2973 Read 3 Done Done New false
7289 1975 Acked 2974 Read 1 Done Done New false
7290 1976 Acked 2975 Read 3 Done Done New false
7291 1977 Acked 2976 Read 4 Done Done New false
7292 1978 Acked 2977 Write 5 Done Done New false
7293 1979 Acked 2978 Write 8 Done Done New false
7294 1980 Acked 2979 Write 1 Done Done New false
7295 1981 Acked 2980 Read 9 Done Done New false
7296 1982 Acked 2981 Read 9 Done Done New false
7297 1983 Acked 2982 Read 1 Done Done New false
7298 1984 Acked 2983 Read 5 Done Done New false
7299 1985 Acked 2984 Read 2 Done Done New false
7300 1986 Acked 2985 Read 3 Done Done New false
7301 1987 Acked 2986 Write 6 Done Done New false
7302 1988 Acked 2987 Read 3 Done Done New false
7303 1989 Acked 2988 Read 7 Done Done New false
7304 1990 Acked 2989 Write 10 Done Done New false
7305 1991 Acked 2990 Write 6 Done Done New false
7306 1992 Acked 2991 Read 7 Done Done New false
7307 1993 Acked 2992 Write 7 Done Done New false
7308 1994 Acked 2993 Read 3 Done Done New false
7309 1995 Acked 2994 Write 2 Done Done New false
7310 1996 Acked 2995 Read 10 Done Done New false
7311 1997 Acked 2996 Flush 0 Done Done New false
7312 1998 Acked 2997 Write 7 Done Done New false
7313 1999 Acked 2998 Write 9 Done Done New false
7314 2000 Acked 2999 Write 3 Done Done New false
7315 2001 Acked 3000 Read 5 Done Done New false
7316 2002 Acked 3001 Flush 0 Done Done New false
7317 2003 Acked 3002 Write 5 Done Done New false
7318 2004 Acked 3003 Flush 0 Done Done New false
7319 2005 Acked 3004 Read 4 Done Done New false
7320 2006 Acked 3005 Write 7 Done Done New false
7321 2007 Acked 3006 Read 5 Done Done New false
7322 2008 Acked 3007 Write 3 Done Done New false
7323 2009 Acked 3008 Read 7 Done Done New false
7324 2010 Acked 3009 Write 5 Done Done New false
7325 2011 Acked 3010 Write 4 Done Done New false
7326 2012 Acked 3011 Flush 0 Done Done New false
7327 2013 Acked 3012 Write 2 Done Done New false
7328 2014 Acked 3013 Flush 0 Done Done New false
7329 2015 Acked 3014 Read 7 Done Done New false
7330 2016 Acked 3015 Write 10 Done Done New false
7331 2017 Acked 3016 Read 1 Done Done New false
7332 2018 Acked 3017 Write 4 Done Done New false
7333 2019 Acked 3018 Write 3 Done Done New false
7334 2020 Acked 3019 Write 9 Done Done New false
7335 2021 Acked 3020 Write 7 Done Done New false
7336 2022 Acked 3021 Flush 0 Done Done New false
7337 2023 Acked 3022 Read 9 Done Done New false
7338 2024 Acked 3023 Read 4 Done Done New false
7339 2025 Acked 3024 Read 6 Done Done New false
7340 2026 Acked 3025 Read 4 Done Done New false
7341 2027 Acked 3026 Flush 0 Done Done New false
7342 2028 Acked 3027 Write 7 Done Done New false
7343 2029 Acked 3028 Write 9 Done Done New false
7344 2030 Acked 3029 Write 9 Done Done New false
7345 2031 Acked 3030 Read 6 Done Done New false
7346 2032 Acked 3031 Write 1 Done Done New false
7347 2033 Acked 3032 Read 2 Done Done New false
7348 2034 Acked 3033 Read 2 Done Done New false
7349 2035 Acked 3034 Write 7 Done Done New false
7350 2036 Acked 3035 Read 3 Done Done New false
7351 2037 Acked 3036 Read 4 Done Done New false
7352 2038 Acked 3037 Write 4 Done Done New false
7353 2039 Acked 3038 Write 7 Done Done New false
7354 2040 Acked 3039 Read 10 Done Done New false
7355 2041 Acked 3040 Write 2 Done Done New false
7356 2042 Acked 3041 Flush 0 Done Done New false
7357 2043 Acked 3042 Write 10 Done Done New false
7358 2044 Acked 3043 Flush 0 Done Done New false
7359 2045 Acked 3044 Read 7 Done Done New false
7360 2046 Acked 3045 Write 9 Done Done New false
7361 2047 Acked 3046 Read 5 Done Done New false
7362 2048 Acked 3047 Read 10 Done Done New false
7363 2049 Acked 3048 Read 7 Done Done New false
7364 2050 Acked 3049 Read 4 Done Done New false
7365 2051 Acked 3050 Flush 0 Done Done New false
7366 2052 Acked 3051 Read 2 Done Done New false
7367 2053 Acked 3052 Flush 0 Done Done New false
7368 2054 Acked 3053 Write 4 Done Done New false
7369 2055 Acked 3054 Write 1 Done Done New false
7370 2056 Acked 3055 Write 3 Done Done New false
7371 2057 Acked 3056 Flush 0 Done Done New false
7372 2058 Acked 3057 Read 6 Done Done New false
7373 2059 Acked 3058 Write 7 Done Done New false
7374 2060 Acked 3059 Flush 0 Done Done New false
7375 2061 Acked 3060 Read 5 Done Done New false
7376 2062 Acked 3061 Read 8 Done Done New false
7377 2063 Acked 3062 Write 4 Done Done New false
7378 2064 Acked 3063 Flush 0 Done Done New false
7379 2065 Acked 3064 Read 2 Done Done New false
7380 2066 Acked 3065 Write 9 Done Done New false
7381 2067 Acked 3066 Read 4 Done Done New false
7382 2068 Acked 3067 Write 2 Done Done New false
7383 2069 Acked 3068 Read 9 Done Done New false
7384 2070 Acked 3069 Flush 0 Done Done New false
7385 2071 Acked 3070 Read 1 Done Done New false
7386 2072 Acked 3071 Flush 0 Done Done New false
7387 2073 Acked 3072 Write 7 Done Done New false
7388 2074 Acked 3073 Write 2 Done Done New false
7389 2075 Acked 3074 Write 9 Done Done New false
7390 2076 Acked 3075 Write 3 Done Done New false
7391 2077 Acked 3076 Write 6 Done Done New false
7392 2078 Acked 3077 Flush 0 Done Done New false
7393 2079 Acked 3078 Read 8 Done Done New false
7394 2080 Acked 3079 Write 1 Done Done New false
7395 2081 Acked 3080 Read 9 Done Done New false
7396 2082 Acked 3081 Read 5 Done Done New false
7397 2083 Acked 3082 Read 10 Done Done New false
7398 2084 Acked 3083 Write 5 Done Done New false
7399 2085 Acked 3084 Read 1 Done Done New false
7400 2086 Acked 3085 Write 4 Done Done New false
7401 2087 Acked 3086 Write 7 Done Done New false
7402 2088 Acked 3087 Read 9 Done Done New false
7403 2089 Acked 3088 Write 2 Done Done New false
7404 2090 Acked 3089 Read 7 Done Done New false
7405 2091 Acked 3090 Read 3 Done Done New false
7406 2092 Acked 3091 Read 5 Done Done New false
7407 2093 Acked 3092 Read 3 Done Done New false
7408 2094 Acked 3093 Read 3 Done Done New false
7409 2095 Acked 3094 Read 4 Done Done New false
7410 2096 Acked 3095 Write 10 Done Done New false
7411 2097 Acked 3096 Read 10 Done Done New false
7412 2098 Acked 3097 Flush 0 Done Done New false
7413 2099 Acked 3098 Read 9 Done Done New false
7414 2100 Acked 3099 Write 2 Done Done New false
7415 2101 Acked 3100 Write 9 Done Done New false
7416 2102 Acked 3101 Write 4 Done Done New false
7417 2103 Acked 3102 Flush 0 Done Done New false
7418 2104 Acked 3103 Flush 0 Done Done New false
7419 2105 Acked 3104 Read 7 Done Done New false
7420 2106 Acked 3105 Write 9 Done Done New false
7421 2107 Acked 3106 Write 5 Done Done New false
7422 2108 Acked 3107 Read 4 Done Done New false
7423 STATES DS:0 DS:1 DS:2 TOTAL
7424 New 0 0 300 300
7425 Sent 0 0 0 0
7426 Done 300 300 0 600
7427 Skipped 0 0 0 0
7428 Error 0 0 0 0
7429 Last Flush: 3103 3103 2807
7430 Downstairs last five completed: 2807 2806 2805 2804 2803
7431 Upstairs last five completed: 2108 2107 2106 2105 2104
7432 CLIENT: Up:0 ds:300 act:2
74332024-02-27T21:46:05.116ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74342024-02-27T21:46:05.116ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74352024-02-27T21:46:05.116ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74362024-02-27T21:46:05.116ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
74372024-02-27T21:46:05.116ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74382024-02-27T21:46:05.117ZINFOcrucible: [2] client re-new 301 jobs since flush 2807 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
74392024-02-27T21:46:05.117ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
7440 [8830] got start action
7441 [8830] I am going to restart
7442 Starting downstairs at port 8830
7443 Make output file at "/tmp/dsc/downstairs-8830.txt"
7444 [8830][2] reports Starting
74452024-02-27T21:46:06.118ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74462024-02-27T21:46:06.119ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74472024-02-27T21:46:06.119ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74482024-02-27T21:46:06.119ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
74492024-02-27T21:46:06.119ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74502024-02-27T21:46:06.119ZINFOcrucible: [2] client re-new 301 jobs since flush 2807 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
74512024-02-27T21:46:06.119ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
7452 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1174)
7453 [8830][2] reports Running
74542024-02-27T21:46:07.120ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74552024-02-27T21:46:07.120ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74562024-02-27T21:46:07.121ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74572024-02-27T21:46:07.121ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74582024-02-27T21:46:07.121ZINFOcrucible: send last flush ID to this DS: 2807 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74592024-02-27T21:46:07.121ZINFOcrucible: Replied this last flush ID: 2807 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74602024-02-27T21:46:07.121ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74612024-02-27T21:46:07.121ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74622024-02-27T21:46:07.121ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74632024-02-27T21:46:07.121ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
7464 ----------------------------------------------------------------
7465 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
7466 Downstairs last five completed: 3108 3107 3106 3105 3104
7467 Upstairs last five completed: 2109 2108 2107 2106 2105
7468 CLIENT: Up:0 ds:0 act:3
7469 Replay: All jobs finished, all DS active.
7470 Main task has work to do, go find it
7471 got dsc Stop(2)
7472 stop 2
7473 Feb 27 21:46:08.464 INFO request completed, latency_us: 362, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 08f2768f-4578-46b2-a5ac-069443baf601, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
7474 [8830] Got stop action so:false kr:false
7475 Feb 27 21:46:08.465 INFO{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-27T21:46:08.465691546Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"io task","client":"2","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
7476 request completed, latency_us: 122, response_code: 200, uri: /state/cid/2, method: GET, req_id{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-27T21:46:08.465797892Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"io task","client":"2","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
7477 : 7f6f32a1-660a-4f95-a8bc-c339039c1942, remote_addr: 127.0.0{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-27T21:46:08.465862194Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
7478 .1:44347, local_addr: 127.0.0.1:9998
74792024-02-27T21:46:08.465ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74802024-02-27T21:46:08.465ZINFOcrucible: [2] client re-new 0 jobs since flush 3108 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
74812024-02-27T21:46:08.465ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
7482 [8830] Exited with: ExitStatus(unix_wait_status(9))
7483 [8830][2] reports Exit
74842024-02-27T21:46:09.467ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74852024-02-27T21:46:09.467ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74862024-02-27T21:46:09.467ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74872024-02-27T21:46:09.467ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
74882024-02-27T21:46:09.467ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74892024-02-27T21:46:09.467ZINFOcrucible: [2] client re-new 0 jobs since flush 3108 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
74902024-02-27T21:46:09.467ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74912024-02-27T21:46:10.468ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74922024-02-27T21:46:10.468ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74932024-02-27T21:46:10.468ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74942024-02-27T21:46:10.468ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
74952024-02-27T21:46:10.468ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74962024-02-27T21:46:10.468ZINFOcrucible: [2] client re-new 0 jobs since flush 3108 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
74972024-02-27T21:46:10.468ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74982024-02-27T21:46:11.469ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
74992024-02-27T21:46:11.469ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
75002024-02-27T21:46:11.469ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
75012024-02-27T21:46:11.469ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
75022024-02-27T21:46:11.469ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
75032024-02-27T21:46:11.469ZINFOcrucible: [2] client re-new 0 jobs since flush 3108 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
75042024-02-27T21:46:11.469ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
7505 Feb 27 21:46:12.468 INFO request completed, latency_us: 221, response_code: 200, uri: /state/cid/2, method: GET, req_id: f9e9918e-3086-47d2-a01f-dea6ee5e1208, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
7506 001/300 Write block 3551 len 40960 data: 2 2 2 2 4 4 3 4 4 4
7507 002/300 Write block 3370 len 24576 data: 3 3 3 4 5 5
7508 003/300 Read block 2321 len 28672
75092024-02-27T21:46:12.470ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
75102024-02-27T21:46:12.470ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
75112024-02-27T21:46:12.470ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
75122024-02-27T21:46:12.470ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
75132024-02-27T21:46:12.470ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
75142024-02-27T21:46:12.470ZINFOcrucible: [2] client re-new 3 jobs since flush 3108 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
75152024-02-27T21:46:12.470ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
7516 004/300 Write block 3844 len 40960 data: 4 4 3 3 2 2 3 3 4 4
7517 005/300 Read block 4800 len 16384
7518 006/300 Write block 326 len 24576 data: 2 2 2 2 2 2
7519 007/300 Write block 3578 len 8192 data: 3 2
7520 008/300 Write block 2642 len 20480 data: 3 3 3 3 4
7521 009/300 Read block 596 len 12288
7522 010/300 Flush
7523 011/300 Read block 1053 len 4096
7524 012/300 Write block 4018 len 24576 data: 3 4 3 4 3 3
7525 013/300 Write block 2381 len 24576 data: 2 2 2 2 2 2
7526 014/300 Read block 1100 len 32768
7527 015/300 Write block 1631 len 36864 data: 2 2 2 2 2 2 2 3 3
7528 016/300 Write block 1889 len 8192 data: 3 3
7529 017/300 Write block 4843 len 40960 data: 2 2 2 2 2 2 2 2 2 2
7530 018/300 Read block 3227 len 28672
7531 019/300 Write block 4673 len 16384 data: 2 2 2 2
7532 020/300 Read block 3025 len 24576
7533 021/300 Write block 333 len 20480 data: 2 2 2 2 2
7534 022/300 Read block 4185 len 16384
7535 023/300 Read block 4409 len 16384
7536 024/300 Write block 4020 len 8192 data: 4 5
7537 025/300 Read block 4567 len 32768
7538 026/300 Read block 876 len 16384
7539 027/300 Read block 2447 len 12288
7540 028/300 Write block 1023 len 24576 data: 2 2 2 2 2 3
7541 029/300 Flush
7542 030/300 Write block 814 len 28672 data: 2 2 2 2 2 3 2
7543 031/300 Write block 3599 len 24576 data: 3 3 3 3 3 2
7544 032/300 Write block 4313 len 36864 data: 3 3 3 3 3 3 3 4 4
7545 033/300 Write block 1433 len 8192 data: 3 2
7546 034/300 Write block 4394 len 4096 data: 3
7547 035/300 Write block 435 len 36864 data: 4 4 4 3 3 3 3 2 2
7548 036/300 Flush
7549 037/300 Write block 59 len 4096 data: 2
7550 038/300 Write block 3028 len 40960 data: 5 5 5 3 4 3 3 3 3 3
7551 039/300 Write block 4539 len 40960 data: 2 2 2 2 2 2 2 2 3 3
7552 040/300 Flush
7553 041/300 Write block 138 len 24576 data: 2 2 2 2 2 2
7554 042/300 Read block 2306 len 16384
7555 043/300 Flush
7556 044/300 Write block 2590 len 4096 data: 3
7557 045/300 Write block 4176 len 16384 data: 2 2 2 2
7558 046/300 Write block 3492 len 24576 data: 4 4 4 4 4 4
7559 047/300 Write block 638 len 16384 data: 2 2 2 2
7560 048/300 Read block 2533 len 20480
7561 049/300 Read block 506 len 20480
7562 050/300 Write block 1854 len 32768 data: 2 2 2 2 2 2 2 2
7563 051/300 Write block 2311 len 28672 data: 3 3 3 4 4 4 4
7564 052/300 Read block 1069 len 36864
7565 053/300 Read block 4464 len 40960
7566 054/300 Write block 3520 len 28672 data: 3 3 3 4 4 5 5
7567 055/300 Read block 209 len 12288
7568 056/300 Read block 1650 len 16384
7569 057/300 Write block 69 len 12288 data: 4 3 3
7570 058/300 Write block 2809 len 8192 data: 2 2
7571 059/300 Write block 2007 len 12288 data: 2 2 2
7572 060/300 Write block 1501 len 40960 data: 2 2 2 2 3 3 3 3 4 4
7573 061/300 Write block 2505 len 16384 data: 5 5 6 5
7574 062/300 Read block 665 len 12288
7575 063/300 Read block 1275 len 40960
7576 064/300 Read block 4428 len 32768
7577 065/300 Write block 1990 len 40960 data: 3 3 3 3 3 3 3 3 2 2
7578 066/300 Flush
7579 067/300 Write block 4493 len 32768 data: 4 3 3 3 3 4 4 4
7580 068/300 Read block 2623 len 12288
7581 069/300 Write block 1328 len 4096 data: 2
7582 070/300 Read block 2227 len 16384
7583 071/300 Read block 4362 len 40960
7584 072/300 Write block 1850 len 24576 data: 4 4 3 3 3 3
7585 073/300 Read block 4715 len 32768
7586 074/300 Read block 4914 len 32768
7587 075/300 Flush
7588 076/300 Write block 3303 len 32768 data: 4 3 3 3 3 3 4 4
7589 077/300 Read block 4044 len 28672
7590 078/300 Read block 3035 len 4096
7591 079/300 Write block 4269 len 16384 data: 4 4 4 3
7592 080/300 Read block 1893 len 28672
7593 081/300 Read block 3836 len 12288
7594 082/300 Read block 4639 len 16384
7595 083/300 Flush
7596 084/300 Write block 180 len 16384 data: 3 3 4 4
7597 085/300 Write block 2002 len 40960 data: 3 2 2 2 2 3 3 3 2 2
7598 086/300 Read block 3962 len 32768
7599 087/300 Write block 1052 len 12288 data: 4 3 3
7600 088/300 Write block 3888 len 16384 data: 2 2 2 2
7601 089/300 Write block 3474 len 12288 data: 3 2 3
7602 090/300 Write block 868 len 36864 data: 2 3 3 3 3 3 4 4 5
7603 091/300 Read block 1903 len 24576
7604 092/300 Write block 4199 len 24576 data: 4 4 2 2 2 2
7605 093/300 Write block 2502 len 32768 data: 4 5 6 6 6 7 6 3
7606 094/300 Write block 4770 len 8192 data: 4 4
7607 095/300 Write block 3784 len 40960 data: 3 3 3 4 4 4 4 3 3 3
7608 096/300 Write block 1280 len 4096 data: 2
7609 097/300 Read block 3054 len 40960
7610 098/300 Write block 4252 len 8192 data: 3 3
7611 099/300 Read block 927 len 20480
7612 100/300 Read block 2833 len 36864
7613 101/300 Read block 4091 len 28672
7614 102/300 Read block 2042 len 32768
7615 103/300 Flush
7616 104/300 Read block 4100 len 8192
7617 105/300 Read block 2717 len 28672
7618 106/300 Write block 1306 len 4096 data: 3
7619 107/300 Write block 2717 len 12288 data: 4 4 3
7620 108/300 Write block 1978 len 12288 data: 4 4 4
7621 109/300 Read block 528 len 40960
7622 110/300 Write block 302 len 8192 data: 2 2
7623 111/300 Write block 1268 len 8192 data: 4 4
7624 112/300 Write block 71 len 4096 data: 4
7625 113/300 Write block 955 len 20480 data: 3 3 4 5 5
7626 114/300 Write block 2759 len 20480 data: 2 2 3 3 3
7627 115/300 Read block 1485 len 32768
7628 116/300 Read block 4161 len 32768
7629 117/300 Read block 3474 len 28672
7630 118/300 Read block 4263 len 24576
7631 119/300 Read block 4575 len 32768
7632 120/300 Read block 3104 len 40960
7633 121/300 Read block 449 len 20480
7634 122/300 Flush
7635 123/300 Read block 3872 len 8192
7636 124/300 Read block 3209 len 24576
7637 125/300 Read block 2487 len 32768
7638 126/300 Write block 1937 len 20480 data: 2 2 2 2 2
7639 127/300 Write block 1082 len 28672 data: 4 5 5 5 3 3 3
7640 128/300 Write block 3020 len 12288 data: 5 5 5
7641 129/300 Read block 2132 len 8192
7642 130/300 Write block 1266 len 28672 data: 3 4 5 5 3 2 2
7643 131/300 Read block 3252 len 36864
7644 132/300 Write block 1953 len 20480 data: 2 2 2 2 2
7645 133/300 Read block 1304 len 32768
7646 134/300 Read block 183 len 4096
7647 135/300 Read block 2098 len 8192
7648 136/300 Read block 735 len 36864
7649 137/300 Write block 2196 len 28672 data: 3 3 3 3 3 2 3
7650 138/300 Read block 2975 len 16384
7651 139/300 Read block 4590 len 24576
7652 140/300 Write block 4836 len 32768 data: 3 3 3 3 2 2 2 3
7653 141/300 Write block 3724 len 12288 data: 4 3 3
7654 142/300 Read block 1115 len 32768
7655 143/300 Read block 988 len 12288
7656 144/300 Read block 4044 len 16384
7657 145/300 Read block 3955 len 8192
7658 146/300 Write block 3641 len 4096 data: 4
7659 147/300 Read block 312 len 16384
7660 148/300 Read block 3372 len 20480
7661 149/300 Write block 759 len 32768 data: 2 2 2 2 3 3 4 4
7662 150/300 Read block 2726 len 40960
7663 151/300 Read block 1174 len 24576
7664 152/300 Read block 3873 len 20480
7665 153/300 Write block 4791 len 16384 data: 3 3 2 2
7666 154/300 Read block 2530 len 28672
7667 155/300 Write block 2215 len 28672 data: 3 3 3 2 2 2 3
7668 156/300 Write block 3472 len 12288 data: 3 3 4
7669 157/300 Write block 552 len 32768 data: 4 3 3 4 4 4 4 3
7670 158/300 Write block 2780 len 12288 data: 4 4 4
7671 159/300 Read block 1900 len 4096
7672 160/300 Write block 2659 len 36864 data: 2 2 2 2 2 2 4 3 3
7673 161/300 Write block 1131 len 36864 data: 3 5 4 3 3 2 3 3 3
7674 162/300 Write block 216 len 12288 data: 2 2 2
7675 163/300 Read block 2916 len 16384
7676 164/300 Write block 1460 len 28672 data: 3 3 3 3 3 3 3
7677 165/300 Write block 4616 len 8192 data: 3 2
7678 166/300 Read block 3156 len 40960
7679 167/300 Write block 1417 len 20480 data: 2 2 2 2 2
7680 168/300 Write block 1662 len 40960 data: 6 6 7 5 4 5 4 4 4 3
7681 169/300 Write block 2014 len 28672 data: 2 2 2 2 2 2 2
7682 170/300 Read block 658 len 12288
7683 171/300 Write block 525 len 36864 data: 3 3 2 3 3 3 2 2 2
7684 172/300 Read block 2233 len 20480
7685 173/300 Read block 4798 len 40960
7686 174/300 Read block 3377 len 4096
7687 175/300 Write block 3677 len 28672 data: 3 5 6 4 3 4 4
7688 176/300 Read block 356 len 20480
7689 177/300 Read block 1694 len 24576
7690 178/300 Read block 4008 len 12288
7691 179/300 Write block 3380 len 16384 data: 2 2 2 2
7692 180/300 Write block 4964 len 28672 data: 2 2 2 2 2 3 4
7693 181/300 Write block 2229 len 4096 data: 2
7694 182/300 Read block 2312 len 8192
7695 183/300 Write block 3953 len 16384 data: 3 3 3 3
7696 184/300 Read block 688 len 32768
7697 185/300 Write block 3219 len 24576 data: 2 2 2 2 2 2
7698 186/300 Read block 2551 len 16384
7699 187/300 Write block 1474 len 36864 data: 6 4 3 4 4 4 4 4 5
7700 188/300 Write block 2158 len 12288 data: 4 4 4
7701 189/300 Write block 2696 len 16384 data: 3 3 2 2
7702 190/300 Write block 2240 len 16384 data: 2 3 3 3
7703 191/300 Write block 162 len 28672 data: 2 2 2 2 2 2 2
7704 192/300 Flush
7705 193/300 Write block 1619 len 40960 data: 4 4 4 4 2 2 2 2 2 2
7706 194/300 Read block 4476 len 20480
7707 195/300 Read block 4082 len 24576
7708 196/300 Write block 3479 len 4096 data: 2
7709 197/300 Read block 3350 len 24576
7710 198/300 Flush
7711 199/300 Write block 4440 len 24576 data: 3 3 2 2 3 3
7712 200/300 Read block 3900 len 32768
7713 201/300 Write block 887 len 4096 data: 2
7714 202/300 Write block 1188 len 16384 data: 3 3 2 2
7715 203/300 Read block 2840 len 12288
7716 204/300 Write block 3906 len 12288 data: 3 2 2
7717 205/300 Read block 2172 len 32768
7718 206/300 Read block 4141 len 16384
7719 207/300 Write block 2597 len 28672 data: 2 3 3 3 3 3 2
7720 208/300 Write block 2854 len 12288 data: 2 2 2
7721 209/300 Write block 4435 len 12288 data: 4 4 4
7722 210/300 Write block 2708 len 8192 data: 2 3
7723 211/300 Flush
7724 212/300 Write block 4475 len 28672 data: 2 2 2 2 2 2 2
7725 213/300 Flush
7726 214/300 Read block 968 len 4096
7727 215/300 Write block 3470 len 16384 data: 3 3 4 4
7728 216/300 Write block 3338 len 4096 data: 3
7729 217/300 Read block 4717 len 8192
7730 218/300 Read block 2963 len 4096
7731 219/300 Read block 160 len 4096
7732 220/300 Flush
7733 221/300 Read block 1601 len 40960
7734 222/300 Write block 4765 len 8192 data: 3 3
7735 223/300 Read block 99 len 20480
7736 224/300 Read block 1339 len 32768
7737 225/300 Flush
7738 226/300 Read block 2822 len 8192
7739 227/300 Read block 749 len 32768
7740 228/300 Read block 3810 len 20480
7741 229/300 Write block 1522 len 20480 data: 2 2 2 2 2
7742 230/300 Write block 23 len 16384 data: 5 5 5 2
7743 231/300 Write block 4427 len 8192 data: 2 3
7744 232/300 Read block 1776 len 4096
7745 233/300 Read block 1624 len 24576
7746 234/300 Read block 1183 len 28672
7747 235/300 Write block 2644 len 32768 data: 4 4 5 6 5 5 4 3
7748 236/300 Flush
7749 237/300 Read block 2879 len 20480
7750 238/300 Write block 4790 len 28672 data: 3 4 4 3 3 2 2
7751 239/300 Write block 3366 len 8192 data: 2 3
7752 240/300 Flush
7753 241/300 Write block 418 len 4096 data: 4
7754 242/300 Write block 4577 len 40960 data: 5 5 4 4 3 3 3 3 2 2
7755 243/300 Read block 4708 len 4096
7756 244/300 Read block 4667 len 40960
7757 245/300 Flush
7758 246/300 Write block 4297 len 32768 data: 3 3 3 3 3 3 4 3
7759 247/300 Write block 176 len 20480 data: 2 2 3 3 4
7760 248/300 Read block 3882 len 12288
7761 249/300 Write block 1716 len 12288 data: 4 4 4
7762 250/300 Read block 4030 len 24576
7763 251/300 Read block 4783 len 36864
7764 252/300 Read block 3673 len 12288
7765 253/300 Read block 1416 len 40960
7766 254/300 Flush
7767 255/300 Write block 2655 len 16384 data: 2 2 2 2
7768 256/300 Read block 2728 len 36864
7769 257/300 Flush
7770 258/300 Write block 2605 len 8192 data: 2 2
7771 259/300 Read block 1382 len 32768
7772 260/300 Read block 1502 len 12288
7773 261/300 Read block 632 len 24576
7774 262/300 Flush
7775 263/300 Read block 2515 len 8192
7776 264/300 Flush
7777 265/300 Read block 4366 len 20480
7778 266/300 Read block 3523 len 12288
7779 267/300 Write block 1544 len 36864 data: 2 2 3 3 3 3 3 3 3
7780 268/300 Flush
7781 269/300 Read block 3912 len 12288
7782 270/300 Read block 492 len 28672
7783 271/300 Write block 1014 len 40960 data: 3 3 3 3 3 3 2 2 2 3
7784 272/300 Write block 838 len 4096 data: 5
7785 273/300 Read block 4936 len 8192
7786 274/300 Write block 4786 len 12288 data: 3 3 3
7787 275/300 Read block 4066 len 24576
7788 276/300 Read block 533 len 28672
7789 277/300 Read block 237 len 28672
7790 278/300 Read block 2232 len 24576
7791 279/300 Write block 2927 len 32768 data: 2 2 2 2 2 2 2 2
7792 280/300 Write block 3945 len 40960 data: 4 4 4 3 2 2 2 3 4 4
7793 281/300 Read block 655 len 4096
7794 282/300 Read block 835 len 20480
7795 283/300 Write block 1940 len 8192 data: 3 3
7796 284/300 Write block 2641 len 24576 data: 3 4 4 5 5 6
7797 285/300 Write block 4316 len 16384 data: 4 4 4 4
7798 286/300 Write block 3110 len 32768 data: 3 3 2 2 3 4 4 4
7799 287/300 Write block 2204 len 40960 data: 2 2 3 3 2 2 2 3 3 3
7800 288/300 Flush
7801 289/300 Read block 804 len 36864
7802 290/300 Write block 3232 len 16384 data: 2 2 2 3
7803 291/300 Read block 4372 len 8192
7804 292/300 Read block 3695 len 20480
7805 293/300 Flush
7806 294/300 Read block 2848 len 32768
7807 295/300 Write block 2204 len 20480 data: 3 3 4 4 3
7808 296/300 Read block 2435 len 40960
7809 297/300 Read block 3332 len 20480
7810 298/300 Read block 140 len 4096
7811 299/300 Read block 2302 len 20480
7812 300/300 Write block 811 len 8192 data: 2 2
7813 Main task has work to do, go find it
7814 got dsc Start(2)
7815 start 2
7816 Feb 27 21:46:12.806 INFO request completed, latency_us: 215, response_code: 204, uri: /start/cid/2, method: POST, req_id: d6c3780b-0380-4149-b605-7a56f5285e74, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
7817 Replay: started 2, returned:Ok(())
7818 ----------------------------------------------------------------
7819 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
7820 States: Active Active Offline
7821 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
7822 2110 Acked 3109 Write 10 Done Done New true
7823 2111 Acked 3110 Write 6 Done Done New true
7824 2112 Acked 3111 Read 7 Done Done New true
7825 2113 Acked 3112 Write 10 Done Done New false
7826 2114 Acked 3113 Read 4 Done Done New false
7827 2115 Acked 3114 Write 6 Done Done New false
7828 2116 Acked 3115 Write 2 Done Done New false
7829 2117 Acked 3116 Write 5 Done Done New false
7830 2118 Acked 3117 Read 3 Done Done New false
7831 2119 Acked 3118 Flush 0 Done Done New false
7832 2120 Acked 3119 Read 1 Done Done New false
7833 2121 Acked 3120 Write 6 Done Done New false
7834 2122 Acked 3121 Write 6 Done Done New false
7835 2123 Acked 3122 Read 8 Done Done New false
7836 2124 Acked 3123 Write 9 Done Done New false
7837 2125 Acked 3124 Write 2 Done Done New false
7838 2126 Acked 3125 Write 10 Done Done New false
7839 2127 Acked 3126 Read 7 Done Done New false
7840 2128 Acked 3127 Write 4 Done Done New false
7841 2129 Acked 3128 Read 6 Done Done New false
7842 2130 Acked 3129 Write 5 Done Done New false
7843 2131 Acked 3130 Read 4 Done Done New false
7844 2132 Acked 3131 Read 4 Done Done New false
7845 2133 Acked 3132 Write 2 Done Done New false
7846 2134 Acked 3133 Read 8 Done Done New false
7847 2135 Acked 3134 Read 4 Done Done New false
7848 2136 Acked 3135 Read 3 Done Done New false
7849 2137 Acked 3136 Write 6 Done Done New false
7850 2138 Acked 3137 Flush 0 Done Done New false
7851 2139 Acked 3138 Write 7 Done Done New false
7852 2140 Acked 3139 Write 6 Done Done New false
7853 2141 Acked 3140 Write 9 Done Done New false
7854 2142 Acked 3141 Write 2 Done Done New false
7855 2143 Acked 3142 Write 1 Done Done New false
7856 2144 Acked 3143 Write 9 Done Done New false
7857 2145 Acked 3144 Flush 0 Done Done New false
7858 2146 Acked 3145 Write 1 Done Done New false
7859 2147 Acked 3146 Write 10 Done Done New false
7860 2148 Acked 3147 Write 10 Done Done New false
7861 2149 Acked 3148 Flush 0 Done Done New false
7862 2150 Acked 3149 Write 6 Done Done New false
7863 2151 Acked 3150 Read 4 Done Done New false
7864 2152 Acked 3151 Flush 0 Done Done New false
7865 2153 Acked 3152 Write 1 Done Done New false
7866 2154 Acked 3153 Write 4 Done Done New false
7867 2155 Acked 3154 Write 6 Done Done New false
7868 2156 Acked 3155 Write 4 Done Done New false
7869 2157 Acked 3156 Read 5 Done Done New false
7870 2158 Acked 3157 Read 5 Done Done New false
7871 2159 Acked 3158 Write 8 Done Done New false
7872 2160 Acked 3159 Write 7 Done Done New false
7873 2161 Acked 3160 Read 9 Done Done New false
7874 2162 Acked 3161 Read 10 Done Done New false
7875 2163 Acked 3162 Write 7 Done Done New false
7876 2164 Acked 3163 Read 3 Done Done New false
7877 2165 Acked 3164 Read 4 Done Done New false
7878 2166 Acked 3165 Write 3 Done Done New false
7879 2167 Acked 3166 Write 2 Done Done New false
7880 2168 Acked 3167 Write 3 Done Done New false
7881 2169 Acked 3168 Write 10 Done Done New false
7882 2170 Acked 3169 Write 4 Done Done New false
7883 2171 Acked 3170 Read 3 Done Done New false
7884 2172 Acked 3171 Read 10 Done Done New false
7885 2173 Acked 3172 Read 8 Done Done New false
7886 2174 Acked 3173 Write 10 Done Done New false
7887 2175 Acked 3174 Flush 0 Done Done New false
7888 2176 Acked 3175 Write 8 Done Done New false
7889 2177 Acked 3176 Read 3 Done Done New false
7890 2178 Acked 3177 Write 1 Done Done New false
7891 2179 Acked 3178 Read 4 Done Done New false
7892 2180 Acked 3179 Read 10 Done Done New false
7893 2181 Acked 3180 Write 6 Done Done New false
7894 2182 Acked 3181 Read 8 Done Done New false
7895 2183 Acked 3182 Read 8 Done Done New false
7896 2184 Acked 3183 Flush 0 Done Done New false
7897 2185 Acked 3184 Write 8 Done Done New false
7898 2186 Acked 3185 Read 7 Done Done New false
7899 2187 Acked 3186 Read 1 Done Done New false
7900 2188 Acked 3187 Write 4 Done Done New false
7901 2189 Acked 3188 Read 7 Done Done New false
7902 2190 Acked 3189 Read 3 Done Done New false
7903 2191 Acked 3190 Read 4 Done Done New false
7904 2192 Acked 3191 Flush 0 Done Done New false
7905 2193 Acked 3192 Write 4 Done Done New false
7906 2194 Acked 3193 Write 10 Done Done New false
7907 2195 Acked 3194 Read 8 Done Done New false
7908 2196 Acked 3195 Write 3 Done Done New false
7909 2197 Acked 3196 Write 4 Done Done New false
7910 2198 Acked 3197 Write 3 Done Done New false
7911 2199 Acked 3198 Write 9 Done Done New false
7912 2200 Acked 3199 Read 6 Done Done New false
7913 2201 Acked 3200 Write 6 Done Done New false
7914 2202 Acked 3201 Write 8 Done Done New false
7915 2203 Acked 3202 Write 2 Done Done New false
7916 2204 Acked 3203 Write 10 Done Done New false
7917 2205 Acked 3204 Write 1 Done Done New false
7918 2206 Acked 3205 Read 10 Done Done New false
7919 2207 Acked 3206 Write 2 Done Done New false
7920 2208 Acked 3207 Read 5 Done Done New false
7921 2209 Acked 3208 Read 9 Done Done New false
7922 2210 Acked 3209 Read 7 Done Done New false
7923 2211 Acked 3210 Read 8 Done Done New false
7924 2212 Acked 3211 Flush 0 Done Done New false
7925 2213 Acked 3212 Read 2 Done Done New false
7926 2214 Acked 3213 Read 7 Done Done New false
7927 2215 Acked 3214 Write 1 Done Done New false
7928 2216 Acked 3215 Write 3 Done Done New false
7929 2217 Acked 3216 Write 3 Done Done New false
7930 2218 Acked 3217 Read 10 Done Done New false
7931 2219 Acked 3218 Write 2 Done Done New false
7932 2220 Acked 3219 Write 2 Done Done New false
7933 2221 Acked 3220 Write 1 Done Done New false
7934 2222 Acked 3221 Write 5 Done Done New false
7935 2223 Acked 3222 Write 5 Done Done New false
7936 2224 Acked 3223 Read 8 Done Done New false
7937 2225 Acked 3224 Read 8 Done Done New false
7938 2226 Acked 3225 Read 7 Done Done New false
7939 2227 Acked 3226 Read 6 Done Done New false
7940 2228 Acked 3227 Read 8 Done Done New false
7941 2229 Acked 3228 Read 10 Done Done New false
7942 2230 Acked 3229 Read 5 Done Done New false
7943 2231 Acked 3230 Flush 0 Done Done New false
7944 2232 Acked 3231 Read 2 Done Done New false
7945 2233 Acked 3232 Read 6 Done Done New false
7946 2234 Acked 3233 Read 8 Done Done New false
7947 2235 Acked 3234 Write 5 Done Done New false
7948 2236 Acked 3235 Write 7 Done Done New false
7949 2237 Acked 3236 Write 3 Done Done New false
7950 2238 Acked 3237 Read 2 Done Done New false
7951 2239 Acked 3238 Write 7 Done Done New false
7952 2240 Acked 3239 Read 9 Done Done New false
7953 2241 Acked 3240 Write 5 Done Done New false
7954 2242 Acked 3241 Read 8 Done Done New false
7955 2243 Acked 3242 Read 1 Done Done New false
7956 2244 Acked 3243 Read 2 Done Done New false
7957 2245 Acked 3244 Read 9 Done Done New false
7958 2246 Acked 3245 Write 7 Done Done New false
7959 2247 Acked 3246 Read 4 Done Done New false
7960 2248 Acked 3247 Read 6 Done Done New false
7961 2249 Acked 3248 Write 8 Done Done New false
7962 2250 Acked 3249 Write 3 Done Done New false
7963 2251 Acked 3250 Read 8 Done Done New false
7964 2252 Acked 3251 Read 3 Done Done New false
7965 2253 Acked 3252 Read 4 Done Done New false
7966 2254 Acked 3253 Read 2 Done Done New false
7967 2255 Acked 3254 Write 1 Done Done New false
7968 2256 Acked 3255 Read 4 Done Done New false
7969 2257 Acked 3256 Read 5 Done Done New false
7970 2258 Acked 3257 Write 8 Done Done New false
7971 2259 Acked 3258 Read 10 Done Done New false
7972 2260 Acked 3259 Read 6 Done Done New false
7973 2261 Acked 3260 Read 5 Done Done New false
7974 2262 Acked 3261 Write 4 Done Done New false
7975 2263 Acked 3262 Read 7 Done Done New false
7976 2264 Acked 3263 Write 7 Done Done New false
7977 2265 Acked 3264 Write 3 Done Done New false
7978 2266 Acked 3265 Write 8 Done Done New false
7979 2267 Acked 3266 Write 3 Done Done New false
7980 2268 Acked 3267 Read 1 Done Done New false
7981 2269 Acked 3268 Write 9 Done Done New false
7982 2270 Acked 3269 Write 9 Done Done New false
7983 2271 Acked 3270 Write 3 Done Done New false
7984 2272 Acked 3271 Read 4 Done Done New false
7985 2273 Acked 3272 Write 7 Done Done New false
7986 2274 Acked 3273 Write 2 Done Done New false
7987 2275 Acked 3274 Read 10 Done Done New false
7988 2276 Acked 3275 Write 5 Done Done New false
7989 2277 Acked 3276 Write 10 Done Done New false
7990 2278 Acked 3277 Write 7 Done Done New false
7991 2279 Acked 3278 Read 3 Done Done New false
7992 2280 Acked 3279 Write 9 Done Done New false
7993 2281 Acked 3280 Read 5 Done Done New false
7994 2282 Acked 3281 Read 10 Done Done New false
7995 2283 Acked 3282 Read 1 Done Done New false
7996 2284 Acked 3283 Write 7 Done Done New false
7997 2285 Acked 3284 Read 5 Done Done New false
7998 2286 Acked 3285 Read 6 Done Done New false
7999 2287 Acked 3286 Read 3 Done Done New false
8000 2288 Acked 3287 Write 4 Done Done New false
8001 2289 Acked 3288 Write 7 Done Done New false
8002 2290 Acked 3289 Write 1 Done Done New false
8003 2291 Acked 3290 Read 2 Done Done New false
8004 2292 Acked 3291 Write 4 Done Done New false
8005 2293 Acked 3292 Read 8 Done Done New false
8006 2294 Acked 3293 Write 6 Done Done New false
8007 2295 Acked 3294 Read 4 Done Done New false
8008 2296 Acked 3295 Write 9 Done Done New false
8009 2297 Acked 3296 Write 3 Done Done New false
8010 2298 Acked 3297 Write 4 Done Done New false
8011 2299 Acked 3298 Write 4 Done Done New false
8012 2300 Acked 3299 Write 7 Done Done New false
8013 2301 Acked 3300 Flush 0 Done Done New false
8014 2302 Acked 3301 Write 10 Done Done New false
8015 2303 Acked 3302 Read 5 Done Done New false
8016 2304 Acked 3303 Read 6 Done Done New false
8017 2305 Acked 3304 Write 1 Done Done New false
8018 2306 Acked 3305 Read 6 Done Done New false
8019 2307 Acked 3306 Flush 0 Done Done New false
8020 2308 Acked 3307 Write 6 Done Done New false
8021 2309 Acked 3308 Read 8 Done Done New false
8022 2310 Acked 3309 Write 1 Done Done New false
8023 2311 Acked 3310 Write 4 Done Done New false
8024 2312 Acked 3311 Read 3 Done Done New false
8025 2313 Acked 3312 Write 3 Done Done New false
8026 2314 Acked 3313 Read 8 Done Done New false
8027 2315 Acked 3314 Read 4 Done Done New false
8028 2316 Acked 3315 Write 7 Done Done New false
8029 2317 Acked 3316 Write 3 Done Done New false
8030 2318 Acked 3317 Write 3 Done Done New false
8031 2319 Acked 3318 Write 2 Done Done New false
8032 2320 Acked 3319 Flush 0 Done Done New false
8033 2321 Acked 3320 Write 7 Done Done New false
8034 2322 Acked 3321 Flush 0 Done Done New false
8035 2323 Acked 3322 Read 1 Done Done New false
8036 2324 Acked 3323 Write 4 Done Done New false
8037 2325 Acked 3324 Write 1 Done Done New false
8038 2326 Acked 3325 Read 2 Done Done New false
8039 2327 Acked 3326 Read 1 Done Done New false
8040 2328 Acked 3327 Read 1 Done Done New false
8041 2329 Acked 3328 Flush 0 Done Done New false
8042 2330 Acked 3329 Read 10 Done Done New false
8043 2331 Acked 3330 Write 2 Done Done New false
8044 2332 Acked 3331 Read 5 Done Done New false
8045 2333 Acked 3332 Read 8 Done Done New false
8046 2334 Acked 3333 Flush 0 Done Done New false
8047 2335 Acked 3334 Read 2 Done Done New false
8048 2336 Acked 3335 Read 8 Done Done New false
8049 2337 Acked 3336 Read 5 Done Done New false
8050 2338 Acked 3337 Write 5 Done Done New false
8051 2339 Acked 3338 Write 4 Done Done New false
8052 2340 Acked 3339 Write 2 Done Done New false
8053 2341 Acked 3340 Read 1 Done Done New false
8054 2342 Acked 3341 Read 6 Done Done New false
8055 2343 Acked 3342 Read 7 Done Done New false
8056 2344 Acked 3343 Write 8 Done Done New false
8057 2345 Acked 3344 Flush 0 Done Done New false
8058 2346 Acked 3345 Read 5 Done Done New false
8059 2347 Acked 3346 Write 7 Done Done New false
8060 2348 Acked 3347 Write 2 Done Done New false
8061 2349 Acked 3348 Flush 0 Done Done New false
8062 2350 Acked 3349 Write 1 Done Done New false
8063 2351 Acked 3350 Write 10 Done Done New false
8064 2352 Acked 3351 Read 1 Done Done New false
8065 2353 Acked 3352 Read 10 Done Done New false
8066 2354 Acked 3353 Flush 0 Done Done New false
8067 2355 Acked 3354 Write 8 Done Done New false
8068 2356 Acked 3355 Write 5 Done Done New false
8069 2357 Acked 3356 Read 3 Done Done New false
8070 2358 Acked 3357 Write 3 Done Done New false
8071 2359 Acked 3358 Read 6 Done Done New false
8072 2360 Acked 3359 Read 9 Done Done New false
8073 2361 Acked 3360 Read 3 Done Done New false
8074 2362 Acked 3361 Read 10 Done Done New false
8075 2363 Acked 3362 Flush 0 Done Done New false
8076 2364 Acked 3363 Write 4 Done Done New false
8077 2365 Acked 3364 Read 9 Done Done New false
8078 2366 Acked 3365 Flush 0 Done Done New false
8079 2367 Acked 3366 Write 2 Done Done New false
8080 2368 Acked 3367 Read 8 Done Done New false
8081 2369 Acked 3368 Read 3 Done Done New false
8082 2370 Acked 3369 Read 6 Done Done New false
8083 2371 Acked 3370 Flush 0 Done Done New false
8084 2372 Acked 3371 Read 2 Done Done New false
8085 2373 Acked 3372 Flush 0 Done Done New false
8086 2374 Acked 3373 Read 5 Done Done New false
8087 2375 Acked 3374 Read 3 Done Done New false
8088 2376 Acked 3375 Write 9 Done Done New false
8089 2377 Acked 3376 Flush 0 Done Done New false
8090 2378 Acked 3377 Read 3 Done Done New false
8091 2379 Acked 3378 Read 7 Done Done New false
8092 2380 Acked 3379 Write 10 Done Done New false
8093 2381 Acked 3380 Write 1 Done Done New false
8094 2382 Acked 3381 Read 2 Done Done New false
8095 2383 Acked 3382 Write 3 Done Done New false
8096 2384 Acked 3383 Read 6 Done Done New false
8097 2385 Acked 3384 Read 7 Done Done New false
8098 2386 Acked 3385 Read 7 Done Done New false
8099 2387 Acked 3386 Read 6 Done Done New false
8100 2388 Acked 3387 Write 8 Done Done New false
8101 2389 Acked 3388 Write 10 Done Done New false
8102 2390 Acked 3389 Read 1 Done Done New false
8103 2391 Acked 3390 Read 5 Done Done New false
8104 2392 Acked 3391 Write 2 Done Done New false
8105 2393 Acked 3392 Write 6 Done Done New false
8106 2394 Acked 3393 Write 4 Done Done New false
8107 2395 Acked 3394 Write 8 Done Done New false
8108 2396 Acked 3395 Write 10 Done Done New false
8109 2397 Acked 3396 Flush 0 Done Done New false
8110 2398 Acked 3397 Read 9 Done Done New false
8111 2399 Acked 3398 Write 4 Done Done New false
8112 2400 Acked 3399 Read 2 Done Done New false
8113 2401 Acked 3400 Read 5 Done Done New false
8114 2402 Acked 3401 Flush 0 Done Done New false
8115 2403 Acked 3402 Read 8 Done Done New false
8116 2404 Acked 3403 Write 5 Done Done New false
8117 2405 Acked 3404 Read 10 Done Done New false
8118 2406 Acked 3405 Read 5 Done Done New false
8119 2407 Acked 3406 Read 1 Done Done New false
8120 2408 Acked 3407 Read 5 Done Done New false
8121 2409 Acked 3408 Write 2 Done Done New false
8122 STATES DS:0 DS:1 DS:2 TOTAL
8123 New 0 0 300 300
8124 Sent 0 0 0 0
8125 Done 300 300 0 600
8126 Skipped 0 0 0 0
8127 Error 0 0 0 0
8128 Last Flush: 3401 3401 3108
8129 Downstairs last five completed: 3108 3107 3106 3105 3104
8130 Upstairs last five completed: 2409 2408 2407 2406 2405
8131 CLIENT: Up:0 ds:300 act:2
81322024-02-27T21:46:13.471ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81332024-02-27T21:46:13.471ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81342024-02-27T21:46:13.471ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81352024-02-27T21:46:13.471ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
81362024-02-27T21:46:13.471ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81372024-02-27T21:46:13.471ZINFOcrucible: [2] client re-new 301 jobs since flush 3108 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
81382024-02-27T21:46:13.472ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81392024-02-27T21:46:14.472ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81402024-02-27T21:46:14.472ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81412024-02-27T21:46:14.473ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81422024-02-27T21:46:14.473ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
81432024-02-27T21:46:14.473ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81442024-02-27T21:46:14.473ZINFOcrucible: [2] client re-new 301 jobs since flush 3108 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
81452024-02-27T21:46:14.473ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
8146 [8830] got start action
8147 [8830] I am going to restart
8148 Starting downstairs at port 8830
8149 Make output file at "/tmp/dsc/downstairs-8830.txt"
8150 [8830][2] reports Starting
8151 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1175)
8152 [8830][2] reports Running
81532024-02-27T21:46:15.475ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81542024-02-27T21:46:15.475ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81552024-02-27T21:46:15.476ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81562024-02-27T21:46:15.476ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81572024-02-27T21:46:15.476ZINFOcrucible: send last flush ID to this DS: 3108 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81582024-02-27T21:46:15.476ZINFOcrucible: Replied this last flush ID: 3108 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81592024-02-27T21:46:15.476ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81602024-02-27T21:46:15.476ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81612024-02-27T21:46:15.476ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81622024-02-27T21:46:15.476ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
8163 ----------------------------------------------------------------
8164 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
8165 Downstairs last five completed: 3409 3408 3407 3406 3405
8166 Upstairs last five completed: 2410 2409 2408 2407 2406
8167 CLIENT: Up:0 ds:0 act:3
8168 Replay: All jobs finished, all DS active.
8169 Main task has work to do, go find it
8170 got dsc Stop(2)
8171 stop 2
8172 Feb 27 21:46:16.809 INFO request completed, latency_us: 361, response_code: 204, uri: [8830] Got stop action so:false kr:false
8173 /stop/cid/2, method: POST, req_id: 234333ea-d6f2-4a8d-959a-c9224be4fe9a, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
8174 Feb 27 21:46:16.810 INFO request completed, latency_us: 110, response_code: 200, uri: /state/cid/2, method: GET, req_id: 36988d6c-29a4-4d91-be68-10e343d23bba, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
81752024-02-27T21:46:16.810ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81762024-02-27T21:46:16.810ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81772024-02-27T21:46:16.810ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = e6a5a974-925a-494f-8845-facc441c2690
81782024-02-27T21:46:16.810ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81792024-02-27T21:46:16.810ZINFOcrucible: [2] client re-new 0 jobs since flush 3409 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
81802024-02-27T21:46:16.810ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
8181 [8830] Exited with: ExitStatus(unix_wait_status(9))
8182 [8830][2] reports Exit
81832024-02-27T21:46:17.811ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81842024-02-27T21:46:17.811ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81852024-02-27T21:46:17.811ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81862024-02-27T21:46:17.811ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
81872024-02-27T21:46:17.811ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81882024-02-27T21:46:17.811ZINFOcrucible: [2] client re-new 0 jobs since flush 3409 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
81892024-02-27T21:46:17.811ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81902024-02-27T21:46:18.812ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81912024-02-27T21:46:18.812ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81922024-02-27T21:46:18.812ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81932024-02-27T21:46:18.812ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
81942024-02-27T21:46:18.812ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81952024-02-27T21:46:18.812ZINFOcrucible: [2] client re-new 0 jobs since flush 3409 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
81962024-02-27T21:46:18.812ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81972024-02-27T21:46:19.813ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81982024-02-27T21:46:19.814ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
81992024-02-27T21:46:19.814ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
82002024-02-27T21:46:19.814ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
82012024-02-27T21:46:19.814ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
82022024-02-27T21:46:19.814ZINFOcrucible: [2] client re-new 0 jobs since flush 3409 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
82032024-02-27T21:46:19.814ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
8204 Feb 27 21:46:20.813 INFO request completed, latency_us: 206, response_code: 200, uri: /state/cid/2, method: GET, req_id: 1da1e1ac-11d2-4ba4-ac8d-e848ca8a4491, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
8205 001/300 Write block 3798 len 32768 data: 5 5 5 5 4 4 3 3
8206 002/300 Read block 3492 len 36864
82072024-02-27T21:46:20.815ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
82082024-02-27T21:46:20.815ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
82092024-02-27T21:46:20.815ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
82102024-02-27T21:46:20.815ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
82112024-02-27T21:46:20.815ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
82122024-02-27T21:46:20.815ZINFOcrucible: [2] client re-new 2 jobs since flush 3409 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
82132024-02-27T21:46:20.815ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
8214 003/300 Write block 2536 len 24576 data: 2 3 3 3 3 3
8215 004/300 Read block 3962 len 8192
8216 005/300 Write block 4147 len 12288 data: 3 4 4
8217 006/300 Write block 3542 len 36864 data: 3 3 3 2 2 2 2 2 2
8218 007/300 Write block 151 len 36864 data: 2 2 2 2 2 2 2 2 2
8219 008/300 Read block 2284 len 4096
8220 009/300 Read block 4889 len 40960
8221 010/300 Read block 3637 len 8192
8222 011/300 Write block 2607 len 24576 data: 2 2 2 2 2 2
8223 012/300 Read block 747 len 40960
8224 013/300 Write block 4449 len 20480 data: 2 2 3 3 3
8225 014/300 Flush
8226 015/300 Read block 4539 len 32768
8227 016/300 Read block 4621 len 36864
8228 017/300 Write block 4317 len 16384 data: 5 5 5 5
8229 018/300 Write block 1359 len 24576 data: 2 2 4 4 4 4
8230 019/300 Write block 2096 len 32768 data: 3 3 3 3 3 3 2 2
8231 020/300 Flush
8232 021/300 Read block 1924 len 12288
8233 022/300 Read block 898 len 12288
8234 023/300 Read block 2893 len 36864
8235 024/300 Write block 1598 len 16384 data: 2 3 3 2
8236 025/300 Read block 2329 len 8192
8237 026/300 Write block 3743 len 8192 data: 2 2
8238 027/300 Write block 2771 len 16384 data: 2 2 2 2
8239 028/300 Read block 3787 len 8192
8240 029/300 Read block 2585 len 32768
8241 030/300 Flush
8242 031/300 Flush
8243 032/300 Write block 454 len 36864 data: 3 3 2 2 2 2 2 2 2
8244 033/300 Read block 4169 len 36864
8245 034/300 Read block 446 len 32768
8246 035/300 Read block 802 len 8192
8247 036/300 Write block 907 len 40960 data: 4 4 4 4 4 4 4 4 4 3
8248 037/300 Read block 3525 len 24576
8249 038/300 Read block 2688 len 24576
8250 039/300 Read block 1923 len 8192
8251 040/300 Write block 4359 len 24576 data: 3 3 4 4 4 4
8252 041/300 Read block 1963 len 32768
8253 042/300 Read block 4611 len 4096
8254 043/300 Write block 190 len 32768 data: 2 2 3 3 3 3 3 3
8255 044/300 Write block 969 len 36864 data: 3 3 3 3 3 2 2 2 2
8256 045/300 Read block 400 len 24576
8257 046/300 Write block 2368 len 16384 data: 3 3 3 3
8258 047/300 Write block 1493 len 16384 data: 2 2 2 2
8259 048/300 Write block 826 len 28672 data: 2 2 2 2 2 2 2
8260 049/300 Read block 1566 len 20480
8261 050/300 Read block 1227 len 24576
8262 051/300 Read block 223 len 16384
8263 052/300 Read block 827 len 40960
8264 053/300 Read block 3964 len 4096
8265 054/300 Read block 2964 len 16384
8266 055/300 Write block 1196 len 40960 data: 2 2 2 2 3 3 3 3 3 2
8267 056/300 Read block 2175 len 12288
8268 057/300 Write block 347 len 16384 data: 2 2 2 2
8269 058/300 Flush
8270 059/300 Read block 4197 len 12288
8271 060/300 Read block 1894 len 12288
8272 061/300 Read block 829 len 24576
8273 062/300 Flush
8274 063/300 Read block 71 len 4096
8275 064/300 Write block 3164 len 12288 data: 3 3 2
8276 065/300 Write block 4870 len 12288 data: 2 2 2
8277 066/300 Write block 2079 len 24576 data: 2 2 2 3 3 3
8278 067/300 Read block 3172 len 28672
8279 068/300 Read block 2069 len 28672
8280 069/300 Read block 3321 len 36864
8281 070/300 Read block 1996 len 16384
8282 071/300 Write block 495 len 28672 data: 3 3 4 4 5 5 4
8283 072/300 Read block 2832 len 8192
8284 073/300 Read block 4478 len 16384
8285 074/300 Read block 1788 len 12288
8286 075/300 Write block 909 len 36864 data: 5 5 5 5 5 5 5 4 3
8287 076/300 Read block 368 len 24576
8288 077/300 Read block 3717 len 16384
8289 078/300 Flush
8290 079/300 Write block 1710 len 24576 data: 4 4 6 6 4 4
8291 080/300 Read block 3056 len 12288
8292 081/300 Read block 3204 len 4096
8293 082/300 Write block 3900 len 40960 data: 3 3 3 3 3 3 4 3 3 2
8294 083/300 Read block 3335 len 16384
8295 084/300 Read block 3684 len 32768
8296 085/300 Write block 680 len 4096 data: 3
8297 086/300 Write block 4128 len 8192 data: 3 3
8298 087/300 Read block 2110 len 4096
8299 088/300 Read block 1099 len 40960
8300 089/300 Write block 134 len 20480 data: 2 2 2 2 3
8301 090/300 Read block 3825 len 36864
8302 091/300 Read block 4597 len 12288
8303 092/300 Read block 1704 len 40960
8304 093/300 Read block 4826 len 24576
8305 094/300 Read block 2334 len 4096
8306 095/300 Read block 4237 len 12288
8307 096/300 Write block 3123 len 32768 data: 2 2 2 2 2 2 2 2
8308 097/300 Read block 3633 len 8192
8309 098/300 Read block 1796 len 24576
8310 099/300 Flush
8311 100/300 Read block 3468 len 28672
8312 101/300 Write block 3461 len 4096 data: 2
8313 102/300 Read block 3902 len 24576
8314 103/300 Write block 4928 len 32768 data: 3 4 3 3 3 3 4 3
8315 104/300 Read block 3263 len 32768
8316 105/300 Write block 4852 len 20480 data: 3 2 2 2 2
8317 106/300 Read block 1605 len 12288
8318 107/300 Read block 1977 len 12288
8319 108/300 Read block 4199 len 36864
8320 109/300 Write block 3039 len 4096 data: 3
8321 110/300 Write block 2905 len 40960 data: 3 3 3 4 4 4 4 3 3 2
8322 111/300 Write block 3887 len 20480 data: 2 3 3 3 3
8323 112/300 Read block 1709 len 8192
8324 113/300 Read block 4974 len 28672
8325 114/300 Read block 1997 len 28672
8326 115/300 Read block 690 len 28672
8327 116/300 Read block 37 len 40960
8328 117/300 Flush
8329 118/300 Read block 1038 len 12288
8330 119/300 Read block 1066 len 36864
8331 120/300 Write block 146 len 24576 data: 2 3 2 2 2 3
8332 121/300 Flush
8333 122/300 Flush
8334 123/300 Read block 1194 len 28672
8335 124/300 Write block 1823 len 40960 data: 2 2 3 3 3 3 2 2 2 2
8336 125/300 Flush
8337 126/300 Flush
8338 127/300 Write block 2046 len 16384 data: 3 3 4 4
8339 128/300 Read block 4672 len 16384
8340 129/300 Read block 3692 len 36864
8341 130/300 Write block 503 len 20480 data: 5 2 4 4 4
8342 131/300 Read block 4179 len 20480
8343 132/300 Write block 735 len 32768 data: 3 3 3 3 3 2 2 2
8344 133/300 Write block 1566 len 20480 data: 3 2 2 3 3
8345 134/300 Write block 3914 len 32768 data: 2 2 2 2 2 2 2 2
8346 135/300 Write block 2023 len 8192 data: 4 5
8347 136/300 Read block 288 len 36864
8348 137/300 Read block 541 len 28672
8349 138/300 Read block 240 len 8192
8350 139/300 Read block 4763 len 20480
8351 140/300 Write block 607 len 28672 data: 2 2 2 2 2 2 3
8352 141/300 Write block 1286 len 16384 data: 2 3 4 4
8353 142/300 Write block 3099 len 4096 data: 2
8354 143/300 Write block 1528 len 4096 data: 2
8355 144/300 Write block 4543 len 24576 data: 3 3 3 3 4 4
8356 145/300 Write block 2813 len 36864 data: 2 2 3 3 3 4 4 3 3
8357 146/300 Read block 3113 len 20480
8358 147/300 Read block 2000 len 36864
8359 148/300 Read block 2240 len 24576
8360 149/300 Write block 4854 len 24576 data: 3 3 3 2 2 2
8361 150/300 Write block 2699 len 24576 data: 3 2 2 2 2 2
8362 151/300 Read block 510 len 20480
8363 152/300 Read block 2987 len 36864
8364 153/300 Read block 2432 len 4096
8365 154/300 Write block 2334 len 32768 data: 4 5 5 5 4 5 4 4
8366 155/300 Write block 4793 len 32768 data: 4 4 3 3 2 2 2 2
8367 156/300 Read block 271 len 16384
8368 157/300 Read block 76 len 16384
8369 158/300 Read block 1683 len 32768
8370 159/300 Write block 1981 len 12288 data: 4 4 3
8371 160/300 Read block 752 len 20480
8372 161/300 Flush
8373 162/300 Read block 580 len 12288
8374 163/300 Read block 3886 len 28672
8375 164/300 Read block 1587 len 32768
8376 165/300 Write block 316 len 12288 data: 5 3 3
8377 166/300 Write block 1190 len 8192 data: 3 3
8378 167/300 Write block 152 len 4096 data: 3
8379 168/300 Read block 4730 len 4096
8380 169/300 Read block 716 len 36864
8381 170/300 Read block 1389 len 40960
8382 171/300 Read block 3837 len 40960
8383 172/300 Write block 4557 len 8192 data: 3 3
8384 173/300 Read block 598 len 12288
8385 174/300 Flush
8386 175/300 Flush
8387 176/300 Read block 4807 len 32768
8388 177/300 Read block 66 len 32768
8389 178/300 Write block 962 len 16384 data: 2 2 2 3
8390 179/300 Read block 3705 len 28672
8391 180/300 Write block 2978 len 20480 data: 3 3 2 2 2
8392 181/300 Read block 467 len 32768
8393 182/300 Read block 1357 len 36864
8394 183/300 Write block 400 len 36864 data: 3 3 3 2 2 2 2 4 4
8395 184/300 Write block 4427 len 28672 data: 3 4 2 3 3 3 3
8396 185/300 Read block 3581 len 8192
8397 186/300 Write block 586 len 20480 data: 4 3 3 3 3
8398 187/300 Read block 3929 len 32768
8399 188/300 Read block 1306 len 24576
8400 189/300 Write block 1839 len 12288 data: 3 3 3
8401 190/300 Read block 4050 len 28672
8402 191/300 Write block 4436 len 40960 data: 5 5 4 4 4 4 3 3 4 4
8403 192/300 Write block 3256 len 24576 data: 3 4 4 3 2 2
8404 193/300 Read block 1134 len 8192
8405 194/300 Flush
8406 195/300 Read block 2239 len 32768
8407 196/300 Read block 1379 len 36864
8408 197/300 Write block 2661 len 8192 data: 3 3
8409 198/300 Write block 1154 len 32768 data: 2 2 2 3 3 3 3 3
8410 199/300 Write block 4830 len 24576 data: 4 3 4 4 4 3
8411 200/300 Read block 979 len 8192
8412 201/300 Read block 4476 len 4096
8413 202/300 Read block 4775 len 12288
8414 203/300 Write block 4185 len 32768 data: 2 2 2 2 2 2 3 3
8415 204/300 Read block 3216 len 28672
8416 205/300 Read block 3103 len 36864
8417 206/300 Write block 1319 len 32768 data: 6 6 4 4 5 5 4 3
8418 207/300 Read block 772 len 24576
8419 208/300 Read block 1279 len 8192
8420 209/300 Write block 4678 len 12288 data: 3 3 3
8421 210/300 Write block 997 len 16384 data: 2 2 2 2
8422 211/300 Write block 4693 len 12288 data: 5 4 4
8423 212/300 Read block 633 len 8192
8424 213/300 Read block 1417 len 12288
8425 214/300 Read block 2319 len 4096
8426 215/300 Read block 1835 len 8192
8427 216/300 Flush
8428 217/300 Flush
8429 218/300 Write block 3443 len 8192 data: 4 4
8430 219/300 Read block 4027 len 32768
8431 220/300 Read block 3746 len 40960
8432 221/300 Write block 3699 len 4096 data: 2
8433 222/300 Read block 342 len 36864
8434 223/300 Write block 4449 len 36864 data: 3 3 4 4 4 3 3 3 3
8435 224/300 Write block 1130 len 12288 data: 3 4 6
8436 225/300 Flush
8437 226/300 Read block 3745 len 36864
8438 227/300 Read block 3622 len 4096
8439 228/300 Write block 4140 len 20480 data: 2 3 3 3 2
8440 229/300 Write block 3527 len 28672 data: 5 3 3 3 4 4 3
8441 230/300 Write block 2348 len 20480 data: 4 6 5 4 4
8442 231/300 Read block 134 len 40960
8443 232/300 Write block 2756 len 36864 data: 2 2 2 3 3 4 4 4 3
8444 233/300 Read block 975 len 20480
8445 234/300 Read block 1144 len 40960
8446 235/300 Read block 4197 len 4096
8447 236/300 Write block 1639 len 8192 data: 4 3
8448 237/300 Read block 4275 len 24576
8449 238/300 Read block 2039 len 8192
8450 239/300 Write block 3109 len 40960 data: 3 4 4 3 3 4 5 5 5 4
8451 240/300 Write block 2225 len 40960 data: 3 3 3 2 3 2 3 3 3 3
8452 241/300 Write block 2775 len 4096 data: 2
8453 242/300 Read block 1307 len 4096
8454 243/300 Read block 2265 len 12288
8455 244/300 Flush
8456 245/300 Write block 1036 len 20480 data: 2 2 2 2 2
8457 246/300 Read block 3158 len 40960
8458 247/300 Read block 3081 len 20480
8459 248/300 Read block 1133 len 32768
8460 249/300 Read block 3492 len 4096
8461 250/300 Read block 1003 len 36864
8462 251/300 Flush
8463 252/300 Flush
8464 253/300 Read block 4343 len 40960
8465 254/300 Write block 4686 len 28672 data: 5 5 4 4 4 4 5
8466 255/300 Read block 1219 len 28672
8467 256/300 Write block 1509 len 20480 data: 5 5 4 4 3
8468 257/300 Write block 3978 len 4096 data: 5
8469 258/300 Read block 1302 len 28672
8470 259/300 Read block 2225 len 16384
8471 260/300 Write block 2920 len 16384 data: 2 2 2 2
8472 261/300 Read block 3338 len 12288
8473 262/300 Flush
8474 263/300 Read block 4543 len 4096
8475 264/300 Read block 3003 len 32768
8476 265/300 Read block 3339 len 12288
8477 266/300 Write block 4272 len 16384 data: 4 3 3 3
8478 267/300 Write block 956 len 4096 data: 4
8479 268/300 Flush
8480 269/300 Write block 1487 len 8192 data: 3 2
8481 270/300 Read block 1514 len 32768
8482 271/300 Read block 3160 len 8192
8483 272/300 Flush
8484 273/300 Write block 3677 len 8192 data: 4 6
8485 274/300 Read block 3472 len 12288
8486 275/300 Write block 3942 len 4096 data: 2
8487 276/300 Read block 244 len 12288
8488 277/300 Read block 3580 len 28672
8489 278/300 Read block 3159 len 12288
8490 279/300 Read block 4901 len 4096
8491 280/300 Read block 1897 len 20480
8492 281/300 Write block 652 len 8192 data: 2 2
8493 282/300 Write block 2050 len 20480 data: 4 4 3 4 4
8494 283/300 Read block 433 len 8192
8495 284/300 Write block 1145 len 28672 data: 3 3 3 3 2 2 2
8496 285/300 Read block 2736 len 24576
8497 286/300 Flush
8498 287/300 Write block 2942 len 36864 data: 3 3 3 3 2 2 2 2 2
8499 288/300 Read block 1278 len 20480
8500 289/300 Write block 1140 len 32768 data: 3 3 3 4 3 4 4 4
8501 290/300 Read block 2971 len 8192
8502 291/300 Write block 2532 len 36864 data: 2 2 2 2 3 4 4 4 4
8503 292/300 Read block 4969 len 20480
8504 293/300 Read block 4680 len 12288
8505 294/300 Read block 1630 len 24576
8506 295/300 Read block 3390 len 4096
8507 296/300 Write block 3964 len 8192 data: 4 4
8508 297/300 Read block 66 len 12288
8509 298/300 Read block 1841 len 4096
8510 299/300 Write block 424 len 4096 data: 3
8511 300/300 Read block 3375 len 32768
8512 Main task has work to do, go find it
8513 got dsc Start(2)
8514 start 2
8515 Feb 27 21:46:21.147 INFO request completed, latency_us: 208, response_code: 204, uri: /start/cid/2, method: POST, req_id: beecec76-f866-42ec-87d1-e2b6957556ca, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
8516 Replay: started 2, returned:Ok(())
8517 ----------------------------------------------------------------
8518 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8519 States: Active Active Offline
8520 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
8521 2411 Acked 3410 Write 8 Done Done New true
8522 2412 Acked 3411 Read 9 Done Done New true
8523 2413 Acked 3412 Write 6 Done Done New false
8524 2414 Acked 3413 Read 2 Done Done New false
8525 2415 Acked 3414 Write 3 Done Done New false
8526 2416 Acked 3415 Write 9 Done Done New false
8527 2417 Acked 3416 Write 9 Done Done New false
8528 2418 Acked 3417 Read 1 Done Done New false
8529 2419 Acked 3418 Read 10 Done Done New false
8530 2420 Acked 3419 Read 2 Done Done New false
8531 2421 Acked 3420 Write 6 Done Done New false
8532 2422 Acked 3421 Read 10 Done Done New false
8533 2423 Acked 3422 Write 5 Done Done New false
8534 2424 Acked 3423 Flush 0 Done Done New false
8535 2425 Acked 3424 Read 8 Done Done New false
8536 2426 Acked 3425 Read 9 Done Done New false
8537 2427 Acked 3426 Write 4 Done Done New false
8538 2428 Acked 3427 Write 6 Done Done New false
8539 2429 Acked 3428 Write 8 Done Done New false
8540 2430 Acked 3429 Flush 0 Done Done New false
8541 2431 Acked 3430 Read 3 Done Done New false
8542 2432 Acked 3431 Read 3 Done Done New false
8543 2433 Acked 3432 Read 9 Done Done New false
8544 2434 Acked 3433 Write 4 Done Done New false
8545 2435 Acked 3434 Read 2 Done Done New false
8546 2436 Acked 3435 Write 2 Done Done New false
8547 2437 Acked 3436 Write 4 Done Done New false
8548 2438 Acked 3437 Read 2 Done Done New false
8549 2439 Acked 3438 Read 8 Done Done New false
8550 2440 Acked 3439 Flush 0 Done Done New false
8551 2441 Acked 3440 Flush 0 Done Done New false
8552 2442 Acked 3441 Write 9 Done Done New false
8553 2443 Acked 3442 Read 9 Done Done New false
8554 2444 Acked 3443 Read 8 Done Done New false
8555 2445 Acked 3444 Read 2 Done Done New false
8556 2446 Acked 3445 Write 10 Done Done New false
8557 2447 Acked 3446 Read 6 Done Done New false
8558 2448 Acked 3447 Read 6 Done Done New false
8559 2449 Acked 3448 Read 2 Done Done New false
8560 2450 Acked 3449 Write 6 Done Done New false
8561 2451 Acked 3450 Read 8 Done Done New false
8562 2452 Acked 3451 Read 1 Done Done New false
8563 2453 Acked 3452 Write 8 Done Done New false
8564 2454 Acked 3453 Write 9 Done Done New false
8565 2455 Acked 3454 Read 6 Done Done New false
8566 2456 Acked 3455 Write 4 Done Done New false
8567 2457 Acked 3456 Write 4 Done Done New false
8568 2458 Acked 3457 Write 7 Done Done New false
8569 2459 Acked 3458 Read 5 Done Done New false
8570 2460 Acked 3459 Read 6 Done Done New false
8571 2461 Acked 3460 Read 4 Done Done New false
8572 2462 Acked 3461 Read 10 Done Done New false
8573 2463 Acked 3462 Read 1 Done Done New false
8574 2464 Acked 3463 Read 4 Done Done New false
8575 2465 Acked 3464 Write 10 Done Done New false
8576 2466 Acked 3465 Read 3 Done Done New false
8577 2467 Acked 3466 Write 4 Done Done New false
8578 2468 Acked 3467 Flush 0 Done Done New false
8579 2469 Acked 3468 Read 3 Done Done New false
8580 2470 Acked 3469 Read 3 Done Done New false
8581 2471 Acked 3470 Read 6 Done Done New false
8582 2472 Acked 3471 Flush 0 Done Done New false
8583 2473 Acked 3472 Read 1 Done Done New false
8584 2474 Acked 3473 Write 3 Done Done New false
8585 2475 Acked 3474 Write 3 Done Done New false
8586 2476 Acked 3475 Write 6 Done Done New false
8587 2477 Acked 3476 Read 7 Done Done New false
8588 2478 Acked 3477 Read 7 Done Done New false
8589 2479 Acked 3478 Read 9 Done Done New false
8590 2480 Acked 3479 Read 4 Done Done New false
8591 2481 Acked 3480 Write 7 Done Done New false
8592 2482 Acked 3481 Read 2 Done Done New false
8593 2483 Acked 3482 Read 4 Done Done New false
8594 2484 Acked 3483 Read 3 Done Done New false
8595 2485 Acked 3484 Write 9 Done Done New false
8596 2486 Acked 3485 Read 6 Done Done New false
8597 2487 Acked 3486 Read 4 Done Done New false
8598 2488 Acked 3487 Flush 0 Done Done New false
8599 2489 Acked 3488 Write 6 Done Done New false
8600 2490 Acked 3489 Read 3 Done Done New false
8601 2491 Acked 3490 Read 1 Done Done New false
8602 2492 Acked 3491 Write 10 Done Done New false
8603 2493 Acked 3492 Read 4 Done Done New false
8604 2494 Acked 3493 Read 8 Done Done New false
8605 2495 Acked 3494 Write 1 Done Done New false
8606 2496 Acked 3495 Write 2 Done Done New false
8607 2497 Acked 3496 Read 1 Done Done New false
8608 2498 Acked 3497 Read 10 Done Done New false
8609 2499 Acked 3498 Write 5 Done Done New false
8610 2500 Acked 3499 Read 9 Done Done New false
8611 2501 Acked 3500 Read 3 Done Done New false
8612 2502 Acked 3501 Read 10 Done Done New false
8613 2503 Acked 3502 Read 6 Done Done New false
8614 2504 Acked 3503 Read 1 Done Done New false
8615 2505 Acked 3504 Read 3 Done Done New false
8616 2506 Acked 3505 Write 8 Done Done New false
8617 2507 Acked 3506 Read 2 Done Done New false
8618 2508 Acked 3507 Read 6 Done Done New false
8619 2509 Acked 3508 Flush 0 Done Done New false
8620 2510 Acked 3509 Read 7 Done Done New false
8621 2511 Acked 3510 Write 1 Done Done New false
8622 2512 Acked 3511 Read 6 Done Done New false
8623 2513 Acked 3512 Write 8 Done Done New false
8624 2514 Acked 3513 Read 8 Done Done New false
8625 2515 Acked 3514 Write 5 Done Done New false
8626 2516 Acked 3515 Read 3 Done Done New false
8627 2517 Acked 3516 Read 3 Done Done New false
8628 2518 Acked 3517 Read 9 Done Done New false
8629 2519 Acked 3518 Write 1 Done Done New false
8630 2520 Acked 3519 Write 10 Done Done New false
8631 2521 Acked 3520 Write 5 Done Done New false
8632 2522 Acked 3521 Read 2 Done Done New false
8633 2523 Acked 3522 Read 7 Done Done New false
8634 2524 Acked 3523 Read 7 Done Done New false
8635 2525 Acked 3524 Read 7 Done Done New false
8636 2526 Acked 3525 Read 10 Done Done New false
8637 2527 Acked 3526 Flush 0 Done Done New false
8638 2528 Acked 3527 Read 3 Done Done New false
8639 2529 Acked 3528 Read 9 Done Done New false
8640 2530 Acked 3529 Write 6 Done Done New false
8641 2531 Acked 3530 Flush 0 Done Done New false
8642 2532 Acked 3531 Flush 0 Done Done New false
8643 2533 Acked 3532 Read 7 Done Done New false
8644 2534 Acked 3533 Write 10 Done Done New false
8645 2535 Acked 3534 Flush 0 Done Done New false
8646 2536 Acked 3535 Flush 0 Done Done New false
8647 2537 Acked 3536 Write 4 Done Done New false
8648 2538 Acked 3537 Read 4 Done Done New false
8649 2539 Acked 3538 Read 9 Done Done New false
8650 2540 Acked 3539 Write 5 Done Done New false
8651 2541 Acked 3540 Read 5 Done Done New false
8652 2542 Acked 3541 Write 8 Done Done New false
8653 2543 Acked 3542 Write 5 Done Done New false
8654 2544 Acked 3543 Write 8 Done Done New false
8655 2545 Acked 3544 Write 2 Done Done New false
8656 2546 Acked 3545 Read 9 Done Done New false
8657 2547 Acked 3546 Read 7 Done Done New false
8658 2548 Acked 3547 Read 2 Done Done New false
8659 2549 Acked 3548 Read 5 Done Done New false
8660 2550 Acked 3549 Write 7 Done Done New false
8661 2551 Acked 3550 Write 4 Done Done New false
8662 2552 Acked 3551 Write 1 Done Done New false
8663 2553 Acked 3552 Write 1 Done Done New false
8664 2554 Acked 3553 Write 6 Done Done New false
8665 2555 Acked 3554 Write 9 Done Done New false
8666 2556 Acked 3555 Read 5 Done Done New false
8667 2557 Acked 3556 Read 9 Done Done New false
8668 2558 Acked 3557 Read 6 Done Done New false
8669 2559 Acked 3558 Write 6 Done Done New false
8670 2560 Acked 3559 Write 6 Done Done New false
8671 2561 Acked 3560 Read 5 Done Done New false
8672 2562 Acked 3561 Read 9 Done Done New false
8673 2563 Acked 3562 Read 1 Done Done New false
8674 2564 Acked 3563 Write 8 Done Done New false
8675 2565 Acked 3564 Write 8 Done Done New false
8676 2566 Acked 3565 Read 4 Done Done New false
8677 2567 Acked 3566 Read 4 Done Done New false
8678 2568 Acked 3567 Read 8 Done Done New false
8679 2569 Acked 3568 Write 3 Done Done New false
8680 2570 Acked 3569 Read 5 Done Done New false
8681 2571 Acked 3570 Flush 0 Done Done New false
8682 2572 Acked 3571 Read 3 Done Done New false
8683 2573 Acked 3572 Read 7 Done Done New false
8684 2574 Acked 3573 Read 8 Done Done New false
8685 2575 Acked 3574 Write 3 Done Done New false
8686 2576 Acked 3575 Write 2 Done Done New false
8687 2577 Acked 3576 Write 1 Done Done New false
8688 2578 Acked 3577 Read 1 Done Done New false
8689 2579 Acked 3578 Read 9 Done Done New false
8690 2580 Acked 3579 Read 10 Done Done New false
8691 2581 Acked 3580 Read 10 Done Done New false
8692 2582 Acked 3581 Write 2 Done Done New false
8693 2583 Acked 3582 Read 3 Done Done New false
8694 2584 Acked 3583 Flush 0 Done Done New false
8695 2585 Acked 3584 Flush 0 Done Done New false
8696 2586 Acked 3585 Read 8 Done Done New false
8697 2587 Acked 3586 Read 8 Done Done New false
8698 2588 Acked 3587 Write 4 Done Done New false
8699 2589 Acked 3588 Read 7 Done Done New false
8700 2590 Acked 3589 Write 5 Done Done New false
8701 2591 Acked 3590 Read 8 Done Done New false
8702 2592 Acked 3591 Read 9 Done Done New false
8703 2593 Acked 3592 Write 9 Done Done New false
8704 2594 Acked 3593 Write 7 Done Done New false
8705 2595 Acked 3594 Read 2 Done Done New false
8706 2596 Acked 3595 Write 5 Done Done New false
8707 2597 Acked 3596 Read 8 Done Done New false
8708 2598 Acked 3597 Read 6 Done Done New false
8709 2599 Acked 3598 Write 3 Done Done New false
8710 2600 Acked 3599 Read 7 Done Done New false
8711 2601 Acked 3600 Write 10 Done Done New false
8712 2602 Acked 3601 Write 6 Done Done New false
8713 2603 Acked 3602 Read 2 Done Done New false
8714 2604 Acked 3603 Flush 0 Done Done New false
8715 2605 Acked 3604 Read 8 Done Done New false
8716 2606 Acked 3605 Read 9 Done Done New false
8717 2607 Acked 3606 Write 2 Done Done New false
8718 2608 Acked 3607 Write 8 Done Done New false
8719 2609 Acked 3608 Write 6 Done Done New false
8720 2610 Acked 3609 Read 2 Done Done New false
8721 2611 Acked 3610 Read 1 Done Done New false
8722 2612 Acked 3611 Read 3 Done Done New false
8723 2613 Acked 3612 Write 8 Done Done New false
8724 2614 Acked 3613 Read 7 Done Done New false
8725 2615 Acked 3614 Read 9 Done Done New false
8726 2616 Acked 3615 Write 8 Done Done New false
8727 2617 Acked 3616 Read 6 Done Done New false
8728 2618 Acked 3617 Read 2 Done Done New false
8729 2619 Acked 3618 Write 3 Done Done New false
8730 2620 Acked 3619 Write 4 Done Done New false
8731 2621 Acked 3620 Write 3 Done Done New false
8732 2622 Acked 3621 Read 2 Done Done New false
8733 2623 Acked 3622 Read 3 Done Done New false
8734 2624 Acked 3623 Read 1 Done Done New false
8735 2625 Acked 3624 Read 2 Done Done New false
8736 2626 Acked 3625 Flush 0 Done Done New false
8737 2627 Acked 3626 Flush 0 Done Done New false
8738 2628 Acked 3627 Write 2 Done Done New false
8739 2629 Acked 3628 Read 8 Done Done New false
8740 2630 Acked 3629 Read 10 Done Done New false
8741 2631 Acked 3630 Write 1 Done Done New false
8742 2632 Acked 3631 Read 9 Done Done New false
8743 2633 Acked 3632 Write 9 Done Done New false
8744 2634 Acked 3633 Write 3 Done Done New false
8745 2635 Acked 3634 Flush 0 Done Done New false
8746 2636 Acked 3635 Read 9 Done Done New false
8747 2637 Acked 3636 Read 1 Done Done New false
8748 2638 Acked 3637 Write 5 Done Done New false
8749 2639 Acked 3638 Write 7 Done Done New false
8750 2640 Acked 3639 Write 5 Done Done New false
8751 2641 Acked 3640 Read 10 Done Done New false
8752 2642 Acked 3641 Write 9 Done Done New false
8753 2643 Acked 3642 Read 5 Done Done New false
8754 2644 Acked 3643 Read 10 Done Done New false
8755 2645 Acked 3644 Read 1 Done Done New false
8756 2646 Acked 3645 Write 2 Done Done New false
8757 2647 Acked 3646 Read 6 Done Done New false
8758 2648 Acked 3647 Read 2 Done Done New false
8759 2649 Acked 3648 Write 10 Done Done New false
8760 2650 Acked 3649 Write 10 Done Done New false
8761 2651 Acked 3650 Write 1 Done Done New false
8762 2652 Acked 3651 Read 1 Done Done New false
8763 2653 Acked 3652 Read 3 Done Done New false
8764 2654 Acked 3653 Flush 0 Done Done New false
8765 2655 Acked 3654 Write 5 Done Done New false
8766 2656 Acked 3655 Read 10 Done Done New false
8767 2657 Acked 3656 Read 5 Done Done New false
8768 2658 Acked 3657 Read 8 Done Done New false
8769 2659 Acked 3658 Read 1 Done Done New false
8770 2660 Acked 3659 Read 9 Done Done New false
8771 2661 Acked 3660 Flush 0 Done Done New false
8772 2662 Acked 3661 Flush 0 Done Done New false
8773 2663 Acked 3662 Read 10 Done Done New false
8774 2664 Acked 3663 Write 7 Done Done New false
8775 2665 Acked 3664 Read 7 Done Done New false
8776 2666 Acked 3665 Write 5 Done Done New false
8777 2667 Acked 3666 Write 1 Done Done New false
8778 2668 Acked 3667 Read 7 Done Done New false
8779 2669 Acked 3668 Read 4 Done Done New false
8780 2670 Acked 3669 Write 4 Done Done New false
8781 2671 Acked 3670 Read 3 Done Done New false
8782 2672 Acked 3671 Flush 0 Done Done New false
8783 2673 Acked 3672 Read 1 Done Done New false
8784 2674 Acked 3673 Read 8 Done Done New false
8785 2675 Acked 3674 Read 3 Done Done New false
8786 2676 Acked 3675 Write 4 Done Done New false
8787 2677 Acked 3676 Write 1 Done Done New false
8788 2678 Acked 3677 Flush 0 Done Done New false
8789 2679 Acked 3678 Write 2 Done Done New false
8790 2680 Acked 3679 Read 8 Done Done New false
8791 2681 Acked 3680 Read 2 Done Done New false
8792 2682 Acked 3681 Flush 0 Done Done New false
8793 2683 Acked 3682 Write 2 Done Done New false
8794 2684 Acked 3683 Read 3 Done Done New false
8795 2685 Acked 3684 Write 1 Done Done New false
8796 2686 Acked 3685 Read 3 Done Done New false
8797 2687 Acked 3686 Read 7 Done Done New false
8798 2688 Acked 3687 Read 3 Done Done New false
8799 2689 Acked 3688 Read 1 Done Done New false
8800 2690 Acked 3689 Read 5 Done Done New false
8801 2691 Acked 3690 Write 2 Done Done New false
8802 2692 Acked 3691 Flush 0 Done Done New false
8803 2693 Acked 3692 Write 5 Done Done New false
8804 2694 Acked 3693 Read 2 Done Done New false
8805 2695 Acked 3694 Write 7 Done Done New false
8806 2696 Acked 3695 Read 6 Done Done New false
8807 2697 Acked 3696 Flush 0 Done Done New false
8808 2698 Acked 3697 Write 9 Done Done New false
8809 2699 Acked 3698 Read 5 Done Done New false
8810 2700 Acked 3699 Write 8 Done Done New false
8811 2701 Acked 3700 Read 2 Done Done New false
8812 2702 Acked 3701 Write 9 Done Done New false
8813 2703 Acked 3702 Read 5 Done Done New false
8814 2704 Acked 3703 Read 3 Done Done New false
8815 2705 Acked 3704 Read 6 Done Done New false
8816 2706 Acked 3705 Read 1 Done Done New false
8817 2707 Acked 3706 Write 2 Done Done New false
8818 2708 Acked 3707 Read 3 Done Done New false
8819 2709 Acked 3708 Read 1 Done Done New false
8820 2710 Acked 3709 Write 1 Done Done New false
8821 2711 Acked 3710 Read 8 Done Done New false
8822 STATES DS:0 DS:1 DS:2 TOTAL
8823 New 0 0 301 301
8824 Sent 0 0 0 0
8825 Done 301 301 0 602
8826 Skipped 0 0 0 0
8827 Error 0 0 0 0
8828 Last Flush: 3696 3696 3409
8829 Downstairs last five completed: 3409 3408 3407 3406 3405
8830 Upstairs last five completed: 2711 2710 2709 2708 2707
8831 CLIENT: Up:0 ds:301 act:2
88322024-02-27T21:46:21.816ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
88332024-02-27T21:46:21.816ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
88342024-02-27T21:46:21.816ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
88352024-02-27T21:46:21.816ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
88362024-02-27T21:46:21.816ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
88372024-02-27T21:46:21.816ZINFOcrucible: [2] client re-new 302 jobs since flush 3409 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
88382024-02-27T21:46:21.816ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
88392024-02-27T21:46:22.817ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
88402024-02-27T21:46:22.817ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
88412024-02-27T21:46:22.817ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
88422024-02-27T21:46:22.817ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
88432024-02-27T21:46:22.817ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
88442024-02-27T21:46:22.817ZINFOcrucible: [2] client re-new 302 jobs since flush 3409 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
88452024-02-27T21:46:22.817ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
8846 [8830] got start action
8847 [8830] I am going to restart
8848 Starting downstairs at port 8830
8849 Make output file at "/tmp/dsc/downstairs-8830.txt"
8850 [8830][2] reports Starting
8851 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1176)
8852 [8830][2] reports Running
88532024-02-27T21:46:23.818ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
88542024-02-27T21:46:23.818ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
88552024-02-27T21:46:23.820ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
88562024-02-27T21:46:23.820ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
88572024-02-27T21:46:23.820ZINFOcrucible: send last flush ID to this DS: 3409 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
88582024-02-27T21:46:23.820ZINFOcrucible: Replied this last flush ID: 3409 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
88592024-02-27T21:46:23.820ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
88602024-02-27T21:46:23.820ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
88612024-02-27T21:46:23.820ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
88622024-02-27T21:46:23.820ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
8863 ----------------------------------------------------------------
8864 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
8865 Downstairs last five completed: 3711 3710 3709 3708 3707
8866 Upstairs last five completed: 2712 2711 2710 2709 2708
8867 CLIENT: Up:0 ds:0 act:3
8868 Replay: All jobs finished, all DS active.
8869 Main task has work to do, go find it
8870 got dsc Stop(1)
8871 stop 1
8872 Feb 27 21:46:25.150 INFO request completed, latency_us: 332, response_code: 204, uri: /stop/cid/1, method: POST, req_id: f3c9617e-83b6-4380-93f6-53c3e60d2127, remote_addr: 127.0.0.1:44347, local_addr[8820] Got stop action so:false kr:false
8873 : 127.0.0.1:9998
8874 Feb 27 21:46:25.151 {"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-27T21:46:25.151307751Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"io task","client":"1","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
8875 INFO request completed, latency_us: 101, response_code: 200, uri: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-27T21:46:25.151361814Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"io task","client":"1","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
8876 /state/cid/1, method: GET, req_id: 65295da3-bfb5-45dc-a61e-0e8a637689b2, remote_addr:{"msg":"downstairs task for 1 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-27T21:46:25.151399169Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
8877 127.0.0.1:44347, {"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:25.151434955Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"client":"1","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
8878 local_addr: 127.0.0.1:9998
88792024-02-27T21:46:25.151ZINFOcrucible: [1] client re-new 0 jobs since flush 3711 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
88802024-02-27T21:46:25.151ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
8881 [8820] Exited with: ExitStatus(unix_wait_status(9))
8882 [8820][1] reports Exit
88832024-02-27T21:46:26.152ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
88842024-02-27T21:46:26.152ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
88852024-02-27T21:46:26.152ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
88862024-02-27T21:46:26.152ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
88872024-02-27T21:46:26.152ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
88882024-02-27T21:46:26.152ZINFOcrucible: [1] client re-new 0 jobs since flush 3711 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
88892024-02-27T21:46:26.152ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
88902024-02-27T21:46:27.154ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
88912024-02-27T21:46:27.154ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
88922024-02-27T21:46:27.154ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
88932024-02-27T21:46:27.154ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
88942024-02-27T21:46:27.154ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
88952024-02-27T21:46:27.154ZINFOcrucible: [1] client re-new 0 jobs since flush 3711 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
88962024-02-27T21:46:27.154ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
88972024-02-27T21:46:28.155ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
88982024-02-27T21:46:28.155ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
88992024-02-27T21:46:28.155ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
89002024-02-27T21:46:28.155ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
89012024-02-27T21:46:28.155ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
89022024-02-27T21:46:28.155ZINFOcrucible: [1] client re-new 0 jobs since flush 3711 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
89032024-02-27T21:46:28.155ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
8904 Feb 27 21:46:29.153 INFO request completed, latency_us: 293, response_code: 200, uri: /state/cid/1, method: GET, req_id: 13b94668-7703-4f5d-be6f-9d14f4b3441a, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
8905 001/300 Write block 2963 len 24576 data: 3 2 2 2 2 2
8906 002/300 Read block 3580 len 32768
8907 003/300 Read block 4223 len 32768
89082024-02-27T21:46:29.156ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
89092024-02-27T21:46:29.156ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
89102024-02-27T21:46:29.156ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
89112024-02-27T21:46:29.156ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
89122024-02-27T21:46:29.156ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
89132024-02-27T21:46:29.156ZINFOcrucible: [1] client re-new 3 jobs since flush 3711 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
89142024-02-27T21:46:29.156ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
8915 004/300 Read block 88 len 36864
8916 005/300 Read block 113 len 16384
8917 006/300 Write block 1270 len 4096 data: 4
8918 007/300 Read block 1771 len 4096
8919 008/300 Flush
8920 009/300 Write block 1811 len 32768 data: 3 3 3 3 3 2 2 2
8921 010/300 Read block 3190 len 12288
8922 011/300 Write block 1321 len 4096 data: 5
8923 012/300 Read block 537 len 40960
8924 013/300 Write block 807 len 28672 data: 2 2 2 2 3 3 2
8925 014/300 Write block 1669 len 20480 data: 5 5 4 3 3
8926 015/300 Read block 3465 len 8192
8927 016/300 Read block 2764 len 28672
8928 017/300 Read block 4695 len 32768
8929 018/300 Read block 653 len 12288
8930 019/300 Read block 3977 len 32768
8931 020/300 Read block 506 len 24576
8932 021/300 Read block 1182 len 4096
8933 022/300 Write block 3963 len 32768 data: 4 5 5 3 3 3 3 3
8934 023/300 Write block 4588 len 40960 data: 3 2 2 2 2 2 2 2 3 3
8935 024/300 Write block 3588 len 8192 data: 3 3
8936 025/300 Write block 362 len 4096 data: 3
8937 026/300 Read block 1444 len 32768
8938 027/300 Read block 503 len 32768
8939 028/300 Flush
8940 029/300 Flush
8941 030/300 Write block 3225 len 8192 data: 2 2
8942 031/300 Write block 830 len 36864 data: 3 3 3 2 2 2 3 4 6
8943 032/300 Read block 2021 len 4096
8944 033/300 Flush
8945 034/300 Flush
8946 035/300 Read block 1092 len 24576
8947 036/300 Write block 1611 len 24576 data: 2 3 4 4 6 6
8948 037/300 Write block 4332 len 32768 data: 2 2 2 2 2 2 2 2
8949 038/300 Read block 1253 len 28672
8950 039/300 Read block 3189 len 40960
8951 040/300 Read block 2220 len 40960
8952 041/300 Read block 3512 len 12288
8953 042/300 Write block 4617 len 16384 data: 3 2 3 3
8954 043/300 Write block 2589 len 20480 data: 3 4 3 4 5
8955 044/300 Read block 2819 len 32768
8956 045/300 Write block 1927 len 40960 data: 4 5 5 5 4 3 3 3 3 2
8957 046/300 Write block 4595 len 40960 data: 3 4 4 3 3 2 2 2 2 2
8958 047/300 Read block 1703 len 32768
8959 048/300 Write block 3214 len 40960 data: 2 2 2 2 2 3 3 3 3 3
8960 049/300 Read block 3559 len 12288
8961 050/300 Write block 2734 len 36864 data: 3 3 3 3 2 2 2 2 2
8962 051/300 Write block 2594 len 4096 data: 4
8963 052/300 Read block 1407 len 12288
8964 053/300 Write block 2575 len 24576 data: 4 4 4 3 3 2
8965 054/300 Read block 4948 len 8192
8966 055/300 Read block 1746 len 8192
8967 056/300 Read block 709 len 40960
8968 057/300 Read block 3467 len 12288
8969 058/300 Write block 4401 len 36864 data: 4 4 2 2 2 3 3 3 3
8970 059/300 Write block 3094 len 24576 data: 2 2 2 2 2 3
8971 060/300 Write block 4095 len 40960 data: 3 3 3 3 3 3 2 2 3 3
8972 061/300 Read block 3641 len 28672
8973 062/300 Write block 2323 len 12288 data: 5 6 5
8974 063/300 Write block 3159 len 32768 data: 3 2 2 3 3 4 4 3
8975 064/300 Write block 2935 len 4096 data: 2
8976 065/300 Write block 1505 len 32768 data: 4 4 4 4 6 6 5 5
8977 066/300 Read block 1485 len 16384
8978 067/300 Write block 4297 len 16384 data: 4 4 4 4
8979 068/300 Read block 3430 len 16384
8980 069/300 Read block 3578 len 20480
8981 070/300 Read block 1120 len 24576
8982 071/300 Read block 1293 len 16384
8983 072/300 Write block 3149 len 20480 data: 2 2 2 2 3
8984 073/300 Write block 3056 len 36864 data: 4 4 4 4 4 3 3 4 4
8985 074/300 Read block 4520 len 32768
8986 075/300 Write block 1018 len 12288 data: 4 4 3
8987 076/300 Write block 2030 len 20480 data: 4 3 2 2 2
8988 077/300 Write block 3818 len 12288 data: 4 4 4
8989 078/300 Write block 2277 len 36864 data: 4 4 4 4 3 3 2 3 3
8990 079/300 Write block 4720 len 20480 data: 3 3 2 2 2
8991 080/300 Read block 4563 len 40960
8992 081/300 Write block 1800 len 8192 data: 2 2
8993 082/300 Read block 3637 len 32768
8994 083/300 Read block 3308 len 4096
8995 084/300 Write block 3254 len 24576 data: 3 4 4 5 5 4
8996 085/300 Read block 1599 len 4096
8997 086/300 Write block 3979 len 4096 data: 5
8998 087/300 Write block 3606 len 28672 data: 3 3 3 4 4 4 3
8999 088/300 Write block 1419 len 36864 data: 3 3 3 2 2 2 2 2 2
9000 089/300 Flush
9001 090/300 Write block 3326 len 12288 data: 2 3 3
9002 091/300 Write block 485 len 16384 data: 2 2 3 3
9003 092/300 Write block 4487 len 40960 data: 5 5 5 7 5 5 5 4 4 4
9004 093/300 Read block 1492 len 16384
9005 094/300 Flush
9006 095/300 Read block 3229 len 32768
9007 096/300 Read block 1257 len 28672
9008 097/300 Write block 3993 len 40960 data: 2 2 2 2 2 2 2 2 2 2
9009 098/300 Read block 533 len 28672
9010 099/300 Read block 2226 len 36864
9011 100/300 Read block 4974 len 28672
9012 101/300 Read block 708 len 40960
9013 102/300 Flush
9014 103/300 Read block 3570 len 12288
9015 104/300 Read block 4309 len 20480
9016 105/300 Write block 4969 len 40960 data: 4 5 4 4 4 4 4 2 2 3
9017 106/300 Read block 28 len 28672
9018 107/300 Read block 4309 len 28672
9019 108/300 Read block 1933 len 4096
9020 109/300 Read block 3741 len 4096
9021 110/300 Write block 1099 len 24576 data: 3 4 3 2 2 2
9022 111/300 Read block 906 len 8192
9023 112/300 Write block 4564 len 8192 data: 4 4
9024 113/300 Read block 3501 len 8192
9025 114/300 Write block 4028 len 20480 data: 4 4 4 4 3
9026 115/300 Write block 3878 len 24576 data: 3 3 3 2 2 2
9027 116/300 Read block 2349 len 24576
9028 117/300 Read block 3506 len 16384
9029 118/300 Write block 983 len 4096 data: 3
9030 119/300 Read block 197 len 8192
9031 120/300 Flush
9032 121/300 Read block 3953 len 20480
9033 122/300 Write block 3109 len 28672 data: 4 5 5 4 4 5 6
9034 123/300 Read block 2434 len 8192
9035 124/300 Read block 1943 len 12288
9036 125/300 Read block 3483 len 4096
9037 126/300 Read block 3331 len 16384
9038 127/300 Write block 468 len 40960 data: 2 3 3 3 3 3 3 2 2 2
9039 128/300 Read block 2548 len 40960
9040 129/300 Write block 259 len 8192 data: 4 4
9041 130/300 Read block 67 len 20480
9042 131/300 Read block 4627 len 28672
9043 132/300 Write block 497 len 32768 data: 5 5 6 6 5 5 6 3
9044 133/300 Write block 3024 len 16384 data: 7 5 5 5
9045 134/300 Read block 4252 len 12288
9046 135/300 Flush
9047 136/300 Write block 824 len 8192 data: 2 2
9048 137/300 Write block 1228 len 40960 data: 4 4 4 4 3 3 3 3 3 3
9049 138/300 Read block 2589 len 12288
9050 139/300 Write block 4024 len 8192 data: 4 5
9051 140/300 Read block 4395 len 28672
9052 141/300 Write block 987 len 16384 data: 3 4 4 4
9053 142/300 Write block 2189 len 12288 data: 2 2 4
9054 143/300 Flush
9055 144/300 Read block 4333 len 32768
9056 145/300 Read block 2817 len 28672
9057 146/300 Read block 4344 len 28672
9058 147/300 Read block 3915 len 16384
9059 148/300 Write block 2668 len 24576 data: 4 4 4 4 4 3
9060 149/300 Read block 3340 len 32768
9061 150/300 Read block 2203 len 16384
9062 151/300 Read block 3413 len 36864
9063 152/300 Read block 463 len 8192
9064 153/300 Write block 1021 len 8192 data: 3 3
9065 154/300 Write block 782 len 4096 data: 3
9066 155/300 Read block 593 len 4096
9067 156/300 Flush
9068 157/300 Write block 591 len 16384 data: 3 2 2 3
9069 158/300 Write block 2869 len 36864 data: 2 3 3 3 2 2 2 2 2
9070 159/300 Read block 3049 len 28672
9071 160/300 Read block 4073 len 32768
9072 161/300 Read block 4298 len 8192
9073 162/300 Read block 1127 len 8192
9074 163/300 Write block 4218 len 16384 data: 2 2 2 3
9075 164/300 Write block 607 len 32768 data: 3 3 3 3 3 3 4 4
9076 165/300 Read block 2639 len 24576
9077 166/300 Flush
9078 167/300 Write block 4867 len 28672 data: 3 3 2 3 3 3 3
9079 168/300 Read block 4367 len 36864
9080 169/300 Write block 2488 len 36864 data: 4 4 4 4 4 3 3 3 2
9081 170/300 Flush
9082 171/300 Read block 1878 len 4096
9083 172/300 Write block 4430 len 20480 data: 4 4 4 4 3
9084 173/300 Read block 4439 len 12288
9085 174/300 Write block 1976 len 12288 data: 5 5 5
9086 175/300 Write block 1968 len 16384 data: 2 2 2 2
9087 176/300 Write block 2914 len 8192 data: 3 2
9088 177/300 Read block 1775 len 28672
9089 178/300 Write block 3945 len 8192 data: 5 5
9090 179/300 Write block 1456 len 12288 data: 3 3 4
9091 180/300 Write block 3651 len 8192 data: 2 3
9092 181/300 Read block 1694 len 28672
9093 182/300 Read block 2922 len 32768
9094 183/300 Write block 2866 len 36864 data: 2 2 2 3 4 4 4 3 3
9095 184/300 Read block 3313 len 12288
9096 185/300 Write block 3485 len 32768 data: 3 5 5 4 4 4 4 5
9097 186/300 Write block 2792 len 36864 data: 3 3 3 2 3 3 3 3 3
9098 187/300 Flush
9099 188/300 Write block 3410 len 28672 data: 5 4 4 4 3 3 3
9100 189/300 Read block 67 len 32768
9101 190/300 Read block 274 len 20480
9102 191/300 Read block 2720 len 8192
9103 192/300 Write block 4128 len 4096 data: 4
9104 193/300 Flush
9105 194/300 Read block 807 len 12288
9106 195/300 Write block 2845 len 12288 data: 3 3 3
9107 196/300 Write block 2065 len 12288 data: 2 2 2
9108 197/300 Write block 4423 len 12288 data: 2 2 2
9109 198/300 Flush
9110 199/300 Read block 1930 len 32768
9111 200/300 Read block 1618 len 4096
9112 201/300 Write block 1694 len 12288 data: 2 2 2
9113 202/300 Flush
9114 203/300 Write block 2807 len 40960 data: 2 2 3 3 2 2 3 3 4 4
9115 204/300 Write block 4476 len 24576 data: 3 3 3 3 3 3
9116 205/300 Write block 4219 len 32768 data: 3 3 4 3 3 2 2 3
9117 206/300 Write block 1511 len 12288 data: 6 6 4
9118 207/300 Write block 887 len 36864 data: 3 2 2 2 2 2 2 2 2
9119 208/300 Read block 1687 len 16384
9120 209/300 Write block 510 len 16384 data: 5 5 5 4
9121 210/300 Write block 2973 len 24576 data: 3 3 3 3 3 4
9122 211/300 Read block 1803 len 24576
9123 212/300 Write block 2954 len 24576 data: 2 3 3 3 4 3
9124 213/300 Read block 315 len 40960
9125 214/300 Read block 4833 len 36864
9126 215/300 Write block 3891 len 4096 data: 4
9127 216/300 Read block 1958 len 16384
9128 217/300 Read block 2420 len 12288
9129 218/300 Read block 54 len 4096
9130 219/300 Write block 2337 len 36864 data: 6 5 6 5 5 4 4 4 4
9131 220/300 Read block 2470 len 40960
9132 221/300 Read block 2833 len 28672
9133 222/300 Read block 2351 len 32768
9134 223/300 Write block 2570 len 28672 data: 5 5 5 4 4 5 5
9135 224/300 Write block 1647 len 4096 data: 5
9136 225/300 Read block 4452 len 36864
9137 226/300 Write block 1737 len 20480 data: 3 4 4 4 3
9138 227/300 Read block 3304 len 40960
9139 228/300 Read block 2884 len 24576
9140 229/300 Write block 1673 len 28672 data: 4 3 3 2 2 3 3
9141 230/300 Write block 3953 len 12288 data: 5 5 4
9142 231/300 Write block 1114 len 12288 data: 3 3 3
9143 232/300 Write block 1404 len 36864 data: 3 3 4 4 4 3 3 3 2
9144 233/300 Flush
9145 234/300 Read block 2250 len 24576
9146 235/300 Read block 3166 len 36864
9147 236/300 Flush
9148 237/300 Read block 2061 len 8192
9149 238/300 Write block 4874 len 12288 data: 3 3 3
9150 239/300 Read block 4526 len 24576
9151 240/300 Write block 1294 len 12288 data: 3 3 3
9152 241/300 Write block 1436 len 4096 data: 2
9153 242/300 Write block 1543 len 36864 data: 2 3 3 4 4 4 4 4 4
9154 243/300 Flush
9155 244/300 Write block 203 len 40960 data: 3 3 3 4 4 4 3 2 2 2
9156 245/300 Read block 3956 len 36864
9157 246/300 Write block 1932 len 4096 data: 4
9158 247/300 Write block 3180 len 12288 data: 3 5 4
9159 248/300 Read block 3487 len 36864
9160 249/300 Read block 4072 len 24576
9161 250/300 Read block 1413 len 40960
9162 251/300 Read block 3964 len 16384
9163 252/300 Read block 4605 len 8192
9164 253/300 Read block 834 len 36864
9165 254/300 Read block 2356 len 4096
9166 255/300 Read block 1611 len 40960
9167 256/300 Read block 3145 len 20480
9168 257/300 Read block 3139 len 24576
9169 258/300 Flush
9170 259/300 Write block 2646 len 40960 data: 7 7 6 6 5 4 2 2 2 3
9171 260/300 Read block 2966 len 32768
9172 261/300 Read block 1041 len 32768
9173 262/300 Read block 1504 len 28672
9174 263/300 Read block 1947 len 16384
9175 264/300 Read block 4611 len 8192
9176 265/300 Read block 4416 len 40960
9177 266/300 Write block 3679 len 40960 data: 7 5 4 5 5 3 3 4 4 3
9178 267/300 Read block 3022 len 16384
9179 268/300 Write block 2519 len 32768 data: 4 4 4 4 4 3 3 2
9180 269/300 Write block 933 len 12288 data: 3 3 3
9181 270/300 Write block 2587 len 12288 data: 3 3 4
9182 271/300 Write block 4444 len 40960 data: 5 5 3 3 2 4 4 5 5 5
9183 272/300 Write block 4207 len 4096 data: 3
9184 273/300 Write block 3344 len 28672 data: 3 5 4 4 3 3 3
9185 274/300 Write block 4181 len 32768 data: 2 2 2 2 3 3 3 3
9186 275/300 Read block 3538 len 28672
9187 276/300 Flush
9188 277/300 Read block 3103 len 16384
9189 278/300 Flush
9190 279/300 Write block 2678 len 20480 data: 2 2 2 2 2
9191 280/300 Read block 4925 len 20480
9192 281/300 Read block 2156 len 32768
9193 282/300 Write block 2673 len 40960 data: 4 2 2 2 2 3 3 3 3 3
9194 283/300 Write block 4397 len 16384 data: 5 5 5 5
9195 284/300 Write block 1113 len 36864 data: 3 4 4 4 3 4 3 3 3
9196 285/300 Read block 2875 len 12288
9197 286/300 Read block 2768 len 4096
9198 287/300 Read block 2065 len 24576
9199 288/300 Read block 300 len 28672
9200 289/300 Write block 4526 len 28672 data: 2 2 2 2 2 2 2
9201 290/300 Write block 4542 len 12288 data: 3 4 4
9202 291/300 Read block 719 len 36864
9203 292/300 Read block 873 len 32768
9204 293/300 Write block 1837 len 28672 data: 3 3 4 4 4 3 3
9205 294/300 Write block 4748 len 28672 data: 6 7 6 3 3 3 3
9206 295/300 Read block 1008 len 8192
9207 296/300 Write block 218 len 12288 data: 3 2 2
9208 297/300 Read block 1582 len 12288
9209 298/300 Read block 3765 len 36864
9210 299/300 Read block 3889 len 12288
9211 300/300 Read block 1263 len 36864
9212 Main task has work to do, go find it
9213 got dsc Start(1)
9214 start 1
9215 Feb 27 21:46:29.515 INFO request completed, latency_us: 221, response_code: 204, uri: /start/cid/1, method: POST, req_id: acca5399-6f52-4946-8770-3ee493990e24, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
9216 Replay: started 1, returned:Ok(())
9217 ----------------------------------------------------------------
9218 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
9219 States: Active Offline Active
9220 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
9221 2713 Acked 3712 Write 6 Done New Done true
9222 2714 Acked 3713 Read 8 Done New Done true
9223 2715 Acked 3714 Read 8 Done New Done true
9224 2716 Acked 3715 Read 9 Done New Done false
9225 2717 Acked 3716 Read 4 Done New Done false
9226 2718 Acked 3717 Write 1 Done New Done false
9227 2719 Acked 3718 Read 1 Done New Done false
9228 2720 Acked 3719 Flush 0 Done New Done false
9229 2721 Acked 3720 Write 8 Done New Done false
9230 2722 Acked 3721 Read 3 Done New Done false
9231 2723 Acked 3722 Write 1 Done New Done false
9232 2724 Acked 3723 Read 10 Done New Done false
9233 2725 Acked 3724 Write 7 Done New Done false
9234 2726 Acked 3725 Write 5 Done New Done false
9235 2727 Acked 3726 Read 2 Done New Done false
9236 2728 Acked 3727 Read 7 Done New Done false
9237 2729 Acked 3728 Read 8 Done New Done false
9238 2730 Acked 3729 Read 3 Done New Done false
9239 2731 Acked 3730 Read 8 Done New Done false
9240 2732 Acked 3731 Read 6 Done New Done false
9241 2733 Acked 3732 Read 1 Done New Done false
9242 2734 Acked 3733 Write 8 Done New Done false
9243 2735 Acked 3734 Write 10 Done New Done false
9244 2736 Acked 3735 Write 2 Done New Done false
9245 2737 Acked 3736 Write 1 Done New Done false
9246 2738 Acked 3737 Read 8 Done New Done false
9247 2739 Acked 3738 Read 8 Done New Done false
9248 2740 Acked 3739 Flush 0 Done New Done false
9249 2741 Acked 3740 Flush 0 Done New Done false
9250 2742 Acked 3741 Write 2 Done New Done false
9251 2743 Acked 3742 Write 9 Done New Done false
9252 2744 Acked 3743 Read 1 Done New Done false
9253 2745 Acked 3744 Flush 0 Done New Done false
9254 2746 Acked 3745 Flush 0 Done New Done false
9255 2747 Acked 3746 Read 6 Done New Done false
9256 2748 Acked 3747 Write 6 Done New Done false
9257 2749 Acked 3748 Write 8 Done New Done false
9258 2750 Acked 3749 Read 7 Done New Done false
9259 2751 Acked 3750 Read 10 Done New Done false
9260 2752 Acked 3751 Read 10 Done New Done false
9261 2753 Acked 3752 Read 3 Done New Done false
9262 2754 Acked 3753 Write 4 Done New Done false
9263 2755 Acked 3754 Write 5 Done New Done false
9264 2756 Acked 3755 Read 8 Done New Done false
9265 2757 Acked 3756 Write 10 Done New Done false
9266 2758 Acked 3757 Write 10 Done New Done false
9267 2759 Acked 3758 Read 8 Done New Done false
9268 2760 Acked 3759 Write 10 Done New Done false
9269 2761 Acked 3760 Read 3 Done New Done false
9270 2762 Acked 3761 Write 9 Done New Done false
9271 2763 Acked 3762 Write 1 Done New Done false
9272 2764 Acked 3763 Read 3 Done New Done false
9273 2765 Acked 3764 Write 6 Done New Done false
9274 2766 Acked 3765 Read 2 Done New Done false
9275 2767 Acked 3766 Read 2 Done New Done false
9276 2768 Acked 3767 Read 10 Done New Done false
9277 2769 Acked 3768 Read 3 Done New Done false
9278 2770 Acked 3769 Write 9 Done New Done false
9279 2771 Acked 3770 Write 6 Done New Done false
9280 2772 Acked 3771 Write 10 Done New Done false
9281 2773 Acked 3772 Read 7 Done New Done false
9282 2774 Acked 3773 Write 3 Done New Done false
9283 2775 Acked 3774 Write 8 Done New Done false
9284 2776 Acked 3775 Write 1 Done New Done false
9285 2777 Acked 3776 Write 8 Done New Done false
9286 2778 Acked 3777 Read 4 Done New Done false
9287 2779 Acked 3778 Write 4 Done New Done false
9288 2780 Acked 3779 Read 4 Done New Done false
9289 2781 Acked 3780 Read 5 Done New Done false
9290 2782 Acked 3781 Read 6 Done New Done false
9291 2783 Acked 3782 Read 4 Done New Done false
9292 2784 Acked 3783 Write 5 Done New Done false
9293 2785 Acked 3784 Write 9 Done New Done false
9294 2786 Acked 3785 Read 8 Done New Done false
9295 2787 Acked 3786 Write 3 Done New Done false
9296 2788 Acked 3787 Write 5 Done New Done false
9297 2789 Acked 3788 Write 3 Done New Done false
9298 2790 Acked 3789 Write 9 Done New Done false
9299 2791 Acked 3790 Write 5 Done New Done false
9300 2792 Acked 3791 Read 10 Done New Done false
9301 2793 Acked 3792 Write 2 Done New Done false
9302 2794 Acked 3793 Read 8 Done New Done false
9303 2795 Acked 3794 Read 1 Done New Done false
9304 2796 Acked 3795 Write 6 Done New Done false
9305 2797 Acked 3796 Read 1 Done New Done false
9306 2798 Acked 3797 Write 1 Done New Done false
9307 2799 Acked 3798 Write 7 Done New Done false
9308 2800 Acked 3799 Write 9 Done New Done false
9309 2801 Acked 3800 Flush 0 Done New Done false
9310 2802 Acked 3801 Write 3 Done New Done false
9311 2803 Acked 3802 Write 4 Done New Done false
9312 2804 Acked 3803 Write 10 Done New Done false
9313 2805 Acked 3804 Read 4 Done New Done false
9314 2806 Acked 3805 Flush 0 Done New Done false
9315 2807 Acked 3806 Read 8 Done New Done false
9316 2808 Acked 3807 Read 7 Done New Done false
9317 2809 Acked 3808 Write 10 Done New Done false
9318 2810 Acked 3809 Read 7 Done New Done false
9319 2811 Acked 3810 Read 9 Done New Done false
9320 2812 Acked 3811 Read 7 Done New Done false
9321 2813 Acked 3812 Read 10 Done New Done false
9322 2814 Acked 3813 Flush 0 Done New Done false
9323 2815 Acked 3814 Read 3 Done New Done false
9324 2816 Acked 3815 Read 5 Done New Done false
9325 2817 Acked 3816 Write 10 Done New Done false
9326 2818 Acked 3817 Read 7 Done New Done false
9327 2819 Acked 3818 Read 7 Done New Done false
9328 2820 Acked 3819 Read 1 Done New Done false
9329 2821 Acked 3820 Read 1 Done New Done false
9330 2822 Acked 3821 Write 6 Done New Done false
9331 2823 Acked 3822 Read 2 Done New Done false
9332 2824 Acked 3823 Write 2 Done New Done false
9333 2825 Acked 3824 Read 2 Done New Done false
9334 2826 Acked 3825 Write 5 Done New Done false
9335 2827 Acked 3826 Write 6 Done New Done false
9336 2828 Acked 3827 Read 6 Done New Done false
9337 2829 Acked 3828 Read 4 Done New Done false
9338 2830 Acked 3829 Write 1 Done New Done false
9339 2831 Acked 3830 Read 2 Done New Done false
9340 2832 Acked 3831 Flush 0 Done New Done false
9341 2833 Acked 3832 Read 5 Done New Done false
9342 2834 Acked 3833 Write 7 Done New Done false
9343 2835 Acked 3834 Read 2 Done New Done false
9344 2836 Acked 3835 Read 3 Done New Done false
9345 2837 Acked 3836 Read 1 Done New Done false
9346 2838 Acked 3837 Read 4 Done New Done false
9347 2839 Acked 3838 Write 10 Done New Done false
9348 2840 Acked 3839 Read 10 Done New Done false
9349 2841 Acked 3840 Write 2 Done New Done false
9350 2842 Acked 3841 Read 5 Done New Done false
9351 2843 Acked 3842 Read 7 Done New Done false
9352 2844 Acked 3843 Write 8 Done New Done false
9353 2845 Acked 3844 Write 4 Done New Done false
9354 2846 Acked 3845 Read 3 Done New Done false
9355 2847 Acked 3846 Flush 0 Done New Done false
9356 2848 Acked 3847 Write 2 Done New Done false
9357 2849 Acked 3848 Write 10 Done New Done false
9358 2850 Acked 3849 Read 3 Done New Done false
9359 2851 Acked 3850 Write 2 Done New Done false
9360 2852 Acked 3851 Read 7 Done New Done false
9361 2853 Acked 3852 Write 4 Done New Done false
9362 2854 Acked 3853 Write 3 Done New Done false
9363 2855 Acked 3854 Flush 0 Done New Done false
9364 2856 Acked 3855 Read 8 Done New Done false
9365 2857 Acked 3856 Read 7 Done New Done false
9366 2858 Acked 3857 Read 7 Done New Done false
9367 2859 Acked 3858 Read 4 Done New Done false
9368 2860 Acked 3859 Write 6 Done New Done false
9369 2861 Acked 3860 Read 8 Done New Done false
9370 2862 Acked 3861 Read 4 Done New Done false
9371 2863 Acked 3862 Read 9 Done New Done false
9372 2864 Acked 3863 Read 2 Done New Done false
9373 2865 Acked 3864 Write 2 Done New Done false
9374 2866 Acked 3865 Write 1 Done New Done false
9375 2867 Acked 3866 Read 1 Done New Done false
9376 2868 Acked 3867 Flush 0 Done New Done false
9377 2869 Acked 3868 Write 4 Done New Done false
9378 2870 Acked 3869 Write 9 Done New Done false
9379 2871 Acked 3870 Read 7 Done New Done false
9380 2872 Acked 3871 Read 8 Done New Done false
9381 2873 Acked 3872 Read 2 Done New Done false
9382 2874 Acked 3873 Read 2 Done New Done false
9383 2875 Acked 3874 Write 4 Done New Done false
9384 2876 Acked 3875 Write 8 Done New Done false
9385 2877 Acked 3876 Read 6 Done New Done false
9386 2878 Acked 3877 Flush 0 Done New Done false
9387 2879 Acked 3878 Write 7 Done New Done false
9388 2880 Acked 3879 Read 9 Done New Done false
9389 2881 Acked 3880 Write 9 Done New Done false
9390 2882 Acked 3881 Flush 0 Done New Done false
9391 2883 Acked 3882 Read 1 Done New Done false
9392 2884 Acked 3883 Write 5 Done New Done false
9393 2885 Acked 3884 Read 3 Done New Done false
9394 2886 Acked 3885 Write 3 Done New Done false
9395 2887 Acked 3886 Write 4 Done New Done false
9396 2888 Acked 3887 Write 2 Done New Done false
9397 2889 Acked 3888 Read 7 Done New Done false
9398 2890 Acked 3889 Write 2 Done New Done false
9399 2891 Acked 3890 Write 3 Done New Done false
9400 2892 Acked 3891 Write 2 Done New Done false
9401 2893 Acked 3892 Read 7 Done New Done false
9402 2894 Acked 3893 Read 8 Done New Done false
9403 2895 Acked 3894 Write 9 Done New Done false
9404 2896 Acked 3895 Read 3 Done New Done false
9405 2897 Acked 3896 Write 8 Done New Done false
9406 2898 Acked 3897 Write 9 Done New Done false
9407 2899 Acked 3898 Flush 0 Done New Done false
9408 2900 Acked 3899 Write 7 Done New Done false
9409 2901 Acked 3900 Read 8 Done New Done false
9410 2902 Acked 3901 Read 5 Done New Done false
9411 2903 Acked 3902 Read 2 Done New Done false
9412 2904 Acked 3903 Write 1 Done New Done false
9413 2905 Acked 3904 Flush 0 Done New Done false
9414 2906 Acked 3905 Read 3 Done New Done false
9415 2907 Acked 3906 Write 3 Done New Done false
9416 2908 Acked 3907 Write 3 Done New Done false
9417 2909 Acked 3908 Write 3 Done New Done false
9418 2910 Acked 3909 Flush 0 Done New Done false
9419 2911 Acked 3910 Read 8 Done New Done false
9420 2912 Acked 3911 Read 1 Done New Done false
9421 2913 Acked 3912 Write 3 Done New Done false
9422 2914 Acked 3913 Flush 0 Done New Done false
9423 2915 Acked 3914 Write 10 Done New Done false
9424 2916 Acked 3915 Write 6 Done New Done false
9425 2917 Acked 3916 Write 8 Done New Done false
9426 2918 Acked 3917 Write 3 Done New Done false
9427 2919 Acked 3918 Write 9 Done New Done false
9428 2920 Acked 3919 Read 4 Done New Done false
9429 2921 Acked 3920 Write 4 Done New Done false
9430 2922 Acked 3921 Write 6 Done New Done false
9431 2923 Acked 3922 Read 6 Done New Done false
9432 2924 Acked 3923 Write 6 Done New Done false
9433 2925 Acked 3924 Read 10 Done New Done false
9434 2926 Acked 3925 Read 9 Done New Done false
9435 2927 Acked 3926 Write 1 Done New Done false
9436 2928 Acked 3927 Read 4 Done New Done false
9437 2929 Acked 3928 Read 3 Done New Done false
9438 2930 Acked 3929 Read 1 Done New Done false
9439 2931 Acked 3930 Write 9 Done New Done false
9440 2932 Acked 3931 Read 10 Done New Done false
9441 2933 Acked 3932 Read 7 Done New Done false
9442 2934 Acked 3933 Read 8 Done New Done false
9443 2935 Acked 3934 Write 7 Done New Done false
9444 2936 Acked 3935 Write 1 Done New Done false
9445 2937 Acked 3936 Read 9 Done New Done false
9446 2938 Acked 3937 Write 5 Done New Done false
9447 2939 Acked 3938 Read 10 Done New Done false
9448 2940 Acked 3939 Read 6 Done New Done false
9449 2941 Acked 3940 Write 7 Done New Done false
9450 2942 Acked 3941 Write 3 Done New Done false
9451 2943 Acked 3942 Write 3 Done New Done false
9452 2944 Acked 3943 Write 9 Done New Done false
9453 2945 Acked 3944 Flush 0 Done New Done false
9454 2946 Acked 3945 Read 6 Done New Done false
9455 2947 Acked 3946 Read 9 Done New Done false
9456 2948 Acked 3947 Flush 0 Done New Done false
9457 2949 Acked 3948 Read 2 Done New Done false
9458 2950 Acked 3949 Write 3 Done New Done false
9459 2951 Acked 3950 Read 6 Done New Done false
9460 2952 Acked 3951 Write 3 Done New Done false
9461 2953 Acked 3952 Write 1 Done New Done false
9462 2954 Acked 3953 Write 9 Done New Done false
9463 2955 Acked 3954 Flush 0 Done New Done false
9464 2956 Acked 3955 Write 10 Done New Done false
9465 2957 Acked 3956 Read 9 Done New Done false
9466 2958 Acked 3957 Write 1 Done New Done false
9467 2959 Acked 3958 Write 3 Done New Done false
9468 2960 Acked 3959 Read 9 Done New Done false
9469 2961 Acked 3960 Read 6 Done New Done false
9470 2962 Acked 3961 Read 10 Done New Done false
9471 2963 Acked 3962 Read 4 Done New Done false
9472 2964 Acked 3963 Read 2 Done New Done false
9473 2965 Acked 3964 Read 9 Done New Done false
9474 2966 Acked 3965 Read 1 Done New Done false
9475 2967 Acked 3966 Read 10 Done New Done false
9476 2968 Acked 3967 Read 5 Done New Done false
9477 2969 Acked 3968 Read 6 Done New Done false
9478 2970 Acked 3969 Flush 0 Done New Done false
9479 2971 Acked 3970 Write 10 Done New Done false
9480 2972 Acked 3971 Read 8 Done New Done false
9481 2973 Acked 3972 Read 8 Done New Done false
9482 2974 Acked 3973 Read 7 Done New Done false
9483 2975 Acked 3974 Read 4 Done New Done false
9484 2976 Acked 3975 Read 2 Done New Done false
9485 2977 Acked 3976 Read 10 Done New Done false
9486 2978 Acked 3977 Write 10 Done New Done false
9487 2979 Acked 3978 Read 4 Done New Done false
9488 2980 Acked 3979 Write 8 Done New Done false
9489 2981 Acked 3980 Write 3 Done New Done false
9490 2982 Acked 3981 Write 3 Done New Done false
9491 2983 Acked 3982 Write 10 Done New Done false
9492 2984 Acked 3983 Write 1 Done New Done false
9493 2985 Acked 3984 Write 7 Done New Done false
9494 2986 Acked 3985 Write 8 Done New Done false
9495 2987 Acked 3986 Read 7 Done New Done false
9496 2988 Acked 3987 Flush 0 Done New Done false
9497 2989 Acked 3988 Read 4 Done New Done false
9498 2990 Acked 3989 Flush 0 Done New Done false
9499 2991 Acked 3990 Write 5 Done New Done false
9500 2992 Acked 3991 Read 5 Done New Done false
9501 2993 Acked 3992 Read 8 Done New Done false
9502 2994 Acked 3993 Write 10 Done New Done false
9503 2995 Acked 3994 Write 4 Done New Done false
9504 2996 Acked 3995 Write 9 Done New Done false
9505 2997 Acked 3996 Read 3 Done New Done false
9506 2998 Acked 3997 Read 1 Done New Done false
9507 2999 Acked 3998 Read 6 Done New Done false
9508 3000 Acked 3999 Read 7 Done New Done false
9509 3001 Acked 4000 Write 7 Done New Done false
9510 3002 Acked 4001 Write 3 Done New Done false
9511 3003 Acked 4002 Read 9 Done New Done false
9512 3004 Acked 4003 Read 8 Done New Done false
9513 3005 Acked 4004 Write 7 Done New Done false
9514 3006 Acked 4005 Write 7 Done New Done false
9515 3007 Acked 4006 Read 2 Done New Done false
9516 3008 Acked 4007 Write 3 Done New Done false
9517 3009 Acked 4008 Read 3 Done New Done false
9518 3010 Acked 4009 Read 9 Done New Done false
9519 3011 Acked 4010 Read 3 Done New Done false
9520 3012 Acked 4011 Read 9 Done New Done false
9521 STATES DS:0 DS:1 DS:2 TOTAL
9522 New 0 300 0 300
9523 Sent 0 0 0 0
9524 Done 300 0 300 600
9525 Skipped 0 0 0 0
9526 Error 0 0 0 0
9527 Last Flush: 3989 3711 3989
9528 Downstairs last five completed: 3711 3710 3709 3708 3707
9529 Upstairs last five completed: 3012 3011 3010 3009 3008
9530 CLIENT: Up:0 ds:300 act:2
95312024-02-27T21:46:30.157ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
95322024-02-27T21:46:30.157ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
95332024-02-27T21:46:30.157ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
95342024-02-27T21:46:30.157ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
95352024-02-27T21:46:30.157ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
95362024-02-27T21:46:30.157ZINFOcrucible: [1] client re-new 301 jobs since flush 3711 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
95372024-02-27T21:46:30.157ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
95382024-02-27T21:46:31.158ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
95392024-02-27T21:46:31.158ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
95402024-02-27T21:46:31.158ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
95412024-02-27T21:46:31.158ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
95422024-02-27T21:46:31.158ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
95432024-02-27T21:46:31.158ZINFOcrucible: [1] client re-new 301 jobs since flush 3711 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
9544 [8820] got start action
9545 [8820] I am going to restart
9546 Starting downstairs at port 8820
95472024-02-27T21:46:31.158ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
9548 Make output file at "/tmp/dsc/downstairs-8820.txt"
9549 [8820][1] reports Starting
9550 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1177)
9551 [8820][1] reports Running
95522024-02-27T21:46:32.159ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
95532024-02-27T21:46:32.159ZINFOcrucible: ds_connection connected = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
95542024-02-27T21:46:32.160ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
95552024-02-27T21:46:32.160ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
95562024-02-27T21:46:32.161ZINFOcrucible: send last flush ID to this DS: 3711 = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
95572024-02-27T21:46:32.161ZINFOcrucible: Replied this last flush ID: 3711 = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
95582024-02-27T21:46:32.161ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
95592024-02-27T21:46:32.161ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
95602024-02-27T21:46:32.161ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
95612024-02-27T21:46:32.161ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
9562 ----------------------------------------------------------------
9563 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
9564 Downstairs last five completed: 4012 4011 4010 4009 4008
9565 Upstairs last five completed: 3013 3012 3011 3010 3009
9566 CLIENT: Up:0 ds:0 act:3
9567 Replay: All jobs finished, all DS active.
9568 Main task has work to do, go find it
9569 got dsc Stop(0)
9570 stop 0
9571 Feb 27 21:46:33.518 INFO request completed, [8810] Got stop action so:false kr:false
9572 latency_us: 367, response_code: 204, uri: /stop/cid/0, method: POST, req_id: b68bbb43-aa3c-4e33-bdb0-1260203ded05, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
95732024-02-27T21:46:33.519ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
95742024-02-27T21:46:33.519ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
95752024-02-27T21:46:33.519ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = e6a5a974-925a-494f-8845-facc441c2690
95762024-02-27T21:46:33.519ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
95772024-02-27T21:46:33.519ZINFOcrucible: [0] client re-new 0 jobs since flush 4012 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
9578 Feb 27 21:46:33.519{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:46:33.519648801Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"io task","client":"0","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
9579 INFO request completed, latency_us: 140, response_code: 200, uri: /state/cid/0, method: GET, req_id: 8dd3935d-cfef-47df-a737-920488ac0e55, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
9580 [8810] Exited with: ExitStatus(unix_wait_status(9))
9581 [8810][0] reports Exit
95822024-02-27T21:46:34.520ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
95832024-02-27T21:46:34.520ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
95842024-02-27T21:46:34.520ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
95852024-02-27T21:46:34.520ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
95862024-02-27T21:46:34.520ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
95872024-02-27T21:46:34.520ZINFOcrucible: [0] client re-new 0 jobs since flush 4012 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
95882024-02-27T21:46:34.520ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
95892024-02-27T21:46:35.521ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
95902024-02-27T21:46:35.521ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
95912024-02-27T21:46:35.521ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
95922024-02-27T21:46:35.521ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
95932024-02-27T21:46:35.521ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
95942024-02-27T21:46:35.521ZINFOcrucible: [0] client re-new 0 jobs since flush 4012 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
95952024-02-27T21:46:35.521ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
95962024-02-27T21:46:36.522ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
95972024-02-27T21:46:36.522ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
95982024-02-27T21:46:36.522ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
95992024-02-27T21:46:36.522ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
96002024-02-27T21:46:36.522ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
96012024-02-27T21:46:36.523ZINFOcrucible: [0] client re-new 0 jobs since flush 4012 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
96022024-02-27T21:46:36.523ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
9603 Feb 27 21:46:37.521 INFO request completed, latency_us: 202, response_code: 200, uri: /state/cid/0, method: GET, req_id: 022b49d9-2612-4662-831a-8846a48687be, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
9604 001/300 Flush
9605 002/300 Write block 735 len 36864 data: 4 4 4 4 4 3 3 3 2
9606 003/300 Read block 2436 len 16384
96072024-02-27T21:46:37.524ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
96082024-02-27T21:46:37.524ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
96092024-02-27T21:46:37.524ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
96102024-02-27T21:46:37.524ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
96112024-02-27T21:46:37.524ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
96122024-02-27T21:46:37.524ZINFOcrucible: [0] client re-new 3 jobs since flush 4012 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
96132024-02-27T21:46:37.524ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
9614 004/300 Read block 1372 len 28672
9615 005/300 Write block 4261 len 16384 data: 3 4 5 6
9616 006/300 Read block 1589 len 12288
9617 007/300 Write block 4390 len 20480 data: 3 3 3 3 4
9618 008/300 Write block 2948 len 24576 data: 3 3 3 2 2 2
9619 009/300 Write block 4479 len 20480 data: 4 4 4 2 3
9620 010/300 Read block 2697 len 40960
9621 011/300 Read block 4451 len 4096
9622 012/300 Write block 583 len 28672 data: 4 4 4 5 4 4 4
9623 013/300 Write block 1308 len 24576 data: 5 5 6 6 4 4
9624 014/300 Read block 1544 len 12288
9625 015/300 Read block 4732 len 12288
9626 016/300 Write block 2339 len 8192 data: 7 6
9627 017/300 Write block 1388 len 36864 data: 2 2 2 2 2 2 3 3 3
9628 018/300 Read block 206 len 28672
9629 019/300 Write block 500 len 16384 data: 7 6 6 7
9630 020/300 Write block 2661 len 40960 data: 4 4 3 3 5 4 4 5 5 5
9631 021/300 Write block 3099 len 16384 data: 4 2 2 3
9632 022/300 Write block 1772 len 4096 data: 3
9633 023/300 Write block 4211 len 40960 data: 2 2 2 2 2 2 2 3 4 4
9634 024/300 Read block 4053 len 28672
9635 025/300 Write block 4298 len 24576 data: 5 5 5 4 4 5
9636 026/300 Read block 889 len 4096
9637 027/300 Read block 230 len 24576
9638 028/300 Read block 433 len 4096
9639 029/300 Read block 4629 len 4096
9640 030/300 Write block 2047 len 12288 data: 4 5 5
9641 031/300 Write block 2532 len 24576 data: 3 3 3 3 4 5
9642 032/300 Read block 1281 len 28672
9643 033/300 Write block 1384 len 20480 data: 2 2 2 2 3
9644 034/300 Flush
9645 035/300 Read block 4014 len 16384
9646 036/300 Flush
9647 037/300 Write block 939 len 40960 data: 3 3 4 5 5 5 4 5 4 4
9648 038/300 Read block 3563 len 12288
9649 039/300 Read block 4757 len 36864
9650 040/300 Read block 1814 len 4096
9651 041/300 Read block 3799 len 20480
9652 042/300 Flush
9653 043/300 Read block 1775 len 4096
9654 044/300 Read block 4368 len 8192
9655 045/300 Flush
9656 046/300 Read block 4773 len 36864
9657 047/300 Flush
9658 048/300 Write block 618 len 16384 data: 3 3 2 2
9659 049/300 Write block 4173 len 12288 data: 2 2 2
9660 050/300 Read block 1072 len 20480
9661 051/300 Write block 1996 len 28672 data: 4 4 3 3 2 2 4
9662 052/300 Read block 4703 len 16384
9663 053/300 Write block 4665 len 16384 data: 4 4 3 3
9664 054/300 Flush
9665 055/300 Read block 1545 len 4096
9666 056/300 Read block 1 len 8192
9667 057/300 Read block 3202 len 36864
9668 058/300 Write block 4380 len 28672 data: 4 4 3 3 2 3 3
9669 059/300 Write block 4981 len 16384 data: 4 4 4 4
9670 060/300 Read block 3614 len 20480
9671 061/300 Flush
9672 062/300 Read block 225 len 8192
9673 063/300 Write block 1202 len 24576 data: 4 4 4 3 2 3
9674 064/300 Read block 4588 len 32768
9675 065/300 Write block 3537 len 32768 data: 3 3 3 3 3 4 4 4
9676 066/300 Read block 3667 len 24576
9677 067/300 Read block 1784 len 8192
9678 068/300 Read block 3979 len 20480
9679 069/300 Write block 1052 len 36864 data: 5 4 4 3 3 3 2 3 3
9680 070/300 Write block 2373 len 32768 data: 3 2 3 3 2 2 2 2
9681 071/300 Write block 1486 len 36864 data: 3 4 3 2 2 2 2 3 3
9682 072/300 Read block 4938 len 12288
9683 073/300 Read block 1486 len 4096
9684 074/300 Write block 4597 len 32768 data: 5 4 4 3 3 3 3 3
9685 075/300 Read block 2266 len 16384
9686 076/300 Read block 894 len 36864
9687 077/300 Read block 3930 len 36864
9688 078/300 Read block 2116 len 4096
9689 079/300 Flush
9690 080/300 Flush
9691 081/300 Read block 4777 len 12288
9692 082/300 Write block 660 len 16384 data: 2 2 2 2
9693 083/300 Write block 3712 len 12288 data: 2 2 2
9694 084/300 Write block 4082 len 16384 data: 2 2 2 2
9695 085/300 Write block 1390 len 28672 data: 3 3 3 3 4 4 4
9696 086/300 Read block 4481 len 40960
9697 087/300 Write block 4247 len 40960 data: 2 2 3 4 4 4 4 3 3 2
9698 088/300 Read block 1346 len 4096
9699 089/300 Write block 3492 len 12288 data: 6 5 5
9700 090/300 Read block 3979 len 32768
9701 091/300 Read block 3101 len 20480
9702 092/300 Read block 3575 len 8192
9703 093/300 Read block 4890 len 36864
9704 094/300 Read block 3111 len 24576
9705 095/300 Read block 230 len 12288
9706 096/300 Read block 4464 len 12288
9707 097/300 Read block 23 len 28672
9708 098/300 Write block 1403 len 32768 data: 3 4 4 5 5 5 4 4
9709 099/300 Read block 2004 len 36864
9710 100/300 Read block 106 len 32768
9711 101/300 Write block 4158 len 24576 data: 2 2 2 2 2 2
9712 102/300 Write block 4248 len 8192 data: 3 4
9713 103/300 Flush
9714 104/300 Read block 288 len 40960
9715 105/300 Read block 4413 len 24576
9716 106/300 Flush
9717 107/300 Flush
9718 108/300 Flush
9719 109/300 Flush
9720 110/300 Read block 1352 len 28672
9721 111/300 Flush
9722 112/300 Read block 3192 len 8192
9723 113/300 Read block 3941 len 16384
9724 114/300 Read block 2742 len 8192
9725 115/300 Write block 750 len 24576 data: 2 2 2 2 2 2
9726 116/300 Flush
9727 117/300 Flush
9728 118/300 Write block 392 len 40960 data: 3 3 4 4 4 4 3 3 4 4
9729 119/300 Write block 1712 len 28672 data: 7 7 5 5 5 5 5
9730 120/300 Read block 915 len 36864
9731 121/300 Flush
9732 122/300 Write block 2802 len 24576 data: 3 3 3 3 2 3
9733 123/300 Read block 1564 len 8192
9734 124/300 Read block 4 len 28672
9735 125/300 Write block 4511 len 40960 data: 3 3 2 2 2 2 2 2 2 2
9736 126/300 Read block 4226 len 32768
9737 127/300 Read block 1807 len 20480
9738 128/300 Flush
9739 129/300 Write block 3499 len 16384 data: 2 2 2 2
9740 130/300 Write block 4481 len 20480 data: 5 3 4 4 3
9741 131/300 Read block 4223 len 12288
9742 132/300 Write block 1545 len 28672 data: 4 5 5 5 5 5 5
9743 133/300 Read block 4833 len 28672
9744 134/300 Read block 1532 len 4096
9745 135/300 Read block 3557 len 8192
9746 136/300 Read block 1800 len 40960
9747 137/300 Write block 4891 len 16384 data: 2 3 3 3
9748 138/300 Write block 591 len 16384 data: 4 3 3 4
9749 139/300 Read block 3185 len 4096
9750 140/300 Write block 1110 len 8192 data: 2 3
9751 141/300 Write block 2871 len 36864 data: 5 5 4 4 3 3 3 3 3
9752 142/300 Read block 2285 len 36864
9753 143/300 Read block 4825 len 40960
9754 144/300 Write block 1634 len 36864 data: 3 3 3 3 4 5 4 4 4
9755 145/300 Read block 1890 len 28672
9756 146/300 Read block 2432 len 40960
9757 147/300 Read block 1276 len 40960
9758 148/300 Write block 1362 len 24576 data: 5 5 5 4 4 4
9759 149/300 Write block 3966 len 8192 data: 4 4
9760 150/300 Read block 711 len 24576
9761 151/300 Write block 843 len 32768 data: 4 4 4 3 3 3 3 2
9762 152/300 Read block 3454 len 36864
9763 153/300 Write block 3120 len 24576 data: 3 3 3 3 3 3
9764 154/300 Flush
9765 155/300 Read block 765 len 20480
9766 156/300 Read block 3252 len 4096
9767 157/300 Read block 909 len 4096
9768 158/300 Read block 1726 len 24576
9769 159/300 Read block 982 len 16384
9770 160/300 Write block 828 len 8192 data: 3 3
9771 161/300 Read block 3422 len 36864
9772 162/300 Flush
9773 163/300 Read block 465 len 20480
9774 164/300 Read block 2640 len 28672
9775 165/300 Read block 2687 len 4096
9776 166/300 Flush
9777 167/300 Read block 4311 len 24576
9778 168/300 Read block 3533 len 24576
9779 169/300 Read block 676 len 36864
9780 170/300 Read block 3805 len 36864
9781 171/300 Read block 309 len 40960
9782 172/300 Read block 2111 len 16384
9783 173/300 Write block 4549 len 8192 data: 4 5
9784 174/300 Read block 1328 len 40960
9785 175/300 Write block 2840 len 36864 data: 2 2 2 3 3 4 4 4 3
9786 176/300 Write block 3167 len 16384 data: 2 2 2 2
9787 177/300 Write block 1973 len 36864 data: 2 2 5 6 6 6 5 5 5
9788 178/300 Read block 4982 len 16384
9789 179/300 Write block 3188 len 8192 data: 2 2
9790 180/300 Read block 372 len 12288
9791 181/300 Read block 256 len 4096
9792 182/300 Write block 3659 len 32768 data: 3 4 3 4 4 3 4 4
9793 183/300 Write block 4660 len 36864 data: 4 4 4 4 4 5 5 4 4
9794 184/300 Write block 3683 len 32768 data: 6 4 4 5 5 4 4 3
9795 185/300 Flush
9796 186/300 Read block 3562 len 28672
9797 187/300 Read block 1122 len 36864
9798 188/300 Read block 4459 len 36864
9799 189/300 Write block 2300 len 8192 data: 2 2
9800 190/300 Flush
9801 191/300 Read block 4535 len 12288
9802 192/300 Write block 36 len 16384 data: 2 2 2 2
9803 193/300 Write block 4328 len 4096 data: 4
9804 194/300 Write block 633 len 12288 data: 2 2 2
9805 195/300 Flush
9806 196/300 Read block 711 len 4096
9807 197/300 Read block 4054 len 12288
9808 198/300 Write block 4625 len 40960 data: 4 3 3 4 5 5 4 4 4 3
9809 199/300 Flush
9810 200/300 Read block 2675 len 20480
9811 201/300 Write block 2483 len 16384 data: 3 3 4 4
9812 202/300 Flush
9813 203/300 Flush
9814 204/300 Read block 2504 len 16384
9815 205/300 Read block 4118 len 12288
9816 206/300 Read block 2736 len 28672
9817 207/300 Read block 4584 len 16384
9818 208/300 Write block 519 len 40960 data: 3 3 3 3 4 3 4 4 3 4
9819 209/300 Write block 3301 len 16384 data: 4 4 5 4
9820 210/300 Write block 3033 len 4096 data: 4
9821 211/300 Read block 4305 len 4096
9822 212/300 Write block 2257 len 20480 data: 4 4 4 4 4
9823 213/300 Write block 1438 len 4096 data: 2
9824 214/300 Read block 1179 len 40960
9825 215/300 Read block 3682 len 16384
9826 216/300 Read block 4061 len 28672
9827 217/300 Read block 975 len 8192
9828 218/300 Write block 3214 len 36864 data: 3 3 3 3 3 4 4 4 4
9829 219/300 Read block 3925 len 12288
9830 220/300 Read block 1505 len 4096
9831 221/300 Write block 1191 len 36864 data: 4 2 2 2 2 3 3 3 3
9832 222/300 Write block 191 len 32768 data: 3 4 4 4 4 4 4 4
9833 223/300 Read block 255 len 20480
9834 224/300 Read block 3600 len 24576
9835 225/300 Flush
9836 226/300 Write block 4406 len 20480 data: 4 4 4 4 3
9837 227/300 Read block 170 len 4096
9838 228/300 Write block 2654 len 40960 data: 3 4 3 3 3 3 3 5 5 4
9839 229/300 Read block 1793 len 20480
9840 230/300 Read block 2551 len 12288
9841 231/300 Write block 1392 len 32768 data: 4 4 5 5 5 3 4 3
9842 232/300 Read block 3876 len 32768
9843 233/300 Write block 275 len 32768 data: 3 3 3 2 2 3 3 3
9844 234/300 Read block 1780 len 40960
9845 235/300 Read block 1998 len 12288
9846 236/300 Write block 4989 len 20480 data: 2 2 3 3 2
9847 237/300 Read block 4490 len 24576
9848 238/300 Read block 4495 len 4096
9849 239/300 Read block 2105 len 32768
9850 240/300 Write block 2431 len 4096 data: 2
9851 241/300 Write block 4652 len 16384 data: 4 4 4 4
9852 242/300 Write block 2915 len 12288 data: 3 2 2
9853 243/300 Read block 2326 len 20480
9854 244/300 Read block 3748 len 24576
9855 245/300 Write block 2874 len 8192 data: 5 4
9856 246/300 Read block 134 len 12288
9857 247/300 Write block 4752 len 4096 data: 4
9858 248/300 Read block 642 len 28672
9859 249/300 Read block 1133 len 24576
9860 250/300 Read block 750 len 28672
9861 251/300 Read block 1289 len 24576
9862 252/300 Read block 1744 len 20480
9863 253/300 Write block 209 len 16384 data: 4 3 3 3
9864 254/300 Flush
9865 255/300 Read block 4065 len 24576
9866 256/300 Read block 4014 len 40960
9867 257/300 Flush
9868 258/300 Write block 398 len 12288 data: 4 4 5
9869 259/300 Write block 2442 len 28672 data: 3 3 3 3 3 3 3
9870 260/300 Read block 2693 len 24576
9871 261/300 Write block 4110 len 16384 data: 2 2 3 3
9872 262/300 Read block 3184 len 40960
9873 263/300 Read block 618 len 24576
9874 264/300 Write block 4431 len 32768 data: 5 5 5 4 5 6 6 5
9875 265/300 Read block 557 len 36864
9876 266/300 Read block 726 len 8192
9877 267/300 Write block 2568 len 20480 data: 3 4 6 6 6
9878 268/300 Flush
9879 269/300 Read block 601 len 24576
9880 270/300 Read block 4862 len 12288
9881 271/300 Read block 2030 len 16384
9882 272/300 Write block 4306 len 24576 data: 3 2 2 2 2 2
9883 273/300 Write block 1760 len 40960 data: 4 4 4 4 4 4 2 3 3 3
9884 274/300 Read block 4845 len 40960
9885 275/300 Write block 268 len 4096 data: 3
9886 276/300 Read block 3937 len 16384
9887 277/300 Read block 4209 len 12288
9888 278/300 Read block 3210 len 28672
9889 279/300 Read block 2381 len 28672
9890 280/300 Write block 399 len 36864 data: 5 6 5 4 3 3 3 3 5
9891 281/300 Write block 2123 len 40960 data: 3 3 3 3 3 3 3 2 2 2
9892 282/300 Write block 407 len 28672 data: 6 5 4 4 4 4 5
9893 283/300 Write block 1486 len 36864 data: 4 5 4 3 3 3 3 4 4
9894 284/300 Read block 308 len 4096
9895 285/300 Write block 1285 len 20480 data: 2 3 4 5 5
9896 286/300 Read block 838 len 12288
9897 287/300 Read block 3777 len 16384
9898 288/300 Read block 2794 len 28672
9899 289/300 Read block 4174 len 36864
9900 290/300 Read block 1546 len 28672
9901 291/300 Read block 494 len 16384
9902 292/300 Read block 2539 len 32768
9903 293/300 Read block 1174 len 16384
9904 294/300 Write block 3147 len 24576 data: 4 3 3 3 3 3
9905 295/300 Read block 530 len 40960
9906 296/300 Write block 221 len 40960 data: 2 2 2 2 2 3 3 2 2 2
9907 297/300 Read block 1018 len 20480
9908 298/300 Flush
9909 299/300 Write block 2400 len 28672 data: 2 2 2 2 2 3 3
9910 300/300 Read block 4058 len 12288
9911 Main task has work to do, go find it
9912 got dsc Start(0)
9913 start 0
9914 Feb 27 21:46:37.857 INFO request completed, latency_us: 224, response_code: 204, uri: /start/cid/0, method: POST, req_id: c3fec770-79b6-4a46-ac5f-c0476ce3f845, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
9915 Replay: started 0, returned:Ok(())
9916 ----------------------------------------------------------------
9917 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
9918 States: Offline Active Active
9919 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
9920 3014 Acked 4013 Flush 0 New Done Done true
9921 3015 Acked 4014 Write 9 New Done Done true
9922 3016 Acked 4015 Read 4 New Done Done true
9923 3017 Acked 4016 Read 7 New Done Done false
9924 3018 Acked 4017 Write 4 New Done Done false
9925 3019 Acked 4018 Read 3 New Done Done false
9926 3020 Acked 4019 Write 5 New Done Done false
9927 3021 Acked 4020 Write 6 New Done Done false
9928 3022 Acked 4021 Write 5 New Done Done false
9929 3023 Acked 4022 Read 10 New Done Done false
9930 3024 Acked 4023 Read 1 New Done Done false
9931 3025 Acked 4024 Write 7 New Done Done false
9932 3026 Acked 4025 Write 6 New Done Done false
9933 3027 Acked 4026 Read 3 New Done Done false
9934 3028 Acked 4027 Read 3 New Done Done false
9935 3029 Acked 4028 Write 2 New Done Done false
9936 3030 Acked 4029 Write 9 New Done Done false
9937 3031 Acked 4030 Read 7 New Done Done false
9938 3032 Acked 4031 Write 4 New Done Done false
9939 3033 Acked 4032 Write 10 New Done Done false
9940 3034 Acked 4033 Write 4 New Done Done false
9941 3035 Acked 4034 Write 1 New Done Done false
9942 3036 Acked 4035 Write 10 New Done Done false
9943 3037 Acked 4036 Read 7 New Done Done false
9944 3038 Acked 4037 Write 6 New Done Done false
9945 3039 Acked 4038 Read 1 New Done Done false
9946 3040 Acked 4039 Read 6 New Done Done false
9947 3041 Acked 4040 Read 1 New Done Done false
9948 3042 Acked 4041 Read 1 New Done Done false
9949 3043 Acked 4042 Write 3 New Done Done false
9950 3044 Acked 4043 Write 6 New Done Done false
9951 3045 Acked 4044 Read 7 New Done Done false
9952 3046 Acked 4045 Write 5 New Done Done false
9953 3047 Acked 4046 Flush 0 New Done Done false
9954 3048 Acked 4047 Read 4 New Done Done false
9955 3049 Acked 4048 Flush 0 New Done Done false
9956 3050 Acked 4049 Write 10 New Done Done false
9957 3051 Acked 4050 Read 3 New Done Done false
9958 3052 Acked 4051 Read 9 New Done Done false
9959 3053 Acked 4052 Read 1 New Done Done false
9960 3054 Acked 4053 Read 5 New Done Done false
9961 3055 Acked 4054 Flush 0 New Done Done false
9962 3056 Acked 4055 Read 1 New Done Done false
9963 3057 Acked 4056 Read 2 New Done Done false
9964 3058 Acked 4057 Flush 0 New Done Done false
9965 3059 Acked 4058 Read 9 New Done Done false
9966 3060 Acked 4059 Flush 0 New Done Done false
9967 3061 Acked 4060 Write 4 New Done Done false
9968 3062 Acked 4061 Write 3 New Done Done false
9969 3063 Acked 4062 Read 5 New Done Done false
9970 3064 Acked 4063 Write 7 New Done Done false
9971 3065 Acked 4064 Read 4 New Done Done false
9972 3066 Acked 4065 Write 4 New Done Done false
9973 3067 Acked 4066 Flush 0 New Done Done false
9974 3068 Acked 4067 Read 1 New Done Done false
9975 3069 Acked 4068 Read 2 New Done Done false
9976 3070 Acked 4069 Read 9 New Done Done false
9977 3071 Acked 4070 Write 7 New Done Done false
9978 3072 Acked 4071 Write 4 New Done Done false
9979 3073 Acked 4072 Read 5 New Done Done false
9980 3074 Acked 4073 Flush 0 New Done Done false
9981 3075 Acked 4074 Read 2 New Done Done false
9982 3076 Acked 4075 Write 6 New Done Done false
9983 3077 Acked 4076 Read 8 New Done Done false
9984 3078 Acked 4077 Write 8 New Done Done false
9985 3079 Acked 4078 Read 6 New Done Done false
9986 3080 Acked 4079 Read 2 New Done Done false
9987 3081 Acked 4080 Read 5 New Done Done false
9988 3082 Acked 4081 Write 9 New Done Done false
9989 3083 Acked 4082 Write 8 New Done Done false
9990 3084 Acked 4083 Write 9 New Done Done false
9991 3085 Acked 4084 Read 3 New Done Done false
9992 3086 Acked 4085 Read 1 New Done Done false
9993 3087 Acked 4086 Write 8 New Done Done false
9994 3088 Acked 4087 Read 4 New Done Done false
9995 3089 Acked 4088 Read 9 New Done Done false
9996 3090 Acked 4089 Read 9 New Done Done false
9997 3091 Acked 4090 Read 1 New Done Done false
9998 3092 Acked 4091 Flush 0 New Done Done false
9999 3093 Acked 4092 Flush 0 New Done Done false
10000 3094 Acked 4093 Read 3 New Done Done false
10001 3095 Acked 4094 Write 4 New Done Done false
10002 3096 Acked 4095 Write 3 New Done Done false
10003 3097 Acked 4096 Write 4 New Done Done false
10004 3098 Acked 4097 Write 7 New Done Done false
10005 3099 Acked 4098 Read 10 New Done Done false
10006 3100 Acked 4099 Write 10 New Done Done false
10007 3101 Acked 4100 Read 1 New Done Done false
10008 3102 Acked 4101 Write 3 New Done Done false
10009 3103 Acked 4102 Read 8 New Done Done false
10010 3104 Acked 4103 Read 5 New Done Done false
10011 3105 Acked 4104 Read 2 New Done Done false
10012 3106 Acked 4105 Read 9 New Done Done false
10013 3107 Acked 4106 Read 6 New Done Done false
10014 3108 Acked 4107 Read 3 New Done Done false
10015 3109 Acked 4108 Read 3 New Done Done false
10016 3110 Acked 4109 Read 7 New Done Done false
10017 3111 Acked 4110 Write 8 New Done Done false
10018 3112 Acked 4111 Read 9 New Done Done false
10019 3113 Acked 4112 Read 8 New Done Done false
10020 3114 Acked 4113 Write 6 New Done Done false
10021 3115 Acked 4114 Write 2 New Done Done false
10022 3116 Acked 4115 Flush 0 New Done Done false
10023 3117 Acked 4116 Read 10 New Done Done false
10024 3118 Acked 4117 Read 6 New Done Done false
10025 3119 Acked 4118 Flush 0 New Done Done false
10026 3120 Acked 4119 Flush 0 New Done Done false
10027 3121 Acked 4120 Flush 0 New Done Done false
10028 3122 Acked 4121 Flush 0 New Done Done false
10029 3123 Acked 4122 Read 7 New Done Done false
10030 3124 Acked 4123 Flush 0 New Done Done false
10031 3125 Acked 4124 Read 2 New Done Done false
10032 3126 Acked 4125 Read 4 New Done Done false
10033 3127 Acked 4126 Read 2 New Done Done false
10034 3128 Acked 4127 Write 6 New Done Done false
10035 3129 Acked 4128 Flush 0 New Done Done false
10036 3130 Acked 4129 Flush 0 New Done Done false
10037 3131 Acked 4130 Write 10 New Done Done false
10038 3132 Acked 4131 Write 7 New Done Done false
10039 3133 Acked 4132 Read 9 New Done Done false
10040 3134 Acked 4133 Flush 0 New Done Done false
10041 3135 Acked 4134 Write 6 New Done Done false
10042 3136 Acked 4135 Read 2 New Done Done false
10043 3137 Acked 4136 Read 7 New Done Done false
10044 3138 Acked 4137 Write 10 New Done Done false
10045 3139 Acked 4138 Read 8 New Done Done false
10046 3140 Acked 4139 Read 5 New Done Done false
10047 3141 Acked 4140 Flush 0 New Done Done false
10048 3142 Acked 4141 Write 4 New Done Done false
10049 3143 Acked 4142 Write 5 New Done Done false
10050 3144 Acked 4143 Read 3 New Done Done false
10051 3145 Acked 4144 Write 7 New Done Done false
10052 3146 Acked 4145 Read 7 New Done Done false
10053 3147 Acked 4146 Read 1 New Done Done false
10054 3148 Acked 4147 Read 2 New Done Done false
10055 3149 Acked 4148 Read 10 New Done Done false
10056 3150 Acked 4149 Write 4 New Done Done false
10057 3151 Acked 4150 Flush 0 New Done Done false
10058 3152 Acked 4151 Write 4 New Done Done false
10059 3153 Acked 4152 Read 1 New Done Done false
10060 3154 Acked 4153 Write 2 New Done Done false
10061 3155 Acked 4154 Write 9 New Done Done false
10062 3156 Acked 4155 Read 9 New Done Done false
10063 3157 Acked 4156 Read 10 New Done Done false
10064 3158 Acked 4157 Write 9 New Done Done false
10065 3159 Acked 4158 Read 7 New Done Done false
10066 3160 Acked 4159 Read 10 New Done Done false
10067 3161 Acked 4160 Read 10 New Done Done false
10068 3162 Acked 4161 Write 6 New Done Done false
10069 3163 Acked 4162 Write 2 New Done Done false
10070 3164 Acked 4163 Read 6 New Done Done false
10071 3165 Acked 4164 Write 8 New Done Done false
10072 3166 Acked 4165 Read 9 New Done Done false
10073 3167 Acked 4166 Write 6 New Done Done false
10074 3168 Acked 4167 Flush 0 New Done Done false
10075 3169 Acked 4168 Read 5 New Done Done false
10076 3170 Acked 4169 Read 1 New Done Done false
10077 3171 Acked 4170 Read 1 New Done Done false
10078 3172 Acked 4171 Read 6 New Done Done false
10079 3173 Acked 4172 Read 4 New Done Done false
10080 3174 Acked 4173 Write 2 New Done Done false
10081 3175 Acked 4174 Read 9 New Done Done false
10082 3176 Acked 4175 Flush 0 New Done Done false
10083 3177 Acked 4176 Read 5 New Done Done false
10084 3178 Acked 4177 Read 7 New Done Done false
10085 3179 Acked 4178 Read 1 New Done Done false
10086 3180 Acked 4179 Flush 0 New Done Done false
10087 3181 Acked 4180 Read 6 New Done Done false
10088 3182 Acked 4181 Read 6 New Done Done false
10089 3183 Acked 4182 Read 9 New Done Done false
10090 3184 Acked 4183 Read 9 New Done Done false
10091 3185 Acked 4184 Read 10 New Done Done false
10092 3186 Acked 4185 Read 4 New Done Done false
10093 3187 Acked 4186 Write 2 New Done Done false
10094 3188 Acked 4187 Read 10 New Done Done false
10095 3189 Acked 4188 Write 9 New Done Done false
10096 3190 Acked 4189 Write 4 New Done Done false
10097 3191 Acked 4190 Write 9 New Done Done false
10098 3192 Acked 4191 Read 4 New Done Done false
10099 3193 Acked 4192 Write 2 New Done Done false
10100 3194 Acked 4193 Read 3 New Done Done false
10101 3195 Acked 4194 Read 1 New Done Done false
10102 3196 Acked 4195 Write 8 New Done Done false
10103 3197 Acked 4196 Write 9 New Done Done false
10104 3198 Acked 4197 Write 8 New Done Done false
10105 3199 Acked 4198 Flush 0 New Done Done false
10106 3200 Acked 4199 Read 7 New Done Done false
10107 3201 Acked 4200 Read 9 New Done Done false
10108 3202 Acked 4201 Read 9 New Done Done false
10109 3203 Acked 4202 Write 2 New Done Done false
10110 3204 Acked 4203 Flush 0 New Done Done false
10111 3205 Acked 4204 Read 3 New Done Done false
10112 3206 Acked 4205 Write 4 New Done Done false
10113 3207 Acked 4206 Write 1 New Done Done false
10114 3208 Acked 4207 Write 3 New Done Done false
10115 3209 Acked 4208 Flush 0 New Done Done false
10116 3210 Acked 4209 Read 1 New Done Done false
10117 3211 Acked 4210 Read 3 New Done Done false
10118 3212 Acked 4211 Write 10 New Done Done false
10119 3213 Acked 4212 Flush 0 New Done Done false
10120 3214 Acked 4213 Read 5 New Done Done false
10121 3215 Acked 4214 Write 4 New Done Done false
10122 3216 Acked 4215 Flush 0 New Done Done false
10123 3217 Acked 4216 Flush 0 New Done Done false
10124 3218 Acked 4217 Read 4 New Done Done false
10125 3219 Acked 4218 Read 3 New Done Done false
10126 3220 Acked 4219 Read 7 New Done Done false
10127 3221 Acked 4220 Read 4 New Done Done false
10128 3222 Acked 4221 Write 10 New Done Done false
10129 3223 Acked 4222 Write 4 New Done Done false
10130 3224 Acked 4223 Write 1 New Done Done false
10131 3225 Acked 4224 Read 1 New Done Done false
10132 3226 Acked 4225 Write 5 New Done Done false
10133 3227 Acked 4226 Write 1 New Done Done false
10134 3228 Acked 4227 Read 10 New Done Done false
10135 3229 Acked 4228 Read 4 New Done Done false
10136 3230 Acked 4229 Read 7 New Done Done false
10137 3231 Acked 4230 Read 2 New Done Done false
10138 3232 Acked 4231 Write 9 New Done Done false
10139 3233 Acked 4232 Read 3 New Done Done false
10140 3234 Acked 4233 Read 1 New Done Done false
10141 3235 Acked 4234 Write 9 New Done Done false
10142 3236 Acked 4235 Write 8 New Done Done false
10143 3237 Acked 4236 Read 5 New Done Done false
10144 3238 Acked 4237 Read 6 New Done Done false
10145 3239 Acked 4238 Flush 0 New Done Done false
10146 3240 Acked 4239 Write 5 New Done Done false
10147 3241 Acked 4240 Read 1 New Done Done false
10148 3242 Acked 4241 Write 10 New Done Done false
10149 3243 Acked 4242 Read 5 New Done Done false
10150 3244 Acked 4243 Read 3 New Done Done false
10151 3245 Acked 4244 Write 8 New Done Done false
10152 3246 Acked 4245 Read 8 New Done Done false
10153 3247 Acked 4246 Write 8 New Done Done false
10154 3248 Acked 4247 Read 10 New Done Done false
10155 3249 Acked 4248 Read 3 New Done Done false
10156 3250 Acked 4249 Write 5 New Done Done false
10157 3251 Acked 4250 Read 6 New Done Done false
10158 3252 Acked 4251 Read 1 New Done Done false
10159 3253 Acked 4252 Read 8 New Done Done false
10160 3254 Acked 4253 Write 1 New Done Done false
10161 3255 Acked 4254 Write 4 New Done Done false
10162 3256 Acked 4255 Write 3 New Done Done false
10163 3257 Acked 4256 Read 5 New Done Done false
10164 3258 Acked 4257 Read 6 New Done Done false
10165 3259 Acked 4258 Write 2 New Done Done false
10166 3260 Acked 4259 Read 3 New Done Done false
10167 3261 Acked 4260 Write 1 New Done Done false
10168 3262 Acked 4261 Read 7 New Done Done false
10169 3263 Acked 4262 Read 6 New Done Done false
10170 3264 Acked 4263 Read 7 New Done Done false
10171 3265 Acked 4264 Read 6 New Done Done false
10172 3266 Acked 4265 Read 5 New Done Done false
10173 3267 Acked 4266 Write 4 New Done Done false
10174 3268 Acked 4267 Flush 0 New Done Done false
10175 3269 Acked 4268 Read 6 New Done Done false
10176 3270 Acked 4269 Read 10 New Done Done false
10177 3271 Acked 4270 Flush 0 New Done Done false
10178 3272 Acked 4271 Write 3 New Done Done false
10179 3273 Acked 4272 Write 7 New Done Done false
10180 3274 Acked 4273 Read 6 New Done Done false
10181 3275 Acked 4274 Write 4 New Done Done false
10182 3276 Acked 4275 Read 10 New Done Done false
10183 3277 Acked 4276 Read 6 New Done Done false
10184 3278 Acked 4277 Write 8 New Done Done false
10185 3279 Acked 4278 Read 9 New Done Done false
10186 3280 Acked 4279 Read 2 New Done Done false
10187 3281 Acked 4280 Write 5 New Done Done false
10188 3282 Acked 4281 Flush 0 New Done Done false
10189 3283 Acked 4282 Read 6 New Done Done false
10190 3284 Acked 4283 Read 3 New Done Done false
10191 3285 Acked 4284 Read 4 New Done Done false
10192 3286 Acked 4285 Write 6 New Done Done false
10193 3287 Acked 4286 Write 10 New Done Done false
10194 3288 Acked 4287 Read 10 New Done Done false
10195 3289 Acked 4288 Write 1 New Done Done false
10196 3290 Acked 4289 Read 4 New Done Done false
10197 3291 Acked 4290 Read 3 New Done Done false
10198 3292 Acked 4291 Read 7 New Done Done false
10199 3293 Acked 4292 Read 7 New Done Done false
10200 3294 Acked 4293 Write 9 New Done Done false
10201 3295 Acked 4294 Write 10 New Done Done false
10202 3296 Acked 4295 Write 7 New Done Done false
10203 3297 Acked 4296 Write 9 New Done Done false
10204 3298 Acked 4297 Read 1 New Done Done false
10205 3299 Acked 4298 Write 5 New Done Done false
10206 3300 Acked 4299 Read 3 New Done Done false
10207 3301 Acked 4300 Read 4 New Done Done false
10208 3302 Acked 4301 Read 7 New Done Done false
10209 3303 Acked 4302 Read 9 New Done Done false
10210 3304 Acked 4303 Read 7 New Done Done false
10211 3305 Acked 4304 Read 4 New Done Done false
10212 3306 Acked 4305 Read 8 New Done Done false
10213 3307 Acked 4306 Read 4 New Done Done false
10214 3308 Acked 4307 Write 6 New Done Done false
10215 3309 Acked 4308 Read 10 New Done Done false
10216 3310 Acked 4309 Write 10 New Done Done false
10217 3311 Acked 4310 Read 5 New Done Done false
10218 3312 Acked 4311 Flush 0 New Done Done false
10219 3313 Acked 4312 Write 7 New Done Done false
10220 3314 Acked 4313 Read 3 New Done Done false
10221 STATES DS:0 DS:1 DS:2 TOTAL
10222 New 301 0 0 301
10223 Sent 0 0 0 0
10224 Done 0 301 301 602
10225 Skipped 0 0 0 0
10226 Error 0 0 0 0
10227 Last Flush: 4012 4311 4311
10228 Downstairs last five completed: 4012 4011 4010 4009 4008
10229 Upstairs last five completed: 3314 3313 3312 3311 3310
10230 CLIENT: Up:0 ds:301 act:2
102312024-02-27T21:46:38.525ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
102322024-02-27T21:46:38.525ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
102332024-02-27T21:46:38.525ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
102342024-02-27T21:46:38.525ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
102352024-02-27T21:46:38.525ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
102362024-02-27T21:46:38.525ZINFOcrucible: [0] client re-new 302 jobs since flush 4012 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
102372024-02-27T21:46:38.525ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
102382024-02-27T21:46:39.526ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
102392024-02-27T21:46:39.526ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
102402024-02-27T21:46:39.526ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
102412024-02-27T21:46:39.526ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
102422024-02-27T21:46:39.526ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
102432024-02-27T21:46:39.526ZINFOcrucible: [0] client re-new 302 jobs since flush 4012 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
102442024-02-27T21:46:39.527ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
10245 [8810] got start action
10246 [8810] I am going to restart
10247 Starting downstairs at port 8810
10248 Make output file at "/tmp/dsc/downstairs-8810.txt"
10249 [8810][0] reports Starting
10250 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1178)
10251 [8810][0] reports Running
102522024-02-27T21:46:40.528ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
102532024-02-27T21:46:40.528ZINFOcrucible: ds_connection connected = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
102542024-02-27T21:46:40.529ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
102552024-02-27T21:46:40.529ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
102562024-02-27T21:46:40.529ZINFOcrucible: send last flush ID to this DS: 4012 = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
102572024-02-27T21:46:40.529ZINFOcrucible: Replied this last flush ID: 4012 = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
102582024-02-27T21:46:40.529ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
102592024-02-27T21:46:40.529ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
102602024-02-27T21:46:40.529ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
102612024-02-27T21:46:40.529ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
10262 ----------------------------------------------------------------
10263 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10264 Downstairs last five completed: 4314 4313 4312 4311 4310
10265 Upstairs last five completed: 3315 3314 3313 3312 3311
10266 CLIENT: Up:0 ds:0 act:3
10267 Replay: All jobs finished, all DS active.
10268 Main task has work to do, go find it
10269 got dsc Stop(2)
10270 stop 2
10271 Feb 27 21:46:41.860 INFO request completed, latency_us:[8830] Got stop action so:false kr:false
10272 498, response_code: 204, uri: /stop/cid/2, method: POST, req_id: a3081613-4a4d-4a42-bf69-77358879a77f, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
102732024-02-27T21:46:41.861ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
10274 Feb 27 21:46:41.861 INFO request completed, latency_us: 126, response_code: 200, uri{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-27T21:46:41.861481704Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"io task","client":"2","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
10275 : /state/cid/2, method: GET, req_id: 161b8cd3-3154-45dc-a0a4-31d50572fe67, remote_addr: 127.0.0.1{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-27T21:46:41.86151194Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
10276 :44347, local_addr: 127.0.0.1:9998
102772024-02-27T21:46:41.861ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
102782024-02-27T21:46:41.861ZINFOcrucible: [2] client re-new 0 jobs since flush 4314 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
102792024-02-27T21:46:41.861ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
10280 [8830] Exited with: ExitStatus(unix_wait_status(9))
10281 [8830][2] reports Exit
102822024-02-27T21:46:42.863ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
102832024-02-27T21:46:42.863ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
102842024-02-27T21:46:42.863ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
102852024-02-27T21:46:42.863ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
102862024-02-27T21:46:42.863ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
102872024-02-27T21:46:42.863ZINFOcrucible: [2] client re-new 0 jobs since flush 4314 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
102882024-02-27T21:46:42.863ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
102892024-02-27T21:46:43.865ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
102902024-02-27T21:46:43.865ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
102912024-02-27T21:46:43.865ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
102922024-02-27T21:46:43.865ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
102932024-02-27T21:46:43.865ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
102942024-02-27T21:46:43.865ZINFOcrucible: [2] client re-new 0 jobs since flush 4314 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
102952024-02-27T21:46:43.865ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
102962024-02-27T21:46:44.866ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
102972024-02-27T21:46:44.866ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
102982024-02-27T21:46:44.866ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
102992024-02-27T21:46:44.866ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
103002024-02-27T21:46:44.866ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
103012024-02-27T21:46:44.866ZINFOcrucible: [2] client re-new 0 jobs since flush 4314 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
103022024-02-27T21:46:44.866ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
10303 Feb 27 21:46:45.862 INFO request completed, latency_us: 194, response_code: 200, uri: /state/cid/2, method: GET, req_id: 860f3b2b-a07c-488c-857b-b993de5e34cd, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
10304 001/300 Write block 2795 len 16384 data: 3 4 4 4
10305 002/300 Read block 1291 len 20480
10306 003/300 Read block 2080 len 4096
10307 004/300 Read block 4023 len 4096
10308 005/300 Read block 643 len 20480
103092024-02-27T21:46:45.867ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
103102024-02-27T21:46:45.867ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
103112024-02-27T21:46:45.867ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
103122024-02-27T21:46:45.867ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
103132024-02-27T21:46:45.867ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
103142024-02-27T21:46:45.867ZINFOcrucible: [2] client re-new 5 jobs since flush 4314 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
103152024-02-27T21:46:45.867ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
10316 006/300 Read block 560 len 28672
10317 007/300 Read block 4030 len 4096
10318 008/300 Read block 1184 len 36864
10319 009/300 Read block 1176 len 32768
10320 010/300 Write block 4786 len 8192 data: 4 4
10321 011/300 Write block 4834 len 24576 data: 5 4 4 4 4 4
10322 012/300 Read block 4646 len 4096
10323 013/300 Read block 2906 len 20480
10324 014/300 Read block 2811 len 12288
10325 015/300 Read block 741 len 16384
10326 016/300 Write block 1429 len 32768 data: 2 2 2 3 4 3 2 3
10327 017/300 Write block 2594 len 40960 data: 5 3 2 3 4 4 4 4 4 3
10328 018/300 Read block 4050 len 4096
10329 019/300 Read block 4386 len 20480
10330 020/300 Read block 811 len 28672
10331 021/300 Write block 203 len 40960 data: 4 4 4 5 5 5 5 4 4 4
10332 022/300 Read block 2794 len 28672
10333 023/300 Read block 365 len 4096
10334 024/300 Read block 3471 len 32768
10335 025/300 Read block 232 len 20480
10336 026/300 Write block 2196 len 20480 data: 4 4 4 4 4
10337 027/300 Write block 1117 len 24576 data: 4 5 4 4 4 3
10338 028/300 Write block 3693 len 8192 data: 3 3
10339 029/300 Read block 175 len 28672
10340 030/300 Flush
10341 031/300 Write block 1005 len 4096 data: 3
10342 032/300 Flush
10343 033/300 Read block 401 len 36864
10344 034/300 Write block 1722 len 12288 data: 3 3 3
10345 035/300 Read block 3160 len 28672
10346 036/300 Read block 2043 len 40960
10347 037/300 Read block 3407 len 8192
10348 038/300 Read block 2236 len 28672
10349 039/300 Read block 3793 len 20480
10350 040/300 Read block 1797 len 12288
10351 041/300 Read block 1893 len 36864
10352 042/300 Read block 4949 len 16384
10353 043/300 Flush
10354 044/300 Read block 3099 len 16384
10355 045/300 Write block 803 len 8192 data: 2 2
10356 046/300 Write block 1239 len 8192 data: 2 2
10357 047/300 Read block 2582 len 36864
10358 048/300 Read block 4562 len 28672
10359 049/300 Read block 1505 len 36864
10360 050/300 Write block 2975 len 24576 data: 4 4 4 5 4 3
10361 051/300 Write block 865 len 4096 data: 2
10362 052/300 Write block 4808 len 8192 data: 4 2
10363 053/300 Write block 2383 len 12288 data: 3 3 3
10364 054/300 Write block 989 len 4096 data: 5
10365 055/300 Write block 158 len 20480 data: 3 3 2 2 3
10366 056/300 Write block 118 len 36864 data: 4 4 4 5 5 3 3 2 2
10367 057/300 Read block 2752 len 8192
10368 058/300 Write block 3383 len 32768 data: 3 2 2 3 3 3 2 2
10369 059/300 Read block 3334 len 24576
10370 060/300 Read block 776 len 4096
10371 061/300 Read block 1838 len 8192
10372 062/300 Flush
10373 063/300 Write block 22 len 24576 data: 4 6 6 6 3 2
10374 064/300 Read block 2816 len 12288
10375 065/300 Write block 4605 len 32768 data: 2 2 2 3 3 3 3 3
10376 066/300 Read block 64 len 20480
10377 067/300 Read block 3756 len 32768
10378 068/300 Write block 3556 len 8192 data: 5 4
10379 069/300 Read block 3031 len 24576
10380 070/300 Read block 3573 len 4096
10381 071/300 Write block 2992 len 28672 data: 3 3 3 3 3 3 5
10382 072/300 Write block 2456 len 36864 data: 3 4 3 3 5 6 6 5 5
10383 073/300 Flush
10384 074/300 Read block 4256 len 36864
10385 075/300 Flush
10386 076/300 Write block 1426 len 12288 data: 3 3 2
10387 077/300 Read block 4139 len 36864
10388 078/300 Read block 2636 len 36864
10389 079/300 Read block 2253 len 36864
10390 080/300 Read block 1655 len 12288
10391 081/300 Write block 3442 len 24576 data: 3 5 5 3 4 4
10392 082/300 Read block 2033 len 28672
10393 083/300 Write block 85 len 20480 data: 2 2 2 2 2
10394 084/300 Write block 983 len 12288 data: 4 3 4
10395 085/300 Read block 3163 len 20480
10396 086/300 Read block 4649 len 24576
10397 087/300 Write block 4294 len 24576 data: 2 3 3 5 6 6
10398 088/300 Write block 3226 len 40960 data: 3 2 2 2 2 2 3 3 3 4
10399 089/300 Read block 1566 len 28672
10400 090/300 Read block 2411 len 12288
10401 091/300 Read block 496 len 20480
10402 092/300 Write block 3411 len 40960 data: 5 5 5 4 4 4 4 3 3 3
10403 093/300 Write block 429 len 4096 data: 2
10404 094/300 Read block 647 len 24576
10405 095/300 Read block 3197 len 20480
10406 096/300 Read block 4330 len 8192
10407 097/300 Read block 1688 len 12288
10408 098/300 Write block 1032 len 8192 data: 2 2
10409 099/300 Write block 2990 len 16384 data: 3 3 4 4
10410 100/300 Read block 3210 len 8192
10411 101/300 Write block 2403 len 16384 data: 3 3 4 4
10412 102/300 Flush
10413 103/300 Write block 4733 len 40960 data: 4 4 5 5 5 6 5 5 5 4
10414 104/300 Write block 1638 len 36864 data: 5 6 5 5 5 5 4 4 5
10415 105/300 Write block 1494 len 4096 data: 5
10416 106/300 Write block 1813 len 40960 data: 4 4 4 3 3 3 2 2 2 2
10417 107/300 Write block 588 len 32768 data: 5 5 4 5 4 4 5 4
10418 108/300 Read block 363 len 28672
10419 109/300 Read block 1826 len 16384
10420 110/300 Read block 3350 len 40960
10421 111/300 Read block 1785 len 4096
10422 112/300 Write block 4577 len 16384 data: 6 6 5 5
10423 113/300 Write block 3747 len 28672 data: 2 2 2 2 2 2 3
10424 114/300 Read block 3229 len 8192
10425 115/300 Write block 2118 len 24576 data: 2 2 2 2 3 4
10426 116/300 Read block 901 len 40960
10427 117/300 Read block 725 len 24576
10428 118/300 Write block 873 len 8192 data: 4 5
10429 119/300 Read block 254 len 36864
10430 120/300 Write block 3694 len 24576 data: 4 3 3 3 3 3
10431 121/300 Read block 3506 len 12288
10432 122/300 Read block 1076 len 24576
10433 123/300 Write block 2538 len 24576 data: 5 5 5 4 4 4
10434 124/300 Read block 996 len 20480
10435 125/300 Write block 862 len 4096 data: 4
10436 126/300 Write block 1125 len 28672 data: 2 2 2 3 3 4 5
10437 127/300 Flush
10438 128/300 Read block 357 len 24576
10439 129/300 Write block 2955 len 32768 data: 4 4 4 5 4 3 3 3
10440 130/300 Write block 1976 len 36864 data: 7 7 7 6 6 6 5 4 4
10441 131/300 Write block 4698 len 20480 data: 2 2 2 2 2
10442 132/300 Read block 3121 len 40960
10443 133/300 Read block 2152 len 16384
10444 134/300 Write block 2206 len 40960 data: 5 5 4 3 3 4 4 4 3 4
10445 135/300 Write block 1853 len 8192 data: 4 4
10446 136/300 Read block 2652 len 24576
10447 137/300 Write block 1934 len 28672 data: 4 4 3 3 3 3 4
10448 138/300 Write block 1024 len 36864 data: 3 3 3 3 4 3 3 2 3
10449 139/300 Read block 3598 len 4096
10450 140/300 Read block 1608 len 36864
10451 141/300 Read block 4200 len 40960
10452 142/300 Read block 3655 len 16384
10453 143/300 Read block 1864 len 24576
10454 144/300 Write block 559 len 32768 data: 4 2 3 3 3 3 3 4
10455 145/300 Read block 2300 len 24576
10456 146/300 Write block 2964 len 16384 data: 3 3 3 3
10457 147/300 Read block 2511 len 4096
10458 148/300 Write block 3169 len 40960 data: 3 3 2 2 2 2 4 4 4 3
10459 149/300 Read block 834 len 32768
10460 150/300 Read block 2682 len 8192
10461 151/300 Flush
10462 152/300 Read block 1123 len 32768
10463 153/300 Read block 3426 len 20480
10464 154/300 Read block 1464 len 16384
10465 155/300 Read block 4662 len 24576
10466 156/300 Flush
10467 157/300 Flush
10468 158/300 Write block 2446 len 40960 data: 4 4 4 3 3 3 4 4 4 3
10469 159/300 Read block 495 len 12288
10470 160/300 Write block 1753 len 16384 data: 2 2 2 2
10471 161/300 Write block 1871 len 28672 data: 2 3 3 3 3 4 3
10472 162/300 Read block 4551 len 12288
10473 163/300 Flush
10474 164/300 Write block 4557 len 12288 data: 4 4 3
10475 165/300 Write block 3355 len 16384 data: 3 3 2 2
10476 166/300 Write block 1467 len 32768 data: 3 3 3 3 4 4 6 7
10477 167/300 Read block 4664 len 32768
10478 168/300 Write block 2029 len 36864 data: 4 5 4 3 3 3 2 2 3
10479 169/300 Write block 4703 len 40960 data: 3 3 4 4 3 3 3 3 3 4
10480 170/300 Read block 1815 len 12288
10481 171/300 Read block 1080 len 32768
10482 172/300 Read block 4575 len 40960
10483 173/300 Read block 1580 len 8192
10484 174/300 Write block 3587 len 12288 data: 3 4 4
10485 175/300 Write block 3862 len 40960 data: 2 2 2 2 2 2 2 2 2 2
10486 176/300 Write block 3922 len 24576 data: 2 2 3 3 3 3
10487 177/300 Read block 1079 len 28672
10488 178/300 Write block 1352 len 36864 data: 3 3 2 2 2 2 2 3 3
10489 179/300 Read block 1677 len 32768
10490 180/300 Read block 2311 len 28672
10491 181/300 Read block 2966 len 40960
10492 182/300 Write block 618 len 40960 data: 4 4 3 3 2 2 2 2 2 2
10493 183/300 Write block 3498 len 28672 data: 3 3 3 3 3 2 2
10494 184/300 Write block 2161 len 32768 data: 4 4 3 2 2 2 3 4
10495 185/300 Flush
10496 186/300 Write block 4665 len 40960 data: 6 6 5 5 3 3 3 3 3 3
10497 187/300 Write block 2503 len 28672 data: 6 7 7 7 8 7 4
10498 188/300 Read block 4614 len 8192
10499 189/300 Write block 4599 len 20480 data: 5 4 4 4 4
10500 190/300 Read block 1096 len 40960
10501 191/300 Write block 145 len 8192 data: 2 3
10502 192/300 Read block 4870 len 12288
10503 193/300 Read block 117 len 32768
10504 194/300 Read block 3031 len 12288
10505 195/300 Read block 3875 len 12288
10506 196/300 Read block 2358 len 40960
10507 197/300 Read block 2636 len 16384
10508 198/300 Write block 2790 len 4096 data: 2
10509 199/300 Read block 3673 len 4096
10510 200/300 Read block 2902 len 40960
10511 201/300 Read block 1632 len 24576
10512 202/300 Read block 1613 len 40960
10513 203/300 Read block 4500 len 28672
10514 204/300 Write block 3343 len 16384 data: 3 4 6 5
10515 205/300 Flush
10516 206/300 Write block 2858 len 24576 data: 3 3 2 2 2 2
10517 207/300 Read block 4045 len 4096
10518 208/300 Flush
10519 209/300 Read block 2250 len 20480
10520 210/300 Read block 2088 len 12288
10521 211/300 Write block 1831 len 28672 data: 3 3 3 3 3 3 4
10522 212/300 Write block 2879 len 24576 data: 4 3 3 3 2 2
10523 213/300 Flush
10524 214/300 Read block 2889 len 8192
10525 215/300 Write block 2266 len 12288 data: 4 4 4
10526 216/300 Write block 3862 len 36864 data: 3 3 3 3 3 3 3 3 3
10527 217/300 Write block 711 len 28672 data: 2 2 2 2 2 3 3
10528 218/300 Write block 2906 len 36864 data: 4 4 5 5 5 5 4 4 4
10529 219/300 Read block 2998 len 12288
10530 220/300 Read block 3522 len 20480
10531 221/300 Read block 1060 len 36864
10532 222/300 Flush
10533 223/300 Read block 2135 len 24576
10534 224/300 Write block 2377 len 20480 data: 3 3 3 3 3
10535 225/300 Read block 4827 len 32768
10536 226/300 Read block 3901 len 32768
10537 227/300 Read block 3550 len 28672
10538 228/300 Write block 4754 len 8192 data: 4 3
10539 229/300 Read block 3771 len 32768
10540 230/300 Write block 2502 len 28672 data: 5 7 8 8 8 9 8
10541 231/300 Write block 4867 len 24576 data: 4 4 3 4 4 4
10542 232/300 Read block 2035 len 28672
10543 233/300 Write block 615 len 4096 data: 3
10544 234/300 Flush
10545 235/300 Read block 4174 len 12288
10546 236/300 Flush
10547 237/300 Write block 3828 len 20480 data: 2 2 3 3 3
10548 238/300 Write block 1140 len 28672 data: 4 4 4 5 4 5 5
10549 239/300 Read block 875 len 12288
10550 240/300 Read block 896 len 32768
10551 241/300 Write block 517 len 8192 data: 4 3
10552 242/300 Write block 492 len 20480 data: 3 2 3 4 4
10553 243/300 Read block 2768 len 36864
10554 244/300 Read block 2545 len 8192
10555 245/300 Read block 476 len 8192
10556 246/300 Write block 4092 len 20480 data: 4 4 3 4 4
10557 247/300 Read block 2807 len 12288
10558 248/300 Flush
10559 249/300 Write block 282 len 28672 data: 4 3 3 2 2 2 2
10560 250/300 Write block 3620 len 36864 data: 2 2 2 2 2 2 2 2 2
10561 251/300 Read block 446 len 32768
10562 252/300 Write block 3600 len 32768 data: 4 4 4 4 3 2 4 4
10563 253/300 Read block 3829 len 8192
10564 254/300 Read block 1255 len 36864
10565 255/300 Read block 3409 len 28672
10566 256/300 Write block 1971 len 28672 data: 3 2 3 3 6 8 8
10567 257/300 Read block 557 len 40960
10568 258/300 Read block 2079 len 24576
10569 259/300 Read block 1573 len 28672
10570 260/300 Flush
10571 261/300 Read block 1866 len 36864
10572 262/300 Read block 824 len 28672
10573 263/300 Read block 4531 len 28672
10574 264/300 Write block 2134 len 24576 data: 2 2 2 2 2 3
10575 265/300 Write block 4763 len 20480 data: 3 3 4 4 3
10576 266/300 Write block 3097 len 24576 data: 3 3 5 3 3 4
10577 267/300 Write block 2592 len 20480 data: 5 6 6 4 3
10578 268/300 Write block 3323 len 32768 data: 2 2 2 3 4 4 3 3
10579 269/300 Write block 2816 len 24576 data: 5 4 5 5 4 4
10580 270/300 Flush
10581 271/300 Write block 1621 len 28672 data: 5 5 3 3 3 3 3
10582 272/300 Read block 1967 len 24576
10583 273/300 Write block 120 len 16384 data: 5 6 6 4
10584 274/300 Write block 4608 len 4096 data: 4
10585 275/300 Write block 1061 len 16384 data: 3 3 3 3
10586 276/300 Write block 3895 len 36864 data: 3 3 2 2 2 4 4 4 4
10587 277/300 Read block 4711 len 36864
10588 278/300 Write block 1374 len 8192 data: 2 2
10589 279/300 Write block 122 len 8192 data: 7 5
10590 280/300 Read block 3822 len 28672
10591 281/300 Read block 414 len 40960
10592 282/300 Read block 2318 len 4096
10593 283/300 Read block 615 len 40960
10594 284/300 Write block 483 len 40960 data: 2 2 3 3 4 4 3 3 3 4
10595 285/300 Write block 889 len 32768 data: 3 3 3 3 3 3 3 2
10596 286/300 Write block 2194 len 4096 data: 4
10597 287/300 Write block 4356 len 16384 data: 2 2 2 4
10598 288/300 Write block 2026 len 8192 data: 4 5
10599 289/300 Read block 496 len 32768
10600 290/300 Read block 1348 len 24576
10601 291/300 Read block 3510 len 4096
10602 292/300 Write block 2900 len 16384 data: 3 3 2 2
10603 293/300 Read block 1614 len 32768
10604 294/300 Write block 3516 len 28672 data: 3 3 3 3 4 4 4
10605 295/300 Flush
10606 296/300 Read block 2315 len 36864
10607 297/300 Read block 1056 len 12288
10608 298/300 Flush
10609 299/300 Read block 457 len 32768
10610 300/300 Write block 1594 len 8192 data: 3 3
10611 Main task has work to do, go find it
10612 got dsc Start(2)
10613 start 2
10614 Feb 27 21:46:46.203 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/2, method: POST, req_id: 1e2bbbcb-9a5b-400b-858f-33b1be174925, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
10615 Replay: started 2, returned:Ok(())
10616 ----------------------------------------------------------------
10617 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
10618 States: Active Active Offline
10619 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
10620 3316 Acked 4315 Write 4 Done Done New true
10621 3317 Acked 4316 Read 5 Done Done New true
10622 3318 Acked 4317 Read 1 Done Done New true
10623 3319 Acked 4318 Read 1 Done Done New true
10624 3320 Acked 4319 Read 5 Done Done New true
10625 3321 Acked 4320 Read 7 Done Done New false
10626 3322 Acked 4321 Read 1 Done Done New false
10627 3323 Acked 4322 Read 9 Done Done New false
10628 3324 Acked 4323 Read 8 Done Done New false
10629 3325 Acked 4324 Write 2 Done Done New false
10630 3326 Acked 4325 Write 6 Done Done New false
10631 3327 Acked 4326 Read 1 Done Done New false
10632 3328 Acked 4327 Read 5 Done Done New false
10633 3329 Acked 4328 Read 3 Done Done New false
10634 3330 Acked 4329 Read 4 Done Done New false
10635 3331 Acked 4330 Write 8 Done Done New false
10636 3332 Acked 4331 Write 10 Done Done New false
10637 3333 Acked 4332 Read 1 Done Done New false
10638 3334 Acked 4333 Read 5 Done Done New false
10639 3335 Acked 4334 Read 7 Done Done New false
10640 3336 Acked 4335 Write 10 Done Done New false
10641 3337 Acked 4336 Read 7 Done Done New false
10642 3338 Acked 4337 Read 1 Done Done New false
10643 3339 Acked 4338 Read 8 Done Done New false
10644 3340 Acked 4339 Read 5 Done Done New false
10645 3341 Acked 4340 Write 5 Done Done New false
10646 3342 Acked 4341 Write 6 Done Done New false
10647 3343 Acked 4342 Write 2 Done Done New false
10648 3344 Acked 4343 Read 7 Done Done New false
10649 3345 Acked 4344 Flush 0 Done Done New false
10650 3346 Acked 4345 Write 1 Done Done New false
10651 3347 Acked 4346 Flush 0 Done Done New false
10652 3348 Acked 4347 Read 9 Done Done New false
10653 3349 Acked 4348 Write 3 Done Done New false
10654 3350 Acked 4349 Read 7 Done Done New false
10655 3351 Acked 4350 Read 10 Done Done New false
10656 3352 Acked 4351 Read 2 Done Done New false
10657 3353 Acked 4352 Read 7 Done Done New false
10658 3354 Acked 4353 Read 5 Done Done New false
10659 3355 Acked 4354 Read 3 Done Done New false
10660 3356 Acked 4355 Read 9 Done Done New false
10661 3357 Acked 4356 Read 4 Done Done New false
10662 3358 Acked 4357 Flush 0 Done Done New false
10663 3359 Acked 4358 Read 4 Done Done New false
10664 3360 Acked 4359 Write 2 Done Done New false
10665 3361 Acked 4360 Write 2 Done Done New false
10666 3362 Acked 4361 Read 9 Done Done New false
10667 3363 Acked 4362 Read 7 Done Done New false
10668 3364 Acked 4363 Read 9 Done Done New false
10669 3365 Acked 4364 Write 6 Done Done New false
10670 3366 Acked 4365 Write 1 Done Done New false
10671 3367 Acked 4366 Write 2 Done Done New false
10672 3368 Acked 4367 Write 3 Done Done New false
10673 3369 Acked 4368 Write 1 Done Done New false
10674 3370 Acked 4369 Write 5 Done Done New false
10675 3371 Acked 4370 Write 9 Done Done New false
10676 3372 Acked 4371 Read 2 Done Done New false
10677 3373 Acked 4372 Write 8 Done Done New false
10678 3374 Acked 4373 Read 6 Done Done New false
10679 3375 Acked 4374 Read 1 Done Done New false
10680 3376 Acked 4375 Read 2 Done Done New false
10681 3377 Acked 4376 Flush 0 Done Done New false
10682 3378 Acked 4377 Write 6 Done Done New false
10683 3379 Acked 4378 Read 3 Done Done New false
10684 3380 Acked 4379 Write 8 Done Done New false
10685 3381 Acked 4380 Read 5 Done Done New false
10686 3382 Acked 4381 Read 8 Done Done New false
10687 3383 Acked 4382 Write 2 Done Done New false
10688 3384 Acked 4383 Read 6 Done Done New false
10689 3385 Acked 4384 Read 1 Done Done New false
10690 3386 Acked 4385 Write 7 Done Done New false
10691 3387 Acked 4386 Write 9 Done Done New false
10692 3388 Acked 4387 Flush 0 Done Done New false
10693 3389 Acked 4388 Read 9 Done Done New false
10694 3390 Acked 4389 Flush 0 Done Done New false
10695 3391 Acked 4390 Write 3 Done Done New false
10696 3392 Acked 4391 Read 9 Done Done New false
10697 3393 Acked 4392 Read 9 Done Done New false
10698 3394 Acked 4393 Read 9 Done Done New false
10699 3395 Acked 4394 Read 3 Done Done New false
10700 3396 Acked 4395 Write 6 Done Done New false
10701 3397 Acked 4396 Read 7 Done Done New false
10702 3398 Acked 4397 Write 5 Done Done New false
10703 3399 Acked 4398 Write 3 Done Done New false
10704 3400 Acked 4399 Read 5 Done Done New false
10705 3401 Acked 4400 Read 6 Done Done New false
10706 3402 Acked 4401 Write 6 Done Done New false
10707 3403 Acked 4402 Write 10 Done Done New false
10708 3404 Acked 4403 Read 7 Done Done New false
10709 3405 Acked 4404 Read 3 Done Done New false
10710 3406 Acked 4405 Read 5 Done Done New false
10711 3407 Acked 4406 Write 10 Done Done New false
10712 3408 Acked 4407 Write 1 Done Done New false
10713 3409 Acked 4408 Read 6 Done Done New false
10714 3410 Acked 4409 Read 5 Done Done New false
10715 3411 Acked 4410 Read 2 Done Done New false
10716 3412 Acked 4411 Read 3 Done Done New false
10717 3413 Acked 4412 Write 2 Done Done New false
10718 3414 Acked 4413 Write 4 Done Done New false
10719 3415 Acked 4414 Read 2 Done Done New false
10720 3416 Acked 4415 Write 4 Done Done New false
10721 3417 Acked 4416 Flush 0 Done Done New false
10722 3418 Acked 4417 Write 10 Done Done New false
10723 3419 Acked 4418 Write 9 Done Done New false
10724 3420 Acked 4419 Write 1 Done Done New false
10725 3421 Acked 4420 Write 10 Done Done New false
10726 3422 Acked 4421 Write 8 Done Done New false
10727 3423 Acked 4422 Read 7 Done Done New false
10728 3424 Acked 4423 Read 4 Done Done New false
10729 3425 Acked 4424 Read 10 Done Done New false
10730 3426 Acked 4425 Read 1 Done Done New false
10731 3427 Acked 4426 Write 4 Done Done New false
10732 3428 Acked 4427 Write 7 Done Done New false
10733 3429 Acked 4428 Read 2 Done Done New false
10734 3430 Acked 4429 Write 6 Done Done New false
10735 3431 Acked 4430 Read 10 Done Done New false
10736 3432 Acked 4431 Read 6 Done Done New false
10737 3433 Acked 4432 Write 2 Done Done New false
10738 3434 Acked 4433 Read 9 Done Done New false
10739 3435 Acked 4434 Write 6 Done Done New false
10740 3436 Acked 4435 Read 3 Done Done New false
10741 3437 Acked 4436 Read 6 Done Done New false
10742 3438 Acked 4437 Write 6 Done Done New false
10743 3439 Acked 4438 Read 5 Done Done New false
10744 3440 Acked 4439 Write 1 Done Done New false
10745 3441 Acked 4440 Write 7 Done Done New false
10746 3442 Acked 4441 Flush 0 Done Done New false
10747 3443 Acked 4442 Read 6 Done Done New false
10748 3444 Acked 4443 Write 8 Done Done New false
10749 3445 Acked 4444 Write 9 Done Done New false
10750 3446 Acked 4445 Write 5 Done Done New false
10751 3447 Acked 4446 Read 10 Done Done New false
10752 3448 Acked 4447 Read 4 Done Done New false
10753 3449 Acked 4448 Write 10 Done Done New false
10754 3450 Acked 4449 Write 2 Done Done New false
10755 3451 Acked 4450 Read 6 Done Done New false
10756 3452 Acked 4451 Write 7 Done Done New false
10757 3453 Acked 4452 Write 9 Done Done New false
10758 3454 Acked 4453 Read 1 Done Done New false
10759 3455 Acked 4454 Read 9 Done Done New false
10760 3456 Acked 4455 Read 10 Done Done New false
10761 3457 Acked 4456 Read 4 Done Done New false
10762 3458 Acked 4457 Read 6 Done Done New false
10763 3459 Acked 4458 Write 8 Done Done New false
10764 3460 Acked 4459 Read 6 Done Done New false
10765 3461 Acked 4460 Write 4 Done Done New false
10766 3462 Acked 4461 Read 1 Done Done New false
10767 3463 Acked 4462 Write 10 Done Done New false
10768 3464 Acked 4463 Read 8 Done Done New false
10769 3465 Acked 4464 Read 2 Done Done New false
10770 3466 Acked 4465 Flush 0 Done Done New false
10771 3467 Acked 4466 Read 8 Done Done New false
10772 3468 Acked 4467 Read 5 Done Done New false
10773 3469 Acked 4468 Read 4 Done Done New false
10774 3470 Acked 4469 Read 6 Done Done New false
10775 3471 Acked 4470 Flush 0 Done Done New false
10776 3472 Acked 4471 Flush 0 Done Done New false
10777 3473 Acked 4472 Write 10 Done Done New false
10778 3474 Acked 4473 Read 3 Done Done New false
10779 3475 Acked 4474 Write 4 Done Done New false
10780 3476 Acked 4475 Write 7 Done Done New false
10781 3477 Acked 4476 Read 3 Done Done New false
10782 3478 Acked 4477 Flush 0 Done Done New false
10783 3479 Acked 4478 Write 3 Done Done New false
10784 3480 Acked 4479 Write 4 Done Done New false
10785 3481 Acked 4480 Write 8 Done Done New false
10786 3482 Acked 4481 Read 8 Done Done New false
10787 3483 Acked 4482 Write 9 Done Done New false
10788 3484 Acked 4483 Write 10 Done Done New false
10789 3485 Acked 4484 Read 3 Done Done New false
10790 3486 Acked 4485 Read 8 Done Done New false
10791 3487 Acked 4486 Read 10 Done Done New false
10792 3488 Acked 4487 Read 2 Done Done New false
10793 3489 Acked 4488 Write 3 Done Done New false
10794 3490 Acked 4489 Write 10 Done Done New false
10795 3491 Acked 4490 Write 6 Done Done New false
10796 3492 Acked 4491 Read 7 Done Done New false
10797 3493 Acked 4492 Write 9 Done Done New false
10798 3494 Acked 4493 Read 8 Done Done New false
10799 3495 Acked 4494 Read 7 Done Done New false
10800 3496 Acked 4495 Read 10 Done Done New false
10801 3497 Acked 4496 Write 10 Done Done New false
10802 3498 Acked 4497 Write 7 Done Done New false
10803 3499 Acked 4498 Write 8 Done Done New false
10804 3500 Acked 4499 Flush 0 Done Done New false
10805 3501 Acked 4500 Write 10 Done Done New false
10806 3502 Acked 4501 Write 7 Done Done New false
10807 3503 Acked 4502 Read 2 Done Done New false
10808 3504 Acked 4503 Write 5 Done Done New false
10809 3505 Acked 4504 Read 10 Done Done New false
10810 3506 Acked 4505 Write 2 Done Done New false
10811 3507 Acked 4506 Read 3 Done Done New false
10812 3508 Acked 4507 Read 8 Done Done New false
10813 3509 Acked 4508 Read 3 Done Done New false
10814 3510 Acked 4509 Read 3 Done Done New false
10815 3511 Acked 4510 Read 10 Done Done New false
10816 3512 Acked 4511 Read 4 Done Done New false
10817 3513 Acked 4512 Write 1 Done Done New false
10818 3514 Acked 4513 Read 1 Done Done New false
10819 3515 Acked 4514 Read 10 Done Done New false
10820 3516 Acked 4515 Read 6 Done Done New false
10821 3517 Acked 4516 Read 10 Done Done New false
10822 3518 Acked 4517 Read 7 Done Done New false
10823 3519 Acked 4518 Write 4 Done Done New false
10824 3520 Acked 4519 Flush 0 Done Done New false
10825 3521 Acked 4520 Write 6 Done Done New false
10826 3522 Acked 4521 Read 1 Done Done New false
10827 3523 Acked 4522 Flush 0 Done Done New false
10828 3524 Acked 4523 Read 5 Done Done New false
10829 3525 Acked 4524 Read 3 Done Done New false
10830 3526 Acked 4525 Write 7 Done Done New false
10831 3527 Acked 4526 Write 6 Done Done New false
10832 3528 Acked 4527 Flush 0 Done Done New false
10833 3529 Acked 4528 Read 2 Done Done New false
10834 3530 Acked 4529 Write 3 Done Done New false
10835 3531 Acked 4530 Write 9 Done Done New false
10836 3532 Acked 4531 Write 7 Done Done New false
10837 3533 Acked 4532 Write 9 Done Done New false
10838 3534 Acked 4533 Read 3 Done Done New false
10839 3535 Acked 4534 Read 5 Done Done New false
10840 3536 Acked 4535 Read 9 Done Done New false
10841 3537 Acked 4536 Flush 0 Done Done New false
10842 3538 Acked 4537 Read 6 Done Done New false
10843 3539 Acked 4538 Write 5 Done Done New false
10844 3540 Acked 4539 Read 8 Done Done New false
10845 3541 Acked 4540 Read 8 Done Done New false
10846 3542 Acked 4541 Read 7 Done Done New false
10847 3543 Acked 4542 Write 2 Done Done New false
10848 3544 Acked 4543 Read 8 Done Done New false
10849 3545 Acked 4544 Write 7 Done Done New false
10850 3546 Acked 4545 Write 6 Done Done New false
10851 3547 Acked 4546 Read 7 Done Done New false
10852 3548 Acked 4547 Write 1 Done Done New false
10853 3549 Acked 4548 Flush 0 Done Done New false
10854 3550 Acked 4549 Read 3 Done Done New false
10855 3551 Acked 4550 Flush 0 Done Done New false
10856 3552 Acked 4551 Write 5 Done Done New false
10857 3553 Acked 4552 Write 7 Done Done New false
10858 3554 Acked 4553 Read 3 Done Done New false
10859 3555 Acked 4554 Read 8 Done Done New false
10860 3556 Acked 4555 Write 2 Done Done New false
10861 3557 Acked 4556 Write 5 Done Done New false
10862 3558 Acked 4557 Read 9 Done Done New false
10863 3559 Acked 4558 Read 2 Done Done New false
10864 3560 Acked 4559 Read 2 Done Done New false
10865 3561 Acked 4560 Write 5 Done Done New false
10866 3562 Acked 4561 Read 3 Done Done New false
10867 3563 Acked 4562 Flush 0 Done Done New false
10868 3564 Acked 4563 Write 7 Done Done New false
10869 3565 Acked 4564 Write 9 Done Done New false
10870 3566 Acked 4565 Read 8 Done Done New false
10871 3567 Acked 4566 Write 8 Done Done New false
10872 3568 Acked 4567 Read 2 Done Done New false
10873 3569 Acked 4568 Read 9 Done Done New false
10874 3570 Acked 4569 Read 7 Done Done New false
10875 3571 Acked 4570 Write 7 Done Done New false
10876 3572 Acked 4571 Read 10 Done Done New false
10877 3573 Acked 4572 Read 6 Done Done New false
10878 3574 Acked 4573 Read 7 Done Done New false
10879 3575 Acked 4574 Flush 0 Done Done New false
10880 3576 Acked 4575 Read 9 Done Done New false
10881 3577 Acked 4576 Read 7 Done Done New false
10882 3578 Acked 4577 Read 7 Done Done New false
10883 3579 Acked 4578 Write 6 Done Done New false
10884 3580 Acked 4579 Write 5 Done Done New false
10885 3581 Acked 4580 Write 6 Done Done New false
10886 3582 Acked 4581 Write 5 Done Done New false
10887 3583 Acked 4582 Write 8 Done Done New false
10888 3584 Acked 4583 Write 6 Done Done New false
10889 3585 Acked 4584 Flush 0 Done Done New false
10890 3586 Acked 4585 Write 7 Done Done New false
10891 3587 Acked 4586 Read 6 Done Done New false
10892 3588 Acked 4587 Write 4 Done Done New false
10893 3589 Acked 4588 Write 1 Done Done New false
10894 3590 Acked 4589 Write 4 Done Done New false
10895 3591 Acked 4590 Write 9 Done Done New false
10896 3592 Acked 4591 Read 9 Done Done New false
10897 3593 Acked 4592 Write 2 Done Done New false
10898 3594 Acked 4593 Write 2 Done Done New false
10899 3595 Acked 4594 Read 7 Done Done New false
10900 3596 Acked 4595 Read 10 Done Done New false
10901 3597 Acked 4596 Read 1 Done Done New false
10902 3598 Acked 4597 Read 10 Done Done New false
10903 3599 Acked 4598 Flush 0 Done Done New false
10904 3600 Acked 4599 Write 10 Done Done New false
10905 3601 Acked 4600 Write 8 Done Done New false
10906 3602 Acked 4601 Write 1 Done Done New false
10907 3603 Acked 4602 Write 4 Done Done New false
10908 3604 Acked 4603 Write 2 Done Done New false
10909 3605 Acked 4604 Read 8 Done Done New false
10910 3606 Acked 4605 Read 6 Done Done New false
10911 3607 Acked 4606 Read 1 Done Done New false
10912 3608 Acked 4607 Write 4 Done Done New false
10913 3609 Acked 4608 Read 8 Done Done New false
10914 3610 Acked 4609 Write 7 Done Done New false
10915 3611 Acked 4610 Flush 0 Done Done New false
10916 3612 Acked 4611 Read 9 Done Done New false
10917 3613 Acked 4612 Read 3 Done Done New false
10918 3614 Acked 4613 Flush 0 Done Done New false
10919 3615 Acked 4614 Read 8 Done Done New false
10920 3616 Acked 4615 Write 2 Done Done New false
10921 STATES DS:0 DS:1 DS:2 TOTAL
10922 New 0 0 301 301
10923 Sent 0 0 0 0
10924 Done 301 301 0 602
10925 Skipped 0 0 0 0
10926 Error 0 0 0 0
10927 Last Flush: 4613 4613 4314
10928 Downstairs last five completed: 4314 4313 4312 4311 4310
10929 Upstairs last five completed: 3616 3615 3614 3613 3612
10930 CLIENT: Up:0 ds:301 act:2
109312024-02-27T21:46:46.868ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
109322024-02-27T21:46:46.868ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
109332024-02-27T21:46:46.868ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
109342024-02-27T21:46:46.868ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
109352024-02-27T21:46:46.868ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
109362024-02-27T21:46:46.868ZINFOcrucible: [2] client re-new 302 jobs since flush 4314 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
109372024-02-27T21:46:46.868ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
10938 [8830] got start action
10939 [8830] I am going to restart
10940 Starting downstairs at port 8830
10941 Make output file at "/tmp/dsc/downstairs-8830.txt"
10942 [8830][2] reports Starting
109432024-02-27T21:46:47.869ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
109442024-02-27T21:46:47.869ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
109452024-02-27T21:46:47.869ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
109462024-02-27T21:46:47.869ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
109472024-02-27T21:46:47.869ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
109482024-02-27T21:46:47.869ZINFOcrucible: [2] client re-new 302 jobs since flush 4314 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
109492024-02-27T21:46:47.869ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
10950 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1179)
10951 [8830][2] reports Running
109522024-02-27T21:46:48.870ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
109532024-02-27T21:46:48.870ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
109542024-02-27T21:46:48.872ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
109552024-02-27T21:46:48.872ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
109562024-02-27T21:46:48.872ZINFOcrucible: send last flush ID to this DS: 4314 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
109572024-02-27T21:46:48.872ZINFOcrucible: Replied this last flush ID: 4314 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
109582024-02-27T21:46:48.872ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
109592024-02-27T21:46:48.872ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
109602024-02-27T21:46:48.872ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
109612024-02-27T21:46:48.872ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
10962 ----------------------------------------------------------------
10963 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10964 Downstairs last five completed: 4616 4615 4614 4613 4612
10965 Upstairs last five completed: 3617 3616 3615 3614 3613
10966 CLIENT: Up:0 ds:0 act:3
10967 Replay: All jobs finished, all DS active.
10968 Main task has work to do, go find it
10969 got dsc Stop(0)
10970 stop 0
10971 Feb 27 21:46:50.206 INFO request completed, latency_us: 369, response_code: 204[8810] Got stop action so:false kr:false
10972 , uri: /stop/cid/0, method: POST, req_id: 6f20fc54-ac0a-48f2-91f2-92ec15063197, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
10973 Feb 27 21:46:50.206 INFO accepted connection, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
109742024-02-27T21:46:50.207ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
109752024-02-27T21:46:50.207ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
109762024-02-27T21:46:50.207ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = e6a5a974-925a-494f-8845-facc441c2690
109772024-02-27T21:46:50.207ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
109782024-02-27T21:46:50.207ZINFOcrucible: [0] client re-new 0 jobs since flush 4616 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
109792024-02-27T21:46:50.207ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
10980 Feb 27 21:46:50.207 INFO request completed, latency_us: 155, response_code: 200, uri: /state/cid/0, method: GET, req_id: 73f4d82d-b484-425d-8361-2dc6f6c98250, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
10981 [8810] Exited with: ExitStatus(unix_wait_status(9))
10982 [8810][0] reports Exit
109832024-02-27T21:46:51.208ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
109842024-02-27T21:46:51.208ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
109852024-02-27T21:46:51.208ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
109862024-02-27T21:46:51.208ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
109872024-02-27T21:46:51.208ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
109882024-02-27T21:46:51.208ZINFOcrucible: [0] client re-new 0 jobs since flush 4616 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
109892024-02-27T21:46:51.209ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
109902024-02-27T21:46:52.209ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
109912024-02-27T21:46:52.209ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
109922024-02-27T21:46:52.209ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
109932024-02-27T21:46:52.209ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
109942024-02-27T21:46:52.209ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
109952024-02-27T21:46:52.209ZINFOcrucible: [0] client re-new 0 jobs since flush 4616 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
109962024-02-27T21:46:52.209ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
109972024-02-27T21:46:53.210ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
109982024-02-27T21:46:53.210ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
109992024-02-27T21:46:53.210ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
110002024-02-27T21:46:53.210ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
110012024-02-27T21:46:53.210ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
110022024-02-27T21:46:53.210ZINFOcrucible: [0] client re-new 0 jobs since flush 4616 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
110032024-02-27T21:46:53.210ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
11004 Feb 27 21:46:54.209 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/0, method: GET, req_id: 048506ef-476a-4eae-89cc-3ecfbffb16e9, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
11005 001/300 Write block 4324 len 28672 data: 3 3 3 4 5 4 3
11006 002/300 Read block 861 len 8192
110072024-02-27T21:46:54.211ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
110082024-02-27T21:46:54.211ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
110092024-02-27T21:46:54.211ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
110102024-02-27T21:46:54.211ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
110112024-02-27T21:46:54.211ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
110122024-02-27T21:46:54.211ZINFOcrucible: [0] client re-new 2 jobs since flush 4616 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
110132024-02-27T21:46:54.211ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
11014 003/300 Read block 4069 len 16384
11015 004/300 Read block 4194 len 20480
11016 005/300 Write block 2197 len 24576 data: 5 5 5 5 3 4
11017 006/300 Read block 266 len 4096
11018 007/300 Read block 3165 len 16384
11019 008/300 Read block 2215 len 36864
11020 009/300 Write block 1800 len 36864 data: 3 3 3 3 3 3 3 3 3
11021 010/300 Write block 1039 len 28672 data: 3 3 2 2 2 2 3
11022 011/300 Read block 3227 len 16384
11023 012/300 Read block 4455 len 8192
11024 013/300 Read block 1302 len 40960
11025 014/300 Read block 187 len 20480
11026 015/300 Read block 4156 len 24576
11027 016/300 Read block 1759 len 20480
11028 017/300 Read block 3630 len 40960
11029 018/300 Write block 516 len 32768 data: 3 5 4 4 4 4 4 5
11030 019/300 Write block 4586 len 24576 data: 3 2 4 3 3 3
11031 020/300 Write block 1321 len 24576 data: 6 5 6 6 5 4
11032 021/300 Read block 3070 len 36864
11033 022/300 Write block 763 len 20480 data: 4 4 5 5 4
11034 023/300 Flush
11035 024/300 Read block 4490 len 28672
11036 025/300 Read block 2814 len 40960
11037 026/300 Read block 1914 len 40960
11038 027/300 Write block 4663 len 16384 data: 5 5 7 7
11039 028/300 Write block 3007 len 24576 data: 2 3 3 3 3 3
11040 029/300 Flush
11041 030/300 Read block 2186 len 40960
11042 031/300 Write block 4826 len 24576 data: 5 5 4 5 5 4
11043 032/300 Write block 235 len 32768 data: 2 2 2 2 2 2 2 2
11044 033/300 Read block 4157 len 4096
11045 034/300 Read block 353 len 24576
11046 035/300 Write block 3590 len 32768 data: 3 3 4 4 3 4 4 4
11047 036/300 Write block 4809 len 16384 data: 3 3 3 3
11048 037/300 Write block 1364 len 20480 data: 6 5 5 5 4
11049 038/300 Write block 3029 len 4096 data: 6
11050 039/300 Read block 3080 len 20480
11051 040/300 Read block 3204 len 8192
11052 041/300 Flush
11053 042/300 Write block 3164 len 32768 data: 5 5 4 3 3 4 4 3
11054 043/300 Read block 402 len 28672
11055 044/300 Read block 2241 len 4096
11056 045/300 Read block 4470 len 24576
11057 046/300 Write block 2615 len 16384 data: 6 4 4 4
11058 047/300 Write block 1453 len 36864 data: 3 3 3 4 4 5 3 4 4
11059 048/300 Read block 886 len 24576
11060 049/300 Read block 841 len 24576
11061 050/300 Write block 3732 len 16384 data: 2 2 2 2
11062 051/300 Flush
11063 052/300 Read block 4978 len 4096
11064 053/300 Write block 1161 len 12288 data: 4 3 3
11065 054/300 Read block 1617 len 36864
11066 055/300 Write block 2002 len 24576 data: 5 3 3 3 3 4
11067 056/300 Read block 2756 len 32768
11068 057/300 Read block 4034 len 40960
11069 058/300 Write block 2910 len 36864 data: 6 6 5 5 5 4 3 3 2
11070 059/300 Write block 1929 len 40960 data: 6 6 5 5 4 5 5 4 4 4
11071 060/300 Read block 2170 len 12288
11072 061/300 Read block 2054 len 20480
11073 062/300 Write block 1684 len 12288 data: 3 3 3
11074 063/300 Write block 3174 len 36864 data: 3 5 5 5 4 3 4 6 5
11075 064/300 Read block 4437 len 24576
11076 065/300 Read block 1481 len 12288
11077 066/300 Read block 3375 len 32768
11078 067/300 Read block 4734 len 12288
11079 068/300 Flush
11080 069/300 Write block 3739 len 8192 data: 2 2
11081 070/300 Flush
11082 071/300 Flush
11083 072/300 Write block 2067 len 36864 data: 3 2 2 2 2 3 3 3 4
11084 073/300 Read block 1577 len 40960
11085 074/300 Read block 3461 len 4096
11086 075/300 Read block 3091 len 16384
11087 076/300 Write block 3387 len 28672 data: 4 4 3 3 2 3 3
11088 077/300 Read block 4961 len 20480
11089 078/300 Write block 3855 len 36864 data: 4 4 3 2 2 2 2 4 4
11090 079/300 Read block 4890 len 4096
11091 080/300 Write block 4939 len 20480 data: 2 2 4 4 4
11092 081/300 Read block 4876 len 12288
11093 082/300 Read block 517 len 12288
11094 083/300 Read block 4193 len 40960
11095 084/300 Write block 2999 len 36864 data: 5 6 5 5 3 2 2 2 3
11096 085/300 Read block 4000 len 32768
11097 086/300 Read block 4568 len 8192
11098 087/300 Read block 2440 len 24576
11099 088/300 Flush
11100 089/300 Read block 4927 len 36864
11101 090/300 Write block 142 len 4096 data: 3
11102 091/300 Flush
11103 092/300 Read block 1210 len 8192
11104 093/300 Flush
11105 094/300 Read block 1362 len 4096
11106 095/300 Write block 1501 len 16384 data: 3 3 3 3
11107 096/300 Read block 1759 len 24576
11108 097/300 Write block 3638 len 12288 data: 4 4 3
11109 098/300 Flush
11110 099/300 Read block 94 len 8192
11111 100/300 Flush
11112 101/300 Write block 905 len 8192 data: 2 3
11113 102/300 Write block 2410 len 16384 data: 3 3 3 3
11114 103/300 Write block 2813 len 36864 data: 4 4 5 6 5 6 6 5 5
11115 104/300 Write block 3183 len 12288 data: 4 3 3
11116 105/300 Read block 2609 len 28672
11117 106/300 Write block 2745 len 16384 data: 2 2 2 2
11118 107/300 Read block 2826 len 36864
11119 108/300 Flush
11120 109/300 Read block 561 len 4096
11121 110/300 Read block 2938 len 40960
11122 111/300 Read block 2649 len 28672
11123 112/300 Write block 1143 len 8192 data: 6 5
11124 113/300 Write block 4812 len 16384 data: 4 5 5 5
11125 114/300 Read block 2505 len 20480
11126 115/300 Write block 124 len 4096 data: 4
11127 116/300 Read block 2561 len 16384
11128 117/300 Write block 64 len 20480 data: 2 3 3 3 3
11129 118/300 Read block 1093 len 40960
11130 119/300 Flush
11131 120/300 Write block 10 len 36864 data: 2 3 3 3 3 3 5 5 5
11132 121/300 Read block 3587 len 20480
11133 122/300 Write block 2669 len 12288 data: 6 6 5
11134 123/300 Read block 2897 len 36864
11135 124/300 Read block 33 len 24576
11136 125/300 Read block 2705 len 16384
11137 126/300 Write block 3093 len 28672 data: 2 3 3 3 4 4 6
11138 127/300 Write block 2000 len 16384 data: 3 3 6 4
11139 128/300 Flush
11140 129/300 Write block 206 len 8192 data: 6 6
11141 130/300 Flush
11142 131/300 Write block 1208 len 28672 data: 3 3 2 2 2 2 2
11143 132/300 Read block 1698 len 4096
11144 133/300 Read block 478 len 28672
11145 134/300 Read block 592 len 36864
11146 135/300 Write block 2084 len 8192 data: 4 3
11147 136/300 Read block 754 len 4096
11148 137/300 Read block 455 len 8192
11149 138/300 Read block 1373 len 28672
11150 139/300 Read block 2324 len 4096
11151 140/300 Read block 3561 len 16384
11152 141/300 Read block 2251 len 12288
11153 142/300 Flush
11154 143/300 Write block 600 len 40960 data: 3 4 4 4 3 2 2 4 4 4
11155 144/300 Write block 2006 len 4096 data: 4
11156 145/300 Flush
11157 146/300 Write block 2240 len 28672 data: 3 4 4 4 3 3 3
11158 147/300 Read block 2255 len 36864
11159 148/300 Write block 963 len 8192 data: 3 3
11160 149/300 Write block 4593 len 24576 data: 3 3 4 5 6 5
11161 150/300 Write block 2811 len 40960 data: 3 3 5 5 6 7 6 7 7 6
11162 151/300 Write block 1154 len 28672 data: 3 3 3 4 4 4 4
11163 152/300 Read block 1229 len 20480
11164 153/300 Read block 1710 len 28672
11165 154/300 Read block 999 len 20480
11166 155/300 Read block 4546 len 12288
11167 156/300 Read block 4192 len 8192
11168 157/300 Read block 1641 len 20480
11169 158/300 Flush
11170 159/300 Flush
11171 160/300 Write block 1573 len 28672 data: 2 2 2 2 2 3 3
11172 161/300 Read block 310 len 32768
11173 162/300 Read block 750 len 4096
11174 163/300 Read block 4814 len 4096
11175 164/300 Read block 509 len 32768
11176 165/300 Flush
11177 166/300 Read block 2697 len 32768
11178 167/300 Write block 3871 len 12288 data: 3 2 3
11179 168/300 Read block 2044 len 40960
11180 169/300 Read block 4212 len 12288
11181 170/300 Write block 821 len 28672 data: 2 2 2 3 3 3 3
11182 171/300 Flush
11183 172/300 Read block 2461 len 4096
11184 173/300 Write block 510 len 36864 data: 6 6 6 5 3 3 4 6 5
11185 174/300 Read block 869 len 8192
11186 175/300 Flush
11187 176/300 Flush
11188 177/300 Read block 440 len 24576
11189 178/300 Read block 965 len 40960
11190 179/300 Write block 4676 len 24576 data: 3 3 4 4 4 3
11191 180/300 Write block 2810 len 24576 data: 4 4 4 6 6 7
11192 181/300 Read block 1502 len 40960
11193 182/300 Read block 2602 len 28672
11194 183/300 Read block 3968 len 4096
11195 184/300 Read block 4525 len 28672
11196 185/300 Read block 3924 len 20480
11197 186/300 Read block 2984 len 20480
11198 187/300 Write block 668 len 40960 data: 3 3 3 3 3 3 4 4 5 5
11199 188/300 Read block 2613 len 8192
11200 189/300 Read block 2018 len 20480
11201 190/300 Write block 1683 len 32768 data: 3 4 4 4 3 3 3 3
11202 191/300 Read block 440 len 12288
11203 192/300 Write block 2708 len 20480 data: 3 4 2 4 4
11204 193/300 Read block 3139 len 24576
11205 194/300 Flush
11206 195/300 Read block 4273 len 20480
11207 196/300 Read block 2120 len 32768
11208 197/300 Write block 4487 len 4096 data: 6
11209 198/300 Read block 1606 len 36864
11210 199/300 Write block 4192 len 8192 data: 4 3
11211 200/300 Write block 624 len 28672 data: 3 3 3 3 2 2 3
11212 201/300 Write block 4513 len 8192 data: 3 3
11213 202/300 Write block 149 len 28672 data: 3 3 4 4 3 3 3
11214 203/300 Read block 55 len 16384
11215 204/300 Flush
11216 205/300 Read block 15 len 36864
11217 206/300 Read block 3448 len 40960
11218 207/300 Read block 4878 len 16384
11219 208/300 Write block 2055 len 32768 data: 4 3 4 3 3 3 3 4
11220 209/300 Read block 934 len 8192
11221 210/300 Read block 2345 len 40960
11222 211/300 Read block 2474 len 8192
11223 212/300 Read block 3096 len 36864
11224 213/300 Read block 3765 len 20480
11225 214/300 Read block 1490 len 40960
11226 215/300 Flush
11227 216/300 Write block 3558 len 28672 data: 5 5 5 4 5 5 2
11228 217/300 Write block 751 len 28672 data: 3 3 3 3 3 2 2
11229 218/300 Write block 2004 len 20480 data: 4 4 5 5 4
11230 219/300 Read block 1612 len 16384
11231 220/300 Write block 922 len 16384 data: 3 3 3 3
11232 221/300 Read block 1 len 16384
11233 222/300 Read block 3234 len 32768
11234 223/300 Read block 3666 len 8192
11235 224/300 Write block 822 len 36864 data: 3 3 4 4 4 4 4 4 4
11236 225/300 Read block 4541 len 16384
11237 226/300 Flush
11238 227/300 Read block 3023 len 12288
11239 228/300 Read block 3997 len 4096
11240 229/300 Read block 1380 len 20480
11241 230/300 Write block 1618 len 40960 data: 4 5 5 6 6 4 4 4 4 4
11242 231/300 Write block 3511 len 32768 data: 3 2 2 3 3 4 4 4
11243 232/300 Write block 4736 len 40960 data: 6 6 7 6 6 6 5 4 6 6
11244 233/300 Read block 3948 len 4096
11245 234/300 Read block 3281 len 8192
11246 235/300 Write block 1194 len 12288 data: 3 3 4
11247 236/300 Write block 512 len 4096 data: 7
11248 237/300 Write block 1313 len 40960 data: 5 5 6 5 5 6 7 7 7 6
11249 238/300 Write block 355 len 32768 data: 2 2 3 3 3 3 3 4
11250 239/300 Read block 2436 len 40960
11251 240/300 Write block 3629 len 16384 data: 2 2 2 3
11252 241/300 Write block 371 len 40960 data: 3 3 3 3 2 2 2 2 2 2
11253 242/300 Read block 2020 len 36864
11254 243/300 Flush
11255 244/300 Read block 1126 len 8192
11256 245/300 Write block 3757 len 4096 data: 3
11257 246/300 Write block 791 len 28672 data: 3 3 4 4 3 3 3
11258 247/300 Write block 72 len 28672 data: 4 4 4 3 2 2 2
11259 248/300 Write block 2450 len 8192 data: 4 4
11260 249/300 Read block 4316 len 20480
11261 250/300 Read block 3316 len 16384
11262 251/300 Read block 4692 len 40960
11263 252/300 Read block 4218 len 28672
11264 253/300 Flush
11265 254/300 Read block 1481 len 12288
11266 255/300 Read block 2504 len 36864
11267 256/300 Write block 1130 len 20480 data: 5 6 7 5 4
11268 257/300 Read block 1750 len 12288
11269 258/300 Read block 1494 len 32768
11270 259/300 Read block 2744 len 32768
11271 260/300 Read block 1583 len 4096
11272 261/300 Read block 137 len 32768
11273 262/300 Read block 3467 len 28672
11274 263/300 Write block 1007 len 4096 data: 3
11275 264/300 Read block 269 len 12288
11276 265/300 Read block 3260 len 40960
11277 266/300 Read block 4369 len 16384
11278 267/300 Read block 1999 len 36864
11279 268/300 Write block 1995 len 12288 data: 4 5 5
11280 269/300 Read block 2830 len 12288
11281 270/300 Read block 3909 len 28672
11282 271/300 Flush
11283 272/300 Write block 4473 len 4096 data: 2
11284 273/300 Write block 2074 len 12288 data: 4 5 4
11285 274/300 Read block 2173 len 28672
11286 275/300 Write block 955 len 24576 data: 4 5 5 6 6 4
11287 276/300 Read block 1058 len 20480
11288 277/300 Write block 3723 len 20480 data: 4 5 4 4 2
11289 278/300 Read block 2322 len 24576
11290 279/300 Read block 3828 len 16384
11291 280/300 Write block 4886 len 40960 data: 4 4 3 3 2 3 4 4 4 3
11292 281/300 Write block 3554 len 40960 data: 3 5 6 5 6 6 6 5 6 6
11293 282/300 Write block 3191 len 28672 data: 2 2 3 3 3 3 3
11294 283/300 Read block 4769 len 40960
11295 284/300 Read block 4924 len 12288
11296 285/300 Read block 274 len 32768
11297 286/300 Read block 4136 len 28672
11298 287/300 Read block 3557 len 24576
11299 288/300 Read block 4785 len 20480
11300 289/300 Write block 1593 len 28672 data: 3 4 4 3 2 3 4
11301 290/300 Read block 3957 len 36864
11302 291/300 Write block 3151 len 20480 data: 4 4 4 3 3
11303 292/300 Write block 1299 len 24576 data: 2 2 2 3 3 3
11304 293/300 Read block 202 len 20480
11305 294/300 Write block 3774 len 24576 data: 3 3 3 3 3 3
11306 295/300 Write block 3223 len 12288 data: 4 3 3
11307 296/300 Read block 2400 len 12288
11308 297/300 Write block 1154 len 8192 data: 4 4
11309 298/300 Read block 810 len 20480
11310 299/300 Read block 4826 len 16384
11311 300/300 Read block 3756 len 32768
11312 Main task has work to do, go find it
11313 got dsc Start(0)
11314 start 0
11315 Feb 27 21:46:54.553 INFO request completed, latency_us: 236, response_code: 204, uri: /start/cid/0, method: POST, req_id: d941dc43-36f6-4a72-870e-7e830d35e7a7, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
11316 Replay: started 0, returned:Ok(())
11317 ----------------------------------------------------------------
11318 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
11319 States: Offline Active Active
11320 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
11321 3618 Acked 4617 Write 7 New Done Done true
11322 3619 Acked 4618 Read 2 New Done Done true
11323 3620 Acked 4619 Read 4 New Done Done false
11324 3621 Acked 4620 Read 5 New Done Done false
11325 3622 Acked 4621 Write 6 New Done Done false
11326 3623 Acked 4622 Read 1 New Done Done false
11327 3624 Acked 4623 Read 4 New Done Done false
11328 3625 Acked 4624 Read 9 New Done Done false
11329 3626 Acked 4625 Write 9 New Done Done false
11330 3627 Acked 4626 Write 7 New Done Done false
11331 3628 Acked 4627 Read 4 New Done Done false
11332 3629 Acked 4628 Read 2 New Done Done false
11333 3630 Acked 4629 Read 10 New Done Done false
11334 3631 Acked 4630 Read 5 New Done Done false
11335 3632 Acked 4631 Read 6 New Done Done false
11336 3633 Acked 4632 Read 5 New Done Done false
11337 3634 Acked 4633 Read 10 New Done Done false
11338 3635 Acked 4634 Write 8 New Done Done false
11339 3636 Acked 4635 Write 6 New Done Done false
11340 3637 Acked 4636 Write 6 New Done Done false
11341 3638 Acked 4637 Read 9 New Done Done false
11342 3639 Acked 4638 Write 5 New Done Done false
11343 3640 Acked 4639 Flush 0 New Done Done false
11344 3641 Acked 4640 Read 7 New Done Done false
11345 3642 Acked 4641 Read 10 New Done Done false
11346 3643 Acked 4642 Read 10 New Done Done false
11347 3644 Acked 4643 Write 4 New Done Done false
11348 3645 Acked 4644 Write 6 New Done Done false
11349 3646 Acked 4645 Flush 0 New Done Done false
11350 3647 Acked 4646 Read 10 New Done Done false
11351 3648 Acked 4647 Write 6 New Done Done false
11352 3649 Acked 4648 Write 8 New Done Done false
11353 3650 Acked 4649 Read 1 New Done Done false
11354 3651 Acked 4650 Read 6 New Done Done false
11355 3652 Acked 4651 Write 8 New Done Done false
11356 3653 Acked 4652 Write 4 New Done Done false
11357 3654 Acked 4653 Write 5 New Done Done false
11358 3655 Acked 4654 Write 1 New Done Done false
11359 3656 Acked 4655 Read 5 New Done Done false
11360 3657 Acked 4656 Read 2 New Done Done false
11361 3658 Acked 4657 Flush 0 New Done Done false
11362 3659 Acked 4658 Write 8 New Done Done false
11363 3660 Acked 4659 Read 7 New Done Done false
11364 3661 Acked 4660 Read 1 New Done Done false
11365 3662 Acked 4661 Read 6 New Done Done false
11366 3663 Acked 4662 Write 4 New Done Done false
11367 3664 Acked 4663 Write 9 New Done Done false
11368 3665 Acked 4664 Read 6 New Done Done false
11369 3666 Acked 4665 Read 6 New Done Done false
11370 3667 Acked 4666 Write 4 New Done Done false
11371 3668 Acked 4667 Flush 0 New Done Done false
11372 3669 Acked 4668 Read 1 New Done Done false
11373 3670 Acked 4669 Write 3 New Done Done false
11374 3671 Acked 4670 Read 9 New Done Done false
11375 3672 Acked 4671 Write 6 New Done Done false
11376 3673 Acked 4672 Read 8 New Done Done false
11377 3674 Acked 4673 Read 10 New Done Done false
11378 3675 Acked 4674 Write 9 New Done Done false
11379 3676 Acked 4675 Write 10 New Done Done false
11380 3677 Acked 4676 Read 3 New Done Done false
11381 3678 Acked 4677 Read 5 New Done Done false
11382 3679 Acked 4678 Write 3 New Done Done false
11383 3680 Acked 4679 Write 9 New Done Done false
11384 3681 Acked 4680 Read 6 New Done Done false
11385 3682 Acked 4681 Read 3 New Done Done false
11386 3683 Acked 4682 Read 8 New Done Done false
11387 3684 Acked 4683 Read 3 New Done Done false
11388 3685 Acked 4684 Flush 0 New Done Done false
11389 3686 Acked 4685 Write 2 New Done Done false
11390 3687 Acked 4686 Flush 0 New Done Done false
11391 3688 Acked 4687 Flush 0 New Done Done false
11392 3689 Acked 4688 Write 9 New Done Done false
11393 3690 Acked 4689 Read 10 New Done Done false
11394 3691 Acked 4690 Read 1 New Done Done false
11395 3692 Acked 4691 Read 4 New Done Done false
11396 3693 Acked 4692 Write 7 New Done Done false
11397 3694 Acked 4693 Read 5 New Done Done false
11398 3695 Acked 4694 Write 9 New Done Done false
11399 3696 Acked 4695 Read 1 New Done Done false
11400 3697 Acked 4696 Write 5 New Done Done false
11401 3698 Acked 4697 Read 3 New Done Done false
11402 3699 Acked 4698 Read 3 New Done Done false
11403 3700 Acked 4699 Read 10 New Done Done false
11404 3701 Acked 4700 Write 9 New Done Done false
11405 3702 Acked 4701 Read 8 New Done Done false
11406 3703 Acked 4702 Read 2 New Done Done false
11407 3704 Acked 4703 Read 6 New Done Done false
11408 3705 Acked 4704 Flush 0 New Done Done false
11409 3706 Acked 4705 Read 9 New Done Done false
11410 3707 Acked 4706 Write 1 New Done Done false
11411 3708 Acked 4707 Flush 0 New Done Done false
11412 3709 Acked 4708 Read 2 New Done Done false
11413 3710 Acked 4709 Flush 0 New Done Done false
11414 3711 Acked 4710 Read 1 New Done Done false
11415 3712 Acked 4711 Write 4 New Done Done false
11416 3713 Acked 4712 Read 6 New Done Done false
11417 3714 Acked 4713 Write 3 New Done Done false
11418 3715 Acked 4714 Flush 0 New Done Done false
11419 3716 Acked 4715 Read 2 New Done Done false
11420 3717 Acked 4716 Flush 0 New Done Done false
11421 3718 Acked 4717 Write 2 New Done Done false
11422 3719 Acked 4718 Write 4 New Done Done false
11423 3720 Acked 4719 Write 9 New Done Done false
11424 3721 Acked 4720 Write 3 New Done Done false
11425 3722 Acked 4721 Read 7 New Done Done false
11426 3723 Acked 4722 Write 4 New Done Done false
11427 3724 Acked 4723 Read 9 New Done Done false
11428 3725 Acked 4724 Flush 0 New Done Done false
11429 3726 Acked 4725 Read 1 New Done Done false
11430 3727 Acked 4726 Read 10 New Done Done false
11431 3728 Acked 4727 Read 7 New Done Done false
11432 3729 Acked 4728 Write 2 New Done Done false
11433 3730 Acked 4729 Write 4 New Done Done false
11434 3731 Acked 4730 Read 5 New Done Done false
11435 3732 Acked 4731 Write 1 New Done Done false
11436 3733 Acked 4732 Read 4 New Done Done false
11437 3734 Acked 4733 Write 5 New Done Done false
11438 3735 Acked 4734 Read 10 New Done Done false
11439 3736 Acked 4735 Flush 0 New Done Done false
11440 3737 Acked 4736 Write 9 New Done Done false
11441 3738 Acked 4737 Read 5 New Done Done false
11442 3739 Acked 4738 Write 3 New Done Done false
11443 3740 Acked 4739 Read 9 New Done Done false
11444 3741 Acked 4740 Read 6 New Done Done false
11445 3742 Acked 4741 Read 4 New Done Done false
11446 3743 Acked 4742 Write 7 New Done Done false
11447 3744 Acked 4743 Write 4 New Done Done false
11448 3745 Acked 4744 Flush 0 New Done Done false
11449 3746 Acked 4745 Write 2 New Done Done false
11450 3747 Acked 4746 Flush 0 New Done Done false
11451 3748 Acked 4747 Write 7 New Done Done false
11452 3749 Acked 4748 Read 1 New Done Done false
11453 3750 Acked 4749 Read 7 New Done Done false
11454 3751 Acked 4750 Read 9 New Done Done false
11455 3752 Acked 4751 Write 2 New Done Done false
11456 3753 Acked 4752 Read 1 New Done Done false
11457 3754 Acked 4753 Read 2 New Done Done false
11458 3755 Acked 4754 Read 7 New Done Done false
11459 3756 Acked 4755 Read 1 New Done Done false
11460 3757 Acked 4756 Read 4 New Done Done false
11461 3758 Acked 4757 Read 3 New Done Done false
11462 3759 Acked 4758 Flush 0 New Done Done false
11463 3760 Acked 4759 Write 10 New Done Done false
11464 3761 Acked 4760 Write 1 New Done Done false
11465 3762 Acked 4761 Flush 0 New Done Done false
11466 3763 Acked 4762 Write 7 New Done Done false
11467 3764 Acked 4763 Read 9 New Done Done false
11468 3765 Acked 4764 Write 2 New Done Done false
11469 3766 Acked 4765 Write 6 New Done Done false
11470 3767 Acked 4766 Write 10 New Done Done false
11471 3768 Acked 4767 Write 7 New Done Done false
11472 3769 Acked 4768 Read 5 New Done Done false
11473 3770 Acked 4769 Read 7 New Done Done false
11474 3771 Acked 4770 Read 5 New Done Done false
11475 3772 Acked 4771 Read 3 New Done Done false
11476 3773 Acked 4772 Read 2 New Done Done false
11477 3774 Acked 4773 Read 5 New Done Done false
11478 3775 Acked 4774 Flush 0 New Done Done false
11479 3776 Acked 4775 Flush 0 New Done Done false
11480 3777 Acked 4776 Write 7 New Done Done false
11481 3778 Acked 4777 Read 8 New Done Done false
11482 3779 Acked 4778 Read 1 New Done Done false
11483 3780 Acked 4779 Read 1 New Done Done false
11484 3781 Acked 4780 Read 8 New Done Done false
11485 3782 Acked 4781 Flush 0 New Done Done false
11486 3783 Acked 4782 Read 8 New Done Done false
11487 3784 Acked 4783 Write 3 New Done Done false
11488 3785 Acked 4784 Read 10 New Done Done false
11489 3786 Acked 4785 Read 3 New Done Done false
11490 3787 Acked 4786 Write 7 New Done Done false
11491 3788 Acked 4787 Flush 0 New Done Done false
11492 3789 Acked 4788 Read 1 New Done Done false
11493 3790 Acked 4789 Write 9 New Done Done false
11494 3791 Acked 4790 Read 2 New Done Done false
11495 3792 Acked 4791 Flush 0 New Done Done false
11496 3793 Acked 4792 Flush 0 New Done Done false
11497 3794 Acked 4793 Read 6 New Done Done false
11498 3795 Acked 4794 Read 10 New Done Done false
11499 3796 Acked 4795 Write 6 New Done Done false
11500 3797 Acked 4796 Write 6 New Done Done false
11501 3798 Acked 4797 Read 10 New Done Done false
11502 3799 Acked 4798 Read 7 New Done Done false
11503 3800 Acked 4799 Read 1 New Done Done false
11504 3801 Acked 4800 Read 7 New Done Done false
11505 3802 Acked 4801 Read 5 New Done Done false
11506 3803 Acked 4802 Read 5 New Done Done false
11507 3804 Acked 4803 Write 10 New Done Done false
11508 3805 Acked 4804 Read 2 New Done Done false
11509 3806 Acked 4805 Read 5 New Done Done false
11510 3807 Acked 4806 Write 8 New Done Done false
11511 3808 Acked 4807 Read 3 New Done Done false
11512 3809 Acked 4808 Write 5 New Done Done false
11513 3810 Acked 4809 Read 6 New Done Done false
11514 3811 Acked 4810 Flush 0 New Done Done false
11515 3812 Acked 4811 Read 5 New Done Done false
11516 3813 Acked 4812 Read 8 New Done Done false
11517 3814 Acked 4813 Write 1 New Done Done false
11518 3815 Acked 4814 Read 9 New Done Done false
11519 3816 Acked 4815 Write 2 New Done Done false
11520 3817 Acked 4816 Write 7 New Done Done false
11521 3818 Acked 4817 Write 2 New Done Done false
11522 3819 Acked 4818 Write 7 New Done Done false
11523 3820 Acked 4819 Read 4 New Done Done false
11524 3821 Acked 4820 Flush 0 New Done Done false
11525 3822 Acked 4821 Read 9 New Done Done false
11526 3823 Acked 4822 Read 10 New Done Done false
11527 3824 Acked 4823 Read 4 New Done Done false
11528 3825 Acked 4824 Write 8 New Done Done false
11529 3826 Acked 4825 Read 2 New Done Done false
11530 3827 Acked 4826 Read 10 New Done Done false
11531 3828 Acked 4827 Read 2 New Done Done false
11532 3829 Acked 4828 Read 9 New Done Done false
11533 3830 Acked 4829 Read 5 New Done Done false
11534 3831 Acked 4830 Read 10 New Done Done false
11535 3832 Acked 4831 Flush 0 New Done Done false
11536 3833 Acked 4832 Write 7 New Done Done false
11537 3834 Acked 4833 Write 7 New Done Done false
11538 3835 Acked 4834 Write 5 New Done Done false
11539 3836 Acked 4835 Read 4 New Done Done false
11540 3837 Acked 4836 Write 4 New Done Done false
11541 3838 Acked 4837 Read 4 New Done Done false
11542 3839 Acked 4838 Read 8 New Done Done false
11543 3840 Acked 4839 Read 2 New Done Done false
11544 3841 Acked 4840 Write 9 New Done Done false
11545 3842 Acked 4841 Read 4 New Done Done false
11546 3843 Acked 4842 Flush 0 New Done Done false
11547 3844 Acked 4843 Read 3 New Done Done false
11548 3845 Acked 4844 Read 1 New Done Done false
11549 3846 Acked 4845 Read 5 New Done Done false
11550 3847 Acked 4846 Write 10 New Done Done false
11551 3848 Acked 4847 Write 8 New Done Done false
11552 3849 Acked 4848 Write 10 New Done Done false
11553 3850 Acked 4849 Read 1 New Done Done false
11554 3851 Acked 4850 Read 2 New Done Done false
11555 3852 Acked 4851 Write 3 New Done Done false
11556 3853 Acked 4852 Write 1 New Done Done false
11557 3854 Acked 4853 Write 10 New Done Done false
11558 3855 Acked 4854 Write 8 New Done Done false
11559 3856 Acked 4855 Read 10 New Done Done false
11560 3857 Acked 4856 Write 4 New Done Done false
11561 3858 Acked 4857 Write 10 New Done Done false
11562 3859 Acked 4858 Read 9 New Done Done false
11563 3860 Acked 4859 Flush 0 New Done Done false
11564 3861 Acked 4860 Read 2 New Done Done false
11565 3862 Acked 4861 Write 1 New Done Done false
11566 3863 Acked 4862 Write 7 New Done Done false
11567 3864 Acked 4863 Write 7 New Done Done false
11568 3865 Acked 4864 Write 2 New Done Done false
11569 3866 Acked 4865 Read 5 New Done Done false
11570 3867 Acked 4866 Read 4 New Done Done false
11571 3868 Acked 4867 Read 10 New Done Done false
11572 3869 Acked 4868 Read 7 New Done Done false
11573 3870 Acked 4869 Flush 0 New Done Done false
11574 3871 Acked 4870 Read 3 New Done Done false
11575 3872 Acked 4871 Read 9 New Done Done false
11576 3873 Acked 4872 Write 5 New Done Done false
11577 3874 Acked 4873 Read 3 New Done Done false
11578 3875 Acked 4874 Read 8 New Done Done false
11579 3876 Acked 4875 Read 8 New Done Done false
11580 3877 Acked 4876 Read 1 New Done Done false
11581 3878 Acked 4877 Read 8 New Done Done false
11582 3879 Acked 4878 Read 7 New Done Done false
11583 3880 Acked 4879 Write 1 New Done Done false
11584 3881 Acked 4880 Read 3 New Done Done false
11585 3882 Acked 4881 Read 10 New Done Done false
11586 3883 Acked 4882 Read 4 New Done Done false
11587 3884 Acked 4883 Read 9 New Done Done false
11588 3885 Acked 4884 Write 3 New Done Done false
11589 3886 Acked 4885 Read 3 New Done Done false
11590 3887 Acked 4886 Read 7 New Done Done false
11591 3888 Acked 4887 Flush 0 New Done Done false
11592 3889 Acked 4888 Write 1 New Done Done false
11593 3890 Acked 4889 Write 3 New Done Done false
11594 3891 Acked 4890 Read 7 New Done Done false
11595 3892 Acked 4891 Write 6 New Done Done false
11596 3893 Acked 4892 Read 5 New Done Done false
11597 3894 Acked 4893 Write 5 New Done Done false
11598 3895 Acked 4894 Read 6 New Done Done false
11599 3896 Acked 4895 Read 4 New Done Done false
11600 3897 Acked 4896 Write 10 New Done Done false
11601 3898 Acked 4897 Write 10 New Done Done false
11602 3899 Acked 4898 Write 7 New Done Done false
11603 3900 Acked 4899 Read 10 New Done Done false
11604 3901 Acked 4900 Read 3 New Done Done false
11605 3902 Acked 4901 Read 8 New Done Done false
11606 3903 Acked 4902 Read 7 New Done Done false
11607 3904 Acked 4903 Read 6 New Done Done false
11608 3905 Acked 4904 Read 5 New Done Done false
11609 3906 Acked 4905 Write 7 New Done Done false
11610 3907 Acked 4906 Read 9 New Done Done false
11611 3908 Acked 4907 Write 5 New Done Done false
11612 3909 Acked 4908 Write 6 New Done Done false
11613 3910 Acked 4909 Read 5 New Done Done false
11614 3911 Acked 4910 Write 6 New Done Done false
11615 3912 Acked 4911 Write 3 New Done Done false
11616 3913 Acked 4912 Read 3 New Done Done false
11617 3914 Acked 4913 Write 2 New Done Done false
11618 3915 Acked 4914 Read 5 New Done Done false
11619 3916 Acked 4915 Read 4 New Done Done false
11620 3917 Acked 4916 Read 8 New Done Done false
11621 STATES DS:0 DS:1 DS:2 TOTAL
11622 New 300 0 0 300
11623 Sent 0 0 0 0
11624 Done 0 300 300 600
11625 Skipped 0 0 0 0
11626 Error 0 0 0 0
11627 Last Flush: 4616 4887 4887
11628 Downstairs last five completed: 4616 4615 4614 4613 4612
11629 Upstairs last five completed: 3917 3916 3915 3914 3913
11630 CLIENT: Up:0 ds:300 act:2
116312024-02-27T21:46:55.212ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
116322024-02-27T21:46:55.212ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
116332024-02-27T21:46:55.212ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
116342024-02-27T21:46:55.212ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
116352024-02-27T21:46:55.212ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
116362024-02-27T21:46:55.212ZINFOcrucible: [0] client re-new 301 jobs since flush 4616 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
116372024-02-27T21:46:55.213ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
116382024-02-27T21:46:56.213ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
116392024-02-27T21:46:56.214ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
116402024-02-27T21:46:56.214ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
116412024-02-27T21:46:56.214ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
116422024-02-27T21:46:56.214ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
116432024-02-27T21:46:56.214ZINFOcrucible: [0] client re-new 301 jobs since flush 4616 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
116442024-02-27T21:46:56.214ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
116452024-02-27T21:46:57.215ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
116462024-02-27T21:46:57.216ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
116472024-02-27T21:46:57.216ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
116482024-02-27T21:46:57.216ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
116492024-02-27T21:46:57.216ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
116502024-02-27T21:46:57.216ZINFOcrucible: [0] client re-new 301 jobs since flush 4616 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
116512024-02-27T21:46:57.216ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
11652 [8810] got start action
11653 [8810] I am going to restart
11654 Starting downstairs at port 8810
11655 Make output file at "/tmp/dsc/downstairs-8810.txt"
11656 [8810][0] reports Starting
116572024-02-27T21:46:58.218ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
116582024-02-27T21:46:58.218ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
116592024-02-27T21:46:58.218ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
116602024-02-27T21:46:58.218ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
116612024-02-27T21:46:58.218ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
116622024-02-27T21:46:58.218ZINFOcrucible: [0] client re-new 301 jobs since flush 4616 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
116632024-02-27T21:46:58.218ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
11664 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1180)
11665 [8810][0] reports Running
11666 ----------------------------------------------------------------
11667 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11668 States: Offline Active Active
11669 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
11670 3618 Acked 4617 Write 7 New Done Done true
11671 3619 Acked 4618 Read 2 New Done Done true
11672 3620 Acked 4619 Read 4 New Done Done true
11673 3621 Acked 4620 Read 5 New Done Done true
11674 3622 Acked 4621 Write 6 New Done Done true
11675 3623 Acked 4622 Read 1 New Done Done true
11676 3624 Acked 4623 Read 4 New Done Done true
11677 3625 Acked 4624 Read 9 New Done Done true
11678 3626 Acked 4625 Write 9 New Done Done true
11679 3627 Acked 4626 Write 7 New Done Done true
11680 3628 Acked 4627 Read 4 New Done Done true
11681 3629 Acked 4628 Read 2 New Done Done true
11682 3630 Acked 4629 Read 10 New Done Done true
11683 3631 Acked 4630 Read 5 New Done Done true
11684 3632 Acked 4631 Read 6 New Done Done true
11685 3633 Acked 4632 Read 5 New Done Done true
11686 3634 Acked 4633 Read 10 New Done Done true
11687 3635 Acked 4634 Write 8 New Done Done true
11688 3636 Acked 4635 Write 6 New Done Done true
11689 3637 Acked 4636 Write 6 New Done Done true
11690 3638 Acked 4637 Read 9 New Done Done true
11691 3639 Acked 4638 Write 5 New Done Done true
11692 3640 Acked 4639 Flush 0 New Done Done true
11693 3641 Acked 4640 Read 7 New Done Done true
11694 3642 Acked 4641 Read 10 New Done Done true
11695 3643 Acked 4642 Read 10 New Done Done true
11696 3644 Acked 4643 Write 4 New Done Done true
11697 3645 Acked 4644 Write 6 New Done Done true
11698 3646 Acked 4645 Flush 0 New Done Done true
11699 3647 Acked 4646 Read 10 New Done Done true
11700 3648 Acked 4647 Write 6 New Done Done true
11701 3649 Acked 4648 Write 8 New Done Done true
11702 3650 Acked 4649 Read 1 New Done Done true
11703 3651 Acked 4650 Read 6 New Done Done true
11704 3652 Acked 4651 Write 8 New Done Done true
11705 3653 Acked 4652 Write 4 New Done Done true
11706 3654 Acked 4653 Write 5 New Done Done true
11707 3655 Acked 4654 Write 1 New Done Done true
11708 3656 Acked 4655 Read 5 New Done Done true
11709 3657 Acked 4656 Read 2 New Done Done true
11710 3658 Acked 4657 Flush 0 New Done Done true
11711 3659 Acked 4658 Write 8 New Done Done true
11712 3660 Acked 4659 Read 7 New Done Done true
11713 3661 Acked 4660 Read 1 New Done Done true
11714 3662 Acked 4661 Read 6 New Done Done true
11715 3663 Acked 4662 Write 4 New Done Done true
11716 3664 Acked 4663 Write 9 New Done Done true
11717 3665 Acked 4664 Read 6 New Done Done true
11718 3666 Acked 4665 Read 6 New Done Done true
11719 3667 Acked 4666 Write 4 New Done Done true
11720 3668 Acked 4667 Flush 0 New Done Done true
11721 3669 Acked 4668 Read 1 New Done Done true
11722 3670 Acked 4669 Write 3 New Done Done true
11723 3671 Acked 4670 Read 9 New Done Done true
11724 3672 Acked 4671 Write 6 New Done Done true
11725 3673 Acked 4672 Read 8 New Done Done true
11726 3674 Acked 4673 Read 10 New Done Done true
11727 3675 Acked 4674 Write 9 New Done Done true
11728 3676 Acked 4675 Write 10 New Done Done true
11729 3677 Acked 4676 Read 3 New Done Done true
11730 3678 Acked 4677 Read 5 New Done Done true
11731 3679 Acked 4678 Write 3 New Done Done true
11732 3680 Acked 4679 Write 9 New Done Done true
11733 3681 Acked 4680 Read 6 New Done Done true
11734 3682 Acked 4681 Read 3 New Done Done true
11735 3683 Acked 4682 Read 8 New Done Done true
11736 3684 Acked 4683 Read 3 New Done Done true
11737 3685 Acked 4684 Flush 0 New Done Done true
11738 3686 Acked 4685 Write 2 New Done Done true
11739 3687 Acked 4686 Flush 0 New Done Done true
11740 3688 Acked 4687 Flush 0 New Done Done true
11741 3689 Acked 4688 Write 9 New Done Done true
11742 3690 Acked 4689 Read 10 New Done Done true
11743 3691 Acked 4690 Read 1 New Done Done true
11744 3692 Acked 4691 Read 4 New Done Done true
11745 3693 Acked 4692 Write 7 New Done Done true
11746 3694 Acked 4693 Read 5 New Done Done true
11747 3695 Acked 4694 Write 9 New Done Done true
11748 3696 Acked 4695 Read 1 New Done Done true
11749 3697 Acked 4696 Write 5 New Done Done true
11750 3698 Acked 4697 Read 3 New Done Done true
11751 3699 Acked 4698 Read 3 New Done Done true
11752 3700 Acked 4699 Read 10 New Done Done true
11753 3701 Acked 4700 Write 9 New Done Done true
11754 3702 Acked 4701 Read 8 New Done Done true
11755 3703 Acked 4702 Read 2 New Done Done true
11756 3704 Acked 4703 Read 6 New Done Done true
11757 3705 Acked 4704 Flush 0 New Done Done true
11758 3706 Acked 4705 Read 9 New Done Done true
11759 3707 Acked 4706 Write 1 New Done Done true
11760 3708 Acked 4707 Flush 0 New Done Done true
11761 3709 Acked 4708 Read 2 New Done Done true
11762 3710 Acked 4709 Flush 0 New Done Done true
11763 3711 Acked 4710 Read 1 New Done Done true
11764 3712 Acked 4711 Write 4 New Done Done true
11765 3713 Acked 4712 Read 6 New Done Done true
11766 3714 Acked 4713 Write 3 New Done Done true
11767 3715 Acked 4714 Flush 0 New Done Done true
11768 3716 Acked 4715 Read 2 New Done Done true
11769 3717 Acked 4716 Flush 0 New Done Done true
11770 3718 Acked 4717 Write 2 New Done Done true
11771 3719 Acked 4718 Write 4 New Done Done true
11772 3720 Acked 4719 Write 9 New Done Done true
11773 3721 Acked 4720 Write 3 New Done Done true
11774 3722 Acked 4721 Read 7 New Done Done true
11775 3723 Acked 4722 Write 4 New Done Done true
11776 3724 Acked 4723 Read 9 New Done Done true
11777 3725 Acked 4724 Flush 0 New Done Done true
11778 3726 Acked 4725 Read 1 New Done Done true
11779 3727 Acked 4726 Read 10 New Done Done true
11780 3728 Acked 4727 Read 7 New Done Done true
11781 3729 Acked 4728 Write 2 New Done Done true
11782 3730 Acked 4729 Write 4 New Done Done true
11783 3731 Acked 4730 Read 5 New Done Done true
11784 3732 Acked 4731 Write 1 New Done Done true
11785 3733 Acked 4732 Read 4 New Done Done true
11786 3734 Acked 4733 Write 5 New Done Done true
11787 3735 Acked 4734 Read 10 New Done Done true
11788 3736 Acked 4735 Flush 0 New Done Done true
11789 3737 Acked 4736 Write 9 New Done Done true
11790 3738 Acked 4737 Read 5 New Done Done true
11791 3739 Acked 4738 Write 3 New Done Done true
11792 3740 Acked 4739 Read 9 New Done Done true
11793 3741 Acked 4740 Read 6 New Done Done true
11794 3742 Acked 4741 Read 4 New Done Done true
11795 3743 Acked 4742 Write 7 New Done Done true
11796 3744 Acked 4743 Write 4 New Done Done true
11797 3745 Acked 4744 Flush 0 New Done Done true
11798 3746 Acked 4745 Write 2 New Done Done true
11799 3747 Acked 4746 Flush 0 New Done Done true
11800 3748 Acked 4747 Write 7 New Done Done true
11801 3749 Acked 4748 Read 1 New Done Done true
11802 3750 Acked 4749 Read 7 New Done Done true
11803 3751 Acked 4750 Read 9 New Done Done true
11804 3752 Acked 4751 Write 2 New Done Done true
11805 3753 Acked 4752 Read 1 New Done Done true
11806 3754 Acked 4753 Read 2 New Done Done true
11807 3755 Acked 4754 Read 7 New Done Done true
11808 3756 Acked 4755 Read 1 New Done Done true
11809 3757 Acked 4756 Read 4 New Done Done true
11810 3758 Acked 4757 Read 3 New Done Done true
11811 3759 Acked 4758 Flush 0 New Done Done true
11812 3760 Acked 4759 Write 10 New Done Done true
11813 3761 Acked 4760 Write 1 New Done Done true
11814 3762 Acked 4761 Flush 0 New Done Done true
11815 3763 Acked 4762 Write 7 New Done Done true
11816 3764 Acked 4763 Read 9 New Done Done true
11817 3765 Acked 4764 Write 2 New Done Done true
11818 3766 Acked 4765 Write 6 New Done Done true
11819 3767 Acked 4766 Write 10 New Done Done true
11820 3768 Acked 4767 Write 7 New Done Done true
11821 3769 Acked 4768 Read 5 New Done Done true
11822 3770 Acked 4769 Read 7 New Done Done true
11823 3771 Acked 4770 Read 5 New Done Done true
11824 3772 Acked 4771 Read 3 New Done Done true
11825 3773 Acked 4772 Read 2 New Done Done true
11826 3774 Acked 4773 Read 5 New Done Done true
11827 3775 Acked 4774 Flush 0 New Done Done true
11828 3776 Acked 4775 Flush 0 New Done Done true
11829 3777 Acked 4776 Write 7 New Done Done true
11830 3778 Acked 4777 Read 8 New Done Done true
11831 3779 Acked 4778 Read 1 New Done Done true
11832 3780 Acked 4779 Read 1 New Done Done true
11833 3781 Acked 4780 Read 8 New Done Done true
11834 3782 Acked 4781 Flush 0 New Done Done true
11835 3783 Acked 4782 Read 8 New Done Done true
11836 3784 Acked 4783 Write 3 New Done Done true
11837 3785 Acked 4784 Read 10 New Done Done true
11838 3786 Acked 4785 Read 3 New Done Done true
11839 3787 Acked 4786 Write 7 New Done Done true
11840 3788 Acked 4787 Flush 0 New Done Done true
11841 3789 Acked 4788 Read 1 New Done Done true
11842 3790 Acked 4789 Write 9 New Done Done true
11843 3791 Acked 4790 Read 2 New Done Done true
11844 3792 Acked 4791 Flush 0 New Done Done true
11845 3793 Acked 4792 Flush 0 New Done Done true
11846 3794 Acked 4793 Read 6 New Done Done true
11847 3795 Acked 4794 Read 10 New Done Done true
11848 3796 Acked 4795 Write 6 New Done Done true
11849 3797 Acked 4796 Write 6 New Done Done true
11850 3798 Acked 4797 Read 10 New Done Done true
11851 3799 Acked 4798 Read 7 New Done Done true
11852 3800 Acked 4799 Read 1 New Done Done true
11853 3801 Acked 4800 Read 7 New Done Done true
11854 3802 Acked 4801 Read 5 New Done Done true
11855 3803 Acked 4802 Read 5 New Done Done true
11856 3804 Acked 4803 Write 10 New Done Done true
11857 3805 Acked 4804 Read 2 New Done Done true
11858 3806 Acked 4805 Read 5 New Done Done true
11859 3807 Acked 4806 Write 8 New Done Done true
11860 3808 Acked 4807 Read 3 New Done Done true
11861 3809 Acked 4808 Write 5 New Done Done true
11862 3810 Acked 4809 Read 6 New Done Done true
11863 3811 Acked 4810 Flush 0 New Done Done true
11864 3812 Acked 4811 Read 5 New Done Done true
11865 3813 Acked 4812 Read 8 New Done Done true
11866 3814 Acked 4813 Write 1 New Done Done true
11867 3815 Acked 4814 Read 9 New Done Done true
11868 3816 Acked 4815 Write 2 New Done Done true
11869 3817 Acked 4816 Write 7 New Done Done true
11870 3818 Acked 4817 Write 2 New Done Done true
11871 3819 Acked 4818 Write 7 New Done Done true
11872 3820 Acked 4819 Read 4 New Done Done true
11873 3821 Acked 4820 Flush 0 New Done Done true
11874 3822 Acked 4821 Read 9 New Done Done true
11875 3823 Acked 4822 Read 10 New Done Done true
11876 3824 Acked 4823 Read 4 New Done Done true
11877 3825 Acked 4824 Write 8 New Done Done true
11878 3826 Acked 4825 Read 2 New Done Done true
11879 3827 Acked 4826 Read 10 New Done Done true
11880 3828 Acked 4827 Read 2 New Done Done true
11881 3829 Acked 4828 Read 9 New Done Done true
11882 3830 Acked 4829 Read 5 New Done Done true
11883 3831 Acked 4830 Read 10 New Done Done true
11884 3832 Acked 4831 Flush 0 New Done Done true
11885 3833 Acked 4832 Write 7 New Done Done true
11886 3834 Acked 4833 Write 7 New Done Done true
11887 3835 Acked 4834 Write 5 New Done Done true
11888 3836 Acked 4835 Read 4 New Done Done true
11889 3837 Acked 4836 Write 4 New Done Done true
11890 3838 Acked 4837 Read 4 New Done Done true
11891 3839 Acked 4838 Read 8 New Done Done true
11892 3840 Acked 4839 Read 2 New Done Done true
11893 3841 Acked 4840 Write 9 New Done Done true
11894 3842 Acked 4841 Read 4 New Done Done true
11895 3843 Acked 4842 Flush 0 New Done Done true
11896 3844 Acked 4843 Read 3 New Done Done true
11897 3845 Acked 4844 Read 1 New Done Done true
11898 3846 Acked 4845 Read 5 New Done Done true
11899 3847 Acked 4846 Write 10 New Done Done true
11900 3848 Acked 4847 Write 8 New Done Done true
11901 3849 Acked 4848 Write 10 New Done Done true
11902 3850 Acked 4849 Read 1 New Done Done true
11903 3851 Acked 4850 Read 2 New Done Done true
11904 3852 Acked 4851 Write 3 New Done Done true
11905 3853 Acked 4852 Write 1 New Done Done true
11906 3854 Acked 4853 Write 10 New Done Done true
11907 3855 Acked 4854 Write 8 New Done Done true
11908 3856 Acked 4855 Read 10 New Done Done true
11909 3857 Acked 4856 Write 4 New Done Done true
11910 3858 Acked 4857 Write 10 New Done Done true
11911 3859 Acked 4858 Read 9 New Done Done true
11912 3860 Acked 4859 Flush 0 New Done Done true
11913 3861 Acked 4860 Read 2 New Done Done true
11914 3862 Acked 4861 Write 1 New Done Done true
11915 3863 Acked 4862 Write 7 New Done Done true
11916 3864 Acked 4863 Write 7 New Done Done true
11917 3865 Acked 4864 Write 2 New Done Done true
11918 3866 Acked 4865 Read 5 New Done Done true
11919 3867 Acked 4866 Read 4 New Done Done true
11920 3868 Acked 4867 Read 10 New Done Done true
11921 3869 Acked 4868 Read 7 New Done Done true
11922 3870 Acked 4869 Flush 0 New Done Done true
11923 3871 Acked 4870 Read 3 New Done Done true
11924 3872 Acked 4871 Read 9 New Done Done true
11925 3873 Acked 4872 Write 5 New Done Done true
11926 3874 Acked 4873 Read 3 New Done Done true
11927 3875 Acked 4874 Read 8 New Done Done true
11928 3876 Acked 4875 Read 8 New Done Done true
11929 3877 Acked 4876 Read 1 New Done Done true
11930 3878 Acked 4877 Read 8 New Done Done true
11931 3879 Acked 4878 Read 7 New Done Done true
11932 3880 Acked 4879 Write 1 New Done Done true
11933 3881 Acked 4880 Read 3 New Done Done true
11934 3882 Acked 4881 Read 10 New Done Done true
11935 3883 Acked 4882 Read 4 New Done Done true
11936 3884 Acked 4883 Read 9 New Done Done true
11937 3885 Acked 4884 Write 3 New Done Done true
11938 3886 Acked 4885 Read 3 New Done Done true
11939 3887 Acked 4886 Read 7 New Done Done true
11940 3888 Acked 4887 Flush 0 New Done Done true
11941 3889 Acked 4888 Write 1 New Done Done true
11942 3890 Acked 4889 Write 3 New Done Done true
11943 3891 Acked 4890 Read 7 New Done Done true
11944 3892 Acked 4891 Write 6 New Done Done true
11945 3893 Acked 4892 Read 5 New Done Done true
11946 3894 Acked 4893 Write 5 New Done Done true
11947 3895 Acked 4894 Read 6 New Done Done true
11948 3896 Acked 4895 Read 4 New Done Done true
11949 3897 Acked 4896 Write 10 New Done Done true
11950 3898 Acked 4897 Write 10 New Done Done true
11951 3899 Acked 4898 Write 7 New Done Done true
11952 3900 Acked 4899 Read 10 New Done Done true
11953 3901 Acked 4900 Read 3 New Done Done true
11954 3902 Acked 4901 Read 8 New Done Done true
11955 3903 Acked 4902 Read 7 New Done Done true
11956 3904 Acked 4903 Read 6 New Done Done true
11957 3905 Acked 4904 Read 5 New Done Done true
11958 3906 Acked 4905 Write 7 New Done Done true
11959 3907 Acked 4906 Read 9 New Done Done true
11960 3908 Acked 4907 Write 5 New Done Done true
11961 3909 Acked 4908 Write 6 New Done Done true
11962 3910 Acked 4909 Read 5 New Done Done true
11963 3911 Acked 4910 Write 6 New Done Done true
11964 3912 Acked 4911 Write 3 New Done Done true
11965 3913 Acked 4912 Read 3 New Done Done true
11966 3914 Acked 4913 Write 2 New Done Done true
11967 3915 Acked 4914 Read 5 New Done Done true
11968 3916 Acked 4915 Read 4 New Done Done true
11969 3917 Acked 4916 Read 8 New Done Done true
11970 3918 Acked 4917 Flush 0 New Done Done true
11971 STATES DS:0 DS:1 DS:2 TOTAL
11972 New 301 0 0 301
11973 Sent 0 0 0 0
11974 Done 0 301 301 602
11975 Skipped 0 0 0 0
11976 Error 0 0 0 0
11977 Last Flush: 4616 4917 4917
11978 Downstairs last five completed: 4616 4615 4614 4613 4612
11979 Upstairs last five completed: 3918 3917 3916 3915 3914
11980 CLIENT: Up:0 ds:301 act:2
119812024-02-27T21:46:59.219ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
119822024-02-27T21:46:59.219ZINFOcrucible: ds_connection connected = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
119832024-02-27T21:46:59.221ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
119842024-02-27T21:46:59.221ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
119852024-02-27T21:46:59.221ZINFOcrucible: send last flush ID to this DS: 4616 = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
119862024-02-27T21:46:59.221ZINFOcrucible: Replied this last flush ID: 4616 = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
119872024-02-27T21:46:59.221ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
119882024-02-27T21:46:59.221ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
119892024-02-27T21:46:59.221ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
119902024-02-27T21:46:59.221ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
11991 ----------------------------------------------------------------
11992 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
11993 Downstairs last five completed: 4917 4916 4915 4914 4913
11994 Upstairs last five completed: 3918 3917 3916 3915 3914
11995 CLIENT: Up:0 ds:0 act:3
11996 Replay: All jobs finished, all DS active.
11997 Main task has work to do, go find it
11998 got dsc Stop(0)
11999 stop 0
12000 Feb 27 21:47:02.559 INFO request completed, latency_us: 369, response_code: 204, uri: /stop/cid/0, method: POST, req_id: e8698f67-5e15-4795-a365-693a6991a055, [8810] Got stop action so:false kr:false
12001 remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
12002 Feb 27 21:47:02.560 INFO request completed, latency_us: 115{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-27T21:47:02.560281297Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"io task","client":"0","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
12003 , response_code: 200, uri: /state/cid/0, method: GET, req_id: d980a069-d692-4128-82e9-79bd1b2c2cfd, remote_addr: {"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-27T21:47:02.560363077Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"io task","client":"0","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
12004 127.0.0.1:44347, local_addr: 127.0.0.1:9998
120052024-02-27T21:47:02.560ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = e6a5a974-925a-494f-8845-facc441c2690
120062024-02-27T21:47:02.560ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
120072024-02-27T21:47:02.560ZINFOcrucible: [0] client re-new 0 jobs since flush 4917 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
120082024-02-27T21:47:02.560ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
12009 [8810] Exited with: ExitStatus(unix_wait_status(9))
12010 [8810][0] reports Exit
120112024-02-27T21:47:03.561ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
120122024-02-27T21:47:03.561ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
120132024-02-27T21:47:03.561ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
120142024-02-27T21:47:03.561ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
120152024-02-27T21:47:03.561ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
120162024-02-27T21:47:03.561ZINFOcrucible: [0] client re-new 0 jobs since flush 4917 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
120172024-02-27T21:47:03.561ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
120182024-02-27T21:47:04.562ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
120192024-02-27T21:47:04.563ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
120202024-02-27T21:47:04.563ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
120212024-02-27T21:47:04.563ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
120222024-02-27T21:47:04.563ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
120232024-02-27T21:47:04.563ZINFOcrucible: [0] client re-new 0 jobs since flush 4917 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
120242024-02-27T21:47:04.563ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
120252024-02-27T21:47:05.565ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
120262024-02-27T21:47:05.565ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
120272024-02-27T21:47:05.565ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
120282024-02-27T21:47:05.565ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
120292024-02-27T21:47:05.565ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
120302024-02-27T21:47:05.565ZINFOcrucible: [0] client re-new 0 jobs since flush 4917 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
120312024-02-27T21:47:05.565ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
12032 Feb 27 21:47:06.561 INFO request completed, latency_us: 216, response_code: 200, uri: /state/cid/0, method: GET, req_id: e2878f00-c089-4e60-9e19-605cc879d4e7, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
12033 001/300 Flush
12034 002/300 Read block 184 len 28672
12035 003/300 Write block 2590 len 16384 data: 5 4 6 7
12036 004/300 Read block 4850 len 16384
120372024-02-27T21:47:06.566ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
120382024-02-27T21:47:06.566ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
120392024-02-27T21:47:06.566ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
120402024-02-27T21:47:06.566ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
120412024-02-27T21:47:06.566ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
120422024-02-27T21:47:06.566ZINFOcrucible: [0] client re-new 4 jobs since flush 4917 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
120432024-02-27T21:47:06.566ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
12044 005/300 Read block 3168 len 24576
12045 006/300 Write block 3317 len 36864 data: 5 4 3 3 2 2 3 3 3
12046 007/300 Flush
12047 008/300 Write block 258 len 16384 data: 4 5 5 4
12048 009/300 Write block 1454 len 16384 data: 4 4 5 5
12049 010/300 Write block 4856 len 4096 data: 4
12050 011/300 Read block 4489 len 20480
12051 012/300 Read block 1389 len 20480
12052 013/300 Read block 3921 len 8192
12053 014/300 Write block 2488 len 28672 data: 5 5 5 5 5 4 4
12054 015/300 Read block 2645 len 12288
12055 016/300 Read block 2271 len 12288
12056 017/300 Flush
12057 018/300 Read block 3246 len 32768
12058 019/300 Flush
12059 020/300 Read block 2269 len 32768
12060 021/300 Read block 4568 len 12288
12061 022/300 Write block 2394 len 20480 data: 2 2 2 2 2
12062 023/300 Write block 4892 len 4096 data: 5
12063 024/300 Read block 1847 len 40960
12064 025/300 Read block 3146 len 36864
12065 026/300 Read block 1416 len 36864
12066 027/300 Read block 3425 len 32768
12067 028/300 Read block 2684 len 12288
12068 029/300 Read block 3285 len 8192
12069 030/300 Read block 707 len 40960
12070 031/300 Write block 4527 len 16384 data: 3 3 3 3
12071 032/300 Read block 1012 len 28672
12072 033/300 Write block 2618 len 36864 data: 5 4 3 3 3 4 5 5 5
12073 034/300 Write block 4042 len 40960 data: 2 2 2 3 3 3 3 3 2 3
12074 035/300 Write block 4325 len 16384 data: 4 4 5 6
12075 036/300 Write block 315 len 4096 data: 3
12076 037/300 Read block 258 len 12288
12077 038/300 Read block 1229 len 32768
12078 039/300 Write block 4588 len 36864 data: 5 4 4 4 3 4 4 5 6
12079 040/300 Flush
12080 041/300 Write block 1108 len 28672 data: 2 2 3 4 3 4 5
12081 042/300 Read block 1539 len 28672
12082 043/300 Read block 4936 len 32768
12083 044/300 Write block 2689 len 24576 data: 2 2 2 4 5 3
12084 045/300 Write block 3840 len 16384 data: 3 3 4 4
12085 046/300 Read block 3982 len 24576
12086 047/300 Read block 1259 len 24576
12087 048/300 Write block 4646 len 12288 data: 4 4 3
12088 049/300 Read block 4095 len 24576
12089 050/300 Read block 792 len 32768
12090 051/300 Flush
12091 052/300 Write block 1709 len 20480 data: 4 5 5 8 8
12092 053/300 Write block 1220 len 28672 data: 5 5 3 3 4 4 5
12093 054/300 Read block 4235 len 24576
12094 055/300 Read block 447 len 20480
12095 056/300 Read block 2873 len 36864
12096 057/300 Read block 4357 len 8192
12097 058/300 Write block 156 len 28672 data: 3 3 4 4 3 3 4
12098 059/300 Read block 1740 len 16384
12099 060/300 Read block 1275 len 4096
12100 061/300 Read block 4186 len 36864
12101 062/300 Write block 3609 len 32768 data: 5 5 5 4 3 3 3 3
12102 063/300 Write block 1584 len 16384 data: 3 3 3 2
12103 064/300 Read block 1550 len 16384
12104 065/300 Flush
12105 066/300 Read block 3517 len 12288
12106 067/300 Read block 2391 len 16384
12107 068/300 Write block 44 len 32768 data: 4 4 4 3 4 4 4 3
12108 069/300 Read block 4783 len 24576
12109 070/300 Read block 2565 len 28672
12110 071/300 Write block 1594 len 12288 data: 5 5 4
12111 072/300 Flush
12112 073/300 Read block 1149 len 40960
12113 074/300 Write block 112 len 32768 data: 3 3 4 4 3 4 5 5
12114 075/300 Write block 2656 len 20480 data: 4 4 4 4 4
12115 076/300 Flush
12116 077/300 Read block 3923 len 20480
12117 078/300 Read block 3810 len 36864
12118 079/300 Write block 663 len 40960 data: 3 2 3 3 3 4 4 4 4 4
12119 080/300 Flush
12120 081/300 Write block 2508 len 24576 data: 9 5 2 2 2 2
12121 082/300 Read block 3966 len 16384
12122 083/300 Write block 3301 len 20480 data: 5 5 6 5 4
12123 084/300 Read block 893 len 36864
12124 085/300 Write block 2427 len 20480 data: 2 2 2 2 3
12125 086/300 Read block 4073 len 8192
12126 087/300 Write block 1968 len 32768 data: 3 3 3 4 3 4 4 7
12127 088/300 Write block 1808 len 20480 data: 4 3 3 4 4
12128 089/300 Write block 1101 len 28672 data: 4 3 3 3 2 2 2
12129 090/300 Write block 856 len 4096 data: 2
12130 091/300 Write block 1006 len 4096 data: 3
12131 092/300 Flush
12132 093/300 Read block 3414 len 28672
12133 094/300 Read block 3219 len 16384
12134 095/300 Read block 3604 len 28672
12135 096/300 Read block 548 len 20480
12136 097/300 Write block 670 len 4096 data: 5
12137 098/300 Read block 821 len 28672
12138 099/300 Flush
12139 100/300 Write block 4013 len 36864 data: 3 3 2 2 2 4 5 5 6
12140 101/300 Flush
12141 102/300 Write block 709 len 8192 data: 2 2
12142 103/300 Read block 468 len 16384
12143 104/300 Read block 2546 len 4096
12144 105/300 Write block 15 len 24576 data: 4 6 6 6 5 5
12145 106/300 Write block 594 len 28672 data: 6 5 4 4 4 3 4
12146 107/300 Read block 4432 len 40960
12147 108/300 Read block 4213 len 4096
12148 109/300 Read block 4372 len 8192
12149 110/300 Write block 2407 len 4096 data: 3
12150 111/300 Read block 2349 len 20480
12151 112/300 Flush
12152 113/300 Read block 4664 len 40960
12153 114/300 Read block 3244 len 16384
12154 115/300 Read block 4740 len 40960
12155 116/300 Write block 791 len 16384 data: 4 4 5 5
12156 117/300 Write block 4570 len 16384 data: 5 5 6 5
12157 118/300 Flush
12158 119/300 Read block 1581 len 12288
12159 120/300 Read block 269 len 4096
12160 121/300 Read block 1530 len 32768
12161 122/300 Read block 149 len 32768
12162 123/300 Write block 1864 len 40960 data: 3 3 3 2 2 2 2 3 4 4
12163 124/300 Read block 3100 len 28672
12164 125/300 Read block 1216 len 4096
12165 126/300 Read block 346 len 28672
12166 127/300 Flush
12167 128/300 Flush
12168 129/300 Write block 2480 len 4096 data: 4
12169 130/300 Read block 2618 len 4096
12170 131/300 Write block 4758 len 24576 data: 2 2 2 2 2 4
12171 132/300 Flush
12172 133/300 Read block 3709 len 12288
12173 134/300 Write block 4856 len 4096 data: 5
12174 135/300 Flush
12175 136/300 Read block 4351 len 8192
12176 137/300 Read block 227 len 12288
12177 138/300 Write block 4611 len 4096 data: 4
12178 139/300 Flush
12179 140/300 Write block 1830 len 20480 data: 3 4 4 4 4
12180 141/300 Read block 2593 len 20480
12181 142/300 Flush
12182 143/300 Write block 2964 len 32768 data: 4 4 4 4 3 2 3 3
12183 144/300 Write block 2713 len 4096 data: 4
12184 145/300 Read block 1985 len 24576
12185 146/300 Read block 1000 len 32768
12186 147/300 Read block 4414 len 20480
12187 148/300 Write block 4716 len 4096 data: 4
12188 149/300 Read block 489 len 36864
12189 150/300 Read block 854 len 12288
12190 151/300 Read block 2733 len 20480
12191 152/300 Read block 3650 len 28672
12192 153/300 Read block 4318 len 24576
12193 154/300 Read block 2579 len 24576
12194 155/300 Flush
12195 156/300 Read block 3531 len 28672
12196 157/300 Read block 4903 len 4096
12197 158/300 Write block 1038 len 32768 data: 3 4 4 3 3 3 3 4
12198 159/300 Read block 4058 len 24576
12199 160/300 Write block 2998 len 24576 data: 6 6 7 6 6 4
12200 161/300 Write block 4632 len 4096 data: 5
12201 162/300 Read block 1823 len 32768
12202 163/300 Read block 391 len 4096
12203 164/300 Write block 552 len 20480 data: 5 4 4 5 5
12204 165/300 Read block 3987 len 8192
12205 166/300 Write block 419 len 16384 data: 4 4 4 3
12206 167/300 Read block 4271 len 16384
12207 168/300 Write block 3563 len 40960 data: 7 3 2 2 2 3 4 5 5 5
12208 169/300 Write block 2347 len 12288 data: 4 5 7
12209 170/300 Write block 4944 len 12288 data: 4 5 4
12210 171/300 Write block 1154 len 32768 data: 5 5 4 5 5 5 5 5
12211 172/300 Read block 1932 len 12288
12212 173/300 Write block 3229 len 24576 data: 3 3 3 4 4 4
12213 174/300 Flush
12214 175/300 Write block 1663 len 8192 data: 7 8
12215 176/300 Read block 2787 len 24576
12216 177/300 Read block 4036 len 28672
12217 178/300 Read block 550 len 12288
12218 179/300 Read block 669 len 36864
12219 180/300 Write block 4702 len 32768 data: 3 4 4 5 5 4 4 4
12220 181/300 Read block 2836 len 24576
12221 182/300 Read block 2723 len 12288
12222 183/300 Write block 717 len 20480 data: 4 3 3 3 3
12223 184/300 Write block 3266 len 36864 data: 4 2 2 2 2 2 4 4 5
12224 185/300 Write block 2809 len 20480 data: 4 5 5 5 7
12225 186/300 Write block 3155 len 32768 data: 4 3 3 3 4 3 3 4
12226 187/300 Read block 1307 len 12288
12227 188/300 Flush
12228 189/300 Read block 2988 len 4096
12229 190/300 Write block 1386 len 8192 data: 3 3
12230 191/300 Read block 3387 len 16384
12231 192/300 Write block 322 len 32768 data: 3 2 2 2 3 3 3 3
12232 193/300 Write block 21 len 32768 data: 4 5 7 7 7 4 3 2
12233 194/300 Read block 872 len 12288
12234 195/300 Read block 2555 len 36864
12235 196/300 Write block 2769 len 16384 data: 3 3 3 3
12236 197/300 Write block 304 len 28672 data: 2 2 2 2 3 3 3
12237 198/300 Write block 3842 len 12288 data: 5 5 5
12238 199/300 Read block 1616 len 40960
12239 200/300 Write block 3861 len 20480 data: 3 5 5 4 4
12240 201/300 Write block 1575 len 28672 data: 3 3 3 4 4 3 3
12241 202/300 Read block 4296 len 36864
12242 203/300 Read block 3046 len 24576
12243 204/300 Read block 4972 len 28672
12244 205/300 Read block 4244 len 32768
12245 206/300 Write block 1748 len 32768 data: 2 2 2 2 2 3 3 3
12246 207/300 Flush
12247 208/300 Read block 4734 len 32768
12248 209/300 Read block 4795 len 28672
12249 210/300 Read block 1122 len 28672
12250 211/300 Write block 1423 len 12288 data: 3 3 3
12251 212/300 Flush
12252 213/300 Write block 4677 len 20480 data: 4 5 5 5 4
12253 214/300 Read block 2240 len 20480
12254 215/300 Write block 304 len 16384 data: 3 3 3 3
12255 216/300 Write block 3239 len 12288 data: 5 5 5
12256 217/300 Read block 4844 len 32768
12257 218/300 Read block 3485 len 36864
12258 219/300 Read block 1705 len 40960
12259 220/300 Read block 4502 len 8192
12260 221/300 Read block 772 len 24576
12261 222/300 Write block 693 len 12288 data: 2 2 2
12262 223/300 Write block 4138 len 32768 data: 4 2 3 4 4 4 3 3
12263 224/300 Read block 4433 len 28672
12264 225/300 Write block 1258 len 24576 data: 3 3 3 2 2 3
12265 226/300 Write block 2870 len 8192 data: 5 6
12266 227/300 Write block 1158 len 40960 data: 6 6 6 6 4 4 3 3 3 3
12267 228/300 Write block 2203 len 28672 data: 2 4 4 6 6 5 4
12268 229/300 Write block 3040 len 28672 data: 3 3 3 4 4 4 3
12269 230/300 Read block 4258 len 40960
12270 231/300 Flush
12271 232/300 Read block 1188 len 16384
12272 233/300 Read block 903 len 32768
12273 234/300 Write block 2554 len 20480 data: 3 4 4 3 3
12274 235/300 Write block 267 len 40960 data: 3 4 3 3 3 3 3 2 4 4
12275 236/300 Read block 3126 len 4096
12276 237/300 Read block 1907 len 16384
12277 238/300 Read block 4792 len 20480
12278 239/300 Read block 2384 len 12288
12279 240/300 Write block 4283 len 8192 data: 3 4
12280 241/300 Flush
12281 242/300 Write block 2389 len 16384 data: 2 2 2 2
12282 243/300 Write block 3965 len 24576 data: 6 5 5 4 4 4
12283 244/300 Read block 1424 len 16384
12284 245/300 Write block 1660 len 24576 data: 7 5 7 8 9 6
12285 246/300 Write block 972 len 32768 data: 4 4 3 3 3 3 2 2
12286 247/300 Read block 1064 len 28672
12287 248/300 Read block 4525 len 16384
12288 249/300 Read block 1061 len 28672
12289 250/300 Write block 4475 len 16384 data: 3 4 4 4
12290 251/300 Read block 3885 len 24576
12291 252/300 Flush
12292 253/300 Write block 1188 len 24576 data: 4 4 4 5 3 3
12293 254/300 Read block 3777 len 24576
12294 255/300 Write block 1036 len 12288 data: 3 3 4
12295 256/300 Write block 3860 len 24576 data: 3 4 6 6 5 5
12296 257/300 Write block 2331 len 36864 data: 3 2 2 5 6 6 7 6 8
12297 258/300 Read block 861 len 28672
12298 259/300 Read block 4790 len 20480
12299 260/300 Write block 253 len 4096 data: 2
12300 261/300 Read block 1081 len 12288
12301 262/300 Flush
12302 263/300 Write block 224 len 20480 data: 3 3 4 4 3
12303 264/300 Read block 4002 len 12288
12304 265/300 Flush
12305 266/300 Read block 140 len 40960
12306 267/300 Read block 2052 len 28672
12307 268/300 Read block 1895 len 40960
12308 269/300 Read block 1563 len 8192
12309 270/300 Read block 826 len 40960
12310 271/300 Write block 222 len 20480 data: 3 3 4 4 5
12311 272/300 Read block 4899 len 20480
12312 273/300 Read block 669 len 20480
12313 274/300 Flush
12314 275/300 Flush
12315 276/300 Read block 2785 len 20480
12316 277/300 Write block 2624 len 24576 data: 6 6 6 5 4 4
12317 278/300 Write block 686 len 8192 data: 3 4
12318 279/300 Read block 3135 len 36864
12319 280/300 Read block 4294 len 8192
12320 281/300 Flush
12321 282/300 Write block 1975 len 8192 data: 8 9
12322 283/300 Read block 526 len 20480
12323 284/300 Read block 4271 len 12288
12324 285/300 Write block 3454 len 40960 data: 3 3 3 4 3 3 3 3 2 2
12325 286/300 Write block 4492 len 16384 data: 6 6 5 5
12326 287/300 Write block 1670 len 12288 data: 6 5 4
12327 288/300 Write block 358 len 16384 data: 4 4 4 4
12328 289/300 Write block 2668 len 20480 data: 6 7 7 6 5
12329 290/300 Write block 2309 len 20480 data: 2 3 4 4 4
12330 291/300 Read block 2229 len 12288
12331 292/300 Read block 3733 len 8192
12332 293/300 Write block 2210 len 4096 data: 4
12333 294/300 Read block 1133 len 8192
12334 295/300 Read block 208 len 32768
12335 296/300 Read block 2831 len 12288
12336 297/300 Read block 4075 len 12288
12337 298/300 Write block 1456 len 20480 data: 6 6 6 4 5
12338 299/300 Write block 4642 len 36864 data: 4 4 3 4 5 5 4 4 4
12339 300/300 Flush
12340 Main task has work to do, go find it
12341 got dsc Start(0)
12342 start 0
12343 Feb 27 21:47:06.906 INFO request completed, latency_us: 232, response_code: 204, uri: /start/cid/0, method: POST, req_id: a2174a91-9442-43be-863b-60651e448271, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
12344 Replay: started 0, returned:Ok(())
12345 ----------------------------------------------------------------
12346 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
12347 States: Offline Active Active
12348 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
12349 3919 Acked 4918 Flush 0 New Done Done true
12350 3920 Acked 4919 Read 7 New Done Done true
12351 3921 Acked 4920 Write 4 New Done Done true
12352 3922 Acked 4921 Read 4 New Done Done true
12353 3923 Acked 4922 Read 6 New Done Done false
12354 3924 Acked 4923 Write 9 New Done Done false
12355 3925 Acked 4924 Flush 0 New Done Done false
12356 3926 Acked 4925 Write 4 New Done Done false
12357 3927 Acked 4926 Write 4 New Done Done false
12358 3928 Acked 4927 Write 1 New Done Done false
12359 3929 Acked 4928 Read 5 New Done Done false
12360 3930 Acked 4929 Read 5 New Done Done false
12361 3931 Acked 4930 Read 2 New Done Done false
12362 3932 Acked 4931 Write 7 New Done Done false
12363 3933 Acked 4932 Read 3 New Done Done false
12364 3934 Acked 4933 Read 3 New Done Done false
12365 3935 Acked 4934 Flush 0 New Done Done false
12366 3936 Acked 4935 Read 8 New Done Done false
12367 3937 Acked 4936 Flush 0 New Done Done false
12368 3938 Acked 4937 Read 8 New Done Done false
12369 3939 Acked 4938 Read 3 New Done Done false
12370 3940 Acked 4939 Write 5 New Done Done false
12371 3941 Acked 4940 Write 1 New Done Done false
12372 3942 Acked 4941 Read 10 New Done Done false
12373 3943 Acked 4942 Read 9 New Done Done false
12374 3944 Acked 4943 Read 9 New Done Done false
12375 3945 Acked 4944 Read 8 New Done Done false
12376 3946 Acked 4945 Read 3 New Done Done false
12377 3947 Acked 4946 Read 2 New Done Done false
12378 3948 Acked 4947 Read 10 New Done Done false
12379 3949 Acked 4948 Write 4 New Done Done false
12380 3950 Acked 4949 Read 7 New Done Done false
12381 3951 Acked 4950 Write 9 New Done Done false
12382 3952 Acked 4951 Write 10 New Done Done false
12383 3953 Acked 4952 Write 4 New Done Done false
12384 3954 Acked 4953 Write 1 New Done Done false
12385 3955 Acked 4954 Read 3 New Done Done false
12386 3956 Acked 4955 Read 8 New Done Done false
12387 3957 Acked 4956 Write 9 New Done Done false
12388 3958 Acked 4957 Flush 0 New Done Done false
12389 3959 Acked 4958 Write 7 New Done Done false
12390 3960 Acked 4959 Read 7 New Done Done false
12391 3961 Acked 4960 Read 8 New Done Done false
12392 3962 Acked 4961 Write 6 New Done Done false
12393 3963 Acked 4962 Write 4 New Done Done false
12394 3964 Acked 4963 Read 6 New Done Done false
12395 3965 Acked 4964 Read 6 New Done Done false
12396 3966 Acked 4965 Write 3 New Done Done false
12397 3967 Acked 4966 Read 6 New Done Done false
12398 3968 Acked 4967 Read 8 New Done Done false
12399 3969 Acked 4968 Flush 0 New Done Done false
12400 3970 Acked 4969 Write 5 New Done Done false
12401 3971 Acked 4970 Write 7 New Done Done false
12402 3972 Acked 4971 Read 6 New Done Done false
12403 3973 Acked 4972 Read 5 New Done Done false
12404 3974 Acked 4973 Read 9 New Done Done false
12405 3975 Acked 4974 Read 2 New Done Done false
12406 3976 Acked 4975 Write 7 New Done Done false
12407 3977 Acked 4976 Read 4 New Done Done false
12408 3978 Acked 4977 Read 1 New Done Done false
12409 3979 Acked 4978 Read 9 New Done Done false
12410 3980 Acked 4979 Write 8 New Done Done false
12411 3981 Acked 4980 Write 4 New Done Done false
12412 3982 Acked 4981 Read 4 New Done Done false
12413 3983 Acked 4982 Flush 0 New Done Done false
12414 3984 Acked 4983 Read 3 New Done Done false
12415 3985 Acked 4984 Read 4 New Done Done false
12416 3986 Acked 4985 Write 8 New Done Done false
12417 3987 Acked 4986 Read 6 New Done Done false
12418 3988 Acked 4987 Read 7 New Done Done false
12419 3989 Acked 4988 Write 3 New Done Done false
12420 3990 Acked 4989 Flush 0 New Done Done false
12421 3991 Acked 4990 Read 10 New Done Done false
12422 3992 Acked 4991 Write 8 New Done Done false
12423 3993 Acked 4992 Write 5 New Done Done false
12424 3994 Acked 4993 Flush 0 New Done Done false
12425 3995 Acked 4994 Read 5 New Done Done false
12426 3996 Acked 4995 Read 9 New Done Done false
12427 3997 Acked 4996 Write 10 New Done Done false
12428 3998 Acked 4997 Flush 0 New Done Done false
12429 3999 Acked 4998 Write 6 New Done Done false
12430 4000 Acked 4999 Read 4 New Done Done false
12431 4001 Acked 5000 Write 5 New Done Done false
12432 4002 Acked 5001 Read 9 New Done Done false
12433 4003 Acked 5002 Write 5 New Done Done false
12434 4004 Acked 5003 Read 2 New Done Done false
12435 4005 Acked 5004 Write 8 New Done Done false
12436 4006 Acked 5005 Write 5 New Done Done false
12437 4007 Acked 5006 Write 7 New Done Done false
12438 4008 Acked 5007 Write 1 New Done Done false
12439 4009 Acked 5008 Write 1 New Done Done false
12440 4010 Acked 5009 Flush 0 New Done Done false
12441 4011 Acked 5010 Read 7 New Done Done false
12442 4012 Acked 5011 Read 4 New Done Done false
12443 4013 Acked 5012 Read 7 New Done Done false
12444 4014 Acked 5013 Read 5 New Done Done false
12445 4015 Acked 5014 Write 1 New Done Done false
12446 4016 Acked 5015 Read 7 New Done Done false
12447 4017 Acked 5016 Flush 0 New Done Done false
12448 4018 Acked 5017 Write 9 New Done Done false
12449 4019 Acked 5018 Flush 0 New Done Done false
12450 4020 Acked 5019 Write 2 New Done Done false
12451 4021 Acked 5020 Read 4 New Done Done false
12452 4022 Acked 5021 Read 1 New Done Done false
12453 4023 Acked 5022 Write 6 New Done Done false
12454 4024 Acked 5023 Write 7 New Done Done false
12455 4025 Acked 5024 Read 10 New Done Done false
12456 4026 Acked 5025 Read 1 New Done Done false
12457 4027 Acked 5026 Read 2 New Done Done false
12458 4028 Acked 5027 Write 1 New Done Done false
12459 4029 Acked 5028 Read 5 New Done Done false
12460 4030 Acked 5029 Flush 0 New Done Done false
12461 4031 Acked 5030 Read 10 New Done Done false
12462 4032 Acked 5031 Read 4 New Done Done false
12463 4033 Acked 5032 Read 10 New Done Done false
12464 4034 Acked 5033 Write 4 New Done Done false
12465 4035 Acked 5034 Write 4 New Done Done false
12466 4036 Acked 5035 Flush 0 New Done Done false
12467 4037 Acked 5036 Read 3 New Done Done false
12468 4038 Acked 5037 Read 1 New Done Done false
12469 4039 Acked 5038 Read 8 New Done Done false
12470 4040 Acked 5039 Read 8 New Done Done false
12471 4041 Acked 5040 Write 10 New Done Done false
12472 4042 Acked 5041 Read 7 New Done Done false
12473 4043 Acked 5042 Read 1 New Done Done false
12474 4044 Acked 5043 Read 7 New Done Done false
12475 4045 Acked 5044 Flush 0 New Done Done false
12476 4046 Acked 5045 Flush 0 New Done Done false
12477 4047 Acked 5046 Write 1 New Done Done false
12478 4048 Acked 5047 Read 1 New Done Done false
12479 4049 Acked 5048 Write 6 New Done Done false
12480 4050 Acked 5049 Flush 0 New Done Done false
12481 4051 Acked 5050 Read 3 New Done Done false
12482 4052 Acked 5051 Write 1 New Done Done false
12483 4053 Acked 5052 Flush 0 New Done Done false
12484 4054 Acked 5053 Read 2 New Done Done false
12485 4055 Acked 5054 Read 3 New Done Done false
12486 4056 Acked 5055 Write 1 New Done Done false
12487 4057 Acked 5056 Flush 0 New Done Done false
12488 4058 Acked 5057 Write 5 New Done Done false
12489 4059 Acked 5058 Read 5 New Done Done false
12490 4060 Acked 5059 Flush 0 New Done Done false
12491 4061 Acked 5060 Write 8 New Done Done false
12492 4062 Acked 5061 Write 1 New Done Done false
12493 4063 Acked 5062 Read 6 New Done Done false
12494 4064 Acked 5063 Flush 0 New Done Done false
12495 4065 Acked 5064 Read 8 New Done Done false
12496 4066 Acked 5065 Read 5 New Done Done false
12497 4067 Acked 5066 Write 1 New Done Done false
12498 4068 Acked 5067 Read 9 New Done Done false
12499 4069 Acked 5068 Read 3 New Done Done false
12500 4070 Acked 5069 Read 5 New Done Done false
12501 4071 Acked 5070 Read 7 New Done Done false
12502 4072 Acked 5071 Read 6 New Done Done false
12503 4073 Acked 5072 Read 6 New Done Done false
12504 4074 Acked 5073 Flush 0 New Done Done false
12505 4075 Acked 5074 Read 7 New Done Done false
12506 4076 Acked 5075 Read 1 New Done Done false
12507 4077 Acked 5076 Write 8 New Done Done false
12508 4078 Acked 5077 Read 6 New Done Done false
12509 4079 Acked 5078 Write 6 New Done Done false
12510 4080 Acked 5079 Write 1 New Done Done false
12511 4081 Acked 5080 Read 8 New Done Done false
12512 4082 Acked 5081 Read 1 New Done Done false
12513 4083 Acked 5082 Write 5 New Done Done false
12514 4084 Acked 5083 Read 2 New Done Done false
12515 4085 Acked 5084 Write 4 New Done Done false
12516 4086 Acked 5085 Read 4 New Done Done false
12517 4087 Acked 5086 Write 10 New Done Done false
12518 4088 Acked 5087 Write 3 New Done Done false
12519 4089 Acked 5088 Write 3 New Done Done false
12520 4090 Acked 5089 Write 8 New Done Done false
12521 4091 Acked 5090 Read 3 New Done Done false
12522 4092 Acked 5091 Write 6 New Done Done false
12523 4093 Acked 5092 Flush 0 New Done Done false
12524 4094 Acked 5093 Write 2 New Done Done false
12525 4095 Acked 5094 Read 6 New Done Done false
12526 4096 Acked 5095 Read 7 New Done Done false
12527 4097 Acked 5096 Read 3 New Done Done false
12528 4098 Acked 5097 Read 9 New Done Done false
12529 4099 Acked 5098 Write 8 New Done Done false
12530 4100 Acked 5099 Read 6 New Done Done false
12531 4101 Acked 5100 Read 3 New Done Done false
12532 4102 Acked 5101 Write 5 New Done Done false
12533 4103 Acked 5102 Write 9 New Done Done false
12534 4104 Acked 5103 Write 5 New Done Done false
12535 4105 Acked 5104 Write 8 New Done Done false
12536 4106 Acked 5105 Read 3 New Done Done false
12537 4107 Acked 5106 Flush 0 New Done Done false
12538 4108 Acked 5107 Read 1 New Done Done false
12539 4109 Acked 5108 Write 2 New Done Done false
12540 4110 Acked 5109 Read 4 New Done Done false
12541 4111 Acked 5110 Write 8 New Done Done false
12542 4112 Acked 5111 Write 8 New Done Done false
12543 4113 Acked 5112 Read 3 New Done Done false
12544 4114 Acked 5113 Read 9 New Done Done false
12545 4115 Acked 5114 Write 4 New Done Done false
12546 4116 Acked 5115 Write 7 New Done Done false
12547 4117 Acked 5116 Write 3 New Done Done false
12548 4118 Acked 5117 Read 10 New Done Done false
12549 4119 Acked 5118 Write 5 New Done Done false
12550 4120 Acked 5119 Write 7 New Done Done false
12551 4121 Acked 5120 Read 9 New Done Done false
12552 4122 Acked 5121 Read 6 New Done Done false
12553 4123 Acked 5122 Read 7 New Done Done false
12554 4124 Acked 5123 Read 8 New Done Done false
12555 4125 Acked 5124 Write 8 New Done Done false
12556 4126 Acked 5125 Flush 0 New Done Done false
12557 4127 Acked 5126 Read 8 New Done Done false
12558 4128 Acked 5127 Read 7 New Done Done false
12559 4129 Acked 5128 Read 7 New Done Done false
12560 4130 Acked 5129 Write 3 New Done Done false
12561 4131 Acked 5130 Flush 0 New Done Done false
12562 4132 Acked 5131 Write 5 New Done Done false
12563 4133 Acked 5132 Read 5 New Done Done false
12564 4134 Acked 5133 Write 4 New Done Done false
12565 4135 Acked 5134 Write 3 New Done Done false
12566 4136 Acked 5135 Read 8 New Done Done false
12567 4137 Acked 5136 Read 9 New Done Done false
12568 4138 Acked 5137 Read 10 New Done Done false
12569 4139 Acked 5138 Read 2 New Done Done false
12570 4140 Acked 5139 Read 6 New Done Done false
12571 4141 Acked 5140 Write 3 New Done Done false
12572 4142 Acked 5141 Write 8 New Done Done false
12573 4143 Acked 5142 Read 7 New Done Done false
12574 4144 Acked 5143 Write 6 New Done Done false
12575 4145 Acked 5144 Write 2 New Done Done false
12576 4146 Acked 5145 Write 10 New Done Done false
12577 4147 Acked 5146 Write 7 New Done Done false
12578 4148 Acked 5147 Write 7 New Done Done false
12579 4149 Acked 5148 Read 10 New Done Done false
12580 4150 Acked 5149 Flush 0 New Done Done false
12581 4151 Acked 5150 Read 4 New Done Done false
12582 4152 Acked 5151 Read 8 New Done Done false
12583 4153 Acked 5152 Write 5 New Done Done false
12584 4154 Acked 5153 Write 10 New Done Done false
12585 4155 Acked 5154 Read 1 New Done Done false
12586 4156 Acked 5155 Read 4 New Done Done false
12587 4157 Acked 5156 Read 5 New Done Done false
12588 4158 Acked 5157 Read 3 New Done Done false
12589 4159 Acked 5158 Write 2 New Done Done false
12590 4160 Acked 5159 Flush 0 New Done Done false
12591 4161 Acked 5160 Write 4 New Done Done false
12592 4162 Acked 5161 Write 6 New Done Done false
12593 4163 Acked 5162 Read 4 New Done Done false
12594 4164 Acked 5163 Write 6 New Done Done false
12595 4165 Acked 5164 Write 8 New Done Done false
12596 4166 Acked 5165 Read 7 New Done Done false
12597 4167 Acked 5166 Read 4 New Done Done false
12598 4168 Acked 5167 Read 7 New Done Done false
12599 4169 Acked 5168 Write 4 New Done Done false
12600 4170 Acked 5169 Read 6 New Done Done false
12601 4171 Acked 5170 Flush 0 New Done Done false
12602 4172 Acked 5171 Write 6 New Done Done false
12603 4173 Acked 5172 Read 6 New Done Done false
12604 4174 Acked 5173 Write 3 New Done Done false
12605 4175 Acked 5174 Write 6 New Done Done false
12606 4176 Acked 5175 Write 9 New Done Done false
12607 4177 Acked 5176 Read 7 New Done Done false
12608 4178 Acked 5177 Read 5 New Done Done false
12609 4179 Acked 5178 Write 1 New Done Done false
12610 4180 Acked 5179 Read 3 New Done Done false
12611 4181 Acked 5180 Flush 0 New Done Done false
12612 4182 Acked 5181 Write 5 New Done Done false
12613 4183 Acked 5182 Read 3 New Done Done false
12614 4184 Acked 5183 Flush 0 New Done Done false
12615 4185 Acked 5184 Read 10 New Done Done false
12616 4186 Acked 5185 Read 7 New Done Done false
12617 4187 Acked 5186 Read 10 New Done Done false
12618 4188 Acked 5187 Read 2 New Done Done false
12619 4189 Acked 5188 Read 10 New Done Done false
12620 4190 Acked 5189 Write 5 New Done Done false
12621 4191 Acked 5190 Read 5 New Done Done false
12622 4192 Acked 5191 Read 5 New Done Done false
12623 4193 Acked 5192 Flush 0 New Done Done false
12624 4194 Acked 5193 Flush 0 New Done Done false
12625 4195 Acked 5194 Read 5 New Done Done false
12626 4196 Acked 5195 Write 6 New Done Done false
12627 4197 Acked 5196 Write 2 New Done Done false
12628 4198 Acked 5197 Read 9 New Done Done false
12629 4199 Acked 5198 Read 2 New Done Done false
12630 4200 Acked 5199 Flush 0 New Done Done false
12631 4201 Acked 5200 Write 2 New Done Done false
12632 4202 Acked 5201 Read 5 New Done Done false
12633 4203 Acked 5202 Read 3 New Done Done false
12634 4204 Acked 5203 Write 10 New Done Done false
12635 4205 Acked 5204 Write 4 New Done Done false
12636 4206 Acked 5205 Write 3 New Done Done false
12637 4207 Acked 5206 Write 4 New Done Done false
12638 4208 Acked 5207 Write 5 New Done Done false
12639 4209 Acked 5208 Write 5 New Done Done false
12640 4210 Acked 5209 Read 3 New Done Done false
12641 4211 Acked 5210 Read 2 New Done Done false
12642 4212 Acked 5211 Write 1 New Done Done false
12643 4213 Acked 5212 Read 2 New Done Done false
12644 4214 Acked 5213 Read 8 New Done Done false
12645 4215 Acked 5214 Read 3 New Done Done false
12646 4216 Acked 5215 Read 3 New Done Done false
12647 4217 Acked 5216 Write 5 New Done Done false
12648 4218 Acked 5217 Write 9 New Done Done false
12649 4219 Acked 5218 Flush 0 New Done Done false
12650 STATES DS:0 DS:1 DS:2 TOTAL
12651 New 301 0 0 301
12652 Sent 0 0 0 0
12653 Done 0 301 301 602
12654 Skipped 0 0 0 0
12655 Error 0 0 0 0
12656 Last Flush: 4917 5218 5218
12657 Downstairs last five completed: 4917 4916 4915 4914 4913
12658 Upstairs last five completed: 4219 4218 4217 4216 4215
12659 CLIENT: Up:0 ds:301 act:2
126602024-02-27T21:47:07.567ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
126612024-02-27T21:47:07.567ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
126622024-02-27T21:47:07.567ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
126632024-02-27T21:47:07.567ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
126642024-02-27T21:47:07.567ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
126652024-02-27T21:47:07.567ZINFOcrucible: [0] client re-new 301 jobs since flush 4917 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
126662024-02-27T21:47:07.567ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
126672024-02-27T21:47:08.568ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
126682024-02-27T21:47:08.568ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
126692024-02-27T21:47:08.568ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
126702024-02-27T21:47:08.568ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
126712024-02-27T21:47:08.568ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
126722024-02-27T21:47:08.568ZINFOcrucible: [0] client re-new 301 jobs since flush 4917 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
126732024-02-27T21:47:08.568ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
12674 [8810] got start action
12675 [8810] I am going to restart
12676 Starting downstairs at port 8810
12677 Make output file at "/tmp/dsc/downstairs-8810.txt"
12678 [8810][0] reports Starting
12679 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1181)
12680 [8810][0] reports Running
126812024-02-27T21:47:09.569ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
126822024-02-27T21:47:09.569ZINFOcrucible: ds_connection connected = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
126832024-02-27T21:47:09.570ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
126842024-02-27T21:47:09.570ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
126852024-02-27T21:47:09.570ZINFOcrucible: send last flush ID to this DS: 4917 = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
126862024-02-27T21:47:09.571ZINFOcrucible: Replied this last flush ID: 4917 = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
126872024-02-27T21:47:09.571ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
126882024-02-27T21:47:09.571ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
126892024-02-27T21:47:09.571ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
126902024-02-27T21:47:09.571ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
12691 ----------------------------------------------------------------
12692 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12693 Downstairs last five completed: 5218 5217 5216 5215 5214
12694 Upstairs last five completed: 4219 4218 4217 4216 4215
12695 CLIENT: Up:0 ds:0 act:3
12696 Replay: All jobs finished, all DS active.
12697 Main task has work to do, go find it
12698 got dsc Stop(2)
12699 stop 2
12700 Feb 27 21:47:10.911 INFO request completed, latency_us: 346, response_code: 204, uri: /stop/cid/2, method: POST[8830] Got stop action so:false kr:false
12701 , req_id: 37e8365a-834c-4019-97ee-747c08b745f5, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
12702 Feb 27 21:47:10.911 INFO request completed, latency_us: 134, response_code: 200, uri: /state/cid/2, method: GET, req_id: b87530ef-a1b4-4933-aa75-1e8d43963aac, remote_addr:{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-27T21:47:10.911932188Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"io task","client":"2","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
12703 127.0.0.1:44347, local_addr: 127.0.0.1:9998
127042024-02-27T21:47:10.912ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
127052024-02-27T21:47:10.912ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = e6a5a974-925a-494f-8845-facc441c2690
127062024-02-27T21:47:10.912ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
127072024-02-27T21:47:10.912ZINFOcrucible: [2] client re-new 0 jobs since flush 5218 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
127082024-02-27T21:47:10.912ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
12709 [8830] Exited with: ExitStatus(unix_wait_status(9))
12710 [8830][2] reports Exit
127112024-02-27T21:47:11.913ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
127122024-02-27T21:47:11.913ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
127132024-02-27T21:47:11.913ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
127142024-02-27T21:47:11.913ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
127152024-02-27T21:47:11.913ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
127162024-02-27T21:47:11.913ZINFOcrucible: [2] client re-new 0 jobs since flush 5218 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
127172024-02-27T21:47:11.913ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
127182024-02-27T21:47:12.914ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
127192024-02-27T21:47:12.914ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
127202024-02-27T21:47:12.914ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
127212024-02-27T21:47:12.914ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
127222024-02-27T21:47:12.914ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
127232024-02-27T21:47:12.914ZINFOcrucible: [2] client re-new 0 jobs since flush 5218 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
127242024-02-27T21:47:12.914ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
127252024-02-27T21:47:13.916ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
127262024-02-27T21:47:13.916ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
127272024-02-27T21:47:13.916ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
127282024-02-27T21:47:13.916ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
127292024-02-27T21:47:13.916ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
127302024-02-27T21:47:13.916ZINFOcrucible: [2] client re-new 0 jobs since flush 5218 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
127312024-02-27T21:47:13.916ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
12732 Feb 27 21:47:14.914 INFO request completed, latency_us: 207, response_code: 200, uri: /state/cid/2, method: GET, req_id: ea40f608-8332-478a-a320-722a2df040d5, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
12733 001/300 Write block 826 len 4096 data: 5
12734 002/300 Write block 1560 len 12288 data: 4 4 3
12735 003/300 Read block 4914 len 28672
12736 004/300 Read block 678 len 32768
127372024-02-27T21:47:14.918ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
127382024-02-27T21:47:14.918ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
127392024-02-27T21:47:14.918ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
127402024-02-27T21:47:14.918ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
127412024-02-27T21:47:14.918ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
127422024-02-27T21:47:14.918ZINFOcrucible: [2] client re-new 4 jobs since flush 5218 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
127432024-02-27T21:47:14.918ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
12744 005/300 Flush
12745 006/300 Write block 3945 len 40960 data: 6 6 5 4 3 3 3 4 6 6
12746 007/300 Write block 3520 len 24576 data: 5 5 5 5 5 6
12747 008/300 Read block 854 len 4096
12748 009/300 Flush
12749 010/300 Write block 2424 len 24576 data: 3 3 3 3 3 3
12750 011/300 Write block 3171 len 8192 data: 4 3
12751 012/300 Write block 977 len 40960 data: 4 3 3 2 2 3 5 4 5 4
12752 013/300 Read block 4887 len 28672
12753 014/300 Write block 1827 len 36864 data: 4 4 3 4 5 5 5 5 4
12754 015/300 Write block 4954 len 28672 data: 3 2 3 3 3 3 3
12755 016/300 Read block 2508 len 24576
12756 017/300 Read block 2416 len 16384
12757 018/300 Read block 1382 len 28672
12758 019/300 Read block 1630 len 4096
12759 020/300 Write block 2137 len 12288 data: 3 3 4
12760 021/300 Write block 1996 len 28672 data: 6 6 4 4 4 4 7
12761 022/300 Write block 883 len 28672 data: 3 2 2 2 4 3 4
12762 023/300 Write block 668 len 20480 data: 5 5 6 5 5
12763 024/300 Write block 1247 len 20480 data: 3 3 3 2 2
12764 025/300 Read block 3242 len 24576
12765 026/300 Write block 3839 len 12288 data: 3 4 4
12766 027/300 Write block 2778 len 40960 data: 2 3 5 5 5 2 2 2 2 2
12767 028/300 Write block 3930 len 36864 data: 3 2 3 4 3 3 3 3 3
12768 029/300 Write block 2441 len 16384 data: 3 4 4 4
12769 030/300 Read block 2125 len 28672
12770 031/300 Read block 2879 len 36864
12771 032/300 Read block 4691 len 8192
12772 033/300 Flush
12773 034/300 Read block 2677 len 40960
12774 035/300 Read block 4917 len 8192
12775 036/300 Flush
12776 037/300 Read block 2745 len 40960
12777 038/300 Write block 548 len 20480 data: 3 3 3 4 6
12778 039/300 Read block 3888 len 36864
12779 040/300 Read block 3908 len 8192
12780 041/300 Write block 1597 len 4096 data: 3
12781 042/300 Flush
12782 043/300 Flush
12783 044/300 Write block 157 len 20480 data: 4 5 5 4 4
12784 045/300 Read block 2652 len 36864
12785 046/300 Write block 882 len 40960 data: 3 4 3 3 3 5 4 5 4 4
12786 047/300 Write block 2985 len 32768 data: 2 2 2 3 3 4 4 5
12787 048/300 Read block 2300 len 20480
12788 049/300 Flush
12789 050/300 Flush
12790 051/300 Write block 3946 len 4096 data: 7
12791 052/300 Write block 1959 len 28672 data: 2 2 2 2 2 3 2
12792 053/300 Read block 3931 len 28672
12793 054/300 Read block 1937 len 20480
12794 055/300 Write block 2885 len 4096 data: 2
12795 056/300 Read block 3530 len 4096
12796 057/300 Write block 1497 len 24576 data: 2 2 2 2 4 4
12797 058/300 Read block 260 len 40960
12798 059/300 Read block 963 len 12288
12799 060/300 Read block 3953 len 4096
12800 061/300 Write block 3795 len 24576 data: 4 5 4 6 6 6
12801 062/300 Read block 3408 len 28672
12802 063/300 Flush
12803 064/300 Flush
12804 065/300 Write block 3859 len 8192 data: 3 4
12805 066/300 Write block 2375 len 8192 data: 4 4
12806 067/300 Read block 3668 len 32768
12807 068/300 Read block 1943 len 20480
12808 069/300 Write block 4069 len 28672 data: 2 2 2 2 2 2 3
12809 070/300 Read block 2936 len 16384
12810 071/300 Read block 1503 len 8192
12811 072/300 Read block 3620 len 4096
12812 073/300 Read block 3145 len 16384
12813 074/300 Read block 3770 len 16384
12814 075/300 Write block 3410 len 24576 data: 6 6 6 6 5 5
12815 076/300 Write block 224 len 28672 data: 5 5 6 5 4 3 3
12816 077/300 Read block 2422 len 4096
12817 078/300 Write block 3246 len 8192 data: 3 2
12818 079/300 Write block 3039 len 12288 data: 4 4 4
12819 080/300 Write block 364 len 28672 data: 2 2 2 2 2 2 2
12820 081/300 Write block 2463 len 4096 data: 6
12821 082/300 Write block 2328 len 36864 data: 3 3 3 4 3 3 6 7 7
12822 083/300 Flush
12823 084/300 Read block 2077 len 28672
12824 085/300 Read block 1150 len 12288
12825 086/300 Write block 4881 len 12288 data: 3 3 4
12826 087/300 Write block 3728 len 24576 data: 2 2 2 2 3 3
12827 088/300 Flush
12828 089/300 Read block 3344 len 8192
12829 090/300 Read block 2024 len 32768
12830 091/300 Write block 4264 len 24576 data: 7 7 7 6 5 5
12831 092/300 Read block 1985 len 24576
12832 093/300 Read block 3248 len 20480
12833 094/300 Read block 3591 len 28672
12834 095/300 Read block 1721 len 12288
12835 096/300 Write block 44 len 8192 data: 5 5
12836 097/300 Read block 4649 len 24576
12837 098/300 Write block 2663 len 16384 data: 5 4 6 5
12838 099/300 Write block 2197 len 16384 data: 6 6 6 6
12839 100/300 Flush
12840 101/300 Write block 4219 len 16384 data: 5 5 5 4
12841 102/300 Read block 588 len 20480
12842 103/300 Read block 1429 len 32768
12843 104/300 Write block 3546 len 8192 data: 3 3
12844 105/300 Flush
12845 106/300 Read block 3611 len 8192
12846 107/300 Read block 4130 len 20480
12847 108/300 Write block 362 len 4096 data: 5
12848 109/300 Write block 1313 len 40960 data: 6 6 7 6 6 7 8 8 8 7
12849 110/300 Read block 2524 len 24576
12850 111/300 Write block 2267 len 12288 data: 5 5 4
12851 112/300 Read block 67 len 24576
12852 113/300 Read block 3310 len 16384
12853 114/300 Read block 3123 len 8192
12854 115/300 Read block 1346 len 4096
12855 116/300 Read block 3292 len 36864
12856 117/300 Read block 1613 len 32768
12857 118/300 Read block 2047 len 36864
12858 119/300 Flush
12859 120/300 Read block 2887 len 20480
12860 121/300 Write block 221 len 4096 data: 3
12861 122/300 Read block 2947 len 8192
12862 123/300 Write block 2056 len 12288 data: 4 5 4
12863 124/300 Read block 3672 len 28672
12864 125/300 Flush
12865 126/300 Write block 4793 len 8192 data: 5 5
12866 127/300 Write block 2579 len 16384 data: 4 3 2 2
12867 128/300 Write block 2789 len 40960 data: 2 3 3 4 4 4 4 5 5 5
12868 129/300 Flush
12869 130/300 Read block 95 len 32768
12870 131/300 Write block 515 len 4096 data: 4
12871 132/300 Write block 1794 len 36864 data: 5 4 3 2 2 2 4 4 4
12872 133/300 Read block 4107 len 4096
12873 134/300 Read block 2103 len 8192
12874 135/300 Read block 4368 len 24576
12875 136/300 Read block 102 len 4096
12876 137/300 Read block 2014 len 8192
12877 138/300 Read block 1174 len 40960
12878 139/300 Flush
12879 140/300 Write block 2097 len 28672 data: 4 4 4 4 4 3 3
12880 141/300 Read block 4408 len 4096
12881 142/300 Write block 879 len 24576 data: 2 2 2 4 5 4
12882 143/300 Write block 4670 len 16384 data: 4 4 4 4
12883 144/300 Read block 1157 len 36864
12884 145/300 Flush
12885 146/300 Read block 4316 len 4096
12886 147/300 Read block 65 len 12288
12887 148/300 Read block 3303 len 4096
12888 149/300 Write block 675 len 12288 data: 5 6 6
12889 150/300 Read block 775 len 8192
12890 151/300 Write block 3224 len 36864 data: 4 4 4 3 3 4 4 4 5
12891 152/300 Read block 4774 len 40960
12892 153/300 Read block 4273 len 12288
12893 154/300 Read block 3064 len 36864
12894 155/300 Flush
12895 156/300 Read block 4665 len 4096
12896 157/300 Read block 1930 len 24576
12897 158/300 Read block 879 len 16384
12898 159/300 Read block 1675 len 24576
12899 160/300 Write block 2277 len 28672 data: 5 5 5 5 4 4 3
12900 161/300 Read block 2462 len 16384
12901 162/300 Write block 891 len 36864 data: 5 4 4 4 4 3 2 2 2
12902 163/300 Read block 1935 len 36864
12903 164/300 Read block 437 len 4096
12904 165/300 Write block 1579 len 28672 data: 5 4 4 3 3 4 4
12905 166/300 Read block 2697 len 28672
12906 167/300 Read block 777 len 28672
12907 168/300 Flush
12908 169/300 Write block 1077 len 32768 data: 4 4 3 4 4 5 6 6
12909 170/300 Read block 3072 len 24576
12910 171/300 Write block 3573 len 40960 data: 5 4 4 4 3 4 3 2 2 2
12911 172/300 Write block 1696 len 16384 data: 3 2 2 2
12912 173/300 Write block 2622 len 20480 data: 4 5 7 7 7
12913 174/300 Write block 2438 len 4096 data: 2
12914 175/300 Read block 2819 len 12288
12915 176/300 Write block 3203 len 40960 data: 3 3 3 2 2 2 2 2 2 2
12916 177/300 Write block 4319 len 8192 data: 6 6
12917 178/300 Write block 2148 len 24576 data: 2 2 2 2 2 3
12918 179/300 Read block 2121 len 24576
12919 180/300 Read block 531 len 28672
12920 181/300 Read block 2544 len 24576
12921 182/300 Write block 345 len 20480 data: 2 2 3 3 3
12922 183/300 Write block 2967 len 28672 data: 5 4 3 4 4 3 4
12923 184/300 Write block 2043 len 4096 data: 2
12924 185/300 Read block 34 len 32768
12925 186/300 Write block 3818 len 36864 data: 5 5 5 2 2 2 2 3 2
12926 187/300 Write block 3987 len 12288 data: 4 2 2
12927 188/300 Write block 3445 len 36864 data: 4 5 5 5 4 4 4 4 3
12928 189/300 Write block 4234 len 8192 data: 2 2
12929 190/300 Write block 3008 len 36864 data: 4 4 4 4 4 3 3 4 5
12930 191/300 Write block 4615 len 24576 data: 3 4 4 3 4 4
12931 192/300 Write block 4775 len 16384 data: 5 5 5 5
12932 193/300 Read block 2116 len 28672
12933 194/300 Read block 3109 len 28672
12934 195/300 Read block 4252 len 32768
12935 196/300 Write block 3583 len 40960 data: 2 2 2 2 4 5 5 4 4 5
12936 197/300 Write block 266 len 40960 data: 3 4 5 4 4 4 4 4 3 5
12937 198/300 Write block 4458 len 4096 data: 3
12938 199/300 Read block 2751 len 32768
12939 200/300 Flush
12940 201/300 Write block 144 len 12288 data: 2 3 4
12941 202/300 Write block 2167 len 8192 data: 4 5
12942 203/300 Read block 2746 len 32768
12943 204/300 Read block 139 len 36864
12944 205/300 Read block 3363 len 8192
12945 206/300 Read block 1758 len 16384
12946 207/300 Read block 3776 len 20480
12947 208/300 Read block 2040 len 8192
12948 209/300 Write block 4647 len 4096 data: 6
12949 210/300 Write block 1257 len 36864 data: 3 4 4 4 3 3 4 3 3
12950 211/300 Write block 2363 len 20480 data: 3 3 2 3 3
12951 212/300 Write block 532 len 36864 data: 3 3 2 2 3 4 3 2 2
12952 213/300 Read block 2517 len 4096
12953 214/300 Read block 3269 len 24576
12954 215/300 Flush
12955 216/300 Write block 509 len 32768 data: 5 7 7 8 6 4 5 5
12956 217/300 Write block 1890 len 32768 data: 4 3 2 2 2 2 2 2
12957 218/300 Flush
12958 219/300 Write block 4727 len 32768 data: 2 2 2 2 2 3 5 5
12959 220/300 Flush
12960 221/300 Read block 2400 len 4096
12961 222/300 Read block 614 len 32768
12962 223/300 Write block 1865 len 16384 data: 4 4 3 3
12963 224/300 Write block 3241 len 12288 data: 6 4 4
12964 225/300 Write block 3052 len 36864 data: 3 2 3 4 5 5 5 5 5
12965 226/300 Read block 2368 len 28672
12966 227/300 Read block 3776 len 28672
12967 228/300 Flush
12968 229/300 Write block 2942 len 36864 data: 4 4 4 4 3 3 4 4 4
12969 230/300 Write block 3445 len 8192 data: 5 6
12970 231/300 Read block 1703 len 16384
12971 232/300 Read block 2124 len 4096
12972 233/300 Write block 1202 len 40960 data: 5 5 5 4 3 4 4 4 3 3
12973 234/300 Write block 717 len 12288 data: 5 4 4
12974 235/300 Flush
12975 236/300 Write block 4089 len 20480 data: 3 3 3 5 5
12976 237/300 Flush
12977 238/300 Read block 4533 len 32768
12978 239/300 Write block 3798 len 36864 data: 7 7 7 6 5 5 4 4 3
12979 240/300 Read block 3573 len 32768
12980 241/300 Write block 535 len 16384 data: 3 4 5 4
12981 242/300 Write block 470 len 36864 data: 4 4 4 4 4 3 3 3 2
12982 243/300 Read block 1971 len 40960
12983 244/300 Flush
12984 245/300 Read block 662 len 4096
12985 246/300 Read block 2339 len 8192
12986 247/300 Read block 1488 len 32768
12987 248/300 Read block 2961 len 24576
12988 249/300 Read block 1461 len 12288
12989 250/300 Write block 1490 len 16384 data: 4 4 4 5
12990 251/300 Write block 484 len 20480 data: 3 4 4 5 5
12991 252/300 Read block 4690 len 8192
12992 253/300 Read block 2024 len 16384
12993 254/300 Read block 4850 len 20480
12994 255/300 Write block 4374 len 8192 data: 3 4
12995 256/300 Read block 269 len 28672
12996 257/300 Write block 1382 len 16384 data: 3 3 3 3
12997 258/300 Write block 4148 len 40960 data: 5 5 4 4 4 4 4 3 2 2
12998 259/300 Read block 570 len 4096
12999 260/300 Read block 2858 len 12288
13000 261/300 Write block 1628 len 16384 data: 3 2 2 3
13001 262/300 Read block 1766 len 20480
13002 263/300 Write block 4533 len 12288 data: 2 2 2
13003 264/300 Read block 4513 len 24576
13004 265/300 Write block 3763 len 36864 data: 2 2 2 2 2 3 3 3 4
13005 266/300 Write block 163 len 16384 data: 3 3 3 3
13006 267/300 Read block 1241 len 12288
13007 268/300 Read block 2700 len 16384
13008 269/300 Write block 910 len 12288 data: 6 6 6
13009 270/300 Read block 2572 len 12288
13010 271/300 Write block 4014 len 8192 data: 4 3
13011 272/300 Write block 4356 len 36864 data: 3 3 3 5 4 5 5 5 5
13012 273/300 Read block 2265 len 32768
13013 274/300 Read block 2823 len 40960
13014 275/300 Write block 4015 len 24576 data: 4 3 3 5 6 6
13015 276/300 Read block 4896 len 32768
13016 277/300 Flush
13017 278/300 Flush
13018 279/300 Write block 1427 len 32768 data: 4 3 3 3 3 4 5 4
13019 280/300 Write block 1745 len 4096 data: 2
13020 281/300 Flush
13021 282/300 Write block 3920 len 36864 data: 3 3 3 3 4 4 4 4 3
13022 283/300 Write block 3055 len 16384 data: 5 6 6 6
13023 284/300 Read block 2043 len 32768
13024 285/300 Read block 147 len 12288
13025 286/300 Flush
13026 287/300 Read block 255 len 4096
13027 288/300 Read block 484 len 40960
13028 289/300 Flush
13029 290/300 Read block 1600 len 16384
13030 291/300 Write block 507 len 36864 data: 5 5 6 8 8 9 7 5 6
13031 292/300 Read block 2785 len 12288
13032 293/300 Write block 4693 len 8192 data: 6 5
13033 294/300 Read block 889 len 16384
13034 295/300 Read block 2754 len 20480
13035 296/300 Read block 2436 len 20480
13036 297/300 Read block 2197 len 12288
13037 298/300 Read block 4988 len 12288
13038 299/300 Write block 1697 len 28672 data: 3 3 3 2 2 2 3
13039 300/300 Read block 40 len 16384
13040 Main task has work to do, go find it
13041 got dsc Start(2)
13042 start 2
13043 Feb 27 21:47:15.257 INFO request completed, latency_us: 209, response_code: 204, uri: /start/cid/2, method: POST, req_id: bda3f865-7ce6-4c29-adae-b8fa027c1a2e, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
13044 Replay: started 2, returned:Ok(())
13045 ----------------------------------------------------------------
13046 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13047 States: Active Active Offline
13048 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
13049 4220 Acked 5219 Write 1 Done Done New true
13050 4221 Acked 5220 Write 3 Done Done New true
13051 4222 Acked 5221 Read 7 Done Done New true
13052 4223 Acked 5222 Read 8 Done Done New true
13053 4224 Acked 5223 Flush 0 Done Done New false
13054 4225 Acked 5224 Write 10 Done Done New false
13055 4226 Acked 5225 Write 6 Done Done New false
13056 4227 Acked 5226 Read 1 Done Done New false
13057 4228 Acked 5227 Flush 0 Done Done New false
13058 4229 Acked 5228 Write 6 Done Done New false
13059 4230 Acked 5229 Write 2 Done Done New false
13060 4231 Acked 5230 Write 10 Done Done New false
13061 4232 Acked 5231 Read 7 Done Done New false
13062 4233 Acked 5232 Write 9 Done Done New false
13063 4234 Acked 5233 Write 7 Done Done New false
13064 4235 Acked 5234 Read 6 Done Done New false
13065 4236 Acked 5235 Read 4 Done Done New false
13066 4237 Acked 5236 Read 7 Done Done New false
13067 4238 Acked 5237 Read 1 Done Done New false
13068 4239 Acked 5238 Write 3 Done Done New false
13069 4240 Acked 5239 Write 7 Done Done New false
13070 4241 Acked 5240 Write 7 Done Done New false
13071 4242 Acked 5241 Write 5 Done Done New false
13072 4243 Acked 5242 Write 5 Done Done New false
13073 4244 Acked 5243 Read 6 Done Done New false
13074 4245 Acked 5244 Write 3 Done Done New false
13075 4246 Acked 5245 Write 10 Done Done New false
13076 4247 Acked 5246 Write 9 Done Done New false
13077 4248 Acked 5247 Write 4 Done Done New false
13078 4249 Acked 5248 Read 7 Done Done New false
13079 4250 Acked 5249 Read 9 Done Done New false
13080 4251 Acked 5250 Read 2 Done Done New false
13081 4252 Acked 5251 Flush 0 Done Done New false
13082 4253 Acked 5252 Read 10 Done Done New false
13083 4254 Acked 5253 Read 2 Done Done New false
13084 4255 Acked 5254 Flush 0 Done Done New false
13085 4256 Acked 5255 Read 10 Done Done New false
13086 4257 Acked 5256 Write 5 Done Done New false
13087 4258 Acked 5257 Read 9 Done Done New false
13088 4259 Acked 5258 Read 2 Done Done New false
13089 4260 Acked 5259 Write 1 Done Done New false
13090 4261 Acked 5260 Flush 0 Done Done New false
13091 4262 Acked 5261 Flush 0 Done Done New false
13092 4263 Acked 5262 Write 5 Done Done New false
13093 4264 Acked 5263 Read 9 Done Done New false
13094 4265 Acked 5264 Write 10 Done Done New false
13095 4266 Acked 5265 Write 8 Done Done New false
13096 4267 Acked 5266 Read 5 Done Done New false
13097 4268 Acked 5267 Flush 0 Done Done New false
13098 4269 Acked 5268 Flush 0 Done Done New false
13099 4270 Acked 5269 Write 1 Done Done New false
13100 4271 Acked 5270 Write 7 Done Done New false
13101 4272 Acked 5271 Read 7 Done Done New false
13102 4273 Acked 5272 Read 5 Done Done New false
13103 4274 Acked 5273 Write 1 Done Done New false
13104 4275 Acked 5274 Read 1 Done Done New false
13105 4276 Acked 5275 Write 6 Done Done New false
13106 4277 Acked 5276 Read 10 Done Done New false
13107 4278 Acked 5277 Read 3 Done Done New false
13108 4279 Acked 5278 Read 1 Done Done New false
13109 4280 Acked 5279 Write 6 Done Done New false
13110 4281 Acked 5280 Read 7 Done Done New false
13111 4282 Acked 5281 Flush 0 Done Done New false
13112 4283 Acked 5282 Flush 0 Done Done New false
13113 4284 Acked 5283 Write 2 Done Done New false
13114 4285 Acked 5284 Write 2 Done Done New false
13115 4286 Acked 5285 Read 8 Done Done New false
13116 4287 Acked 5286 Read 5 Done Done New false
13117 4288 Acked 5287 Write 7 Done Done New false
13118 4289 Acked 5288 Read 4 Done Done New false
13119 4290 Acked 5289 Read 2 Done Done New false
13120 4291 Acked 5290 Read 1 Done Done New false
13121 4292 Acked 5291 Read 4 Done Done New false
13122 4293 Acked 5292 Read 4 Done Done New false
13123 4294 Acked 5293 Write 6 Done Done New false
13124 4295 Acked 5294 Write 7 Done Done New false
13125 4296 Acked 5295 Read 1 Done Done New false
13126 4297 Acked 5296 Write 2 Done Done New false
13127 4298 Acked 5297 Write 3 Done Done New false
13128 4299 Acked 5298 Write 7 Done Done New false
13129 4300 Acked 5299 Write 1 Done Done New false
13130 4301 Acked 5300 Write 9 Done Done New false
13131 4302 Acked 5301 Flush 0 Done Done New false
13132 4303 Acked 5302 Read 7 Done Done New false
13133 4304 Acked 5303 Read 3 Done Done New false
13134 4305 Acked 5304 Write 3 Done Done New false
13135 4306 Acked 5305 Write 6 Done Done New false
13136 4307 Acked 5306 Flush 0 Done Done New false
13137 4308 Acked 5307 Read 2 Done Done New false
13138 4309 Acked 5308 Read 8 Done Done New false
13139 4310 Acked 5309 Write 6 Done Done New false
13140 4311 Acked 5310 Read 6 Done Done New false
13141 4312 Acked 5311 Read 5 Done Done New false
13142 4313 Acked 5312 Read 7 Done Done New false
13143 4314 Acked 5313 Read 3 Done Done New false
13144 4315 Acked 5314 Write 2 Done Done New false
13145 4316 Acked 5315 Read 6 Done Done New false
13146 4317 Acked 5316 Write 4 Done Done New false
13147 4318 Acked 5317 Write 4 Done Done New false
13148 4319 Acked 5318 Flush 0 Done Done New false
13149 4320 Acked 5319 Write 4 Done Done New false
13150 4321 Acked 5320 Read 5 Done Done New false
13151 4322 Acked 5321 Read 8 Done Done New false
13152 4323 Acked 5322 Write 2 Done Done New false
13153 4324 Acked 5323 Flush 0 Done Done New false
13154 4325 Acked 5324 Read 2 Done Done New false
13155 4326 Acked 5325 Read 5 Done Done New false
13156 4327 Acked 5326 Write 1 Done Done New false
13157 4328 Acked 5327 Write 10 Done Done New false
13158 4329 Acked 5328 Read 6 Done Done New false
13159 4330 Acked 5329 Write 3 Done Done New false
13160 4331 Acked 5330 Read 6 Done Done New false
13161 4332 Acked 5331 Read 4 Done Done New false
13162 4333 Acked 5332 Read 2 Done Done New false
13163 4334 Acked 5333 Read 1 Done Done New false
13164 4335 Acked 5334 Read 9 Done Done New false
13165 4336 Acked 5335 Read 8 Done Done New false
13166 4337 Acked 5336 Read 9 Done Done New false
13167 4338 Acked 5337 Flush 0 Done Done New false
13168 4339 Acked 5338 Read 5 Done Done New false
13169 4340 Acked 5339 Write 1 Done Done New false
13170 4341 Acked 5340 Read 2 Done Done New false
13171 4342 Acked 5341 Write 3 Done Done New false
13172 4343 Acked 5342 Read 7 Done Done New false
13173 4344 Acked 5343 Flush 0 Done Done New false
13174 4345 Acked 5344 Write 2 Done Done New false
13175 4346 Acked 5345 Write 4 Done Done New false
13176 4347 Acked 5346 Write 10 Done Done New false
13177 4348 Acked 5347 Flush 0 Done Done New false
13178 4349 Acked 5348 Read 8 Done Done New false
13179 4350 Acked 5349 Write 1 Done Done New false
13180 4351 Acked 5350 Write 9 Done Done New false
13181 4352 Acked 5351 Read 1 Done Done New false
13182 4353 Acked 5352 Read 2 Done Done New false
13183 4354 Acked 5353 Read 6 Done Done New false
13184 4355 Acked 5354 Read 1 Done Done New false
13185 4356 Acked 5355 Read 2 Done Done New false
13186 4357 Acked 5356 Read 10 Done Done New false
13187 4358 Acked 5357 Flush 0 Done Done New false
13188 4359 Acked 5358 Write 7 Done Done New false
13189 4360 Acked 5359 Read 1 Done Done New false
13190 4361 Acked 5360 Write 6 Done Done New false
13191 4362 Acked 5361 Write 4 Done Done New false
13192 4363 Acked 5362 Read 9 Done Done New false
13193 4364 Acked 5363 Flush 0 Done Done New false
13194 4365 Acked 5364 Read 1 Done Done New false
13195 4366 Acked 5365 Read 3 Done Done New false
13196 4367 Acked 5366 Read 1 Done Done New false
13197 4368 Acked 5367 Write 3 Done Done New false
13198 4369 Acked 5368 Read 2 Done Done New false
13199 4370 Acked 5369 Write 9 Done Done New false
13200 4371 Acked 5370 Read 10 Done Done New false
13201 4372 Acked 5371 Read 3 Done Done New false
13202 4373 Acked 5372 Read 9 Done Done New false
13203 4374 Acked 5373 Flush 0 Done Done New false
13204 4375 Acked 5374 Read 1 Done Done New false
13205 4376 Acked 5375 Read 6 Done Done New false
13206 4377 Acked 5376 Read 4 Done Done New false
13207 4378 Acked 5377 Read 6 Done Done New false
13208 4379 Acked 5378 Write 7 Done Done New false
13209 4380 Acked 5379 Read 4 Done Done New false
13210 4381 Acked 5380 Write 9 Done Done New false
13211 4382 Acked 5381 Read 9 Done Done New false
13212 4383 Acked 5382 Read 1 Done Done New false
13213 4384 Acked 5383 Write 7 Done Done New false
13214 4385 Acked 5384 Read 7 Done Done New false
13215 4386 Acked 5385 Read 7 Done Done New false
13216 4387 Acked 5386 Flush 0 Done Done New false
13217 4388 Acked 5387 Write 8 Done Done New false
13218 4389 Acked 5388 Read 6 Done Done New false
13219 4390 Acked 5389 Write 10 Done Done New false
13220 4391 Acked 5390 Write 4 Done Done New false
13221 4392 Acked 5391 Write 5 Done Done New false
13222 4393 Acked 5392 Write 1 Done Done New false
13223 4394 Acked 5393 Read 3 Done Done New false
13224 4395 Acked 5394 Write 10 Done Done New false
13225 4396 Acked 5395 Write 2 Done Done New false
13226 4397 Acked 5396 Write 6 Done Done New false
13227 4398 Acked 5397 Read 6 Done Done New false
13228 4399 Acked 5398 Read 7 Done Done New false
13229 4400 Acked 5399 Read 6 Done Done New false
13230 4401 Acked 5400 Write 5 Done Done New false
13231 4402 Acked 5401 Write 7 Done Done New false
13232 4403 Acked 5402 Write 1 Done Done New false
13233 4404 Acked 5403 Read 8 Done Done New false
13234 4405 Acked 5404 Write 9 Done Done New false
13235 4406 Acked 5405 Write 3 Done Done New false
13236 4407 Acked 5406 Write 9 Done Done New false
13237 4408 Acked 5407 Write 2 Done Done New false
13238 4409 Acked 5408 Write 9 Done Done New false
13239 4410 Acked 5409 Write 6 Done Done New false
13240 4411 Acked 5410 Write 4 Done Done New false
13241 4412 Acked 5411 Read 7 Done Done New false
13242 4413 Acked 5412 Read 7 Done Done New false
13243 4414 Acked 5413 Read 8 Done Done New false
13244 4415 Acked 5414 Write 10 Done Done New false
13245 4416 Acked 5415 Write 10 Done Done New false
13246 4417 Acked 5416 Write 1 Done Done New false
13247 4418 Acked 5417 Read 8 Done Done New false
13248 4419 Acked 5418 Flush 0 Done Done New false
13249 4420 Acked 5419 Write 3 Done Done New false
13250 4421 Acked 5420 Write 2 Done Done New false
13251 4422 Acked 5421 Read 8 Done Done New false
13252 4423 Acked 5422 Read 9 Done Done New false
13253 4424 Acked 5423 Read 2 Done Done New false
13254 4425 Acked 5424 Read 4 Done Done New false
13255 4426 Acked 5425 Read 5 Done Done New false
13256 4427 Acked 5426 Read 2 Done Done New false
13257 4428 Acked 5427 Write 1 Done Done New false
13258 4429 Acked 5428 Write 9 Done Done New false
13259 4430 Acked 5429 Write 5 Done Done New false
13260 4431 Acked 5430 Write 9 Done Done New false
13261 4432 Acked 5431 Read 1 Done Done New false
13262 4433 Acked 5432 Read 6 Done Done New false
13263 4434 Acked 5433 Flush 0 Done Done New false
13264 4435 Acked 5434 Write 8 Done Done New false
13265 4436 Acked 5435 Write 8 Done Done New false
13266 4437 Acked 5436 Flush 0 Done Done New false
13267 4438 Acked 5437 Write 8 Done Done New false
13268 4439 Acked 5438 Flush 0 Done Done New false
13269 4440 Acked 5439 Read 1 Done Done New false
13270 4441 Acked 5440 Read 8 Done Done New false
13271 4442 Acked 5441 Write 4 Done Done New false
13272 4443 Acked 5442 Write 3 Done Done New false
13273 4444 Acked 5443 Write 9 Done Done New false
13274 4445 Acked 5444 Read 7 Done Done New false
13275 4446 Acked 5445 Read 7 Done Done New false
13276 4447 Acked 5446 Flush 0 Done Done New false
13277 4448 Acked 5447 Write 9 Done Done New false
13278 4449 Acked 5448 Write 2 Done Done New false
13279 4450 Acked 5449 Read 4 Done Done New false
13280 4451 Acked 5450 Read 1 Done Done New false
13281 4452 Acked 5451 Write 10 Done Done New false
13282 4453 Acked 5452 Write 3 Done Done New false
13283 4454 Acked 5453 Flush 0 Done Done New false
13284 4455 Acked 5454 Write 5 Done Done New false
13285 4456 Acked 5455 Flush 0 Done Done New false
13286 4457 Acked 5456 Read 8 Done Done New false
13287 4458 Acked 5457 Write 9 Done Done New false
13288 4459 Acked 5458 Read 8 Done Done New false
13289 4460 Acked 5459 Write 4 Done Done New false
13290 4461 Acked 5460 Write 9 Done Done New false
13291 4462 Acked 5461 Read 10 Done Done New false
13292 4463 Acked 5462 Flush 0 Done Done New false
13293 4464 Acked 5463 Read 1 Done Done New false
13294 4465 Acked 5464 Read 2 Done Done New false
13295 4466 Acked 5465 Read 8 Done Done New false
13296 4467 Acked 5466 Read 6 Done Done New false
13297 4468 Acked 5467 Read 3 Done Done New false
13298 4469 Acked 5468 Write 4 Done Done New false
13299 4470 Acked 5469 Write 5 Done Done New false
13300 4471 Acked 5470 Read 2 Done Done New false
13301 4472 Acked 5471 Read 4 Done Done New false
13302 4473 Acked 5472 Read 5 Done Done New false
13303 4474 Acked 5473 Write 2 Done Done New false
13304 4475 Acked 5474 Read 7 Done Done New false
13305 4476 Acked 5475 Write 4 Done Done New false
13306 4477 Acked 5476 Write 10 Done Done New false
13307 4478 Acked 5477 Read 1 Done Done New false
13308 4479 Acked 5478 Read 3 Done Done New false
13309 4480 Acked 5479 Write 4 Done Done New false
13310 4481 Acked 5480 Read 5 Done Done New false
13311 4482 Acked 5481 Write 3 Done Done New false
13312 4483 Acked 5482 Read 6 Done Done New false
13313 4484 Acked 5483 Write 9 Done Done New false
13314 4485 Acked 5484 Write 4 Done Done New false
13315 4486 Acked 5485 Read 3 Done Done New false
13316 4487 Acked 5486 Read 4 Done Done New false
13317 4488 Acked 5487 Write 3 Done Done New false
13318 4489 Acked 5488 Read 3 Done Done New false
13319 4490 Acked 5489 Write 2 Done Done New false
13320 4491 Acked 5490 Write 9 Done Done New false
13321 4492 Acked 5491 Read 8 Done Done New false
13322 4493 Acked 5492 Read 10 Done Done New false
13323 4494 Acked 5493 Write 6 Done Done New false
13324 4495 Acked 5494 Read 8 Done Done New false
13325 4496 Acked 5495 Flush 0 Done Done New false
13326 4497 Acked 5496 Flush 0 Done Done New false
13327 4498 Acked 5497 Write 8 Done Done New false
13328 4499 Acked 5498 Write 1 Done Done New false
13329 4500 Acked 5499 Flush 0 Done Done New false
13330 4501 Acked 5500 Write 9 Done Done New false
13331 4502 Acked 5501 Write 4 Done Done New false
13332 4503 Acked 5502 Read 8 Done Done New false
13333 4504 Acked 5503 Read 3 Done Done New false
13334 4505 Acked 5504 Flush 0 Done Done New false
13335 4506 Acked 5505 Read 1 Done Done New false
13336 4507 Acked 5506 Read 10 Done Done New false
13337 4508 Acked 5507 Flush 0 Done Done New false
13338 4509 Acked 5508 Read 4 Done Done New false
13339 4510 Acked 5509 Write 9 Done Done New false
13340 4511 Acked 5510 Read 3 Done Done New false
13341 4512 Acked 5511 Flush 0 Done Done New false
13342 4513 Acked 5512 Write 2 Done Done New false
13343 4514 Acked 5513 Read 4 Done Done New false
13344 4515 Acked 5514 Read 5 Done Done New false
13345 4516 Acked 5515 Read 5 Done Done New false
13346 4517 Acked 5516 Read 3 Done Done New false
13347 4518 Acked 5517 Read 3 Done Done New false
13348 4519 Acked 5518 Write 7 Done Done New false
13349 4520 Acked 5519 Read 4 Done Done New false
13350 STATES DS:0 DS:1 DS:2 TOTAL
13351 New 0 0 301 301
13352 Sent 0 0 0 0
13353 Done 301 301 0 602
13354 Skipped 0 0 0 0
13355 Error 0 0 0 0
13356 Last Flush: 5511 5511 5218
13357 Downstairs last five completed: 5218 5217 5216 5215 5214
13358 Upstairs last five completed: 4520 4519 4518 4517 4516
13359 CLIENT: Up:0 ds:301 act:2
133602024-02-27T21:47:15.919ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
133612024-02-27T21:47:15.919ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
133622024-02-27T21:47:15.919ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
133632024-02-27T21:47:15.919ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
133642024-02-27T21:47:15.919ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
133652024-02-27T21:47:15.919ZINFOcrucible: [2] client re-new 302 jobs since flush 5218 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
133662024-02-27T21:47:15.919ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
13367 [8830] got start action
13368 [8830] I am going to restart
13369 Starting downstairs at port 8830
13370 Make output file at "/tmp/dsc/downstairs-8830.txt"
13371 [8830][2] reports Starting
133722024-02-27T21:47:16.920ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
133732024-02-27T21:47:16.921ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
133742024-02-27T21:47:16.921ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
133752024-02-27T21:47:16.921ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
133762024-02-27T21:47:16.921ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
133772024-02-27T21:47:16.921ZINFOcrucible: [2] client re-new 302 jobs since flush 5218 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
133782024-02-27T21:47:16.921ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
13379 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1182)
13380 [8830][2] reports Running
133812024-02-27T21:47:17.922ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
133822024-02-27T21:47:17.922ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
133832024-02-27T21:47:17.924ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
133842024-02-27T21:47:17.924ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
133852024-02-27T21:47:17.924ZINFOcrucible: send last flush ID to this DS: 5218 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
133862024-02-27T21:47:17.924ZINFOcrucible: Replied this last flush ID: 5218 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
133872024-02-27T21:47:17.924ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
133882024-02-27T21:47:17.924ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
133892024-02-27T21:47:17.924ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
133902024-02-27T21:47:17.924ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
13391 ----------------------------------------------------------------
13392 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
13393 Downstairs last five completed: 5520 5519 5518 5517 5516
13394 Upstairs last five completed: 4521 4520 4519 4518 4517
13395 CLIENT: Up:0 ds:0 act:3
13396 Replay: All jobs finished, all DS active.
13397 Main task has work to do, go find it
13398 got dsc Stop(2)
13399 stop 2
13400 Feb 27 21:47:19.260 INFO request completed, latency_us: 342, response_code: 204, uri: /stop/cid/2, method[8830] Got stop action so:false kr:false
13401 : POST, req_id: 12643840-a132-4f2a-8521-a4b0847232ce, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
134022024-02-27T21:47:19.261ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
134032024-02-27T21:47:19.261ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
13404 Feb 27 21:47:19.261 INFO request completed, latency_us: {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-27T21:47:19.261584783Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
13405 165, response_code: 200, uri: /state/cid/2, method:{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:47:19.26160473Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"client":"2","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
13406 GET, req_id: 9e4d33b4-8f3f-4069-b50b-52ef288f6ae8, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
134072024-02-27T21:47:19.261ZINFOcrucible: [2] client re-new 0 jobs since flush 5520 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
134082024-02-27T21:47:19.261ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
13409 [8830] Exited with: ExitStatus(unix_wait_status(9))
13410 [8830][2] reports Exit
134112024-02-27T21:47:20.262ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
134122024-02-27T21:47:20.262ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
134132024-02-27T21:47:20.262ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
134142024-02-27T21:47:20.262ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
134152024-02-27T21:47:20.262ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
134162024-02-27T21:47:20.262ZINFOcrucible: [2] client re-new 0 jobs since flush 5520 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
134172024-02-27T21:47:20.262ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
134182024-02-27T21:47:21.264ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
134192024-02-27T21:47:21.264ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
134202024-02-27T21:47:21.264ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
134212024-02-27T21:47:21.264ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
134222024-02-27T21:47:21.264ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
134232024-02-27T21:47:21.264ZINFOcrucible: [2] client re-new 0 jobs since flush 5520 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
134242024-02-27T21:47:21.264ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
134252024-02-27T21:47:22.265ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
134262024-02-27T21:47:22.265ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
134272024-02-27T21:47:22.266ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
134282024-02-27T21:47:22.266ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
134292024-02-27T21:47:22.266ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
134302024-02-27T21:47:22.266ZINFOcrucible: [2] client re-new 0 jobs since flush 5520 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
134312024-02-27T21:47:22.266ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
13432 Feb 27 21:47:23.263 INFO request completed, latency_us: 214, response_code: 200, uri: /state/cid/2, method: GET, req_id: c13700ca-4c44-49ab-a61e-dd0ab96b00be, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
13433 001/300 Read block 2960 len 28672
13434 002/300 Read block 3398 len 8192
13435 003/300 Read block 3373 len 28672
134362024-02-27T21:47:23.267ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
134372024-02-27T21:47:23.267ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
134382024-02-27T21:47:23.267ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
134392024-02-27T21:47:23.267ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
134402024-02-27T21:47:23.267ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
134412024-02-27T21:47:23.267ZINFOcrucible: [2] client re-new 3 jobs since flush 5520 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
134422024-02-27T21:47:23.267ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
13443 004/300 Write block 1616 len 4096 data: 7
13444 005/300 Write block 4504 len 20480 data: 2 3 3 3 3
13445 006/300 Read block 325 len 32768
13446 007/300 Read block 2422 len 40960
13447 008/300 Write block 2626 len 32768 data: 8 6 5 5 4 3 2 2
13448 009/300 Write block 2156 len 20480 data: 4 4 5 5 5
13449 010/300 Read block 2588 len 40960
13450 011/300 Read block 949 len 24576
13451 012/300 Write block 3947 len 32768 data: 6 5 4 4 4 5 7 7
13452 013/300 Write block 2797 len 36864 data: 6 6 4 4 3 4 4 4 4
13453 014/300 Read block 4249 len 24576
13454 015/300 Read block 4865 len 36864
13455 016/300 Write block 2282 len 20480 data: 5 4 4 4 4
13456 017/300 Read block 3027 len 40960
13457 018/300 Write block 3196 len 16384 data: 4 4 3 3
13458 019/300 Read block 2551 len 8192
13459 020/300 Write block 97 len 36864 data: 4 4 3 3 3 3 3 3 2
13460 021/300 Read block 2751 len 8192
13461 022/300 Write block 1019 len 8192 data: 5 4
13462 023/300 Flush
13463 024/300 Write block 4026 len 20480 data: 4 4 5 5 5
13464 025/300 Read block 4694 len 32768
13465 026/300 Write block 4613 len 20480 data: 3 3 4 5 5
13466 027/300 Write block 2679 len 36864 data: 4 4 4 4 3 3 3 3 2
13467 028/300 Write block 2356 len 8192 data: 4 4
13468 029/300 Read block 3839 len 32768
13469 030/300 Read block 1720 len 20480
13470 031/300 Read block 3572 len 40960
13471 032/300 Read block 4226 len 24576
13472 033/300 Write block 1171 len 16384 data: 3 3 3 3
13473 034/300 Read block 3980 len 20480
13474 035/300 Write block 678 len 4096 data: 5
13475 036/300 Read block 3900 len 8192
13476 037/300 Read block 4605 len 36864
13477 038/300 Write block 3816 len 8192 data: 5 4
13478 039/300 Write block 522 len 20480 data: 5 6 4 5 5
13479 040/300 Write block 2878 len 36864 data: 4 5 4 4 4 3 3 3 2
13480 041/300 Write block 3102 len 40960 data: 5 3 3 4 4 4 4 5 6 6
13481 042/300 Read block 547 len 20480
13482 043/300 Flush
13483 044/300 Read block 1112 len 16384
13484 045/300 Write block 2674 len 40960 data: 3 3 3 3 4 5 5 5 5 4
13485 046/300 Flush
13486 047/300 Write block 1752 len 12288 data: 3 4 4
13487 048/300 Write block 3414 len 40960 data: 6 6 5 5 4 4 4 3 3 3
13488 049/300 Flush
13489 050/300 Write block 2423 len 20480 data: 3 4 4 4 4
13490 051/300 Write block 4870 len 12288 data: 5 5 5
13491 052/300 Read block 1197 len 24576
13492 053/300 Read block 3527 len 20480
13493 054/300 Read block 3906 len 16384
13494 055/300 Write block 1197 len 8192 data: 4 4
13495 056/300 Write block 1469 len 4096 data: 4
13496 057/300 Write block 730 len 32768 data: 3 4 4 4 4 5 5 5
13497 058/300 Read block 4522 len 12288
13498 059/300 Write block 4502 len 12288 data: 3 3 3
13499 060/300 Read block 2219 len 36864
13500 061/300 Write block 256 len 16384 data: 3 4 5 6
13501 062/300 Read block 3062 len 20480
13502 063/300 Write block 415 len 36864 data: 5 4 3 5 5 5 5 4 3
13503 064/300 Read block 4154 len 16384
13504 065/300 Flush
13505 066/300 Write block 1084 len 40960 data: 7 6 4 4 4 4 5 6 5 4
13506 067/300 Write block 2493 len 28672 data: 5 5 4 3 2 2 3
13507 068/300 Write block 4664 len 4096 data: 6
13508 069/300 Read block 1892 len 20480
13509 070/300 Write block 2667 len 32768 data: 5 7 8 8 7 6 5 4
13510 071/300 Write block 26 len 12288 data: 5 4 3
13511 072/300 Write block 4067 len 36864 data: 3 2 3 3 3 3 3 3 4
13512 073/300 Write block 2956 len 12288 data: 5 5 6
13513 074/300 Read block 4953 len 4096
13514 075/300 Read block 1222 len 4096
13515 076/300 Write block 383 len 40960 data: 2 2 3 3 3 3 3 2 2 4
13516 077/300 Write block 3482 len 16384 data: 4 3 3 4
13517 078/300 Read block 2837 len 16384
13518 079/300 Read block 2181 len 32768
13519 080/300 Write block 671 len 36864 data: 6 6 4 5 6 7 7 6 3
13520 081/300 Read block 2001 len 4096
13521 082/300 Read block 3522 len 4096
13522 083/300 Write block 2169 len 32768 data: 5 4 3 3 4 3 2 2
13523 084/300 Read block 769 len 36864
13524 085/300 Write block 251 len 28672 data: 3 3 3 2 3 4 5
13525 086/300 Write block 4654 len 16384 data: 5 5 5 4
13526 087/300 Read block 91 len 20480
13527 088/300 Read block 3490 len 24576
13528 089/300 Write block 3132 len 4096 data: 2
13529 090/300 Read block 4928 len 40960
13530 091/300 Write block 867 len 28672 data: 2 3 4 4 4 4 5
13531 092/300 Write block 695 len 36864 data: 3 3 3 3 3 3 2 2 2
13532 093/300 Flush
13533 094/300 Read block 1137 len 32768
13534 095/300 Write block 1967 len 8192 data: 2 4
13535 096/300 Write block 3310 len 40960 data: 5 4 4 4 4 4 5 6 5 4
13536 097/300 Write block 3727 len 36864 data: 3 3 3 3 3 4 4 3 3
13537 098/300 Read block 3320 len 16384
13538 099/300 Read block 1156 len 8192
13539 100/300 Write block 3477 len 28672 data: 3 3 3 2 3 5 4
13540 101/300 Read block 2654 len 20480
13541 102/300 Read block 2366 len 32768
13542 103/300 Read block 2958 len 8192
13543 104/300 Read block 2975 len 36864
13544 105/300 Write block 2845 len 28672 data: 5 5 5 4 3 3 2
13545 106/300 Read block 2833 len 40960
13546 107/300 Write block 3523 len 8192 data: 6 6
13547 108/300 Write block 441 len 8192 data: 4 3
13548 109/300 Read block 3344 len 24576
13549 110/300 Write block 3303 len 4096 data: 7
13550 111/300 Read block 1928 len 16384
13551 112/300 Read block 2376 len 28672
13552 113/300 Write block 157 len 16384 data: 5 6 6 5
13553 114/300 Read block 3066 len 28672
13554 115/300 Read block 64 len 32768
13555 116/300 Write block 1499 len 16384 data: 3 3 5 5
13556 117/300 Read block 3031 len 20480
13557 118/300 Write block 4630 len 20480 data: 6 5 6 5 4
13558 119/300 Write block 4458 len 12288 data: 4 4 4
13559 120/300 Read block 2251 len 28672
13560 121/300 Read block 3175 len 12288
13561 122/300 Read block 2382 len 24576
13562 123/300 Write block 4029 len 12288 data: 6 6 5
13563 124/300 Read block 4095 len 20480
13564 125/300 Flush
13565 126/300 Write block 2466 len 4096 data: 4
13566 127/300 Write block 1702 len 12288 data: 3 4 3
13567 128/300 Write block 3447 len 20480 data: 6 6 5 5 5
13568 129/300 Read block 3097 len 36864
13569 130/300 Write block 2280 len 16384 data: 6 5 6 5
13570 131/300 Write block 871 len 16384 data: 5 5 6 6
13571 132/300 Read block 1577 len 24576
13572 133/300 Write block 4785 len 4096 data: 3
13573 134/300 Write block 3189 len 36864 data: 3 2 3 3 4 4 4 5 5
13574 135/300 Write block 675 len 4096 data: 7
13575 136/300 Read block 3480 len 28672
13576 137/300 Write block 3092 len 12288 data: 2 3 4
13577 138/300 Read block 3902 len 4096
13578 139/300 Write block 4478 len 8192 data: 5 5
13579 140/300 Read block 2268 len 24576
13580 141/300 Write block 3316 len 28672 data: 6 7 6 5 4 3 3
13581 142/300 Read block 148 len 36864
13582 143/300 Write block 2962 len 36864 data: 4 4 5 5 5 6 5 4 5
13583 144/300 Read block 4039 len 24576
13584 145/300 Write block 2352 len 40960 data: 5 4 4 4 5 5 4 3 3 4
13585 146/300 Write block 3937 len 32768 data: 4 4 3 2 2 3 4 4
13586 147/300 Read block 3778 len 16384
13587 148/300 Read block 601 len 16384
13588 149/300 Write block 229 len 36864 data: 4 4 2 2 2 2 3 3 3
13589 150/300 Write block 3098 len 40960 data: 5 7 4 4 6 4 4 5 5 5
13590 151/300 Write block 3592 len 4096 data: 6
13591 152/300 Read block 928 len 12288
13592 153/300 Write block 1270 len 8192 data: 5 3
13593 154/300 Read block 1020 len 4096
13594 155/300 Write block 3481 len 16384 data: 4 6 5 4
13595 156/300 Read block 4255 len 4096
13596 157/300 Read block 4010 len 4096
13597 158/300 Write block 665 len 40960 data: 4 4 4 6 6 7 7 7 5 6
13598 159/300 Read block 2473 len 32768
13599 160/300 Write block 649 len 8192 data: 2 2
13600 161/300 Write block 2742 len 16384 data: 3 2 2 3
13601 162/300 Read block 3327 len 24576
13602 163/300 Write block 1949 len 4096 data: 3
13603 164/300 Write block 2793 len 4096 data: 5
13604 165/300 Read block 2777 len 12288
13605 166/300 Read block 2330 len 36864
13606 167/300 Write block 4056 len 32768 data: 4 4 4 5 4 4 4 3
13607 168/300 Read block 1509 len 36864
13608 169/300 Flush
13609 170/300 Read block 3528 len 12288
13610 171/300 Read block 962 len 24576
13611 172/300 Flush
13612 173/300 Write block 3809 len 8192 data: 4 4
13613 174/300 Read block 3368 len 40960
13614 175/300 Flush
13615 176/300 Write block 4268 len 20480 data: 6 6 5 5 5
13616 177/300 Write block 3505 len 28672 data: 2 2 2 2 2 3 4
13617 178/300 Flush
13618 179/300 Write block 3810 len 32768 data: 5 5 4 5 5 5 6 5
13619 180/300 Read block 4686 len 24576
13620 181/300 Read block 3718 len 36864
13621 182/300 Read block 4238 len 20480
13622 183/300 Flush
13623 184/300 Write block 991 len 16384 data: 3 3 4 3
13624 185/300 Read block 431 len 12288
13625 186/300 Read block 1406 len 8192
13626 187/300 Read block 3257 len 8192
13627 188/300 Write block 586 len 4096 data: 6
13628 189/300 Read block 4259 len 24576
13629 190/300 Write block 493 len 20480 data: 3 4 5 5 6
13630 191/300 Write block 2786 len 40960 data: 3 3 2 3 4 4 5 6 5 5
13631 192/300 Write block 4332 len 40960 data: 3 3 3 3 3 3 3 3 2 2
13632 193/300 Flush
13633 194/300 Read block 4398 len 16384
13634 195/300 Read block 3689 len 24576
13635 196/300 Write block 2199 len 20480 data: 7 7 4 5 3
13636 197/300 Write block 1928 len 24576 data: 6 7 7 6 6 5
13637 198/300 Write block 424 len 20480 data: 4 3 2 2 2
13638 199/300 Write block 3849 len 32768 data: 3 4 4 5 5 4 5 5
13639 200/300 Write block 3740 len 24576 data: 3 2 2 3 3 2
13640 201/300 Write block 537 len 16384 data: 6 5 3 3
13641 202/300 Read block 2682 len 16384
13642 203/300 Write block 4748 len 8192 data: 7 8
13643 204/300 Read block 1417 len 20480
13644 205/300 Read block 1691 len 8192
13645 206/300 Read block 3431 len 16384
13646 207/300 Write block 614 len 16384 data: 5 4 3 3
13647 208/300 Write block 3215 len 32768 data: 4 4 4 4 5 5 5 5
13648 209/300 Write block 2810 len 12288 data: 6 6 6
13649 210/300 Read block 2285 len 36864
13650 211/300 Write block 2331 len 4096 data: 5
13651 212/300 Write block 3958 len 36864 data: 4 4 2 3 3 5 6 7 6
13652 213/300 Write block 1081 len 16384 data: 5 6 7 8
13653 214/300 Write block 3337 len 32768 data: 3 4 2 3 3 3 4 5
13654 215/300 Read block 96 len 12288
13655 216/300 Write block 4690 len 28672 data: 5 5 6 7 6 5 3
13656 217/300 Write block 329 len 12288 data: 4 3 3
13657 218/300 Read block 2277 len 32768
13658 219/300 Write block 4323 len 4096 data: 3
13659 220/300 Read block 4882 len 20480
13660 221/300 Write block 2160 len 4096 data: 6
13661 222/300 Read block 1295 len 40960
13662 223/300 Write block 770 len 28672 data: 3 3 3 3 2 2 2
13663 224/300 Read block 234 len 4096
13664 225/300 Read block 4070 len 28672
13665 226/300 Write block 328 len 40960 data: 4 5 4 4 2 3 3 3 3 3
13666 227/300 Read block 1705 len 28672
13667 228/300 Write block 1914 len 32768 data: 4 3 3 3 2 2 3 3
13668 229/300 Write block 2337 len 12288 data: 8 7 9
13669 230/300 Write block 1002 len 4096 data: 2
13670 231/300 Read block 49 len 28672
13671 232/300 Flush
13672 233/300 Read block 3997 len 28672
13673 234/300 Write block 3001 len 28672 data: 7 7 5 3 3 3 4
13674 235/300 Read block 2123 len 4096
13675 236/300 Read block 917 len 28672
13676 237/300 Write block 87 len 24576 data: 3 3 3 3 4 4
13677 238/300 Read block 3147 len 16384
13678 239/300 Read block 829 len 32768
13679 240/300 Flush
13680 241/300 Read block 3186 len 16384
13681 242/300 Read block 2409 len 12288
13682 243/300 Write block 4958 len 4096 data: 4
13683 244/300 Read block 2762 len 12288
13684 245/300 Flush
13685 246/300 Read block 1744 len 8192
13686 247/300 Write block 3849 len 20480 data: 4 5 5 6 6
13687 248/300 Read block 2811 len 12288
13688 249/300 Read block 2175 len 20480
13689 250/300 Read block 1693 len 32768
13690 251/300 Write block 3333 len 16384 data: 5 5 5 4
13691 252/300 Write block 2732 len 16384 data: 3 3 4 4
13692 253/300 Write block 2445 len 12288 data: 4 5 5
13693 254/300 Write block 26 len 16384 data: 6 5 4 2
13694 255/300 Write block 4365 len 36864 data: 5 5 6 5 4 4 3 3 3
13695 256/300 Write block 4300 len 28672 data: 6 5 5 6 4 3 4
13696 257/300 Write block 3747 len 36864 data: 3 3 3 3 3 3 4 3 3
13697 258/300 Read block 1112 len 28672
13698 259/300 Flush
13699 260/300 Read block 1521 len 40960
13700 261/300 Read block 445 len 16384
13701 262/300 Write block 29 len 20480 data: 3 2 2 2 2
13702 263/300 Read block 1124 len 36864
13703 264/300 Read block 1899 len 40960
13704 265/300 Read block 1227 len 20480
13705 266/300 Write block 2938 len 4096 data: 3
13706 267/300 Write block 1330 len 32768 data: 3 2 2 2 2 3 3 3
13707 268/300 Read block 1839 len 40960
13708 269/300 Read block 4989 len 4096
13709 270/300 Write block 1538 len 12288 data: 2 2 2
13710 271/300 Write block 4397 len 24576 data: 6 6 6 6 5 5
13711 272/300 Read block 2754 len 8192
13712 273/300 Write block 3533 len 32768 data: 4 3 3 3 4 4 4 4
13713 274/300 Read block 4843 len 40960
13714 275/300 Read block 4237 len 20480
13715 276/300 Read block 41 len 40960
13716 277/300 Read block 2457 len 40960
13717 278/300 Read block 620 len 12288
13718 279/300 Write block 895 len 8192 data: 5 4
13719 280/300 Read block 1762 len 20480
13720 281/300 Write block 687 len 36864 data: 5 3 3 2 2 2 3 3 4
13721 282/300 Write block 2652 len 28672 data: 3 3 4 5 5 5 5
13722 283/300 Read block 2737 len 24576
13723 284/300 Write block 4990 len 32768 data: 3 4 4 3 2 2 2 2
13724 285/300 Write block 2779 len 12288 data: 4 6 6
13725 286/300 Read block 3813 len 40960
13726 287/300 Flush
13727 288/300 Read block 3263 len 4096
13728 289/300 Read block 63 len 36864
13729 290/300 Write block 4840 len 8192 data: 3 3
13730 291/300 Read block 107 len 8192
13731 292/300 Read block 4464 len 20480
13732 293/300 Write block 680 len 24576 data: 4 3 3 2 2 3
13733 294/300 Flush
13734 295/300 Read block 2798 len 16384
13735 296/300 Flush
13736 297/300 Read block 576 len 28672
13737 298/300 Write block 4984 len 20480 data: 5 4 3 2 2
13738 299/300 Write block 3556 len 32768 data: 7 6 7 7 7 6 7 8
13739 300/300 Write block 589 len 36864 data: 6 5 6 5 5 7 6 5 5
13740 Main task has work to do, go find it
13741 got dsc Start(2)
13742 start 2
13743 Feb 27 21:47:23.594 INFO request completed, latency_us: 210, response_code: 204, uri: /start/cid/2, method: POST, req_id: 61a88ac1-8588-461e-8550-1bf4fe3b8238, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
13744 Replay: started 2, returned:Ok(())
13745 ----------------------------------------------------------------
13746 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
13747 States: Active Active Offline
13748 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
13749 4522 Acked 5521 Read 7 Done Done New true
13750 4523 Acked 5522 Read 2 Done Done New true
13751 4524 Acked 5523 Read 7 Done Done New true
13752 4525 Acked 5524 Write 1 Done Done New false
13753 4526 Acked 5525 Write 5 Done Done New false
13754 4527 Acked 5526 Read 8 Done Done New false
13755 4528 Acked 5527 Read 10 Done Done New false
13756 4529 Acked 5528 Write 8 Done Done New false
13757 4530 Acked 5529 Write 5 Done Done New false
13758 4531 Acked 5530 Read 10 Done Done New false
13759 4532 Acked 5531 Read 6 Done Done New false
13760 4533 Acked 5532 Write 8 Done Done New false
13761 4534 Acked 5533 Write 9 Done Done New false
13762 4535 Acked 5534 Read 6 Done Done New false
13763 4536 Acked 5535 Read 9 Done Done New false
13764 4537 Acked 5536 Write 5 Done Done New false
13765 4538 Acked 5537 Read 10 Done Done New false
13766 4539 Acked 5538 Write 4 Done Done New false
13767 4540 Acked 5539 Read 2 Done Done New false
13768 4541 Acked 5540 Write 9 Done Done New false
13769 4542 Acked 5541 Read 2 Done Done New false
13770 4543 Acked 5542 Write 2 Done Done New false
13771 4544 Acked 5543 Flush 0 Done Done New false
13772 4545 Acked 5544 Write 5 Done Done New false
13773 4546 Acked 5545 Read 8 Done Done New false
13774 4547 Acked 5546 Write 5 Done Done New false
13775 4548 Acked 5547 Write 9 Done Done New false
13776 4549 Acked 5548 Write 2 Done Done New false
13777 4550 Acked 5549 Read 8 Done Done New false
13778 4551 Acked 5550 Read 5 Done Done New false
13779 4552 Acked 5551 Read 10 Done Done New false
13780 4553 Acked 5552 Read 6 Done Done New false
13781 4554 Acked 5553 Write 4 Done Done New false
13782 4555 Acked 5554 Read 5 Done Done New false
13783 4556 Acked 5555 Write 1 Done Done New false
13784 4557 Acked 5556 Read 2 Done Done New false
13785 4558 Acked 5557 Read 9 Done Done New false
13786 4559 Acked 5558 Write 2 Done Done New false
13787 4560 Acked 5559 Write 5 Done Done New false
13788 4561 Acked 5560 Write 9 Done Done New false
13789 4562 Acked 5561 Write 10 Done Done New false
13790 4563 Acked 5562 Read 5 Done Done New false
13791 4564 Acked 5563 Flush 0 Done Done New false
13792 4565 Acked 5564 Read 4 Done Done New false
13793 4566 Acked 5565 Write 10 Done Done New false
13794 4567 Acked 5566 Flush 0 Done Done New false
13795 4568 Acked 5567 Write 3 Done Done New false
13796 4569 Acked 5568 Write 10 Done Done New false
13797 4570 Acked 5569 Flush 0 Done Done New false
13798 4571 Acked 5570 Write 5 Done Done New false
13799 4572 Acked 5571 Write 3 Done Done New false
13800 4573 Acked 5572 Read 6 Done Done New false
13801 4574 Acked 5573 Read 5 Done Done New false
13802 4575 Acked 5574 Read 4 Done Done New false
13803 4576 Acked 5575 Write 2 Done Done New false
13804 4577 Acked 5576 Write 1 Done Done New false
13805 4578 Acked 5577 Write 8 Done Done New false
13806 4579 Acked 5578 Read 3 Done Done New false
13807 4580 Acked 5579 Write 3 Done Done New false
13808 4581 Acked 5580 Read 9 Done Done New false
13809 4582 Acked 5581 Write 4 Done Done New false
13810 4583 Acked 5582 Read 5 Done Done New false
13811 4584 Acked 5583 Write 9 Done Done New false
13812 4585 Acked 5584 Read 4 Done Done New false
13813 4586 Acked 5585 Flush 0 Done Done New false
13814 4587 Acked 5586 Write 10 Done Done New false
13815 4588 Acked 5587 Write 7 Done Done New false
13816 4589 Acked 5588 Write 1 Done Done New false
13817 4590 Acked 5589 Read 5 Done Done New false
13818 4591 Acked 5590 Write 8 Done Done New false
13819 4592 Acked 5591 Write 3 Done Done New false
13820 4593 Acked 5592 Write 9 Done Done New false
13821 4594 Acked 5593 Write 3 Done Done New false
13822 4595 Acked 5594 Read 1 Done Done New false
13823 4596 Acked 5595 Read 1 Done Done New false
13824 4597 Acked 5596 Write 10 Done Done New false
13825 4598 Acked 5597 Write 4 Done Done New false
13826 4599 Acked 5598 Read 4 Done Done New false
13827 4600 Acked 5599 Read 8 Done Done New false
13828 4601 Acked 5600 Write 9 Done Done New false
13829 4602 Acked 5601 Read 1 Done Done New false
13830 4603 Acked 5602 Read 1 Done Done New false
13831 4604 Acked 5603 Write 8 Done Done New false
13832 4605 Acked 5604 Read 9 Done Done New false
13833 4606 Acked 5605 Write 7 Done Done New false
13834 4607 Acked 5606 Write 4 Done Done New false
13835 4608 Acked 5607 Read 5 Done Done New false
13836 4609 Acked 5608 Read 6 Done Done New false
13837 4610 Acked 5609 Write 1 Done Done New false
13838 4611 Acked 5610 Read 10 Done Done New false
13839 4612 Acked 5611 Write 7 Done Done New false
13840 4613 Acked 5612 Write 9 Done Done New false
13841 4614 Acked 5613 Flush 0 Done Done New false
13842 4615 Acked 5614 Read 8 Done Done New false
13843 4616 Acked 5615 Write 2 Done Done New false
13844 4617 Acked 5616 Write 10 Done Done New false
13845 4618 Acked 5617 Write 9 Done Done New false
13846 4619 Acked 5618 Read 4 Done Done New false
13847 4620 Acked 5619 Read 2 Done Done New false
13848 4621 Acked 5620 Write 7 Done Done New false
13849 4622 Acked 5621 Read 5 Done Done New false
13850 4623 Acked 5622 Read 8 Done Done New false
13851 4624 Acked 5623 Read 2 Done Done New false
13852 4625 Acked 5624 Read 9 Done Done New false
13853 4626 Acked 5625 Write 7 Done Done New false
13854 4627 Acked 5626 Read 10 Done Done New false
13855 4628 Acked 5627 Write 2 Done Done New false
13856 4629 Acked 5628 Write 2 Done Done New false
13857 4630 Acked 5629 Read 6 Done Done New false
13858 4631 Acked 5630 Write 1 Done Done New false
13859 4632 Acked 5631 Read 4 Done Done New false
13860 4633 Acked 5632 Read 7 Done Done New false
13861 4634 Acked 5633 Write 4 Done Done New false
13862 4635 Acked 5634 Read 7 Done Done New false
13863 4636 Acked 5635 Read 8 Done Done New false
13864 4637 Acked 5636 Write 4 Done Done New false
13865 4638 Acked 5637 Read 5 Done Done New false
13866 4639 Acked 5638 Write 5 Done Done New false
13867 4640 Acked 5639 Write 3 Done Done New false
13868 4641 Acked 5640 Read 7 Done Done New false
13869 4642 Acked 5641 Read 3 Done Done New false
13870 4643 Acked 5642 Read 6 Done Done New false
13871 4644 Acked 5643 Write 3 Done Done New false
13872 4645 Acked 5644 Read 5 Done Done New false
13873 4646 Acked 5645 Flush 0 Done Done New false
13874 4647 Acked 5646 Write 1 Done Done New false
13875 4648 Acked 5647 Write 3 Done Done New false
13876 4649 Acked 5648 Write 5 Done Done New false
13877 4650 Acked 5649 Read 9 Done Done New false
13878 4651 Acked 5650 Write 4 Done Done New false
13879 4652 Acked 5651 Write 4 Done Done New false
13880 4653 Acked 5652 Read 6 Done Done New false
13881 4654 Acked 5653 Write 1 Done Done New false
13882 4655 Acked 5654 Write 9 Done Done New false
13883 4656 Acked 5655 Write 1 Done Done New false
13884 4657 Acked 5656 Read 7 Done Done New false
13885 4658 Acked 5657 Write 3 Done Done New false
13886 4659 Acked 5658 Read 1 Done Done New false
13887 4660 Acked 5659 Write 2 Done Done New false
13888 4661 Acked 5660 Read 6 Done Done New false
13889 4662 Acked 5661 Write 7 Done Done New false
13890 4663 Acked 5662 Read 9 Done Done New false
13891 4664 Acked 5663 Write 9 Done Done New false
13892 4665 Acked 5664 Read 6 Done Done New false
13893 4666 Acked 5665 Write 10 Done Done New false
13894 4667 Acked 5666 Write 8 Done Done New false
13895 4668 Acked 5667 Read 4 Done Done New false
13896 4669 Acked 5668 Read 4 Done Done New false
13897 4670 Acked 5669 Write 9 Done Done New false
13898 4671 Acked 5670 Write 10 Done Done New false
13899 4672 Acked 5671 Write 1 Done Done New false
13900 4673 Acked 5672 Read 3 Done Done New false
13901 4674 Acked 5673 Write 2 Done Done New false
13902 4675 Acked 5674 Read 1 Done Done New false
13903 4676 Acked 5675 Write 4 Done Done New false
13904 4677 Acked 5676 Read 1 Done Done New false
13905 4678 Acked 5677 Read 1 Done Done New false
13906 4679 Acked 5678 Write 10 Done Done New false
13907 4680 Acked 5679 Read 8 Done Done New false
13908 4681 Acked 5680 Write 2 Done Done New false
13909 4682 Acked 5681 Write 4 Done Done New false
13910 4683 Acked 5682 Read 6 Done Done New false
13911 4684 Acked 5683 Write 1 Done Done New false
13912 4685 Acked 5684 Write 1 Done Done New false
13913 4686 Acked 5685 Read 3 Done Done New false
13914 4687 Acked 5686 Read 9 Done Done New false
13915 4688 Acked 5687 Write 8 Done Done New false
13916 4689 Acked 5688 Read 9 Done Done New false
13917 4690 Acked 5689 Flush 0 Done Done New false
13918 4691 Acked 5690 Read 3 Done Done New false
13919 4692 Acked 5691 Read 6 Done Done New false
13920 4693 Acked 5692 Flush 0 Done Done New false
13921 4694 Acked 5693 Write 2 Done Done New false
13922 4695 Acked 5694 Read 10 Done Done New false
13923 4696 Acked 5695 Flush 0 Done Done New false
13924 4697 Acked 5696 Write 5 Done Done New false
13925 4698 Acked 5697 Write 7 Done Done New false
13926 4699 Acked 5698 Flush 0 Done Done New false
13927 4700 Acked 5699 Write 8 Done Done New false
13928 4701 Acked 5700 Read 6 Done Done New false
13929 4702 Acked 5701 Read 9 Done Done New false
13930 4703 Acked 5702 Read 5 Done Done New false
13931 4704 Acked 5703 Flush 0 Done Done New false
13932 4705 Acked 5704 Write 4 Done Done New false
13933 4706 Acked 5705 Read 3 Done Done New false
13934 4707 Acked 5706 Read 2 Done Done New false
13935 4708 Acked 5707 Read 2 Done Done New false
13936 4709 Acked 5708 Write 1 Done Done New false
13937 4710 Acked 5709 Read 6 Done Done New false
13938 4711 Acked 5710 Write 5 Done Done New false
13939 4712 Acked 5711 Write 10 Done Done New false
13940 4713 Acked 5712 Write 10 Done Done New false
13941 4714 Acked 5713 Flush 0 Done Done New false
13942 4715 Acked 5714 Read 4 Done Done New false
13943 4716 Acked 5715 Read 6 Done Done New false
13944 4717 Acked 5716 Write 5 Done Done New false
13945 4718 Acked 5717 Write 6 Done Done New false
13946 4719 Acked 5718 Write 5 Done Done New false
13947 4720 Acked 5719 Write 8 Done Done New false
13948 4721 Acked 5720 Write 6 Done Done New false
13949 4722 Acked 5721 Write 4 Done Done New false
13950 4723 Acked 5722 Read 4 Done Done New false
13951 4724 Acked 5723 Write 2 Done Done New false
13952 4725 Acked 5724 Read 5 Done Done New false
13953 4726 Acked 5725 Read 2 Done Done New false
13954 4727 Acked 5726 Read 4 Done Done New false
13955 4728 Acked 5727 Write 4 Done Done New false
13956 4729 Acked 5728 Write 8 Done Done New false
13957 4730 Acked 5729 Write 3 Done Done New false
13958 4731 Acked 5730 Read 9 Done Done New false
13959 4732 Acked 5731 Write 1 Done Done New false
13960 4733 Acked 5732 Write 9 Done Done New false
13961 4734 Acked 5733 Write 4 Done Done New false
13962 4735 Acked 5734 Write 8 Done Done New false
13963 4736 Acked 5735 Read 3 Done Done New false
13964 4737 Acked 5736 Write 7 Done Done New false
13965 4738 Acked 5737 Write 3 Done Done New false
13966 4739 Acked 5738 Read 8 Done Done New false
13967 4740 Acked 5739 Write 1 Done Done New false
13968 4741 Acked 5740 Read 5 Done Done New false
13969 4742 Acked 5741 Write 1 Done Done New false
13970 4743 Acked 5742 Read 10 Done Done New false
13971 4744 Acked 5743 Write 7 Done Done New false
13972 4745 Acked 5744 Read 1 Done Done New false
13973 4746 Acked 5745 Read 7 Done Done New false
13974 4747 Acked 5746 Write 10 Done Done New false
13975 4748 Acked 5747 Read 7 Done Done New false
13976 4749 Acked 5748 Write 8 Done Done New false
13977 4750 Acked 5749 Write 3 Done Done New false
13978 4751 Acked 5750 Write 1 Done Done New false
13979 4752 Acked 5751 Read 7 Done Done New false
13980 4753 Acked 5752 Flush 0 Done Done New false
13981 4754 Acked 5753 Read 7 Done Done New false
13982 4755 Acked 5754 Write 7 Done Done New false
13983 4756 Acked 5755 Read 1 Done Done New false
13984 4757 Acked 5756 Read 7 Done Done New false
13985 4758 Acked 5757 Write 6 Done Done New false
13986 4759 Acked 5758 Read 4 Done Done New false
13987 4760 Acked 5759 Read 8 Done Done New false
13988 4761 Acked 5760 Flush 0 Done Done New false
13989 4762 Acked 5761 Read 4 Done Done New false
13990 4763 Acked 5762 Read 3 Done Done New false
13991 4764 Acked 5763 Write 1 Done Done New false
13992 4765 Acked 5764 Read 3 Done Done New false
13993 4766 Acked 5765 Flush 0 Done Done New false
13994 4767 Acked 5766 Read 2 Done Done New false
13995 4768 Acked 5767 Write 5 Done Done New false
13996 4769 Acked 5768 Read 3 Done Done New false
13997 4770 Acked 5769 Read 5 Done Done New false
13998 4771 Acked 5770 Read 8 Done Done New false
13999 4772 Acked 5771 Write 4 Done Done New false
14000 4773 Acked 5772 Write 4 Done Done New false
14001 4774 Acked 5773 Write 3 Done Done New false
14002 4775 Acked 5774 Write 4 Done Done New false
14003 4776 Acked 5775 Write 9 Done Done New false
14004 4777 Acked 5776 Write 7 Done Done New false
14005 4778 Acked 5777 Write 9 Done Done New false
14006 4779 Acked 5778 Read 7 Done Done New false
14007 4780 Acked 5779 Flush 0 Done Done New false
14008 4781 Acked 5780 Read 10 Done Done New false
14009 4782 Acked 5781 Read 4 Done Done New false
14010 4783 Acked 5782 Write 5 Done Done New false
14011 4784 Acked 5783 Read 9 Done Done New false
14012 4785 Acked 5784 Read 10 Done Done New false
14013 4786 Acked 5785 Read 5 Done Done New false
14014 4787 Acked 5786 Write 1 Done Done New false
14015 4788 Acked 5787 Write 8 Done Done New false
14016 4789 Acked 5788 Read 10 Done Done New false
14017 4790 Acked 5789 Read 1 Done Done New false
14018 4791 Acked 5790 Write 3 Done Done New false
14019 4792 Acked 5791 Write 6 Done Done New false
14020 4793 Acked 5792 Read 2 Done Done New false
14021 4794 Acked 5793 Write 8 Done Done New false
14022 4795 Acked 5794 Read 10 Done Done New false
14023 4796 Acked 5795 Read 5 Done Done New false
14024 4797 Acked 5796 Read 10 Done Done New false
14025 4798 Acked 5797 Read 10 Done Done New false
14026 4799 Acked 5798 Read 3 Done Done New false
14027 4800 Acked 5799 Write 2 Done Done New false
14028 4801 Acked 5800 Read 5 Done Done New false
14029 4802 Acked 5801 Write 9 Done Done New false
14030 4803 Acked 5802 Write 7 Done Done New false
14031 4804 Acked 5803 Read 6 Done Done New false
14032 4805 Acked 5804 Write 8 Done Done New false
14033 4806 Acked 5805 Write 3 Done Done New false
14034 4807 Acked 5806 Read 10 Done Done New false
14035 4808 Acked 5807 Flush 0 Done Done New false
14036 4809 Acked 5808 Read 1 Done Done New false
14037 4810 Acked 5809 Read 9 Done Done New false
14038 4811 Acked 5810 Write 2 Done Done New false
14039 4812 Acked 5811 Read 2 Done Done New false
14040 4813 Acked 5812 Read 5 Done Done New false
14041 4814 Acked 5813 Write 6 Done Done New false
14042 4815 Acked 5814 Flush 0 Done Done New false
14043 4816 Acked 5815 Read 4 Done Done New false
14044 4817 Acked 5816 Flush 0 Done Done New false
14045 4818 Acked 5817 Read 7 Done Done New false
14046 4819 Acked 5818 Write 5 Done Done New false
14047 4820 Acked 5819 Write 8 Done Done New false
14048 4821 Acked 5820 Write 9 Done Done New false
14049 STATES DS:0 DS:1 DS:2 TOTAL
14050 New 0 0 300 300
14051 Sent 0 0 0 0
14052 Done 300 300 0 600
14053 Skipped 0 0 0 0
14054 Error 0 0 0 0
14055 Last Flush: 5816 5816 5520
14056 Downstairs last five completed: 5520 5519 5518 5517 5516
14057 Upstairs last five completed: 4821 4820 4819 4818 4817
14058 CLIENT: Up:0 ds:300 act:2
140592024-02-27T21:47:24.268ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
140602024-02-27T21:47:24.269ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
140612024-02-27T21:47:24.269ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
140622024-02-27T21:47:24.269ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
140632024-02-27T21:47:24.269ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
140642024-02-27T21:47:24.269ZINFOcrucible: [2] client re-new 301 jobs since flush 5520 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
140652024-02-27T21:47:24.269ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
140662024-02-27T21:47:25.270ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
140672024-02-27T21:47:25.270ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
140682024-02-27T21:47:25.270ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
140692024-02-27T21:47:25.270ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
140702024-02-27T21:47:25.270ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
140712024-02-27T21:47:25.271ZINFOcrucible: [2] client re-new 301 jobs since flush 5520 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
140722024-02-27T21:47:25.271ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
140732024-02-27T21:47:26.272ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
140742024-02-27T21:47:26.273ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
140752024-02-27T21:47:26.273ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
140762024-02-27T21:47:26.273ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
140772024-02-27T21:47:26.273ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
140782024-02-27T21:47:26.273ZINFOcrucible: [2] client re-new 301 jobs since flush 5520 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
140792024-02-27T21:47:26.273ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
140802024-02-27T21:47:27.274ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
140812024-02-27T21:47:27.274ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
140822024-02-27T21:47:27.275ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
140832024-02-27T21:47:27.275ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
140842024-02-27T21:47:27.275ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
140852024-02-27T21:47:27.275ZINFOcrucible: [2] client re-new 301 jobs since flush 5520 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
140862024-02-27T21:47:27.275ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
14087 ----------------------------------------------------------------
14088 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14089 States: Active Active Offline
14090 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
14091 4522 Acked 5521 Read 7 Done Done New true
14092 4523 Acked 5522 Read 2 Done Done New true
14093 4524 Acked 5523 Read 7 Done Done New true
14094 4525 Acked 5524 Write 1 Done Done New true
14095 4526 Acked 5525 Write 5 Done Done New true
14096 4527 Acked 5526 Read 8 Done Done New true
14097 4528 Acked 5527 Read 10 Done Done New true
14098 4529 Acked 5528 Write 8 Done Done New true
14099 4530 Acked 5529 Write 5 Done Done New true
14100 4531 Acked 5530 Read 10 Done Done New true
14101 4532 Acked 5531 Read 6 Done Done New true
14102 4533 Acked 5532 Write 8 Done Done New true
14103 4534 Acked 5533 Write 9 Done Done New true
14104 4535 Acked 5534 Read 6 Done Done New true
14105 4536 Acked 5535 Read 9 Done Done New true
14106 4537 Acked 5536 Write 5 Done Done New true
14107 4538 Acked 5537 Read 10 Done Done New true
14108 4539 Acked 5538 Write 4 Done Done New true
14109 4540 Acked 5539 Read 2 Done Done New true
14110 4541 Acked 5540 Write 9 Done Done New true
14111 4542 Acked 5541 Read 2 Done Done New true
14112 4543 Acked 5542 Write 2 Done Done New true
14113 4544 Acked 5543 Flush 0 Done Done New true
14114 4545 Acked 5544 Write 5 Done Done New true
14115 4546 Acked 5545 Read 8 Done Done New true
14116 4547 Acked 5546 Write 5 Done Done New true
14117 4548 Acked 5547 Write 9 Done Done New true
14118 4549 Acked 5548 Write 2 Done Done New true
14119 4550 Acked 5549 Read 8 Done Done New true
14120 4551 Acked 5550 Read 5 Done Done New true
14121 4552 Acked 5551 Read 10 Done Done New true
14122 4553 Acked 5552 Read 6 Done Done New true
14123 4554 Acked 5553 Write 4 Done Done New true
14124 4555 Acked 5554 Read 5 Done Done New true
14125 4556 Acked 5555 Write 1 Done Done New true
14126 4557 Acked 5556 Read 2 Done Done New true
14127 4558 Acked 5557 Read 9 Done Done New true
14128 4559 Acked 5558 Write 2 Done Done New true
14129 4560 Acked 5559 Write 5 Done Done New true
14130 4561 Acked 5560 Write 9 Done Done New true
14131 4562 Acked 5561 Write 10 Done Done New true
14132 4563 Acked 5562 Read 5 Done Done New true
14133 4564 Acked 5563 Flush 0 Done Done New true
14134 4565 Acked 5564 Read 4 Done Done New true
14135 4566 Acked 5565 Write 10 Done Done New true
14136 4567 Acked 5566 Flush 0 Done Done New true
14137 4568 Acked 5567 Write 3 Done Done New true
14138 4569 Acked 5568 Write 10 Done Done New true
14139 4570 Acked 5569 Flush 0 Done Done New true
14140 4571 Acked 5570 Write 5 Done Done New true
14141 4572 Acked 5571 Write 3 Done Done New true
14142 4573 Acked 5572 Read 6 Done Done New true
14143 4574 Acked 5573 Read 5 Done Done New true
14144 4575 Acked 5574 Read 4 Done Done New true
14145 4576 Acked 5575 Write 2 Done Done New true
14146 4577 Acked 5576 Write 1 Done Done New true
14147 4578 Acked 5577 Write 8 Done Done New true
14148 4579 Acked 5578 Read 3 Done Done New true
14149 4580 Acked 5579 Write 3 Done Done New true
14150 4581 Acked 5580 Read 9 Done Done New true
14151 4582 Acked 5581 Write 4 Done Done New true
14152 4583 Acked 5582 Read 5 Done Done New true
14153 4584 Acked 5583 Write 9 Done Done New true
14154 4585 Acked 5584 Read 4 Done Done New true
14155 4586 Acked 5585 Flush 0 Done Done New true
14156 4587 Acked 5586 Write 10 Done Done New true
14157 4588 Acked 5587 Write 7 Done Done New true
14158 4589 Acked 5588 Write 1 Done Done New true
14159 4590 Acked 5589 Read 5 Done Done New true
14160 4591 Acked 5590 Write 8 Done Done New true
14161 4592 Acked 5591 Write 3 Done Done New true
14162 4593 Acked 5592 Write 9 Done Done New true
14163 4594 Acked 5593 Write 3 Done Done New true
14164 4595 Acked 5594 Read 1 Done Done New true
14165 4596 Acked 5595 Read 1 Done Done New true
14166 4597 Acked 5596 Write 10 Done Done New true
14167 4598 Acked 5597 Write 4 Done Done New true
14168 4599 Acked 5598 Read 4 Done Done New true
14169 4600 Acked 5599 Read 8 Done Done New true
14170 4601 Acked 5600 Write 9 Done Done New true
14171 4602 Acked 5601 Read 1 Done Done New true
14172 4603 Acked 5602 Read 1 Done Done New true
14173 4604 Acked 5603 Write 8 Done Done New true
14174 4605 Acked 5604 Read 9 Done Done New true
14175 4606 Acked 5605 Write 7 Done Done New true
14176 4607 Acked 5606 Write 4 Done Done New true
14177 4608 Acked 5607 Read 5 Done Done New true
14178 4609 Acked 5608 Read 6 Done Done New true
14179 4610 Acked 5609 Write 1 Done Done New true
14180 4611 Acked 5610 Read 10 Done Done New true
14181 4612 Acked 5611 Write 7 Done Done New true
14182 4613 Acked 5612 Write 9 Done Done New true
14183 4614 Acked 5613 Flush 0 Done Done New true
14184 4615 Acked 5614 Read 8 Done Done New true
14185 4616 Acked 5615 Write 2 Done Done New true
14186 4617 Acked 5616 Write 10 Done Done New true
14187 4618 Acked 5617 Write 9 Done Done New true
14188 4619 Acked 5618 Read 4 Done Done New true
14189 4620 Acked 5619 Read 2 Done Done New true
14190 4621 Acked 5620 Write 7 Done Done New true
14191 4622 Acked 5621 Read 5 Done Done New true
14192 4623 Acked 5622 Read 8 Done Done New true
14193 4624 Acked 5623 Read 2 Done Done New true
14194 4625 Acked 5624 Read 9 Done Done New true
14195 4626 Acked 5625 Write 7 Done Done New true
14196 4627 Acked 5626 Read 10 Done Done New true
14197 4628 Acked 5627 Write 2 Done Done New true
14198 4629 Acked 5628 Write 2 Done Done New true
14199 4630 Acked 5629 Read 6 Done Done New true
14200 4631 Acked 5630 Write 1 Done Done New true
14201 4632 Acked 5631 Read 4 Done Done New true
14202 4633 Acked 5632 Read 7 Done Done New true
14203 4634 Acked 5633 Write 4 Done Done New true
14204 4635 Acked 5634 Read 7 Done Done New true
14205 4636 Acked 5635 Read 8 Done Done New true
14206 4637 Acked 5636 Write 4 Done Done New true
14207 4638 Acked 5637 Read 5 Done Done New true
14208 4639 Acked 5638 Write 5 Done Done New true
14209 4640 Acked 5639 Write 3 Done Done New true
14210 4641 Acked 5640 Read 7 Done Done New true
14211 4642 Acked 5641 Read 3 Done Done New true
14212 4643 Acked 5642 Read 6 Done Done New true
14213 4644 Acked 5643 Write 3 Done Done New true
14214 4645 Acked 5644 Read 5 Done Done New true
14215 4646 Acked 5645 Flush 0 Done Done New true
14216 4647 Acked 5646 Write 1 Done Done New true
14217 4648 Acked 5647 Write 3 Done Done New true
14218 4649 Acked 5648 Write 5 Done Done New true
14219 4650 Acked 5649 Read 9 Done Done New true
14220 4651 Acked 5650 Write 4 Done Done New true
14221 4652 Acked 5651 Write 4 Done Done New true
14222 4653 Acked 5652 Read 6 Done Done New true
14223 4654 Acked 5653 Write 1 Done Done New true
14224 4655 Acked 5654 Write 9 Done Done New true
14225 4656 Acked 5655 Write 1 Done Done New true
14226 4657 Acked 5656 Read 7 Done Done New true
14227 4658 Acked 5657 Write 3 Done Done New true
14228 4659 Acked 5658 Read 1 Done Done New true
14229 4660 Acked 5659 Write 2 Done Done New true
14230 4661 Acked 5660 Read 6 Done Done New true
14231 4662 Acked 5661 Write 7 Done Done New true
14232 4663 Acked 5662 Read 9 Done Done New true
14233 4664 Acked 5663 Write 9 Done Done New true
14234 4665 Acked 5664 Read 6 Done Done New true
14235 4666 Acked 5665 Write 10 Done Done New true
14236 4667 Acked 5666 Write 8 Done Done New true
14237 4668 Acked 5667 Read 4 Done Done New true
14238 4669 Acked 5668 Read 4 Done Done New true
14239 4670 Acked 5669 Write 9 Done Done New true
14240 4671 Acked 5670 Write 10 Done Done New true
14241 4672 Acked 5671 Write 1 Done Done New true
14242 4673 Acked 5672 Read 3 Done Done New true
14243 4674 Acked 5673 Write 2 Done Done New true
14244 4675 Acked 5674 Read 1 Done Done New true
14245 4676 Acked 5675 Write 4 Done Done New true
14246 4677 Acked 5676 Read 1 Done Done New true
14247 4678 Acked 5677 Read 1 Done Done New true
14248 4679 Acked 5678 Write 10 Done Done New true
14249 4680 Acked 5679 Read 8 Done Done New true
14250 4681 Acked 5680 Write 2 Done Done New true
14251 4682 Acked 5681 Write 4 Done Done New true
14252 4683 Acked 5682 Read 6 Done Done New true
14253 4684 Acked 5683 Write 1 Done Done New true
14254 4685 Acked 5684 Write 1 Done Done New true
14255 4686 Acked 5685 Read 3 Done Done New true
14256 4687 Acked 5686 Read 9 Done Done New true
14257 4688 Acked 5687 Write 8 Done Done New true
14258 4689 Acked 5688 Read 9 Done Done New true
14259 4690 Acked 5689 Flush 0 Done Done New true
14260 4691 Acked 5690 Read 3 Done Done New true
14261 4692 Acked 5691 Read 6 Done Done New true
14262 4693 Acked 5692 Flush 0 Done Done New true
14263 4694 Acked 5693 Write 2 Done Done New true
14264 4695 Acked 5694 Read 10 Done Done New true
14265 4696 Acked 5695 Flush 0 Done Done New true
14266 4697 Acked 5696 Write 5 Done Done New true
14267 4698 Acked 5697 Write 7 Done Done New true
14268 4699 Acked 5698 Flush 0 Done Done New true
14269 4700 Acked 5699 Write 8 Done Done New true
14270 4701 Acked 5700 Read 6 Done Done New true
14271 4702 Acked 5701 Read 9 Done Done New true
14272 4703 Acked 5702 Read 5 Done Done New true
14273 4704 Acked 5703 Flush 0 Done Done New true
14274 4705 Acked 5704 Write 4 Done Done New true
14275 4706 Acked 5705 Read 3 Done Done New true
14276 4707 Acked 5706 Read 2 Done Done New true
14277 4708 Acked 5707 Read 2 Done Done New true
14278 4709 Acked 5708 Write 1 Done Done New true
14279 4710 Acked 5709 Read 6 Done Done New true
14280 4711 Acked 5710 Write 5 Done Done New true
14281 4712 Acked 5711 Write 10 Done Done New true
14282 4713 Acked 5712 Write 10 Done Done New true
14283 4714 Acked 5713 Flush 0 Done Done New true
14284 4715 Acked 5714 Read 4 Done Done New true
14285 4716 Acked 5715 Read 6 Done Done New true
14286 4717 Acked 5716 Write 5 Done Done New true
14287 4718 Acked 5717 Write 6 Done Done New true
14288 4719 Acked 5718 Write 5 Done Done New true
14289 4720 Acked 5719 Write 8 Done Done New true
14290 4721 Acked 5720 Write 6 Done Done New true
14291 4722 Acked 5721 Write 4 Done Done New true
14292 4723 Acked 5722 Read 4 Done Done New true
14293 4724 Acked 5723 Write 2 Done Done New true
14294 4725 Acked 5724 Read 5 Done Done New true
14295 4726 Acked 5725 Read 2 Done Done New true
14296 4727 Acked 5726 Read 4 Done Done New true
14297 4728 Acked 5727 Write 4 Done Done New true
14298 4729 Acked 5728 Write 8 Done Done New true
14299 4730 Acked 5729 Write 3 Done Done New true
14300 4731 Acked 5730 Read 9 Done Done New true
14301 4732 Acked 5731 Write 1 Done Done New true
14302 4733 Acked 5732 Write 9 Done Done New true
14303 4734 Acked 5733 Write 4 Done Done New true
14304 4735 Acked 5734 Write 8 Done Done New true
14305 4736 Acked 5735 Read 3 Done Done New true
14306 4737 Acked 5736 Write 7 Done Done New true
14307 4738 Acked 5737 Write 3 Done Done New true
14308 4739 Acked 5738 Read 8 Done Done New true
14309 4740 Acked 5739 Write 1 Done Done New true
14310 4741 Acked 5740 Read 5 Done Done New true
14311 4742 Acked 5741 Write 1 Done Done New true
14312 4743 Acked 5742 Read 10 Done Done New true
14313 4744 Acked 5743 Write 7 Done Done New true
14314 4745 Acked 5744 Read 1 Done Done New true
14315 4746 Acked 5745 Read 7 Done Done New true
14316 4747 Acked 5746 Write 10 Done Done New true
14317 4748 Acked 5747 Read 7 Done Done New true
14318 4749 Acked 5748 Write 8 Done Done New true
14319 4750 Acked 5749 Write 3 Done Done New true
14320 4751 Acked 5750 Write 1 Done Done New true
14321 4752 Acked 5751 Read 7 Done Done New true
14322 4753 Acked 5752 Flush 0 Done Done New true
14323 4754 Acked 5753 Read 7 Done Done New true
14324 4755 Acked 5754 Write 7 Done Done New true
14325 4756 Acked 5755 Read 1 Done Done New true
14326 4757 Acked 5756 Read 7 Done Done New true
14327 4758 Acked 5757 Write 6 Done Done New true
14328 4759 Acked 5758 Read 4 Done Done New true
14329 4760 Acked 5759 Read 8 Done Done New true
14330 4761 Acked 5760 Flush 0 Done Done New true
14331 4762 Acked 5761 Read 4 Done Done New true
14332 4763 Acked 5762 Read 3 Done Done New true
14333 4764 Acked 5763 Write 1 Done Done New true
14334 4765 Acked 5764 Read 3 Done Done New true
14335 4766 Acked 5765 Flush 0 Done Done New true
14336 4767 Acked 5766 Read 2 Done Done New true
14337 4768 Acked 5767 Write 5 Done Done New true
14338 4769 Acked 5768 Read 3 Done Done New true
14339 4770 Acked 5769 Read 5 Done Done New true
14340 4771 Acked 5770 Read 8 Done Done New true
14341 4772 Acked 5771 Write 4 Done Done New true
14342 4773 Acked 5772 Write 4 Done Done New true
14343 4774 Acked 5773 Write 3 Done Done New true
14344 4775 Acked 5774 Write 4 Done Done New true
14345 4776 Acked 5775 Write 9 Done Done New true
14346 4777 Acked 5776 Write 7 Done Done New true
14347 4778 Acked 5777 Write 9 Done Done New true
14348 4779 Acked 5778 Read 7 Done Done New true
14349 4780 Acked 5779 Flush 0 Done Done New true
14350 4781 Acked 5780 Read 10 Done Done New true
14351 4782 Acked 5781 Read 4 Done Done New true
14352 4783 Acked 5782 Write 5 Done Done New true
14353 4784 Acked 5783 Read 9 Done Done New true
14354 4785 Acked 5784 Read 10 Done Done New true
14355 4786 Acked 5785 Read 5 Done Done New true
14356 4787 Acked 5786 Write 1 Done Done New true
14357 4788 Acked 5787 Write 8 Done Done New true
14358 4789 Acked 5788 Read 10 Done Done New true
14359 4790 Acked 5789 Read 1 Done Done New true
14360 4791 Acked 5790 Write 3 Done Done New true
14361 4792 Acked 5791 Write 6 Done Done New true
14362 4793 Acked 5792 Read 2 Done Done New true
14363 4794 Acked 5793 Write 8 Done Done New true
14364 4795 Acked 5794 Read 10 Done Done New true
14365 4796 Acked 5795 Read 5 Done Done New true
14366 4797 Acked 5796 Read 10 Done Done New true
14367 4798 Acked 5797 Read 10 Done Done New true
14368 4799 Acked 5798 Read 3 Done Done New true
14369 4800 Acked 5799 Write 2 Done Done New true
14370 4801 Acked 5800 Read 5 Done Done New true
14371 4802 Acked 5801 Write 9 Done Done New true
14372 4803 Acked 5802 Write 7 Done Done New true
14373 4804 Acked 5803 Read 6 Done Done New true
14374 4805 Acked 5804 Write 8 Done Done New true
14375 4806 Acked 5805 Write 3 Done Done New true
14376 4807 Acked 5806 Read 10 Done Done New true
14377 4808 Acked 5807 Flush 0 Done Done New true
14378 4809 Acked 5808 Read 1 Done Done New true
14379 4810 Acked 5809 Read 9 Done Done New true
14380 4811 Acked 5810 Write 2 Done Done New true
14381 4812 Acked 5811 Read 2 Done Done New true
14382 4813 Acked 5812 Read 5 Done Done New true
14383 4814 Acked 5813 Write 6 Done Done New true
14384 4815 Acked 5814 Flush 0 Done Done New true
14385 4816 Acked 5815 Read 4 Done Done New true
14386 4817 Acked 5816 Flush 0 Done Done New true
14387 4818 Acked 5817 Read 7 Done Done New true
14388 4819 Acked 5818 Write 5 Done Done New true
14389 4820 Acked 5819 Write 8 Done Done New true
14390 4821 Acked 5820 Write 9 Done Done New true
14391 4822 Acked 5821 Flush 0 Done Done New true
14392 STATES DS:0 DS:1 DS:2 TOTAL
14393 New 0 0 301 301
14394 Sent 0 0 0 0
14395 Done 301 301 0 602
14396 Skipped 0 0 0 0
14397 Error 0 0 0 0
14398 Last Flush: 5821 5821 5520
14399 Downstairs last five completed: 5520 5519 5518 5517 5516
14400 Upstairs last five completed: 4822 4821 4820 4819 4818
14401 CLIENT: Up:0 ds:301 act:2
144022024-02-27T21:47:28.276ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
144032024-02-27T21:47:28.276ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
144042024-02-27T21:47:28.276ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
144052024-02-27T21:47:28.276ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
144062024-02-27T21:47:28.276ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
144072024-02-27T21:47:28.276ZINFOcrucible: [2] client re-new 301 jobs since flush 5520 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
144082024-02-27T21:47:28.276ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
144092024-02-27T21:47:29.277ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
144102024-02-27T21:47:29.277ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
144112024-02-27T21:47:29.277ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
144122024-02-27T21:47:29.277ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
144132024-02-27T21:47:29.277ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
144142024-02-27T21:47:29.277ZINFOcrucible: [2] client re-new 301 jobs since flush 5520 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
144152024-02-27T21:47:29.277ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
144162024-02-27T21:47:30.278ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
144172024-02-27T21:47:30.278ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
144182024-02-27T21:47:30.278ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
144192024-02-27T21:47:30.278ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
144202024-02-27T21:47:30.278ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
144212024-02-27T21:47:30.278ZINFOcrucible: [2] client re-new 301 jobs since flush 5520 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
144222024-02-27T21:47:30.278ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
14423 [8830] got start action
14424 [8830] I am going to restart
14425 Starting downstairs at port 8830
14426 Make output file at "/tmp/dsc/downstairs-8830.txt"
14427 [8830][2] reports Starting
14428 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1183)
14429 [8830][2] reports Running
144302024-02-27T21:47:31.280ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
144312024-02-27T21:47:31.280ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
144322024-02-27T21:47:31.280ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
144332024-02-27T21:47:31.280ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
144342024-02-27T21:47:31.280ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
144352024-02-27T21:47:31.280ZINFOcrucible: [2] client re-new 301 jobs since flush 5520 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
144362024-02-27T21:47:31.280ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
14437 ----------------------------------------------------------------
14438 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14439 States: Active Active Offline
14440 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
14441 4522 Acked 5521 Read 7 Done Done New true
14442 4523 Acked 5522 Read 2 Done Done New true
14443 4524 Acked 5523 Read 7 Done Done New true
14444 4525 Acked 5524 Write 1 Done Done New true
14445 4526 Acked 5525 Write 5 Done Done New true
14446 4527 Acked 5526 Read 8 Done Done New true
14447 4528 Acked 5527 Read 10 Done Done New true
14448 4529 Acked 5528 Write 8 Done Done New true
14449 4530 Acked 5529 Write 5 Done Done New true
14450 4531 Acked 5530 Read 10 Done Done New true
14451 4532 Acked 5531 Read 6 Done Done New true
14452 4533 Acked 5532 Write 8 Done Done New true
14453 4534 Acked 5533 Write 9 Done Done New true
14454 4535 Acked 5534 Read 6 Done Done New true
14455 4536 Acked 5535 Read 9 Done Done New true
14456 4537 Acked 5536 Write 5 Done Done New true
14457 4538 Acked 5537 Read 10 Done Done New true
14458 4539 Acked 5538 Write 4 Done Done New true
14459 4540 Acked 5539 Read 2 Done Done New true
14460 4541 Acked 5540 Write 9 Done Done New true
14461 4542 Acked 5541 Read 2 Done Done New true
14462 4543 Acked 5542 Write 2 Done Done New true
14463 4544 Acked 5543 Flush 0 Done Done New true
14464 4545 Acked 5544 Write 5 Done Done New true
14465 4546 Acked 5545 Read 8 Done Done New true
14466 4547 Acked 5546 Write 5 Done Done New true
14467 4548 Acked 5547 Write 9 Done Done New true
14468 4549 Acked 5548 Write 2 Done Done New true
14469 4550 Acked 5549 Read 8 Done Done New true
14470 4551 Acked 5550 Read 5 Done Done New true
14471 4552 Acked 5551 Read 10 Done Done New true
14472 4553 Acked 5552 Read 6 Done Done New true
14473 4554 Acked 5553 Write 4 Done Done New true
14474 4555 Acked 5554 Read 5 Done Done New true
14475 4556 Acked 5555 Write 1 Done Done New true
14476 4557 Acked 5556 Read 2 Done Done New true
14477 4558 Acked 5557 Read 9 Done Done New true
14478 4559 Acked 5558 Write 2 Done Done New true
14479 4560 Acked 5559 Write 5 Done Done New true
14480 4561 Acked 5560 Write 9 Done Done New true
14481 4562 Acked 5561 Write 10 Done Done New true
14482 4563 Acked 5562 Read 5 Done Done New true
14483 4564 Acked 5563 Flush 0 Done Done New true
14484 4565 Acked 5564 Read 4 Done Done New true
14485 4566 Acked 5565 Write 10 Done Done New true
14486 4567 Acked 5566 Flush 0 Done Done New true
14487 4568 Acked 5567 Write 3 Done Done New true
14488 4569 Acked 5568 Write 10 Done Done New true
14489 4570 Acked 5569 Flush 0 Done Done New true
14490 4571 Acked 5570 Write 5 Done Done New true
14491 4572 Acked 5571 Write 3 Done Done New true
14492 4573 Acked 5572 Read 6 Done Done New true
14493 4574 Acked 5573 Read 5 Done Done New true
14494 4575 Acked 5574 Read 4 Done Done New true
14495 4576 Acked 5575 Write 2 Done Done New true
14496 4577 Acked 5576 Write 1 Done Done New true
14497 4578 Acked 5577 Write 8 Done Done New true
14498 4579 Acked 5578 Read 3 Done Done New true
14499 4580 Acked 5579 Write 3 Done Done New true
14500 4581 Acked 5580 Read 9 Done Done New true
14501 4582 Acked 5581 Write 4 Done Done New true
14502 4583 Acked 5582 Read 5 Done Done New true
14503 4584 Acked 5583 Write 9 Done Done New true
14504 4585 Acked 5584 Read 4 Done Done New true
14505 4586 Acked 5585 Flush 0 Done Done New true
14506 4587 Acked 5586 Write 10 Done Done New true
14507 4588 Acked 5587 Write 7 Done Done New true
14508 4589 Acked 5588 Write 1 Done Done New true
14509 4590 Acked 5589 Read 5 Done Done New true
14510 4591 Acked 5590 Write 8 Done Done New true
14511 4592 Acked 5591 Write 3 Done Done New true
14512 4593 Acked 5592 Write 9 Done Done New true
14513 4594 Acked 5593 Write 3 Done Done New true
14514 4595 Acked 5594 Read 1 Done Done New true
14515 4596 Acked 5595 Read 1 Done Done New true
14516 4597 Acked 5596 Write 10 Done Done New true
14517 4598 Acked 5597 Write 4 Done Done New true
14518 4599 Acked 5598 Read 4 Done Done New true
14519 4600 Acked 5599 Read 8 Done Done New true
14520 4601 Acked 5600 Write 9 Done Done New true
14521 4602 Acked 5601 Read 1 Done Done New true
14522 4603 Acked 5602 Read 1 Done Done New true
14523 4604 Acked 5603 Write 8 Done Done New true
14524 4605 Acked 5604 Read 9 Done Done New true
14525 4606 Acked 5605 Write 7 Done Done New true
14526 4607 Acked 5606 Write 4 Done Done New true
14527 4608 Acked 5607 Read 5 Done Done New true
14528 4609 Acked 5608 Read 6 Done Done New true
14529 4610 Acked 5609 Write 1 Done Done New true
14530 4611 Acked 5610 Read 10 Done Done New true
14531 4612 Acked 5611 Write 7 Done Done New true
14532 4613 Acked 5612 Write 9 Done Done New true
14533 4614 Acked 5613 Flush 0 Done Done New true
14534 4615 Acked 5614 Read 8 Done Done New true
14535 4616 Acked 5615 Write 2 Done Done New true
14536 4617 Acked 5616 Write 10 Done Done New true
14537 4618 Acked 5617 Write 9 Done Done New true
14538 4619 Acked 5618 Read 4 Done Done New true
14539 4620 Acked 5619 Read 2 Done Done New true
14540 4621 Acked 5620 Write 7 Done Done New true
14541 4622 Acked 5621 Read 5 Done Done New true
14542 4623 Acked 5622 Read 8 Done Done New true
14543 4624 Acked 5623 Read 2 Done Done New true
14544 4625 Acked 5624 Read 9 Done Done New true
14545 4626 Acked 5625 Write 7 Done Done New true
14546 4627 Acked 5626 Read 10 Done Done New true
14547 4628 Acked 5627 Write 2 Done Done New true
14548 4629 Acked 5628 Write 2 Done Done New true
14549 4630 Acked 5629 Read 6 Done Done New true
14550 4631 Acked 5630 Write 1 Done Done New true
14551 4632 Acked 5631 Read 4 Done Done New true
14552 4633 Acked 5632 Read 7 Done Done New true
14553 4634 Acked 5633 Write 4 Done Done New true
14554 4635 Acked 5634 Read 7 Done Done New true
14555 4636 Acked 5635 Read 8 Done Done New true
14556 4637 Acked 5636 Write 4 Done Done New true
14557 4638 Acked 5637 Read 5 Done Done New true
14558 4639 Acked 5638 Write 5 Done Done New true
14559 4640 Acked 5639 Write 3 Done Done New true
14560 4641 Acked 5640 Read 7 Done Done New true
14561 4642 Acked 5641 Read 3 Done Done New true
14562 4643 Acked 5642 Read 6 Done Done New true
14563 4644 Acked 5643 Write 3 Done Done New true
14564 4645 Acked 5644 Read 5 Done Done New true
14565 4646 Acked 5645 Flush 0 Done Done New true
14566 4647 Acked 5646 Write 1 Done Done New true
14567 4648 Acked 5647 Write 3 Done Done New true
14568 4649 Acked 5648 Write 5 Done Done New true
14569 4650 Acked 5649 Read 9 Done Done New true
14570 4651 Acked 5650 Write 4 Done Done New true
14571 4652 Acked 5651 Write 4 Done Done New true
14572 4653 Acked 5652 Read 6 Done Done New true
14573 4654 Acked 5653 Write 1 Done Done New true
14574 4655 Acked 5654 Write 9 Done Done New true
14575 4656 Acked 5655 Write 1 Done Done New true
14576 4657 Acked 5656 Read 7 Done Done New true
14577 4658 Acked 5657 Write 3 Done Done New true
14578 4659 Acked 5658 Read 1 Done Done New true
14579 4660 Acked 5659 Write 2 Done Done New true
14580 4661 Acked 5660 Read 6 Done Done New true
14581 4662 Acked 5661 Write 7 Done Done New true
14582 4663 Acked 5662 Read 9 Done Done New true
14583 4664 Acked 5663 Write 9 Done Done New true
14584 4665 Acked 5664 Read 6 Done Done New true
14585 4666 Acked 5665 Write 10 Done Done New true
14586 4667 Acked 5666 Write 8 Done Done New true
14587 4668 Acked 5667 Read 4 Done Done New true
14588 4669 Acked 5668 Read 4 Done Done New true
14589 4670 Acked 5669 Write 9 Done Done New true
14590 4671 Acked 5670 Write 10 Done Done New true
14591 4672 Acked 5671 Write 1 Done Done New true
14592 4673 Acked 5672 Read 3 Done Done New true
14593 4674 Acked 5673 Write 2 Done Done New true
14594 4675 Acked 5674 Read 1 Done Done New true
14595 4676 Acked 5675 Write 4 Done Done New true
14596 4677 Acked 5676 Read 1 Done Done New true
14597 4678 Acked 5677 Read 1 Done Done New true
14598 4679 Acked 5678 Write 10 Done Done New true
14599 4680 Acked 5679 Read 8 Done Done New true
14600 4681 Acked 5680 Write 2 Done Done New true
14601 4682 Acked 5681 Write 4 Done Done New true
14602 4683 Acked 5682 Read 6 Done Done New true
14603 4684 Acked 5683 Write 1 Done Done New true
14604 4685 Acked 5684 Write 1 Done Done New true
14605 4686 Acked 5685 Read 3 Done Done New true
14606 4687 Acked 5686 Read 9 Done Done New true
14607 4688 Acked 5687 Write 8 Done Done New true
14608 4689 Acked 5688 Read 9 Done Done New true
14609 4690 Acked 5689 Flush 0 Done Done New true
14610 4691 Acked 5690 Read 3 Done Done New true
14611 4692 Acked 5691 Read 6 Done Done New true
14612 4693 Acked 5692 Flush 0 Done Done New true
14613 4694 Acked 5693 Write 2 Done Done New true
14614 4695 Acked 5694 Read 10 Done Done New true
14615 4696 Acked 5695 Flush 0 Done Done New true
14616 4697 Acked 5696 Write 5 Done Done New true
14617 4698 Acked 5697 Write 7 Done Done New true
14618 4699 Acked 5698 Flush 0 Done Done New true
14619 4700 Acked 5699 Write 8 Done Done New true
14620 4701 Acked 5700 Read 6 Done Done New true
14621 4702 Acked 5701 Read 9 Done Done New true
14622 4703 Acked 5702 Read 5 Done Done New true
14623 4704 Acked 5703 Flush 0 Done Done New true
14624 4705 Acked 5704 Write 4 Done Done New true
14625 4706 Acked 5705 Read 3 Done Done New true
14626 4707 Acked 5706 Read 2 Done Done New true
14627 4708 Acked 5707 Read 2 Done Done New true
14628 4709 Acked 5708 Write 1 Done Done New true
14629 4710 Acked 5709 Read 6 Done Done New true
14630 4711 Acked 5710 Write 5 Done Done New true
14631 4712 Acked 5711 Write 10 Done Done New true
14632 4713 Acked 5712 Write 10 Done Done New true
14633 4714 Acked 5713 Flush 0 Done Done New true
14634 4715 Acked 5714 Read 4 Done Done New true
14635 4716 Acked 5715 Read 6 Done Done New true
14636 4717 Acked 5716 Write 5 Done Done New true
14637 4718 Acked 5717 Write 6 Done Done New true
14638 4719 Acked 5718 Write 5 Done Done New true
14639 4720 Acked 5719 Write 8 Done Done New true
14640 4721 Acked 5720 Write 6 Done Done New true
14641 4722 Acked 5721 Write 4 Done Done New true
14642 4723 Acked 5722 Read 4 Done Done New true
14643 4724 Acked 5723 Write 2 Done Done New true
14644 4725 Acked 5724 Read 5 Done Done New true
14645 4726 Acked 5725 Read 2 Done Done New true
14646 4727 Acked 5726 Read 4 Done Done New true
14647 4728 Acked 5727 Write 4 Done Done New true
14648 4729 Acked 5728 Write 8 Done Done New true
14649 4730 Acked 5729 Write 3 Done Done New true
14650 4731 Acked 5730 Read 9 Done Done New true
14651 4732 Acked 5731 Write 1 Done Done New true
14652 4733 Acked 5732 Write 9 Done Done New true
14653 4734 Acked 5733 Write 4 Done Done New true
14654 4735 Acked 5734 Write 8 Done Done New true
14655 4736 Acked 5735 Read 3 Done Done New true
14656 4737 Acked 5736 Write 7 Done Done New true
14657 4738 Acked 5737 Write 3 Done Done New true
14658 4739 Acked 5738 Read 8 Done Done New true
14659 4740 Acked 5739 Write 1 Done Done New true
14660 4741 Acked 5740 Read 5 Done Done New true
14661 4742 Acked 5741 Write 1 Done Done New true
14662 4743 Acked 5742 Read 10 Done Done New true
14663 4744 Acked 5743 Write 7 Done Done New true
14664 4745 Acked 5744 Read 1 Done Done New true
14665 4746 Acked 5745 Read 7 Done Done New true
14666 4747 Acked 5746 Write 10 Done Done New true
14667 4748 Acked 5747 Read 7 Done Done New true
14668 4749 Acked 5748 Write 8 Done Done New true
14669 4750 Acked 5749 Write 3 Done Done New true
14670 4751 Acked 5750 Write 1 Done Done New true
14671 4752 Acked 5751 Read 7 Done Done New true
14672 4753 Acked 5752 Flush 0 Done Done New true
14673 4754 Acked 5753 Read 7 Done Done New true
14674 4755 Acked 5754 Write 7 Done Done New true
14675 4756 Acked 5755 Read 1 Done Done New true
14676 4757 Acked 5756 Read 7 Done Done New true
14677 4758 Acked 5757 Write 6 Done Done New true
14678 4759 Acked 5758 Read 4 Done Done New true
14679 4760 Acked 5759 Read 8 Done Done New true
14680 4761 Acked 5760 Flush 0 Done Done New true
14681 4762 Acked 5761 Read 4 Done Done New true
14682 4763 Acked 5762 Read 3 Done Done New true
14683 4764 Acked 5763 Write 1 Done Done New true
14684 4765 Acked 5764 Read 3 Done Done New true
14685 4766 Acked 5765 Flush 0 Done Done New true
14686 4767 Acked 5766 Read 2 Done Done New true
14687 4768 Acked 5767 Write 5 Done Done New true
14688 4769 Acked 5768 Read 3 Done Done New true
14689 4770 Acked 5769 Read 5 Done Done New true
14690 4771 Acked 5770 Read 8 Done Done New true
14691 4772 Acked 5771 Write 4 Done Done New true
14692 4773 Acked 5772 Write 4 Done Done New true
14693 4774 Acked 5773 Write 3 Done Done New true
14694 4775 Acked 5774 Write 4 Done Done New true
14695 4776 Acked 5775 Write 9 Done Done New true
14696 4777 Acked 5776 Write 7 Done Done New true
14697 4778 Acked 5777 Write 9 Done Done New true
14698 4779 Acked 5778 Read 7 Done Done New true
14699 4780 Acked 5779 Flush 0 Done Done New true
14700 4781 Acked 5780 Read 10 Done Done New true
14701 4782 Acked 5781 Read 4 Done Done New true
14702 4783 Acked 5782 Write 5 Done Done New true
14703 4784 Acked 5783 Read 9 Done Done New true
14704 4785 Acked 5784 Read 10 Done Done New true
14705 4786 Acked 5785 Read 5 Done Done New true
14706 4787 Acked 5786 Write 1 Done Done New true
14707 4788 Acked 5787 Write 8 Done Done New true
14708 4789 Acked 5788 Read 10 Done Done New true
14709 4790 Acked 5789 Read 1 Done Done New true
14710 4791 Acked 5790 Write 3 Done Done New true
14711 4792 Acked 5791 Write 6 Done Done New true
14712 4793 Acked 5792 Read 2 Done Done New true
14713 4794 Acked 5793 Write 8 Done Done New true
14714 4795 Acked 5794 Read 10 Done Done New true
14715 4796 Acked 5795 Read 5 Done Done New true
14716 4797 Acked 5796 Read 10 Done Done New true
14717 4798 Acked 5797 Read 10 Done Done New true
14718 4799 Acked 5798 Read 3 Done Done New true
14719 4800 Acked 5799 Write 2 Done Done New true
14720 4801 Acked 5800 Read 5 Done Done New true
14721 4802 Acked 5801 Write 9 Done Done New true
14722 4803 Acked 5802 Write 7 Done Done New true
14723 4804 Acked 5803 Read 6 Done Done New true
14724 4805 Acked 5804 Write 8 Done Done New true
14725 4806 Acked 5805 Write 3 Done Done New true
14726 4807 Acked 5806 Read 10 Done Done New true
14727 4808 Acked 5807 Flush 0 Done Done New true
14728 4809 Acked 5808 Read 1 Done Done New true
14729 4810 Acked 5809 Read 9 Done Done New true
14730 4811 Acked 5810 Write 2 Done Done New true
14731 4812 Acked 5811 Read 2 Done Done New true
14732 4813 Acked 5812 Read 5 Done Done New true
14733 4814 Acked 5813 Write 6 Done Done New true
14734 4815 Acked 5814 Flush 0 Done Done New true
14735 4816 Acked 5815 Read 4 Done Done New true
14736 4817 Acked 5816 Flush 0 Done Done New true
14737 4818 Acked 5817 Read 7 Done Done New true
14738 4819 Acked 5818 Write 5 Done Done New true
14739 4820 Acked 5819 Write 8 Done Done New true
14740 4821 Acked 5820 Write 9 Done Done New true
14741 4822 Acked 5821 Flush 0 Done Done New true
14742 STATES DS:0 DS:1 DS:2 TOTAL
14743 New 0 0 301 301
14744 Sent 0 0 0 0
14745 Done 301 301 0 602
14746 Skipped 0 0 0 0
14747 Error 0 0 0 0
14748 Last Flush: 5821 5821 5520
14749 Downstairs last five completed: 5520 5519 5518 5517 5516
14750 Upstairs last five completed: 4822 4821 4820 4819 4818
14751 CLIENT: Up:0 ds:301 act:2
147522024-02-27T21:47:32.281ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147532024-02-27T21:47:32.281ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147542024-02-27T21:47:32.283ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147552024-02-27T21:47:32.283ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147562024-02-27T21:47:32.283ZINFOcrucible: send last flush ID to this DS: 5520 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147572024-02-27T21:47:32.283ZINFOcrucible: Replied this last flush ID: 5520 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147582024-02-27T21:47:32.283ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147592024-02-27T21:47:32.283ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147602024-02-27T21:47:32.283ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147612024-02-27T21:47:32.283ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
14762 ----------------------------------------------------------------
14763 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
14764 Downstairs last five completed: 5821 5820 5819 5818 5817
14765 Upstairs last five completed: 4822 4821 4820 4819 4818
14766 CLIENT: Up:0 ds:0 act:3
14767 Replay: All jobs finished, all DS active.
14768 Main task has work to do, go find it
14769 got dsc Stop(2)
14770 stop 2
14771 Feb 27 21:47:35.601 INFO request completed, latency_us: 400, response_code[8830] Got stop action so:false kr:false
14772 : 204, uri: /stop/cid/2, method: POST, req_id: 39019e16-f68e-4dfd-b6c9-a34213f0448d, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
147732024-02-27T21:47:35.602ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147742024-02-27T21:47:35.602ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147752024-02-27T21:47:35.602ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = e6a5a974-925a-494f-8845-facc441c2690
14776 Feb 27 21:47:35.602 INFO request completed, latency_us: 129, response_code: 200, uri: /state/cid/2{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:47:35.602752178Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"client":"2","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
14777 , method: GET, req_id: 254e3ffb-3ba5-4dec-868f-9ecc976b75b7, remote_addr: 127.0.0.1:{"msg":"[2] client re-new 0 jobs since flush 5821","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:47:35.602775695Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"downstairs","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
14778 44347{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:47:35.602808301Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"io task","client":"2","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
14779 , local_addr: 127.0.0.1:9998
14780 [8830] Exited with: ExitStatus(unix_wait_status(9))
14781 [8830][2] reports Exit
147822024-02-27T21:47:36.603ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147832024-02-27T21:47:36.604ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147842024-02-27T21:47:36.604ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147852024-02-27T21:47:36.604ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
147862024-02-27T21:47:36.604ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147872024-02-27T21:47:36.604ZINFOcrucible: [2] client re-new 0 jobs since flush 5821 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
147882024-02-27T21:47:36.604ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147892024-02-27T21:47:37.605ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147902024-02-27T21:47:37.605ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147912024-02-27T21:47:37.605ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147922024-02-27T21:47:37.605ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
147932024-02-27T21:47:37.605ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147942024-02-27T21:47:37.605ZINFOcrucible: [2] client re-new 0 jobs since flush 5821 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
147952024-02-27T21:47:37.605ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147962024-02-27T21:47:38.606ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147972024-02-27T21:47:38.606ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147982024-02-27T21:47:38.606ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
147992024-02-27T21:47:38.606ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
148002024-02-27T21:47:38.606ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
148012024-02-27T21:47:38.606ZINFOcrucible: [2] client re-new 0 jobs since flush 5821 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
148022024-02-27T21:47:38.606ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
14803 Feb 27 21:47:39.603 INFO request completed, latency_us: 200, response_code: 200, uri: /state/cid/2, method: GET, req_id: 51e8fe83-9558-44f7-8057-73e1a886b45e, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
14804 001/300 Flush
14805 002/300 Write block 3836 len 36864 data: 3 3 3 4 5 5 6 6 6
14806 003/300 Read block 4290 len 20480
148072024-02-27T21:47:39.607ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
148082024-02-27T21:47:39.607ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
148092024-02-27T21:47:39.607ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
148102024-02-27T21:47:39.607ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
148112024-02-27T21:47:39.607ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
148122024-02-27T21:47:39.607ZINFOcrucible: [2] client re-new 3 jobs since flush 5821 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
148132024-02-27T21:47:39.607ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
14814 004/300 Write block 3328 len 36864 data: 5 4 4 4 4 6 6 6 5
14815 005/300 Write block 4625 len 28672 data: 5 4 4 5 6 7 6
14816 006/300 Read block 4851 len 24576
14817 007/300 Read block 2130 len 28672
14818 008/300 Write block 1639 len 32768 data: 7 6 6 6 6 5 5 6
14819 009/300 Read block 3111 len 28672
14820 010/300 Write block 791 len 16384 data: 5 5 6 6
14821 011/300 Read block 3566 len 16384
14822 012/300 Read block 2136 len 28672
14823 013/300 Write block 2733 len 16384 data: 4 5 5 4
14824 014/300 Read block 3818 len 12288
14825 015/300 Write block 317 len 16384 data: 4 4 3 3
14826 016/300 Write block 4345 len 32768 data: 3 3 3 3 3 3 3 4
14827 017/300 Read block 95 len 40960
14828 018/300 Read block 2864 len 28672
14829 019/300 Write block 4493 len 36864 data: 7 6 6 5 4 5 5 5 3
14830 020/300 Read block 3479 len 20480
14831 021/300 Write block 1903 len 36864 data: 2 2 2 2 2 2 2 3 4
14832 022/300 Read block 4866 len 8192
14833 023/300 Read block 4695 len 20480
14834 024/300 Read block 2005 len 20480
14835 025/300 Read block 1166 len 16384
14836 026/300 Read block 2894 len 32768
14837 027/300 Flush
14838 028/300 Write block 3234 len 36864 data: 5 5 3 3 4 6 6 7 5
14839 029/300 Flush
14840 030/300 Write block 634 len 16384 data: 3 3 2 2
14841 031/300 Read block 3519 len 4096
14842 032/300 Read block 2154 len 40960
14843 033/300 Write block 4830 len 12288 data: 6 5 5
14844 034/300 Read block 4593 len 24576
14845 035/300 Flush
14846 036/300 Read block 3625 len 8192
14847 037/300 Write block 943 len 32768 data: 6 6 5 6 5 5 4 5
14848 038/300 Write block 2885 len 36864 data: 4 3 2 2 2 2 2 2 2
14849 039/300 Write block 419 len 28672 data: 6 6 6 5 4 5 4
14850 040/300 Read block 1434 len 12288
14851 041/300 Write block 4408 len 40960 data: 5 5 4 3 3 3 3 3 2 2
14852 042/300 Write block 562 len 28672 data: 4 4 4 4 5 4 3
14853 043/300 Read block 3436 len 16384
14854 044/300 Read block 4330 len 40960
14855 045/300 Read block 4832 len 16384
14856 046/300 Write block 4424 len 36864 data: 3 3 2 4 5 3 5 6 6
14857 047/300 Write block 4816 len 24576 data: 4 4 4 4 3 3
14858 048/300 Write block 3157 len 4096 data: 4
14859 049/300 Flush
14860 050/300 Read block 2054 len 20480
14861 051/300 Read block 463 len 24576
14862 052/300 Write block 1692 len 4096 data: 2
14863 053/300 Write block 2110 len 32768 data: 2 2 2 2 2 2 2 2
14864 054/300 Flush
14865 055/300 Read block 1234 len 20480
14866 056/300 Write block 943 len 12288 data: 7 7 6
14867 057/300 Write block 4964 len 40960 data: 3 3 3 3 3 5 6 5 5 5
14868 058/300 Read block 4681 len 36864
14869 059/300 Read block 2877 len 8192
14870 060/300 Read block 4156 len 32768
14871 061/300 Write block 2899 len 32768 data: 3 4 4 3 3 2 4 5
14872 062/300 Write block 3570 len 28672 data: 6 6 6 6 5 5 5
14873 063/300 Read block 2335 len 40960
14874 064/300 Flush
14875 065/300 Read block 2622 len 4096
14876 066/300 Write block 3671 len 36864 data: 3 3 3 3 3 3 5 7 8
14877 067/300 Read block 2707 len 28672
14878 068/300 Read block 4322 len 16384
14879 069/300 Read block 4379 len 12288
14880 070/300 Write block 4652 len 12288 data: 5 5 6
14881 071/300 Flush
14882 072/300 Write block 2354 len 12288 data: 5 5 6
14883 073/300 Write block 1398 len 12288 data: 5 4 4
14884 074/300 Read block 2828 len 24576
14885 075/300 Write block 2468 len 32768 data: 3 4 4 4 4 4 4 3
14886 076/300 Write block 1326 len 36864 data: 5 3 3 3 4 3 3 3 3
14887 077/300 Write block 3319 len 16384 data: 6 5 4 4
14888 078/300 Read block 3535 len 16384
14889 079/300 Read block 1573 len 36864
14890 080/300 Read block 4193 len 32768
14891 081/300 Write block 4673 len 8192 data: 5 4
14892 082/300 Read block 4791 len 8192
14893 083/300 Write block 1012 len 12288 data: 3 4 4
14894 084/300 Flush
14895 085/300 Read block 3530 len 40960
14896 086/300 Read block 4624 len 8192
14897 087/300 Read block 852 len 4096
14898 088/300 Read block 4084 len 12288
14899 089/300 Read block 2408 len 16384
14900 090/300 Write block 1128 len 24576 data: 4 4 6 7 8 6
14901 091/300 Read block 2978 len 28672
14902 092/300 Read block 4046 len 28672
14903 093/300 Write block 4547 len 32768 data: 5 5 5 6 5 5 4 3
14904 094/300 Read block 1657 len 40960
14905 095/300 Read block 1393 len 24576
14906 096/300 Write block 429 len 40960 data: 3 2 3 4 4 4 5 5 5 4
14907 097/300 Write block 2547 len 4096 data: 4
14908 098/300 Read block 3997 len 24576
14909 099/300 Write block 2187 len 36864 data: 2 2 3 3 5 4 5 5 3
14910 100/300 Write block 4668 len 32768 data: 6 4 5 5 5 6 5 3
14911 101/300 Read block 2785 len 40960
14912 102/300 Read block 1372 len 32768
14913 103/300 Write block 1797 len 8192 data: 3 3
14914 104/300 Read block 3484 len 28672
14915 105/300 Write block 4435 len 24576 data: 6 7 7 6 5 5
14916 106/300 Read block 3261 len 36864
14917 107/300 Write block 584 len 20480 data: 5 5 7 5 6
14918 108/300 Read block 1784 len 36864
14919 109/300 Read block 2805 len 16384
14920 110/300 Write block 1427 len 12288 data: 5 4 4
14921 111/300 Write block 992 len 40960 data: 4 5 4 2 2 3 3 3 3 2
14922 112/300 Read block 3263 len 40960
14923 113/300 Write block 2570 len 40960 data: 7 7 7 5 5 6 6 5 4 5
14924 114/300 Read block 4251 len 20480
14925 115/300 Read block 2568 len 36864
14926 116/300 Write block 3402 len 32768 data: 4 4 2 2 2 3 3 4
14927 117/300 Write block 3280 len 36864 data: 3 3 2 2 3 2 3 3 3
14928 118/300 Write block 2663 len 24576 data: 6 5 7 6 6 8
14929 119/300 Read block 967 len 32768
14930 120/300 Write block 2610 len 28672 data: 3 3 3 5 4 7 5
14931 121/300 Flush
14932 122/300 Write block 618 len 4096 data: 5
14933 123/300 Write block 3255 len 4096 data: 5
14934 124/300 Write block 151 len 24576 data: 5 5 4 4 4 4
14935 125/300 Read block 1187 len 24576
14936 126/300 Read block 2449 len 32768
14937 127/300 Read block 236 len 12288
14938 128/300 Read block 1408 len 4096
14939 129/300 Flush
14940 130/300 Read block 0 len 20480
14941 131/300 Read block 3909 len 24576
14942 132/300 Write block 709 len 20480 data: 3 3 3 3 3
14943 133/300 Read block 716 len 8192
14944 134/300 Write block 4527 len 8192 data: 4 4
14945 135/300 Read block 2842 len 16384
14946 136/300 Read block 370 len 4096
14947 137/300 Read block 3722 len 40960
14948 138/300 Read block 2793 len 4096
14949 139/300 Write block 3463 len 16384 data: 3 2 2 2
14950 140/300 Read block 4001 len 16384
14951 141/300 Read block 1997 len 8192
14952 142/300 Flush
14953 143/300 Read block 4169 len 12288
14954 144/300 Read block 4835 len 12288
14955 145/300 Write block 2669 len 8192 data: 9 9
14956 146/300 Read block 2545 len 28672
14957 147/300 Write block 2617 len 12288 data: 5 6 5
14958 148/300 Read block 3255 len 8192
14959 149/300 Read block 1173 len 4096
14960 150/300 Write block 2487 len 40960 data: 4 6 6 6 6 6 6 6 5 4
14961 151/300 Write block 1186 len 28672 data: 3 3 5 5 5 6 4
14962 152/300 Write block 1094 len 24576 data: 4 3 2 2 3 4
14963 153/300 Read block 3215 len 36864
14964 154/300 Read block 3011 len 28672
14965 155/300 Write block 288 len 28672 data: 3 2 2 2 2 2 3
14966 156/300 Read block 782 len 24576
14967 157/300 Read block 2824 len 40960
14968 158/300 Read block 2724 len 40960
14969 159/300 Read block 4255 len 20480
14970 160/300 Write block 4694 len 20480 data: 7 6 4 3 3
14971 161/300 Read block 4438 len 40960
14972 162/300 Read block 2742 len 8192
14973 163/300 Read block 1002 len 4096
14974 164/300 Write block 455 len 32768 data: 4 3 3 3 3 3 3 3
14975 165/300 Write block 4498 len 40960 data: 6 6 6 4 4 4 4 4 4 4
14976 166/300 Read block 272 len 16384
14977 167/300 Read block 4492 len 16384
14978 168/300 Write block 3383 len 20480 data: 4 3 3 4 5
14979 169/300 Read block 2195 len 20480
14980 170/300 Write block 3111 len 28672 data: 7 5 5 6 7 6 6
14981 171/300 Read block 3476 len 20480
14982 172/300 Read block 201 len 40960
14983 173/300 Write block 740 len 24576 data: 4 4 4 3 2 2
14984 174/300 Flush
14985 175/300 Write block 4498 len 24576 data: 7 7 7 5 5 5
14986 176/300 Read block 599 len 8192
14987 177/300 Write block 3847 len 28672 data: 4 3 5 6 6 7 7
14988 178/300 Write block 3537 len 28672 data: 5 5 5 5 4 5 5
14989 179/300 Read block 2645 len 40960
14990 180/300 Write block 3232 len 40960 data: 6 5 6 6 4 4 5 7 7 8
14991 181/300 Read block 1963 len 16384
14992 182/300 Read block 1824 len 28672
14993 183/300 Write block 4846 len 12288 data: 3 3 3
14994 184/300 Write block 1881 len 32768 data: 3 3 3 3 3 3 3 3
14995 185/300 Read block 4314 len 28672
14996 186/300 Write block 3418 len 28672 data: 5 5 5 4 4 4 2
14997 187/300 Flush
14998 188/300 Read block 3015 len 20480
14999 189/300 Write block 838 len 20480 data: 7 4 4 5 5
15000 190/300 Write block 4422 len 4096 data: 2
15001 191/300 Read block 691 len 4096
15002 192/300 Flush
15003 193/300 Read block 2309 len 28672
15004 194/300 Write block 2963 len 16384 data: 5 6 6 6
15005 195/300 Read block 841 len 16384
15006 196/300 Read block 1871 len 36864
15007 197/300 Write block 1509 len 16384 data: 7 7 7 7
15008 198/300 Read block 56 len 36864
15009 199/300 Write block 1232 len 20480 data: 4 4 4 4 4
15010 200/300 Flush
15011 201/300 Read block 3315 len 24576
15012 202/300 Write block 896 len 28672 data: 5 3 3 3 2 2 2
15013 203/300 Read block 1065 len 40960
15014 204/300 Read block 679 len 40960
15015 205/300 Read block 4560 len 28672
15016 206/300 Write block 3323 len 20480 data: 4 4 4 4 5
15017 207/300 Read block 1887 len 28672
15018 208/300 Read block 2714 len 36864
15019 209/300 Read block 596 len 40960
15020 210/300 Write block 1432 len 32768 data: 5 6 5 3 4 2 3 3
15021 211/300 Flush
15022 212/300 Flush
15023 213/300 Read block 325 len 12288
15024 214/300 Read block 1907 len 32768
15025 215/300 Read block 1318 len 28672
15026 216/300 Read block 4869 len 20480
15027 217/300 Write block 2121 len 32768 data: 3 4 5 4 4 4 4 4
15028 218/300 Read block 3849 len 40960
15029 219/300 Read block 1987 len 20480
15030 220/300 Write block 3696 len 24576 data: 4 4 4 4 3 3
15031 221/300 Read block 2405 len 16384
15032 222/300 Write block 4982 len 4096 data: 5
15033 223/300 Write block 4916 len 12288 data: 3 3 3
15034 224/300 Write block 2086 len 4096 data: 3
15035 225/300 Read block 4489 len 8192
15036 226/300 Write block 2578 len 36864 data: 5 6 4 3 3 2 2 2 2
15037 227/300 Read block 3428 len 20480
15038 228/300 Write block 823 len 40960 data: 4 5 5 6 5 5 5 5 4 4
15039 229/300 Write block 1996 len 16384 data: 7 7 5 5
15040 230/300 Flush
15041 231/300 Read block 2531 len 16384
15042 232/300 Flush
15043 233/300 Read block 70 len 32768
15044 234/300 Read block 4644 len 16384
15045 235/300 Flush
15046 236/300 Flush
15047 237/300 Read block 1797 len 20480
15048 238/300 Flush
15049 239/300 Write block 732 len 16384 data: 5 5 5 6
15050 240/300 Read block 3588 len 36864
15051 241/300 Write block 1746 len 40960 data: 2 2 3 3 3 3 4 5 5 4
15052 242/300 Write block 1035 len 28672 data: 2 4 4 5 5 5 4
15053 243/300 Read block 552 len 4096
15054 244/300 Write block 2976 len 36864 data: 5 5 6 5 4 3 3 2 2
15055 245/300 Read block 4847 len 32768
15056 246/300 Write block 3022 len 4096 data: 6
15057 247/300 Write block 2137 len 24576 data: 4 4 5 3 3 3
15058 248/300 Write block 3818 len 32768 data: 6 6 6 3 3 3 3 4
15059 249/300 Read block 1055 len 8192
15060 250/300 Flush
15061 251/300 Write block 2662 len 36864 data: 6 7 6 8 7 7 9 10 10
15062 252/300 Flush
15063 253/300 Read block 3064 len 40960
15064 254/300 Read block 966 len 20480
15065 255/300 Write block 3466 len 4096 data: 3
15066 256/300 Read block 4835 len 4096
15067 257/300 Flush
15068 258/300 Flush
15069 259/300 Read block 3893 len 40960
15070 260/300 Write block 2035 len 32768 data: 3 3 4 3 3 2 2 2
15071 261/300 Read block 2755 len 24576
15072 262/300 Read block 3842 len 16384
15073 263/300 Write block 3321 len 40960 data: 5 5 5 5 5 5 6 6 5 5
15074 264/300 Write block 4075 len 8192 data: 5 2
15075 265/300 Write block 3372 len 20480 data: 4 5 6 6 4
15076 266/300 Flush
15077 267/300 Write block 2224 len 12288 data: 3 4 4
15078 268/300 Write block 3661 len 40960 data: 4 5 5 4 5 5 3 3 3 3
15079 269/300 Read block 69 len 12288
15080 270/300 Read block 1141 len 8192
15081 271/300 Read block 2166 len 24576
15082 272/300 Read block 2159 len 12288
15083 273/300 Read block 4958 len 24576
15084 274/300 Read block 2782 len 16384
15085 275/300 Write block 3489 len 24576 data: 5 5 5 7 6 6
15086 276/300 Write block 1457 len 40960 data: 7 7 5 6 5 4 4 4 4 4
15087 277/300 Write block 4399 len 32768 data: 7 7 6 6 3 3 3 5
15088 278/300 Read block 1329 len 32768
15089 279/300 Read block 1976 len 40960
15090 280/300 Write block 4470 len 4096 data: 2
15091 281/300 Read block 1639 len 20480
15092 282/300 Write block 3037 len 20480 data: 4 3 5 5 5
15093 283/300 Flush
15094 284/300 Read block 2344 len 16384
15095 285/300 Read block 2743 len 8192
15096 286/300 Read block 2829 len 24576
15097 287/300 Read block 2376 len 36864
15098 288/300 Write block 970 len 8192 data: 4 4
15099 289/300 Read block 1435 len 20480
15100 290/300 Read block 2 len 32768
15101 291/300 Write block 4398 len 20480 data: 7 8 8 7 7
15102 292/300 Read block 3787 len 12288
15103 293/300 Write block 2288 len 40960 data: 3 3 3 3 3 4 3 3 4 4
15104 294/300 Read block 4883 len 24576
15105 295/300 Read block 3779 len 20480
15106 296/300 Write block 2478 len 40960 data: 3 4 5 5 4 4 4 5 5 5
15107 297/300 Read block 2542 len 40960
15108 298/300 Read block 4061 len 4096
15109 299/300 Read block 2728 len 36864
15110 300/300 Read block 2536 len 28672
15111 Main task has work to do, go find it
15112 got dsc Start(2)
15113 start 2
15114 Feb 27 21:47:39.959 INFO request completed, latency_us: 226, response_code: 204, uri: /start/cid/2, method: POST, req_id: aa26d163-8558-412b-9633-a833d24460f7, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
15115 Replay: started 2, returned:Ok(())
15116 ----------------------------------------------------------------
15117 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15118 States: Active Active Offline
15119 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
15120 4823 Acked 5822 Flush 0 Done Done New true
15121 4824 Acked 5823 Write 9 Done Done New true
15122 4825 Acked 5824 Read 5 Done Done New true
15123 4826 Acked 5825 Write 9 Done Done New false
15124 4827 Acked 5826 Write 7 Done Done New false
15125 4828 Acked 5827 Read 6 Done Done New false
15126 4829 Acked 5828 Read 7 Done Done New false
15127 4830 Acked 5829 Write 8 Done Done New false
15128 4831 Acked 5830 Read 7 Done Done New false
15129 4832 Acked 5831 Write 4 Done Done New false
15130 4833 Acked 5832 Read 4 Done Done New false
15131 4834 Acked 5833 Read 7 Done Done New false
15132 4835 Acked 5834 Write 4 Done Done New false
15133 4836 Acked 5835 Read 3 Done Done New false
15134 4837 Acked 5836 Write 4 Done Done New false
15135 4838 Acked 5837 Write 8 Done Done New false
15136 4839 Acked 5838 Read 10 Done Done New false
15137 4840 Acked 5839 Read 7 Done Done New false
15138 4841 Acked 5840 Write 9 Done Done New false
15139 4842 Acked 5841 Read 5 Done Done New false
15140 4843 Acked 5842 Write 9 Done Done New false
15141 4844 Acked 5843 Read 2 Done Done New false
15142 4845 Acked 5844 Read 5 Done Done New false
15143 4846 Acked 5845 Read 5 Done Done New false
15144 4847 Acked 5846 Read 4 Done Done New false
15145 4848 Acked 5847 Read 8 Done Done New false
15146 4849 Acked 5848 Flush 0 Done Done New false
15147 4850 Acked 5849 Write 9 Done Done New false
15148 4851 Acked 5850 Flush 0 Done Done New false
15149 4852 Acked 5851 Write 4 Done Done New false
15150 4853 Acked 5852 Read 1 Done Done New false
15151 4854 Acked 5853 Read 10 Done Done New false
15152 4855 Acked 5854 Write 3 Done Done New false
15153 4856 Acked 5855 Read 6 Done Done New false
15154 4857 Acked 5856 Flush 0 Done Done New false
15155 4858 Acked 5857 Read 2 Done Done New false
15156 4859 Acked 5858 Write 8 Done Done New false
15157 4860 Acked 5859 Write 9 Done Done New false
15158 4861 Acked 5860 Write 7 Done Done New false
15159 4862 Acked 5861 Read 3 Done Done New false
15160 4863 Acked 5862 Write 10 Done Done New false
15161 4864 Acked 5863 Write 7 Done Done New false
15162 4865 Acked 5864 Read 4 Done Done New false
15163 4866 Acked 5865 Read 10 Done Done New false
15164 4867 Acked 5866 Read 4 Done Done New false
15165 4868 Acked 5867 Write 9 Done Done New false
15166 4869 Acked 5868 Write 6 Done Done New false
15167 4870 Acked 5869 Write 1 Done Done New false
15168 4871 Acked 5870 Flush 0 Done Done New false
15169 4872 Acked 5871 Read 5 Done Done New false
15170 4873 Acked 5872 Read 6 Done Done New false
15171 4874 Acked 5873 Write 1 Done Done New false
15172 4875 Acked 5874 Write 8 Done Done New false
15173 4876 Acked 5875 Flush 0 Done Done New false
15174 4877 Acked 5876 Read 5 Done Done New false
15175 4878 Acked 5877 Write 3 Done Done New false
15176 4879 Acked 5878 Write 10 Done Done New false
15177 4880 Acked 5879 Read 9 Done Done New false
15178 4881 Acked 5880 Read 2 Done Done New false
15179 4882 Acked 5881 Read 8 Done Done New false
15180 4883 Acked 5882 Write 8 Done Done New false
15181 4884 Acked 5883 Write 7 Done Done New false
15182 4885 Acked 5884 Read 10 Done Done New false
15183 4886 Acked 5885 Flush 0 Done Done New false
15184 4887 Acked 5886 Read 1 Done Done New false
15185 4888 Acked 5887 Write 9 Done Done New false
15186 4889 Acked 5888 Read 7 Done Done New false
15187 4890 Acked 5889 Read 4 Done Done New false
15188 4891 Acked 5890 Read 3 Done Done New false
15189 4892 Acked 5891 Write 3 Done Done New false
15190 4893 Acked 5892 Flush 0 Done Done New false
15191 4894 Acked 5893 Write 3 Done Done New false
15192 4895 Acked 5894 Write 3 Done Done New false
15193 4896 Acked 5895 Read 6 Done Done New false
15194 4897 Acked 5896 Write 8 Done Done New false
15195 4898 Acked 5897 Write 9 Done Done New false
15196 4899 Acked 5898 Write 4 Done Done New false
15197 4900 Acked 5899 Read 4 Done Done New false
15198 4901 Acked 5900 Read 9 Done Done New false
15199 4902 Acked 5901 Read 8 Done Done New false
15200 4903 Acked 5902 Write 2 Done Done New false
15201 4904 Acked 5903 Read 2 Done Done New false
15202 4905 Acked 5904 Write 3 Done Done New false
15203 4906 Acked 5905 Flush 0 Done Done New false
15204 4907 Acked 5906 Read 10 Done Done New false
15205 4908 Acked 5907 Read 2 Done Done New false
15206 4909 Acked 5908 Read 1 Done Done New false
15207 4910 Acked 5909 Read 3 Done Done New false
15208 4911 Acked 5910 Read 4 Done Done New false
15209 4912 Acked 5911 Write 6 Done Done New false
15210 4913 Acked 5912 Read 7 Done Done New false
15211 4914 Acked 5913 Read 7 Done Done New false
15212 4915 Acked 5914 Write 8 Done Done New false
15213 4916 Acked 5915 Read 10 Done Done New false
15214 4917 Acked 5916 Read 6 Done Done New false
15215 4918 Acked 5917 Write 10 Done Done New false
15216 4919 Acked 5918 Write 1 Done Done New false
15217 4920 Acked 5919 Read 6 Done Done New false
15218 4921 Acked 5920 Write 9 Done Done New false
15219 4922 Acked 5921 Write 8 Done Done New false
15220 4923 Acked 5922 Read 10 Done Done New false
15221 4924 Acked 5923 Read 8 Done Done New false
15222 4925 Acked 5924 Write 2 Done Done New false
15223 4926 Acked 5925 Read 7 Done Done New false
15224 4927 Acked 5926 Write 6 Done Done New false
15225 4928 Acked 5927 Read 9 Done Done New false
15226 4929 Acked 5928 Write 5 Done Done New false
15227 4930 Acked 5929 Read 9 Done Done New false
15228 4931 Acked 5930 Read 4 Done Done New false
15229 4932 Acked 5931 Write 3 Done Done New false
15230 4933 Acked 5932 Write 10 Done Done New false
15231 4934 Acked 5933 Read 10 Done Done New false
15232 4935 Acked 5934 Write 10 Done Done New false
15233 4936 Acked 5935 Read 5 Done Done New false
15234 4937 Acked 5936 Read 9 Done Done New false
15235 4938 Acked 5937 Write 8 Done Done New false
15236 4939 Acked 5938 Write 9 Done Done New false
15237 4940 Acked 5939 Write 6 Done Done New false
15238 4941 Acked 5940 Read 8 Done Done New false
15239 4942 Acked 5941 Write 7 Done Done New false
15240 4943 Acked 5942 Flush 0 Done Done New false
15241 4944 Acked 5943 Write 1 Done Done New false
15242 4945 Acked 5944 Write 1 Done Done New false
15243 4946 Acked 5945 Write 6 Done Done New false
15244 4947 Acked 5946 Read 6 Done Done New false
15245 4948 Acked 5947 Read 8 Done Done New false
15246 4949 Acked 5948 Read 3 Done Done New false
15247 4950 Acked 5949 Read 1 Done Done New false
15248 4951 Acked 5950 Flush 0 Done Done New false
15249 4952 Acked 5951 Read 5 Done Done New false
15250 4953 Acked 5952 Read 6 Done Done New false
15251 4954 Acked 5953 Write 5 Done Done New false
15252 4955 Acked 5954 Read 2 Done Done New false
15253 4956 Acked 5955 Write 2 Done Done New false
15254 4957 Acked 5956 Read 4 Done Done New false
15255 4958 Acked 5957 Read 1 Done Done New false
15256 4959 Acked 5958 Read 10 Done Done New false
15257 4960 Acked 5959 Read 1 Done Done New false
15258 4961 Acked 5960 Write 4 Done Done New false
15259 4962 Acked 5961 Read 4 Done Done New false
15260 4963 Acked 5962 Read 2 Done Done New false
15261 4964 Acked 5963 Flush 0 Done Done New false
15262 4965 Acked 5964 Read 3 Done Done New false
15263 4966 Acked 5965 Read 3 Done Done New false
15264 4967 Acked 5966 Write 2 Done Done New false
15265 4968 Acked 5967 Read 7 Done Done New false
15266 4969 Acked 5968 Write 3 Done Done New false
15267 4970 Acked 5969 Read 2 Done Done New false
15268 4971 Acked 5970 Read 1 Done Done New false
15269 4972 Acked 5971 Write 10 Done Done New false
15270 4973 Acked 5972 Write 7 Done Done New false
15271 4974 Acked 5973 Write 6 Done Done New false
15272 4975 Acked 5974 Read 9 Done Done New false
15273 4976 Acked 5975 Read 7 Done Done New false
15274 4977 Acked 5976 Write 7 Done Done New false
15275 4978 Acked 5977 Read 6 Done Done New false
15276 4979 Acked 5978 Read 10 Done Done New false
15277 4980 Acked 5979 Read 10 Done Done New false
15278 4981 Acked 5980 Read 5 Done Done New false
15279 4982 Acked 5981 Write 5 Done Done New false
15280 4983 Acked 5982 Read 10 Done Done New false
15281 4984 Acked 5983 Read 2 Done Done New false
15282 4985 Acked 5984 Read 1 Done Done New false
15283 4986 Acked 5985 Write 8 Done Done New false
15284 4987 Acked 5986 Write 10 Done Done New false
15285 4988 Acked 5987 Read 4 Done Done New false
15286 4989 Acked 5988 Read 4 Done Done New false
15287 4990 Acked 5989 Flush 0 Done Done New false
15288 4991 Acked 5990 Write 5 Done Done New false
15289 4992 Acked 5991 Read 5 Done Done New false
15290 4993 Acked 5992 Write 7 Done Done New false
15291 4994 Acked 5993 Read 5 Done Done New false
15292 4995 Acked 5994 Read 10 Done Done New false
15293 4996 Acked 5995 Write 6 Done Done New false
15294 4997 Acked 5996 Flush 0 Done Done New false
15295 4998 Acked 5997 Write 6 Done Done New false
15296 4999 Acked 5998 Read 2 Done Done New false
15297 5000 Acked 5999 Write 7 Done Done New false
15298 5001 Acked 6000 Write 7 Done Done New false
15299 5002 Acked 6001 Read 10 Done Done New false
15300 5003 Acked 6002 Write 10 Done Done New false
15301 5004 Acked 6003 Read 4 Done Done New false
15302 5005 Acked 6004 Read 7 Done Done New false
15303 5006 Acked 6005 Write 3 Done Done New false
15304 5007 Acked 6006 Write 8 Done Done New false
15305 5008 Acked 6007 Read 7 Done Done New false
15306 5009 Acked 6008 Write 7 Done Done New false
15307 5010 Acked 6009 Flush 0 Done Done New false
15308 5011 Acked 6010 Read 5 Done Done New false
15309 5012 Acked 6011 Write 5 Done Done New false
15310 5013 Acked 6012 Write 1 Done Done New false
15311 5014 Acked 6013 Read 1 Done Done New false
15312 5015 Acked 6014 Flush 0 Done Done New false
15313 5016 Acked 6015 Read 7 Done Done New false
15314 5017 Acked 6016 Write 4 Done Done New false
15315 5018 Acked 6017 Read 4 Done Done New false
15316 5019 Acked 6018 Read 9 Done Done New false
15317 5020 Acked 6019 Write 4 Done Done New false
15318 5021 Acked 6020 Read 9 Done Done New false
15319 5022 Acked 6021 Write 5 Done Done New false
15320 5023 Acked 6022 Flush 0 Done Done New false
15321 5024 Acked 6023 Read 6 Done Done New false
15322 5025 Acked 6024 Write 7 Done Done New false
15323 5026 Acked 6025 Read 10 Done Done New false
15324 5027 Acked 6026 Read 10 Done Done New false
15325 5028 Acked 6027 Read 7 Done Done New false
15326 5029 Acked 6028 Write 5 Done Done New false
15327 5030 Acked 6029 Read 7 Done Done New false
15328 5031 Acked 6030 Read 9 Done Done New false
15329 5032 Acked 6031 Read 10 Done Done New false
15330 5033 Acked 6032 Write 8 Done Done New false
15331 5034 Acked 6033 Flush 0 Done Done New false
15332 5035 Acked 6034 Flush 0 Done Done New false
15333 5036 Acked 6035 Read 3 Done Done New false
15334 5037 Acked 6036 Read 8 Done Done New false
15335 5038 Acked 6037 Read 7 Done Done New false
15336 5039 Acked 6038 Read 5 Done Done New false
15337 5040 Acked 6039 Write 8 Done Done New false
15338 5041 Acked 6040 Read 10 Done Done New false
15339 5042 Acked 6041 Read 5 Done Done New false
15340 5043 Acked 6042 Write 6 Done Done New false
15341 5044 Acked 6043 Read 4 Done Done New false
15342 5045 Acked 6044 Write 1 Done Done New false
15343 5046 Acked 6045 Write 3 Done Done New false
15344 5047 Acked 6046 Write 1 Done Done New false
15345 5048 Acked 6047 Read 2 Done Done New false
15346 5049 Acked 6048 Write 9 Done Done New false
15347 5050 Acked 6049 Read 5 Done Done New false
15348 5051 Acked 6050 Write 10 Done Done New false
15349 5052 Acked 6051 Write 4 Done Done New false
15350 5053 Acked 6052 Flush 0 Done Done New false
15351 5054 Acked 6053 Read 4 Done Done New false
15352 5055 Acked 6054 Flush 0 Done Done New false
15353 5056 Acked 6055 Read 8 Done Done New false
15354 5057 Acked 6056 Read 4 Done Done New false
15355 5058 Acked 6057 Flush 0 Done Done New false
15356 5059 Acked 6058 Flush 0 Done Done New false
15357 5060 Acked 6059 Read 5 Done Done New false
15358 5061 Acked 6060 Flush 0 Done Done New false
15359 5062 Acked 6061 Write 4 Done Done New false
15360 5063 Acked 6062 Read 9 Done Done New false
15361 5064 Acked 6063 Write 10 Done Done New false
15362 5065 Acked 6064 Write 7 Done Done New false
15363 5066 Acked 6065 Read 1 Done Done New false
15364 5067 Acked 6066 Write 9 Done Done New false
15365 5068 Acked 6067 Read 8 Done Done New false
15366 5069 Acked 6068 Write 1 Done Done New false
15367 5070 Acked 6069 Write 6 Done Done New false
15368 5071 Acked 6070 Write 8 Done Done New false
15369 5072 Acked 6071 Read 2 Done Done New false
15370 5073 Acked 6072 Flush 0 Done Done New false
15371 5074 Acked 6073 Write 9 Done Done New false
15372 5075 Acked 6074 Flush 0 Done Done New false
15373 5076 Acked 6075 Read 10 Done Done New false
15374 5077 Acked 6076 Read 5 Done Done New false
15375 5078 Acked 6077 Write 1 Done Done New false
15376 5079 Acked 6078 Read 1 Done Done New false
15377 5080 Acked 6079 Flush 0 Done Done New false
15378 5081 Acked 6080 Flush 0 Done Done New false
15379 5082 Acked 6081 Read 10 Done Done New false
15380 5083 Acked 6082 Write 8 Done Done New false
15381 5084 Acked 6083 Read 6 Done Done New false
15382 5085 Acked 6084 Read 4 Done Done New false
15383 5086 Acked 6085 Write 10 Done Done New false
15384 5087 Acked 6086 Write 2 Done Done New false
15385 5088 Acked 6087 Write 5 Done Done New false
15386 5089 Acked 6088 Flush 0 Done Done New false
15387 5090 Acked 6089 Write 3 Done Done New false
15388 5091 Acked 6090 Write 10 Done Done New false
15389 5092 Acked 6091 Read 3 Done Done New false
15390 5093 Acked 6092 Read 2 Done Done New false
15391 5094 Acked 6093 Read 6 Done Done New false
15392 5095 Acked 6094 Read 3 Done Done New false
15393 5096 Acked 6095 Read 6 Done Done New false
15394 5097 Acked 6096 Read 4 Done Done New false
15395 5098 Acked 6097 Write 6 Done Done New false
15396 5099 Acked 6098 Write 10 Done Done New false
15397 5100 Acked 6099 Write 8 Done Done New false
15398 5101 Acked 6100 Read 8 Done Done New false
15399 5102 Acked 6101 Read 10 Done Done New false
15400 5103 Acked 6102 Write 1 Done Done New false
15401 5104 Acked 6103 Read 5 Done Done New false
15402 5105 Acked 6104 Write 5 Done Done New false
15403 5106 Acked 6105 Flush 0 Done Done New false
15404 5107 Acked 6106 Read 4 Done Done New false
15405 5108 Acked 6107 Read 2 Done Done New false
15406 5109 Acked 6108 Read 6 Done Done New false
15407 5110 Acked 6109 Read 9 Done Done New false
15408 5111 Acked 6110 Write 2 Done Done New false
15409 5112 Acked 6111 Read 5 Done Done New false
15410 5113 Acked 6112 Read 8 Done Done New false
15411 5114 Acked 6113 Write 5 Done Done New false
15412 5115 Acked 6114 Read 3 Done Done New false
15413 5116 Acked 6115 Write 10 Done Done New false
15414 5117 Acked 6116 Read 6 Done Done New false
15415 5118 Acked 6117 Read 5 Done Done New false
15416 5119 Acked 6118 Write 10 Done Done New false
15417 5120 Acked 6119 Read 10 Done Done New false
15418 5121 Acked 6120 Read 1 Done Done New false
15419 5122 Acked 6121 Read 9 Done Done New false
15420 5123 Acked 6122 Read 7 Done Done New false
15421 STATES DS:0 DS:1 DS:2 TOTAL
15422 New 0 0 301 301
15423 Sent 0 0 0 0
15424 Done 301 301 0 602
15425 Skipped 0 0 0 0
15426 Error 0 0 0 0
15427 Last Flush: 6105 6105 5821
15428 Downstairs last five completed: 5821 5820 5819 5818 5817
15429 Upstairs last five completed: 5123 5122 5121 5120 5119
15430 CLIENT: Up:0 ds:301 act:2
154312024-02-27T21:47:40.608ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154322024-02-27T21:47:40.608ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154332024-02-27T21:47:40.608ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154342024-02-27T21:47:40.609ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
154352024-02-27T21:47:40.609ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154362024-02-27T21:47:40.609ZINFOcrucible: [2] client re-new 302 jobs since flush 5821 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
154372024-02-27T21:47:40.609ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154382024-02-27T21:47:41.610ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154392024-02-27T21:47:41.610ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154402024-02-27T21:47:41.610ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154412024-02-27T21:47:41.610ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
154422024-02-27T21:47:41.610ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154432024-02-27T21:47:41.610ZINFOcrucible: [2] client re-new 302 jobs since flush 5821 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
154442024-02-27T21:47:41.610ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
15445 [8830] got start action
15446 [8830] I am going to restart
15447 Starting downstairs at port 8830
15448 Make output file at "/tmp/dsc/downstairs-8830.txt"
15449 [8830][2] reports Starting
15450 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1184)
15451 [8830][2] reports Running
154522024-02-27T21:47:42.611ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154532024-02-27T21:47:42.611ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154542024-02-27T21:47:42.613ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154552024-02-27T21:47:42.613ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154562024-02-27T21:47:42.613ZINFOcrucible: send last flush ID to this DS: 5821 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154572024-02-27T21:47:42.613ZINFOcrucible: Replied this last flush ID: 5821 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154582024-02-27T21:47:42.613ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154592024-02-27T21:47:42.613ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154602024-02-27T21:47:42.613ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154612024-02-27T21:47:42.613ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
15462 ----------------------------------------------------------------
15463 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
15464 Downstairs last five completed: 6123 6122 6121 6120 6119
15465 Upstairs last five completed: 5124 5123 5122 5121 5120
15466 CLIENT: Up:0 ds:0 act:3
15467 Replay: All jobs finished, all DS active.
15468 Main task has work to do, go find it
15469 got dsc Stop(2)
15470 stop 2
15471 Feb 27 21:47:43.963 INFO request completed, latency_us: 321, response_code: [8830] Got stop action so:false kr:false
15472 204, uri: /stop/cid/2, method: POST, req_id: 2024f7db-ec59-4a38-b7b4-2769f81fc983, remote_addr: 127.0.0.1:44347, local_addr: 127.0.0.1:9998
15473 Feb 27 21:47:43.963 INFO request completed, latency_us: 107, response_code: 200, uri: /state/cid/2, method: GET, req_id: ac80d1af-0b29-4897-b408-9cd028d2255b, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
154742024-02-27T21:47:43.963ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154752024-02-27T21:47:43.964ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154762024-02-27T21:47:43.964ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = e6a5a974-925a-494f-8845-facc441c2690
154772024-02-27T21:47:43.964ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154782024-02-27T21:47:43.964ZINFOcrucible: [2] client re-new 0 jobs since flush 6123 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
154792024-02-27T21:47:43.964ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
15480 [8830] Exited with: ExitStatus(unix_wait_status(9))
15481 [8830][2] reports Exit
154822024-02-27T21:47:44.965ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154832024-02-27T21:47:44.965ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154842024-02-27T21:47:44.965ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154852024-02-27T21:47:44.965ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
154862024-02-27T21:47:44.965ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154872024-02-27T21:47:44.965ZINFOcrucible: [2] client re-new 0 jobs since flush 6123 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
154882024-02-27T21:47:44.965ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154892024-02-27T21:47:45.966ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154902024-02-27T21:47:45.967ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154912024-02-27T21:47:45.967ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154922024-02-27T21:47:45.967ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
154932024-02-27T21:47:45.967ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154942024-02-27T21:47:45.967ZINFOcrucible: [2] client re-new 0 jobs since flush 6123 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
154952024-02-27T21:47:45.967ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154962024-02-27T21:47:46.968ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154972024-02-27T21:47:46.968ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154982024-02-27T21:47:46.968ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
154992024-02-27T21:47:46.968ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
155002024-02-27T21:47:46.968ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
155012024-02-27T21:47:46.968ZINFOcrucible: [2] client re-new 0 jobs since flush 6123 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
155022024-02-27T21:47:46.968ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
15503 Feb 27 21:47:47.966 INFO request completed, latency_us: 249, response_code: 200, uri: /state/cid/2, method: GET, req_id: f4a823bc-9d63-45bd-9947-4753cd3ba0a5, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
15504 001/300 Read block 3151 len 28672
15505 002/300 Write block 662 len 20480 data: 3 4 3 5 5
15506 003/300 Read block 4502 len 8192
155072024-02-27T21:47:47.969ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
155082024-02-27T21:47:47.969ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
155092024-02-27T21:47:47.969ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
155102024-02-27T21:47:47.969ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
155112024-02-27T21:47:47.969ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
155122024-02-27T21:47:47.969ZINFOcrucible: [2] client re-new 3 jobs since flush 6123 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
155132024-02-27T21:47:47.969ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
15514 004/300 Write block 1563 len 32768 data: 3 3 3 4 3 3 4 4
15515 005/300 Write block 2691 len 12288 data: 3 5 6
15516 006/300 Write block 2816 len 40960 data: 8 7 8 8 7 6 3 4 4 4
15517 007/300 Read block 1999 len 28672
15518 008/300 Write block 2158 len 16384 data: 6 6 7 5
15519 009/300 Read block 4088 len 24576
15520 010/300 Flush
15521 011/300 Write block 4229 len 32768 data: 2 2 2 2 2 3 3 2
15522 012/300 Write block 4000 len 28672 data: 3 3 3 2 2 2 2
15523 013/300 Flush
15524 014/300 Read block 2607 len 24576
15525 015/300 Read block 140 len 40960
15526 016/300 Read block 4241 len 8192
15527 017/300 Read block 2881 len 36864
15528 018/300 Write block 3226 len 4096 data: 5
15529 019/300 Read block 1041 len 40960
15530 020/300 Write block 1325 len 36864 data: 6 6 4 4 4 5 4 4 4
15531 021/300 Read block 2604 len 16384
15532 022/300 Flush
15533 023/300 Write block 2238 len 24576 data: 4 4 4 5 5 5
15534 024/300 Read block 2808 len 40960
15535 025/300 Read block 4833 len 36864
15536 026/300 Read block 2338 len 12288
15537 027/300 Read block 1920 len 8192
15538 028/300 Write block 3916 len 8192 data: 3 3
15539 029/300 Write block 4068 len 32768 data: 3 4 4 4 4 4 4 6
15540 030/300 Read block 168 len 40960
15541 031/300 Write block 243 len 20480 data: 3 3 3 3 3
15542 032/300 Write block 1645 len 28672 data: 6 7 6 4 5 4 3
15543 033/300 Read block 2325 len 8192
15544 034/300 Write block 822 len 12288 data: 4 5 6
15545 035/300 Write block 1265 len 16384 data: 4 4 5 6
15546 036/300 Flush
15547 037/300 Read block 98 len 28672
15548 038/300 Flush
15549 039/300 Read block 470 len 24576
15550 040/300 Read block 2260 len 4096
15551 041/300 Flush
15552 042/300 Write block 3301 len 4096 data: 6
15553 043/300 Read block 788 len 28672
15554 044/300 Write block 2416 len 8192 data: 2 2
15555 045/300 Read block 3635 len 8192
15556 046/300 Read block 4427 len 24576
15557 047/300 Read block 3693 len 4096
15558 048/300 Write block 4489 len 4096 data: 6
15559 049/300 Write block 4750 len 16384 data: 7 4 5 4
15560 050/300 Write block 1989 len 16384 data: 3 4 4 4
15561 051/300 Read block 4584 len 4096
15562 052/300 Flush
15563 053/300 Read block 1268 len 8192
15564 054/300 Read block 3779 len 40960
15565 055/300 Write block 44 len 28672 data: 6 6 5 4 5 5 5
15566 056/300 Flush
15567 057/300 Write block 3405 len 28672 data: 3 3 4 4 5 7 7
15568 058/300 Read block 2877 len 28672
15569 059/300 Read block 4296 len 4096
15570 060/300 Read block 554 len 24576
15571 061/300 Write block 173 len 32768 data: 3 2 2 3 3 4 4 5
15572 062/300 Read block 3391 len 24576
15573 063/300 Read block 1874 len 4096
15574 064/300 Write block 3636 len 36864 data: 4 5 5 5 4 5 4 3 3
15575 065/300 Read block 2180 len 36864
15576 066/300 Read block 804 len 20480
15577 067/300 Write block 2119 len 36864 data: 3 3 4 5 6 5 5 5 5
15578 068/300 Read block 2182 len 28672
15579 069/300 Read block 1848 len 12288
15580 070/300 Read block 3978 len 20480
15581 071/300 Read block 11 len 8192
15582 072/300 Read block 2997 len 12288
15583 073/300 Read block 4147 len 28672
15584 074/300 Flush
15585 075/300 Read block 3560 len 28672
15586 076/300 Write block 678 len 40960 data: 7 4 5 4 4 3 3 4 4 6
15587 077/300 Read block 579 len 24576
15588 078/300 Read block 2590 len 36864
15589 079/300 Write block 1710 len 40960 data: 6 6 9 9 6 6 6 6 6 4
15590 080/300 Read block 767 len 28672
15591 081/300 Write block 2657 len 8192 data: 6 6
15592 082/300 Write block 4125 len 12288 data: 3 2 3
15593 083/300 Write block 4106 len 8192 data: 2 2
15594 084/300 Read block 2283 len 12288
15595 085/300 Write block 3838 len 32768 data: 4 5 6 6 7 7 7 5
15596 086/300 Write block 2298 len 40960 data: 3 3 3 3 3 3 3 3 2 2
15597 087/300 Read block 2008 len 20480
15598 088/300 Write block 1681 len 8192 data: 2 3
15599 089/300 Read block 2945 len 8192
15600 090/300 Write block 2014 len 8192 data: 3 3
15601 091/300 Read block 4362 len 28672
15602 092/300 Read block 4912 len 32768
15603 093/300 Read block 1348 len 20480
15604 094/300 Read block 3438 len 4096
15605 095/300 Write block 1827 len 4096 data: 5
15606 096/300 Write block 4945 len 32768 data: 6 5 4 5 5 4 5 5
15607 097/300 Write block 2182 len 36864 data: 3 3 3 2 3 3 3 4 4
15608 098/300 Read block 4170 len 36864
15609 099/300 Flush
15610 100/300 Read block 3062 len 36864
15611 101/300 Read block 2916 len 24576
15612 102/300 Write block 1519 len 16384 data: 2 3 3 3
15613 103/300 Write block 2447 len 20480 data: 6 5 4 5 5
15614 104/300 Read block 3090 len 40960
15615 105/300 Write block 4576 len 8192 data: 5 7
15616 106/300 Write block 1194 len 12288 data: 4 4 5
15617 107/300 Read block 3244 len 4096
15618 108/300 Read block 99 len 28672
15619 109/300 Read block 1390 len 20480
15620 110/300 Write block 927 len 8192 data: 3 3
15621 111/300 Read block 2961 len 4096
15622 112/300 Read block 1217 len 8192
15623 113/300 Write block 4766 len 24576 data: 5 4 3 4 5 5
15624 114/300 Read block 2253 len 40960
15625 115/300 Write block 831 len 16384 data: 5 5 3 3
15626 116/300 Write block 482 len 8192 data: 2 3
15627 117/300 Read block 1210 len 8192
15628 118/300 Read block 392 len 8192
15629 119/300 Write block 1724 len 32768 data: 4 3 2 2 2 2 2 2
15630 120/300 Write block 1090 len 12288 data: 6 7 6
15631 121/300 Read block 900 len 4096
15632 122/300 Write block 4656 len 4096 data: 6
15633 123/300 Write block 4349 len 24576 data: 4 4 4 5 4 3
15634 124/300 Read block 4801 len 16384
15635 125/300 Write block 4245 len 16384 data: 2 2 3 4
15636 126/300 Read block 3935 len 32768
15637 127/300 Read block 4345 len 36864
15638 128/300 Write block 2229 len 12288 data: 4 3 4
15639 129/300 Read block 1306 len 32768
15640 130/300 Read block 1310 len 24576
15641 131/300 Read block 3374 len 8192
15642 132/300 Read block 3361 len 24576
15643 133/300 Read block 2745 len 16384
15644 134/300 Read block 994 len 16384
15645 135/300 Write block 171 len 40960 data: 3 4 4 3 3 4 4 5 5 6
15646 136/300 Read block 4005 len 24576
15647 137/300 Read block 4742 len 36864
15648 138/300 Read block 1561 len 16384
15649 139/300 Write block 3530 len 12288 data: 4 5 5
15650 140/300 Read block 851 len 4096
15651 141/300 Write block 3400 len 28672 data: 5 5 5 5 3 4 4
15652 142/300 Read block 1483 len 12288
15653 143/300 Write block 393 len 32768 data: 4 5 5 5 5 5 6 7
15654 144/300 Flush
15655 145/300 Write block 3917 len 8192 data: 4 3
15656 146/300 Read block 1582 len 16384
15657 147/300 Write block 3291 len 24576 data: 2 2 2 2 2 3
15658 148/300 Read block 3366 len 12288
15659 149/300 Flush
15660 150/300 Read block 4380 len 4096
15661 151/300 Read block 3934 len 16384
15662 152/300 Write block 3254 len 16384 data: 4 6 5 6
15663 153/300 Write block 2392 len 40960 data: 3 2 3 3 3 3 3 2 3 3
15664 154/300 Read block 3951 len 12288
15665 155/300 Read block 906 len 16384
15666 156/300 Read block 320 len 36864
15667 157/300 Write block 711 len 4096 data: 4
15668 158/300 Read block 1515 len 20480
15669 159/300 Read block 3836 len 20480
15670 160/300 Read block 4336 len 36864
15671 161/300 Read block 3921 len 24576
15672 162/300 Write block 3513 len 20480 data: 3 4 4 5 5
15673 163/300 Read block 3879 len 36864
15674 164/300 Flush
15675 165/300 Read block 3237 len 36864
15676 166/300 Write block 113 len 32768 data: 4 5 5 4 5 6 6 6
15677 167/300 Read block 4997 len 12288
15678 168/300 Write block 4278 len 8192 data: 2 2
15679 169/300 Read block 3928 len 20480
15680 170/300 Read block 4599 len 8192
15681 171/300 Read block 3934 len 40960
15682 172/300 Write block 3247 len 36864 data: 3 2 3 3 3 3 3 5 7
15683 173/300 Write block 451 len 28672 data: 5 4 3 4 5 4 4
15684 174/300 Read block 3781 len 8192
15685 175/300 Read block 3275 len 32768
15686 176/300 Write block 1174 len 16384 data: 4 3 4 3
15687 177/300 Read block 2085 len 32768
15688 178/300 Write block 3120 len 36864 data: 4 4 4 4 4 4 3 3 3
15689 179/300 Write block 2996 len 8192 data: 4 4
15690 180/300 Write block 2503 len 36864 data: 8 9 9 9 10 10 6 3 3
15691 181/300 Read block 3374 len 32768
15692 182/300 Write block 4814 len 28672 data: 6 6 5 5 5 5 4
15693 183/300 Read block 476 len 32768
15694 184/300 Write block 3613 len 40960 data: 4 4 4 4 3 2 2 3 3 3
15695 185/300 Read block 3603 len 16384
15696 186/300 Write block 4095 len 36864 data: 5 5 4 4 4 4 3 3 4
15697 187/300 Read block 3110 len 4096
15698 188/300 Read block 4388 len 28672
15699 189/300 Flush
15700 190/300 Read block 547 len 32768
15701 191/300 Read block 2698 len 36864
15702 192/300 Write block 4873 len 16384 data: 4 4 4 4
15703 193/300 Write block 188 len 12288 data: 3 3 3
15704 194/300 Read block 4584 len 20480
15705 195/300 Read block 719 len 8192
15706 196/300 Write block 2931 len 36864 data: 3 3 3 3 3 3 3 4 3
15707 197/300 Write block 1696 len 32768 data: 4 4 4 4 3 3 4 5
15708 198/300 Read block 3985 len 36864
15709 199/300 Read block 684 len 20480
15710 200/300 Read block 3824 len 12288
15711 201/300 Write block 3890 len 36864 data: 4 5 2 2 2 4 4 3 3
15712 202/300 Read block 334 len 28672
15713 203/300 Flush
15714 204/300 Write block 4918 len 28672 data: 4 3 2 3 3 3 3
15715 205/300 Read block 2648 len 40960
15716 206/300 Read block 216 len 16384
15717 207/300 Read block 1437 len 8192
15718 208/300 Read block 1995 len 12288
15719 209/300 Write block 4627 len 8192 data: 5 6
15720 210/300 Read block 1144 len 24576
15721 211/300 Write block 2782 len 8192 data: 6 3
15722 212/300 Read block 3201 len 20480
15723 213/300 Write block 2394 len 16384 data: 4 4 4 4
15724 214/300 Read block 1573 len 20480
15725 215/300 Write block 3691 len 16384 data: 2 3 4 5
15726 216/300 Read block 1420 len 40960
15727 217/300 Read block 4882 len 24576
15728 218/300 Write block 3258 len 32768 data: 6 5 3 3 2 3 3 4
15729 219/300 Read block 2071 len 32768
15730 220/300 Read block 289 len 24576
15731 221/300 Write block 1583 len 40960 data: 4 5 5 4 3 2 2 2 3 3
15732 222/300 Read block 4764 len 16384
15733 223/300 Read block 1297 len 40960
15734 224/300 Flush
15735 225/300 Write block 1781 len 16384 data: 2 2 3 3
15736 226/300 Read block 3261 len 16384
15737 227/300 Read block 1782 len 8192
15738 228/300 Write block 902 len 32768 data: 3 2 2 3 4 5 5 6
15739 229/300 Read block 214 len 28672
15740 230/300 Flush
15741 231/300 Read block 4355 len 28672
15742 232/300 Read block 800 len 28672
15743 233/300 Read block 811 len 24576
15744 234/300 Read block 2978 len 28672
15745 235/300 Flush
15746 236/300 Flush
15747 237/300 Read block 2147 len 16384
15748 238/300 Flush
15749 239/300 Read block 4204 len 20480
15750 240/300 Flush
15751 241/300 Write block 3903 len 36864 data: 5 4 4 5 4 4 3 2 2
15752 242/300 Read block 3575 len 28672
15753 243/300 Read block 2312 len 16384
15754 244/300 Read block 4542 len 40960
15755 245/300 Flush
15756 246/300 Read block 914 len 32768
15757 247/300 Write block 3736 len 28672 data: 2 2 2 3 4 3 3
15758 248/300 Read block 393 len 16384
15759 249/300 Flush
15760 250/300 Flush
15761 251/300 Read block 1693 len 20480
15762 252/300 Write block 69 len 12288 data: 5 4 5
15763 253/300 Read block 709 len 16384
15764 254/300 Read block 4459 len 24576
15765 255/300 Write block 1143 len 32768 data: 7 6 6 6 5 4 3 3
15766 256/300 Read block 1666 len 40960
15767 257/300 Read block 4088 len 40960
15768 258/300 Read block 2368 len 32768
15769 259/300 Read block 1393 len 24576
15770 260/300 Write block 265 len 32768 data: 3 4 5 6 5 5 5 5
15771 261/300 Read block 3924 len 8192
15772 262/300 Flush
15773 263/300 Write block 3382 len 4096 data: 3
15774 264/300 Flush
15775 265/300 Read block 4864 len 28672
15776 266/300 Write block 4287 len 20480 data: 2 2 2 2 2
15777 267/300 Write block 2562 len 4096 data: 3
15778 268/300 Flush
15779 269/300 Write block 631 len 20480 data: 3 2 3 4 4
15780 270/300 Write block 2130 len 40960 data: 3 3 3 2 3 3 3 5 5 6
15781 271/300 Flush
15782 272/300 Read block 4099 len 8192
15783 273/300 Read block 3274 len 20480
15784 274/300 Read block 4228 len 20480
15785 275/300 Read block 2451 len 12288
15786 276/300 Write block 3591 len 20480 data: 5 7 5 4 5
15787 277/300 Read block 2659 len 32768
15788 278/300 Read block 1099 len 8192
15789 279/300 Read block 586 len 20480
15790 280/300 Flush
15791 281/300 Write block 327 len 28672 data: 4 5 6 5 5 3 4
15792 282/300 Read block 2945 len 32768
15793 283/300 Read block 1686 len 20480
15794 284/300 Flush
15795 285/300 Write block 779 len 24576 data: 2 3 3 4 3 4
15796 286/300 Write block 2820 len 24576 data: 8 7 4 5 5 5
15797 287/300 Flush
15798 288/300 Write block 3169 len 12288 data: 5 5 5
15799 289/300 Read block 374 len 12288
15800 290/300 Read block 4503 len 36864
15801 291/300 Write block 4655 len 16384 data: 6 7 5 4
15802 292/300 Write block 4488 len 36864 data: 6 7 8 6 7 8 7 7 6
15803 293/300 Read block 4188 len 8192
15804 294/300 Write block 507 len 4096 data: 6
15805 295/300 Write block 822 len 28672 data: 5 6 7 6 7 6 6
15806 296/300 Read block 1384 len 28672
15807 297/300 Flush
15808 298/300 Read block 622 len 24576
15809 299/300 Read block 2957 len 4096
15810 300/300 Read block 4934 len 12288
15811 Main task has work to do, go find it
15812 got dsc Start(2)
15813 start 2
15814 Feb 27 21:47:48.335 INFO request completed, latency_us: 228, response_code: 204, uri: /start/cid/2, method: POST, req_id: f40edf68-9639-4b03-937d-03b3a46e38d8, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
15815 Replay: started 2, returned:Ok(())
15816 ----------------------------------------------------------------
15817 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15818 States: Active Active Offline
15819 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
15820 5125 Acked 6124 Read 7 Done Done New true
15821 5126 Acked 6125 Write 5 Done Done New true
15822 5127 Acked 6126 Read 2 Done Done New true
15823 5128 Acked 6127 Write 8 Done Done New false
15824 5129 Acked 6128 Write 3 Done Done New false
15825 5130 Acked 6129 Write 10 Done Done New false
15826 5131 Acked 6130 Read 7 Done Done New false
15827 5132 Acked 6131 Write 4 Done Done New false
15828 5133 Acked 6132 Read 6 Done Done New false
15829 5134 Acked 6133 Flush 0 Done Done New false
15830 5135 Acked 6134 Write 8 Done Done New false
15831 5136 Acked 6135 Write 7 Done Done New false
15832 5137 Acked 6136 Flush 0 Done Done New false
15833 5138 Acked 6137 Read 6 Done Done New false
15834 5139 Acked 6138 Read 10 Done Done New false
15835 5140 Acked 6139 Read 2 Done Done New false
15836 5141 Acked 6140 Read 9 Done Done New false
15837 5142 Acked 6141 Write 1 Done Done New false
15838 5143 Acked 6142 Read 10 Done Done New false
15839 5144 Acked 6143 Write 9 Done Done New false
15840 5145 Acked 6144 Read 4 Done Done New false
15841 5146 Acked 6145 Flush 0 Done Done New false
15842 5147 Acked 6146 Write 6 Done Done New false
15843 5148 Acked 6147 Read 10 Done Done New false
15844 5149 Acked 6148 Read 9 Done Done New false
15845 5150 Acked 6149 Read 3 Done Done New false
15846 5151 Acked 6150 Read 2 Done Done New false
15847 5152 Acked 6151 Write 2 Done Done New false
15848 5153 Acked 6152 Write 8 Done Done New false
15849 5154 Acked 6153 Read 10 Done Done New false
15850 5155 Acked 6154 Write 5 Done Done New false
15851 5156 Acked 6155 Write 7 Done Done New false
15852 5157 Acked 6156 Read 2 Done Done New false
15853 5158 Acked 6157 Write 3 Done Done New false
15854 5159 Acked 6158 Write 4 Done Done New false
15855 5160 Acked 6159 Flush 0 Done Done New false
15856 5161 Acked 6160 Read 7 Done Done New false
15857 5162 Acked 6161 Flush 0 Done Done New false
15858 5163 Acked 6162 Read 6 Done Done New false
15859 5164 Acked 6163 Read 1 Done Done New false
15860 5165 Acked 6164 Flush 0 Done Done New false
15861 5166 Acked 6165 Write 1 Done Done New false
15862 5167 Acked 6166 Read 7 Done Done New false
15863 5168 Acked 6167 Write 2 Done Done New false
15864 5169 Acked 6168 Read 2 Done Done New false
15865 5170 Acked 6169 Read 6 Done Done New false
15866 5171 Acked 6170 Read 1 Done Done New false
15867 5172 Acked 6171 Write 1 Done Done New false
15868 5173 Acked 6172 Write 4 Done Done New false
15869 5174 Acked 6173 Write 4 Done Done New false
15870 5175 Acked 6174 Read 1 Done Done New false
15871 5176 Acked 6175 Flush 0 Done Done New false
15872 5177 Acked 6176 Read 2 Done Done New false
15873 5178 Acked 6177 Read 10 Done Done New false
15874 5179 Acked 6178 Write 7 Done Done New false
15875 5180 Acked 6179 Flush 0 Done Done New false
15876 5181 Acked 6180 Write 7 Done Done New false
15877 5182 Acked 6181 Read 7 Done Done New false
15878 5183 Acked 6182 Read 1 Done Done New false
15879 5184 Acked 6183 Read 6 Done Done New false
15880 5185 Acked 6184 Write 8 Done Done New false
15881 5186 Acked 6185 Read 6 Done Done New false
15882 5187 Acked 6186 Read 1 Done Done New false
15883 5188 Acked 6187 Write 9 Done Done New false
15884 5189 Acked 6188 Read 9 Done Done New false
15885 5190 Acked 6189 Read 5 Done Done New false
15886 5191 Acked 6190 Write 9 Done Done New false
15887 5192 Acked 6191 Read 7 Done Done New false
15888 5193 Acked 6192 Read 3 Done Done New false
15889 5194 Acked 6193 Read 5 Done Done New false
15890 5195 Acked 6194 Read 2 Done Done New false
15891 5196 Acked 6195 Read 3 Done Done New false
15892 5197 Acked 6196 Read 7 Done Done New false
15893 5198 Acked 6197 Flush 0 Done Done New false
15894 5199 Acked 6198 Read 7 Done Done New false
15895 5200 Acked 6199 Write 10 Done Done New false
15896 5201 Acked 6200 Read 6 Done Done New false
15897 5202 Acked 6201 Read 9 Done Done New false
15898 5203 Acked 6202 Write 10 Done Done New false
15899 5204 Acked 6203 Read 7 Done Done New false
15900 5205 Acked 6204 Write 2 Done Done New false
15901 5206 Acked 6205 Write 3 Done Done New false
15902 5207 Acked 6206 Write 2 Done Done New false
15903 5208 Acked 6207 Read 3 Done Done New false
15904 5209 Acked 6208 Write 8 Done Done New false
15905 5210 Acked 6209 Write 10 Done Done New false
15906 5211 Acked 6210 Read 5 Done Done New false
15907 5212 Acked 6211 Write 2 Done Done New false
15908 5213 Acked 6212 Read 2 Done Done New false
15909 5214 Acked 6213 Write 2 Done Done New false
15910 5215 Acked 6214 Read 7 Done Done New false
15911 5216 Acked 6215 Read 8 Done Done New false
15912 5217 Acked 6216 Read 5 Done Done New false
15913 5218 Acked 6217 Read 1 Done Done New false
15914 5219 Acked 6218 Write 1 Done Done New false
15915 5220 Acked 6219 Write 8 Done Done New false
15916 5221 Acked 6220 Write 9 Done Done New false
15917 5222 Acked 6221 Read 9 Done Done New false
15918 5223 Acked 6222 Flush 0 Done Done New false
15919 5224 Acked 6223 Read 9 Done Done New false
15920 5225 Acked 6224 Read 6 Done Done New false
15921 5226 Acked 6225 Write 4 Done Done New false
15922 5227 Acked 6226 Write 5 Done Done New false
15923 5228 Acked 6227 Read 10 Done Done New false
15924 5229 Acked 6228 Write 2 Done Done New false
15925 5230 Acked 6229 Write 3 Done Done New false
15926 5231 Acked 6230 Read 1 Done Done New false
15927 5232 Acked 6231 Read 7 Done Done New false
15928 5233 Acked 6232 Read 5 Done Done New false
15929 5234 Acked 6233 Write 2 Done Done New false
15930 5235 Acked 6234 Read 1 Done Done New false
15931 5236 Acked 6235 Read 2 Done Done New false
15932 5237 Acked 6236 Write 6 Done Done New false
15933 5238 Acked 6237 Read 10 Done Done New false
15934 5239 Acked 6238 Write 4 Done Done New false
15935 5240 Acked 6239 Write 2 Done Done New false
15936 5241 Acked 6240 Read 2 Done Done New false
15937 5242 Acked 6241 Read 2 Done Done New false
15938 5243 Acked 6242 Write 8 Done Done New false
15939 5244 Acked 6243 Write 3 Done Done New false
15940 5245 Acked 6244 Read 1 Done Done New false
15941 5246 Acked 6245 Write 1 Done Done New false
15942 5247 Acked 6246 Write 6 Done Done New false
15943 5248 Acked 6247 Read 4 Done Done New false
15944 5249 Acked 6248 Write 4 Done Done New false
15945 5250 Acked 6249 Read 8 Done Done New false
15946 5251 Acked 6250 Read 9 Done Done New false
15947 5252 Acked 6251 Write 3 Done Done New false
15948 5253 Acked 6252 Read 8 Done Done New false
15949 5254 Acked 6253 Read 6 Done Done New false
15950 5255 Acked 6254 Read 2 Done Done New false
15951 5256 Acked 6255 Read 6 Done Done New false
15952 5257 Acked 6256 Read 4 Done Done New false
15953 5258 Acked 6257 Read 4 Done Done New false
15954 5259 Acked 6258 Write 10 Done Done New false
15955 5260 Acked 6259 Read 6 Done Done New false
15956 5261 Acked 6260 Read 9 Done Done New false
15957 5262 Acked 6261 Read 4 Done Done New false
15958 5263 Acked 6262 Write 3 Done Done New false
15959 5264 Acked 6263 Read 1 Done Done New false
15960 5265 Acked 6264 Write 7 Done Done New false
15961 5266 Acked 6265 Read 3 Done Done New false
15962 5267 Acked 6266 Write 8 Done Done New false
15963 5268 Acked 6267 Flush 0 Done Done New false
15964 5269 Acked 6268 Write 2 Done Done New false
15965 5270 Acked 6269 Read 4 Done Done New false
15966 5271 Acked 6270 Write 6 Done Done New false
15967 5272 Acked 6271 Read 3 Done Done New false
15968 5273 Acked 6272 Flush 0 Done Done New false
15969 5274 Acked 6273 Read 1 Done Done New false
15970 5275 Acked 6274 Read 4 Done Done New false
15971 5276 Acked 6275 Write 4 Done Done New false
15972 5277 Acked 6276 Write 10 Done Done New false
15973 5278 Acked 6277 Read 3 Done Done New false
15974 5279 Acked 6278 Read 4 Done Done New false
15975 5280 Acked 6279 Read 9 Done Done New false
15976 5281 Acked 6280 Write 1 Done Done New false
15977 5282 Acked 6281 Read 5 Done Done New false
15978 5283 Acked 6282 Read 5 Done Done New false
15979 5284 Acked 6283 Read 9 Done Done New false
15980 5285 Acked 6284 Read 6 Done Done New false
15981 5286 Acked 6285 Write 5 Done Done New false
15982 5287 Acked 6286 Read 9 Done Done New false
15983 5288 Acked 6287 Flush 0 Done Done New false
15984 5289 Acked 6288 Read 9 Done Done New false
15985 5290 Acked 6289 Write 8 Done Done New false
15986 5291 Acked 6290 Read 3 Done Done New false
15987 5292 Acked 6291 Write 2 Done Done New false
15988 5293 Acked 6292 Read 5 Done Done New false
15989 5294 Acked 6293 Read 2 Done Done New false
15990 5295 Acked 6294 Read 10 Done Done New false
15991 5296 Acked 6295 Write 9 Done Done New false
15992 5297 Acked 6296 Write 7 Done Done New false
15993 5298 Acked 6297 Read 2 Done Done New false
15994 5299 Acked 6298 Read 8 Done Done New false
15995 5300 Acked 6299 Write 4 Done Done New false
15996 5301 Acked 6300 Read 8 Done Done New false
15997 5302 Acked 6301 Write 9 Done Done New false
15998 5303 Acked 6302 Write 2 Done Done New false
15999 5304 Acked 6303 Write 9 Done Done New false
16000 5305 Acked 6304 Read 8 Done Done New false
16001 5306 Acked 6305 Write 7 Done Done New false
16002 5307 Acked 6306 Read 8 Done Done New false
16003 5308 Acked 6307 Write 10 Done Done New false
16004 5309 Acked 6308 Read 4 Done Done New false
16005 5310 Acked 6309 Write 9 Done Done New false
16006 5311 Acked 6310 Read 1 Done Done New false
16007 5312 Acked 6311 Read 7 Done Done New false
16008 5313 Acked 6312 Flush 0 Done Done New false
16009 5314 Acked 6313 Read 8 Done Done New false
16010 5315 Acked 6314 Read 9 Done Done New false
16011 5316 Acked 6315 Write 4 Done Done New false
16012 5317 Acked 6316 Write 3 Done Done New false
16013 5318 Acked 6317 Read 5 Done Done New false
16014 5319 Acked 6318 Read 2 Done Done New false
16015 5320 Acked 6319 Write 9 Done Done New false
16016 5321 Acked 6320 Write 8 Done Done New false
16017 5322 Acked 6321 Read 9 Done Done New false
16018 5323 Acked 6322 Read 5 Done Done New false
16019 5324 Acked 6323 Read 3 Done Done New false
16020 5325 Acked 6324 Write 9 Done Done New false
16021 5326 Acked 6325 Read 7 Done Done New false
16022 5327 Acked 6326 Flush 0 Done Done New false
16023 5328 Acked 6327 Write 7 Done Done New false
16024 5329 Acked 6328 Read 10 Done Done New false
16025 5330 Acked 6329 Read 4 Done Done New false
16026 5331 Acked 6330 Read 2 Done Done New false
16027 5332 Acked 6331 Read 3 Done Done New false
16028 5333 Acked 6332 Write 2 Done Done New false
16029 5334 Acked 6333 Read 6 Done Done New false
16030 5335 Acked 6334 Write 2 Done Done New false
16031 5336 Acked 6335 Read 5 Done Done New false
16032 5337 Acked 6336 Write 4 Done Done New false
16033 5338 Acked 6337 Read 5 Done Done New false
16034 5339 Acked 6338 Write 4 Done Done New false
16035 5340 Acked 6339 Read 10 Done Done New false
16036 5341 Acked 6340 Read 6 Done Done New false
16037 5342 Acked 6341 Write 8 Done Done New false
16038 5343 Acked 6342 Read 8 Done Done New false
16039 5344 Acked 6343 Read 6 Done Done New false
16040 5345 Acked 6344 Write 10 Done Done New false
16041 5346 Acked 6345 Read 4 Done Done New false
16042 5347 Acked 6346 Read 10 Done Done New false
16043 5348 Acked 6347 Flush 0 Done Done New false
16044 5349 Acked 6348 Write 4 Done Done New false
16045 5350 Acked 6349 Read 4 Done Done New false
16046 5351 Acked 6350 Read 2 Done Done New false
16047 5352 Acked 6351 Write 8 Done Done New false
16048 5353 Acked 6352 Read 7 Done Done New false
16049 5354 Acked 6353 Flush 0 Done Done New false
16050 5355 Acked 6354 Read 7 Done Done New false
16051 5356 Acked 6355 Read 7 Done Done New false
16052 5357 Acked 6356 Read 6 Done Done New false
16053 5358 Acked 6357 Read 7 Done Done New false
16054 5359 Acked 6358 Flush 0 Done Done New false
16055 5360 Acked 6359 Flush 0 Done Done New false
16056 5361 Acked 6360 Read 4 Done Done New false
16057 5362 Acked 6361 Flush 0 Done Done New false
16058 5363 Acked 6362 Read 5 Done Done New false
16059 5364 Acked 6363 Flush 0 Done Done New false
16060 5365 Acked 6364 Write 9 Done Done New false
16061 5366 Acked 6365 Read 7 Done Done New false
16062 5367 Acked 6366 Read 4 Done Done New false
16063 5368 Acked 6367 Read 10 Done Done New false
16064 5369 Acked 6368 Flush 0 Done Done New false
16065 5370 Acked 6369 Read 8 Done Done New false
16066 5371 Acked 6370 Write 7 Done Done New false
16067 5372 Acked 6371 Read 4 Done Done New false
16068 5373 Acked 6372 Flush 0 Done Done New false
16069 5374 Acked 6373 Flush 0 Done Done New false
16070 5375 Acked 6374 Read 5 Done Done New false
16071 5376 Acked 6375 Write 3 Done Done New false
16072 5377 Acked 6376 Read 4 Done Done New false
16073 5378 Acked 6377 Read 6 Done Done New false
16074 5379 Acked 6378 Write 8 Done Done New false
16075 5380 Acked 6379 Read 10 Done Done New false
16076 5381 Acked 6380 Read 10 Done Done New false
16077 5382 Acked 6381 Read 8 Done Done New false
16078 5383 Acked 6382 Read 6 Done Done New false
16079 5384 Acked 6383 Write 8 Done Done New false
16080 5385 Acked 6384 Read 2 Done Done New false
16081 5386 Acked 6385 Flush 0 Done Done New false
16082 5387 Acked 6386 Write 1 Done Done New false
16083 5388 Acked 6387 Flush 0 Done Done New false
16084 5389 Acked 6388 Read 7 Done Done New false
16085 5390 Acked 6389 Write 5 Done Done New false
16086 5391 Acked 6390 Write 1 Done Done New false
16087 5392 Acked 6391 Flush 0 Done Done New false
16088 5393 Acked 6392 Write 5 Done Done New false
16089 5394 Acked 6393 Write 10 Done Done New false
16090 5395 Acked 6394 Flush 0 Done Done New false
16091 5396 Acked 6395 Read 2 Done Done New false
16092 5397 Acked 6396 Read 5 Done Done New false
16093 5398 Acked 6397 Read 5 Done Done New false
16094 5399 Acked 6398 Read 3 Done Done New false
16095 5400 Acked 6399 Write 5 Done Done New false
16096 5401 Acked 6400 Read 8 Done Done New false
16097 5402 Acked 6401 Read 2 Done Done New false
16098 5403 Acked 6402 Read 5 Done Done New false
16099 5404 Acked 6403 Flush 0 Done Done New false
16100 5405 Acked 6404 Write 7 Done Done New false
16101 5406 Acked 6405 Read 8 Done Done New false
16102 5407 Acked 6406 Flush 0 Done Done New false
16103 5408 Acked 6407 Read 5 Done Done New false
16104 5409 Acked 6408 Flush 0 Done Done New false
16105 5410 Acked 6409 Write 6 Done Done New false
16106 5411 Acked 6410 Write 6 Done Done New false
16107 5412 Acked 6411 Flush 0 Done Done New false
16108 5413 Acked 6412 Write 3 Done Done New false
16109 5414 Acked 6413 Read 3 Done Done New false
16110 5415 Acked 6414 Read 9 Done Done New false
16111 5416 Acked 6415 Write 4 Done Done New false
16112 5417 Acked 6416 Write 9 Done Done New false
16113 5418 Acked 6417 Read 2 Done Done New false
16114 5419 Acked 6418 Write 1 Done Done New false
16115 5420 Acked 6419 Write 7 Done Done New false
16116 5421 Acked 6420 Read 7 Done Done New false
16117 5422 Acked 6421 Flush 0 Done Done New false
16118 5423 Acked 6422 Read 6 Done Done New false
16119 5424 Acked 6423 Read 1 Done Done New false
16120 5425 Acked 6424 Read 3 Done Done New false
16121 STATES DS:0 DS:1 DS:2 TOTAL
16122 New 0 0 301 301
16123 Sent 0 0 0 0
16124 Done 301 301 0 602
16125 Skipped 0 0 0 0
16126 Error 0 0 0 0
16127 Last Flush: 6421 6421 6123
16128 Downstairs last five completed: 6123 6122 6121 6120 6119
16129 Upstairs last five completed: 5425 5424 5423 5422 5421
16130 CLIENT: Up:0 ds:301 act:2
161312024-02-27T21:47:48.970ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161322024-02-27T21:47:48.970ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161332024-02-27T21:47:48.970ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161342024-02-27T21:47:48.970ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
161352024-02-27T21:47:48.970ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161362024-02-27T21:47:48.970ZINFOcrucible: [2] client re-new 302 jobs since flush 6123 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
161372024-02-27T21:47:48.971ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161382024-02-27T21:47:49.971ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161392024-02-27T21:47:49.971ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161402024-02-27T21:47:49.971ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161412024-02-27T21:47:49.971ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
16142 [8830] got start action
16143 [8830] I am going to restart
16144 Starting downstairs at port 8830
161452024-02-27T21:47:49.971ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161462024-02-27T21:47:49.972ZINFOcrucible: [2] client re-new 302 jobs since flush 6123 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
16147 Make output file at "/tmp/dsc/downstairs-8830.txt"
16148 [8830][2] reports Starting
161492024-02-27T21:47:49.972ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
16150 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1185)
16151 [8830][2] reports Running
161522024-02-27T21:47:50.974ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161532024-02-27T21:47:50.974ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161542024-02-27T21:47:50.975ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161552024-02-27T21:47:50.975ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161562024-02-27T21:47:50.975ZINFOcrucible: send last flush ID to this DS: 6123 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161572024-02-27T21:47:50.975ZINFOcrucible: Replied this last flush ID: 6123 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161582024-02-27T21:47:50.975ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161592024-02-27T21:47:50.975ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161602024-02-27T21:47:50.976ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161612024-02-27T21:47:50.976ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
16162 ----------------------------------------------------------------
16163 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
16164 Downstairs last five completed: 6425 6424 6423 6422 6421
16165 Upstairs last five completed: 5426 5425 5424 5423 5422
16166 CLIENT: Up:0 ds:0 act:3
16167 Replay: All jobs finished, all DS active.
16168 Main task has work to do, go find it
16169 got dsc Stop(2)
16170 stop 2
16171 Feb 27 21:47:52.338 INFO request completed, latency_us:[8830] Got stop action so:false kr:false
16172 365, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 7b195fc6-ffbd-4920-91fc-4931e2e6a2ac, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
161732024-02-27T21:47:52.339ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161742024-02-27T21:47:52.339ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161752024-02-27T21:47:52.339ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = e6a5a974-925a-494f-8845-facc441c2690
161762024-02-27T21:47:52.339ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
16177 Feb 27 21:47:52.339 INFO request completed, latency_us: 120, response_code: 200, {"msg":"[2] client re-new 0 jobs since flush 6425","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:47:52.339858024Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"downstairs","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
16178 uri: /state/cid/2, method: GET, req_id: 2ce34927-eb11-4e3b-bb4d-c72ea2aa9270, remote_addr: 127.0.0.1:42901, local_addr: 127.0{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:47:52.33989583Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"io task","client":"2","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
16179 .0.1:9998
16180 [8830] Exited with: ExitStatus(unix_wait_status(9))
16181 [8830][2] reports Exit
161822024-02-27T21:47:53.340ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161832024-02-27T21:47:53.340ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161842024-02-27T21:47:53.340ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161852024-02-27T21:47:53.340ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
161862024-02-27T21:47:53.340ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161872024-02-27T21:47:53.341ZINFOcrucible: [2] client re-new 0 jobs since flush 6425 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
161882024-02-27T21:47:53.341ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161892024-02-27T21:47:54.342ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161902024-02-27T21:47:54.342ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161912024-02-27T21:47:54.342ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161922024-02-27T21:47:54.342ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
161932024-02-27T21:47:54.342ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161942024-02-27T21:47:54.342ZINFOcrucible: [2] client re-new 0 jobs since flush 6425 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
161952024-02-27T21:47:54.343ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161962024-02-27T21:47:55.343ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161972024-02-27T21:47:55.343ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161982024-02-27T21:47:55.343ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
161992024-02-27T21:47:55.343ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
162002024-02-27T21:47:55.343ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
162012024-02-27T21:47:55.343ZINFOcrucible: [2] client re-new 0 jobs since flush 6425 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
162022024-02-27T21:47:55.343ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
16203 Feb 27 21:47:56.342 INFO request completed, latency_us: 236, response_code: 200, uri: /state/cid/2, method: GET, req_id: a44f2d24-113e-4056-94ec-79f70473701d, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
16204 001/300 Write block 3736 len 24576 data: 3 3 3 4 5 4
16205 002/300 Write block 1826 len 32768 data: 4 6 5 4 5 6 6 6
162062024-02-27T21:47:56.344ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
162072024-02-27T21:47:56.344ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
162082024-02-27T21:47:56.344ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
162092024-02-27T21:47:56.344ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
162102024-02-27T21:47:56.344ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
162112024-02-27T21:47:56.344ZINFOcrucible: [2] client re-new 2 jobs since flush 6425 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
162122024-02-27T21:47:56.344ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
16213 003/300 Write block 4523 len 36864 data: 2 2 2 3 5 5 4 4 3
16214 004/300 Flush
16215 005/300 Read block 2318 len 28672
16216 006/300 Write block 1097 len 4096 data: 3
16217 007/300 Flush
16218 008/300 Read block 1320 len 12288
16219 009/300 Write block 1227 len 16384 data: 4 5 5 5
16220 010/300 Flush
16221 011/300 Write block 2043 len 20480 data: 3 2 3 4 5
16222 012/300 Write block 4417 len 4096 data: 3
16223 013/300 Read block 3678 len 4096
16224 014/300 Write block 2305 len 36864 data: 4 3 3 2 3 4 5 5 5
16225 015/300 Read block 1904 len 20480
16226 016/300 Read block 2071 len 4096
16227 017/300 Read block 351 len 36864
16228 018/300 Read block 1674 len 12288
16229 019/300 Read block 445 len 4096
16230 020/300 Read block 2487 len 28672
16231 021/300 Write block 1789 len 12288 data: 3 3 3
16232 022/300 Flush
16233 023/300 Read block 1813 len 32768
16234 024/300 Read block 1304 len 32768
16235 025/300 Flush
16236 026/300 Write block 26 len 40960 data: 7 6 5 4 3 3 3 3 2 2
16237 027/300 Flush
16238 028/300 Read block 2624 len 16384
16239 029/300 Read block 493 len 4096
16240 030/300 Write block 3109 len 36864 data: 6 7 8 6 6 7 8 7 7
16241 031/300 Read block 4292 len 12288
16242 032/300 Write block 2603 len 8192 data: 4 2
16243 033/300 Write block 4624 len 12288 data: 5 6 5
16244 034/300 Write block 2882 len 32768 data: 5 4 4 5 4 3 3 3
16245 035/300 Flush
16246 036/300 Write block 3448 len 24576 data: 7 6 6 6 5 4
16247 037/300 Read block 3643 len 36864
16248 038/300 Read block 2997 len 24576
16249 039/300 Read block 4385 len 28672
16250 040/300 Read block 89 len 20480
16251 041/300 Read block 598 len 12288
16252 042/300 Read block 763 len 12288
16253 043/300 Read block 3429 len 8192
16254 044/300 Read block 1853 len 16384
16255 045/300 Read block 4202 len 24576
16256 046/300 Read block 3402 len 8192
16257 047/300 Read block 3934 len 20480
16258 048/300 Read block 2224 len 4096
16259 049/300 Write block 4398 len 20480 data: 8 9 9 8 8
16260 050/300 Read block 3172 len 16384
16261 051/300 Write block 1373 len 32768 data: 2 3 3 3 3 4 4 3
16262 052/300 Write block 1918 len 20480 data: 3 3 4 4 2
16263 053/300 Read block 2992 len 16384
16264 054/300 Read block 4228 len 28672
16265 055/300 Write block 3612 len 28672 data: 5 5 5 5 5 4 3
16266 056/300 Write block 2468 len 28672 data: 4 5 5 5 5 5 5
16267 057/300 Write block 1979 len 4096 data: 7
16268 058/300 Flush
16269 059/300 Read block 1967 len 20480
16270 060/300 Write block 3938 len 16384 data: 5 4 3 3
16271 061/300 Write block 1380 len 28672 data: 4 3 4 4 4 4 4
16272 062/300 Write block 2331 len 20480 data: 6 4 4 7 8
16273 063/300 Read block 5 len 40960
16274 064/300 Read block 3322 len 20480
16275 065/300 Write block 3635 len 32768 data: 4 5 6 6 6 5 6 5
16276 066/300 Write block 4159 len 16384 data: 3 3 3 3
16277 067/300 Read block 1153 len 36864
16278 068/300 Read block 1123 len 12288
16279 069/300 Write block 2 len 36864 data: 2 2 3 3 2 2 2 2 3
16280 070/300 Read block 1583 len 8192
16281 071/300 Read block 4170 len 16384
16282 072/300 Read block 905 len 12288
16283 073/300 Read block 3668 len 24576
16284 074/300 Read block 3898 len 36864
16285 075/300 Flush
16286 076/300 Read block 1620 len 32768
16287 077/300 Write block 4455 len 4096 data: 4
16288 078/300 Write block 804 len 24576 data: 3 2 2 3 3 3
16289 079/300 Read block 502 len 32768
16290 080/300 Write block 130 len 20480 data: 2 2 2 2 3
16291 081/300 Read block 3584 len 12288
16292 082/300 Read block 3554 len 24576
16293 083/300 Write block 97 len 24576 data: 5 5 4 4 4 4
16294 084/300 Read block 2164 len 40960
16295 085/300 Flush
16296 086/300 Write block 3512 len 4096 data: 3
16297 087/300 Read block 1192 len 28672
16298 088/300 Read block 1068 len 24576
16299 089/300 Write block 4252 len 40960 data: 5 5 4 4 3 2 2 3 3 4
16300 090/300 Write block 2680 len 12288 data: 6 6 6
16301 091/300 Read block 3161 len 32768
16302 092/300 Write block 2387 len 4096 data: 2
16303 093/300 Write block 1170 len 20480 data: 3 4 4 4 5
16304 094/300 Read block 2272 len 16384
16305 095/300 Write block 1945 len 20480 data: 4 4 4 4 4
16306 096/300 Write block 2323 len 20480 data: 6 7 6 5 5
16307 097/300 Write block 1808 len 24576 data: 5 4 4 5 5 5
16308 098/300 Read block 4752 len 32768
16309 099/300 Read block 3069 len 40960
16310 100/300 Read block 3444 len 28672
16311 101/300 Read block 567 len 36864
16312 102/300 Flush
16313 103/300 Write block 4703 len 8192 data: 5 5
16314 104/300 Write block 1498 len 16384 data: 3 4 4 6
16315 105/300 Read block 3033 len 8192
16316 106/300 Write block 1594 len 40960 data: 6 6 5 4 4 5 4 3 2 3
16317 107/300 Flush
16318 108/300 Read block 4525 len 16384
16319 109/300 Write block 684 len 20480 data: 4 5 5 7 4
16320 110/300 Write block 3678 len 8192 data: 8 9
16321 111/300 Read block 1749 len 36864
16322 112/300 Read block 3005 len 32768
16323 113/300 Write block 4780 len 28672 data: 2 2 2 3 3 4 5
16324 114/300 Flush
16325 115/300 Read block 3308 len 28672
16326 116/300 Read block 1520 len 12288
16327 117/300 Read block 2104 len 24576
16328 118/300 Read block 1184 len 20480
16329 119/300 Read block 2457 len 4096
16330 120/300 Flush
16331 121/300 Write block 4258 len 32768 data: 3 4 4 5 5 6 8 8
16332 122/300 Write block 4440 len 20480 data: 6 5 4 4 6
16333 123/300 Write block 3709 len 24576 data: 4 2 2 3 3 3
16334 124/300 Read block 4946 len 16384
16335 125/300 Write block 242 len 36864 data: 3 4 4 4 4 4 3 3 3
16336 126/300 Flush
16337 127/300 Read block 23 len 20480
16338 128/300 Read block 3727 len 16384
16339 129/300 Read block 1460 len 40960
16340 130/300 Read block 3615 len 20480
16341 131/300 Write block 4229 len 16384 data: 3 3 3 3
16342 132/300 Read block 2462 len 40960
16343 133/300 Write block 4144 len 40960 data: 4 4 3 4 6 6 5 5 5 5
16344 134/300 Flush
16345 135/300 Read block 4962 len 4096
16346 136/300 Read block 1122 len 16384
16347 137/300 Write block 4345 len 36864 data: 4 4 4 4 5 5 5 6 5
16348 138/300 Write block 4667 len 40960 data: 6 7 5 6 6 6 7 6 4 4
16349 139/300 Read block 578 len 20480
16350 140/300 Read block 2611 len 36864
16351 141/300 Write block 1965 len 40960 data: 3 2 3 5 4 4 5 4 5 5
16352 142/300 Read block 1262 len 20480
16353 143/300 Write block 139 len 32768 data: 3 3 3 4 3 3 4 5
16354 144/300 Write block 4923 len 24576 data: 4 4 3 3 3 4
16355 145/300 Write block 4835 len 16384 data: 5 5 5 5
16356 146/300 Write block 4512 len 8192 data: 4 4
16357 147/300 Read block 1522 len 4096
16358 148/300 Write block 4408 len 40960 data: 6 6 5 4 4 4 4 4 3 4
16359 149/300 Read block 1957 len 36864
16360 150/300 Write block 412 len 40960 data: 5 6 6 6 5 4 6 7 7 7
16361 151/300 Write block 3439 len 16384 data: 2 2 2 4
16362 152/300 Read block 4282 len 32768
16363 153/300 Read block 714 len 20480
16364 154/300 Write block 1673 len 4096 data: 5
16365 155/300 Write block 1350 len 16384 data: 3 3 4 4
16366 156/300 Write block 2511 len 24576 data: 4 3 3 2 3 3
16367 157/300 Read block 4759 len 8192
16368 158/300 Read block 2372 len 32768
16369 159/300 Write block 3434 len 24576 data: 2 2 2 2 2 3
16370 160/300 Flush
16371 161/300 Write block 3409 len 16384 data: 6 8 8 7
16372 162/300 Write block 4425 len 16384 data: 4 3 5 6
16373 163/300 Write block 4511 len 40960 data: 4 5 5 4 3 3 3 3 3 3
16374 164/300 Read block 20 len 28672
16375 165/300 Flush
16376 166/300 Write block 4028 len 36864 data: 6 7 7 6 4 2 2 4 4
16377 167/300 Write block 3788 len 20480 data: 5 5 5 4 4
16378 168/300 Read block 3402 len 28672
16379 169/300 Read block 2681 len 8192
16380 170/300 Write block 2918 len 40960 data: 3 2 3 3 3 3 2 2 2 3
16381 171/300 Write block 3195 len 20480 data: 5 6 6 4 4
16382 172/300 Write block 1032 len 12288 data: 4 3 2
16383 173/300 Write block 3679 len 24576 data: 10 6 5 6 7 5
16384 174/300 Read block 1399 len 40960
16385 175/300 Read block 2757 len 32768
16386 176/300 Flush
16387 177/300 Read block 3109 len 36864
16388 178/300 Write block 3291 len 20480 data: 3 3 3 3 3
16389 179/300 Write block 3850 len 24576 data: 7 7 8 8 5 6
16390 180/300 Write block 4989 len 16384 data: 3 4 5 5
16391 181/300 Read block 365 len 12288
16392 182/300 Write block 1417 len 4096 data: 3
16393 183/300 Write block 4010 len 36864 data: 3 3 3 4 5 5 4 4 6
16394 184/300 Write block 4473 len 8192 data: 3 2
16395 185/300 Read block 2887 len 16384
16396 186/300 Write block 1963 len 28672 data: 3 4 4 3 4 6 5
16397 187/300 Write block 3317 len 28672 data: 8 7 7 6 6 6 6
16398 188/300 Read block 1565 len 16384
16399 189/300 Write block 1352 len 32768 data: 5 5 3 3 3 3 3 4
16400 190/300 Read block 432 len 4096
16401 191/300 Read block 3351 len 20480
16402 192/300 Read block 2121 len 40960
16403 193/300 Read block 2104 len 8192
16404 194/300 Read block 2698 len 8192
16405 195/300 Write block 4876 len 20480 data: 5 4 4 3 3
16406 196/300 Read block 519 len 40960
16407 197/300 Write block 781 len 12288 data: 4 5 4
16408 198/300 Read block 3861 len 28672
16409 199/300 Write block 2728 len 32768 data: 2 2 2 3 4 5 6 6
16410 200/300 Write block 726 len 20480 data: 3 3 3 3 4
16411 201/300 Read block 98 len 28672
16412 202/300 Read block 877 len 32768
16413 203/300 Write block 207 len 40960 data: 7 6 6 5 5 5 2 2 2 3
16414 204/300 Read block 1489 len 28672
16415 205/300 Write block 1126 len 40960 data: 3 3 5 5 7 8 9 7 5 4
16416 206/300 Read block 2527 len 24576
16417 207/300 Read block 1680 len 12288
16418 208/300 Write block 4101 len 36864 data: 4 4 5 4 3 3 3 2 2
16419 209/300 Write block 2339 len 28672 data: 10 7 6 5 5 5 5
16420 210/300 Flush
16421 211/300 Read block 4435 len 24576
16422 212/300 Read block 1985 len 4096
16423 213/300 Write block 2861 len 12288 data: 3 3 3
16424 214/300 Write block 3043 len 8192 data: 5 5
16425 215/300 Read block 3559 len 40960
16426 216/300 Read block 1148 len 8192
16427 217/300 Write block 280 len 28672 data: 4 4 5 4 4 3 3
16428 218/300 Write block 344 len 8192 data: 2 3
16429 219/300 Read block 3052 len 12288
16430 220/300 Read block 3667 len 32768
16431 221/300 Read block 593 len 32768
16432 222/300 Read block 1374 len 20480
16433 223/300 Read block 4280 len 20480
16434 224/300 Write block 1494 len 8192 data: 6 3
16435 225/300 Read block 364 len 28672
16436 226/300 Write block 2132 len 4096 data: 4
16437 227/300 Read block 4232 len 16384
16438 228/300 Read block 389 len 36864
16439 229/300 Read block 1830 len 24576
16440 230/300 Read block 2822 len 36864
16441 231/300 Read block 3180 len 24576
16442 232/300 Read block 4460 len 16384
16443 233/300 Write block 2843 len 24576 data: 4 4 6 6 6 5
16444 234/300 Write block 3585 len 12288 data: 3 3 5
16445 235/300 Read block 3110 len 20480
16446 236/300 Read block 36 len 8192
16447 237/300 Write block 2910 len 32768 data: 7 7 6 6 6 5 4 4
16448 238/300 Read block 2631 len 40960
16449 239/300 Flush
16450 240/300 Write block 3346 len 24576 data: 6 5 4 4 4 2
16451 241/300 Read block 891 len 40960
16452 242/300 Read block 3334 len 20480
16453 243/300 Read block 965 len 8192
16454 244/300 Write block 3835 len 16384 data: 3 4 4 5
16455 245/300 Write block 450 len 16384 data: 5 6 5 4
16456 246/300 Read block 1531 len 36864
16457 247/300 Flush
16458 248/300 Read block 1119 len 32768
16459 249/300 Write block 3168 len 4096 data: 4
16460 250/300 Write block 3145 len 32768 data: 4 4 5 4 4 4 5 5
16461 251/300 Read block 2691 len 32768
16462 252/300 Read block 3230 len 4096
16463 253/300 Read block 2448 len 36864
16464 254/300 Flush
16465 255/300 Write block 3067 len 16384 data: 3 3 3 3
16466 256/300 Write block 2983 len 8192 data: 3 3
16467 257/300 Read block 2959 len 32768
16468 258/300 Write block 3914 len 32768 data: 3 3 4 5 4 3 4 4
16469 259/300 Write block 2024 len 8192 data: 6 4
16470 260/300 Read block 2298 len 28672
16471 261/300 Write block 3819 len 36864 data: 7 7 4 4 4 4 5 3 2
16472 262/300 Read block 4192 len 28672
16473 263/300 Flush
16474 264/300 Read block 3806 len 8192
16475 265/300 Write block 4485 len 4096 data: 4
16476 266/300 Read block 3703 len 36864
16477 267/300 Write block 6 len 32768 data: 3 3 3 3 4 4 4 4
16478 268/300 Write block 4696 len 28672 data: 5 4 4 3 3 3 4
16479 269/300 Write block 612 len 4096 data: 4
16480 270/300 Read block 1811 len 4096
16481 271/300 Write block 622 len 12288 data: 3 3 4
16482 272/300 Write block 1351 len 12288 data: 4 6 6
16483 273/300 Write block 4221 len 8192 data: 6 5
16484 274/300 Write block 1634 len 28672 data: 4 4 4 4 6 8 7
16485 275/300 Write block 4972 len 8192 data: 6 6
16486 276/300 Write block 910 len 36864 data: 7 7 7 6 6 6 5 4 2
16487 277/300 Read block 4992 len 28672
16488 278/300 Flush
16489 279/300 Read block 1061 len 28672
16490 280/300 Write block 248 len 28672 data: 4 4 4 4 4 4 3
16491 281/300 Write block 427 len 16384 data: 3 3 4 3
16492 282/300 Read block 4970 len 16384
16493 283/300 Flush
16494 284/300 Read block 1406 len 36864
16495 285/300 Read block 4589 len 4096
16496 286/300 Read block 4471 len 28672
16497 287/300 Write block 2251 len 32768 data: 2 2 2 3 4 4 5 5
16498 288/300 Read block 3608 len 4096
16499 289/300 Write block 1923 len 24576 data: 2 3 3 4 5 7
16500 290/300 Read block 270 len 20480
16501 291/300 Write block 600 len 20480 data: 5 5 5 5 4
16502 292/300 Read block 776 len 28672
16503 293/300 Flush
16504 294/300 Write block 3062 len 28672 data: 4 5 5 4 4 4 4
16505 295/300 Read block 2174 len 4096
16506 296/300 Write block 139 len 16384 data: 4 4 4 5
16507 297/300 Read block 3190 len 12288
16508 298/300 Write block 1189 len 32768 data: 6 6 7 5 4 5 5 6
16509 299/300 Flush
16510 300/300 Read block 400 len 8192
16511 Main task has work to do, go find it
16512 got dsc Start(2)
16513 start 2
16514 Feb 27 21:47:56.685 INFO request completed, latency_us: 211, response_code: 204, uri: /start/cid/2, method: POST, req_id: 6106d50a-bbc0-40cb-9495-905d7275dedc, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
16515 Replay: started 2, returned:Ok(())
16516 ----------------------------------------------------------------
16517 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
16518 States: Active Active Offline
16519 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
16520 5427 Acked 6426 Write 6 Done Done New true
16521 5428 Acked 6427 Write 8 Done Done New true
16522 5429 Acked 6428 Write 9 Done Done New false
16523 5430 Acked 6429 Flush 0 Done Done New false
16524 5431 Acked 6430 Read 7 Done Done New false
16525 5432 Acked 6431 Write 1 Done Done New false
16526 5433 Acked 6432 Flush 0 Done Done New false
16527 5434 Acked 6433 Read 3 Done Done New false
16528 5435 Acked 6434 Write 4 Done Done New false
16529 5436 Acked 6435 Flush 0 Done Done New false
16530 5437 Acked 6436 Write 5 Done Done New false
16531 5438 Acked 6437 Write 1 Done Done New false
16532 5439 Acked 6438 Read 1 Done Done New false
16533 5440 Acked 6439 Write 9 Done Done New false
16534 5441 Acked 6440 Read 5 Done Done New false
16535 5442 Acked 6441 Read 1 Done Done New false
16536 5443 Acked 6442 Read 9 Done Done New false
16537 5444 Acked 6443 Read 3 Done Done New false
16538 5445 Acked 6444 Read 1 Done Done New false
16539 5446 Acked 6445 Read 7 Done Done New false
16540 5447 Acked 6446 Write 3 Done Done New false
16541 5448 Acked 6447 Flush 0 Done Done New false
16542 5449 Acked 6448 Read 8 Done Done New false
16543 5450 Acked 6449 Read 8 Done Done New false
16544 5451 Acked 6450 Flush 0 Done Done New false
16545 5452 Acked 6451 Write 10 Done Done New false
16546 5453 Acked 6452 Flush 0 Done Done New false
16547 5454 Acked 6453 Read 4 Done Done New false
16548 5455 Acked 6454 Read 1 Done Done New false
16549 5456 Acked 6455 Write 9 Done Done New false
16550 5457 Acked 6456 Read 3 Done Done New false
16551 5458 Acked 6457 Write 2 Done Done New false
16552 5459 Acked 6458 Write 3 Done Done New false
16553 5460 Acked 6459 Write 8 Done Done New false
16554 5461 Acked 6460 Flush 0 Done Done New false
16555 5462 Acked 6461 Write 6 Done Done New false
16556 5463 Acked 6462 Read 9 Done Done New false
16557 5464 Acked 6463 Read 6 Done Done New false
16558 5465 Acked 6464 Read 7 Done Done New false
16559 5466 Acked 6465 Read 5 Done Done New false
16560 5467 Acked 6466 Read 3 Done Done New false
16561 5468 Acked 6467 Read 3 Done Done New false
16562 5469 Acked 6468 Read 2 Done Done New false
16563 5470 Acked 6469 Read 4 Done Done New false
16564 5471 Acked 6470 Read 6 Done Done New false
16565 5472 Acked 6471 Read 2 Done Done New false
16566 5473 Acked 6472 Read 5 Done Done New false
16567 5474 Acked 6473 Read 1 Done Done New false
16568 5475 Acked 6474 Write 5 Done Done New false
16569 5476 Acked 6475 Read 4 Done Done New false
16570 5477 Acked 6476 Write 8 Done Done New false
16571 5478 Acked 6477 Write 5 Done Done New false
16572 5479 Acked 6478 Read 4 Done Done New false
16573 5480 Acked 6479 Read 7 Done Done New false
16574 5481 Acked 6480 Write 7 Done Done New false
16575 5482 Acked 6481 Write 7 Done Done New false
16576 5483 Acked 6482 Write 1 Done Done New false
16577 5484 Acked 6483 Flush 0 Done Done New false
16578 5485 Acked 6484 Read 5 Done Done New false
16579 5486 Acked 6485 Write 4 Done Done New false
16580 5487 Acked 6486 Write 7 Done Done New false
16581 5488 Acked 6487 Write 5 Done Done New false
16582 5489 Acked 6488 Read 10 Done Done New false
16583 5490 Acked 6489 Read 5 Done Done New false
16584 5491 Acked 6490 Write 8 Done Done New false
16585 5492 Acked 6491 Write 4 Done Done New false
16586 5493 Acked 6492 Read 9 Done Done New false
16587 5494 Acked 6493 Read 3 Done Done New false
16588 5495 Acked 6494 Write 9 Done Done New false
16589 5496 Acked 6495 Read 2 Done Done New false
16590 5497 Acked 6496 Read 4 Done Done New false
16591 5498 Acked 6497 Read 3 Done Done New false
16592 5499 Acked 6498 Read 6 Done Done New false
16593 5500 Acked 6499 Read 9 Done Done New false
16594 5501 Acked 6500 Flush 0 Done Done New false
16595 5502 Acked 6501 Read 8 Done Done New false
16596 5503 Acked 6502 Write 1 Done Done New false
16597 5504 Acked 6503 Write 6 Done Done New false
16598 5505 Acked 6504 Read 8 Done Done New false
16599 5506 Acked 6505 Write 5 Done Done New false
16600 5507 Acked 6506 Read 3 Done Done New false
16601 5508 Acked 6507 Read 6 Done Done New false
16602 5509 Acked 6508 Write 6 Done Done New false
16603 5510 Acked 6509 Read 10 Done Done New false
16604 5511 Acked 6510 Flush 0 Done Done New false
16605 5512 Acked 6511 Write 1 Done Done New false
16606 5513 Acked 6512 Read 7 Done Done New false
16607 5514 Acked 6513 Read 6 Done Done New false
16608 5515 Acked 6514 Write 10 Done Done New false
16609 5516 Acked 6515 Write 3 Done Done New false
16610 5517 Acked 6516 Read 8 Done Done New false
16611 5518 Acked 6517 Write 1 Done Done New false
16612 5519 Acked 6518 Write 5 Done Done New false
16613 5520 Acked 6519 Read 4 Done Done New false
16614 5521 Acked 6520 Write 5 Done Done New false
16615 5522 Acked 6521 Write 5 Done Done New false
16616 5523 Acked 6522 Write 6 Done Done New false
16617 5524 Acked 6523 Read 8 Done Done New false
16618 5525 Acked 6524 Read 10 Done Done New false
16619 5526 Acked 6525 Read 7 Done Done New false
16620 5527 Acked 6526 Read 9 Done Done New false
16621 5528 Acked 6527 Flush 0 Done Done New false
16622 5529 Acked 6528 Write 2 Done Done New false
16623 5530 Acked 6529 Write 4 Done Done New false
16624 5531 Acked 6530 Read 2 Done Done New false
16625 5532 Acked 6531 Write 10 Done Done New false
16626 5533 Acked 6532 Flush 0 Done Done New false
16627 5534 Acked 6533 Read 4 Done Done New false
16628 5535 Acked 6534 Write 5 Done Done New false
16629 5536 Acked 6535 Write 2 Done Done New false
16630 5537 Acked 6536 Read 9 Done Done New false
16631 5538 Acked 6537 Read 8 Done Done New false
16632 5539 Acked 6538 Write 7 Done Done New false
16633 5540 Acked 6539 Flush 0 Done Done New false
16634 5541 Acked 6540 Read 7 Done Done New false
16635 5542 Acked 6541 Read 3 Done Done New false
16636 5543 Acked 6542 Read 6 Done Done New false
16637 5544 Acked 6543 Read 5 Done Done New false
16638 5545 Acked 6544 Read 1 Done Done New false
16639 5546 Acked 6545 Flush 0 Done Done New false
16640 5547 Acked 6546 Write 8 Done Done New false
16641 5548 Acked 6547 Write 5 Done Done New false
16642 5549 Acked 6548 Write 6 Done Done New false
16643 5550 Acked 6549 Read 4 Done Done New false
16644 5551 Acked 6550 Write 9 Done Done New false
16645 5552 Acked 6551 Flush 0 Done Done New false
16646 5553 Acked 6552 Read 5 Done Done New false
16647 5554 Acked 6553 Read 4 Done Done New false
16648 5555 Acked 6554 Read 10 Done Done New false
16649 5556 Acked 6555 Read 5 Done Done New false
16650 5557 Acked 6556 Write 4 Done Done New false
16651 5558 Acked 6557 Read 10 Done Done New false
16652 5559 Acked 6558 Write 10 Done Done New false
16653 5560 Acked 6559 Flush 0 Done Done New false
16654 5561 Acked 6560 Read 1 Done Done New false
16655 5562 Acked 6561 Read 4 Done Done New false
16656 5563 Acked 6562 Write 9 Done Done New false
16657 5564 Acked 6563 Write 10 Done Done New false
16658 5565 Acked 6564 Read 5 Done Done New false
16659 5566 Acked 6565 Read 9 Done Done New false
16660 5567 Acked 6566 Write 10 Done Done New false
16661 5568 Acked 6567 Read 5 Done Done New false
16662 5569 Acked 6568 Write 8 Done Done New false
16663 5570 Acked 6569 Write 6 Done Done New false
16664 5571 Acked 6570 Write 4 Done Done New false
16665 5572 Acked 6571 Write 2 Done Done New false
16666 5573 Acked 6572 Read 1 Done Done New false
16667 5574 Acked 6573 Write 10 Done Done New false
16668 5575 Acked 6574 Read 9 Done Done New false
16669 5576 Acked 6575 Write 10 Done Done New false
16670 5577 Acked 6576 Write 4 Done Done New false
16671 5578 Acked 6577 Read 8 Done Done New false
16672 5579 Acked 6578 Read 5 Done Done New false
16673 5580 Acked 6579 Write 1 Done Done New false
16674 5581 Acked 6580 Write 4 Done Done New false
16675 5582 Acked 6581 Write 6 Done Done New false
16676 5583 Acked 6582 Read 2 Done Done New false
16677 5584 Acked 6583 Read 8 Done Done New false
16678 5585 Acked 6584 Write 6 Done Done New false
16679 5586 Acked 6585 Flush 0 Done Done New false
16680 5587 Acked 6586 Write 4 Done Done New false
16681 5588 Acked 6587 Write 4 Done Done New false
16682 5589 Acked 6588 Write 10 Done Done New false
16683 5590 Acked 6589 Read 7 Done Done New false
16684 5591 Acked 6590 Flush 0 Done Done New false
16685 5592 Acked 6591 Write 9 Done Done New false
16686 5593 Acked 6592 Write 5 Done Done New false
16687 5594 Acked 6593 Read 7 Done Done New false
16688 5595 Acked 6594 Read 2 Done Done New false
16689 5596 Acked 6595 Write 10 Done Done New false
16690 5597 Acked 6596 Write 5 Done Done New false
16691 5598 Acked 6597 Write 3 Done Done New false
16692 5599 Acked 6598 Write 6 Done Done New false
16693 5600 Acked 6599 Read 10 Done Done New false
16694 5601 Acked 6600 Read 8 Done Done New false
16695 5602 Acked 6601 Flush 0 Done Done New false
16696 5603 Acked 6602 Read 9 Done Done New false
16697 5604 Acked 6603 Write 5 Done Done New false
16698 5605 Acked 6604 Write 6 Done Done New false
16699 5606 Acked 6605 Write 4 Done Done New false
16700 5607 Acked 6606 Read 3 Done Done New false
16701 5608 Acked 6607 Write 1 Done Done New false
16702 5609 Acked 6608 Write 9 Done Done New false
16703 5610 Acked 6609 Write 2 Done Done New false
16704 5611 Acked 6610 Read 4 Done Done New false
16705 5612 Acked 6611 Write 7 Done Done New false
16706 5613 Acked 6612 Write 7 Done Done New false
16707 5614 Acked 6613 Read 4 Done Done New false
16708 5615 Acked 6614 Write 8 Done Done New false
16709 5616 Acked 6615 Read 1 Done Done New false
16710 5617 Acked 6616 Read 5 Done Done New false
16711 5618 Acked 6617 Read 10 Done Done New false
16712 5619 Acked 6618 Read 2 Done Done New false
16713 5620 Acked 6619 Read 2 Done Done New false
16714 5621 Acked 6620 Write 5 Done Done New false
16715 5622 Acked 6621 Read 10 Done Done New false
16716 5623 Acked 6622 Write 3 Done Done New false
16717 5624 Acked 6623 Read 7 Done Done New false
16718 5625 Acked 6624 Write 8 Done Done New false
16719 5626 Acked 6625 Write 5 Done Done New false
16720 5627 Acked 6626 Read 7 Done Done New false
16721 5628 Acked 6627 Read 8 Done Done New false
16722 5629 Acked 6628 Write 10 Done Done New false
16723 5630 Acked 6629 Read 7 Done Done New false
16724 5631 Acked 6630 Write 10 Done Done New false
16725 5632 Acked 6631 Read 6 Done Done New false
16726 5633 Acked 6632 Read 3 Done Done New false
16727 5634 Acked 6633 Write 9 Done Done New false
16728 5635 Acked 6634 Write 7 Done Done New false
16729 5636 Acked 6635 Flush 0 Done Done New false
16730 5637 Acked 6636 Read 6 Done Done New false
16731 5638 Acked 6637 Read 1 Done Done New false
16732 5639 Acked 6638 Write 3 Done Done New false
16733 5640 Acked 6639 Write 2 Done Done New false
16734 5641 Acked 6640 Read 10 Done Done New false
16735 5642 Acked 6641 Read 2 Done Done New false
16736 5643 Acked 6642 Write 7 Done Done New false
16737 5644 Acked 6643 Write 2 Done Done New false
16738 5645 Acked 6644 Read 3 Done Done New false
16739 5646 Acked 6645 Read 8 Done Done New false
16740 5647 Acked 6646 Read 8 Done Done New false
16741 5648 Acked 6647 Read 5 Done Done New false
16742 5649 Acked 6648 Read 5 Done Done New false
16743 5650 Acked 6649 Write 2 Done Done New false
16744 5651 Acked 6650 Read 7 Done Done New false
16745 5652 Acked 6651 Write 1 Done Done New false
16746 5653 Acked 6652 Read 4 Done Done New false
16747 5654 Acked 6653 Read 9 Done Done New false
16748 5655 Acked 6654 Read 6 Done Done New false
16749 5656 Acked 6655 Read 9 Done Done New false
16750 5657 Acked 6656 Read 6 Done Done New false
16751 5658 Acked 6657 Read 4 Done Done New false
16752 5659 Acked 6658 Write 6 Done Done New false
16753 5660 Acked 6659 Write 3 Done Done New false
16754 5661 Acked 6660 Read 5 Done Done New false
16755 5662 Acked 6661 Read 2 Done Done New false
16756 5663 Acked 6662 Write 8 Done Done New false
16757 5664 Acked 6663 Read 10 Done Done New false
16758 5665 Acked 6664 Flush 0 Done Done New false
16759 5666 Acked 6665 Write 6 Done Done New false
16760 5667 Acked 6666 Read 10 Done Done New false
16761 5668 Acked 6667 Read 5 Done Done New false
16762 5669 Acked 6668 Read 2 Done Done New false
16763 5670 Acked 6669 Write 4 Done Done New false
16764 5671 Acked 6670 Write 4 Done Done New false
16765 5672 Acked 6671 Read 9 Done Done New false
16766 5673 Acked 6672 Flush 0 Done Done New false
16767 5674 Acked 6673 Read 8 Done Done New false
16768 5675 Acked 6674 Write 1 Done Done New false
16769 5676 Acked 6675 Write 8 Done Done New false
16770 5677 Acked 6676 Read 8 Done Done New false
16771 5678 Acked 6677 Read 1 Done Done New false
16772 5679 Acked 6678 Read 9 Done Done New false
16773 5680 Acked 6679 Flush 0 Done Done New false
16774 5681 Acked 6680 Write 4 Done Done New false
16775 5682 Acked 6681 Write 2 Done Done New false
16776 5683 Acked 6682 Read 8 Done Done New false
16777 5684 Acked 6683 Write 8 Done Done New false
16778 5685 Acked 6684 Write 2 Done Done New false
16779 5686 Acked 6685 Read 7 Done Done New false
16780 5687 Acked 6686 Write 9 Done Done New false
16781 5688 Acked 6687 Read 7 Done Done New false
16782 5689 Acked 6688 Flush 0 Done Done New false
16783 5690 Acked 6689 Read 2 Done Done New false
16784 5691 Acked 6690 Write 1 Done Done New false
16785 5692 Acked 6691 Read 9 Done Done New false
16786 5693 Acked 6692 Write 8 Done Done New false
16787 5694 Acked 6693 Write 7 Done Done New false
16788 5695 Acked 6694 Write 1 Done Done New false
16789 5696 Acked 6695 Read 1 Done Done New false
16790 5697 Acked 6696 Write 3 Done Done New false
16791 5698 Acked 6697 Write 3 Done Done New false
16792 5699 Acked 6698 Write 2 Done Done New false
16793 5700 Acked 6699 Write 7 Done Done New false
16794 5701 Acked 6700 Write 2 Done Done New false
16795 5702 Acked 6701 Write 9 Done Done New false
16796 5703 Acked 6702 Read 7 Done Done New false
16797 5704 Acked 6703 Flush 0 Done Done New false
16798 5705 Acked 6704 Read 7 Done Done New false
16799 5706 Acked 6705 Write 7 Done Done New false
16800 5707 Acked 6706 Write 4 Done Done New false
16801 5708 Acked 6707 Read 4 Done Done New false
16802 5709 Acked 6708 Flush 0 Done Done New false
16803 5710 Acked 6709 Read 9 Done Done New false
16804 5711 Acked 6710 Read 1 Done Done New false
16805 5712 Acked 6711 Read 7 Done Done New false
16806 5713 Acked 6712 Write 8 Done Done New false
16807 5714 Acked 6713 Read 1 Done Done New false
16808 5715 Acked 6714 Write 6 Done Done New false
16809 5716 Acked 6715 Read 5 Done Done New false
16810 5717 Acked 6716 Write 5 Done Done New false
16811 5718 Acked 6717 Read 7 Done Done New false
16812 5719 Acked 6718 Flush 0 Done Done New false
16813 5720 Acked 6719 Write 7 Done Done New false
16814 5721 Acked 6720 Read 1 Done Done New false
16815 5722 Acked 6721 Write 4 Done Done New false
16816 5723 Acked 6722 Read 3 Done Done New false
16817 5724 Acked 6723 Write 8 Done Done New false
16818 5725 Acked 6724 Flush 0 Done Done New false
16819 5726 Acked 6725 Read 2 Done Done New false
16820 STATES DS:0 DS:1 DS:2 TOTAL
16821 New 0 0 300 300
16822 Sent 0 0 0 0
16823 Done 300 300 0 600
16824 Skipped 0 0 0 0
16825 Error 0 0 0 0
16826 Last Flush: 6724 6724 6425
16827 Downstairs last five completed: 6425 6424 6423 6422 6421
16828 Upstairs last five completed: 5726 5725 5724 5723 5722
16829 CLIENT: Up:0 ds:300 act:2
168302024-02-27T21:47:57.345ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
168312024-02-27T21:47:57.345ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
168322024-02-27T21:47:57.345ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
168332024-02-27T21:47:57.345ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
168342024-02-27T21:47:57.345ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
168352024-02-27T21:47:57.345ZINFOcrucible: [2] client re-new 301 jobs since flush 6425 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
168362024-02-27T21:47:57.345ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
168372024-02-27T21:47:58.346ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
168382024-02-27T21:47:58.346ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
168392024-02-27T21:47:58.346ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
168402024-02-27T21:47:58.346ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
168412024-02-27T21:47:58.346ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
168422024-02-27T21:47:58.346ZINFOcrucible: [2] client re-new 301 jobs since flush 6425 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
168432024-02-27T21:47:58.346ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
168442024-02-27T21:47:59.347ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
168452024-02-27T21:47:59.347ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
168462024-02-27T21:47:59.347ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
168472024-02-27T21:47:59.347ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
168482024-02-27T21:47:59.348ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
168492024-02-27T21:47:59.348ZINFOcrucible: [2] client re-new 301 jobs since flush 6425 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
168502024-02-27T21:47:59.348ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
16851 [8830] got start action
16852 [8830] I am going to restart
16853 Starting downstairs at port 8830
16854 Make output file at "/tmp/dsc/downstairs-8830.txt"
16855 [8830][2] reports Starting
168562024-02-27T21:48:00.349ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
168572024-02-27T21:48:00.349ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
168582024-02-27T21:48:00.349ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
168592024-02-27T21:48:00.349ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
168602024-02-27T21:48:00.349ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
168612024-02-27T21:48:00.349ZINFOcrucible: [2] client re-new 301 jobs since flush 6425 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
168622024-02-27T21:48:00.349ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
16863 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1186)
16864 [8830][2] reports Running
16865 ----------------------------------------------------------------
16866 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
16867 States: Active Active Offline
16868 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
16869 5427 Acked 6426 Write 6 Done Done New true
16870 5428 Acked 6427 Write 8 Done Done New true
16871 5429 Acked 6428 Write 9 Done Done New true
16872 5430 Acked 6429 Flush 0 Done Done New true
16873 5431 Acked 6430 Read 7 Done Done New true
16874 5432 Acked 6431 Write 1 Done Done New true
16875 5433 Acked 6432 Flush 0 Done Done New true
16876 5434 Acked 6433 Read 3 Done Done New true
16877 5435 Acked 6434 Write 4 Done Done New true
16878 5436 Acked 6435 Flush 0 Done Done New true
16879 5437 Acked 6436 Write 5 Done Done New true
16880 5438 Acked 6437 Write 1 Done Done New true
16881 5439 Acked 6438 Read 1 Done Done New true
16882 5440 Acked 6439 Write 9 Done Done New true
16883 5441 Acked 6440 Read 5 Done Done New true
16884 5442 Acked 6441 Read 1 Done Done New true
16885 5443 Acked 6442 Read 9 Done Done New true
16886 5444 Acked 6443 Read 3 Done Done New true
16887 5445 Acked 6444 Read 1 Done Done New true
16888 5446 Acked 6445 Read 7 Done Done New true
16889 5447 Acked 6446 Write 3 Done Done New true
16890 5448 Acked 6447 Flush 0 Done Done New true
16891 5449 Acked 6448 Read 8 Done Done New true
16892 5450 Acked 6449 Read 8 Done Done New true
16893 5451 Acked 6450 Flush 0 Done Done New true
16894 5452 Acked 6451 Write 10 Done Done New true
16895 5453 Acked 6452 Flush 0 Done Done New true
16896 5454 Acked 6453 Read 4 Done Done New true
16897 5455 Acked 6454 Read 1 Done Done New true
16898 5456 Acked 6455 Write 9 Done Done New true
16899 5457 Acked 6456 Read 3 Done Done New true
16900 5458 Acked 6457 Write 2 Done Done New true
16901 5459 Acked 6458 Write 3 Done Done New true
16902 5460 Acked 6459 Write 8 Done Done New true
16903 5461 Acked 6460 Flush 0 Done Done New true
16904 5462 Acked 6461 Write 6 Done Done New true
16905 5463 Acked 6462 Read 9 Done Done New true
16906 5464 Acked 6463 Read 6 Done Done New true
16907 5465 Acked 6464 Read 7 Done Done New true
16908 5466 Acked 6465 Read 5 Done Done New true
16909 5467 Acked 6466 Read 3 Done Done New true
16910 5468 Acked 6467 Read 3 Done Done New true
16911 5469 Acked 6468 Read 2 Done Done New true
16912 5470 Acked 6469 Read 4 Done Done New true
16913 5471 Acked 6470 Read 6 Done Done New true
16914 5472 Acked 6471 Read 2 Done Done New true
16915 5473 Acked 6472 Read 5 Done Done New true
16916 5474 Acked 6473 Read 1 Done Done New true
16917 5475 Acked 6474 Write 5 Done Done New true
16918 5476 Acked 6475 Read 4 Done Done New true
16919 5477 Acked 6476 Write 8 Done Done New true
16920 5478 Acked 6477 Write 5 Done Done New true
16921 5479 Acked 6478 Read 4 Done Done New true
16922 5480 Acked 6479 Read 7 Done Done New true
16923 5481 Acked 6480 Write 7 Done Done New true
16924 5482 Acked 6481 Write 7 Done Done New true
16925 5483 Acked 6482 Write 1 Done Done New true
16926 5484 Acked 6483 Flush 0 Done Done New true
16927 5485 Acked 6484 Read 5 Done Done New true
16928 5486 Acked 6485 Write 4 Done Done New true
16929 5487 Acked 6486 Write 7 Done Done New true
16930 5488 Acked 6487 Write 5 Done Done New true
16931 5489 Acked 6488 Read 10 Done Done New true
16932 5490 Acked 6489 Read 5 Done Done New true
16933 5491 Acked 6490 Write 8 Done Done New true
16934 5492 Acked 6491 Write 4 Done Done New true
16935 5493 Acked 6492 Read 9 Done Done New true
16936 5494 Acked 6493 Read 3 Done Done New true
16937 5495 Acked 6494 Write 9 Done Done New true
16938 5496 Acked 6495 Read 2 Done Done New true
16939 5497 Acked 6496 Read 4 Done Done New true
16940 5498 Acked 6497 Read 3 Done Done New true
16941 5499 Acked 6498 Read 6 Done Done New true
16942 5500 Acked 6499 Read 9 Done Done New true
16943 5501 Acked 6500 Flush 0 Done Done New true
16944 5502 Acked 6501 Read 8 Done Done New true
16945 5503 Acked 6502 Write 1 Done Done New true
16946 5504 Acked 6503 Write 6 Done Done New true
16947 5505 Acked 6504 Read 8 Done Done New true
16948 5506 Acked 6505 Write 5 Done Done New true
16949 5507 Acked 6506 Read 3 Done Done New true
16950 5508 Acked 6507 Read 6 Done Done New true
16951 5509 Acked 6508 Write 6 Done Done New true
16952 5510 Acked 6509 Read 10 Done Done New true
16953 5511 Acked 6510 Flush 0 Done Done New true
16954 5512 Acked 6511 Write 1 Done Done New true
16955 5513 Acked 6512 Read 7 Done Done New true
16956 5514 Acked 6513 Read 6 Done Done New true
16957 5515 Acked 6514 Write 10 Done Done New true
16958 5516 Acked 6515 Write 3 Done Done New true
16959 5517 Acked 6516 Read 8 Done Done New true
16960 5518 Acked 6517 Write 1 Done Done New true
16961 5519 Acked 6518 Write 5 Done Done New true
16962 5520 Acked 6519 Read 4 Done Done New true
16963 5521 Acked 6520 Write 5 Done Done New true
16964 5522 Acked 6521 Write 5 Done Done New true
16965 5523 Acked 6522 Write 6 Done Done New true
16966 5524 Acked 6523 Read 8 Done Done New true
16967 5525 Acked 6524 Read 10 Done Done New true
16968 5526 Acked 6525 Read 7 Done Done New true
16969 5527 Acked 6526 Read 9 Done Done New true
16970 5528 Acked 6527 Flush 0 Done Done New true
16971 5529 Acked 6528 Write 2 Done Done New true
16972 5530 Acked 6529 Write 4 Done Done New true
16973 5531 Acked 6530 Read 2 Done Done New true
16974 5532 Acked 6531 Write 10 Done Done New true
16975 5533 Acked 6532 Flush 0 Done Done New true
16976 5534 Acked 6533 Read 4 Done Done New true
16977 5535 Acked 6534 Write 5 Done Done New true
16978 5536 Acked 6535 Write 2 Done Done New true
16979 5537 Acked 6536 Read 9 Done Done New true
16980 5538 Acked 6537 Read 8 Done Done New true
16981 5539 Acked 6538 Write 7 Done Done New true
16982 5540 Acked 6539 Flush 0 Done Done New true
16983 5541 Acked 6540 Read 7 Done Done New true
16984 5542 Acked 6541 Read 3 Done Done New true
16985 5543 Acked 6542 Read 6 Done Done New true
16986 5544 Acked 6543 Read 5 Done Done New true
16987 5545 Acked 6544 Read 1 Done Done New true
16988 5546 Acked 6545 Flush 0 Done Done New true
16989 5547 Acked 6546 Write 8 Done Done New true
16990 5548 Acked 6547 Write 5 Done Done New true
16991 5549 Acked 6548 Write 6 Done Done New true
16992 5550 Acked 6549 Read 4 Done Done New true
16993 5551 Acked 6550 Write 9 Done Done New true
16994 5552 Acked 6551 Flush 0 Done Done New true
16995 5553 Acked 6552 Read 5 Done Done New true
16996 5554 Acked 6553 Read 4 Done Done New true
16997 5555 Acked 6554 Read 10 Done Done New true
16998 5556 Acked 6555 Read 5 Done Done New true
16999 5557 Acked 6556 Write 4 Done Done New true
17000 5558 Acked 6557 Read 10 Done Done New true
17001 5559 Acked 6558 Write 10 Done Done New true
17002 5560 Acked 6559 Flush 0 Done Done New true
17003 5561 Acked 6560 Read 1 Done Done New true
17004 5562 Acked 6561 Read 4 Done Done New true
17005 5563 Acked 6562 Write 9 Done Done New true
17006 5564 Acked 6563 Write 10 Done Done New true
17007 5565 Acked 6564 Read 5 Done Done New true
17008 5566 Acked 6565 Read 9 Done Done New true
17009 5567 Acked 6566 Write 10 Done Done New true
17010 5568 Acked 6567 Read 5 Done Done New true
17011 5569 Acked 6568 Write 8 Done Done New true
17012 5570 Acked 6569 Write 6 Done Done New true
17013 5571 Acked 6570 Write 4 Done Done New true
17014 5572 Acked 6571 Write 2 Done Done New true
17015 5573 Acked 6572 Read 1 Done Done New true
17016 5574 Acked 6573 Write 10 Done Done New true
17017 5575 Acked 6574 Read 9 Done Done New true
17018 5576 Acked 6575 Write 10 Done Done New true
17019 5577 Acked 6576 Write 4 Done Done New true
17020 5578 Acked 6577 Read 8 Done Done New true
17021 5579 Acked 6578 Read 5 Done Done New true
17022 5580 Acked 6579 Write 1 Done Done New true
17023 5581 Acked 6580 Write 4 Done Done New true
17024 5582 Acked 6581 Write 6 Done Done New true
17025 5583 Acked 6582 Read 2 Done Done New true
17026 5584 Acked 6583 Read 8 Done Done New true
17027 5585 Acked 6584 Write 6 Done Done New true
17028 5586 Acked 6585 Flush 0 Done Done New true
17029 5587 Acked 6586 Write 4 Done Done New true
17030 5588 Acked 6587 Write 4 Done Done New true
17031 5589 Acked 6588 Write 10 Done Done New true
17032 5590 Acked 6589 Read 7 Done Done New true
17033 5591 Acked 6590 Flush 0 Done Done New true
17034 5592 Acked 6591 Write 9 Done Done New true
17035 5593 Acked 6592 Write 5 Done Done New true
17036 5594 Acked 6593 Read 7 Done Done New true
17037 5595 Acked 6594 Read 2 Done Done New true
17038 5596 Acked 6595 Write 10 Done Done New true
17039 5597 Acked 6596 Write 5 Done Done New true
17040 5598 Acked 6597 Write 3 Done Done New true
17041 5599 Acked 6598 Write 6 Done Done New true
17042 5600 Acked 6599 Read 10 Done Done New true
17043 5601 Acked 6600 Read 8 Done Done New true
17044 5602 Acked 6601 Flush 0 Done Done New true
17045 5603 Acked 6602 Read 9 Done Done New true
17046 5604 Acked 6603 Write 5 Done Done New true
17047 5605 Acked 6604 Write 6 Done Done New true
17048 5606 Acked 6605 Write 4 Done Done New true
17049 5607 Acked 6606 Read 3 Done Done New true
17050 5608 Acked 6607 Write 1 Done Done New true
17051 5609 Acked 6608 Write 9 Done Done New true
17052 5610 Acked 6609 Write 2 Done Done New true
17053 5611 Acked 6610 Read 4 Done Done New true
17054 5612 Acked 6611 Write 7 Done Done New true
17055 5613 Acked 6612 Write 7 Done Done New true
17056 5614 Acked 6613 Read 4 Done Done New true
17057 5615 Acked 6614 Write 8 Done Done New true
17058 5616 Acked 6615 Read 1 Done Done New true
17059 5617 Acked 6616 Read 5 Done Done New true
17060 5618 Acked 6617 Read 10 Done Done New true
17061 5619 Acked 6618 Read 2 Done Done New true
17062 5620 Acked 6619 Read 2 Done Done New true
17063 5621 Acked 6620 Write 5 Done Done New true
17064 5622 Acked 6621 Read 10 Done Done New true
17065 5623 Acked 6622 Write 3 Done Done New true
17066 5624 Acked 6623 Read 7 Done Done New true
17067 5625 Acked 6624 Write 8 Done Done New true
17068 5626 Acked 6625 Write 5 Done Done New true
17069 5627 Acked 6626 Read 7 Done Done New true
17070 5628 Acked 6627 Read 8 Done Done New true
17071 5629 Acked 6628 Write 10 Done Done New true
17072 5630 Acked 6629 Read 7 Done Done New true
17073 5631 Acked 6630 Write 10 Done Done New true
17074 5632 Acked 6631 Read 6 Done Done New true
17075 5633 Acked 6632 Read 3 Done Done New true
17076 5634 Acked 6633 Write 9 Done Done New true
17077 5635 Acked 6634 Write 7 Done Done New true
17078 5636 Acked 6635 Flush 0 Done Done New true
17079 5637 Acked 6636 Read 6 Done Done New true
17080 5638 Acked 6637 Read 1 Done Done New true
17081 5639 Acked 6638 Write 3 Done Done New true
17082 5640 Acked 6639 Write 2 Done Done New true
17083 5641 Acked 6640 Read 10 Done Done New true
17084 5642 Acked 6641 Read 2 Done Done New true
17085 5643 Acked 6642 Write 7 Done Done New true
17086 5644 Acked 6643 Write 2 Done Done New true
17087 5645 Acked 6644 Read 3 Done Done New true
17088 5646 Acked 6645 Read 8 Done Done New true
17089 5647 Acked 6646 Read 8 Done Done New true
17090 5648 Acked 6647 Read 5 Done Done New true
17091 5649 Acked 6648 Read 5 Done Done New true
17092 5650 Acked 6649 Write 2 Done Done New true
17093 5651 Acked 6650 Read 7 Done Done New true
17094 5652 Acked 6651 Write 1 Done Done New true
17095 5653 Acked 6652 Read 4 Done Done New true
17096 5654 Acked 6653 Read 9 Done Done New true
17097 5655 Acked 6654 Read 6 Done Done New true
17098 5656 Acked 6655 Read 9 Done Done New true
17099 5657 Acked 6656 Read 6 Done Done New true
17100 5658 Acked 6657 Read 4 Done Done New true
17101 5659 Acked 6658 Write 6 Done Done New true
17102 5660 Acked 6659 Write 3 Done Done New true
17103 5661 Acked 6660 Read 5 Done Done New true
17104 5662 Acked 6661 Read 2 Done Done New true
17105 5663 Acked 6662 Write 8 Done Done New true
17106 5664 Acked 6663 Read 10 Done Done New true
17107 5665 Acked 6664 Flush 0 Done Done New true
17108 5666 Acked 6665 Write 6 Done Done New true
17109 5667 Acked 6666 Read 10 Done Done New true
17110 5668 Acked 6667 Read 5 Done Done New true
17111 5669 Acked 6668 Read 2 Done Done New true
17112 5670 Acked 6669 Write 4 Done Done New true
17113 5671 Acked 6670 Write 4 Done Done New true
17114 5672 Acked 6671 Read 9 Done Done New true
17115 5673 Acked 6672 Flush 0 Done Done New true
17116 5674 Acked 6673 Read 8 Done Done New true
17117 5675 Acked 6674 Write 1 Done Done New true
17118 5676 Acked 6675 Write 8 Done Done New true
17119 5677 Acked 6676 Read 8 Done Done New true
17120 5678 Acked 6677 Read 1 Done Done New true
17121 5679 Acked 6678 Read 9 Done Done New true
17122 5680 Acked 6679 Flush 0 Done Done New true
17123 5681 Acked 6680 Write 4 Done Done New true
17124 5682 Acked 6681 Write 2 Done Done New true
17125 5683 Acked 6682 Read 8 Done Done New true
17126 5684 Acked 6683 Write 8 Done Done New true
17127 5685 Acked 6684 Write 2 Done Done New true
17128 5686 Acked 6685 Read 7 Done Done New true
17129 5687 Acked 6686 Write 9 Done Done New true
17130 5688 Acked 6687 Read 7 Done Done New true
17131 5689 Acked 6688 Flush 0 Done Done New true
17132 5690 Acked 6689 Read 2 Done Done New true
17133 5691 Acked 6690 Write 1 Done Done New true
17134 5692 Acked 6691 Read 9 Done Done New true
17135 5693 Acked 6692 Write 8 Done Done New true
17136 5694 Acked 6693 Write 7 Done Done New true
17137 5695 Acked 6694 Write 1 Done Done New true
17138 5696 Acked 6695 Read 1 Done Done New true
17139 5697 Acked 6696 Write 3 Done Done New true
17140 5698 Acked 6697 Write 3 Done Done New true
17141 5699 Acked 6698 Write 2 Done Done New true
17142 5700 Acked 6699 Write 7 Done Done New true
17143 5701 Acked 6700 Write 2 Done Done New true
17144 5702 Acked 6701 Write 9 Done Done New true
17145 5703 Acked 6702 Read 7 Done Done New true
17146 5704 Acked 6703 Flush 0 Done Done New true
17147 5705 Acked 6704 Read 7 Done Done New true
17148 5706 Acked 6705 Write 7 Done Done New true
17149 5707 Acked 6706 Write 4 Done Done New true
17150 5708 Acked 6707 Read 4 Done Done New true
17151 5709 Acked 6708 Flush 0 Done Done New true
17152 5710 Acked 6709 Read 9 Done Done New true
17153 5711 Acked 6710 Read 1 Done Done New true
17154 5712 Acked 6711 Read 7 Done Done New true
17155 5713 Acked 6712 Write 8 Done Done New true
17156 5714 Acked 6713 Read 1 Done Done New true
17157 5715 Acked 6714 Write 6 Done Done New true
17158 5716 Acked 6715 Read 5 Done Done New true
17159 5717 Acked 6716 Write 5 Done Done New true
17160 5718 Acked 6717 Read 7 Done Done New true
17161 5719 Acked 6718 Flush 0 Done Done New true
17162 5720 Acked 6719 Write 7 Done Done New true
17163 5721 Acked 6720 Read 1 Done Done New true
17164 5722 Acked 6721 Write 4 Done Done New true
17165 5723 Acked 6722 Read 3 Done Done New true
17166 5724 Acked 6723 Write 8 Done Done New true
17167 5725 Acked 6724 Flush 0 Done Done New true
17168 5726 Acked 6725 Read 2 Done Done New true
17169 5727 Acked 6726 Flush 0 Done Done New true
17170 STATES DS:0 DS:1 DS:2 TOTAL
17171 New 0 0 301 301
17172 Sent 0 0 0 0
17173 Done 301 301 0 602
17174 Skipped 0 0 0 0
17175 Error 0 0 0 0
17176 Last Flush: 6726 6726 6425
17177 Downstairs last five completed: 6425 6424 6423 6422 6421
17178 Upstairs last five completed: 5727 5726 5725 5724 5723
17179 CLIENT: Up:0 ds:301 act:2
171802024-02-27T21:48:01.350ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
171812024-02-27T21:48:01.350ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
171822024-02-27T21:48:01.352ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
171832024-02-27T21:48:01.352ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
171842024-02-27T21:48:01.352ZINFOcrucible: send last flush ID to this DS: 6425 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
171852024-02-27T21:48:01.352ZINFOcrucible: Replied this last flush ID: 6425 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
171862024-02-27T21:48:01.352ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
171872024-02-27T21:48:01.352ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
171882024-02-27T21:48:01.352ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
171892024-02-27T21:48:01.352ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
17190 ----------------------------------------------------------------
17191 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
17192 Downstairs last five completed: 6726 6725 6724 6723 6722
17193 Upstairs last five completed: 5727 5726 5725 5724 5723
17194 CLIENT: Up:0 ds:0 act:3
17195 Replay: All jobs finished, all DS active.
17196 Main task has work to do, go find it
17197 got dsc Stop(2)
17198 stop 2
17199 Feb 27 21:48:04.692 INFO request completed[8830] Got stop action so:false kr:false
17200 , latency_us: 346, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 308b04ed-74e3-4f29-bbc7-d08fa545f1ef, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
17201 Feb 27 21:48:04.692 INFO request completed, latency_us{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-27T21:48:04.692884746Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"io task","client":"2","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
17202 : 128, response_code: 200, uri: /state/cid/2, method: GET, req_id: 5dc26672-203c-4a0d-9a68-361725729349, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
172032024-02-27T21:48:04.693ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
172042024-02-27T21:48:04.693ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = e6a5a974-925a-494f-8845-facc441c2690
172052024-02-27T21:48:04.693ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
172062024-02-27T21:48:04.693ZINFOcrucible: [2] client re-new 0 jobs since flush 6726 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
172072024-02-27T21:48:04.693ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
17208 [8830] Exited with: ExitStatus(unix_wait_status(9))
17209 [8830][2] reports Exit
172102024-02-27T21:48:05.694ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
172112024-02-27T21:48:05.694ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
172122024-02-27T21:48:05.694ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
172132024-02-27T21:48:05.694ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
172142024-02-27T21:48:05.694ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
172152024-02-27T21:48:05.694ZINFOcrucible: [2] client re-new 0 jobs since flush 6726 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
172162024-02-27T21:48:05.694ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
172172024-02-27T21:48:06.695ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
172182024-02-27T21:48:06.695ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
172192024-02-27T21:48:06.695ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
172202024-02-27T21:48:06.695ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
172212024-02-27T21:48:06.695ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
172222024-02-27T21:48:06.695ZINFOcrucible: [2] client re-new 0 jobs since flush 6726 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
172232024-02-27T21:48:06.695ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
172242024-02-27T21:48:07.696ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
172252024-02-27T21:48:07.696ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
172262024-02-27T21:48:07.696ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
172272024-02-27T21:48:07.696ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
172282024-02-27T21:48:07.696ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
172292024-02-27T21:48:07.696ZINFOcrucible: [2] client re-new 0 jobs since flush 6726 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
172302024-02-27T21:48:07.696ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
17231 Feb 27 21:48:08.695 INFO request completed, latency_us: 204, response_code: 200, uri: /state/cid/2, method: GET, req_id: 3bcf6836-3711-43e4-aa7c-0e114cde45e2, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
17232 001/300 Flush
17233 002/300 Flush
172342024-02-27T21:48:08.697ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
172352024-02-27T21:48:08.697ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
172362024-02-27T21:48:08.697ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
172372024-02-27T21:48:08.697ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
172382024-02-27T21:48:08.697ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
172392024-02-27T21:48:08.697ZINFOcrucible: [2] client re-new 2 jobs since flush 6726 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
172402024-02-27T21:48:08.697ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
17241 003/300 Write block 3641 len 40960 data: 7 6 4 4 2 2 2 2 2 2
17242 004/300 Read block 1556 len 16384
17243 005/300 Write block 4713 len 20480 data: 3 3 3 5 3
17244 006/300 Read block 3541 len 16384
17245 007/300 Flush
17246 008/300 Read block 1227 len 16384
17247 009/300 Read block 3284 len 20480
17248 010/300 Read block 986 len 32768
17249 011/300 Read block 15 len 32768
17250 012/300 Read block 776 len 40960
17251 013/300 Read block 4736 len 12288
17252 014/300 Read block 4305 len 32768
17253 015/300 Write block 3001 len 40960 data: 8 8 6 4 4 4 5 5 5 5
17254 016/300 Write block 728 len 20480 data: 4 4 5 5 6
17255 017/300 Write block 2019 len 24576 data: 3 3 3 4 5 7
17256 018/300 Flush
17257 019/300 Read block 61 len 36864
17258 020/300 Read block 552 len 8192
17259 021/300 Read block 1735 len 40960
17260 022/300 Read block 3779 len 36864
17261 023/300 Read block 4523 len 24576
17262 024/300 Read block 2550 len 16384
17263 025/300 Write block 1728 len 32768 data: 3 3 3 3 3 3 3 3
17264 026/300 Read block 1547 len 28672
17265 027/300 Read block 1659 len 32768
17266 028/300 Write block 449 len 24576 data: 5 6 7 6 5 5
17267 029/300 Read block 1005 len 32768
17268 030/300 Read block 3197 len 36864
17269 031/300 Read block 3122 len 40960
17270 032/300 Read block 790 len 4096
17271 033/300 Read block 4270 len 24576
17272 034/300 Write block 181 len 28672 data: 4 5 5 4 4 5 3
17273 035/300 Read block 2695 len 4096
17274 036/300 Write block 1814 len 36864 data: 5 5 4 4 4 3 3 3 3
17275 037/300 Read block 3807 len 12288
17276 038/300 Write block 245 len 16384 data: 5 5 5 5
17277 039/300 Read block 685 len 40960
17278 040/300 Write block 2888 len 32768 data: 4 4 3 3 3 3 2 2
17279 041/300 Flush
17280 042/300 Write block 2601 len 12288 data: 5 5 5
17281 043/300 Read block 2542 len 16384
17282 044/300 Read block 3166 len 28672
17283 045/300 Read block 755 len 40960
17284 046/300 Write block 2778 len 28672 data: 3 5 7 7 7 4 3
17285 047/300 Write block 2856 len 20480 data: 3 2 4 4 3
17286 048/300 Write block 3911 len 40960 data: 3 2 2 4 4 5 6 5 4 5
17287 049/300 Write block 1611 len 16384 data: 3 4 5 5
17288 050/300 Write block 3326 len 12288 data: 6 7 7
17289 051/300 Read block 231 len 8192
17290 052/300 Write block 2856 len 4096 data: 4
17291 053/300 Read block 1377 len 28672
17292 054/300 Read block 1963 len 36864
17293 055/300 Write block 980 len 20480 data: 3 3 4 6 5
17294 056/300 Read block 27 len 16384
17295 057/300 Write block 1554 len 4096 data: 4
17296 058/300 Write block 2632 len 36864 data: 3 3 2 2 3 4 4 4 4
17297 059/300 Read block 437 len 40960
17298 060/300 Read block 4596 len 40960
17299 061/300 Read block 948 len 4096
17300 062/300 Write block 2326 len 12288 data: 6 6 4
17301 063/300 Flush
17302 064/300 Read block 3225 len 20480
17303 065/300 Write block 2673 len 36864 data: 6 5 4 4 4 5 6 7 7
17304 066/300 Write block 4017 len 20480 data: 5 7 7 7 7
17305 067/300 Read block 3650 len 16384
17306 068/300 Write block 2383 len 4096 data: 4
17307 069/300 Flush
17308 070/300 Write block 2650 len 8192 data: 6 5
17309 071/300 Flush
17310 072/300 Flush
17311 073/300 Write block 2830 len 4096 data: 4
17312 074/300 Read block 2006 len 20480
17313 075/300 Write block 4959 len 36864 data: 4 4 2 2 2 4 4 4 4
17314 076/300 Read block 2360 len 24576
17315 077/300 Read block 3912 len 8192
17316 078/300 Write block 1970 len 12288 data: 5 6 5
17317 079/300 Read block 2048 len 4096
17318 080/300 Read block 1319 len 32768
17319 081/300 Read block 476 len 24576
17320 082/300 Write block 2817 len 36864 data: 8 9 9 9 8 5 6 6 6
17321 083/300 Read block 4231 len 4096
17322 084/300 Write block 4892 len 28672 data: 6 5 5 4 3 3 3
17323 085/300 Write block 4875 len 36864 data: 5 6 5 5 4 4 4 4 5
17324 086/300 Write block 4515 len 24576 data: 4 4 4 4 4 4
17325 087/300 Write block 3163 len 24576 data: 4 6 6 5 4 5
17326 088/300 Write block 2010 len 28672 data: 3 3 2 2 4 4 3
17327 089/300 Write block 391 len 28672 data: 3 5 5 6 6 6 6
17328 090/300 Read block 252 len 32768
17329 091/300 Read block 4648 len 20480
17330 092/300 Write block 4872 len 40960 data: 6 5 5 6 7 6 6 5 5 5
17331 093/300 Read block 269 len 16384
17332 094/300 Read block 2305 len 8192
17333 095/300 Write block 4141 len 24576 data: 5 5 5 5 5 4
17334 096/300 Read block 2647 len 24576
17335 097/300 Write block 819 len 32768 data: 4 3 3 6 7 8 7 8
17336 098/300 Read block 4763 len 20480
17337 099/300 Read block 3119 len 8192
17338 100/300 Read block 4478 len 16384
17339 101/300 Read block 3379 len 28672
17340 102/300 Read block 2294 len 20480
17341 103/300 Read block 3737 len 36864
17342 104/300 Read block 1231 len 36864
17343 105/300 Write block 1781 len 8192 data: 3 3
17344 106/300 Write block 2835 len 40960 data: 4 4 3 2 2 3 3 3 5 5
17345 107/300 Write block 3372 len 32768 data: 5 6 7 7 5 3 2 2
17346 108/300 Read block 2645 len 16384
17347 109/300 Read block 2778 len 4096
17348 110/300 Write block 4513 len 4096 data: 6
17349 111/300 Write block 1084 len 12288 data: 9 7 5
17350 112/300 Flush
17351 113/300 Read block 2510 len 4096
17352 114/300 Write block 1886 len 24576 data: 4 4 4 4 5 4
17353 115/300 Write block 606 len 32768 data: 3 5 5 5 4 4 5 5
17354 116/300 Write block 2417 len 40960 data: 3 2 2 2 3 3 4 5 5 5
17355 117/300 Read block 3912 len 12288
17356 118/300 Read block 3530 len 32768
17357 119/300 Read block 4631 len 8192
17358 120/300 Write block 695 len 32768 data: 5 4 4 4 4 4 3 3
17359 121/300 Write block 3364 len 8192 data: 2 2
17360 122/300 Write block 4882 len 36864 data: 5 6 4 5 5 5 4 4 3
17361 123/300 Read block 1772 len 24576
17362 124/300 Write block 3620 len 24576 data: 4 4 4 3 3 3
17363 125/300 Read block 825 len 12288
17364 126/300 Write block 2473 len 28672 data: 6 6 4 4 3 4 5
17365 127/300 Write block 739 len 8192 data: 5 5
17366 128/300 Write block 2193 len 32768 data: 6 6 4 5 7 7 8 8
17367 129/300 Read block 593 len 8192
17368 130/300 Write block 4712 len 28672 data: 5 4 4 4 6 4 3
17369 131/300 Write block 2288 len 20480 data: 4 4 4 4 4
17370 132/300 Read block 1458 len 32768
17371 133/300 Write block 3983 len 12288 data: 5 4 4
17372 134/300 Read block 2969 len 32768
17373 135/300 Read block 2242 len 8192
17374 136/300 Read block 4988 len 36864
17375 137/300 Read block 3910 len 16384
17376 138/300 Read block 427 len 4096
17377 139/300 Write block 675 len 20480 data: 8 8 8 8 5
17378 140/300 Write block 3341 len 36864 data: 4 4 5 6 7 7 6 5 5
17379 141/300 Write block 4368 len 24576 data: 6 5 5 4 4 4
17380 142/300 Read block 2828 len 4096
17381 143/300 Write block 3773 len 40960 data: 3 4 4 4 4 4 4 3 2 2
17382 144/300 Read block 4277 len 40960
17383 145/300 Write block 760 len 24576 data: 3 3 3 5 5 6
17384 146/300 Read block 746 len 4096
17385 147/300 Read block 1875 len 16384
17386 148/300 Write block 4746 len 36864 data: 6 6 8 9 8 5 6 5 5
17387 149/300 Read block 2620 len 8192
17388 150/300 Read block 363 len 24576
17389 151/300 Write block 2614 len 4096 data: 5
17390 152/300 Read block 2213 len 16384
17391 153/300 Write block 375 len 36864 data: 3 3 3 3 3 3 2 2 3
17392 154/300 Write block 4809 len 4096 data: 4
17393 155/300 Read block 3336 len 36864
17394 156/300 Read block 3629 len 4096
17395 157/300 Write block 3081 len 40960 data: 2 2 2 2 2 2 2 2 2 2
17396 158/300 Write block 1746 len 28672 data: 3 3 4 4 4 4 5
17397 159/300 Flush
17398 160/300 Read block 340 len 36864
17399 161/300 Read block 1610 len 28672
17400 162/300 Read block 3913 len 28672
17401 163/300 Write block 9 len 24576 data: 4 5 5 5 5 4
17402 164/300 Write block 4962 len 40960 data: 3 3 5 5 5 5 4 6 7 6
17403 165/300 Read block 2196 len 40960
17404 166/300 Read block 316 len 40960
17405 167/300 Read block 4133 len 36864
17406 168/300 Write block 3702 len 16384 data: 3 3 4 4
17407 169/300 Flush
17408 170/300 Read block 2873 len 4096
17409 171/300 Read block 11 len 8192
17410 172/300 Read block 4261 len 40960
17411 173/300 Write block 2608 len 28672 data: 3 3 4 4 4 6 6
17412 174/300 Read block 3297 len 24576
17413 175/300 Read block 489 len 4096
17414 176/300 Write block 2972 len 20480 data: 4 5 4 5 6
17415 177/300 Read block 1574 len 36864
17416 178/300 Read block 2864 len 40960
17417 179/300 Write block 4337 len 32768 data: 4 4 4 3 3 2 2 2
17418 180/300 Write block 4603 len 36864 data: 5 4 3 3 3 5 4 4 5
17419 181/300 Read block 3869 len 28672
17420 182/300 Write block 3433 len 24576 data: 2 3 3 3 3 3
17421 183/300 Read block 1799 len 32768
17422 184/300 Read block 2292 len 24576
17423 185/300 Write block 4894 len 12288 data: 6 5 4
17424 186/300 Write block 3135 len 28672 data: 2 2 2 3 3 3 3
17425 187/300 Read block 451 len 36864
17426 188/300 Read block 4910 len 4096
17427 189/300 Write block 4516 len 12288 data: 5 5 5
17428 190/300 Write block 2129 len 36864 data: 4 4 4 5 3 4 4 4 6
17429 191/300 Flush
17430 192/300 Flush
17431 193/300 Write block 4033 len 8192 data: 3 3
17432 194/300 Flush
17433 195/300 Read block 1626 len 16384
17434 196/300 Write block 4119 len 36864 data: 3 4 4 4 4 4 4 3 4
17435 197/300 Write block 629 len 16384 data: 3 4 4 3
17436 198/300 Write block 2709 len 8192 data: 5 3
17437 199/300 Read block 1481 len 24576
17438 200/300 Write block 2955 len 36864 data: 5 6 6 7 5 4 4 5 6
17439 201/300 Read block 3627 len 32768
17440 202/300 Read block 2046 len 16384
17441 203/300 Read block 4282 len 32768
17442 204/300 Read block 1444 len 20480
17443 205/300 Read block 1716 len 16384
17444 206/300 Write block 3142 len 16384 data: 3 3 3 5
17445 207/300 Read block 1922 len 28672
17446 208/300 Read block 3747 len 24576
17447 209/300 Write block 2622 len 20480 data: 5 6 8 8 9
17448 210/300 Read block 730 len 40960
17449 211/300 Read block 1739 len 16384
17450 212/300 Read block 1119 len 8192
17451 213/300 Read block 1710 len 36864
17452 214/300 Read block 930 len 36864
17453 215/300 Write block 2802 len 40960 data: 5 5 5 5 3 4 3 5 7 7
17454 216/300 Flush
17455 217/300 Read block 2250 len 16384
17456 218/300 Flush
17457 219/300 Read block 4021 len 32768
17458 220/300 Write block 1347 len 24576 data: 3 3 3 4 5 7
17459 221/300 Read block 2496 len 12288
17460 222/300 Read block 1328 len 32768
17461 223/300 Flush
17462 224/300 Write block 3228 len 12288 data: 4 5 5
17463 225/300 Read block 772 len 36864
17464 226/300 Write block 2844 len 16384 data: 6 7 7 7
17465 227/300 Write block 2246 len 16384 data: 4 2 2 2
17466 228/300 Write block 4080 len 20480 data: 3 2 3 3 3
17467 229/300 Read block 4772 len 8192
17468 230/300 Write block 2772 len 16384 data: 4 3 3 3
17469 231/300 Read block 649 len 4096
17470 232/300 Write block 1275 len 24576 data: 2 2 2 2 2 3
17471 233/300 Write block 2598 len 4096 data: 5
17472 234/300 Read block 3502 len 36864
17473 235/300 Write block 3357 len 24576 data: 3 3 3 3 3 2
17474 236/300 Write block 2485 len 20480 data: 6 6 6 7 7
17475 237/300 Flush
17476 238/300 Write block 3124 len 12288 data: 5 5 4
17477 239/300 Read block 3653 len 12288
17478 240/300 Flush
17479 241/300 Read block 389 len 20480
17480 242/300 Read block 396 len 4096
17481 243/300 Write block 711 len 12288 data: 5 4 4
17482 244/300 Read block 838 len 8192
17483 245/300 Read block 3324 len 40960
17484 246/300 Read block 3166 len 4096
17485 247/300 Write block 3595 len 32768 data: 6 5 5 4 4 5 5 5
17486 248/300 Read block 360 len 4096
17487 249/300 Write block 1041 len 8192 data: 5 4
17488 250/300 Read block 2107 len 16384
17489 251/300 Read block 1323 len 36864
17490 252/300 Write block 2925 len 16384 data: 3 3 4 3
17491 253/300 Read block 1862 len 24576
17492 254/300 Write block 401 len 24576 data: 6 5 4 4 4 4
17493 255/300 Read block 941 len 40960
17494 256/300 Write block 3418 len 4096 data: 6
17495 257/300 Write block 4507 len 36864 data: 5 4 3 3 5 6 7 5 5
17496 258/300 Write block 2957 len 8192 data: 7 8
17497 259/300 Write block 1708 len 36864 data: 4 5 7 7 10 10 7 7 7
17498 260/300 Flush
17499 261/300 Read block 2573 len 36864
17500 262/300 Write block 2874 len 16384 data: 6 5 4 4
17501 263/300 Read block 3468 len 8192
17502 264/300 Read block 1795 len 24576
17503 265/300 Read block 52 len 16384
17504 266/300 Write block 2856 len 32768 data: 5 3 5 5 4 4 4 4
17505 267/300 Read block 4516 len 32768
17506 268/300 Read block 4373 len 40960
17507 269/300 Write block 4318 len 28672 data: 6 7 7 5 3 4 4
17508 270/300 Read block 2595 len 32768
17509 271/300 Read block 1672 len 40960
17510 272/300 Read block 119 len 4096
17511 273/300 Write block 1191 len 36864 data: 8 6 5 6 6 7 5 5 4
17512 274/300 Flush
17513 275/300 Write block 3800 len 8192 data: 8 7
17514 276/300 Write block 1226 len 24576 data: 6 5 6 6 6 5
17515 277/300 Read block 7 len 20480
17516 278/300 Read block 2747 len 36864
17517 279/300 Write block 4393 len 28672 data: 4 5 3 4 7 9 10
17518 280/300 Write block 4869 len 4096 data: 4
17519 281/300 Read block 3047 len 4096
17520 282/300 Write block 4475 len 16384 data: 4 5 5 6
17521 283/300 Read block 500 len 16384
17522 284/300 Read block 3191 len 24576
17523 285/300 Read block 2172 len 4096
17524 286/300 Read block 1341 len 12288
17525 287/300 Read block 4976 len 32768
17526 288/300 Write block 1613 len 32768 data: 6 6 7 8 6 5 6 6
17527 289/300 Read block 4782 len 20480
17528 290/300 Read block 3305 len 20480
17529 291/300 Read block 206 len 16384
17530 292/300 Read block 815 len 16384
17531 293/300 Write block 1290 len 8192 data: 4 4
17532 294/300 Read block 922 len 28672
17533 295/300 Read block 3911 len 40960
17534 296/300 Write block 3503 len 8192 data: 3 3
17535 297/300 Read block 3965 len 40960
17536 298/300 Write block 3698 len 16384 data: 5 5 4 4
17537 299/300 Read block 1738 len 4096
17538 300/300 Read block 2668 len 40960
17539 Main task has work to do, go find it
17540 got dsc Start(2)
17541 start 2
17542 Feb 27 21:48:09.038 INFO request completed, latency_us: 210, response_code: 204, uri: /start/cid/2, method: POST, req_id: 446cb701-016e-4f9e-8b47-f62476471572, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
17543 Replay: started 2, returned:Ok(())
17544 ----------------------------------------------------------------
17545 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17546 States: Active Active Offline
17547 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
17548 5728 Acked 6727 Flush 0 Done Done New true
17549 5729 Acked 6728 Flush 0 Done Done New true
17550 5730 Acked 6729 Write 10 Done Done New false
17551 5731 Acked 6730 Read 4 Done Done New false
17552 5732 Acked 6731 Write 5 Done Done New false
17553 5733 Acked 6732 Read 4 Done Done New false
17554 5734 Acked 6733 Flush 0 Done Done New false
17555 5735 Acked 6734 Read 4 Done Done New false
17556 5736 Acked 6735 Read 5 Done Done New false
17557 5737 Acked 6736 Read 8 Done Done New false
17558 5738 Acked 6737 Read 8 Done Done New false
17559 5739 Acked 6738 Read 10 Done Done New false
17560 5740 Acked 6739 Read 3 Done Done New false
17561 5741 Acked 6740 Read 8 Done Done New false
17562 5742 Acked 6741 Write 10 Done Done New false
17563 5743 Acked 6742 Write 5 Done Done New false
17564 5744 Acked 6743 Write 6 Done Done New false
17565 5745 Acked 6744 Flush 0 Done Done New false
17566 5746 Acked 6745 Read 9 Done Done New false
17567 5747 Acked 6746 Read 2 Done Done New false
17568 5748 Acked 6747 Read 10 Done Done New false
17569 5749 Acked 6748 Read 9 Done Done New false
17570 5750 Acked 6749 Read 6 Done Done New false
17571 5751 Acked 6750 Read 4 Done Done New false
17572 5752 Acked 6751 Write 8 Done Done New false
17573 5753 Acked 6752 Read 7 Done Done New false
17574 5754 Acked 6753 Read 8 Done Done New false
17575 5755 Acked 6754 Write 6 Done Done New false
17576 5756 Acked 6755 Read 8 Done Done New false
17577 5757 Acked 6756 Read 9 Done Done New false
17578 5758 Acked 6757 Read 10 Done Done New false
17579 5759 Acked 6758 Read 1 Done Done New false
17580 5760 Acked 6759 Read 6 Done Done New false
17581 5761 Acked 6760 Write 7 Done Done New false
17582 5762 Acked 6761 Read 1 Done Done New false
17583 5763 Acked 6762 Write 9 Done Done New false
17584 5764 Acked 6763 Read 3 Done Done New false
17585 5765 Acked 6764 Write 4 Done Done New false
17586 5766 Acked 6765 Read 10 Done Done New false
17587 5767 Acked 6766 Write 8 Done Done New false
17588 5768 Acked 6767 Flush 0 Done Done New false
17589 5769 Acked 6768 Write 3 Done Done New false
17590 5770 Acked 6769 Read 4 Done Done New false
17591 5771 Acked 6770 Read 7 Done Done New false
17592 5772 Acked 6771 Read 10 Done Done New false
17593 5773 Acked 6772 Write 7 Done Done New false
17594 5774 Acked 6773 Write 5 Done Done New false
17595 5775 Acked 6774 Write 10 Done Done New false
17596 5776 Acked 6775 Write 4 Done Done New false
17597 5777 Acked 6776 Write 3 Done Done New false
17598 5778 Acked 6777 Read 2 Done Done New false
17599 5779 Acked 6778 Write 1 Done Done New false
17600 5780 Acked 6779 Read 7 Done Done New false
17601 5781 Acked 6780 Read 9 Done Done New false
17602 5782 Acked 6781 Write 5 Done Done New false
17603 5783 Acked 6782 Read 4 Done Done New false
17604 5784 Acked 6783 Write 1 Done Done New false
17605 5785 Acked 6784 Write 9 Done Done New false
17606 5786 Acked 6785 Read 10 Done Done New false
17607 5787 Acked 6786 Read 10 Done Done New false
17608 5788 Acked 6787 Read 1 Done Done New false
17609 5789 Acked 6788 Write 3 Done Done New false
17610 5790 Acked 6789 Flush 0 Done Done New false
17611 5791 Acked 6790 Read 5 Done Done New false
17612 5792 Acked 6791 Write 9 Done Done New false
17613 5793 Acked 6792 Write 5 Done Done New false
17614 5794 Acked 6793 Read 4 Done Done New false
17615 5795 Acked 6794 Write 1 Done Done New false
17616 5796 Acked 6795 Flush 0 Done Done New false
17617 5797 Acked 6796 Write 2 Done Done New false
17618 5798 Acked 6797 Flush 0 Done Done New false
17619 5799 Acked 6798 Flush 0 Done Done New false
17620 5800 Acked 6799 Write 1 Done Done New false
17621 5801 Acked 6800 Read 5 Done Done New false
17622 5802 Acked 6801 Write 9 Done Done New false
17623 5803 Acked 6802 Read 6 Done Done New false
17624 5804 Acked 6803 Read 2 Done Done New false
17625 5805 Acked 6804 Write 3 Done Done New false
17626 5806 Acked 6805 Read 1 Done Done New false
17627 5807 Acked 6806 Read 8 Done Done New false
17628 5808 Acked 6807 Read 6 Done Done New false
17629 5809 Acked 6808 Write 9 Done Done New false
17630 5810 Acked 6809 Read 1 Done Done New false
17631 5811 Acked 6810 Write 7 Done Done New false
17632 5812 Acked 6811 Write 9 Done Done New false
17633 5813 Acked 6812 Write 6 Done Done New false
17634 5814 Acked 6813 Write 6 Done Done New false
17635 5815 Acked 6814 Write 7 Done Done New false
17636 5816 Acked 6815 Write 7 Done Done New false
17637 5817 Acked 6816 Read 8 Done Done New false
17638 5818 Acked 6817 Read 5 Done Done New false
17639 5819 Acked 6818 Write 10 Done Done New false
17640 5820 Acked 6819 Read 4 Done Done New false
17641 5821 Acked 6820 Read 2 Done Done New false
17642 5822 Acked 6821 Write 6 Done Done New false
17643 5823 Acked 6822 Read 6 Done Done New false
17644 5824 Acked 6823 Write 8 Done Done New false
17645 5825 Acked 6824 Read 5 Done Done New false
17646 5826 Acked 6825 Read 2 Done Done New false
17647 5827 Acked 6826 Read 4 Done Done New false
17648 5828 Acked 6827 Read 7 Done Done New false
17649 5829 Acked 6828 Read 5 Done Done New false
17650 5830 Acked 6829 Read 9 Done Done New false
17651 5831 Acked 6830 Read 9 Done Done New false
17652 5832 Acked 6831 Write 2 Done Done New false
17653 5833 Acked 6832 Write 10 Done Done New false
17654 5834 Acked 6833 Write 8 Done Done New false
17655 5835 Acked 6834 Read 4 Done Done New false
17656 5836 Acked 6835 Read 1 Done Done New false
17657 5837 Acked 6836 Write 1 Done Done New false
17658 5838 Acked 6837 Write 3 Done Done New false
17659 5839 Acked 6838 Flush 0 Done Done New false
17660 5840 Acked 6839 Read 1 Done Done New false
17661 5841 Acked 6840 Write 6 Done Done New false
17662 5842 Acked 6841 Write 8 Done Done New false
17663 5843 Acked 6842 Write 10 Done Done New false
17664 5844 Acked 6843 Read 3 Done Done New false
17665 5845 Acked 6844 Read 8 Done Done New false
17666 5846 Acked 6845 Read 2 Done Done New false
17667 5847 Acked 6846 Write 8 Done Done New false
17668 5848 Acked 6847 Write 2 Done Done New false
17669 5849 Acked 6848 Write 9 Done Done New false
17670 5850 Acked 6849 Read 6 Done Done New false
17671 5851 Acked 6850 Write 6 Done Done New false
17672 5852 Acked 6851 Read 3 Done Done New false
17673 5853 Acked 6852 Write 7 Done Done New false
17674 5854 Acked 6853 Write 2 Done Done New false
17675 5855 Acked 6854 Write 8 Done Done New false
17676 5856 Acked 6855 Read 2 Done Done New false
17677 5857 Acked 6856 Write 7 Done Done New false
17678 5858 Acked 6857 Write 5 Done Done New false
17679 5859 Acked 6858 Read 8 Done Done New false
17680 5860 Acked 6859 Write 3 Done Done New false
17681 5861 Acked 6860 Read 8 Done Done New false
17682 5862 Acked 6861 Read 2 Done Done New false
17683 5863 Acked 6862 Read 9 Done Done New false
17684 5864 Acked 6863 Read 4 Done Done New false
17685 5865 Acked 6864 Read 1 Done Done New false
17686 5866 Acked 6865 Write 5 Done Done New false
17687 5867 Acked 6866 Write 9 Done Done New false
17688 5868 Acked 6867 Write 6 Done Done New false
17689 5869 Acked 6868 Read 1 Done Done New false
17690 5870 Acked 6869 Write 10 Done Done New false
17691 5871 Acked 6870 Read 10 Done Done New false
17692 5872 Acked 6871 Write 6 Done Done New false
17693 5873 Acked 6872 Read 1 Done Done New false
17694 5874 Acked 6873 Read 4 Done Done New false
17695 5875 Acked 6874 Write 9 Done Done New false
17696 5876 Acked 6875 Read 2 Done Done New false
17697 5877 Acked 6876 Flush 0 Done Done New false
17698 5878 Acked 6877 Read 6 Done Done New false
17699 5879 Acked 6878 Write 1 Done Done New false
17700 5880 Acked 6879 Read 4 Done Done New false
17701 5881 Acked 6880 Write 9 Done Done New false
17702 5882 Acked 6881 Write 1 Done Done New false
17703 5883 Acked 6882 Read 9 Done Done New false
17704 5884 Acked 6883 Read 1 Done Done New false
17705 5885 Acked 6884 Write 10 Done Done New false
17706 5886 Acked 6885 Write 7 Done Done New false
17707 5887 Acked 6886 Flush 0 Done Done New false
17708 5888 Acked 6887 Read 9 Done Done New false
17709 5889 Acked 6888 Read 7 Done Done New false
17710 5890 Acked 6889 Read 7 Done Done New false
17711 5891 Acked 6890 Write 6 Done Done New false
17712 5892 Acked 6891 Write 10 Done Done New false
17713 5893 Acked 6892 Read 10 Done Done New false
17714 5894 Acked 6893 Read 10 Done Done New false
17715 5895 Acked 6894 Read 9 Done Done New false
17716 5896 Acked 6895 Write 4 Done Done New false
17717 5897 Acked 6896 Flush 0 Done Done New false
17718 5898 Acked 6897 Read 1 Done Done New false
17719 5899 Acked 6898 Read 2 Done Done New false
17720 5900 Acked 6899 Read 10 Done Done New false
17721 5901 Acked 6900 Write 7 Done Done New false
17722 5902 Acked 6901 Read 6 Done Done New false
17723 5903 Acked 6902 Read 1 Done Done New false
17724 5904 Acked 6903 Write 5 Done Done New false
17725 5905 Acked 6904 Read 9 Done Done New false
17726 5906 Acked 6905 Read 10 Done Done New false
17727 5907 Acked 6906 Write 8 Done Done New false
17728 5908 Acked 6907 Write 9 Done Done New false
17729 5909 Acked 6908 Read 7 Done Done New false
17730 5910 Acked 6909 Write 6 Done Done New false
17731 5911 Acked 6910 Read 8 Done Done New false
17732 5912 Acked 6911 Read 6 Done Done New false
17733 5913 Acked 6912 Write 3 Done Done New false
17734 5914 Acked 6913 Write 7 Done Done New false
17735 5915 Acked 6914 Read 9 Done Done New false
17736 5916 Acked 6915 Read 1 Done Done New false
17737 5917 Acked 6916 Write 3 Done Done New false
17738 5918 Acked 6917 Write 9 Done Done New false
17739 5919 Acked 6918 Flush 0 Done Done New false
17740 5920 Acked 6919 Flush 0 Done Done New false
17741 5921 Acked 6920 Write 2 Done Done New false
17742 5922 Acked 6921 Flush 0 Done Done New false
17743 5923 Acked 6922 Read 4 Done Done New false
17744 5924 Acked 6923 Write 9 Done Done New false
17745 5925 Acked 6924 Write 4 Done Done New false
17746 5926 Acked 6925 Write 2 Done Done New false
17747 5927 Acked 6926 Read 6 Done Done New false
17748 5928 Acked 6927 Write 9 Done Done New false
17749 5929 Acked 6928 Read 8 Done Done New false
17750 5930 Acked 6929 Read 4 Done Done New false
17751 5931 Acked 6930 Read 8 Done Done New false
17752 5932 Acked 6931 Read 5 Done Done New false
17753 5933 Acked 6932 Read 4 Done Done New false
17754 5934 Acked 6933 Write 4 Done Done New false
17755 5935 Acked 6934 Read 7 Done Done New false
17756 5936 Acked 6935 Read 6 Done Done New false
17757 5937 Acked 6936 Write 5 Done Done New false
17758 5938 Acked 6937 Read 10 Done Done New false
17759 5939 Acked 6938 Read 4 Done Done New false
17760 5940 Acked 6939 Read 2 Done Done New false
17761 5941 Acked 6940 Read 9 Done Done New false
17762 5942 Acked 6941 Read 9 Done Done New false
17763 5943 Acked 6942 Write 10 Done Done New false
17764 5944 Acked 6943 Flush 0 Done Done New false
17765 5945 Acked 6944 Read 4 Done Done New false
17766 5946 Acked 6945 Flush 0 Done Done New false
17767 5947 Acked 6946 Read 8 Done Done New false
17768 5948 Acked 6947 Write 6 Done Done New false
17769 5949 Acked 6948 Read 3 Done Done New false
17770 5950 Acked 6949 Read 8 Done Done New false
17771 5951 Acked 6950 Flush 0 Done Done New false
17772 5952 Acked 6951 Write 3 Done Done New false
17773 5953 Acked 6952 Read 9 Done Done New false
17774 5954 Acked 6953 Write 4 Done Done New false
17775 5955 Acked 6954 Write 4 Done Done New false
17776 5956 Acked 6955 Write 5 Done Done New false
17777 5957 Acked 6956 Read 2 Done Done New false
17778 5958 Acked 6957 Write 4 Done Done New false
17779 5959 Acked 6958 Read 1 Done Done New false
17780 5960 Acked 6959 Write 6 Done Done New false
17781 5961 Acked 6960 Write 1 Done Done New false
17782 5962 Acked 6961 Read 9 Done Done New false
17783 5963 Acked 6962 Write 6 Done Done New false
17784 5964 Acked 6963 Write 5 Done Done New false
17785 5965 Acked 6964 Flush 0 Done Done New false
17786 5966 Acked 6965 Write 3 Done Done New false
17787 5967 Acked 6966 Read 3 Done Done New false
17788 5968 Acked 6967 Flush 0 Done Done New false
17789 5969 Acked 6968 Read 5 Done Done New false
17790 5970 Acked 6969 Read 1 Done Done New false
17791 5971 Acked 6970 Write 3 Done Done New false
17792 5972 Acked 6971 Read 2 Done Done New false
17793 5973 Acked 6972 Read 10 Done Done New false
17794 5974 Acked 6973 Read 1 Done Done New false
17795 5975 Acked 6974 Write 8 Done Done New false
17796 5976 Acked 6975 Read 1 Done Done New false
17797 5977 Acked 6976 Write 2 Done Done New false
17798 5978 Acked 6977 Read 4 Done Done New false
17799 5979 Acked 6978 Read 9 Done Done New false
17800 5980 Acked 6979 Write 4 Done Done New false
17801 5981 Acked 6980 Read 6 Done Done New false
17802 5982 Acked 6981 Write 6 Done Done New false
17803 5983 Acked 6982 Read 10 Done Done New false
17804 5984 Acked 6983 Write 1 Done Done New false
17805 5985 Acked 6984 Write 9 Done Done New false
17806 5986 Acked 6985 Write 2 Done Done New false
17807 5987 Acked 6986 Write 9 Done Done New false
17808 5988 Acked 6987 Flush 0 Done Done New false
17809 5989 Acked 6988 Read 9 Done Done New false
17810 5990 Acked 6989 Write 4 Done Done New false
17811 5991 Acked 6990 Read 2 Done Done New false
17812 5992 Acked 6991 Read 6 Done Done New false
17813 5993 Acked 6992 Read 4 Done Done New false
17814 5994 Acked 6993 Write 8 Done Done New false
17815 5995 Acked 6994 Read 8 Done Done New false
17816 5996 Acked 6995 Read 10 Done Done New false
17817 5997 Acked 6996 Write 7 Done Done New false
17818 5998 Acked 6997 Read 8 Done Done New false
17819 5999 Acked 6998 Read 10 Done Done New false
17820 6000 Acked 6999 Read 1 Done Done New false
17821 6001 Acked 7000 Write 9 Done Done New false
17822 6002 Acked 7001 Flush 0 Done Done New false
17823 6003 Acked 7002 Write 2 Done Done New false
17824 6004 Acked 7003 Write 6 Done Done New false
17825 6005 Acked 7004 Read 5 Done Done New false
17826 6006 Acked 7005 Read 9 Done Done New false
17827 6007 Acked 7006 Write 7 Done Done New false
17828 6008 Acked 7007 Write 1 Done Done New false
17829 6009 Acked 7008 Read 1 Done Done New false
17830 6010 Acked 7009 Write 4 Done Done New false
17831 6011 Acked 7010 Read 4 Done Done New false
17832 6012 Acked 7011 Read 6 Done Done New false
17833 6013 Acked 7012 Read 1 Done Done New false
17834 6014 Acked 7013 Read 3 Done Done New false
17835 6015 Acked 7014 Read 8 Done Done New false
17836 6016 Acked 7015 Write 8 Done Done New false
17837 6017 Acked 7016 Read 5 Done Done New false
17838 6018 Acked 7017 Read 5 Done Done New false
17839 6019 Acked 7018 Read 4 Done Done New false
17840 6020 Acked 7019 Read 4 Done Done New false
17841 6021 Acked 7020 Write 2 Done Done New false
17842 6022 Acked 7021 Read 7 Done Done New false
17843 6023 Acked 7022 Read 10 Done Done New false
17844 6024 Acked 7023 Write 2 Done Done New false
17845 6025 Acked 7024 Read 10 Done Done New false
17846 6026 Acked 7025 Write 4 Done Done New false
17847 6027 Acked 7026 Read 1 Done Done New false
17848 6028 Acked 7027 Read 10 Done Done New false
17849 STATES DS:0 DS:1 DS:2 TOTAL
17850 New 0 0 301 301
17851 Sent 0 0 0 0
17852 Done 301 301 0 602
17853 Skipped 0 0 0 0
17854 Error 0 0 0 0
17855 Last Flush: 7001 7001 6726
17856 Downstairs last five completed: 6726 6725 6724 6723 6722
17857 Upstairs last five completed: 6028 6027 6026 6025 6024
17858 CLIENT: Up:0 ds:301 act:2
178592024-02-27T21:48:09.699ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
178602024-02-27T21:48:09.699ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
178612024-02-27T21:48:09.699ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
178622024-02-27T21:48:09.699ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
178632024-02-27T21:48:09.699ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
178642024-02-27T21:48:09.699ZINFOcrucible: [2] client re-new 302 jobs since flush 6726 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
178652024-02-27T21:48:09.699ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
178662024-02-27T21:48:10.700ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
178672024-02-27T21:48:10.700ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
178682024-02-27T21:48:10.700ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
178692024-02-27T21:48:10.700ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
178702024-02-27T21:48:10.700ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
178712024-02-27T21:48:10.700ZINFOcrucible: [2] client re-new 302 jobs since flush 6726 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
178722024-02-27T21:48:10.700ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
17873 [8830] got start action
17874 [8830] I am going to restart
17875 Starting downstairs at port 8830
17876 Make output file at "/tmp/dsc/downstairs-8830.txt"
17877 [8830][2] reports Starting
17878 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1187)
17879 [8830][2] reports Running
178802024-02-27T21:48:11.701ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
178812024-02-27T21:48:11.701ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
178822024-02-27T21:48:11.703ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
178832024-02-27T21:48:11.703ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
178842024-02-27T21:48:11.703ZINFOcrucible: send last flush ID to this DS: 6726 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
178852024-02-27T21:48:11.703ZINFOcrucible: Replied this last flush ID: 6726 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
178862024-02-27T21:48:11.703ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
178872024-02-27T21:48:11.703ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
178882024-02-27T21:48:11.703ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
178892024-02-27T21:48:11.703ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
17890 ----------------------------------------------------------------
17891 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
17892 Downstairs last five completed: 7028 7027 7026 7025 7024
17893 Upstairs last five completed: 6029 6028 6027 6026 6025
17894 CLIENT: Up:0 ds:0 act:3
17895 Replay: All jobs finished, all DS active.
17896 Main task has work to do, go find it
17897 got dsc Stop(2)
17898 stop 2
17899 Feb 27 21:48:13.042 INFO[8830] Got stop action so:false kr:false
17900 request completed, latency_us: 349, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 2a6aede8-6f8a-461a-9c66-44ff0f1ff49e, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
179012024-02-27T21:48:13.042ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
179022024-02-27T21:48:13.042ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
17903 Feb 27 21:48:13.042 INFO request completed, latency_us: 149, response_code: 200, uri: {"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-27T21:48:13.04293344Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
17904 /state/cid/2, method: GET, req_id: f8b83b92-6352-4a9f-bc2d-122f35fd63a0, remote_addr: 127.{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:48:13.042972395Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"client":"2","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
17905 0.0.1:42901, local_addr: 127.0.0.1:9998
179062024-02-27T21:48:13.043ZINFOcrucible: [2] client re-new 0 jobs since flush 7028 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
179072024-02-27T21:48:13.043ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
17908 [8830] Exited with: ExitStatus(unix_wait_status(9))
17909 [8830][2] reports Exit
179102024-02-27T21:48:14.044ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
179112024-02-27T21:48:14.044ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
179122024-02-27T21:48:14.044ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
179132024-02-27T21:48:14.044ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
179142024-02-27T21:48:14.044ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
179152024-02-27T21:48:14.044ZINFOcrucible: [2] client re-new 0 jobs since flush 7028 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
179162024-02-27T21:48:14.044ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
179172024-02-27T21:48:15.045ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
179182024-02-27T21:48:15.045ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
179192024-02-27T21:48:15.045ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
179202024-02-27T21:48:15.045ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
179212024-02-27T21:48:15.045ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
179222024-02-27T21:48:15.045ZINFOcrucible: [2] client re-new 0 jobs since flush 7028 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
179232024-02-27T21:48:15.045ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
179242024-02-27T21:48:16.046ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
179252024-02-27T21:48:16.047ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
179262024-02-27T21:48:16.047ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
179272024-02-27T21:48:16.047ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
179282024-02-27T21:48:16.047ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
179292024-02-27T21:48:16.047ZINFOcrucible: [2] client re-new 0 jobs since flush 7028 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
179302024-02-27T21:48:16.047ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
17931 Feb 27 21:48:17.045 INFO request completed, latency_us: 203, response_code: 200, uri: /state/cid/2, method: GET, req_id: 069f8a06-5ac8-491f-bdc0-915c884e68ac, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
17932 001/300 Read block 2272 len 4096
17933 002/300 Read block 577 len 8192
17934 003/300 Flush
17935 004/300 Read block 3432 len 8192
179362024-02-27T21:48:17.048ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
179372024-02-27T21:48:17.048ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
179382024-02-27T21:48:17.048ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
179392024-02-27T21:48:17.048ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
179402024-02-27T21:48:17.048ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
179412024-02-27T21:48:17.048ZINFOcrucible: [2] client re-new 4 jobs since flush 7028 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
179422024-02-27T21:48:17.048ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
17943 005/300 Write block 1488 len 32768 data: 5 4 5 5 5 6 7 4
17944 006/300 Write block 705 len 20480 data: 2 2 2 2 4
17945 007/300 Flush
17946 008/300 Write block 3329 len 16384 data: 6 6 5 5
17947 009/300 Read block 3588 len 28672
17948 010/300 Read block 462 len 12288
17949 011/300 Read block 4384 len 32768
17950 012/300 Read block 2303 len 40960
17951 013/300 Write block 813 len 12288 data: 3 3 3
17952 014/300 Write block 1677 len 8192 data: 3 4
17953 015/300 Read block 832 len 40960
17954 016/300 Read block 1276 len 24576
17955 017/300 Read block 1898 len 16384
17956 018/300 Read block 3309 len 28672
17957 019/300 Read block 5 len 36864
17958 020/300 Write block 3753 len 32768 data: 5 4 4 3 4 3 4 3
17959 021/300 Read block 2356 len 8192
17960 022/300 Flush
17961 023/300 Read block 1876 len 16384
17962 024/300 Read block 2158 len 16384
17963 025/300 Read block 4826 len 4096
17964 026/300 Read block 2169 len 20480
17965 027/300 Flush
17966 028/300 Read block 3301 len 40960
17967 029/300 Read block 1833 len 40960
17968 030/300 Read block 3434 len 40960
17969 031/300 Write block 587 len 8192 data: 6 7
17970 032/300 Write block 4649 len 8192 data: 5 5
17971 033/300 Write block 2747 len 16384 data: 3 3 2 2
17972 034/300 Read block 4471 len 40960
17973 035/300 Write block 4772 len 28672 data: 4 5 5 6 6 6 6
17974 036/300 Read block 2762 len 12288
17975 037/300 Read block 1772 len 28672
17976 038/300 Read block 2289 len 24576
17977 039/300 Write block 4548 len 28672 data: 6 6 7 6 6 5 4
17978 040/300 Read block 3260 len 32768
17979 041/300 Write block 3300 len 4096 data: 3
17980 042/300 Read block 4384 len 16384
17981 043/300 Write block 1610 len 20480 data: 2 4 5 7 7
17982 044/300 Read block 2477 len 36864
17983 045/300 Write block 131 len 16384 data: 3 3 3 4
17984 046/300 Read block 4711 len 36864
17985 047/300 Write block 4569 len 12288 data: 5 6 6
17986 048/300 Read block 2965 len 28672
17987 049/300 Read block 293 len 16384
17988 050/300 Read block 2514 len 20480
17989 051/300 Read block 4153 len 28672
17990 052/300 Write block 4249 len 28672 data: 5 5 5 6 6 5 5
17991 053/300 Write block 1467 len 16384 data: 4 4 5 4
17992 054/300 Read block 1974 len 12288
17993 055/300 Read block 2788 len 8192
17994 056/300 Write block 2828 len 16384 data: 3 3 5 4
17995 057/300 Write block 1721 len 20480 data: 3 4 4 5 4
17996 058/300 Read block 4312 len 24576
17997 059/300 Read block 4777 len 20480
17998 060/300 Read block 3999 len 28672
17999 061/300 Read block 4466 len 24576
18000 062/300 Write block 266 len 28672 data: 5 6 7 6 6 6 6
18001 063/300 Write block 2673 len 20480 data: 7 6 5 5 5
18002 064/300 Write block 1694 len 32768 data: 3 3 5 5 5 5 4 4
18003 065/300 Write block 3956 len 16384 data: 4 4 5 5
18004 066/300 Flush
18005 067/300 Read block 1080 len 24576
18006 068/300 Read block 4502 len 32768
18007 069/300 Read block 4896 len 24576
18008 070/300 Read block 4932 len 4096
18009 071/300 Write block 424 len 40960 data: 6 5 3 4 4 5 4 4 5 5
18010 072/300 Read block 560 len 4096
18011 073/300 Read block 1208 len 8192
18012 074/300 Read block 478 len 36864
18013 075/300 Write block 2994 len 8192 data: 4 4
18014 076/300 Read block 3087 len 28672
18015 077/300 Write block 1817 len 16384 data: 5 5 4 4
18016 078/300 Read block 4180 len 36864
18017 079/300 Flush
18018 080/300 Flush
18019 081/300 Write block 487 len 36864 data: 6 6 4 4 4 5 4 5 6
18020 082/300 Read block 239 len 32768
18021 083/300 Flush
18022 084/300 Read block 1359 len 4096
18023 085/300 Read block 4976 len 36864
18024 086/300 Read block 1877 len 8192
18025 087/300 Read block 2881 len 8192
18026 088/300 Read block 980 len 36864
18027 089/300 Write block 1775 len 8192 data: 2 2
18028 090/300 Write block 2750 len 16384 data: 3 2 2 2
18029 091/300 Write block 332 len 40960 data: 4 5 4 4 4 4 3 3 3 3
18030 092/300 Write block 161 len 28672 data: 5 5 4 4 4 4 3
18031 093/300 Flush
18032 094/300 Read block 3767 len 40960
18033 095/300 Read block 2118 len 32768
18034 096/300 Write block 3204 len 4096 data: 4
18035 097/300 Read block 2206 len 16384
18036 098/300 Read block 673 len 28672
18037 099/300 Flush
18038 100/300 Read block 3818 len 20480
18039 101/300 Read block 3147 len 16384
18040 102/300 Write block 1235 len 4096 data: 5
18041 103/300 Flush
18042 104/300 Read block 4694 len 28672
18043 105/300 Read block 4639 len 8192
18044 106/300 Flush
18045 107/300 Write block 1309 len 16384 data: 6 7 7 5
18046 108/300 Read block 4158 len 4096
18047 109/300 Write block 367 len 4096 data: 3
18048 110/300 Read block 2647 len 36864
18049 111/300 Read block 3785 len 12288
18050 112/300 Write block 1063 len 36864 data: 4 4 3 3 3 2 3 3 3
18051 113/300 Write block 2017 len 16384 data: 3 3 4 4
18052 114/300 Read block 2405 len 20480
18053 115/300 Read block 1546 len 12288
18054 116/300 Write block 1854 len 4096 data: 5
18055 117/300 Read block 2706 len 16384
18056 118/300 Read block 4401 len 36864
18057 119/300 Write block 3372 len 36864 data: 6 7 8 8 6 4 3 3 3
18058 120/300 Write block 1024 len 24576 data: 4 4 4 4 5 4
18059 121/300 Read block 3754 len 28672
18060 122/300 Read block 95 len 4096
18061 123/300 Write block 357 len 32768 data: 4 5 5 5 5 6 3 3
18062 124/300 Read block 2510 len 4096
18063 125/300 Read block 3073 len 36864
18064 126/300 Read block 2695 len 8192
18065 127/300 Read block 39 len 28672
18066 128/300 Read block 3463 len 24576
18067 129/300 Read block 3438 len 28672
18068 130/300 Read block 1762 len 12288
18069 131/300 Read block 430 len 12288
18070 132/300 Flush
18071 133/300 Read block 2467 len 28672
18072 134/300 Write block 1170 len 36864 data: 4 5 5 5 6 4 5 4 4
18073 135/300 Write block 2529 len 24576 data: 2 2 2 4 4 4
18074 136/300 Read block 3110 len 28672
18075 137/300 Read block 150 len 4096
18076 138/300 Write block 4939 len 24576 data: 3 3 5 5 5 5
18077 139/300 Read block 2078 len 24576
18078 140/300 Read block 4312 len 8192
18079 141/300 Write block 3397 len 16384 data: 4 4 5 6
18080 142/300 Flush
18081 143/300 Write block 957 len 32768 data: 6 7 7 5 3 3 4 4
18082 144/300 Flush
18083 145/300 Read block 2543 len 16384
18084 146/300 Read block 1913 len 40960
18085 147/300 Write block 3548 len 32768 data: 3 3 3 3 3 3 4 6
18086 148/300 Read block 4888 len 20480
18087 149/300 Read block 1769 len 12288
18088 150/300 Flush
18089 151/300 Write block 3110 len 28672 data: 8 9 7 7 8 9 8
18090 152/300 Read block 2069 len 28672
18091 153/300 Write block 897 len 40960 data: 4 4 4 3 3 4 3 3 4 5
18092 154/300 Read block 3523 len 32768
18093 155/300 Write block 2239 len 8192 data: 5 5
18094 156/300 Write block 4010 len 20480 data: 4 4 4 5 6
18095 157/300 Flush
18096 158/300 Write block 174 len 40960 data: 4 4 5 5 6 6 7 5 6 6
18097 159/300 Read block 284 len 32768
18098 160/300 Read block 3129 len 36864
18099 161/300 Read block 3732 len 8192
18100 162/300 Read block 2730 len 4096
18101 163/300 Read block 4865 len 12288
18102 164/300 Write block 175 len 36864 data: 5 6 6 7 7 8 6 7 7
18103 165/300 Write block 1165 len 40960 data: 4 4 4 3 3 5 6 6 6 7
18104 166/300 Write block 3494 len 4096 data: 7
18105 167/300 Write block 1401 len 12288 data: 4 4 4
18106 168/300 Read block 4612 len 24576
18107 169/300 Write block 4731 len 40960 data: 3 4 6 6 6 7 7 8 7 7
18108 170/300 Read block 1623 len 20480
18109 171/300 Write block 2283 len 12288 data: 6 5 5
18110 172/300 Write block 4176 len 24576 data: 3 3 3 3 3 3
18111 173/300 Read block 3861 len 4096
18112 174/300 Read block 106 len 24576
18113 175/300 Write block 2099 len 16384 data: 5 5 5 4
18114 176/300 Read block 4031 len 28672
18115 177/300 Write block 4619 len 16384 data: 5 5 4 5
18116 178/300 Read block 1774 len 40960
18117 179/300 Read block 861 len 40960
18118 180/300 Write block 4488 len 12288 data: 7 8 9
18119 181/300 Read block 363 len 8192
18120 182/300 Read block 4815 len 36864
18121 183/300 Read block 3447 len 8192
18122 184/300 Read block 3307 len 40960
18123 185/300 Write block 4078 len 4096 data: 3
18124 186/300 Flush
18125 187/300 Write block 2640 len 40960 data: 5 4 5 5 6 6 8 8 7 7
18126 188/300 Read block 1728 len 40960
18127 189/300 Write block 1010 len 20480 data: 3 4 4 5 5
18128 190/300 Read block 1771 len 4096
18129 191/300 Write block 4504 len 12288 data: 5 5 5
18130 192/300 Flush
18131 193/300 Flush
18132 194/300 Read block 1569 len 40960
18133 195/300 Read block 4839 len 20480
18134 196/300 Write block 3718 len 40960 data: 4 4 4 4 4 5 6 5 5 4
18135 197/300 Read block 3286 len 20480
18136 198/300 Write block 4092 len 4096 data: 6
18137 199/300 Read block 4040 len 16384
18138 200/300 Write block 504 len 24576 data: 4 5 5 7 6 7
18139 201/300 Write block 706 len 40960 data: 3 3 3 5 4 6 5 5 3 3
18140 202/300 Read block 1440 len 16384
18141 203/300 Write block 4258 len 40960 data: 4 5 5 6 6 7 9 9 8 7
18142 204/300 Write block 3733 len 36864 data: 5 4 4 4 4 4 5 6 5
18143 205/300 Write block 2667 len 32768 data: 8 10 11 11 8 7 8 7
18144 206/300 Write block 459 len 20480 data: 4 4 4 4 2
18145 207/300 Write block 1447 len 4096 data: 3
18146 208/300 Write block 2187 len 24576 data: 4 4 5 5 6 5
18147 209/300 Write block 1901 len 16384 data: 2 2 3 3
18148 210/300 Read block 1387 len 36864
18149 211/300 Write block 1315 len 32768 data: 8 7 7 8 9 9 9 8
18150 212/300 Write block 4034 len 20480 data: 4 5 5 3 2
18151 213/300 Flush
18152 214/300 Write block 3094 len 24576 data: 5 4 4 5 6 8
18153 215/300 Write block 3960 len 12288 data: 3 4 4
18154 216/300 Write block 3197 len 32768 data: 7 5 5 3 2 2 4 5
18155 217/300 Write block 2141 len 16384 data: 4 4 3 3
18156 218/300 Read block 2176 len 16384
18157 219/300 Read block 1383 len 36864
18158 220/300 Read block 3800 len 24576
18159 221/300 Write block 4410 len 40960 data: 6 5 5 5 5 5 4 5 2 2
18160 222/300 Write block 1734 len 4096 data: 4
18161 223/300 Write block 907 len 20480 data: 6 6 7 8 8
18162 224/300 Write block 1882 len 8192 data: 4 4
18163 225/300 Write block 1384 len 12288 data: 5 5 5
18164 226/300 Read block 1301 len 4096
18165 227/300 Read block 525 len 16384
18166 228/300 Read block 1044 len 16384
18167 229/300 Write block 2298 len 16384 data: 4 4 4 4
18168 230/300 Write block 1666 len 40960 data: 5 6 5 6 7 6 5 6 4 4
18169 231/300 Write block 2370 len 40960 data: 4 4 3 4 3 5 5 4 4 4
18170 232/300 Write block 1193 len 8192 data: 6 7
18171 233/300 Read block 1776 len 12288
18172 234/300 Read block 2763 len 40960
18173 235/300 Write block 1676 len 40960 data: 3 4 5 4 2 3 4 4 5 5
18174 236/300 Write block 331 len 4096 data: 6
18175 237/300 Read block 4213 len 8192
18176 238/300 Read block 3285 len 28672
18177 239/300 Write block 2929 len 40960 data: 3 3 4 4 4 4 4 4 4 5
18178 240/300 Write block 46 len 36864 data: 6 5 6 6 6 4 3 2 2
18179 241/300 Read block 2066 len 32768
18180 242/300 Read block 3695 len 32768
18181 243/300 Write block 3202 len 36864 data: 3 5 6 4 3 3 3 3 3
18182 244/300 Read block 3768 len 12288
18183 245/300 Write block 3288 len 8192 data: 4 3
18184 246/300 Read block 427 len 8192
18185 247/300 Flush
18186 248/300 Read block 854 len 24576
18187 249/300 Flush
18188 250/300 Read block 4106 len 28672
18189 251/300 Write block 4023 len 32768 data: 4 5 6 5 5 7 8 8
18190 252/300 Read block 4533 len 20480
18191 253/300 Read block 2721 len 12288
18192 254/300 Read block 3648 len 24576
18193 255/300 Read block 721 len 36864
18194 256/300 Read block 103 len 24576
18195 257/300 Read block 690 len 28672
18196 258/300 Flush
18197 259/300 Write block 3894 len 24576 data: 3 5 5 4 4 3
18198 260/300 Flush
18199 261/300 Read block 2308 len 28672
18200 262/300 Read block 1873 len 12288
18201 263/300 Write block 4008 len 4096 data: 2
18202 264/300 Read block 1076 len 32768
18203 265/300 Write block 3880 len 40960 data: 4 3 3 3 2 2 2 3 4 4
18204 266/300 Flush
18205 267/300 Read block 3415 len 12288
18206 268/300 Read block 3232 len 28672
18207 269/300 Write block 1798 len 4096 data: 4
18208 270/300 Write block 3069 len 28672 data: 4 4 2 2 2 2 2
18209 271/300 Write block 1363 len 24576 data: 6 7 6 6 6 5
18210 272/300 Write block 635 len 16384 data: 5 3 3 3
18211 273/300 Read block 2089 len 24576
18212 274/300 Read block 1425 len 24576
18213 275/300 Write block 3018 len 24576 data: 5 4 6 6 7 7
18214 276/300 Read block 4499 len 20480
18215 277/300 Read block 1299 len 12288
18216 278/300 Write block 230 len 4096 data: 5
18217 279/300 Read block 4210 len 8192
18218 280/300 Write block 1266 len 24576 data: 5 6 7 6 6 4
18219 281/300 Read block 3285 len 40960
18220 282/300 Write block 2009 len 4096 data: 4
18221 283/300 Write block 1499 len 8192 data: 5 5
18222 284/300 Read block 4698 len 28672
18223 285/300 Read block 3566 len 8192
18224 286/300 Read block 3667 len 28672
18225 287/300 Write block 4106 len 4096 data: 4
18226 288/300 Read block 2369 len 40960
18227 289/300 Read block 3761 len 32768
18228 290/300 Read block 1092 len 4096
18229 291/300 Write block 1956 len 12288 data: 3 3 2
18230 292/300 Write block 3446 len 20480 data: 7 7 8 7 7
18231 293/300 Read block 153 len 36864
18232 294/300 Flush
18233 295/300 Read block 4351 len 28672
18234 296/300 Flush
18235 297/300 Read block 4712 len 40960
18236 298/300 Write block 517 len 36864 data: 7 6 5 5 5 6 7 5 6
18237 299/300 Write block 2872 len 4096 data: 6
18238 300/300 Read block 1354 len 16384
18239 Main task has work to do, go find it
18240 got dsc Start(2)
18241 start 2
18242 Feb 27 21:48:17.409 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/2, method: POST, req_id: 6ae3d2ba-20e3-496a-a99c-3486c3bce02e, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
18243 Replay: started 2, returned:Ok(())
18244 ----------------------------------------------------------------
18245 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18246 States: Active Active Offline
18247 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
18248 6030 Acked 7029 Read 1 Done Done New true
18249 6031 Acked 7030 Read 2 Done Done New true
18250 6032 Acked 7031 Flush 0 Done Done New true
18251 6033 Acked 7032 Read 2 Done Done New true
18252 6034 Acked 7033 Write 8 Done Done New false
18253 6035 Acked 7034 Write 5 Done Done New false
18254 6036 Acked 7035 Flush 0 Done Done New false
18255 6037 Acked 7036 Write 4 Done Done New false
18256 6038 Acked 7037 Read 7 Done Done New false
18257 6039 Acked 7038 Read 3 Done Done New false
18258 6040 Acked 7039 Read 8 Done Done New false
18259 6041 Acked 7040 Read 10 Done Done New false
18260 6042 Acked 7041 Write 3 Done Done New false
18261 6043 Acked 7042 Write 2 Done Done New false
18262 6044 Acked 7043 Read 10 Done Done New false
18263 6045 Acked 7044 Read 6 Done Done New false
18264 6046 Acked 7045 Read 4 Done Done New false
18265 6047 Acked 7046 Read 7 Done Done New false
18266 6048 Acked 7047 Read 9 Done Done New false
18267 6049 Acked 7048 Write 8 Done Done New false
18268 6050 Acked 7049 Read 2 Done Done New false
18269 6051 Acked 7050 Flush 0 Done Done New false
18270 6052 Acked 7051 Read 4 Done Done New false
18271 6053 Acked 7052 Read 4 Done Done New false
18272 6054 Acked 7053 Read 1 Done Done New false
18273 6055 Acked 7054 Read 5 Done Done New false
18274 6056 Acked 7055 Flush 0 Done Done New false
18275 6057 Acked 7056 Read 10 Done Done New false
18276 6058 Acked 7057 Read 10 Done Done New false
18277 6059 Acked 7058 Read 10 Done Done New false
18278 6060 Acked 7059 Write 2 Done Done New false
18279 6061 Acked 7060 Write 2 Done Done New false
18280 6062 Acked 7061 Write 4 Done Done New false
18281 6063 Acked 7062 Read 10 Done Done New false
18282 6064 Acked 7063 Write 7 Done Done New false
18283 6065 Acked 7064 Read 3 Done Done New false
18284 6066 Acked 7065 Read 7 Done Done New false
18285 6067 Acked 7066 Read 6 Done Done New false
18286 6068 Acked 7067 Write 7 Done Done New false
18287 6069 Acked 7068 Read 8 Done Done New false
18288 6070 Acked 7069 Write 1 Done Done New false
18289 6071 Acked 7070 Read 4 Done Done New false
18290 6072 Acked 7071 Write 5 Done Done New false
18291 6073 Acked 7072 Read 9 Done Done New false
18292 6074 Acked 7073 Write 4 Done Done New false
18293 6075 Acked 7074 Read 9 Done Done New false
18294 6076 Acked 7075 Write 3 Done Done New false
18295 6077 Acked 7076 Read 7 Done Done New false
18296 6078 Acked 7077 Read 4 Done Done New false
18297 6079 Acked 7078 Read 5 Done Done New false
18298 6080 Acked 7079 Read 7 Done Done New false
18299 6081 Acked 7080 Write 7 Done Done New false
18300 6082 Acked 7081 Write 4 Done Done New false
18301 6083 Acked 7082 Read 3 Done Done New false
18302 6084 Acked 7083 Read 2 Done Done New false
18303 6085 Acked 7084 Write 4 Done Done New false
18304 6086 Acked 7085 Write 5 Done Done New false
18305 6087 Acked 7086 Read 6 Done Done New false
18306 6088 Acked 7087 Read 5 Done Done New false
18307 6089 Acked 7088 Read 7 Done Done New false
18308 6090 Acked 7089 Read 6 Done Done New false
18309 6091 Acked 7090 Write 7 Done Done New false
18310 6092 Acked 7091 Write 5 Done Done New false
18311 6093 Acked 7092 Write 8 Done Done New false
18312 6094 Acked 7093 Write 4 Done Done New false
18313 6095 Acked 7094 Flush 0 Done Done New false
18314 6096 Acked 7095 Read 6 Done Done New false
18315 6097 Acked 7096 Read 8 Done Done New false
18316 6098 Acked 7097 Read 6 Done Done New false
18317 6099 Acked 7098 Read 1 Done Done New false
18318 6100 Acked 7099 Write 10 Done Done New false
18319 6101 Acked 7100 Read 1 Done Done New false
18320 6102 Acked 7101 Read 2 Done Done New false
18321 6103 Acked 7102 Read 9 Done Done New false
18322 6104 Acked 7103 Write 2 Done Done New false
18323 6105 Acked 7104 Read 7 Done Done New false
18324 6106 Acked 7105 Write 4 Done Done New false
18325 6107 Acked 7106 Read 9 Done Done New false
18326 6108 Acked 7107 Flush 0 Done Done New false
18327 6109 Acked 7108 Flush 0 Done Done New false
18328 6110 Acked 7109 Write 9 Done Done New false
18329 6111 Acked 7110 Read 8 Done Done New false
18330 6112 Acked 7111 Flush 0 Done Done New false
18331 6113 Acked 7112 Read 1 Done Done New false
18332 6114 Acked 7113 Read 9 Done Done New false
18333 6115 Acked 7114 Read 2 Done Done New false
18334 6116 Acked 7115 Read 2 Done Done New false
18335 6117 Acked 7116 Read 9 Done Done New false
18336 6118 Acked 7117 Write 2 Done Done New false
18337 6119 Acked 7118 Write 4 Done Done New false
18338 6120 Acked 7119 Write 10 Done Done New false
18339 6121 Acked 7120 Write 7 Done Done New false
18340 6122 Acked 7121 Flush 0 Done Done New false
18341 6123 Acked 7122 Read 10 Done Done New false
18342 6124 Acked 7123 Read 8 Done Done New false
18343 6125 Acked 7124 Write 1 Done Done New false
18344 6126 Acked 7125 Read 4 Done Done New false
18345 6127 Acked 7126 Read 7 Done Done New false
18346 6128 Acked 7127 Flush 0 Done Done New false
18347 6129 Acked 7128 Read 5 Done Done New false
18348 6130 Acked 7129 Read 4 Done Done New false
18349 6131 Acked 7130 Write 1 Done Done New false
18350 6132 Acked 7131 Flush 0 Done Done New false
18351 6133 Acked 7132 Read 7 Done Done New false
18352 6134 Acked 7133 Read 2 Done Done New false
18353 6135 Acked 7134 Flush 0 Done Done New false
18354 6136 Acked 7135 Write 4 Done Done New false
18355 6137 Acked 7136 Read 1 Done Done New false
18356 6138 Acked 7137 Write 1 Done Done New false
18357 6139 Acked 7138 Read 9 Done Done New false
18358 6140 Acked 7139 Read 3 Done Done New false
18359 6141 Acked 7140 Write 9 Done Done New false
18360 6142 Acked 7141 Write 4 Done Done New false
18361 6143 Acked 7142 Read 5 Done Done New false
18362 6144 Acked 7143 Read 3 Done Done New false
18363 6145 Acked 7144 Write 1 Done Done New false
18364 6146 Acked 7145 Read 4 Done Done New false
18365 6147 Acked 7146 Read 9 Done Done New false
18366 6148 Acked 7147 Write 9 Done Done New false
18367 6149 Acked 7148 Write 6 Done Done New false
18368 6150 Acked 7149 Read 7 Done Done New false
18369 6151 Acked 7150 Read 1 Done Done New false
18370 6152 Acked 7151 Write 8 Done Done New false
18371 6153 Acked 7152 Read 1 Done Done New false
18372 6154 Acked 7153 Read 9 Done Done New false
18373 6155 Acked 7154 Read 2 Done Done New false
18374 6156 Acked 7155 Read 7 Done Done New false
18375 6157 Acked 7156 Read 6 Done Done New false
18376 6158 Acked 7157 Read 7 Done Done New false
18377 6159 Acked 7158 Read 3 Done Done New false
18378 6160 Acked 7159 Read 3 Done Done New false
18379 6161 Acked 7160 Flush 0 Done Done New false
18380 6162 Acked 7161 Read 7 Done Done New false
18381 6163 Acked 7162 Write 9 Done Done New false
18382 6164 Acked 7163 Write 6 Done Done New false
18383 6165 Acked 7164 Read 7 Done Done New false
18384 6166 Acked 7165 Read 1 Done Done New false
18385 6167 Acked 7166 Write 6 Done Done New false
18386 6168 Acked 7167 Read 6 Done Done New false
18387 6169 Acked 7168 Read 2 Done Done New false
18388 6170 Acked 7169 Write 4 Done Done New false
18389 6171 Acked 7170 Flush 0 Done Done New false
18390 6172 Acked 7171 Write 8 Done Done New false
18391 6173 Acked 7172 Flush 0 Done Done New false
18392 6174 Acked 7173 Read 4 Done Done New false
18393 6175 Acked 7174 Read 10 Done Done New false
18394 6176 Acked 7175 Write 8 Done Done New false
18395 6177 Acked 7176 Read 5 Done Done New false
18396 6178 Acked 7177 Read 3 Done Done New false
18397 6179 Acked 7178 Flush 0 Done Done New false
18398 6180 Acked 7179 Write 7 Done Done New false
18399 6181 Acked 7180 Read 7 Done Done New false
18400 6182 Acked 7181 Write 10 Done Done New false
18401 6183 Acked 7182 Read 8 Done Done New false
18402 6184 Acked 7183 Write 2 Done Done New false
18403 6185 Acked 7184 Write 5 Done Done New false
18404 6186 Acked 7185 Flush 0 Done Done New false
18405 6187 Acked 7186 Write 10 Done Done New false
18406 6188 Acked 7187 Read 8 Done Done New false
18407 6189 Acked 7188 Read 9 Done Done New false
18408 6190 Acked 7189 Read 2 Done Done New false
18409 6191 Acked 7190 Read 1 Done Done New false
18410 6192 Acked 7191 Read 3 Done Done New false
18411 6193 Acked 7192 Write 9 Done Done New false
18412 6194 Acked 7193 Write 10 Done Done New false
18413 6195 Acked 7194 Write 1 Done Done New false
18414 6196 Acked 7195 Write 3 Done Done New false
18415 6197 Acked 7196 Read 6 Done Done New false
18416 6198 Acked 7197 Write 10 Done Done New false
18417 6199 Acked 7198 Read 5 Done Done New false
18418 6200 Acked 7199 Write 3 Done Done New false
18419 6201 Acked 7200 Write 6 Done Done New false
18420 6202 Acked 7201 Read 1 Done Done New false
18421 6203 Acked 7202 Read 6 Done Done New false
18422 6204 Acked 7203 Write 4 Done Done New false
18423 6205 Acked 7204 Read 7 Done Done New false
18424 6206 Acked 7205 Write 4 Done Done New false
18425 6207 Acked 7206 Read 10 Done Done New false
18426 6208 Acked 7207 Read 10 Done Done New false
18427 6209 Acked 7208 Write 3 Done Done New false
18428 6210 Acked 7209 Read 2 Done Done New false
18429 6211 Acked 7210 Read 9 Done Done New false
18430 6212 Acked 7211 Read 2 Done Done New false
18431 6213 Acked 7212 Read 10 Done Done New false
18432 6214 Acked 7213 Write 1 Done Done New false
18433 6215 Acked 7214 Flush 0 Done Done New false
18434 6216 Acked 7215 Write 10 Done Done New false
18435 6217 Acked 7216 Read 10 Done Done New false
18436 6218 Acked 7217 Write 5 Done Done New false
18437 6219 Acked 7218 Read 1 Done Done New false
18438 6220 Acked 7219 Write 3 Done Done New false
18439 6221 Acked 7220 Flush 0 Done Done New false
18440 6222 Acked 7221 Flush 0 Done Done New false
18441 6223 Acked 7222 Read 10 Done Done New false
18442 6224 Acked 7223 Read 5 Done Done New false
18443 6225 Acked 7224 Write 10 Done Done New false
18444 6226 Acked 7225 Read 5 Done Done New false
18445 6227 Acked 7226 Write 1 Done Done New false
18446 6228 Acked 7227 Read 4 Done Done New false
18447 6229 Acked 7228 Write 6 Done Done New false
18448 6230 Acked 7229 Write 10 Done Done New false
18449 6231 Acked 7230 Read 4 Done Done New false
18450 6232 Acked 7231 Write 10 Done Done New false
18451 6233 Acked 7232 Write 9 Done Done New false
18452 6234 Acked 7233 Write 8 Done Done New false
18453 6235 Acked 7234 Write 5 Done Done New false
18454 6236 Acked 7235 Write 1 Done Done New false
18455 6237 Acked 7236 Write 6 Done Done New false
18456 6238 Acked 7237 Write 4 Done Done New false
18457 6239 Acked 7238 Read 9 Done Done New false
18458 6240 Acked 7239 Write 8 Done Done New false
18459 6241 Acked 7240 Write 5 Done Done New false
18460 6242 Acked 7241 Flush 0 Done Done New false
18461 6243 Acked 7242 Write 6 Done Done New false
18462 6244 Acked 7243 Write 3 Done Done New false
18463 6245 Acked 7244 Write 8 Done Done New false
18464 6246 Acked 7245 Write 4 Done Done New false
18465 6247 Acked 7246 Read 4 Done Done New false
18466 6248 Acked 7247 Read 9 Done Done New false
18467 6249 Acked 7248 Read 6 Done Done New false
18468 6250 Acked 7249 Write 10 Done Done New false
18469 6251 Acked 7250 Write 1 Done Done New false
18470 6252 Acked 7251 Write 5 Done Done New false
18471 6253 Acked 7252 Write 2 Done Done New false
18472 6254 Acked 7253 Write 3 Done Done New false
18473 6255 Acked 7254 Read 1 Done Done New false
18474 6256 Acked 7255 Read 4 Done Done New false
18475 6257 Acked 7256 Read 4 Done Done New false
18476 6258 Acked 7257 Write 4 Done Done New false
18477 6259 Acked 7258 Write 10 Done Done New false
18478 6260 Acked 7259 Write 10 Done Done New false
18479 6261 Acked 7260 Write 2 Done Done New false
18480 6262 Acked 7261 Read 3 Done Done New false
18481 6263 Acked 7262 Read 10 Done Done New false
18482 6264 Acked 7263 Write 10 Done Done New false
18483 6265 Acked 7264 Write 1 Done Done New false
18484 6266 Acked 7265 Read 2 Done Done New false
18485 6267 Acked 7266 Read 7 Done Done New false
18486 6268 Acked 7267 Write 10 Done Done New false
18487 6269 Acked 7268 Write 9 Done Done New false
18488 6270 Acked 7269 Read 8 Done Done New false
18489 6271 Acked 7270 Read 8 Done Done New false
18490 6272 Acked 7271 Write 9 Done Done New false
18491 6273 Acked 7272 Read 3 Done Done New false
18492 6274 Acked 7273 Write 2 Done Done New false
18493 6275 Acked 7274 Read 2 Done Done New false
18494 6276 Acked 7275 Flush 0 Done Done New false
18495 6277 Acked 7276 Read 6 Done Done New false
18496 6278 Acked 7277 Flush 0 Done Done New false
18497 6279 Acked 7278 Read 7 Done Done New false
18498 6280 Acked 7279 Write 8 Done Done New false
18499 6281 Acked 7280 Read 5 Done Done New false
18500 6282 Acked 7281 Read 3 Done Done New false
18501 6283 Acked 7282 Read 6 Done Done New false
18502 6284 Acked 7283 Read 9 Done Done New false
18503 6285 Acked 7284 Read 6 Done Done New false
18504 6286 Acked 7285 Read 7 Done Done New false
18505 6287 Acked 7286 Flush 0 Done Done New false
18506 6288 Acked 7287 Write 6 Done Done New false
18507 6289 Acked 7288 Flush 0 Done Done New false
18508 6290 Acked 7289 Read 7 Done Done New false
18509 6291 Acked 7290 Read 3 Done Done New false
18510 6292 Acked 7291 Write 1 Done Done New false
18511 6293 Acked 7292 Read 8 Done Done New false
18512 6294 Acked 7293 Write 10 Done Done New false
18513 6295 Acked 7294 Flush 0 Done Done New false
18514 6296 Acked 7295 Read 3 Done Done New false
18515 6297 Acked 7296 Read 7 Done Done New false
18516 6298 Acked 7297 Write 1 Done Done New false
18517 6299 Acked 7298 Write 7 Done Done New false
18518 6300 Acked 7299 Write 6 Done Done New false
18519 6301 Acked 7300 Flush 0 Done Done New false
18520 6302 Acked 7301 Write 4 Done Done New false
18521 6303 Acked 7302 Read 6 Done Done New false
18522 6304 Acked 7303 Read 6 Done Done New false
18523 6305 Acked 7304 Write 6 Done Done New false
18524 6306 Acked 7305 Read 5 Done Done New false
18525 6307 Acked 7306 Read 3 Done Done New false
18526 6308 Acked 7307 Write 1 Done Done New false
18527 6309 Acked 7308 Read 2 Done Done New false
18528 6310 Acked 7309 Write 6 Done Done New false
18529 6311 Acked 7310 Read 10 Done Done New false
18530 6312 Acked 7311 Write 1 Done Done New false
18531 6313 Acked 7312 Write 2 Done Done New false
18532 6314 Acked 7313 Read 7 Done Done New false
18533 6315 Acked 7314 Read 2 Done Done New false
18534 6316 Acked 7315 Read 7 Done Done New false
18535 6317 Acked 7316 Write 1 Done Done New false
18536 6318 Acked 7317 Read 10 Done Done New false
18537 6319 Acked 7318 Read 8 Done Done New false
18538 6320 Acked 7319 Read 1 Done Done New false
18539 6321 Acked 7320 Write 3 Done Done New false
18540 6322 Acked 7321 Write 5 Done Done New false
18541 6323 Acked 7322 Read 9 Done Done New false
18542 6324 Acked 7323 Flush 0 Done Done New false
18543 6325 Acked 7324 Read 7 Done Done New false
18544 6326 Acked 7325 Flush 0 Done Done New false
18545 6327 Acked 7326 Read 10 Done Done New false
18546 6328 Acked 7327 Write 9 Done Done New false
18547 6329 Acked 7328 Write 1 Done Done New false
18548 6330 Acked 7329 Read 4 Done Done New false
18549 STATES DS:0 DS:1 DS:2 TOTAL
18550 New 0 0 301 301
18551 Sent 0 0 0 0
18552 Done 301 301 0 602
18553 Skipped 0 0 0 0
18554 Error 0 0 0 0
18555 Last Flush: 7325 7325 7028
18556 Downstairs last five completed: 7028 7027 7026 7025 7024
18557 Upstairs last five completed: 6330 6329 6328 6327 6326
18558 CLIENT: Up:0 ds:301 act:2
185592024-02-27T21:48:18.050ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
185602024-02-27T21:48:18.050ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
185612024-02-27T21:48:18.050ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
185622024-02-27T21:48:18.050ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
185632024-02-27T21:48:18.050ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
185642024-02-27T21:48:18.050ZINFOcrucible: [2] client re-new 302 jobs since flush 7028 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
185652024-02-27T21:48:18.050ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
18566 [8830] got start action
18567 [8830] I am going to restart
18568 Starting downstairs at port 8830
18569 Make output file at "/tmp/dsc/downstairs-8830.txt"
18570 [8830][2] reports Starting
185712024-02-27T21:48:19.052ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
185722024-02-27T21:48:19.052ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
185732024-02-27T21:48:19.052ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
185742024-02-27T21:48:19.052ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
185752024-02-27T21:48:19.052ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
185762024-02-27T21:48:19.052ZINFOcrucible: [2] client re-new 302 jobs since flush 7028 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
185772024-02-27T21:48:19.052ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
18578 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1188)
18579 [8830][2] reports Running
185802024-02-27T21:48:20.053ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
185812024-02-27T21:48:20.054ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
185822024-02-27T21:48:20.055ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
185832024-02-27T21:48:20.055ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
185842024-02-27T21:48:20.055ZINFOcrucible: send last flush ID to this DS: 7028 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
185852024-02-27T21:48:20.055ZINFOcrucible: Replied this last flush ID: 7028 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
185862024-02-27T21:48:20.055ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
185872024-02-27T21:48:20.055ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
185882024-02-27T21:48:20.055ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
185892024-02-27T21:48:20.055ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
18590 ----------------------------------------------------------------
18591 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
18592 Downstairs last five completed: 7330 7329 7328 7327 7326
18593 Upstairs last five completed: 6331 6330 6329 6328 6327
18594 CLIENT: Up:0 ds:0 act:3
18595 Replay: All jobs finished, all DS active.
18596 Main task has work to do, go find it
18597 got dsc Stop(0)
18598 stop 0
18599 Feb 27 21:48:21.412 INFO request completed, latency_us: 390, response_code: 204, uri: [8810] Got stop action so:false kr:false
18600 /stop/cid/0, method: POST, req_id: ab8be710-0844-4b09-a887-f44c9380432d, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
18601 Feb 27 21:48:21.413 INFO request completed, latency_us: 125, response_code: 200, uri: /state/cid/0, method: GET, req_id: e3676172-bc2c-4de8-a2b9-97e0a002e636, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998{"msg":"downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2024-02-27T21:48:21.413520724Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"io task","client":"0","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
18602 
186032024-02-27T21:48:21.413ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
186042024-02-27T21:48:21.413ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = e6a5a974-925a-494f-8845-facc441c2690
186052024-02-27T21:48:21.413ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
186062024-02-27T21:48:21.413ZINFOcrucible: [0] client re-new 0 jobs since flush 7330 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
186072024-02-27T21:48:21.413ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
18608 [8810] Exited with: ExitStatus(unix_wait_status(9))
18609 [8810][0] reports Exit
186102024-02-27T21:48:22.415ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
186112024-02-27T21:48:22.415ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
186122024-02-27T21:48:22.415ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
186132024-02-27T21:48:22.415ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
186142024-02-27T21:48:22.415ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
186152024-02-27T21:48:22.415ZINFOcrucible: [0] client re-new 0 jobs since flush 7330 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
186162024-02-27T21:48:22.415ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
186172024-02-27T21:48:23.416ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
186182024-02-27T21:48:23.416ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
186192024-02-27T21:48:23.416ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
186202024-02-27T21:48:23.416ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
186212024-02-27T21:48:23.416ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
186222024-02-27T21:48:23.416ZINFOcrucible: [0] client re-new 0 jobs since flush 7330 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
186232024-02-27T21:48:23.416ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
186242024-02-27T21:48:24.417ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
186252024-02-27T21:48:24.417ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
186262024-02-27T21:48:24.417ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
186272024-02-27T21:48:24.417ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
186282024-02-27T21:48:24.417ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
186292024-02-27T21:48:24.417ZINFOcrucible: [0] client re-new 0 jobs since flush 7330 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
186302024-02-27T21:48:24.417ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
18631 Feb 27 21:48:25.415 INFO request completed, latency_us: 203, response_code: 200, uri: /state/cid/0, method: GET, req_id: 7d661d08-23ea-4e8d-9d25-6f17d1bafb39, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
18632 001/300 Read block 1019 len 28672
18633 002/300 Write block 224 len 16384 data: 6 6 7 6
186342024-02-27T21:48:25.418ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
18635 {"msg":"ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }","v":0,"name":"crucible","level":40003/300 Read block 3401 len 12288
18636 ,"time":"2024-02-27T21:48:25.418335927Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"io task","client":"0","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
186372024-02-27T21:48:25.418ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
186382024-02-27T21:48:25.418ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
186392024-02-27T21:48:25.418ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
186402024-02-27T21:48:25.418ZINFOcrucible: [0] client re-new 2 jobs since flush 7330 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
186412024-02-27T21:48:25.418ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
18642 004/300 Write block 618 len 40960 data: 6 5 4 4 4 4 5 4 4 4
18643 005/300 Read block 1506 len 32768
18644 006/300 Write block 2570 len 28672 data: 8 8 8 6 6 7 7
18645 007/300 Read block 962 len 40960
18646 008/300 Write block 3371 len 24576 data: 4 7 8 9 9 7
18647 009/300 Write block 4740 len 12288 data: 8 7 6
18648 010/300 Read block 3435 len 12288
18649 011/300 Read block 3188 len 4096
18650 012/300 Write block 1917 len 40960 data: 4 4 4 5 5 3 3 4 4 5
18651 013/300 Read block 2321 len 4096
18652 014/300 Write block 355 len 20480 data: 3 3 5 6 6
18653 015/300 Flush
18654 016/300 Read block 1361 len 36864
18655 017/300 Write block 1118 len 24576 data: 6 5 5 5 4 3
18656 018/300 Write block 1825 len 32768 data: 4 5 7 6 5 6 7 7
18657 019/300 Read block 893 len 8192
18658 020/300 Write block 3167 len 16384 data: 5 6 6 6
18659 021/300 Read block 1459 len 8192
18660 022/300 Read block 3788 len 36864
18661 023/300 Write block 2810 len 36864 data: 8 8 7 8 7 8 9 9 10
18662 024/300 Write block 4845 len 12288 data: 3 4 4
18663 025/300 Read block 114 len 4096
18664 026/300 Read block 4697 len 4096
18665 027/300 Read block 4441 len 12288
18666 028/300 Read block 222 len 32768
18667 029/300 Read block 3188 len 4096
18668 030/300 Read block 1058 len 20480
18669 031/300 Read block 160 len 24576
18670 032/300 Read block 1354 len 12288
18671 033/300 Write block 4210 len 28672 data: 3 3 3 3 3 3 3
18672 034/300 Read block 4361 len 40960
18673 035/300 Read block 2559 len 40960
18674 036/300 Read block 1099 len 32768
18675 037/300 Read block 1224 len 24576
18676 038/300 Write block 3671 len 28672 data: 4 4 4 4 4 4 6
18677 039/300 Read block 1494 len 24576
18678 040/300 Write block 4397 len 16384 data: 8 10 11 10
18679 041/300 Write block 46 len 8192 data: 7 6
18680 042/300 Write block 3654 len 20480 data: 3 4 3 3 3
18681 043/300 Read block 3914 len 28672
18682 044/300 Read block 4435 len 16384
18683 045/300 Read block 2231 len 36864
18684 046/300 Read block 1615 len 20480
18685 047/300 Read block 3955 len 8192
18686 048/300 Read block 3224 len 4096
18687 049/300 Read block 4800 len 28672
18688 050/300 Write block 2330 len 8192 data: 4 7
18689 051/300 Read block 886 len 20480
18690 052/300 Write block 4533 len 32768 data: 3 3 3 3 3 2 3 3
18691 053/300 Read block 2529 len 12288
18692 054/300 Read block 2051 len 20480
18693 055/300 Read block 1949 len 12288
18694 056/300 Read block 3808 len 40960
18695 057/300 Write block 1391 len 12288 data: 4 5 5
18696 058/300 Flush
18697 059/300 Read block 4047 len 16384
18698 060/300 Read block 1090 len 32768
18699 061/300 Write block 3042 len 36864 data: 4 6 6 5 4 3 3 3 3
18700 062/300 Write block 2163 len 20480 data: 4 3 3 3 5
18701 063/300 Read block 3022 len 40960
18702 064/300 Write block 4081 len 36864 data: 3 4 4 4 3 3 3 3 4
18703 065/300 Write block 1033 len 36864 data: 4 3 3 5 5 6 6 6 6
18704 066/300 Write block 2191 len 40960 data: 7 6 7 7 5 6 8 8 9 9
18705 067/300 Read block 3682 len 40960
18706 068/300 Read block 3157 len 16384
18707 069/300 Write block 3938 len 24576 data: 6 5 4 4 4 5
18708 070/300 Write block 3522 len 8192 data: 6 7
18709 071/300 Read block 3559 len 32768
18710 072/300 Write block 3912 len 32768 data: 3 3 5 5 6 7 6 5
18711 073/300 Read block 1412 len 8192
18712 074/300 Write block 2052 len 16384 data: 4 5 5 5
18713 075/300 Write block 1366 len 32768 data: 7 7 6 3 3 3 3 3
18714 076/300 Write block 4812 len 24576 data: 5 6 7 7 6 6
18715 077/300 Write block 3462 len 8192 data: 3 4
18716 078/300 Read block 2694 len 24576
18717 079/300 Write block 1697 len 32768 data: 6 6 6 5 5 5 6 4
18718 080/300 Read block 1683 len 16384
18719 081/300 Read block 1435 len 12288
18720 082/300 Read block 2754 len 40960
18721 083/300 Write block 608 len 16384 data: 6 6 5 5
18722 084/300 Read block 4962 len 28672
18723 085/300 Write block 1781 len 20480 data: 4 4 4 4 4
18724 086/300 Write block 3058 len 12288 data: 7 6 6
18725 087/300 Write block 2667 len 24576 data: 9 11 12 12 9 8
18726 088/300 Read block 3409 len 40960
18727 089/300 Write block 3091 len 12288 data: 2 3 4
18728 090/300 Write block 2365 len 40960 data: 3 4 4 4 4 5 5 4 5 4
18729 091/300 Read block 919 len 40960
18730 092/300 Write block 4592 len 20480 data: 4 5 5 6 7
18731 093/300 Write block 4014 len 24576 data: 7 6 5 6 8 8
18732 094/300 Read block 1987 len 20480
18733 095/300 Read block 3124 len 20480
18734 096/300 Write block 1743 len 12288 data: 2 2 3
18735 097/300 Flush
18736 098/300 Write block 4790 len 32768 data: 4 5 5 6 6 4 4 3
18737 099/300 Write block 3135 len 8192 data: 3 3
18738 100/300 Read block 3830 len 12288
18739 101/300 Write block 235 len 28672 data: 4 4 4 3 3 3 3
18740 102/300 Read block 2333 len 28672
18741 103/300 Flush
18742 104/300 Write block 1318 len 40960 data: 9 10 10 10 9 7 7 7 7 5
18743 105/300 Write block 3285 len 40960 data: 3 4 4 5 4 3 4 4 4 4
18744 106/300 Read block 3326 len 12288
18745 107/300 Read block 179 len 4096
18746 108/300 Read block 2920 len 24576
18747 109/300 Read block 86 len 32768
18748 110/300 Flush
18749 111/300 Write block 3471 len 20480 data: 4 5 5 5 3
18750 112/300 Write block 3518 len 16384 data: 5 4 6 6
18751 113/300 Write block 2078 len 16384 data: 3 3 3 3
18752 114/300 Read block 2343 len 20480
18753 115/300 Write block 3763 len 32768 data: 3 3 3 3 3 4 4 4
18754 116/300 Flush
18755 117/300 Write block 1378 len 28672 data: 5 5 5 4 5 5 6
18756 118/300 Write block 3337 len 16384 data: 4 5 3 4
18757 119/300 Write block 4796 len 40960 data: 5 4 3 3 3 3 4 4 4 4
18758 120/300 Write block 3138 len 8192 data: 4 4
18759 121/300 Read block 1427 len 8192
18760 122/300 Read block 2714 len 12288
18761 123/300 Read block 522 len 4096
18762 124/300 Flush
18763 125/300 Write block 704 len 16384 data: 2 3 4 4
18764 126/300 Read block 4531 len 4096
18765 127/300 Read block 1949 len 4096
18766 128/300 Write block 403 len 12288 data: 5 5 5
18767 129/300 Read block 2041 len 20480
18768 130/300 Write block 2181 len 32768 data: 3 4 4 4 3 4 5 5
18769 131/300 Write block 2199 len 36864 data: 10 10 5 6 4 5 5 7 7
18770 132/300 Read block 3595 len 36864
18771 133/300 Read block 1160 len 24576
18772 134/300 Read block 3265 len 12288
18773 135/300 Read block 781 len 16384
18774 136/300 Read block 450 len 36864
18775 137/300 Read block 2162 len 40960
18776 138/300 Read block 601 len 40960
18777 139/300 Write block 504 len 20480 data: 5 6 6 8 7
18778 140/300 Read block 2640 len 28672
18779 141/300 Write block 922 len 28672 data: 4 4 4 4 3 4 4
18780 142/300 Write block 319 len 4096 data: 4
18781 143/300 Write block 3998 len 16384 data: 3 3 4 4
18782 144/300 Read block 3665 len 4096
18783 145/300 Write block 2705 len 40960 data: 2 2 2 4 6 4 5 5 5 4
18784 146/300 Read block 1150 len 32768
18785 147/300 Write block 737 len 36864 data: 6 5 6 6 5 5 4 3 3
18786 148/300 Read block 3550 len 40960
18787 149/300 Flush
18788 150/300 Read block 2671 len 20480
18789 151/300 Read block 1516 len 28672
18790 152/300 Write block 241 len 24576 data: 4 4 5 5 6 6
18791 153/300 Read block 2777 len 16384
18792 154/300 Read block 359 len 20480
18793 155/300 Read block 1385 len 4096
18794 156/300 Read block 2683 len 24576
18795 157/300 Write block 3318 len 32768 data: 8 8 7 7 7 7 6 6
18796 158/300 Write block 4345 len 4096 data: 5
18797 159/300 Read block 3444 len 12288
18798 160/300 Write block 2589 len 16384 data: 5 6 5 7
18799 161/300 Write block 3594 len 24576 data: 5 7 6 6 5 5
18800 162/300 Read block 4186 len 12288
18801 163/300 Read block 1636 len 32768
18802 164/300 Write block 4511 len 20480 data: 6 7 8 6 6
18803 165/300 Write block 1577 len 40960 data: 4 5 6 5 5 4 5 6 6 5
18804 166/300 Write block 1123 len 24576 data: 4 2 3 4 4 6
18805 167/300 Write block 2309 len 4096 data: 4
18806 168/300 Write block 124 len 24576 data: 5 3 3 2 2 2
18807 169/300 Read block 695 len 12288
18808 170/300 Read block 2989 len 40960
18809 171/300 Write block 2909 len 40960 data: 6 8 8 7 7 7 6 5 5 4
18810 172/300 Write block 4560 len 16384 data: 2 2 2 3
18811 173/300 Write block 4516 len 28672 data: 6 6 6 5 5 2 2
18812 174/300 Write block 1208 len 20480 data: 5 5 4 4 3
18813 175/300 Read block 3862 len 4096
18814 176/300 Flush
18815 177/300 Read block 2681 len 28672
18816 178/300 Flush
18817 179/300 Read block 3544 len 32768
18818 180/300 Write block 3998 len 24576 data: 4 4 5 5 4 3
18819 181/300 Write block 1205 len 36864 data: 5 4 5 6 6 5 5 4 3
18820 182/300 Read block 938 len 24576
18821 183/300 Read block 762 len 4096
18822 184/300 Write block 4072 len 16384 data: 5 5 5 7
18823 185/300 Write block 3416 len 28672 data: 6 6 7 6 6 5 5
18824 186/300 Read block 2743 len 20480
18825 187/300 Write block 3285 len 20480 data: 4 5 5 6 5
18826 188/300 Read block 1801 len 32768
18827 189/300 Write block 3055 len 12288 data: 6 7 7
18828 190/300 Read block 3081 len 12288
18829 191/300 Write block 3538 len 28672 data: 6 6 6 5 6 6 5
18830 192/300 Read block 1358 len 12288
18831 193/300 Read block 2747 len 16384
18832 194/300 Read block 4825 len 28672
18833 195/300 Write block 4918 len 32768 data: 5 4 3 4 4 5 5 4
18834 196/300 Read block 3833 len 8192
18835 197/300 Read block 646 len 36864
18836 198/300 Write block 2021 len 4096 data: 4
18837 199/300 Read block 434 len 16384
18838 200/300 Read block 2810 len 28672
18839 201/300 Read block 940 len 16384
18840 202/300 Write block 2428 len 12288 data: 4 4 3
18841 203/300 Write block 1834 len 12288 data: 6 5 4
18842 204/300 Write block 169 len 28672 data: 2 3 4 5 5 5 6
18843 205/300 Read block 4138 len 4096
18844 206/300 Write block 4924 len 20480 data: 6 5 4 4 5
18845 207/300 Write block 4249 len 8192 data: 6 6
18846 208/300 Write block 3105 len 12288 data: 6 6 6
18847 209/300 Read block 3943 len 8192
18848 210/300 Read block 2111 len 4096
18849 211/300 Read block 2206 len 8192
18850 212/300 Write block 886 len 40960 data: 4 6 5 6 5 6 5 5 5 6
18851 213/300 Flush
18852 214/300 Write block 4321 len 16384 data: 6 4 5 5
18853 215/300 Flush
18854 216/300 Read block 2357 len 24576
18855 217/300 Write block 4830 len 24576 data: 7 6 6 5 6 6
18856 218/300 Read block 3950 len 28672
18857 219/300 Write block 3203 len 12288 data: 6 7 5
18858 220/300 Read block 1371 len 24576
18859 221/300 Write block 1601 len 36864 data: 4 3 4 2 2 2 2 2 2
18860 222/300 Write block 1778 len 12288 data: 2 2 2
18861 223/300 Read block 601 len 12288
18862 224/300 Read block 3502 len 4096
18863 225/300 Write block 3271 len 32768 data: 3 5 5 6 4 3 3 3
18864 226/300 Read block 2299 len 20480
18865 227/300 Write block 458 len 24576 data: 4 5 5 5 5 3
18866 228/300 Write block 1795 len 24576 data: 5 4 4 5 3 5
18867 229/300 Read block 4649 len 12288
18868 230/300 Write block 2828 len 24576 data: 4 4 6 5 4 4
18869 231/300 Write block 54 len 4096 data: 3
18870 232/300 Write block 3320 len 32768 data: 8 8 8 8 7 7 7 8
18871 233/300 Write block 2943 len 4096 data: 5
18872 234/300 Flush
18873 235/300 Flush
18874 236/300 Write block 3283 len 40960 data: 3 4 5 6 6 7 6 4 5 5
18875 237/300 Read block 1689 len 36864
18876 238/300 Read block 1827 len 12288
18877 239/300 Read block 12 len 40960
18878 240/300 Write block 178 len 12288 data: 8 8 9
18879 241/300 Write block 1913 len 8192 data: 4 5
18880 242/300 Read block 3757 len 40960
18881 243/300 Flush
18882 244/300 Read block 2494 len 4096
18883 245/300 Read block 2827 len 36864
18884 246/300 Read block 3691 len 40960
18885 247/300 Read block 654 len 28672
18886 248/300 Flush
18887 249/300 Read block 1210 len 28672
18888 250/300 Read block 2462 len 20480
18889 251/300 Read block 3138 len 8192
18890 252/300 Read block 1329 len 24576
18891 253/300 Read block 943 len 16384
18892 254/300 Write block 1019 len 24576 data: 6 5 4 4 4 5
18893 255/300 Read block 4390 len 4096
18894 256/300 Flush
18895 257/300 Flush
18896 258/300 Read block 2941 len 24576
18897 259/300 Write block 3709 len 16384 data: 5 3 3 4
18898 260/300 Flush
18899 261/300 Flush
18900 262/300 Write block 4218 len 8192 data: 4 6
18901 263/300 Read block 1095 len 20480
18902 264/300 Write block 3631 len 28672 data: 3 4 4 4 5 6 7
18903 265/300 Write block 4405 len 32768 data: 4 6 5 7 7 7 6 6
18904 266/300 Read block 4643 len 16384
18905 267/300 Read block 2400 len 20480
18906 268/300 Write block 1590 len 20480 data: 3 4 4 4 7
18907 269/300 Read block 3704 len 28672
18908 270/300 Read block 4297 len 16384
18909 271/300 Write block 4633 len 40960 data: 6 5 3 3 3 3 3 4 4 5
18910 272/300 Read block 4546 len 36864
18911 273/300 Read block 1337 len 32768
18912 274/300 Write block 745 len 8192 data: 4 2
18913 275/300 Flush
18914 276/300 Write block 1457 len 24576 data: 8 8 6 7 6 5
18915 277/300 Read block 4698 len 28672
18916 278/300 Read block 4474 len 28672
18917 279/300 Write block 3728 len 4096 data: 4
18918 280/300 Write block 1301 len 24576 data: 3 4 4 4 3 4
18919 281/300 Write block 3175 len 24576 data: 6 6 6 5 4 5
18920 282/300 Write block 288 len 12288 data: 4 3 3
18921 283/300 Read block 862 len 16384
18922 284/300 Write block 3342 len 8192 data: 5 6
18923 285/300 Flush
18924 286/300 Read block 299 len 40960
18925 287/300 Write block 622 len 16384 data: 5 5 6 5
18926 288/300 Read block 3152 len 36864
18927 289/300 Write block 3003 len 4096 data: 7
18928 290/300 Write block 62 len 4096 data: 2
18929 291/300 Write block 1366 len 12288 data: 8 8 7
18930 292/300 Flush
18931 293/300 Read block 1932 len 24576
18932 294/300 Read block 1526 len 12288
18933 295/300 Read block 4376 len 12288
18934 296/300 Read block 2158 len 8192
18935 297/300 Read block 355 len 20480
18936 298/300 Write block 752 len 40960 data: 4 4 4 4 3 3 2 3 4 4
18937 299/300 Read block 2267 len 4096
18938 300/300 Flush
18939 Main task has work to do, go find it
18940 got dsc Start(0)
18941 start 0
18942 Feb 27 21:48:25.748 INFO request completed, latency_us: 204, response_code: 204, uri: /start/cid/0, method: POST, req_id: c6c5d18c-265f-43fe-8c0a-36f4a75c74bd, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
18943 Replay: started 0, returned:Ok(())
18944 ----------------------------------------------------------------
18945 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
18946 States: Offline Active Active
18947 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
18948 6332 Acked 7331 Read 7 New Done Done true
18949 6333 Acked 7332 Write 4 New Done Done true
18950 6334 Acked 7333 Read 3 New Done Done false
18951 6335 Acked 7334 Write 10 New Done Done false
18952 6336 Acked 7335 Read 8 New Done Done false
18953 6337 Acked 7336 Write 7 New Done Done false
18954 6338 Acked 7337 Read 10 New Done Done false
18955 6339 Acked 7338 Write 6 New Done Done false
18956 6340 Acked 7339 Write 3 New Done Done false
18957 6341 Acked 7340 Read 3 New Done Done false
18958 6342 Acked 7341 Read 1 New Done Done false
18959 6343 Acked 7342 Write 10 New Done Done false
18960 6344 Acked 7343 Read 1 New Done Done false
18961 6345 Acked 7344 Write 5 New Done Done false
18962 6346 Acked 7345 Flush 0 New Done Done false
18963 6347 Acked 7346 Read 9 New Done Done false
18964 6348 Acked 7347 Write 6 New Done Done false
18965 6349 Acked 7348 Write 8 New Done Done false
18966 6350 Acked 7349 Read 2 New Done Done false
18967 6351 Acked 7350 Write 4 New Done Done false
18968 6352 Acked 7351 Read 2 New Done Done false
18969 6353 Acked 7352 Read 9 New Done Done false
18970 6354 Acked 7353 Write 9 New Done Done false
18971 6355 Acked 7354 Write 3 New Done Done false
18972 6356 Acked 7355 Read 1 New Done Done false
18973 6357 Acked 7356 Read 1 New Done Done false
18974 6358 Acked 7357 Read 3 New Done Done false
18975 6359 Acked 7358 Read 8 New Done Done false
18976 6360 Acked 7359 Read 1 New Done Done false
18977 6361 Acked 7360 Read 5 New Done Done false
18978 6362 Acked 7361 Read 6 New Done Done false
18979 6363 Acked 7362 Read 3 New Done Done false
18980 6364 Acked 7363 Write 7 New Done Done false
18981 6365 Acked 7364 Read 10 New Done Done false
18982 6366 Acked 7365 Read 10 New Done Done false
18983 6367 Acked 7366 Read 8 New Done Done false
18984 6368 Acked 7367 Read 6 New Done Done false
18985 6369 Acked 7368 Write 7 New Done Done false
18986 6370 Acked 7369 Read 6 New Done Done false
18987 6371 Acked 7370 Write 4 New Done Done false
18988 6372 Acked 7371 Write 2 New Done Done false
18989 6373 Acked 7372 Write 5 New Done Done false
18990 6374 Acked 7373 Read 7 New Done Done false
18991 6375 Acked 7374 Read 4 New Done Done false
18992 6376 Acked 7375 Read 9 New Done Done false
18993 6377 Acked 7376 Read 5 New Done Done false
18994 6378 Acked 7377 Read 2 New Done Done false
18995 6379 Acked 7378 Read 1 New Done Done false
18996 6380 Acked 7379 Read 7 New Done Done false
18997 6381 Acked 7380 Write 2 New Done Done false
18998 6382 Acked 7381 Read 5 New Done Done false
18999 6383 Acked 7382 Write 8 New Done Done false
19000 6384 Acked 7383 Read 3 New Done Done false
19001 6385 Acked 7384 Read 5 New Done Done false
19002 6386 Acked 7385 Read 3 New Done Done false
19003 6387 Acked 7386 Read 10 New Done Done false
19004 6388 Acked 7387 Write 3 New Done Done false
19005 6389 Acked 7388 Flush 0 New Done Done false
19006 6390 Acked 7389 Read 4 New Done Done false
19007 6391 Acked 7390 Read 8 New Done Done false
19008 6392 Acked 7391 Write 9 New Done Done false
19009 6393 Acked 7392 Write 5 New Done Done false
19010 6394 Acked 7393 Read 10 New Done Done false
19011 6395 Acked 7394 Write 9 New Done Done false
19012 6396 Acked 7395 Write 9 New Done Done false
19013 6397 Acked 7396 Write 10 New Done Done false
19014 6398 Acked 7397 Read 10 New Done Done false
19015 6399 Acked 7398 Read 4 New Done Done false
19016 6400 Acked 7399 Write 6 New Done Done false
19017 6401 Acked 7400 Write 2 New Done Done false
19018 6402 Acked 7401 Read 8 New Done Done false
19019 6403 Acked 7402 Write 8 New Done Done false
19020 6404 Acked 7403 Read 2 New Done Done false
19021 6405 Acked 7404 Write 4 New Done Done false
19022 6406 Acked 7405 Write 8 New Done Done false
19023 6407 Acked 7406 Write 6 New Done Done false
19024 6408 Acked 7407 Write 2 New Done Done false
19025 6409 Acked 7408 Read 6 New Done Done false
19026 6410 Acked 7409 Write 8 New Done Done false
19027 6411 Acked 7410 Read 4 New Done Done false
19028 6412 Acked 7411 Read 3 New Done Done false
19029 6413 Acked 7412 Read 10 New Done Done false
19030 6414 Acked 7413 Write 4 New Done Done false
19031 6415 Acked 7414 Read 7 New Done Done false
19032 6416 Acked 7415 Write 5 New Done Done false
19033 6417 Acked 7416 Write 3 New Done Done false
19034 6418 Acked 7417 Write 6 New Done Done false
19035 6419 Acked 7418 Read 10 New Done Done false
19036 6420 Acked 7419 Write 3 New Done Done false
19037 6421 Acked 7420 Write 10 New Done Done false
19038 6422 Acked 7421 Read 10 New Done Done false
19039 6423 Acked 7422 Write 5 New Done Done false
19040 6424 Acked 7423 Write 6 New Done Done false
19041 6425 Acked 7424 Read 5 New Done Done false
19042 6426 Acked 7425 Read 5 New Done Done false
19043 6427 Acked 7426 Write 3 New Done Done false
19044 6428 Acked 7427 Flush 0 New Done Done false
19045 6429 Acked 7428 Write 8 New Done Done false
19046 6430 Acked 7429 Write 2 New Done Done false
19047 6431 Acked 7430 Read 3 New Done Done false
19048 6432 Acked 7431 Write 7 New Done Done false
19049 6433 Acked 7432 Read 7 New Done Done false
19050 6434 Acked 7433 Flush 0 New Done Done false
19051 6435 Acked 7434 Write 10 New Done Done false
19052 6436 Acked 7435 Write 10 New Done Done false
19053 6437 Acked 7436 Read 3 New Done Done false
19054 6438 Acked 7437 Read 1 New Done Done false
19055 6439 Acked 7438 Read 6 New Done Done false
19056 6440 Acked 7439 Read 8 New Done Done false
19057 6441 Acked 7440 Flush 0 New Done Done false
19058 6442 Acked 7441 Write 5 New Done Done false
19059 6443 Acked 7442 Write 4 New Done Done false
19060 6444 Acked 7443 Write 4 New Done Done false
19061 6445 Acked 7444 Read 5 New Done Done false
19062 6446 Acked 7445 Write 8 New Done Done false
19063 6447 Acked 7446 Flush 0 New Done Done false
19064 6448 Acked 7447 Write 7 New Done Done false
19065 6449 Acked 7448 Write 4 New Done Done false
19066 6450 Acked 7449 Write 10 New Done Done false
19067 6451 Acked 7450 Write 2 New Done Done false
19068 6452 Acked 7451 Read 2 New Done Done false
19069 6453 Acked 7452 Read 3 New Done Done false
19070 6454 Acked 7453 Read 1 New Done Done false
19071 6455 Acked 7454 Flush 0 New Done Done false
19072 6456 Acked 7455 Write 4 New Done Done false
19073 6457 Acked 7456 Read 1 New Done Done false
19074 6458 Acked 7457 Read 1 New Done Done false
19075 6459 Acked 7458 Write 3 New Done Done false
19076 6460 Acked 7459 Read 5 New Done Done false
19077 6461 Acked 7460 Write 8 New Done Done false
19078 6462 Acked 7461 Write 9 New Done Done false
19079 6463 Acked 7462 Read 9 New Done Done false
19080 6464 Acked 7463 Read 6 New Done Done false
19081 6465 Acked 7464 Read 3 New Done Done false
19082 6466 Acked 7465 Read 4 New Done Done false
19083 6467 Acked 7466 Read 9 New Done Done false
19084 6468 Acked 7467 Read 10 New Done Done false
19085 6469 Acked 7468 Read 10 New Done Done false
19086 6470 Acked 7469 Write 5 New Done Done false
19087 6471 Acked 7470 Read 7 New Done Done false
19088 6472 Acked 7471 Write 7 New Done Done false
19089 6473 Acked 7472 Write 1 New Done Done false
19090 6474 Acked 7473 Write 4 New Done Done false
19091 6475 Acked 7474 Read 1 New Done Done false
19092 6476 Acked 7475 Write 10 New Done Done false
19093 6477 Acked 7476 Read 8 New Done Done false
19094 6478 Acked 7477 Write 9 New Done Done false
19095 6479 Acked 7478 Read 10 New Done Done false
19096 6480 Acked 7479 Flush 0 New Done Done false
19097 6481 Acked 7480 Read 5 New Done Done false
19098 6482 Acked 7481 Read 7 New Done Done false
19099 6483 Acked 7482 Write 6 New Done Done false
19100 6484 Acked 7483 Read 4 New Done Done false
19101 6485 Acked 7484 Read 5 New Done Done false
19102 6486 Acked 7485 Read 1 New Done Done false
19103 6487 Acked 7486 Read 6 New Done Done false
19104 6488 Acked 7487 Write 8 New Done Done false
19105 6489 Acked 7488 Write 1 New Done Done false
19106 6490 Acked 7489 Read 3 New Done Done false
19107 6491 Acked 7490 Write 4 New Done Done false
19108 6492 Acked 7491 Write 6 New Done Done false
19109 6493 Acked 7492 Read 3 New Done Done false
19110 6494 Acked 7493 Read 8 New Done Done false
19111 6495 Acked 7494 Write 5 New Done Done false
19112 6496 Acked 7495 Write 10 New Done Done false
19113 6497 Acked 7496 Write 6 New Done Done false
19114 6498 Acked 7497 Write 1 New Done Done false
19115 6499 Acked 7498 Write 6 New Done Done false
19116 6500 Acked 7499 Read 3 New Done Done false
19117 6501 Acked 7500 Read 10 New Done Done false
19118 6502 Acked 7501 Write 10 New Done Done false
19119 6503 Acked 7502 Write 4 New Done Done false
19120 6504 Acked 7503 Write 7 New Done Done false
19121 6505 Acked 7504 Write 5 New Done Done false
19122 6506 Acked 7505 Read 1 New Done Done false
19123 6507 Acked 7506 Flush 0 New Done Done false
19124 6508 Acked 7507 Read 7 New Done Done false
19125 6509 Acked 7508 Flush 0 New Done Done false
19126 6510 Acked 7509 Read 8 New Done Done false
19127 6511 Acked 7510 Write 6 New Done Done false
19128 6512 Acked 7511 Write 9 New Done Done false
19129 6513 Acked 7512 Read 6 New Done Done false
19130 6514 Acked 7513 Read 1 New Done Done false
19131 6515 Acked 7514 Write 4 New Done Done false
19132 6516 Acked 7515 Write 7 New Done Done false
19133 6517 Acked 7516 Read 5 New Done Done false
19134 6518 Acked 7517 Write 5 New Done Done false
19135 6519 Acked 7518 Read 8 New Done Done false
19136 6520 Acked 7519 Write 3 New Done Done false
19137 6521 Acked 7520 Read 3 New Done Done false
19138 6522 Acked 7521 Write 7 New Done Done false
19139 6523 Acked 7522 Read 3 New Done Done false
19140 6524 Acked 7523 Read 4 New Done Done false
19141 6525 Acked 7524 Read 7 New Done Done false
19142 6526 Acked 7525 Write 8 New Done Done false
19143 6527 Acked 7526 Read 2 New Done Done false
19144 6528 Acked 7527 Read 9 New Done Done false
19145 6529 Acked 7528 Write 1 New Done Done false
19146 6530 Acked 7529 Read 4 New Done Done false
19147 6531 Acked 7530 Read 7 New Done Done false
19148 6532 Acked 7531 Read 4 New Done Done false
19149 6533 Acked 7532 Write 3 New Done Done false
19150 6534 Acked 7533 Write 3 New Done Done false
19151 6535 Acked 7534 Write 7 New Done Done false
19152 6536 Acked 7535 Read 1 New Done Done false
19153 6537 Acked 7536 Write 5 New Done Done false
19154 6538 Acked 7537 Write 2 New Done Done false
19155 6539 Acked 7538 Write 3 New Done Done false
19156 6540 Acked 7539 Read 2 New Done Done false
19157 6541 Acked 7540 Read 1 New Done Done false
19158 6542 Acked 7541 Read 2 New Done Done false
19159 6543 Acked 7542 Write 10 New Done Done false
19160 6544 Acked 7543 Flush 0 New Done Done false
19161 6545 Acked 7544 Write 4 New Done Done false
19162 6546 Acked 7545 Flush 0 New Done Done false
19163 6547 Acked 7546 Read 6 New Done Done false
19164 6548 Acked 7547 Write 6 New Done Done false
19165 6549 Acked 7548 Read 7 New Done Done false
19166 6550 Acked 7549 Write 3 New Done Done false
19167 6551 Acked 7550 Read 6 New Done Done false
19168 6552 Acked 7551 Write 9 New Done Done false
19169 6553 Acked 7552 Write 3 New Done Done false
19170 6554 Acked 7553 Read 3 New Done Done false
19171 6555 Acked 7554 Read 1 New Done Done false
19172 6556 Acked 7555 Write 8 New Done Done false
19173 6557 Acked 7556 Read 5 New Done Done false
19174 6558 Acked 7557 Write 6 New Done Done false
19175 6559 Acked 7558 Write 6 New Done Done false
19176 6560 Acked 7559 Read 3 New Done Done false
19177 6561 Acked 7560 Write 6 New Done Done false
19178 6562 Acked 7561 Write 1 New Done Done false
19179 6563 Acked 7562 Write 8 New Done Done false
19180 6564 Acked 7563 Write 1 New Done Done false
19181 6565 Acked 7564 Flush 0 New Done Done false
19182 6566 Acked 7565 Flush 0 New Done Done false
19183 6567 Acked 7566 Write 10 New Done Done false
19184 6568 Acked 7567 Read 9 New Done Done false
19185 6569 Acked 7568 Read 3 New Done Done false
19186 6570 Acked 7569 Read 10 New Done Done false
19187 6571 Acked 7570 Write 3 New Done Done false
19188 6572 Acked 7571 Write 2 New Done Done false
19189 6573 Acked 7572 Read 10 New Done Done false
19190 6574 Acked 7573 Flush 0 New Done Done false
19191 6575 Acked 7574 Read 1 New Done Done false
19192 6576 Acked 7575 Read 9 New Done Done false
19193 6577 Acked 7576 Read 10 New Done Done false
19194 6578 Acked 7577 Read 7 New Done Done false
19195 6579 Acked 7578 Flush 0 New Done Done false
19196 6580 Acked 7579 Read 7 New Done Done false
19197 6581 Acked 7580 Read 5 New Done Done false
19198 6582 Acked 7581 Read 2 New Done Done false
19199 6583 Acked 7582 Read 6 New Done Done false
19200 6584 Acked 7583 Read 4 New Done Done false
19201 6585 Acked 7584 Write 6 New Done Done false
19202 6586 Acked 7585 Read 1 New Done Done false
19203 6587 Acked 7586 Flush 0 New Done Done false
19204 6588 Acked 7587 Flush 0 New Done Done false
19205 6589 Acked 7588 Read 6 New Done Done false
19206 6590 Acked 7589 Write 4 New Done Done false
19207 6591 Acked 7590 Flush 0 New Done Done false
19208 6592 Acked 7591 Flush 0 New Done Done false
19209 6593 Acked 7592 Write 2 New Done Done false
19210 6594 Acked 7593 Read 5 New Done Done false
19211 6595 Acked 7594 Write 7 New Done Done false
19212 6596 Acked 7595 Write 8 New Done Done false
19213 6597 Acked 7596 Read 4 New Done Done false
19214 6598 Acked 7597 Read 5 New Done Done false
19215 6599 Acked 7598 Write 5 New Done Done false
19216 6600 Acked 7599 Read 7 New Done Done false
19217 6601 Acked 7600 Read 4 New Done Done false
19218 6602 Acked 7601 Write 10 New Done Done false
19219 6603 Acked 7602 Read 9 New Done Done false
19220 6604 Acked 7603 Read 8 New Done Done false
19221 6605 Acked 7604 Write 2 New Done Done false
19222 6606 Acked 7605 Flush 0 New Done Done false
19223 6607 Acked 7606 Write 6 New Done Done false
19224 6608 Acked 7607 Read 7 New Done Done false
19225 6609 Acked 7608 Read 7 New Done Done false
19226 6610 Acked 7609 Write 1 New Done Done false
19227 6611 Acked 7610 Write 6 New Done Done false
19228 6612 Acked 7611 Write 6 New Done Done false
19229 6613 Acked 7612 Write 3 New Done Done false
19230 6614 Acked 7613 Read 4 New Done Done false
19231 6615 Acked 7614 Write 2 New Done Done false
19232 6616 Acked 7615 Flush 0 New Done Done false
19233 6617 Acked 7616 Read 10 New Done Done false
19234 6618 Acked 7617 Write 4 New Done Done false
19235 6619 Acked 7618 Read 9 New Done Done false
19236 6620 Acked 7619 Write 1 New Done Done false
19237 6621 Acked 7620 Write 1 New Done Done false
19238 6622 Acked 7621 Write 3 New Done Done false
19239 6623 Acked 7622 Flush 0 New Done Done false
19240 6624 Acked 7623 Read 6 New Done Done false
19241 6625 Acked 7624 Read 3 New Done Done false
19242 6626 Acked 7625 Read 3 New Done Done false
19243 6627 Acked 7626 Read 2 New Done Done false
19244 6628 Acked 7627 Read 5 New Done Done false
19245 6629 Acked 7628 Write 10 New Done Done false
19246 6630 Acked 7629 Read 1 New Done Done false
19247 6631 Acked 7630 Flush 0 New Done Done false
19248 STATES DS:0 DS:1 DS:2 TOTAL
19249 New 300 0 0 300
19250 Sent 0 0 0 0
19251 Done 0 300 300 600
19252 Skipped 0 0 0 0
19253 Error 0 0 0 0
19254 Last Flush: 7330 7630 7630
19255 Downstairs last five completed: 7330 7329 7328 7327 7326
19256 Upstairs last five completed: 6631 6630 6629 6628 6627
19257 CLIENT: Up:0 ds:300 act:2
192582024-02-27T21:48:26.420ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
192592024-02-27T21:48:26.420ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
192602024-02-27T21:48:26.420ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
192612024-02-27T21:48:26.420ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
192622024-02-27T21:48:26.420ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
192632024-02-27T21:48:26.420ZINFOcrucible: [0] client re-new 300 jobs since flush 7330 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
192642024-02-27T21:48:26.420ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
192652024-02-27T21:48:27.421ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
192662024-02-27T21:48:27.422ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
192672024-02-27T21:48:27.422ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
192682024-02-27T21:48:27.422ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
192692024-02-27T21:48:27.422ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
192702024-02-27T21:48:27.422ZINFOcrucible: [0] client re-new 300 jobs since flush 7330 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
192712024-02-27T21:48:27.422ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
192722024-02-27T21:48:28.423ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
192732024-02-27T21:48:28.423ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
192742024-02-27T21:48:28.423ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
192752024-02-27T21:48:28.423ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
192762024-02-27T21:48:28.423ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
192772024-02-27T21:48:28.423ZINFOcrucible: [0] client re-new 300 jobs since flush 7330 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
192782024-02-27T21:48:28.423ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
19279 [8810] got start action
19280 [8810] I am going to restart
19281 Starting downstairs at port 8810
19282 Make output file at "/tmp/dsc/downstairs-8810.txt"
19283 [8810][0] reports Starting
192842024-02-27T21:48:29.424ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
192852024-02-27T21:48:29.424ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
192862024-02-27T21:48:29.424ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
192872024-02-27T21:48:29.424ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
192882024-02-27T21:48:29.424ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
192892024-02-27T21:48:29.424ZINFOcrucible: [0] client re-new 300 jobs since flush 7330 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
192902024-02-27T21:48:29.424ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
19291 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1189)
19292 [8810][0] reports Running
19293 ----------------------------------------------------------------
19294 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
19295 States: Offline Active Active
19296 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
19297 6332 Acked 7331 Read 7 New Done Done true
19298 6333 Acked 7332 Write 4 New Done Done true
19299 6334 Acked 7333 Read 3 New Done Done true
19300 6335 Acked 7334 Write 10 New Done Done true
19301 6336 Acked 7335 Read 8 New Done Done true
19302 6337 Acked 7336 Write 7 New Done Done true
19303 6338 Acked 7337 Read 10 New Done Done true
19304 6339 Acked 7338 Write 6 New Done Done true
19305 6340 Acked 7339 Write 3 New Done Done true
19306 6341 Acked 7340 Read 3 New Done Done true
19307 6342 Acked 7341 Read 1 New Done Done true
19308 6343 Acked 7342 Write 10 New Done Done true
19309 6344 Acked 7343 Read 1 New Done Done true
19310 6345 Acked 7344 Write 5 New Done Done true
19311 6346 Acked 7345 Flush 0 New Done Done true
19312 6347 Acked 7346 Read 9 New Done Done true
19313 6348 Acked 7347 Write 6 New Done Done true
19314 6349 Acked 7348 Write 8 New Done Done true
19315 6350 Acked 7349 Read 2 New Done Done true
19316 6351 Acked 7350 Write 4 New Done Done true
19317 6352 Acked 7351 Read 2 New Done Done true
19318 6353 Acked 7352 Read 9 New Done Done true
19319 6354 Acked 7353 Write 9 New Done Done true
19320 6355 Acked 7354 Write 3 New Done Done true
19321 6356 Acked 7355 Read 1 New Done Done true
19322 6357 Acked 7356 Read 1 New Done Done true
19323 6358 Acked 7357 Read 3 New Done Done true
19324 6359 Acked 7358 Read 8 New Done Done true
19325 6360 Acked 7359 Read 1 New Done Done true
19326 6361 Acked 7360 Read 5 New Done Done true
19327 6362 Acked 7361 Read 6 New Done Done true
19328 6363 Acked 7362 Read 3 New Done Done true
19329 6364 Acked 7363 Write 7 New Done Done true
19330 6365 Acked 7364 Read 10 New Done Done true
19331 6366 Acked 7365 Read 10 New Done Done true
19332 6367 Acked 7366 Read 8 New Done Done true
19333 6368 Acked 7367 Read 6 New Done Done true
19334 6369 Acked 7368 Write 7 New Done Done true
19335 6370 Acked 7369 Read 6 New Done Done true
19336 6371 Acked 7370 Write 4 New Done Done true
19337 6372 Acked 7371 Write 2 New Done Done true
19338 6373 Acked 7372 Write 5 New Done Done true
19339 6374 Acked 7373 Read 7 New Done Done true
19340 6375 Acked 7374 Read 4 New Done Done true
19341 6376 Acked 7375 Read 9 New Done Done true
19342 6377 Acked 7376 Read 5 New Done Done true
19343 6378 Acked 7377 Read 2 New Done Done true
19344 6379 Acked 7378 Read 1 New Done Done true
19345 6380 Acked 7379 Read 7 New Done Done true
19346 6381 Acked 7380 Write 2 New Done Done true
19347 6382 Acked 7381 Read 5 New Done Done true
19348 6383 Acked 7382 Write 8 New Done Done true
19349 6384 Acked 7383 Read 3 New Done Done true
19350 6385 Acked 7384 Read 5 New Done Done true
19351 6386 Acked 7385 Read 3 New Done Done true
19352 6387 Acked 7386 Read 10 New Done Done true
19353 6388 Acked 7387 Write 3 New Done Done true
19354 6389 Acked 7388 Flush 0 New Done Done true
19355 6390 Acked 7389 Read 4 New Done Done true
19356 6391 Acked 7390 Read 8 New Done Done true
19357 6392 Acked 7391 Write 9 New Done Done true
19358 6393 Acked 7392 Write 5 New Done Done true
19359 6394 Acked 7393 Read 10 New Done Done true
19360 6395 Acked 7394 Write 9 New Done Done true
19361 6396 Acked 7395 Write 9 New Done Done true
19362 6397 Acked 7396 Write 10 New Done Done true
19363 6398 Acked 7397 Read 10 New Done Done true
19364 6399 Acked 7398 Read 4 New Done Done true
19365 6400 Acked 7399 Write 6 New Done Done true
19366 6401 Acked 7400 Write 2 New Done Done true
19367 6402 Acked 7401 Read 8 New Done Done true
19368 6403 Acked 7402 Write 8 New Done Done true
19369 6404 Acked 7403 Read 2 New Done Done true
19370 6405 Acked 7404 Write 4 New Done Done true
19371 6406 Acked 7405 Write 8 New Done Done true
19372 6407 Acked 7406 Write 6 New Done Done true
19373 6408 Acked 7407 Write 2 New Done Done true
19374 6409 Acked 7408 Read 6 New Done Done true
19375 6410 Acked 7409 Write 8 New Done Done true
19376 6411 Acked 7410 Read 4 New Done Done true
19377 6412 Acked 7411 Read 3 New Done Done true
19378 6413 Acked 7412 Read 10 New Done Done true
19379 6414 Acked 7413 Write 4 New Done Done true
19380 6415 Acked 7414 Read 7 New Done Done true
19381 6416 Acked 7415 Write 5 New Done Done true
19382 6417 Acked 7416 Write 3 New Done Done true
19383 6418 Acked 7417 Write 6 New Done Done true
19384 6419 Acked 7418 Read 10 New Done Done true
19385 6420 Acked 7419 Write 3 New Done Done true
19386 6421 Acked 7420 Write 10 New Done Done true
19387 6422 Acked 7421 Read 10 New Done Done true
19388 6423 Acked 7422 Write 5 New Done Done true
19389 6424 Acked 7423 Write 6 New Done Done true
19390 6425 Acked 7424 Read 5 New Done Done true
19391 6426 Acked 7425 Read 5 New Done Done true
19392 6427 Acked 7426 Write 3 New Done Done true
19393 6428 Acked 7427 Flush 0 New Done Done true
19394 6429 Acked 7428 Write 8 New Done Done true
19395 6430 Acked 7429 Write 2 New Done Done true
19396 6431 Acked 7430 Read 3 New Done Done true
19397 6432 Acked 7431 Write 7 New Done Done true
19398 6433 Acked 7432 Read 7 New Done Done true
19399 6434 Acked 7433 Flush 0 New Done Done true
19400 6435 Acked 7434 Write 10 New Done Done true
19401 6436 Acked 7435 Write 10 New Done Done true
19402 6437 Acked 7436 Read 3 New Done Done true
19403 6438 Acked 7437 Read 1 New Done Done true
19404 6439 Acked 7438 Read 6 New Done Done true
19405 6440 Acked 7439 Read 8 New Done Done true
19406 6441 Acked 7440 Flush 0 New Done Done true
19407 6442 Acked 7441 Write 5 New Done Done true
19408 6443 Acked 7442 Write 4 New Done Done true
19409 6444 Acked 7443 Write 4 New Done Done true
19410 6445 Acked 7444 Read 5 New Done Done true
19411 6446 Acked 7445 Write 8 New Done Done true
19412 6447 Acked 7446 Flush 0 New Done Done true
19413 6448 Acked 7447 Write 7 New Done Done true
19414 6449 Acked 7448 Write 4 New Done Done true
19415 6450 Acked 7449 Write 10 New Done Done true
19416 6451 Acked 7450 Write 2 New Done Done true
19417 6452 Acked 7451 Read 2 New Done Done true
19418 6453 Acked 7452 Read 3 New Done Done true
19419 6454 Acked 7453 Read 1 New Done Done true
19420 6455 Acked 7454 Flush 0 New Done Done true
19421 6456 Acked 7455 Write 4 New Done Done true
19422 6457 Acked 7456 Read 1 New Done Done true
19423 6458 Acked 7457 Read 1 New Done Done true
19424 6459 Acked 7458 Write 3 New Done Done true
19425 6460 Acked 7459 Read 5 New Done Done true
19426 6461 Acked 7460 Write 8 New Done Done true
19427 6462 Acked 7461 Write 9 New Done Done true
19428 6463 Acked 7462 Read 9 New Done Done true
19429 6464 Acked 7463 Read 6 New Done Done true
19430 6465 Acked 7464 Read 3 New Done Done true
19431 6466 Acked 7465 Read 4 New Done Done true
19432 6467 Acked 7466 Read 9 New Done Done true
19433 6468 Acked 7467 Read 10 New Done Done true
19434 6469 Acked 7468 Read 10 New Done Done true
19435 6470 Acked 7469 Write 5 New Done Done true
19436 6471 Acked 7470 Read 7 New Done Done true
19437 6472 Acked 7471 Write 7 New Done Done true
19438 6473 Acked 7472 Write 1 New Done Done true
19439 6474 Acked 7473 Write 4 New Done Done true
19440 6475 Acked 7474 Read 1 New Done Done true
19441 6476 Acked 7475 Write 10 New Done Done true
19442 6477 Acked 7476 Read 8 New Done Done true
19443 6478 Acked 7477 Write 9 New Done Done true
19444 6479 Acked 7478 Read 10 New Done Done true
19445 6480 Acked 7479 Flush 0 New Done Done true
19446 6481 Acked 7480 Read 5 New Done Done true
19447 6482 Acked 7481 Read 7 New Done Done true
19448 6483 Acked 7482 Write 6 New Done Done true
19449 6484 Acked 7483 Read 4 New Done Done true
19450 6485 Acked 7484 Read 5 New Done Done true
19451 6486 Acked 7485 Read 1 New Done Done true
19452 6487 Acked 7486 Read 6 New Done Done true
19453 6488 Acked 7487 Write 8 New Done Done true
19454 6489 Acked 7488 Write 1 New Done Done true
19455 6490 Acked 7489 Read 3 New Done Done true
19456 6491 Acked 7490 Write 4 New Done Done true
19457 6492 Acked 7491 Write 6 New Done Done true
19458 6493 Acked 7492 Read 3 New Done Done true
19459 6494 Acked 7493 Read 8 New Done Done true
19460 6495 Acked 7494 Write 5 New Done Done true
19461 6496 Acked 7495 Write 10 New Done Done true
19462 6497 Acked 7496 Write 6 New Done Done true
19463 6498 Acked 7497 Write 1 New Done Done true
19464 6499 Acked 7498 Write 6 New Done Done true
19465 6500 Acked 7499 Read 3 New Done Done true
19466 6501 Acked 7500 Read 10 New Done Done true
19467 6502 Acked 7501 Write 10 New Done Done true
19468 6503 Acked 7502 Write 4 New Done Done true
19469 6504 Acked 7503 Write 7 New Done Done true
19470 6505 Acked 7504 Write 5 New Done Done true
19471 6506 Acked 7505 Read 1 New Done Done true
19472 6507 Acked 7506 Flush 0 New Done Done true
19473 6508 Acked 7507 Read 7 New Done Done true
19474 6509 Acked 7508 Flush 0 New Done Done true
19475 6510 Acked 7509 Read 8 New Done Done true
19476 6511 Acked 7510 Write 6 New Done Done true
19477 6512 Acked 7511 Write 9 New Done Done true
19478 6513 Acked 7512 Read 6 New Done Done true
19479 6514 Acked 7513 Read 1 New Done Done true
19480 6515 Acked 7514 Write 4 New Done Done true
19481 6516 Acked 7515 Write 7 New Done Done true
19482 6517 Acked 7516 Read 5 New Done Done true
19483 6518 Acked 7517 Write 5 New Done Done true
19484 6519 Acked 7518 Read 8 New Done Done true
19485 6520 Acked 7519 Write 3 New Done Done true
19486 6521 Acked 7520 Read 3 New Done Done true
19487 6522 Acked 7521 Write 7 New Done Done true
19488 6523 Acked 7522 Read 3 New Done Done true
19489 6524 Acked 7523 Read 4 New Done Done true
19490 6525 Acked 7524 Read 7 New Done Done true
19491 6526 Acked 7525 Write 8 New Done Done true
19492 6527 Acked 7526 Read 2 New Done Done true
19493 6528 Acked 7527 Read 9 New Done Done true
19494 6529 Acked 7528 Write 1 New Done Done true
19495 6530 Acked 7529 Read 4 New Done Done true
19496 6531 Acked 7530 Read 7 New Done Done true
19497 6532 Acked 7531 Read 4 New Done Done true
19498 6533 Acked 7532 Write 3 New Done Done true
19499 6534 Acked 7533 Write 3 New Done Done true
19500 6535 Acked 7534 Write 7 New Done Done true
19501 6536 Acked 7535 Read 1 New Done Done true
19502 6537 Acked 7536 Write 5 New Done Done true
19503 6538 Acked 7537 Write 2 New Done Done true
19504 6539 Acked 7538 Write 3 New Done Done true
19505 6540 Acked 7539 Read 2 New Done Done true
19506 6541 Acked 7540 Read 1 New Done Done true
19507 6542 Acked 7541 Read 2 New Done Done true
19508 6543 Acked 7542 Write 10 New Done Done true
19509 6544 Acked 7543 Flush 0 New Done Done true
19510 6545 Acked 7544 Write 4 New Done Done true
19511 6546 Acked 7545 Flush 0 New Done Done true
19512 6547 Acked 7546 Read 6 New Done Done true
19513 6548 Acked 7547 Write 6 New Done Done true
19514 6549 Acked 7548 Read 7 New Done Done true
19515 6550 Acked 7549 Write 3 New Done Done true
19516 6551 Acked 7550 Read 6 New Done Done true
19517 6552 Acked 7551 Write 9 New Done Done true
19518 6553 Acked 7552 Write 3 New Done Done true
19519 6554 Acked 7553 Read 3 New Done Done true
19520 6555 Acked 7554 Read 1 New Done Done true
19521 6556 Acked 7555 Write 8 New Done Done true
19522 6557 Acked 7556 Read 5 New Done Done true
19523 6558 Acked 7557 Write 6 New Done Done true
19524 6559 Acked 7558 Write 6 New Done Done true
19525 6560 Acked 7559 Read 3 New Done Done true
19526 6561 Acked 7560 Write 6 New Done Done true
19527 6562 Acked 7561 Write 1 New Done Done true
19528 6563 Acked 7562 Write 8 New Done Done true
19529 6564 Acked 7563 Write 1 New Done Done true
19530 6565 Acked 7564 Flush 0 New Done Done true
19531 6566 Acked 7565 Flush 0 New Done Done true
19532 6567 Acked 7566 Write 10 New Done Done true
19533 6568 Acked 7567 Read 9 New Done Done true
19534 6569 Acked 7568 Read 3 New Done Done true
19535 6570 Acked 7569 Read 10 New Done Done true
19536 6571 Acked 7570 Write 3 New Done Done true
19537 6572 Acked 7571 Write 2 New Done Done true
19538 6573 Acked 7572 Read 10 New Done Done true
19539 6574 Acked 7573 Flush 0 New Done Done true
19540 6575 Acked 7574 Read 1 New Done Done true
19541 6576 Acked 7575 Read 9 New Done Done true
19542 6577 Acked 7576 Read 10 New Done Done true
19543 6578 Acked 7577 Read 7 New Done Done true
19544 6579 Acked 7578 Flush 0 New Done Done true
19545 6580 Acked 7579 Read 7 New Done Done true
19546 6581 Acked 7580 Read 5 New Done Done true
19547 6582 Acked 7581 Read 2 New Done Done true
19548 6583 Acked 7582 Read 6 New Done Done true
19549 6584 Acked 7583 Read 4 New Done Done true
19550 6585 Acked 7584 Write 6 New Done Done true
19551 6586 Acked 7585 Read 1 New Done Done true
19552 6587 Acked 7586 Flush 0 New Done Done true
19553 6588 Acked 7587 Flush 0 New Done Done true
19554 6589 Acked 7588 Read 6 New Done Done true
19555 6590 Acked 7589 Write 4 New Done Done true
19556 6591 Acked 7590 Flush 0 New Done Done true
19557 6592 Acked 7591 Flush 0 New Done Done true
19558 6593 Acked 7592 Write 2 New Done Done true
19559 6594 Acked 7593 Read 5 New Done Done true
19560 6595 Acked 7594 Write 7 New Done Done true
19561 6596 Acked 7595 Write 8 New Done Done true
19562 6597 Acked 7596 Read 4 New Done Done true
19563 6598 Acked 7597 Read 5 New Done Done true
19564 6599 Acked 7598 Write 5 New Done Done true
19565 6600 Acked 7599 Read 7 New Done Done true
19566 6601 Acked 7600 Read 4 New Done Done true
19567 6602 Acked 7601 Write 10 New Done Done true
19568 6603 Acked 7602 Read 9 New Done Done true
19569 6604 Acked 7603 Read 8 New Done Done true
19570 6605 Acked 7604 Write 2 New Done Done true
19571 6606 Acked 7605 Flush 0 New Done Done true
19572 6607 Acked 7606 Write 6 New Done Done true
19573 6608 Acked 7607 Read 7 New Done Done true
19574 6609 Acked 7608 Read 7 New Done Done true
19575 6610 Acked 7609 Write 1 New Done Done true
19576 6611 Acked 7610 Write 6 New Done Done true
19577 6612 Acked 7611 Write 6 New Done Done true
19578 6613 Acked 7612 Write 3 New Done Done true
19579 6614 Acked 7613 Read 4 New Done Done true
19580 6615 Acked 7614 Write 2 New Done Done true
19581 6616 Acked 7615 Flush 0 New Done Done true
19582 6617 Acked 7616 Read 10 New Done Done true
19583 6618 Acked 7617 Write 4 New Done Done true
19584 6619 Acked 7618 Read 9 New Done Done true
19585 6620 Acked 7619 Write 1 New Done Done true
19586 6621 Acked 7620 Write 1 New Done Done true
19587 6622 Acked 7621 Write 3 New Done Done true
19588 6623 Acked 7622 Flush 0 New Done Done true
19589 6624 Acked 7623 Read 6 New Done Done true
19590 6625 Acked 7624 Read 3 New Done Done true
19591 6626 Acked 7625 Read 3 New Done Done true
19592 6627 Acked 7626 Read 2 New Done Done true
19593 6628 Acked 7627 Read 5 New Done Done true
19594 6629 Acked 7628 Write 10 New Done Done true
19595 6630 Acked 7629 Read 1 New Done Done true
19596 6631 Acked 7630 Flush 0 New Done Done true
19597 STATES DS:0 DS:1 DS:2 TOTAL
19598 New 300 0 0 300
19599 Sent 0 0 0 0
19600 Done 0 300 300 600
19601 Skipped 0 0 0 0
19602 Error 0 0 0 0
19603 Last Flush: 7330 7630 7630
19604 Downstairs last five completed: 7330 7329 7328 7327 7326
19605 Upstairs last five completed: 6631 6630 6629 6628 6627
19606 CLIENT: Up:0 ds:300 act:2
196072024-02-27T21:48:30.425ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
196082024-02-27T21:48:30.425ZINFOcrucible: ds_connection connected = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
196092024-02-27T21:48:30.427ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
196102024-02-27T21:48:30.427ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
196112024-02-27T21:48:30.427ZINFOcrucible: send last flush ID to this DS: 7330 = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
196122024-02-27T21:48:30.427ZINFOcrucible: Replied this last flush ID: 7330 = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
196132024-02-27T21:48:30.427ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
196142024-02-27T21:48:30.427ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
196152024-02-27T21:48:30.427ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
196162024-02-27T21:48:30.427ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
19617 ----------------------------------------------------------------
19618 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
19619 Downstairs last five completed: 7630 7629 7628 7627 7626
19620 Upstairs last five completed: 6631 6630 6629 6628 6627
19621 CLIENT: Up:0 ds:0 act:3
19622 Replay: All jobs finished, all DS active.
19623 Main task has work to do, go find it
19624 got dsc Stop(1)
19625 stop 1
19626 Feb 27 21:48:33.753[8820] Got stop action so:false kr:false
19627 INFO request completed, latency_us: 337, response_code: 204, uri: /stop/cid/1, method: POST, req_id: e49093c6-5297-41d3-84c0-58cfa3fb6734, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
196282024-02-27T21:48:33.754ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
196292024-02-27T21:48:33.754ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
196302024-02-27T21:48:33.754ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = e6a5a974-925a-494f-8845-facc441c2690
196312024-02-27T21:48:33.754ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
19632 Feb 27 21:48:33.754 INFO request completed, latency_us: 155, response_code: 200, uri: /state/cid/1, method: GET{"msg":"[1] client re-new 0 jobs since flush 7630","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:48:33.754322685Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"downstairs","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
19633 , req_id: e9e0fc51-76b9-4b23-a8c4-1ffebe45042b, remote_addr: 127.0.0.1:42901, local_addr: 127.0.{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:48:33.754352601Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"io task","client":"1","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
19634 0.1:9998
19635 [8820] Exited with: ExitStatus(unix_wait_status(9))
19636 [8820][1] reports Exit
196372024-02-27T21:48:34.755ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
196382024-02-27T21:48:34.755ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
196392024-02-27T21:48:34.755ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
196402024-02-27T21:48:34.755ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
196412024-02-27T21:48:34.755ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
196422024-02-27T21:48:34.755ZINFOcrucible: [1] client re-new 0 jobs since flush 7630 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
196432024-02-27T21:48:34.755ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
196442024-02-27T21:48:35.756ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
196452024-02-27T21:48:35.756ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
196462024-02-27T21:48:35.756ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
196472024-02-27T21:48:35.756ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
196482024-02-27T21:48:35.756ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
196492024-02-27T21:48:35.756ZINFOcrucible: [1] client re-new 0 jobs since flush 7630 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
196502024-02-27T21:48:35.756ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
196512024-02-27T21:48:36.757ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
196522024-02-27T21:48:36.757ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
196532024-02-27T21:48:36.757ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
196542024-02-27T21:48:36.757ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
196552024-02-27T21:48:36.757ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
196562024-02-27T21:48:36.757ZINFOcrucible: [1] client re-new 0 jobs since flush 7630 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
196572024-02-27T21:48:36.757ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
19658 Feb 27 21:48:37.756 INFO request completed, latency_us: 211, response_code: 200, uri: /state/cid/1, method: GET, req_id: 2694487c-c5b4-4eb7-bb4e-ce3d73756a3b, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
19659 001/300 Read block 1641 len 8192
19660 002/300 Write block 4608 len 20480 data: 6 5 5 6 4
196612024-02-27T21:48:37.758ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
196622024-02-27T21:48:37.759ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
196632024-02-27T21:48:37.759ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
196642024-02-27T21:48:37.759ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
196652024-02-27T21:48:37.759ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
196662024-02-27T21:48:37.759ZINFOcrucible: [1] client re-new 2 jobs since flush 7630 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
196672024-02-27T21:48:37.759ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
19668 003/300 Write block 2215 len 36864 data: 5 4 4 3 3 3 4 3 4
19669 004/300 Read block 2413 len 32768
19670 005/300 Write block 663 len 12288 data: 5 4 6
19671 006/300 Read block 4678 len 8192
19672 007/300 Read block 539 len 12288
19673 008/300 Write block 2206 len 40960 data: 8 8 6 5 5 5 5 5 4 6
19674 009/300 Write block 2879 len 32768 data: 6 5 5 6 5 5 6 5
19675 010/300 Read block 3942 len 16384
19676 011/300 Read block 747 len 40960
19677 012/300 Write block 1037 len 16384 data: 6 7 7 7
19678 013/300 Read block 2666 len 4096
19679 014/300 Flush
19680 015/300 Write block 4592 len 4096 data: 5
19681 016/300 Read block 2884 len 36864
19682 017/300 Write block 2862 len 16384 data: 5 5 3 2
19683 018/300 Read block 323 len 32768
19684 019/300 Write block 29 len 40960 data: 5 4 4 4 4 3 3 3 3 3
19685 020/300 Read block 2139 len 28672
19686 021/300 Read block 1363 len 28672
19687 022/300 Read block 167 len 4096
19688 023/300 Write block 1946 len 28672 data: 5 5 5 5 3 3 3
19689 024/300 Write block 3481 len 12288 data: 5 7 6
19690 025/300 Write block 2191 len 4096 data: 8
19691 026/300 Write block 3320 len 20480 data: 9 9 9 9 8
19692 027/300 Write block 3777 len 4096 data: 5
19693 028/300 Read block 3432 len 36864
19694 029/300 Read block 319 len 20480
19695 030/300 Write block 495 len 12288 data: 7 6 7
19696 031/300 Read block 4059 len 28672
19697 032/300 Read block 2330 len 24576
19698 033/300 Read block 1485 len 16384
19699 034/300 Write block 3323 len 4096 data: 10
19700 035/300 Write block 2188 len 20480 data: 6 6 6 9 7
19701 036/300 Flush
19702 037/300 Flush
19703 038/300 Read block 399 len 20480
19704 039/300 Flush
19705 040/300 Write block 4626 len 40960 data: 6 6 7 7 8 7 7 7 6 4
19706 041/300 Read block 1824 len 8192
19707 042/300 Write block 1013 len 12288 data: 6 6 4
19708 043/300 Write block 2757 len 40960 data: 3 3 4 4 5 5 5 4 3 3
19709 044/300 Read block 695 len 36864
19710 045/300 Flush
19711 046/300 Read block 525 len 28672
19712 047/300 Write block 3321 len 12288 data: 10 10 11
19713 048/300 Write block 1874 len 16384 data: 4 4 5 4
19714 049/300 Read block 4747 len 40960
19715 050/300 Write block 4884 len 8192 data: 5 6
19716 051/300 Read block 1760 len 20480
19717 052/300 Write block 1985 len 20480 data: 2 2 2 2 4
19718 053/300 Write block 801 len 20480 data: 2 2 3 4 3
19719 054/300 Read block 4210 len 8192
19720 055/300 Read block 537 len 24576
19721 056/300 Write block 4361 len 20480 data: 6 6 6 6 6
19722 057/300 Read block 1384 len 16384
19723 058/300 Flush
19724 059/300 Write block 2158 len 40960 data: 7 7 8 6 5 5 4 4 4 6
19725 060/300 Flush
19726 061/300 Write block 2293 len 36864 data: 5 4 4 5 5 5 5 5 5
19727 062/300 Write block 560 len 40960 data: 3 4 5 5 5 5 6 5 4 2
19728 063/300 Write block 4336 len 4096 data: 4
19729 064/300 Read block 4745 len 16384
19730 065/300 Write block 4509 len 40960 data: 4 4 7 8 9 7 7 7 7 7
19731 066/300 Write block 2257 len 28672 data: 6 6 5 5 5 5 4
19732 067/300 Write block 3761 len 32768 data: 3 2 4 4 4 4 4 5
19733 068/300 Write block 1913 len 20480 data: 5 6 4 4 5
19734 069/300 Write block 718 len 4096 data: 5
19735 070/300 Write block 3249 len 4096 data: 4
19736 071/300 Read block 85 len 12288
19737 072/300 Write block 178 len 24576 data: 9 9 10 7 8 8
19738 073/300 Write block 2937 len 32768 data: 5 6 4 3 3 5 6 5
19739 074/300 Write block 1126 len 8192 data: 5 5
19740 075/300 Write block 4224 len 16384 data: 3 3 4 3
19741 076/300 Write block 1649 len 32768 data: 6 5 4 3 4 3 4 4
19742 077/300 Write block 796 len 24576 data: 4 4 3 3 3 3
19743 078/300 Read block 622 len 16384
19744 079/300 Read block 2024 len 20480
19745 080/300 Write block 969 len 4096 data: 4
19746 081/300 Flush
19747 082/300 Read block 140 len 40960
19748 083/300 Read block 4091 len 28672
19749 084/300 Flush
19750 085/300 Write block 3567 len 36864 data: 3 4 5 7 7 7 7 6 6
19751 086/300 Write block 1910 len 8192 data: 4 5
19752 087/300 Write block 607 len 8192 data: 6 7
19753 088/300 Read block 80 len 12288
19754 089/300 Read block 1494 len 28672
19755 090/300 Write block 4210 len 24576 data: 4 4 4 4 4 4
19756 091/300 Read block 2385 len 4096
19757 092/300 Write block 3221 len 16384 data: 6 6 5 5
19758 093/300 Flush
19759 094/300 Read block 343 len 20480
19760 095/300 Read block 1088 len 28672
19761 096/300 Read block 901 len 20480
19762 097/300 Write block 2220 len 20480 data: 4 5 4 5 4
19763 098/300 Write block 4323 len 36864 data: 6 6 5 5 6 7 5 4 3
19764 099/300 Write block 3846 len 40960 data: 4 5 4 6 8 8 9 9 6 7
19765 100/300 Read block 1153 len 36864
19766 101/300 Read block 1444 len 28672
19767 102/300 Write block 3236 len 16384 data: 5 5 6 8
19768 103/300 Write block 320 len 24576 data: 4 3 4 3 3 3
19769 104/300 Read block 3614 len 36864
19770 105/300 Flush
19771 106/300 Write block 754 len 8192 data: 5 5
19772 107/300 Read block 293 len 4096
19773 108/300 Read block 306 len 12288
19774 109/300 Write block 3108 len 28672 data: 5 7 9 10 8 8 9
19775 110/300 Read block 4274 len 36864
19776 111/300 Read block 3085 len 20480
19777 112/300 Write block 472 len 4096 data: 5
19778 113/300 Read block 514 len 32768
19779 114/300 Read block 140 len 32768
19780 115/300 Read block 2944 len 36864
19781 116/300 Read block 1274 len 24576
19782 117/300 Write block 2309 len 32768 data: 5 5 6 6 6 5 5 5
19783 118/300 Flush
19784 119/300 Write block 4612 len 8192 data: 5 4
19785 120/300 Read block 3540 len 40960
19786 121/300 Read block 1343 len 8192
19787 122/300 Flush
19788 123/300 Write block 4225 len 24576 data: 4 5 4 3 4 4
19789 124/300 Write block 1196 len 20480 data: 8 6 6 5 4
19790 125/300 Read block 465 len 4096
19791 126/300 Read block 3637 len 28672
19792 127/300 Read block 2710 len 40960
19793 128/300 Write block 2611 len 28672 data: 5 5 7 7 8 6 6
19794 129/300 Read block 1118 len 32768
19795 130/300 Read block 2791 len 12288
19796 131/300 Read block 1330 len 36864
19797 132/300 Write block 4312 len 4096 data: 2
19798 133/300 Read block 4779 len 16384
19799 134/300 Flush
19800 135/300 Write block 2546 len 40960 data: 5 5 5 4 4 3 2 2 4 5
19801 136/300 Read block 153 len 12288
19802 137/300 Write block 3384 len 20480 data: 4 4 5 6 5
19803 138/300 Read block 3718 len 20480
19804 139/300 Write block 2554 len 16384 data: 5 6 5 4
19805 140/300 Write block 23 len 36864 data: 8 8 8 8 7 6 6 5 5
19806 141/300 Read block 1126 len 4096
19807 142/300 Read block 3138 len 16384
19808 143/300 Read block 3887 len 36864
19809 144/300 Write block 4568 len 4096 data: 5
19810 145/300 Read block 3896 len 8192
19811 146/300 Read block 1572 len 24576
19812 147/300 Flush
19813 148/300 Read block 707 len 16384
19814 149/300 Read block 500 len 28672
19815 150/300 Read block 3934 len 4096
19816 151/300 Read block 4605 len 12288
19817 152/300 Flush
19818 153/300 Write block 3544 len 4096 data: 6
19819 154/300 Write block 1914 len 24576 data: 7 5 5 6 5 5
19820 155/300 Read block 3380 len 16384
19821 156/300 Write block 2279 len 12288 data: 6 7 6
19822 157/300 Write block 3766 len 36864 data: 5 5 6 5 5 5 3 4 5
19823 158/300 Read block 2738 len 16384
19824 159/300 Read block 1974 len 28672
19825 160/300 Write block 1218 len 36864 data: 7 6 6 6 4 4 5 5 7
19826 161/300 Flush
19827 162/300 Write block 1059 len 32768 data: 4 4 4 4 5 5 4 4
19828 163/300 Write block 646 len 28672 data: 3 2 2 3 3 2 3
19829 164/300 Flush
19830 165/300 Read block 4537 len 28672
19831 166/300 Read block 1572 len 12288
19832 167/300 Read block 3185 len 16384
19833 168/300 Read block 2051 len 12288
19834 169/300 Flush
19835 170/300 Flush
19836 171/300 Write block 3593 len 16384 data: 6 6 8 7
19837 172/300 Read block 337 len 40960
19838 173/300 Write block 1062 len 36864 data: 5 6 6 5 5 4 3 4 4
19839 174/300 Read block 1574 len 16384
19840 175/300 Write block 3074 len 4096 data: 3
19841 176/300 Read block 367 len 12288
19842 177/300 Flush
19843 178/300 Write block 3643 len 40960 data: 5 5 3 3 3 3 3 3 3 4
19844 179/300 Flush
19845 180/300 Write block 2231 len 36864 data: 5 4 4 4 4 4 4 5 6
19846 181/300 Read block 1632 len 4096
19847 182/300 Write block 3318 len 32768 data: 9 9 10 11 11 12 9 8
19848 183/300 Read block 4724 len 20480
19849 184/300 Read block 4522 len 4096
19850 185/300 Write block 3924 len 12288 data: 5 5 5
19851 186/300 Read block 4405 len 24576
19852 187/300 Read block 3444 len 8192
19853 188/300 Read block 3051 len 28672
19854 189/300 Write block 1412 len 32768 data: 3 2 2 2 2 4 3 4
19855 190/300 Read block 2844 len 16384
19856 191/300 Write block 3838 len 12288 data: 6 6 7
19857 192/300 Write block 2350 len 40960 data: 6 5 6 5 6 6 7 6 5 4
19858 193/300 Write block 428 len 36864 data: 5 6 5 5 6 6 5 6 6
19859 194/300 Read block 1454 len 32768
19860 195/300 Write block 2227 len 8192 data: 4 3
19861 196/300 Write block 1823 len 40960 data: 3 3 5 6 8 7 6 7 8 8
19862 197/300 Write block 3238 len 8192 data: 7 9
19863 198/300 Write block 929 len 28672 data: 4 4 4 3 4 4 4
19864 199/300 Read block 3770 len 36864
19865 200/300 Read block 3033 len 40960
19866 201/300 Read block 3535 len 32768
19867 202/300 Write block 3482 len 8192 data: 8 7
19868 203/300 Read block 3824 len 24576
19869 204/300 Write block 3488 len 32768 data: 5 6 6 6 8 7 8 5
19870 205/300 Flush
19871 206/300 Read block 1625 len 20480
19872 207/300 Read block 3836 len 20480
19873 208/300 Read block 114 len 8192
19874 209/300 Read block 4821 len 4096
19875 210/300 Write block 4471 len 20480 data: 2 2 4 3 5
19876 211/300 Read block 1262 len 12288
19877 212/300 Read block 4469 len 36864
19878 213/300 Read block 1835 len 32768
19879 214/300 Flush
19880 215/300 Write block 4550 len 24576 data: 8 7 7 6 5 3
19881 216/300 Read block 1109 len 8192
19882 217/300 Read block 2256 len 20480
19883 218/300 Write block 3798 len 16384 data: 8 8 9 8
19884 219/300 Write block 951 len 16384 data: 5 4 3 3
19885 220/300 Write block 3216 len 4096 data: 5
19886 221/300 Read block 1187 len 8192
19887 222/300 Write block 2293 len 8192 data: 6 5
19888 223/300 Read block 4408 len 36864
19889 224/300 Write block 383 len 8192 data: 4 3
19890 225/300 Read block 1150 len 40960
19891 226/300 Read block 830 len 24576
19892 227/300 Read block 3972 len 40960
19893 228/300 Write block 3081 len 40960 data: 3 3 3 3 3 3 3 3 3 3
19894 229/300 Read block 2498 len 16384
19895 230/300 Write block 1641 len 8192 data: 7 7
19896 231/300 Read block 2315 len 24576
19897 232/300 Write block 2089 len 20480 data: 3 2 2 2 2
19898 233/300 Read block 4531 len 32768
19899 234/300 Write block 741 len 20480 data: 6 6 5 4 5
19900 235/300 Read block 584 len 20480
19901 236/300 Read block 40 len 20480
19902 237/300 Read block 1352 len 16384
19903 238/300 Write block 413 len 32768 data: 7 7 7 6 5 7 8 8
19904 239/300 Read block 2161 len 16384
19905 240/300 Read block 818 len 24576
19906 241/300 Read block 1766 len 24576
19907 242/300 Flush
19908 243/300 Write block 1259 len 40960 data: 5 5 4 4 5 4 5 6 7 8
19909 244/300 Read block 309 len 40960
19910 245/300 Read block 2445 len 20480
19911 246/300 Flush
19912 247/300 Read block 2118 len 40960
19913 248/300 Write block 3547 len 40960 data: 4 4 4 4 4 4 4 5 7 8
19914 249/300 Read block 4490 len 32768
19915 250/300 Read block 2113 len 28672
19916 251/300 Write block 2895 len 8192 data: 3 2
19917 252/300 Flush
19918 253/300 Write block 334 len 24576 data: 5 5 5 5 4 4
19919 254/300 Write block 2657 len 32768 data: 7 7 5 5 6 7 8 7
19920 255/300 Write block 2021 len 4096 data: 5
19921 256/300 Write block 1778 len 24576 data: 3 3 3 5 5 5
19922 257/300 Flush
19923 258/300 Write block 4608 len 8192 data: 7 6
19924 259/300 Write block 4438 len 32768 data: 7 6 7 6 5 5 7 6
19925 260/300 Write block 518 len 32768 data: 7 6 6 6 7 8 6 7
19926 261/300 Read block 1312 len 8192
19927 262/300 Write block 3803 len 20480 data: 6 5 5 4 3
19928 263/300 Read block 501 len 12288
19929 264/300 Write block 4765 len 4096 data: 5
19930 265/300 Read block 3006 len 4096
19931 266/300 Write block 3028 len 12288 data: 6 7 6
19932 267/300 Read block 3497 len 20480
19933 268/300 Write block 4069 len 36864 data: 5 5 5 6 6 6 8 3 2
19934 269/300 Write block 973 len 32768 data: 5 4 4 4 5 4 4 4
19935 270/300 Read block 2674 len 20480
19936 271/300 Write block 2080 len 8192 data: 4 4
19937 272/300 Write block 571 len 40960 data: 3 3 3 3 4 4 4 4 4 5
19938 273/300 Read block 3703 len 32768
19939 274/300 Read block 482 len 8192
19940 275/300 Read block 4640 len 28672
19941 276/300 Write block 3773 len 40960 data: 5 6 5 5 6 5 5 4 3 3
19942 277/300 Write block 4294 len 12288 data: 3 4 4
19943 278/300 Read block 3633 len 12288
19944 279/300 Read block 4326 len 36864
19945 280/300 Read block 508 len 24576
19946 281/300 Flush
19947 282/300 Flush
19948 283/300 Write block 3537 len 20480 data: 6 7 7 7 6
19949 284/300 Read block 4402 len 4096
19950 285/300 Read block 645 len 8192
19951 286/300 Write block 3008 len 24576 data: 6 6 6 5 5 4
19952 287/300 Read block 4753 len 28672
19953 288/300 Flush
19954 289/300 Read block 4901 len 36864
19955 290/300 Flush
19956 291/300 Write block 3799 len 28672 data: 9 10 9 6 7 6 6
19957 292/300 Flush
19958 293/300 Write block 4429 len 28672 data: 4 6 7 7 6 5 7
19959 294/300 Write block 1784 len 24576 data: 5 5 4 3 2 4
19960 295/300 Read block 3502 len 40960
19961 296/300 Write block 3356 len 4096 data: 4
19962 297/300 Read block 2556 len 8192
19963 298/300 Write block 4382 len 32768 data: 4 4 3 4 4 3 3 3
19964 299/300 Read block 1135 len 40960
19965 300/300 Write block 1219 len 40960 data: 7 7 7 5 5 6 6 8 6 7
19966 Main task has work to do, go find it
19967 got dsc Start(1)
19968 start 1
19969 Feb 27 21:48:38.094 INFO request completed, latency_us: 210, response_code: 204, uri: /start/cid/1, method: POST, req_id: 94886db0-5e82-4415-9512-b9953dea4119, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
19970 Replay: started 1, returned:Ok(())
19971 ----------------------------------------------------------------
19972 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
19973 States: Active Offline Active
19974 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
19975 6632 Acked 7631 Read 2 Done New Done true
19976 6633 Acked 7632 Write 5 Done New Done true
19977 6634 Acked 7633 Write 9 Done New Done false
19978 6635 Acked 7634 Read 8 Done New Done false
19979 6636 Acked 7635 Write 3 Done New Done false
19980 6637 Acked 7636 Read 2 Done New Done false
19981 6638 Acked 7637 Read 3 Done New Done false
19982 6639 Acked 7638 Write 10 Done New Done false
19983 6640 Acked 7639 Write 8 Done New Done false
19984 6641 Acked 7640 Read 4 Done New Done false
19985 6642 Acked 7641 Read 10 Done New Done false
19986 6643 Acked 7642 Write 4 Done New Done false
19987 6644 Acked 7643 Read 1 Done New Done false
19988 6645 Acked 7644 Flush 0 Done New Done false
19989 6646 Acked 7645 Write 1 Done New Done false
19990 6647 Acked 7646 Read 9 Done New Done false
19991 6648 Acked 7647 Write 4 Done New Done false
19992 6649 Acked 7648 Read 8 Done New Done false
19993 6650 Acked 7649 Write 10 Done New Done false
19994 6651 Acked 7650 Read 7 Done New Done false
19995 6652 Acked 7651 Read 7 Done New Done false
19996 6653 Acked 7652 Read 1 Done New Done false
19997 6654 Acked 7653 Write 7 Done New Done false
19998 6655 Acked 7654 Write 3 Done New Done false
19999 6656 Acked 7655 Write 1 Done New Done false
20000 6657 Acked 7656 Write 5 Done New Done false
20001 6658 Acked 7657 Write 1 Done New Done false
20002 6659 Acked 7658 Read 9 Done New Done false
20003 6660 Acked 7659 Read 5 Done New Done false
20004 6661 Acked 7660 Write 3 Done New Done false
20005 6662 Acked 7661 Read 7 Done New Done false
20006 6663 Acked 7662 Read 6 Done New Done false
20007 6664 Acked 7663 Read 4 Done New Done false
20008 6665 Acked 7664 Write 1 Done New Done false
20009 6666 Acked 7665 Write 5 Done New Done false
20010 6667 Acked 7666 Flush 0 Done New Done false
20011 6668 Acked 7667 Flush 0 Done New Done false
20012 6669 Acked 7668 Read 5 Done New Done false
20013 6670 Acked 7669 Flush 0 Done New Done false
20014 6671 Acked 7670 Write 10 Done New Done false
20015 6672 Acked 7671 Read 2 Done New Done false
20016 6673 Acked 7672 Write 3 Done New Done false
20017 6674 Acked 7673 Write 10 Done New Done false
20018 6675 Acked 7674 Read 9 Done New Done false
20019 6676 Acked 7675 Flush 0 Done New Done false
20020 6677 Acked 7676 Read 7 Done New Done false
20021 6678 Acked 7677 Write 3 Done New Done false
20022 6679 Acked 7678 Write 4 Done New Done false
20023 6680 Acked 7679 Read 10 Done New Done false
20024 6681 Acked 7680 Write 2 Done New Done false
20025 6682 Acked 7681 Read 5 Done New Done false
20026 6683 Acked 7682 Write 5 Done New Done false
20027 6684 Acked 7683 Write 5 Done New Done false
20028 6685 Acked 7684 Read 2 Done New Done false
20029 6686 Acked 7685 Read 6 Done New Done false
20030 6687 Acked 7686 Write 5 Done New Done false
20031 6688 Acked 7687 Read 4 Done New Done false
20032 6689 Acked 7688 Flush 0 Done New Done false
20033 6690 Acked 7689 Write 10 Done New Done false
20034 6691 Acked 7690 Flush 0 Done New Done false
20035 6692 Acked 7691 Write 9 Done New Done false
20036 6693 Acked 7692 Write 10 Done New Done false
20037 6694 Acked 7693 Write 1 Done New Done false
20038 6695 Acked 7694 Read 4 Done New Done false
20039 6696 Acked 7695 Write 10 Done New Done false
20040 6697 Acked 7696 Write 7 Done New Done false
20041 6698 Acked 7697 Write 8 Done New Done false
20042 6699 Acked 7698 Write 5 Done New Done false
20043 6700 Acked 7699 Write 1 Done New Done false
20044 6701 Acked 7700 Write 1 Done New Done false
20045 6702 Acked 7701 Read 3 Done New Done false
20046 6703 Acked 7702 Write 6 Done New Done false
20047 6704 Acked 7703 Write 8 Done New Done false
20048 6705 Acked 7704 Write 2 Done New Done false
20049 6706 Acked 7705 Write 4 Done New Done false
20050 6707 Acked 7706 Write 8 Done New Done false
20051 6708 Acked 7707 Write 6 Done New Done false
20052 6709 Acked 7708 Read 4 Done New Done false
20053 6710 Acked 7709 Read 5 Done New Done false
20054 6711 Acked 7710 Write 1 Done New Done false
20055 6712 Acked 7711 Flush 0 Done New Done false
20056 6713 Acked 7712 Read 10 Done New Done false
20057 6714 Acked 7713 Read 7 Done New Done false
20058 6715 Acked 7714 Flush 0 Done New Done false
20059 6716 Acked 7715 Write 9 Done New Done false
20060 6717 Acked 7716 Write 2 Done New Done false
20061 6718 Acked 7717 Write 2 Done New Done false
20062 6719 Acked 7718 Read 3 Done New Done false
20063 6720 Acked 7719 Read 7 Done New Done false
20064 6721 Acked 7720 Write 6 Done New Done false
20065 6722 Acked 7721 Read 1 Done New Done false
20066 6723 Acked 7722 Write 4 Done New Done false
20067 6724 Acked 7723 Flush 0 Done New Done false
20068 6725 Acked 7724 Read 5 Done New Done false
20069 6726 Acked 7725 Read 7 Done New Done false
20070 6727 Acked 7726 Read 5 Done New Done false
20071 6728 Acked 7727 Write 5 Done New Done false
20072 6729 Acked 7728 Write 9 Done New Done false
20073 6730 Acked 7729 Write 10 Done New Done false
20074 6731 Acked 7730 Read 9 Done New Done false
20075 6732 Acked 7731 Read 7 Done New Done false
20076 6733 Acked 7732 Write 4 Done New Done false
20077 6734 Acked 7733 Write 6 Done New Done false
20078 6735 Acked 7734 Read 9 Done New Done false
20079 6736 Acked 7735 Flush 0 Done New Done false
20080 6737 Acked 7736 Write 2 Done New Done false
20081 6738 Acked 7737 Read 1 Done New Done false
20082 6739 Acked 7738 Read 3 Done New Done false
20083 6740 Acked 7739 Write 7 Done New Done false
20084 6741 Acked 7740 Read 9 Done New Done false
20085 6742 Acked 7741 Read 5 Done New Done false
20086 6743 Acked 7742 Write 1 Done New Done false
20087 6744 Acked 7743 Read 8 Done New Done false
20088 6745 Acked 7744 Read 8 Done New Done false
20089 6746 Acked 7745 Read 9 Done New Done false
20090 6747 Acked 7746 Read 6 Done New Done false
20091 6748 Acked 7747 Write 8 Done New Done false
20092 6749 Acked 7748 Flush 0 Done New Done false
20093 6750 Acked 7749 Write 2 Done New Done false
20094 6751 Acked 7750 Read 10 Done New Done false
20095 6752 Acked 7751 Read 2 Done New Done false
20096 6753 Acked 7752 Flush 0 Done New Done false
20097 6754 Acked 7753 Write 6 Done New Done false
20098 6755 Acked 7754 Write 5 Done New Done false
20099 6756 Acked 7755 Read 1 Done New Done false
20100 6757 Acked 7756 Read 7 Done New Done false
20101 6758 Acked 7757 Read 10 Done New Done false
20102 6759 Acked 7758 Write 7 Done New Done false
20103 6760 Acked 7759 Read 8 Done New Done false
20104 6761 Acked 7760 Read 3 Done New Done false
20105 6762 Acked 7761 Read 9 Done New Done false
20106 6763 Acked 7762 Write 1 Done New Done false
20107 6764 Acked 7763 Read 4 Done New Done false
20108 6765 Acked 7764 Flush 0 Done New Done false
20109 6766 Acked 7765 Write 10 Done New Done false
20110 6767 Acked 7766 Read 3 Done New Done false
20111 6768 Acked 7767 Write 5 Done New Done false
20112 6769 Acked 7768 Read 5 Done New Done false
20113 6770 Acked 7769 Write 4 Done New Done false
20114 6771 Acked 7770 Write 9 Done New Done false
20115 6772 Acked 7771 Read 1 Done New Done false
20116 6773 Acked 7772 Read 4 Done New Done false
20117 6774 Acked 7773 Read 9 Done New Done false
20118 6775 Acked 7774 Write 1 Done New Done false
20119 6776 Acked 7775 Read 2 Done New Done false
20120 6777 Acked 7776 Read 6 Done New Done false
20121 6778 Acked 7777 Flush 0 Done New Done false
20122 6779 Acked 7778 Read 4 Done New Done false
20123 6780 Acked 7779 Read 7 Done New Done false
20124 6781 Acked 7780 Read 1 Done New Done false
20125 6782 Acked 7781 Read 3 Done New Done false
20126 6783 Acked 7782 Flush 0 Done New Done false
20127 6784 Acked 7783 Write 1 Done New Done false
20128 6785 Acked 7784 Write 6 Done New Done false
20129 6786 Acked 7785 Read 4 Done New Done false
20130 6787 Acked 7786 Write 3 Done New Done false
20131 6788 Acked 7787 Write 9 Done New Done false
20132 6789 Acked 7788 Read 4 Done New Done false
20133 6790 Acked 7789 Read 7 Done New Done false
20134 6791 Acked 7790 Write 9 Done New Done false
20135 6792 Acked 7791 Flush 0 Done New Done false
20136 6793 Acked 7792 Write 8 Done New Done false
20137 6794 Acked 7793 Write 7 Done New Done false
20138 6795 Acked 7794 Flush 0 Done New Done false
20139 6796 Acked 7795 Read 7 Done New Done false
20140 6797 Acked 7796 Read 3 Done New Done false
20141 6798 Acked 7797 Read 4 Done New Done false
20142 6799 Acked 7798 Read 3 Done New Done false
20143 6800 Acked 7799 Flush 0 Done New Done false
20144 6801 Acked 7800 Flush 0 Done New Done false
20145 6802 Acked 7801 Write 4 Done New Done false
20146 6803 Acked 7802 Read 10 Done New Done false
20147 6804 Acked 7803 Write 9 Done New Done false
20148 6805 Acked 7804 Read 4 Done New Done false
20149 6806 Acked 7805 Write 1 Done New Done false
20150 6807 Acked 7806 Read 3 Done New Done false
20151 6808 Acked 7807 Flush 0 Done New Done false
20152 6809 Acked 7808 Write 10 Done New Done false
20153 6810 Acked 7809 Flush 0 Done New Done false
20154 6811 Acked 7810 Write 9 Done New Done false
20155 6812 Acked 7811 Read 1 Done New Done false
20156 6813 Acked 7812 Write 8 Done New Done false
20157 6814 Acked 7813 Read 5 Done New Done false
20158 6815 Acked 7814 Read 1 Done New Done false
20159 6816 Acked 7815 Write 3 Done New Done false
20160 6817 Acked 7816 Read 6 Done New Done false
20161 6818 Acked 7817 Read 2 Done New Done false
20162 6819 Acked 7818 Read 7 Done New Done false
20163 6820 Acked 7819 Write 8 Done New Done false
20164 6821 Acked 7820 Read 4 Done New Done false
20165 6822 Acked 7821 Write 3 Done New Done false
20166 6823 Acked 7822 Write 10 Done New Done false
20167 6824 Acked 7823 Write 9 Done New Done false
20168 6825 Acked 7824 Read 8 Done New Done false
20169 6826 Acked 7825 Write 2 Done New Done false
20170 6827 Acked 7826 Write 10 Done New Done false
20171 6828 Acked 7827 Write 2 Done New Done false
20172 6829 Acked 7828 Write 7 Done New Done false
20173 6830 Acked 7829 Read 9 Done New Done false
20174 6831 Acked 7830 Read 10 Done New Done false
20175 6832 Acked 7831 Read 8 Done New Done false
20176 6833 Acked 7832 Write 2 Done New Done false
20177 6834 Acked 7833 Read 6 Done New Done false
20178 6835 Acked 7834 Write 8 Done New Done false
20179 6836 Acked 7835 Flush 0 Done New Done false
20180 6837 Acked 7836 Read 5 Done New Done false
20181 6838 Acked 7837 Read 5 Done New Done false
20182 6839 Acked 7838 Read 2 Done New Done false
20183 6840 Acked 7839 Read 1 Done New Done false
20184 6841 Acked 7840 Write 5 Done New Done false
20185 6842 Acked 7841 Read 3 Done New Done false
20186 6843 Acked 7842 Read 9 Done New Done false
20187 6844 Acked 7843 Read 8 Done New Done false
20188 6845 Acked 7844 Flush 0 Done New Done false
20189 6846 Acked 7845 Write 6 Done New Done false
20190 6847 Acked 7846 Read 2 Done New Done false
20191 6848 Acked 7847 Read 5 Done New Done false
20192 6849 Acked 7848 Write 4 Done New Done false
20193 6850 Acked 7849 Write 4 Done New Done false
20194 6851 Acked 7850 Write 1 Done New Done false
20195 6852 Acked 7851 Read 2 Done New Done false
20196 6853 Acked 7852 Write 2 Done New Done false
20197 6854 Acked 7853 Read 9 Done New Done false
20198 6855 Acked 7854 Write 2 Done New Done false
20199 6856 Acked 7855 Read 10 Done New Done false
20200 6857 Acked 7856 Read 6 Done New Done false
20201 6858 Acked 7857 Read 10 Done New Done false
20202 6859 Acked 7858 Write 10 Done New Done false
20203 6860 Acked 7859 Read 4 Done New Done false
20204 6861 Acked 7860 Write 2 Done New Done false
20205 6862 Acked 7861 Read 6 Done New Done false
20206 6863 Acked 7862 Write 5 Done New Done false
20207 6864 Acked 7863 Read 8 Done New Done false
20208 6865 Acked 7864 Write 5 Done New Done false
20209 6866 Acked 7865 Read 5 Done New Done false
20210 6867 Acked 7866 Read 5 Done New Done false
20211 6868 Acked 7867 Read 4 Done New Done false
20212 6869 Acked 7868 Write 8 Done New Done false
20213 6870 Acked 7869 Read 4 Done New Done false
20214 6871 Acked 7870 Read 6 Done New Done false
20215 6872 Acked 7871 Read 6 Done New Done false
20216 6873 Acked 7872 Flush 0 Done New Done false
20217 6874 Acked 7873 Write 10 Done New Done false
20218 6875 Acked 7874 Read 10 Done New Done false
20219 6876 Acked 7875 Read 5 Done New Done false
20220 6877 Acked 7876 Flush 0 Done New Done false
20221 6878 Acked 7877 Read 10 Done New Done false
20222 6879 Acked 7878 Write 10 Done New Done false
20223 6880 Acked 7879 Read 8 Done New Done false
20224 6881 Acked 7880 Read 7 Done New Done false
20225 6882 Acked 7881 Write 2 Done New Done false
20226 6883 Acked 7882 Flush 0 Done New Done false
20227 6884 Acked 7883 Write 6 Done New Done false
20228 6885 Acked 7884 Write 8 Done New Done false
20229 6886 Acked 7885 Write 1 Done New Done false
20230 6887 Acked 7886 Write 6 Done New Done false
20231 6888 Acked 7887 Flush 0 Done New Done false
20232 6889 Acked 7888 Write 2 Done New Done false
20233 6890 Acked 7889 Write 8 Done New Done false
20234 6891 Acked 7890 Write 8 Done New Done false
20235 6892 Acked 7891 Read 2 Done New Done false
20236 6893 Acked 7892 Write 5 Done New Done false
20237 6894 Acked 7893 Read 3 Done New Done false
20238 6895 Acked 7894 Write 1 Done New Done false
20239 6896 Acked 7895 Read 1 Done New Done false
20240 6897 Acked 7896 Write 3 Done New Done false
20241 6898 Acked 7897 Read 5 Done New Done false
20242 6899 Acked 7898 Write 9 Done New Done false
20243 6900 Acked 7899 Write 8 Done New Done false
20244 6901 Acked 7900 Read 5 Done New Done false
20245 6902 Acked 7901 Write 2 Done New Done false
20246 6903 Acked 7902 Write 10 Done New Done false
20247 6904 Acked 7903 Read 8 Done New Done false
20248 6905 Acked 7904 Read 2 Done New Done false
20249 6906 Acked 7905 Read 7 Done New Done false
20250 6907 Acked 7906 Write 10 Done New Done false
20251 6908 Acked 7907 Write 3 Done New Done false
20252 6909 Acked 7908 Read 3 Done New Done false
20253 6910 Acked 7909 Read 9 Done New Done false
20254 6911 Acked 7910 Read 6 Done New Done false
20255 6912 Acked 7911 Flush 0 Done New Done false
20256 6913 Acked 7912 Flush 0 Done New Done false
20257 6914 Acked 7913 Write 5 Done New Done false
20258 6915 Acked 7914 Read 1 Done New Done false
20259 6916 Acked 7915 Read 2 Done New Done false
20260 6917 Acked 7916 Write 6 Done New Done false
20261 6918 Acked 7917 Read 7 Done New Done false
20262 6919 Acked 7918 Flush 0 Done New Done false
20263 6920 Acked 7919 Read 9 Done New Done false
20264 6921 Acked 7920 Flush 0 Done New Done false
20265 6922 Acked 7921 Write 7 Done New Done false
20266 6923 Acked 7922 Flush 0 Done New Done false
20267 6924 Acked 7923 Write 7 Done New Done false
20268 6925 Acked 7924 Write 6 Done New Done false
20269 6926 Acked 7925 Read 10 Done New Done false
20270 6927 Acked 7926 Write 1 Done New Done false
20271 6928 Acked 7927 Read 2 Done New Done false
20272 6929 Acked 7928 Write 8 Done New Done false
20273 6930 Acked 7929 Read 10 Done New Done false
20274 6931 Acked 7930 Write 10 Sent New Done false
20275 STATES DS:0 DS:1 DS:2 TOTAL
20276 New 0 300 0 300
20277 Sent 1 0 0 1
20278 Done 299 0 300 599
20279 Skipped 0 0 0 0
20280 Error 0 0 0 0
20281 Last Flush: 7922 7630 7922
20282 Downstairs last five completed: 7630 7629 7628 7627 7626
20283 Upstairs last five completed: 6931 6930 6929 6928 6927
20284 CLIENT: Up:0 ds:300 act:2
202852024-02-27T21:48:38.760ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
202862024-02-27T21:48:38.760ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
202872024-02-27T21:48:38.760ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
202882024-02-27T21:48:38.760ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
202892024-02-27T21:48:38.760ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
202902024-02-27T21:48:38.760ZINFOcrucible: [1] client re-new 301 jobs since flush 7630 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
202912024-02-27T21:48:38.761ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
202922024-02-27T21:48:39.761ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
202932024-02-27T21:48:39.762ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
202942024-02-27T21:48:39.762ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
202952024-02-27T21:48:39.762ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
202962024-02-27T21:48:39.762ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
202972024-02-27T21:48:39.762ZINFOcrucible: [1] client re-new 301 jobs since flush 7630 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
20298 [8820] got start action
20299 [8820] I am going to restart
20300 Starting downstairs at port 8820
203012024-02-27T21:48:39.762ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
20302 Make output file at "/tmp/dsc/downstairs-8820.txt"
20303 [8820][1] reports Starting
20304 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1190)
20305 [8820][1] reports Running
203062024-02-27T21:48:40.763ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203072024-02-27T21:48:40.763ZINFOcrucible: ds_connection connected = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203082024-02-27T21:48:40.765ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203092024-02-27T21:48:40.765ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203102024-02-27T21:48:40.765ZINFOcrucible: send last flush ID to this DS: 7630 = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203112024-02-27T21:48:40.765ZINFOcrucible: Replied this last flush ID: 7630 = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203122024-02-27T21:48:40.765ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203132024-02-27T21:48:40.765ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203142024-02-27T21:48:40.765ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203152024-02-27T21:48:40.765ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
20316 ----------------------------------------------------------------
20317 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
20318 Downstairs last five completed: 7931 7930 7929 7928 7927
20319 Upstairs last five completed: 6932 6931 6930 6929 6928
20320 CLIENT: Up:0 ds:0 act:3
20321 Replay: All jobs finished, all DS active.
20322 Main task has work to do, go find it
20323 got dsc Stop(1)
20324 stop 1
20325 [8820] Got stop action so:false kr:false
20326 Feb 27 21:48:42.098 INFO request completed, latency_us: 351, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 9011b810-3c85-4e2c-bfa5-7ec05cd5281d, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
20327 Feb 27 21:48:42.099 INFO request completed, latency_us: 123, response_code: 200, uri: /state/cid/1, method: GET, req_id: 3db0486a-32c3-4216-85d5-44df04bcd01c, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
203282024-02-27T21:48:42.099ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203292024-02-27T21:48:42.099ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203302024-02-27T21:48:42.099ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = e6a5a974-925a-494f-8845-facc441c2690
203312024-02-27T21:48:42.099ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203322024-02-27T21:48:42.099ZINFOcrucible: [1] client re-new 0 jobs since flush 7931 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
203332024-02-27T21:48:42.099ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
20334 [8820] Exited with: ExitStatus(unix_wait_status(9))
20335 [8820][1] reports Exit
203362024-02-27T21:48:43.100ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203372024-02-27T21:48:43.100ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203382024-02-27T21:48:43.100ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203392024-02-27T21:48:43.100ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
203402024-02-27T21:48:43.100ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203412024-02-27T21:48:43.100ZINFOcrucible: [1] client re-new 0 jobs since flush 7931 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
203422024-02-27T21:48:43.100ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203432024-02-27T21:48:44.101ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203442024-02-27T21:48:44.101ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203452024-02-27T21:48:44.101ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203462024-02-27T21:48:44.101ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
203472024-02-27T21:48:44.101ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203482024-02-27T21:48:44.101ZINFOcrucible: [1] client re-new 0 jobs since flush 7931 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
203492024-02-27T21:48:44.101ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203502024-02-27T21:48:45.102ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203512024-02-27T21:48:45.102ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203522024-02-27T21:48:45.102ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203532024-02-27T21:48:45.102ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
203542024-02-27T21:48:45.102ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203552024-02-27T21:48:45.102ZINFOcrucible: [1] client re-new 0 jobs since flush 7931 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
203562024-02-27T21:48:45.102ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
20357 Feb 27 21:48:46.101 INFO request completed, latency_us: 210, response_code: 200, uri: /state/cid/1, method: GET, req_id: b407205f-48e0-4d67-9cfa-47433f230c4d, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
20358 001/300 Flush
20359 002/300 Read block 1832 len 20480
203602024-02-27T21:48:46.103ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203612024-02-27T21:48:46.103ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203622024-02-27T21:48:46.103ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203632024-02-27T21:48:46.103ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
203642024-02-27T21:48:46.103ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
203652024-02-27T21:48:46.103ZINFOcrucible: [1] client re-new 2 jobs since flush 7931 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
203662024-02-27T21:48:46.103ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
20367 003/300 Read block 4521 len 16384
20368 004/300 Write block 3139 len 32768 data: 5 4 4 4 4 4 6 5
20369 005/300 Flush
20370 006/300 Flush
20371 007/300 Write block 3522 len 16384 data: 7 8 7 7
20372 008/300 Read block 2439 len 12288
20373 009/300 Write block 1496 len 4096 data: 3
20374 010/300 Write block 3613 len 32768 data: 6 6 6 6 5 4 3 5
20375 011/300 Write block 2240 len 36864 data: 6 6 6 6 4 4 5 3 3
20376 012/300 Write block 2116 len 28672 data: 3 3 3 4 4 5 6
20377 013/300 Read block 1768 len 28672
20378 014/300 Write block 4659 len 12288 data: 4 5 5
20379 015/300 Read block 1723 len 16384
20380 016/300 Write block 1273 len 28672 data: 3 3 3 3 3 3 3
20381 017/300 Read block 1006 len 8192
20382 018/300 Read block 3757 len 28672
20383 019/300 Read block 4377 len 28672
20384 020/300 Write block 3408 len 24576 data: 5 7 9 9 8 7
20385 021/300 Write block 3190 len 16384 data: 3 4 4 5
20386 022/300 Read block 1948 len 4096
20387 023/300 Write block 3845 len 28672 data: 6 5 6 5 7 9 9
20388 024/300 Flush
20389 025/300 Read block 4560 len 24576
20390 026/300 Read block 3233 len 36864
20391 027/300 Write block 3798 len 4096 data: 9
20392 028/300 Read block 2662 len 16384
20393 029/300 Write block 4667 len 40960 data: 7 8 6 7 7 7 8 7 5 5
20394 030/300 Flush
20395 031/300 Write block 2776 len 20480 data: 2 2 4 6 8
20396 032/300 Read block 4781 len 4096
20397 033/300 Write block 3349 len 32768 data: 6 5 3 2 3 2 4 5
20398 034/300 Read block 387 len 12288
20399 035/300 Write block 2271 len 8192 data: 3 2
20400 036/300 Write block 3855 len 28672 data: 8 6 4 3 4 5 5
20401 037/300 Read block 1908 len 4096
20402 038/300 Read block 2858 len 32768
20403 039/300 Read block 3024 len 16384
20404 040/300 Write block 3202 len 16384 data: 4 7 8 6
20405 041/300 Write block 4219 len 16384 data: 7 6 7 6
20406 042/300 Write block 4351 len 28672 data: 6 7 6 4 2 4 4
20407 043/300 Read block 3486 len 40960
20408 044/300 Write block 3083 len 32768 data: 4 4 4 4 4 4 4 4
20409 045/300 Read block 154 len 12288
20410 046/300 Read block 552 len 12288
20411 047/300 Read block 3644 len 32768
20412 048/300 Read block 3614 len 28672
20413 049/300 Read block 3550 len 24576
20414 050/300 Read block 4187 len 40960
20415 051/300 Read block 3683 len 12288
20416 052/300 Write block 4202 len 20480 data: 3 3 3 2 2
20417 053/300 Write block 2347 len 20480 data: 5 6 8 7 6
20418 054/300 Read block 2425 len 8192
20419 055/300 Read block 3410 len 20480
20420 056/300 Read block 1484 len 4096
20421 057/300 Read block 1242 len 8192
20422 058/300 Write block 967 len 28672 data: 3 3 5 5 5 5 6
20423 059/300 Read block 1999 len 28672
20424 060/300 Read block 4102 len 24576
20425 061/300 Read block 1917 len 40960
20426 062/300 Read block 4127 len 28672
20427 063/300 Read block 4750 len 40960
20428 064/300 Read block 3131 len 12288
20429 065/300 Read block 3436 len 36864
20430 066/300 Write block 3593 len 12288 data: 7 7 9
20431 067/300 Write block 2498 len 36864 data: 3 4 4 4 6 9 10 10 10
20432 068/300 Write block 1252 len 8192 data: 2 2
20433 069/300 Flush
20434 070/300 Read block 1753 len 36864
20435 071/300 Write block 4617 len 24576 data: 6 4 6 6 5 6
20436 072/300 Read block 3635 len 8192
20437 073/300 Write block 3628 len 4096 data: 3
20438 074/300 Read block 1801 len 12288
20439 075/300 Write block 3399 len 28672 data: 6 7 6 6 6 4 5
20440 076/300 Write block 3775 len 20480 data: 6 6 7 6 6
20441 077/300 Write block 4140 len 32768 data: 4 6 6 6 6 6 5 5
20442 078/300 Flush
20443 079/300 Write block 3211 len 4096 data: 3
20444 080/300 Write block 1003 len 12288 data: 2 3 4
20445 081/300 Write block 2458 len 40960 data: 4 4 6 7 7 7 6 3 5 4
20446 082/300 Read block 4144 len 8192
20447 083/300 Write block 1194 len 16384 data: 8 7 9 7
20448 084/300 Read block 4142 len 36864
20449 085/300 Write block 4526 len 32768 data: 4 6 6 5 5 4 3 4
20450 086/300 Write block 1145 len 28672 data: 7 7 6 5 4 4 3
20451 087/300 Write block 1076 len 40960 data: 3 5 5 4 5 6 7 8 10 8
20452 088/300 Read block 1041 len 28672
20453 089/300 Read block 2201 len 40960
20454 090/300 Read block 2400 len 40960
20455 091/300 Write block 4871 len 16384 data: 6 7 6 6
20456 092/300 Flush
20457 093/300 Read block 2354 len 28672
20458 094/300 Read block 1862 len 20480
20459 095/300 Write block 2416 len 36864 data: 3 4 3 3 3 4 4 5 6
20460 096/300 Read block 1607 len 20480
20461 097/300 Write block 2601 len 12288 data: 6 6 6
20462 098/300 Write block 3064 len 16384 data: 6 5 5 5
20463 099/300 Write block 4613 len 4096 data: 5
20464 100/300 Read block 4143 len 28672
20465 101/300 Read block 322 len 40960
20466 102/300 Read block 288 len 12288
20467 103/300 Write block 568 len 24576 data: 5 3 2 4 4 4
20468 104/300 Write block 3316 len 12288 data: 7 9 10
20469 105/300 Write block 620 len 8192 data: 5 5
20470 106/300 Write block 4709 len 8192 data: 5 4
20471 107/300 Read block 2929 len 4096
20472 108/300 Read block 1732 len 36864
20473 109/300 Read block 2943 len 36864
20474 110/300 Write block 2065 len 36864 data: 3 3 4 3 3 3 3 4 4
20475 111/300 Read block 1308 len 16384
20476 112/300 Read block 3324 len 32768
20477 113/300 Write block 2770 len 24576 data: 4 4 5 4 4 4
20478 114/300 Write block 624 len 36864 data: 7 6 5 5 3 4 5 5 4
20479 115/300 Read block 4591 len 28672
20480 116/300 Write block 4701 len 36864 data: 4 5 6 6 6 6 5 5 6
20481 117/300 Write block 1365 len 36864 data: 7 9 9 8 4 4 4 4 4
20482 118/300 Read block 2122 len 36864
20483 119/300 Flush
20484 120/300 Write block 2220 len 20480 data: 5 6 5 6 5
20485 121/300 Write block 1937 len 24576 data: 5 5 4 5 4 2
20486 122/300 Write block 2660 len 36864 data: 6 7 8 9 8 9 8 10 12
20487 123/300 Write block 3938 len 24576 data: 7 6 5 5 5 6
20488 124/300 Read block 3406 len 20480
20489 125/300 Read block 2579 len 40960
20490 126/300 Read block 1051 len 4096
20491 127/300 Write block 3083 len 8192 data: 5 5
20492 128/300 Flush
20493 129/300 Read block 2123 len 20480
20494 130/300 Flush
20495 131/300 Write block 1487 len 8192 data: 6 6
20496 132/300 Read block 3545 len 4096
20497 133/300 Write block 3859 len 24576 data: 5 6 6 7 7 6
20498 134/300 Read block 1857 len 4096
20499 135/300 Read block 3971 len 36864
20500 136/300 Read block 1079 len 8192
20501 137/300 Write block 954 len 24576 data: 4 5 6 7 8 8
20502 138/300 Read block 3972 len 28672
20503 139/300 Write block 2788 len 36864 data: 3 4 5 5 6 7 6 6 6
20504 140/300 Write block 1887 len 24576 data: 5 5 5 6 5 3
20505 141/300 Read block 2823 len 16384
20506 142/300 Write block 2995 len 32768 data: 5 5 5 7 7 8 9 9
20507 143/300 Read block 4261 len 40960
20508 144/300 Read block 578 len 4096
20509 145/300 Write block 4550 len 8192 data: 9 8
20510 146/300 Read block 4531 len 24576
20511 147/300 Flush
20512 148/300 Flush
20513 149/300 Write block 2082 len 36864 data: 4 4 5 4 4 3 3 4 3
20514 150/300 Read block 13 len 16384
20515 151/300 Write block 4796 len 32768 data: 6 5 4 4 4 4 5 5
20516 152/300 Write block 2403 len 28672 data: 4 4 5 5 4 3 3
20517 153/300 Write block 3738 len 24576 data: 5 6 7 6 4 4
20518 154/300 Read block 2537 len 24576
20519 155/300 Read block 1142 len 40960
20520 156/300 Read block 2052 len 16384
20521 157/300 Read block 249 len 32768
20522 158/300 Read block 4285 len 16384
20523 159/300 Read block 4476 len 4096
20524 160/300 Read block 2839 len 20480
20525 161/300 Write block 3314 len 16384 data: 5 5 8 10
20526 162/300 Read block 571 len 32768
20527 163/300 Read block 3030 len 40960
20528 164/300 Read block 3949 len 36864
20529 165/300 Write block 1511 len 24576 data: 8 8 5 2 3 3
20530 166/300 Read block 3123 len 24576
20531 167/300 Read block 2098 len 20480
20532 168/300 Flush
20533 169/300 Read block 3572 len 28672
20534 170/300 Read block 104 len 28672
20535 171/300 Write block 605 len 32768 data: 3 4 7 8 7 6 6 6
20536 172/300 Write block 1860 len 8192 data: 3 3
20537 173/300 Read block 3663 len 24576
20538 174/300 Write block 2482 len 8192 data: 5 5
20539 175/300 Read block 1702 len 32768
20540 176/300 Write block 951 len 32768 data: 6 5 4 5 6 7 8 9
20541 177/300 Write block 586 len 8192 data: 8 7
20542 178/300 Read block 3567 len 32768
20543 179/300 Read block 1468 len 16384
20544 180/300 Write block 1385 len 8192 data: 6 6
20545 181/300 Flush
20546 182/300 Flush
20547 183/300 Flush
20548 184/300 Read block 3356 len 16384
20549 185/300 Write block 4020 len 8192 data: 8 8
20550 186/300 Read block 254 len 28672
20551 187/300 Read block 2441 len 40960
20552 188/300 Read block 906 len 16384
20553 189/300 Read block 3070 len 24576
20554 190/300 Write block 343 len 8192 data: 2 3
20555 191/300 Write block 1885 len 8192 data: 4 5
20556 192/300 Write block 4445 len 24576 data: 7 4 4 3 5 5
20557 193/300 Write block 948 len 24576 data: 6 5 6 7 6 5
20558 194/300 Read block 2293 len 12288
20559 195/300 Write block 4942 len 20480 data: 6 6 6 7 6
20560 196/300 Write block 4363 len 36864 data: 7 7 7 6 7 7 6 6 5
20561 197/300 Write block 4068 len 32768 data: 4 6 6 6 7 7 7 9
20562 198/300 Flush
20563 199/300 Flush
20564 200/300 Write block 3166 len 20480 data: 6 6 7 7 7
20565 201/300 Read block 4514 len 28672
20566 202/300 Flush
20567 203/300 Read block 4222 len 16384
20568 204/300 Write block 2147 len 20480 data: 2 3 3 3 3
20569 205/300 Read block 637 len 28672
20570 206/300 Read block 1773 len 8192
20571 207/300 Read block 1354 len 28672
20572 208/300 Read block 650 len 28672
20573 209/300 Write block 2828 len 8192 data: 5 5
20574 210/300 Read block 4324 len 4096
20575 211/300 Read block 682 len 32768
20576 212/300 Write block 1435 len 20480 data: 4 5 3 4 4
20577 213/300 Write block 3419 len 32768 data: 7 7 6 6 5 3 2 2
20578 214/300 Write block 1100 len 36864 data: 5 5 4 4 4 3 3 3 3
20579 215/300 Write block 3437 len 4096 data: 4
20580 216/300 Write block 4916 len 4096 data: 4
20581 217/300 Read block 1880 len 28672
20582 218/300 Flush
20583 219/300 Write block 1312 len 8192 data: 6 7
20584 220/300 Flush
20585 221/300 Write block 705 len 16384 data: 4 5 5 4
20586 222/300 Flush
20587 223/300 Flush
20588 224/300 Read block 1926 len 20480
20589 225/300 Read block 805 len 40960
20590 226/300 Write block 2502 len 28672 data: 7 10 11 11 11 11 11
20591 227/300 Write block 1643 len 12288 data: 7 6 7
20592 228/300 Read block 905 len 4096
20593 229/300 Read block 1209 len 32768
20594 230/300 Write block 2658 len 4096 data: 8
20595 231/300 Flush
20596 232/300 Write block 355 len 36864 data: 4 4 6 7 7 6 6 7 4
20597 233/300 Write block 2836 len 24576 data: 5 4 3 3 4 4
20598 234/300 Read block 4470 len 8192
20599 235/300 Read block 2624 len 40960
20600 236/300 Write block 2183 len 4096 data: 5
20601 237/300 Flush
20602 238/300 Write block 1843 len 8192 data: 4 3
20603 239/300 Write block 589 len 8192 data: 7 6
20604 240/300 Flush
20605 241/300 Read block 4495 len 16384
20606 242/300 Read block 2645 len 12288
20607 243/300 Write block 1409 len 36864 data: 5 5 4 4 3 3 3 3 5
20608 244/300 Write block 998 len 12288 data: 4 4 4
20609 245/300 Flush
20610 246/300 Write block 655 len 4096 data: 2
20611 247/300 Read block 2839 len 32768
20612 248/300 Read block 527 len 36864
20613 249/300 Read block 2726 len 8192
20614 250/300 Read block 3043 len 40960
20615 251/300 Write block 2146 len 32768 data: 2 3 4 4 4 4 3 4
20616 252/300 Read block 770 len 4096
20617 253/300 Flush
20618 254/300 Read block 4849 len 32768
20619 255/300 Flush
20620 256/300 Flush
20621 257/300 Read block 2242 len 40960
20622 258/300 Write block 3595 len 16384 data: 10 8 7 6
20623 259/300 Write block 4512 len 4096 data: 9
20624 260/300 Read block 2603 len 36864
20625 261/300 Write block 3177 len 8192 data: 7 6
20626 262/300 Read block 4526 len 36864
20627 263/300 Write block 1500 len 8192 data: 6 7
20628 264/300 Write block 1043 len 16384 data: 4 4 5 3
20629 265/300 Flush
20630 266/300 Write block 899 len 16384 data: 5 4 4 5
20631 267/300 Read block 3774 len 28672
20632 268/300 Read block 4702 len 8192
20633 269/300 Write block 1325 len 16384 data: 8 8 6 5
20634 270/300 Write block 3633 len 4096 data: 5
20635 271/300 Read block 816 len 20480
20636 272/300 Read block 4585 len 8192
20637 273/300 Read block 4243 len 24576
20638 274/300 Read block 2531 len 28672
20639 275/300 Flush
20640 276/300 Write block 3979 len 16384 data: 6 5 5 6
20641 277/300 Write block 399 len 24576 data: 7 8 7 6 6 6
20642 278/300 Read block 1237 len 16384
20643 279/300 Read block 549 len 20480
20644 280/300 Flush
20645 281/300 Write block 3260 len 4096 data: 4
20646 282/300 Flush
20647 283/300 Write block 137 len 8192 data: 3 4
20648 284/300 Read block 3972 len 32768
20649 285/300 Read block 3354 len 8192
20650 286/300 Write block 1067 len 8192 data: 5 4
20651 287/300 Write block 1607 len 20480 data: 3 3 3 3 5
20652 288/300 Read block 4865 len 16384
20653 289/300 Write block 3107 len 20480 data: 7 6 8 10 11
20654 290/300 Flush
20655 291/300 Write block 741 len 24576 data: 7 7 6 5 6 3
20656 292/300 Write block 4935 len 40960 data: 4 3 2 2 4 4 6 7 7 7
20657 293/300 Write block 4383 len 4096 data: 5
20658 294/300 Flush
20659 295/300 Read block 2406 len 12288
20660 296/300 Read block 2554 len 16384
20661 297/300 Write block 1059 len 36864 data: 5 5 5 6 7 7 6 6 6
20662 298/300 Read block 502 len 12288
20663 299/300 Read block 3884 len 40960
20664 300/300 Read block 400 len 28672
20665 Main task has work to do, go find it
20666 got dsc Start(1)
20667 start 1
20668 Feb 27 21:48:46.446 INFO request completed, latency_us: 212, response_code: 204, uri: /start/cid/1, method: POST, req_id: caed1106-f6d3-43a5-b2a9-7a712319262f, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
20669 Replay: started 1, returned:Ok(())
20670 ----------------------------------------------------------------
20671 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20672 States: Active Offline Active
20673 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
20674 6933 Acked 7932 Flush 0 Done New Done true
20675 6934 Acked 7933 Read 5 Done New Done true
20676 6935 Acked 7934 Read 4 Done New Done false
20677 6936 Acked 7935 Write 8 Done New Done false
20678 6937 Acked 7936 Flush 0 Done New Done false
20679 6938 Acked 7937 Flush 0 Done New Done false
20680 6939 Acked 7938 Write 4 Done New Done false
20681 6940 Acked 7939 Read 3 Done New Done false
20682 6941 Acked 7940 Write 1 Done New Done false
20683 6942 Acked 7941 Write 8 Done New Done false
20684 6943 Acked 7942 Write 9 Done New Done false
20685 6944 Acked 7943 Write 7 Done New Done false
20686 6945 Acked 7944 Read 7 Done New Done false
20687 6946 Acked 7945 Write 3 Done New Done false
20688 6947 Acked 7946 Read 4 Done New Done false
20689 6948 Acked 7947 Write 7 Done New Done false
20690 6949 Acked 7948 Read 2 Done New Done false
20691 6950 Acked 7949 Read 7 Done New Done false
20692 6951 Acked 7950 Read 7 Done New Done false
20693 6952 Acked 7951 Write 6 Done New Done false
20694 6953 Acked 7952 Write 4 Done New Done false
20695 6954 Acked 7953 Read 1 Done New Done false
20696 6955 Acked 7954 Write 7 Done New Done false
20697 6956 Acked 7955 Flush 0 Done New Done false
20698 6957 Acked 7956 Read 6 Done New Done false
20699 6958 Acked 7957 Read 9 Done New Done false
20700 6959 Acked 7958 Write 1 Done New Done false
20701 6960 Acked 7959 Read 4 Done New Done false
20702 6961 Acked 7960 Write 10 Done New Done false
20703 6962 Acked 7961 Flush 0 Done New Done false
20704 6963 Acked 7962 Write 5 Done New Done false
20705 6964 Acked 7963 Read 1 Done New Done false
20706 6965 Acked 7964 Write 8 Done New Done false
20707 6966 Acked 7965 Read 3 Done New Done false
20708 6967 Acked 7966 Write 2 Done New Done false
20709 6968 Acked 7967 Write 7 Done New Done false
20710 6969 Acked 7968 Read 1 Done New Done false
20711 6970 Acked 7969 Read 8 Done New Done false
20712 6971 Acked 7970 Read 4 Done New Done false
20713 6972 Acked 7971 Write 4 Done New Done false
20714 6973 Acked 7972 Write 4 Done New Done false
20715 6974 Acked 7973 Write 7 Done New Done false
20716 6975 Acked 7974 Read 10 Done New Done false
20717 6976 Acked 7975 Write 8 Done New Done false
20718 6977 Acked 7976 Read 3 Done New Done false
20719 6978 Acked 7977 Read 3 Done New Done false
20720 6979 Acked 7978 Read 8 Done New Done false
20721 6980 Acked 7979 Read 7 Done New Done false
20722 6981 Acked 7980 Read 6 Done New Done false
20723 6982 Acked 7981 Read 10 Done New Done false
20724 6983 Acked 7982 Read 3 Done New Done false
20725 6984 Acked 7983 Write 5 Done New Done false
20726 6985 Acked 7984 Write 5 Done New Done false
20727 6986 Acked 7985 Read 2 Done New Done false
20728 6987 Acked 7986 Read 5 Done New Done false
20729 6988 Acked 7987 Read 1 Done New Done false
20730 6989 Acked 7988 Read 2 Done New Done false
20731 6990 Acked 7989 Write 7 Done New Done false
20732 6991 Acked 7990 Read 7 Done New Done false
20733 6992 Acked 7991 Read 6 Done New Done false
20734 6993 Acked 7992 Read 10 Done New Done false
20735 6994 Acked 7993 Read 7 Done New Done false
20736 6995 Acked 7994 Read 10 Done New Done false
20737 6996 Acked 7995 Read 3 Done New Done false
20738 6997 Acked 7996 Read 9 Done New Done false
20739 6998 Acked 7997 Write 3 Done New Done false
20740 6999 Acked 7998 Write 9 Done New Done false
20741 7000 Acked 7999 Write 2 Done New Done false
20742 7001 Acked 8000 Flush 0 Done New Done false
20743 7002 Acked 8001 Read 9 Done New Done false
20744 7003 Acked 8002 Write 6 Done New Done false
20745 7004 Acked 8003 Read 2 Done New Done false
20746 7005 Acked 8004 Write 1 Done New Done false
20747 7006 Acked 8005 Read 3 Done New Done false
20748 7007 Acked 8006 Write 7 Done New Done false
20749 7008 Acked 8007 Write 5 Done New Done false
20750 7009 Acked 8008 Write 8 Done New Done false
20751 7010 Acked 8009 Flush 0 Done New Done false
20752 7011 Acked 8010 Write 1 Done New Done false
20753 7012 Acked 8011 Write 3 Done New Done false
20754 7013 Acked 8012 Write 10 Done New Done false
20755 7014 Acked 8013 Read 2 Done New Done false
20756 7015 Acked 8014 Write 4 Done New Done false
20757 7016 Acked 8015 Read 9 Done New Done false
20758 7017 Acked 8016 Write 8 Done New Done false
20759 7018 Acked 8017 Write 7 Done New Done false
20760 7019 Acked 8018 Write 10 Done New Done false
20761 7020 Acked 8019 Read 7 Done New Done false
20762 7021 Acked 8020 Read 10 Done New Done false
20763 7022 Acked 8021 Read 10 Done New Done false
20764 7023 Acked 8022 Write 4 Done New Done false
20765 7024 Acked 8023 Flush 0 Done New Done false
20766 7025 Acked 8024 Read 7 Done New Done false
20767 7026 Acked 8025 Read 5 Done New Done false
20768 7027 Acked 8026 Write 9 Done New Done false
20769 7028 Acked 8027 Read 5 Done New Done false
20770 7029 Acked 8028 Write 3 Done New Done false
20771 7030 Acked 8029 Write 4 Done New Done false
20772 7031 Acked 8030 Write 1 Done New Done false
20773 7032 Acked 8031 Read 7 Done New Done false
20774 7033 Acked 8032 Read 10 Done New Done false
20775 7034 Acked 8033 Read 3 Done New Done false
20776 7035 Acked 8034 Write 6 Done New Done false
20777 7036 Acked 8035 Write 3 Done New Done false
20778 7037 Acked 8036 Write 2 Done New Done false
20779 7038 Acked 8037 Write 2 Done New Done false
20780 7039 Acked 8038 Read 1 Done New Done false
20781 7040 Acked 8039 Read 9 Done New Done false
20782 7041 Acked 8040 Read 9 Done New Done false
20783 7042 Acked 8041 Write 9 Done New Done false
20784 7043 Acked 8042 Read 4 Done New Done false
20785 7044 Acked 8043 Read 8 Done New Done false
20786 7045 Acked 8044 Write 6 Done New Done false
20787 7046 Acked 8045 Write 9 Done New Done false
20788 7047 Acked 8046 Read 7 Done New Done false
20789 7048 Acked 8047 Write 9 Done New Done false
20790 7049 Acked 8048 Write 9 Done New Done false
20791 7050 Acked 8049 Read 9 Done New Done false
20792 7051 Acked 8050 Flush 0 Done New Done false
20793 7052 Acked 8051 Write 5 Done New Done false
20794 7053 Acked 8052 Write 6 Done New Done false
20795 7054 Acked 8053 Write 9 Done New Done false
20796 7055 Acked 8054 Write 6 Done New Done false
20797 7056 Acked 8055 Read 5 Done New Done false
20798 7057 Acked 8056 Read 10 Done New Done false
20799 7058 Acked 8057 Read 1 Done New Done false
20800 7059 Acked 8058 Write 2 Done New Done false
20801 7060 Acked 8059 Flush 0 Done New Done false
20802 7061 Acked 8060 Read 5 Done New Done false
20803 7062 Acked 8061 Flush 0 Done New Done false
20804 7063 Acked 8062 Write 2 Done New Done false
20805 7064 Acked 8063 Read 1 Done New Done false
20806 7065 Acked 8064 Write 6 Done New Done false
20807 7066 Acked 8065 Read 1 Done New Done false
20808 7067 Acked 8066 Read 9 Done New Done false
20809 7068 Acked 8067 Read 2 Done New Done false
20810 7069 Acked 8068 Write 6 Done New Done false
20811 7070 Acked 8069 Read 7 Done New Done false
20812 7071 Acked 8070 Write 9 Done New Done false
20813 7072 Acked 8071 Write 6 Done New Done false
20814 7073 Acked 8072 Read 4 Done New Done false
20815 7074 Acked 8073 Write 8 Done New Done false
20816 7075 Acked 8074 Read 10 Done New Done false
20817 7076 Acked 8075 Read 1 Done New Done false
20818 7077 Acked 8076 Write 2 Done New Done false
20819 7078 Acked 8077 Read 6 Done New Done false
20820 7079 Acked 8078 Flush 0 Done New Done false
20821 7080 Acked 8079 Flush 0 Done New Done false
20822 7081 Acked 8080 Write 9 Done New Done false
20823 7082 Acked 8081 Read 4 Done New Done false
20824 7083 Acked 8082 Write 8 Done New Done false
20825 7084 Acked 8083 Write 7 Done New Done false
20826 7085 Acked 8084 Write 6 Done New Done false
20827 7086 Acked 8085 Read 6 Done New Done false
20828 7087 Acked 8086 Read 10 Done New Done false
20829 7088 Acked 8087 Read 4 Done New Done false
20830 7089 Acked 8088 Read 8 Done New Done false
20831 7090 Acked 8089 Read 4 Done New Done false
20832 7091 Acked 8090 Read 1 Done New Done false
20833 7092 Acked 8091 Read 5 Done New Done false
20834 7093 Acked 8092 Write 4 Done New Done false
20835 7094 Acked 8093 Read 8 Done New Done false
20836 7095 Acked 8094 Read 10 Done New Done false
20837 7096 Acked 8095 Read 9 Done New Done false
20838 7097 Acked 8096 Write 6 Done New Done false
20839 7098 Acked 8097 Read 6 Done New Done false
20840 7099 Acked 8098 Read 5 Done New Done false
20841 7100 Acked 8099 Flush 0 Done New Done false
20842 7101 Acked 8100 Read 7 Done New Done false
20843 7102 Acked 8101 Read 7 Done New Done false
20844 7103 Acked 8102 Write 8 Done New Done false
20845 7104 Acked 8103 Write 2 Done New Done false
20846 7105 Acked 8104 Read 6 Done New Done false
20847 7106 Acked 8105 Write 2 Done New Done false
20848 7107 Acked 8106 Read 8 Done New Done false
20849 7108 Acked 8107 Write 8 Done New Done false
20850 7109 Acked 8108 Write 2 Done New Done false
20851 7110 Acked 8109 Read 8 Done New Done false
20852 7111 Acked 8110 Read 4 Done New Done false
20853 7112 Acked 8111 Write 2 Done New Done false
20854 7113 Acked 8112 Flush 0 Done New Done false
20855 7114 Acked 8113 Flush 0 Done New Done false
20856 7115 Acked 8114 Flush 0 Done New Done false
20857 7116 Acked 8115 Read 4 Done New Done false
20858 7117 Acked 8116 Write 2 Done New Done false
20859 7118 Acked 8117 Read 7 Done New Done false
20860 7119 Acked 8118 Read 10 Done New Done false
20861 7120 Acked 8119 Read 4 Done New Done false
20862 7121 Acked 8120 Read 6 Done New Done false
20863 7122 Acked 8121 Write 2 Done New Done false
20864 7123 Acked 8122 Write 2 Done New Done false
20865 7124 Acked 8123 Write 6 Done New Done false
20866 7125 Acked 8124 Write 6 Done New Done false
20867 7126 Acked 8125 Read 3 Done New Done false
20868 7127 Acked 8126 Write 5 Done New Done false
20869 7128 Acked 8127 Write 9 Done New Done false
20870 7129 Acked 8128 Write 8 Done New Done false
20871 7130 Acked 8129 Flush 0 Done New Done false
20872 7131 Acked 8130 Flush 0 Done New Done false
20873 7132 Acked 8131 Write 5 Done New Done false
20874 7133 Acked 8132 Read 7 Done New Done false
20875 7134 Acked 8133 Flush 0 Done New Done false
20876 7135 Acked 8134 Read 4 Done New Done false
20877 7136 Acked 8135 Write 5 Done New Done false
20878 7137 Acked 8136 Read 7 Done New Done false
20879 7138 Acked 8137 Read 2 Done New Done false
20880 7139 Acked 8138 Read 7 Done New Done false
20881 7140 Acked 8139 Read 7 Done New Done false
20882 7141 Acked 8140 Write 2 Done New Done false
20883 7142 Acked 8141 Read 1 Done New Done false
20884 7143 Acked 8142 Read 8 Done New Done false
20885 7144 Acked 8143 Write 5 Done New Done false
20886 7145 Acked 8144 Write 8 Done New Done false
20887 7146 Acked 8145 Write 9 Done New Done false
20888 7147 Acked 8146 Write 1 Done New Done false
20889 7148 Acked 8147 Write 1 Done New Done false
20890 7149 Acked 8148 Read 7 Done New Done false
20891 7150 Acked 8149 Flush 0 Done New Done false
20892 7151 Acked 8150 Write 2 Done New Done false
20893 7152 Acked 8151 Flush 0 Done New Done false
20894 7153 Acked 8152 Write 4 Done New Done false
20895 7154 Acked 8153 Flush 0 Done New Done false
20896 7155 Acked 8154 Flush 0 Done New Done false
20897 7156 Acked 8155 Read 5 Done New Done false
20898 7157 Acked 8156 Read 10 Done New Done false
20899 7158 Acked 8157 Write 7 Done New Done false
20900 7159 Acked 8158 Write 3 Done New Done false
20901 7160 Acked 8159 Read 1 Done New Done false
20902 7161 Acked 8160 Read 8 Done New Done false
20903 7162 Acked 8161 Write 1 Done New Done false
20904 7163 Acked 8162 Flush 0 Done New Done false
20905 7164 Acked 8163 Write 9 Done New Done false
20906 7165 Acked 8164 Write 6 Done New Done false
20907 7166 Acked 8165 Read 2 Done New Done false
20908 7167 Acked 8166 Read 10 Done New Done false
20909 7168 Acked 8167 Write 1 Done New Done false
20910 7169 Acked 8168 Flush 0 Done New Done false
20911 7170 Acked 8169 Write 2 Done New Done false
20912 7171 Acked 8170 Write 2 Done New Done false
20913 7172 Acked 8171 Flush 0 Done New Done false
20914 7173 Acked 8172 Read 4 Done New Done false
20915 7174 Acked 8173 Read 3 Done New Done false
20916 7175 Acked 8174 Write 9 Done New Done false
20917 7176 Acked 8175 Write 3 Done New Done false
20918 7177 Acked 8176 Flush 0 Done New Done false
20919 7178 Acked 8177 Write 1 Done New Done false
20920 7179 Acked 8178 Read 8 Done New Done false
20921 7180 Acked 8179 Read 9 Done New Done false
20922 7181 Acked 8180 Read 2 Done New Done false
20923 7182 Acked 8181 Read 10 Done New Done false
20924 7183 Acked 8182 Write 8 Done New Done false
20925 7184 Acked 8183 Read 1 Done New Done false
20926 7185 Acked 8184 Flush 0 Done New Done false
20927 7186 Acked 8185 Read 8 Done New Done false
20928 7187 Acked 8186 Flush 0 Done New Done false
20929 7188 Acked 8187 Flush 0 Done New Done false
20930 7189 Acked 8188 Read 10 Done New Done false
20931 7190 Acked 8189 Write 4 Done New Done false
20932 7191 Acked 8190 Write 1 Done New Done false
20933 7192 Acked 8191 Read 9 Done New Done false
20934 7193 Acked 8192 Write 2 Done New Done false
20935 7194 Acked 8193 Read 9 Done New Done false
20936 7195 Acked 8194 Write 2 Done New Done false
20937 7196 Acked 8195 Write 4 Done New Done false
20938 7197 Acked 8196 Flush 0 Done New Done false
20939 7198 Acked 8197 Write 4 Done New Done false
20940 7199 Acked 8198 Read 7 Done New Done false
20941 7200 Acked 8199 Read 2 Done New Done false
20942 7201 Acked 8200 Write 4 Done New Done false
20943 7202 Acked 8201 Write 1 Done New Done false
20944 7203 Acked 8202 Read 5 Done New Done false
20945 7204 Acked 8203 Read 2 Done New Done false
20946 7205 Acked 8204 Read 6 Done New Done false
20947 7206 Acked 8205 Read 7 Done New Done false
20948 7207 Acked 8206 Flush 0 Done New Done false
20949 7208 Acked 8207 Write 4 Done New Done false
20950 7209 Acked 8208 Write 6 Done New Done false
20951 7210 Acked 8209 Read 4 Done New Done false
20952 7211 Acked 8210 Read 5 Done New Done false
20953 7212 Acked 8211 Flush 0 Done New Done false
20954 7213 Acked 8212 Write 1 Done New Done false
20955 7214 Acked 8213 Flush 0 Done New Done false
20956 7215 Acked 8214 Write 2 Done New Done false
20957 7216 Acked 8215 Read 8 Done New Done false
20958 7217 Acked 8216 Read 2 Done New Done false
20959 7218 Acked 8217 Write 2 Done New Done false
20960 7219 Acked 8218 Write 5 Done New Done false
20961 7220 Acked 8219 Read 4 Done New Done false
20962 7221 Acked 8220 Write 5 Done New Done false
20963 7222 Acked 8221 Flush 0 Done New Done false
20964 7223 Acked 8222 Write 6 Done New Done false
20965 7224 Acked 8223 Write 10 Done New Done false
20966 7225 Acked 8224 Write 1 Done New Done false
20967 7226 Acked 8225 Flush 0 Done New Done false
20968 7227 Acked 8226 Read 3 Done New Done false
20969 7228 Acked 8227 Read 4 Done New Done false
20970 7229 Acked 8228 Write 9 Done New Done false
20971 7230 Acked 8229 Read 3 Done New Done false
20972 7231 Acked 8230 Flush 0 Done New Done false
20973 7232 Acked 8231 Read 10 Done New Done false
20974 7233 Acked 8232 Read 7 Done New Done false
20975 STATES DS:0 DS:1 DS:2 TOTAL
20976 New 0 301 0 301
20977 Sent 0 0 0 0
20978 Done 301 0 301 602
20979 Skipped 0 0 0 0
20980 Error 0 0 0 0
20981 Last Flush: 8230 7931 8230
20982 Downstairs last five completed: 7931 7930 7929 7928 7927
20983 Upstairs last five completed: 7233 7232 7231 7230 7229
20984 CLIENT: Up:0 ds:301 act:2
209852024-02-27T21:48:47.104ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
209862024-02-27T21:48:47.104ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
209872024-02-27T21:48:47.104ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
209882024-02-27T21:48:47.104ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
209892024-02-27T21:48:47.104ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
209902024-02-27T21:48:47.104ZINFOcrucible: [1] client re-new 302 jobs since flush 7931 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
209912024-02-27T21:48:47.105ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
209922024-02-27T21:48:48.106ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
209932024-02-27T21:48:48.106ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
209942024-02-27T21:48:48.106ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
209952024-02-27T21:48:48.106ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
209962024-02-27T21:48:48.106ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
209972024-02-27T21:48:48.106ZINFOcrucible: [1] client re-new 302 jobs since flush 7931 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
209982024-02-27T21:48:48.106ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
209992024-02-27T21:48:49.108ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
210002024-02-27T21:48:49.108ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
210012024-02-27T21:48:49.108ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
210022024-02-27T21:48:49.108ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
210032024-02-27T21:48:49.108ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
210042024-02-27T21:48:49.108ZINFOcrucible: [1] client re-new 302 jobs since flush 7931 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
210052024-02-27T21:48:49.108ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
210062024-02-27T21:48:50.109ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
210072024-02-27T21:48:50.109ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
210082024-02-27T21:48:50.110ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
210092024-02-27T21:48:50.110ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
210102024-02-27T21:48:50.110ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
210112024-02-27T21:48:50.110ZINFOcrucible: [1] client re-new 302 jobs since flush 7931 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
210122024-02-27T21:48:50.110ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
21013 ----------------------------------------------------------------
21014 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
21015 States: Active Offline Active
21016 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
21017 6933 Acked 7932 Flush 0 Done New Done true
21018 6934 Acked 7933 Read 5 Done New Done true
21019 6935 Acked 7934 Read 4 Done New Done true
21020 6936 Acked 7935 Write 8 Done New Done true
21021 6937 Acked 7936 Flush 0 Done New Done true
21022 6938 Acked 7937 Flush 0 Done New Done true
21023 6939 Acked 7938 Write 4 Done New Done true
21024 6940 Acked 7939 Read 3 Done New Done true
21025 6941 Acked 7940 Write 1 Done New Done true
21026 6942 Acked 7941 Write 8 Done New Done true
21027 6943 Acked 7942 Write 9 Done New Done true
21028 6944 Acked 7943 Write 7 Done New Done true
21029 6945 Acked 7944 Read 7 Done New Done true
21030 6946 Acked 7945 Write 3 Done New Done true
21031 6947 Acked 7946 Read 4 Done New Done true
21032 6948 Acked 7947 Write 7 Done New Done true
21033 6949 Acked 7948 Read 2 Done New Done true
21034 6950 Acked 7949 Read 7 Done New Done true
21035 6951 Acked 7950 Read 7 Done New Done true
21036 6952 Acked 7951 Write 6 Done New Done true
21037 6953 Acked 7952 Write 4 Done New Done true
21038 6954 Acked 7953 Read 1 Done New Done true
21039 6955 Acked 7954 Write 7 Done New Done true
21040 6956 Acked 7955 Flush 0 Done New Done true
21041 6957 Acked 7956 Read 6 Done New Done true
21042 6958 Acked 7957 Read 9 Done New Done true
21043 6959 Acked 7958 Write 1 Done New Done true
21044 6960 Acked 7959 Read 4 Done New Done true
21045 6961 Acked 7960 Write 10 Done New Done true
21046 6962 Acked 7961 Flush 0 Done New Done true
21047 6963 Acked 7962 Write 5 Done New Done true
21048 6964 Acked 7963 Read 1 Done New Done true
21049 6965 Acked 7964 Write 8 Done New Done true
21050 6966 Acked 7965 Read 3 Done New Done true
21051 6967 Acked 7966 Write 2 Done New Done true
21052 6968 Acked 7967 Write 7 Done New Done true
21053 6969 Acked 7968 Read 1 Done New Done true
21054 6970 Acked 7969 Read 8 Done New Done true
21055 6971 Acked 7970 Read 4 Done New Done true
21056 6972 Acked 7971 Write 4 Done New Done true
21057 6973 Acked 7972 Write 4 Done New Done true
21058 6974 Acked 7973 Write 7 Done New Done true
21059 6975 Acked 7974 Read 10 Done New Done true
21060 6976 Acked 7975 Write 8 Done New Done true
21061 6977 Acked 7976 Read 3 Done New Done true
21062 6978 Acked 7977 Read 3 Done New Done true
21063 6979 Acked 7978 Read 8 Done New Done true
21064 6980 Acked 7979 Read 7 Done New Done true
21065 6981 Acked 7980 Read 6 Done New Done true
21066 6982 Acked 7981 Read 10 Done New Done true
21067 6983 Acked 7982 Read 3 Done New Done true
21068 6984 Acked 7983 Write 5 Done New Done true
21069 6985 Acked 7984 Write 5 Done New Done true
21070 6986 Acked 7985 Read 2 Done New Done true
21071 6987 Acked 7986 Read 5 Done New Done true
21072 6988 Acked 7987 Read 1 Done New Done true
21073 6989 Acked 7988 Read 2 Done New Done true
21074 6990 Acked 7989 Write 7 Done New Done true
21075 6991 Acked 7990 Read 7 Done New Done true
21076 6992 Acked 7991 Read 6 Done New Done true
21077 6993 Acked 7992 Read 10 Done New Done true
21078 6994 Acked 7993 Read 7 Done New Done true
21079 6995 Acked 7994 Read 10 Done New Done true
21080 6996 Acked 7995 Read 3 Done New Done true
21081 6997 Acked 7996 Read 9 Done New Done true
21082 6998 Acked 7997 Write 3 Done New Done true
21083 6999 Acked 7998 Write 9 Done New Done true
21084 7000 Acked 7999 Write 2 Done New Done true
21085 7001 Acked 8000 Flush 0 Done New Done true
21086 7002 Acked 8001 Read 9 Done New Done true
21087 7003 Acked 8002 Write 6 Done New Done true
21088 7004 Acked 8003 Read 2 Done New Done true
21089 7005 Acked 8004 Write 1 Done New Done true
21090 7006 Acked 8005 Read 3 Done New Done true
21091 7007 Acked 8006 Write 7 Done New Done true
21092 7008 Acked 8007 Write 5 Done New Done true
21093 7009 Acked 8008 Write 8 Done New Done true
21094 7010 Acked 8009 Flush 0 Done New Done true
21095 7011 Acked 8010 Write 1 Done New Done true
21096 7012 Acked 8011 Write 3 Done New Done true
21097 7013 Acked 8012 Write 10 Done New Done true
21098 7014 Acked 8013 Read 2 Done New Done true
21099 7015 Acked 8014 Write 4 Done New Done true
21100 7016 Acked 8015 Read 9 Done New Done true
21101 7017 Acked 8016 Write 8 Done New Done true
21102 7018 Acked 8017 Write 7 Done New Done true
21103 7019 Acked 8018 Write 10 Done New Done true
21104 7020 Acked 8019 Read 7 Done New Done true
21105 7021 Acked 8020 Read 10 Done New Done true
21106 7022 Acked 8021 Read 10 Done New Done true
21107 7023 Acked 8022 Write 4 Done New Done true
21108 7024 Acked 8023 Flush 0 Done New Done true
21109 7025 Acked 8024 Read 7 Done New Done true
21110 7026 Acked 8025 Read 5 Done New Done true
21111 7027 Acked 8026 Write 9 Done New Done true
21112 7028 Acked 8027 Read 5 Done New Done true
21113 7029 Acked 8028 Write 3 Done New Done true
21114 7030 Acked 8029 Write 4 Done New Done true
21115 7031 Acked 8030 Write 1 Done New Done true
21116 7032 Acked 8031 Read 7 Done New Done true
21117 7033 Acked 8032 Read 10 Done New Done true
21118 7034 Acked 8033 Read 3 Done New Done true
21119 7035 Acked 8034 Write 6 Done New Done true
21120 7036 Acked 8035 Write 3 Done New Done true
21121 7037 Acked 8036 Write 2 Done New Done true
21122 7038 Acked 8037 Write 2 Done New Done true
21123 7039 Acked 8038 Read 1 Done New Done true
21124 7040 Acked 8039 Read 9 Done New Done true
21125 7041 Acked 8040 Read 9 Done New Done true
21126 7042 Acked 8041 Write 9 Done New Done true
21127 7043 Acked 8042 Read 4 Done New Done true
21128 7044 Acked 8043 Read 8 Done New Done true
21129 7045 Acked 8044 Write 6 Done New Done true
21130 7046 Acked 8045 Write 9 Done New Done true
21131 7047 Acked 8046 Read 7 Done New Done true
21132 7048 Acked 8047 Write 9 Done New Done true
21133 7049 Acked 8048 Write 9 Done New Done true
21134 7050 Acked 8049 Read 9 Done New Done true
21135 7051 Acked 8050 Flush 0 Done New Done true
21136 7052 Acked 8051 Write 5 Done New Done true
21137 7053 Acked 8052 Write 6 Done New Done true
21138 7054 Acked 8053 Write 9 Done New Done true
21139 7055 Acked 8054 Write 6 Done New Done true
21140 7056 Acked 8055 Read 5 Done New Done true
21141 7057 Acked 8056 Read 10 Done New Done true
21142 7058 Acked 8057 Read 1 Done New Done true
21143 7059 Acked 8058 Write 2 Done New Done true
21144 7060 Acked 8059 Flush 0 Done New Done true
21145 7061 Acked 8060 Read 5 Done New Done true
21146 7062 Acked 8061 Flush 0 Done New Done true
21147 7063 Acked 8062 Write 2 Done New Done true
21148 7064 Acked 8063 Read 1 Done New Done true
21149 7065 Acked 8064 Write 6 Done New Done true
21150 7066 Acked 8065 Read 1 Done New Done true
21151 7067 Acked 8066 Read 9 Done New Done true
21152 7068 Acked 8067 Read 2 Done New Done true
21153 7069 Acked 8068 Write 6 Done New Done true
21154 7070 Acked 8069 Read 7 Done New Done true
21155 7071 Acked 8070 Write 9 Done New Done true
21156 7072 Acked 8071 Write 6 Done New Done true
21157 7073 Acked 8072 Read 4 Done New Done true
21158 7074 Acked 8073 Write 8 Done New Done true
21159 7075 Acked 8074 Read 10 Done New Done true
21160 7076 Acked 8075 Read 1 Done New Done true
21161 7077 Acked 8076 Write 2 Done New Done true
21162 7078 Acked 8077 Read 6 Done New Done true
21163 7079 Acked 8078 Flush 0 Done New Done true
21164 7080 Acked 8079 Flush 0 Done New Done true
21165 7081 Acked 8080 Write 9 Done New Done true
21166 7082 Acked 8081 Read 4 Done New Done true
21167 7083 Acked 8082 Write 8 Done New Done true
21168 7084 Acked 8083 Write 7 Done New Done true
21169 7085 Acked 8084 Write 6 Done New Done true
21170 7086 Acked 8085 Read 6 Done New Done true
21171 7087 Acked 8086 Read 10 Done New Done true
21172 7088 Acked 8087 Read 4 Done New Done true
21173 7089 Acked 8088 Read 8 Done New Done true
21174 7090 Acked 8089 Read 4 Done New Done true
21175 7091 Acked 8090 Read 1 Done New Done true
21176 7092 Acked 8091 Read 5 Done New Done true
21177 7093 Acked 8092 Write 4 Done New Done true
21178 7094 Acked 8093 Read 8 Done New Done true
21179 7095 Acked 8094 Read 10 Done New Done true
21180 7096 Acked 8095 Read 9 Done New Done true
21181 7097 Acked 8096 Write 6 Done New Done true
21182 7098 Acked 8097 Read 6 Done New Done true
21183 7099 Acked 8098 Read 5 Done New Done true
21184 7100 Acked 8099 Flush 0 Done New Done true
21185 7101 Acked 8100 Read 7 Done New Done true
21186 7102 Acked 8101 Read 7 Done New Done true
21187 7103 Acked 8102 Write 8 Done New Done true
21188 7104 Acked 8103 Write 2 Done New Done true
21189 7105 Acked 8104 Read 6 Done New Done true
21190 7106 Acked 8105 Write 2 Done New Done true
21191 7107 Acked 8106 Read 8 Done New Done true
21192 7108 Acked 8107 Write 8 Done New Done true
21193 7109 Acked 8108 Write 2 Done New Done true
21194 7110 Acked 8109 Read 8 Done New Done true
21195 7111 Acked 8110 Read 4 Done New Done true
21196 7112 Acked 8111 Write 2 Done New Done true
21197 7113 Acked 8112 Flush 0 Done New Done true
21198 7114 Acked 8113 Flush 0 Done New Done true
21199 7115 Acked 8114 Flush 0 Done New Done true
21200 7116 Acked 8115 Read 4 Done New Done true
21201 7117 Acked 8116 Write 2 Done New Done true
21202 7118 Acked 8117 Read 7 Done New Done true
21203 7119 Acked 8118 Read 10 Done New Done true
21204 7120 Acked 8119 Read 4 Done New Done true
21205 7121 Acked 8120 Read 6 Done New Done true
21206 7122 Acked 8121 Write 2 Done New Done true
21207 7123 Acked 8122 Write 2 Done New Done true
21208 7124 Acked 8123 Write 6 Done New Done true
21209 7125 Acked 8124 Write 6 Done New Done true
21210 7126 Acked 8125 Read 3 Done New Done true
21211 7127 Acked 8126 Write 5 Done New Done true
21212 7128 Acked 8127 Write 9 Done New Done true
21213 7129 Acked 8128 Write 8 Done New Done true
21214 7130 Acked 8129 Flush 0 Done New Done true
21215 7131 Acked 8130 Flush 0 Done New Done true
21216 7132 Acked 8131 Write 5 Done New Done true
21217 7133 Acked 8132 Read 7 Done New Done true
21218 7134 Acked 8133 Flush 0 Done New Done true
21219 7135 Acked 8134 Read 4 Done New Done true
21220 7136 Acked 8135 Write 5 Done New Done true
21221 7137 Acked 8136 Read 7 Done New Done true
21222 7138 Acked 8137 Read 2 Done New Done true
21223 7139 Acked 8138 Read 7 Done New Done true
21224 7140 Acked 8139 Read 7 Done New Done true
21225 7141 Acked 8140 Write 2 Done New Done true
21226 7142 Acked 8141 Read 1 Done New Done true
21227 7143 Acked 8142 Read 8 Done New Done true
21228 7144 Acked 8143 Write 5 Done New Done true
21229 7145 Acked 8144 Write 8 Done New Done true
21230 7146 Acked 8145 Write 9 Done New Done true
21231 7147 Acked 8146 Write 1 Done New Done true
21232 7148 Acked 8147 Write 1 Done New Done true
21233 7149 Acked 8148 Read 7 Done New Done true
21234 7150 Acked 8149 Flush 0 Done New Done true
21235 7151 Acked 8150 Write 2 Done New Done true
21236 7152 Acked 8151 Flush 0 Done New Done true
21237 7153 Acked 8152 Write 4 Done New Done true
21238 7154 Acked 8153 Flush 0 Done New Done true
21239 7155 Acked 8154 Flush 0 Done New Done true
21240 7156 Acked 8155 Read 5 Done New Done true
21241 7157 Acked 8156 Read 10 Done New Done true
21242 7158 Acked 8157 Write 7 Done New Done true
21243 7159 Acked 8158 Write 3 Done New Done true
21244 7160 Acked 8159 Read 1 Done New Done true
21245 7161 Acked 8160 Read 8 Done New Done true
21246 7162 Acked 8161 Write 1 Done New Done true
21247 7163 Acked 8162 Flush 0 Done New Done true
21248 7164 Acked 8163 Write 9 Done New Done true
21249 7165 Acked 8164 Write 6 Done New Done true
21250 7166 Acked 8165 Read 2 Done New Done true
21251 7167 Acked 8166 Read 10 Done New Done true
21252 7168 Acked 8167 Write 1 Done New Done true
21253 7169 Acked 8168 Flush 0 Done New Done true
21254 7170 Acked 8169 Write 2 Done New Done true
21255 7171 Acked 8170 Write 2 Done New Done true
21256 7172 Acked 8171 Flush 0 Done New Done true
21257 7173 Acked 8172 Read 4 Done New Done true
21258 7174 Acked 8173 Read 3 Done New Done true
21259 7175 Acked 8174 Write 9 Done New Done true
21260 7176 Acked 8175 Write 3 Done New Done true
21261 7177 Acked 8176 Flush 0 Done New Done true
21262 7178 Acked 8177 Write 1 Done New Done true
21263 7179 Acked 8178 Read 8 Done New Done true
21264 7180 Acked 8179 Read 9 Done New Done true
21265 7181 Acked 8180 Read 2 Done New Done true
21266 7182 Acked 8181 Read 10 Done New Done true
21267 7183 Acked 8182 Write 8 Done New Done true
21268 7184 Acked 8183 Read 1 Done New Done true
21269 7185 Acked 8184 Flush 0 Done New Done true
21270 7186 Acked 8185 Read 8 Done New Done true
21271 7187 Acked 8186 Flush 0 Done New Done true
21272 7188 Acked 8187 Flush 0 Done New Done true
21273 7189 Acked 8188 Read 10 Done New Done true
21274 7190 Acked 8189 Write 4 Done New Done true
21275 7191 Acked 8190 Write 1 Done New Done true
21276 7192 Acked 8191 Read 9 Done New Done true
21277 7193 Acked 8192 Write 2 Done New Done true
21278 7194 Acked 8193 Read 9 Done New Done true
21279 7195 Acked 8194 Write 2 Done New Done true
21280 7196 Acked 8195 Write 4 Done New Done true
21281 7197 Acked 8196 Flush 0 Done New Done true
21282 7198 Acked 8197 Write 4 Done New Done true
21283 7199 Acked 8198 Read 7 Done New Done true
21284 7200 Acked 8199 Read 2 Done New Done true
21285 7201 Acked 8200 Write 4 Done New Done true
21286 7202 Acked 8201 Write 1 Done New Done true
21287 7203 Acked 8202 Read 5 Done New Done true
21288 7204 Acked 8203 Read 2 Done New Done true
21289 7205 Acked 8204 Read 6 Done New Done true
21290 7206 Acked 8205 Read 7 Done New Done true
21291 7207 Acked 8206 Flush 0 Done New Done true
21292 7208 Acked 8207 Write 4 Done New Done true
21293 7209 Acked 8208 Write 6 Done New Done true
21294 7210 Acked 8209 Read 4 Done New Done true
21295 7211 Acked 8210 Read 5 Done New Done true
21296 7212 Acked 8211 Flush 0 Done New Done true
21297 7213 Acked 8212 Write 1 Done New Done true
21298 7214 Acked 8213 Flush 0 Done New Done true
21299 7215 Acked 8214 Write 2 Done New Done true
21300 7216 Acked 8215 Read 8 Done New Done true
21301 7217 Acked 8216 Read 2 Done New Done true
21302 7218 Acked 8217 Write 2 Done New Done true
21303 7219 Acked 8218 Write 5 Done New Done true
21304 7220 Acked 8219 Read 4 Done New Done true
21305 7221 Acked 8220 Write 5 Done New Done true
21306 7222 Acked 8221 Flush 0 Done New Done true
21307 7223 Acked 8222 Write 6 Done New Done true
21308 7224 Acked 8223 Write 10 Done New Done true
21309 7225 Acked 8224 Write 1 Done New Done true
21310 7226 Acked 8225 Flush 0 Done New Done true
21311 7227 Acked 8226 Read 3 Done New Done true
21312 7228 Acked 8227 Read 4 Done New Done true
21313 7229 Acked 8228 Write 9 Done New Done true
21314 7230 Acked 8229 Read 3 Done New Done true
21315 7231 Acked 8230 Flush 0 Done New Done true
21316 7232 Acked 8231 Read 10 Done New Done true
21317 7233 Acked 8232 Read 7 Done New Done true
21318 7234 Acked 8233 Flush 0 Done New Done true
21319 STATES DS:0 DS:1 DS:2 TOTAL
21320 New 0 302 0 302
21321 Sent 0 0 0 0
21322 Done 302 0 302 604
21323 Skipped 0 0 0 0
21324 Error 0 0 0 0
21325 Last Flush: 8233 7931 8233
21326 Downstairs last five completed: 7931 7930 7929 7928 7927
21327 Upstairs last five completed: 7234 7233 7232 7231 7230
21328 CLIENT: Up:0 ds:302 act:2
213292024-02-27T21:48:51.111ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213302024-02-27T21:48:51.111ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213312024-02-27T21:48:51.111ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213322024-02-27T21:48:51.111ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
213332024-02-27T21:48:51.111ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213342024-02-27T21:48:51.111ZINFOcrucible: [1] client re-new 302 jobs since flush 7931 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
213352024-02-27T21:48:51.111ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
21336 [8820] got start action
21337 [8820] I am going to restart
21338 Starting downstairs at port 8820
21339 Make output file at "/tmp/dsc/downstairs-8820.txt"
21340 [8820][1] reports Starting
213412024-02-27T21:48:52.112ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213422024-02-27T21:48:52.112ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213432024-02-27T21:48:52.112ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213442024-02-27T21:48:52.112ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
213452024-02-27T21:48:52.112ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213462024-02-27T21:48:52.113ZINFOcrucible: [1] client re-new 302 jobs since flush 7931 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
213472024-02-27T21:48:52.113ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
21348 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1191)
21349 [8820][1] reports Running
213502024-02-27T21:48:53.114ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213512024-02-27T21:48:53.114ZINFOcrucible: ds_connection connected = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213522024-02-27T21:48:53.116ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213532024-02-27T21:48:53.116ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213542024-02-27T21:48:53.116ZINFOcrucible: send last flush ID to this DS: 7931 = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213552024-02-27T21:48:53.116ZINFOcrucible: Replied this last flush ID: 7931 = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213562024-02-27T21:48:53.116ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213572024-02-27T21:48:53.116ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213582024-02-27T21:48:53.116ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213592024-02-27T21:48:53.116ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
21360 ----------------------------------------------------------------
21361 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
21362 Downstairs last five completed: 8233 8232 8231 8230 8229
21363 Upstairs last five completed: 7234 7233 7232 7231 7230
21364 CLIENT: Up:0 ds:0 act:3
21365 Replay: All jobs finished, all DS active.
21366 Main task has work to do, go find it
21367 got dsc Stop(1)
21368 stop 1
21369 Feb 27 21:48:54.452 [8820] Got stop action so:false kr:false
21370 INFO request completed, latency_us: 348, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 52912d10-e031-45db-8c98-c20b98efebef, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
21371 Feb 27 21:48:54.453 INFO request completed, latency_us: 135, response_code: 200, uri: /state/cid/1, method: GET, req_id: b348f0c5-44a4-400d-9f9e-92ced1d9d13d, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
213722024-02-27T21:48:54.453ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213732024-02-27T21:48:54.453ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213742024-02-27T21:48:54.453ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = e6a5a974-925a-494f-8845-facc441c2690
213752024-02-27T21:48:54.453ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213762024-02-27T21:48:54.453ZINFOcrucible: [1] client re-new 0 jobs since flush 8233 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
213772024-02-27T21:48:54.453ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
21378 [8820] Exited with: ExitStatus(unix_wait_status(9))
21379 [8820][1] reports Exit
213802024-02-27T21:48:55.455ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213812024-02-27T21:48:55.455ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213822024-02-27T21:48:55.455ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213832024-02-27T21:48:55.455ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
213842024-02-27T21:48:55.455ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213852024-02-27T21:48:55.455ZINFOcrucible: [1] client re-new 0 jobs since flush 8233 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
213862024-02-27T21:48:55.455ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213872024-02-27T21:48:56.456ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213882024-02-27T21:48:56.456ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213892024-02-27T21:48:56.456ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213902024-02-27T21:48:56.456ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
213912024-02-27T21:48:56.456ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213922024-02-27T21:48:56.456ZINFOcrucible: [1] client re-new 0 jobs since flush 8233 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
213932024-02-27T21:48:56.456ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213942024-02-27T21:48:57.457ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213952024-02-27T21:48:57.457ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213962024-02-27T21:48:57.457ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213972024-02-27T21:48:57.457ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
213982024-02-27T21:48:57.457ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
213992024-02-27T21:48:57.457ZINFOcrucible: [1] client re-new 0 jobs since flush 8233 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
214002024-02-27T21:48:57.458ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
21401 Feb 27 21:48:58.455 INFO request completed, latency_us: 200, response_code: 200, uri: /state/cid/1, method: GET, req_id: 4bf1975f-d0ff-4ced-8943-658853627b6e, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
21402 001/300 Flush
21403 002/300 Read block 4197 len 24576
214042024-02-27T21:48:58.458ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
214052024-02-27T21:48:58.458ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
21406 {"msg":"client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: \"Connection refused\" }))","v":0,"name":"crucible","level":40003/300 Write block 1101 len 12288 data: 6 5 5
21407 ,"time":"2024-02-27T21:48:58.458712784Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"io task","client":"1","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
214082024-02-27T21:48:58.458ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
214092024-02-27T21:48:58.458ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
214102024-02-27T21:48:58.458ZINFOcrucible: [1] client re-new 2 jobs since flush 8233 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
214112024-02-27T21:48:58.458ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
21412 004/300 Read block 4896 len 28672
21413 005/300 Read block 2799 len 12288
21414 006/300 Write block 4793 len 28672 data: 7 7 5 7 6 5 5
21415 007/300 Read block 4337 len 12288
21416 008/300 Read block 1474 len 16384
21417 009/300 Read block 3107 len 28672
21418 010/300 Flush
21419 011/300 Write block 2980 len 24576 data: 5 4 4 4 4 3
21420 012/300 Write block 552 len 36864 data: 7 5 5 6 6 5 5 5 4
21421 013/300 Write block 1233 len 36864 data: 5 5 6 5 4 3 3 3 2
21422 014/300 Read block 140 len 16384
21423 015/300 Read block 3332 len 40960
21424 016/300 Read block 982 len 16384
21425 017/300 Write block 3464 len 36864 data: 3 3 4 2 3 3 4 5 6
21426 018/300 Write block 2540 len 8192 data: 6 5
21427 019/300 Read block 1122 len 8192
21428 020/300 Read block 2779 len 36864
21429 021/300 Write block 1014 len 28672 data: 7 5 4 4 5 7 6
21430 022/300 Write block 1148 len 36864 data: 6 5 5 4 2 2 6 6 5
21431 023/300 Write block 2204 len 12288 data: 6 6 9
21432 024/300 Read block 4388 len 40960
21433 025/300 Read block 1846 len 40960
21434 026/300 Write block 1035 len 8192 data: 4 6
21435 027/300 Write block 4993 len 16384 data: 4 3 3 3
21436 028/300 Read block 479 len 24576
21437 029/300 Write block 4586 len 12288 data: 4 3 6
21438 030/300 Flush
21439 031/300 Read block 2842 len 28672
21440 032/300 Write block 2979 len 36864 data: 6 6 5 5 5 5 4 3 3
21441 033/300 Write block 254 len 32768 data: 4 4 5 6 6 7 6 5
21442 034/300 Write block 3084 len 12288 data: 6 5 5
21443 035/300 Read block 2238 len 12288
21444 036/300 Flush
21445 037/300 Read block 1685 len 8192
21446 038/300 Read block 3790 len 16384
21447 039/300 Write block 2499 len 24576 data: 5 5 5 8 11 12
21448 040/300 Write block 2465 len 4096 data: 4
21449 041/300 Write block 102 len 8192 data: 5 4
21450 042/300 Write block 3922 len 16384 data: 4 4 6 6
21451 043/300 Write block 1432 len 8192 data: 6 7
21452 044/300 Read block 1084 len 28672
21453 045/300 Read block 2519 len 20480
21454 046/300 Read block 744 len 28672
21455 047/300 Flush
21456 048/300 Write block 1866 len 20480 data: 5 4 4 3 3
21457 049/300 Read block 1775 len 12288
21458 050/300 Flush
21459 051/300 Read block 808 len 8192
21460 052/300 Write block 2659 len 4096 data: 6
21461 053/300 Read block 1921 len 36864
21462 054/300 Write block 478 len 28672 data: 3 2 3 2 3 4 4
21463 055/300 Write block 2550 len 4096 data: 5
21464 056/300 Write block 216 len 24576 data: 4 3 4 3 3 4
21465 057/300 Write block 975 len 28672 data: 5 5 6 5 5 5 4
21466 058/300 Read block 1379 len 28672
21467 059/300 Read block 3229 len 16384
21468 060/300 Read block 230 len 8192
21469 061/300 Write block 3748 len 36864 data: 4 4 4 4 4 6 5 5 4
21470 062/300 Write block 3422 len 24576 data: 7 6 4 3 3 3
21471 063/300 Write block 1050 len 24576 data: 3 4 6 5 5 4
21472 064/300 Write block 2305 len 40960 data: 5 4 4 3 6 6 7 7 7 6
21473 065/300 Read block 1473 len 28672
21474 066/300 Write block 4544 len 36864 data: 5 4 4 6 7 7 10 9 8
21475 067/300 Read block 873 len 20480
21476 068/300 Write block 4530 len 24576 data: 6 5 4 5 4 4
21477 069/300 Read block 494 len 40960
21478 070/300 Write block 4385 len 16384 data: 5 5 4 4
21479 071/300 Read block 695 len 4096
21480 072/300 Write block 2293 len 32768 data: 7 6 5 6 6 6 6 6
21481 073/300 Read block 3575 len 36864
21482 074/300 Read block 1125 len 28672
21483 075/300 Read block 4742 len 40960
21484 076/300 Write block 1958 len 28672 data: 3 3 3 3 3 4 5
21485 077/300 Flush
21486 078/300 Read block 3674 len 24576
21487 079/300 Write block 3386 len 28672 data: 6 7 6 4 4 3 4
21488 080/300 Write block 66 len 32768 data: 4 4 4 6 5 6 5 5
21489 081/300 Flush
21490 082/300 Write block 145 len 40960 data: 5 6 4 3 4 4 6 6 5 5
21491 083/300 Write block 2179 len 4096 data: 3
21492 084/300 Read block 4470 len 8192
21493 085/300 Read block 4172 len 24576
21494 086/300 Read block 2488 len 32768
21495 087/300 Write block 337 len 32768 data: 6 5 5 4 4 2 3 4
21496 088/300 Read block 1426 len 16384
21497 089/300 Read block 1422 len 32768
21498 090/300 Write block 3390 len 28672 data: 5 4 5 4 3 3 4
21499 091/300 Write block 2182 len 16384 data: 5 6 5 4
21500 092/300 Read block 896 len 16384
21501 093/300 Read block 4562 len 8192
21502 094/300 Write block 464 len 20480 data: 2 2 2 2 3
21503 095/300 Read block 4177 len 4096
21504 096/300 Flush
21505 097/300 Write block 780 len 24576 data: 4 5 6 5 5 5
21506 098/300 Read block 4050 len 28672
21507 099/300 Write block 4561 len 24576 data: 3 3 4 5 5 5
21508 100/300 Read block 2423 len 24576
21509 101/300 Read block 3045 len 32768
21510 102/300 Write block 3753 len 40960 data: 7 6 6 5 5 4 5 4 4 3
21511 103/300 Flush
21512 104/300 Flush
21513 105/300 Read block 1575 len 4096
21514 106/300 Read block 634 len 36864
21515 107/300 Read block 1239 len 40960
21516 108/300 Read block 3334 len 24576
21517 109/300 Read block 857 len 20480
21518 110/300 Flush
21519 111/300 Read block 4166 len 28672
21520 112/300 Write block 46 len 24576 data: 8 7 7 7 7 5
21521 113/300 Write block 3113 len 28672 data: 9 10 10 9 8 5 4
21522 114/300 Read block 2147 len 4096
21523 115/300 Read block 4094 len 20480
21524 116/300 Read block 4559 len 20480
21525 117/300 Read block 3133 len 8192
21526 118/300 Read block 2198 len 16384
21527 119/300 Write block 3109 len 16384 data: 9 11 12 9
21528 120/300 Write block 1834 len 20480 data: 7 6 5 5 4
21529 121/300 Read block 13 len 24576
21530 122/300 Write block 2500 len 40960 data: 6 6 9 12 13 12 12 12 12 7
21531 123/300 Read block 3109 len 28672
21532 124/300 Write block 1215 len 32768 data: 3 4 7 8 8 8 8 6
21533 125/300 Write block 1606 len 32768 data: 3 4 4 4 4 6 6 8
21534 126/300 Write block 2106 len 24576 data: 2 2 2 2 3 3
21535 127/300 Write block 3010 len 20480 data: 7 6 6 5 4
21536 128/300 Read block 3878 len 20480
21537 129/300 Read block 3517 len 16384
21538 130/300 Read block 1493 len 4096
21539 131/300 Write block 647 len 40960 data: 3 3 4 4 3 4 3 2 3 2
21540 132/300 Read block 3501 len 4096
21541 133/300 Read block 1127 len 32768
21542 134/300 Write block 1549 len 28672 data: 6 6 6 4 3 5 3
21543 135/300 Write block 310 len 20480 data: 4 3 3 3 3
21544 136/300 Read block 132 len 24576
21545 137/300 Read block 2556 len 20480
21546 138/300 Read block 3227 len 4096
21547 139/300 Read block 2593 len 32768
21548 140/300 Read block 3423 len 36864
21549 141/300 Write block 2828 len 24576 data: 6 6 7 6 5 5
21550 142/300 Read block 1955 len 32768
21551 143/300 Write block 4771 len 12288 data: 6 5 6
21552 144/300 Read block 4073 len 20480
21553 145/300 Write block 3337 len 32768 data: 5 6 4 5 5 6 7 7
21554 146/300 Write block 684 len 40960 data: 5 6 6 8 5 4 3 3 3 4
21555 147/300 Write block 4695 len 36864 data: 7 6 5 5 4 4 5 6 7
21556 148/300 Read block 891 len 40960
21557 149/300 Write block 2319 len 32768 data: 5 4 4 4 7 8 7 7
21558 150/300 Write block 4545 len 28672 data: 5 5 7 8 8 11 10
21559 151/300 Read block 1940 len 20480
21560 152/300 Read block 3816 len 32768
21561 153/300 Read block 2130 len 36864
21562 154/300 Write block 2446 len 4096 data: 6
21563 155/300 Read block 3310 len 32768
21564 156/300 Flush
21565 157/300 Read block 3562 len 28672
21566 158/300 Read block 4705 len 8192
21567 159/300 Read block 526 len 20480
21568 160/300 Read block 947 len 12288
21569 161/300 Read block 2986 len 12288
21570 162/300 Read block 1151 len 4096
21571 163/300 Write block 879 len 4096 data: 3
21572 164/300 Flush
21573 165/300 Read block 3135 len 8192
21574 166/300 Write block 262 len 40960 data: 3 3 4 4 6 7 8 7 7 7
21575 167/300 Write block 2544 len 32768 data: 4 4 6 6 6 5 6 4
21576 168/300 Read block 2418 len 12288
21577 169/300 Write block 4600 len 28672 data: 5 5 5 6 5 4 4
21578 170/300 Write block 594 len 8192 data: 8 7
21579 171/300 Read block 1806 len 40960
21580 172/300 Read block 1785 len 20480
21581 173/300 Read block 1554 len 36864
21582 174/300 Read block 2010 len 32768
21583 175/300 Write block 349 len 12288 data: 4 3 2
21584 176/300 Read block 2875 len 40960
21585 177/300 Write block 1848 len 40960 data: 4 3 5 5 4 5 6 4 3 3
21586 178/300 Flush
21587 179/300 Write block 214 len 8192 data: 3 3
21588 180/300 Write block 2237 len 32768 data: 5 6 7 7 7 7 7 5
21589 181/300 Write block 3918 len 36864 data: 7 6 6 5 5 5 7 7 6
21590 182/300 Read block 4231 len 12288
21591 183/300 Read block 374 len 8192
21592 184/300 Read block 319 len 40960
21593 185/300 Read block 838 len 32768
21594 186/300 Write block 4758 len 12288 data: 3 3 3
21595 187/300 Flush
21596 188/300 Read block 2028 len 20480
21597 189/300 Write block 431 len 32768 data: 6 7 7 6 7 7 6 5
21598 190/300 Read block 814 len 12288
21599 191/300 Write block 4308 len 40960 data: 3 3 3 3 3 4 4 4 5 6
21600 192/300 Flush
21601 193/300 Read block 1101 len 36864
21602 194/300 Write block 1179 len 24576 data: 4 3 2 2 2 2
21603 195/300 Read block 666 len 12288
21604 196/300 Read block 2042 len 16384
21605 197/300 Read block 4805 len 28672
21606 198/300 Write block 3264 len 8192 data: 4 5
21607 199/300 Write block 4542 len 36864 data: 4 5 6 6 6 8 9 9 12
21608 200/300 Flush
21609 201/300 Read block 822 len 40960
21610 202/300 Read block 4215 len 24576
21611 203/300 Read block 1015 len 36864
21612 204/300 Read block 3314 len 16384
21613 205/300 Read block 1544 len 20480
21614 206/300 Write block 571 len 4096 data: 5
21615 207/300 Write block 1098 len 12288 data: 4 5 6
21616 208/300 Write block 1501 len 20480 data: 8 6 4 4 5
21617 209/300 Read block 2560 len 32768
21618 210/300 Read block 4518 len 8192
21619 211/300 Read block 1553 len 20480
21620 212/300 Read block 771 len 20480
21621 213/300 Write block 3768 len 36864 data: 7 6 6 6 4 6 7 7 7
21622 214/300 Write block 3247 len 16384 data: 4 3 5 4
21623 215/300 Write block 816 len 8192 data: 3 3
21624 216/300 Read block 2187 len 12288
21625 217/300 Flush
21626 218/300 Write block 3408 len 36864 data: 6 8 10 10 9 8 7 7 7
21627 219/300 Write block 2407 len 20480 data: 5 4 4 4 4
21628 220/300 Write block 4717 len 4096 data: 5
21629 221/300 Read block 409 len 20480
21630 222/300 Flush
21631 223/300 Write block 2604 len 20480 data: 3 3 3 3 4
21632 224/300 Read block 3756 len 24576
21633 225/300 Write block 2784 len 20480 data: 4 3 4 4 4
21634 226/300 Flush
21635 227/300 Read block 4992 len 28672
21636 228/300 Write block 4321 len 24576 data: 7 5 7 7 6 6
21637 229/300 Write block 2701 len 12288 data: 3 3 3
21638 230/300 Write block 4533 len 36864 data: 6 5 5 4 4 3 4 4 3
21639 231/300 Read block 3897 len 20480
21640 232/300 Write block 2050 len 4096 data: 5
21641 233/300 Write block 2382 len 16384 data: 3 5 4 4
21642 234/300 Write block 1479 len 24576 data: 5 5 5 6 3 3
21643 235/300 Flush
21644 236/300 Read block 1624 len 24576
21645 237/300 Read block 2483 len 32768
21646 238/300 Read block 4272 len 32768
21647 239/300 Write block 3161 len 20480 data: 4 5 5 7 7
21648 240/300 Read block 2440 len 36864
21649 241/300 Write block 1940 len 40960 data: 6 5 3 3 3 5 6 6 6 6
21650 242/300 Write block 1254 len 40960 data: 3 3 3 4 5 6 6 5 5 6
21651 243/300 Read block 1894 len 12288
21652 244/300 Write block 1461 len 32768 data: 7 6 5 5 5 5 5 5
21653 245/300 Write block 3005 len 32768 data: 5 5 6 7 7 8 7 7
21654 246/300 Write block 2396 len 36864 data: 5 5 4 3 4 4 3 5 5
21655 247/300 Read block 575 len 12288
21656 248/300 Read block 3431 len 12288
21657 249/300 Write block 3150 len 36864 data: 5 6 6 5 4 5 4 5 4
21658 250/300 Write block 1939 len 20480 data: 5 7 6 4 4
21659 251/300 Read block 3361 len 8192
21660 252/300 Read block 2738 len 28672
21661 253/300 Write block 3497 len 28672 data: 5 4 4 4 4 4 4
21662 254/300 Write block 4003 len 12288 data: 4 3 3
21663 255/300 Read block 4637 len 12288
21664 256/300 Read block 3220 len 20480
21665 257/300 Read block 2709 len 8192
21666 258/300 Read block 1953 len 4096
21667 259/300 Read block 4050 len 40960
21668 260/300 Read block 2404 len 12288
21669 261/300 Read block 3657 len 16384
21670 262/300 Read block 1442 len 28672
21671 263/300 Write block 1608 len 40960 data: 5 5 5 7 7 9 8 8 9 7
21672 264/300 Write block 4690 len 36864 data: 6 6 7 8 8 8 7 6 6
21673 265/300 Read block 67 len 36864
21674 266/300 Write block 804 len 28672 data: 5 4 3 4 4 4 3
21675 267/300 Read block 815 len 36864
21676 268/300 Read block 4338 len 4096
21677 269/300 Read block 2671 len 16384
21678 270/300 Write block 4041 len 28672 data: 2 3 3 3 4 4 4
21679 271/300 Write block 223 len 32768 data: 4 7 7 8 7 5 5 6
21680 272/300 Flush
21681 273/300 Flush
21682 274/300 Write block 4297 len 4096 data: 6
21683 275/300 Read block 3506 len 24576
21684 276/300 Read block 581 len 24576
21685 277/300 Read block 4585 len 8192
21686 278/300 Read block 4981 len 16384
21687 279/300 Flush
21688 280/300 Read block 2130 len 40960
21689 281/300 Read block 4761 len 12288
21690 282/300 Read block 4225 len 32768
21691 283/300 Write block 379 len 4096 data: 4
21692 284/300 Read block 3987 len 16384
21693 285/300 Write block 3170 len 32768 data: 8 6 4 3 4 7 7 8
21694 286/300 Read block 1860 len 16384
21695 287/300 Read block 3482 len 20480
21696 288/300 Write block 1996 len 36864 data: 8 8 6 6 5 5 8 5 5
21697 289/300 Write block 2310 len 20480 data: 7 8 8 8 7
21698 290/300 Read block 1196 len 12288
21699 291/300 Read block 1002 len 20480
21700 292/300 Write block 2246 len 28672 data: 6 4 4 3 3 3 3
21701 293/300 Read block 3055 len 36864
21702 294/300 Read block 4106 len 16384
21703 295/300 Read block 4691 len 20480
21704 296/300 Write block 3544 len 12288 data: 7 3 4
21705 297/300 Read block 1485 len 32768
21706 298/300 Read block 4337 len 28672
21707 299/300 Write block 2632 len 40960 data: 4 4 3 3 4 5 5 5 6 5
21708 300/300 Read block 4762 len 12288
21709 Main task has work to do, go find it
21710 got dsc Start(1)
21711 start 1
21712 Feb 27 21:48:58.801 INFO request completed, latency_us: 223, response_code: 204, uri: /start/cid/1, method: POST, req_id: 62897835-0665-4581-b404-d22470f0597a, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
21713 Replay: started 1, returned:Ok(())
21714 ----------------------------------------------------------------
21715 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
21716 States: Active Offline Active
21717 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
21718 7235 Acked 8234 Flush 0 Done New Done true
21719 7236 Acked 8235 Read 6 Done New Done true
21720 7237 Acked 8236 Write 3 Done New Done false
21721 7238 Acked 8237 Read 7 Done New Done false
21722 7239 Acked 8238 Read 3 Done New Done false
21723 7240 Acked 8239 Write 7 Done New Done false
21724 7241 Acked 8240 Read 3 Done New Done false
21725 7242 Acked 8241 Read 4 Done New Done false
21726 7243 Acked 8242 Flush 0 Done New Done false
21727 7244 Acked 8243 Read 7 Done New Done false
21728 7245 Acked 8244 Flush 0 Done New Done false
21729 7246 Acked 8245 Write 6 Done New Done false
21730 7247 Acked 8246 Write 9 Done New Done false
21731 7248 Acked 8247 Write 9 Done New Done false
21732 7249 Acked 8248 Read 4 Done New Done false
21733 7250 Acked 8249 Read 10 Done New Done false
21734 7251 Acked 8250 Read 4 Done New Done false
21735 7252 Acked 8251 Write 9 Done New Done false
21736 7253 Acked 8252 Write 2 Done New Done false
21737 7254 Acked 8253 Read 2 Done New Done false
21738 7255 Acked 8254 Read 9 Done New Done false
21739 7256 Acked 8255 Write 7 Done New Done false
21740 7257 Acked 8256 Write 9 Done New Done false
21741 7258 Acked 8257 Write 3 Done New Done false
21742 7259 Acked 8258 Read 10 Done New Done false
21743 7260 Acked 8259 Read 10 Done New Done false
21744 7261 Acked 8260 Write 2 Done New Done false
21745 7262 Acked 8261 Write 4 Done New Done false
21746 7263 Acked 8262 Read 6 Done New Done false
21747 7264 Acked 8263 Write 3 Done New Done false
21748 7265 Acked 8264 Flush 0 Done New Done false
21749 7266 Acked 8265 Read 7 Done New Done false
21750 7267 Acked 8266 Write 9 Done New Done false
21751 7268 Acked 8267 Write 8 Done New Done false
21752 7269 Acked 8268 Write 3 Done New Done false
21753 7270 Acked 8269 Read 3 Done New Done false
21754 7271 Acked 8270 Flush 0 Done New Done false
21755 7272 Acked 8271 Read 2 Done New Done false
21756 7273 Acked 8272 Read 4 Done New Done false
21757 7274 Acked 8273 Write 6 Done New Done false
21758 7275 Acked 8274 Write 1 Done New Done false
21759 7276 Acked 8275 Write 2 Done New Done false
21760 7277 Acked 8276 Write 4 Done New Done false
21761 7278 Acked 8277 Write 2 Done New Done false
21762 7279 Acked 8278 Read 7 Done New Done false
21763 7280 Acked 8279 Read 5 Done New Done false
21764 7281 Acked 8280 Read 7 Done New Done false
21765 7282 Acked 8281 Flush 0 Done New Done false
21766 7283 Acked 8282 Write 5 Done New Done false
21767 7284 Acked 8283 Read 3 Done New Done false
21768 7285 Acked 8284 Flush 0 Done New Done false
21769 7286 Acked 8285 Read 2 Done New Done false
21770 7287 Acked 8286 Write 1 Done New Done false
21771 7288 Acked 8287 Read 9 Done New Done false
21772 7289 Acked 8288 Write 7 Done New Done false
21773 7290 Acked 8289 Write 1 Done New Done false
21774 7291 Acked 8290 Write 6 Done New Done false
21775 7292 Acked 8291 Write 7 Done New Done false
21776 7293 Acked 8292 Read 7 Done New Done false
21777 7294 Acked 8293 Read 4 Done New Done false
21778 7295 Acked 8294 Read 2 Done New Done false
21779 7296 Acked 8295 Write 9 Done New Done false
21780 7297 Acked 8296 Write 6 Done New Done false
21781 7298 Acked 8297 Write 6 Done New Done false
21782 7299 Acked 8298 Write 10 Done New Done false
21783 7300 Acked 8299 Read 7 Done New Done false
21784 7301 Acked 8300 Write 9 Done New Done false
21785 7302 Acked 8301 Read 5 Done New Done false
21786 7303 Acked 8302 Write 6 Done New Done false
21787 7304 Acked 8303 Read 10 Done New Done false
21788 7305 Acked 8304 Write 4 Done New Done false
21789 7306 Acked 8305 Read 1 Done New Done false
21790 7307 Acked 8306 Write 8 Done New Done false
21791 7308 Acked 8307 Read 9 Done New Done false
21792 7309 Acked 8308 Read 7 Done New Done false
21793 7310 Acked 8309 Read 10 Done New Done false
21794 7311 Acked 8310 Write 7 Done New Done false
21795 7312 Acked 8311 Flush 0 Done New Done false
21796 7313 Acked 8312 Read 6 Done New Done false
21797 7314 Acked 8313 Write 7 Done New Done false
21798 7315 Acked 8314 Write 8 Done New Done false
21799 7316 Acked 8315 Flush 0 Done New Done false
21800 7317 Acked 8316 Write 10 Done New Done false
21801 7318 Acked 8317 Write 1 Done New Done false
21802 7319 Acked 8318 Read 2 Done New Done false
21803 7320 Acked 8319 Read 6 Done New Done false
21804 7321 Acked 8320 Read 8 Done New Done false
21805 7322 Acked 8321 Write 8 Done New Done false
21806 7323 Acked 8322 Read 4 Done New Done false
21807 7324 Acked 8323 Read 8 Done New Done false
21808 7325 Acked 8324 Write 7 Done New Done false
21809 7326 Acked 8325 Write 4 Done New Done false
21810 7327 Acked 8326 Read 4 Done New Done false
21811 7328 Acked 8327 Read 2 Done New Done false
21812 7329 Acked 8328 Write 5 Done New Done false
21813 7330 Acked 8329 Read 1 Done New Done false
21814 7331 Acked 8330 Flush 0 Done New Done false
21815 7332 Acked 8331 Write 6 Done New Done false
21816 7333 Acked 8332 Read 7 Done New Done false
21817 7334 Acked 8333 Write 6 Done New Done false
21818 7335 Acked 8334 Read 6 Done New Done false
21819 7336 Acked 8335 Read 8 Done New Done false
21820 7337 Acked 8336 Write 10 Done New Done false
21821 7338 Acked 8337 Flush 0 Done New Done false
21822 7339 Acked 8338 Flush 0 Done New Done false
21823 7340 Acked 8339 Read 1 Done New Done false
21824 7341 Acked 8340 Read 9 Done New Done false
21825 7342 Acked 8341 Read 10 Done New Done false
21826 7343 Acked 8342 Read 6 Done New Done false
21827 7344 Acked 8343 Read 5 Done New Done false
21828 7345 Acked 8344 Flush 0 Done New Done false
21829 7346 Acked 8345 Read 7 Done New Done false
21830 7347 Acked 8346 Write 6 Done New Done false
21831 7348 Acked 8347 Write 7 Done New Done false
21832 7349 Acked 8348 Read 1 Done New Done false
21833 7350 Acked 8349 Read 5 Done New Done false
21834 7351 Acked 8350 Read 5 Done New Done false
21835 7352 Acked 8351 Read 2 Done New Done false
21836 7353 Acked 8352 Read 4 Done New Done false
21837 7354 Acked 8353 Write 4 Done New Done false
21838 7355 Acked 8354 Write 5 Done New Done false
21839 7356 Acked 8355 Read 6 Done New Done false
21840 7357 Acked 8356 Write 10 Done New Done false
21841 7358 Acked 8357 Read 7 Done New Done false
21842 7359 Acked 8358 Write 8 Done New Done false
21843 7360 Acked 8359 Write 8 Done New Done false
21844 7361 Acked 8360 Write 6 Done New Done false
21845 7362 Acked 8361 Write 5 Done New Done false
21846 7363 Acked 8362 Read 5 Done New Done false
21847 7364 Acked 8363 Read 4 Done New Done false
21848 7365 Acked 8364 Read 1 Done New Done false
21849 7366 Acked 8365 Write 10 Done New Done false
21850 7367 Acked 8366 Read 1 Done New Done false
21851 7368 Acked 8367 Read 8 Done New Done false
21852 7369 Acked 8368 Write 7 Done New Done false
21853 7370 Acked 8369 Write 5 Done New Done false
21854 7371 Acked 8370 Read 6 Done New Done false
21855 7372 Acked 8371 Read 5 Done New Done false
21856 7373 Acked 8372 Read 1 Done New Done false
21857 7374 Acked 8373 Read 8 Done New Done false
21858 7375 Acked 8374 Read 9 Done New Done false
21859 7376 Acked 8375 Write 6 Done New Done false
21860 7377 Acked 8376 Read 8 Done New Done false
21861 7378 Acked 8377 Write 3 Done New Done false
21862 7379 Acked 8378 Read 5 Done New Done false
21863 7380 Acked 8379 Write 8 Done New Done false
21864 7381 Acked 8380 Write 10 Done New Done false
21865 7382 Acked 8381 Write 9 Done New Done false
21866 7383 Acked 8382 Read 10 Done New Done false
21867 7384 Acked 8383 Write 8 Done New Done false
21868 7385 Acked 8384 Write 7 Done New Done false
21869 7386 Acked 8385 Read 5 Done New Done false
21870 7387 Acked 8386 Read 8 Done New Done false
21871 7388 Acked 8387 Read 9 Done New Done false
21872 7389 Acked 8388 Write 1 Done New Done false
21873 7390 Acked 8389 Read 8 Done New Done false
21874 7391 Acked 8390 Flush 0 Done New Done false
21875 7392 Acked 8391 Read 7 Done New Done false
21876 7393 Acked 8392 Read 2 Done New Done false
21877 7394 Acked 8393 Read 5 Done New Done false
21878 7395 Acked 8394 Read 3 Done New Done false
21879 7396 Acked 8395 Read 3 Done New Done false
21880 7397 Acked 8396 Read 1 Done New Done false
21881 7398 Acked 8397 Write 1 Done New Done false
21882 7399 Acked 8398 Flush 0 Done New Done false
21883 7400 Acked 8399 Read 2 Done New Done false
21884 7401 Acked 8400 Write 10 Done New Done false
21885 7402 Acked 8401 Write 8 Done New Done false
21886 7403 Acked 8402 Read 3 Done New Done false
21887 7404 Acked 8403 Write 7 Done New Done false
21888 7405 Acked 8404 Write 2 Done New Done false
21889 7406 Acked 8405 Read 10 Done New Done false
21890 7407 Acked 8406 Read 5 Done New Done false
21891 7408 Acked 8407 Read 9 Done New Done false
21892 7409 Acked 8408 Read 8 Done New Done false
21893 7410 Acked 8409 Write 3 Done New Done false
21894 7411 Acked 8410 Read 10 Done New Done false
21895 7412 Acked 8411 Write 10 Done New Done false
21896 7413 Acked 8412 Flush 0 Done New Done false
21897 7414 Acked 8413 Write 2 Done New Done false
21898 7415 Acked 8414 Write 8 Done New Done false
21899 7416 Acked 8415 Write 9 Done New Done false
21900 7417 Acked 8416 Read 3 Done New Done false
21901 7418 Acked 8417 Read 2 Done New Done false
21902 7419 Acked 8418 Read 10 Done New Done false
21903 7420 Acked 8419 Read 8 Done New Done false
21904 7421 Acked 8420 Write 3 Done New Done false
21905 7422 Acked 8421 Flush 0 Done New Done false
21906 7423 Acked 8422 Read 5 Done New Done false
21907 7424 Acked 8423 Write 8 Done New Done false
21908 7425 Acked 8424 Read 3 Done New Done false
21909 7426 Acked 8425 Write 10 Done New Done false
21910 7427 Acked 8426 Flush 0 Done New Done false
21911 7428 Acked 8427 Read 9 Done New Done false
21912 7429 Acked 8428 Write 6 Done New Done false
21913 7430 Acked 8429 Read 3 Done New Done false
21914 7431 Acked 8430 Read 4 Done New Done false
21915 7432 Acked 8431 Read 7 Done New Done false
21916 7433 Acked 8432 Write 2 Done New Done false
21917 7434 Acked 8433 Write 9 Done New Done false
21918 7435 Acked 8434 Flush 0 Done New Done false
21919 7436 Acked 8435 Read 10 Done New Done false
21920 7437 Acked 8436 Read 6 Done New Done false
21921 7438 Acked 8437 Read 9 Done New Done false
21922 7439 Acked 8438 Read 4 Done New Done false
21923 7440 Acked 8439 Read 5 Done New Done false
21924 7441 Acked 8440 Write 1 Done New Done false
21925 7442 Acked 8441 Write 3 Done New Done false
21926 7443 Acked 8442 Write 5 Done New Done false
21927 7444 Acked 8443 Read 8 Done New Done false
21928 7445 Acked 8444 Read 2 Done New Done false
21929 7446 Acked 8445 Read 5 Done New Done false
21930 7447 Acked 8446 Read 5 Done New Done false
21931 7448 Acked 8447 Write 9 Done New Done false
21932 7449 Acked 8448 Write 4 Done New Done false
21933 7450 Acked 8449 Write 2 Done New Done false
21934 7451 Acked 8450 Read 3 Done New Done false
21935 7452 Acked 8451 Flush 0 Done New Done false
21936 7453 Acked 8452 Write 9 Done New Done false
21937 7454 Acked 8453 Write 5 Done New Done false
21938 7455 Acked 8454 Write 1 Done New Done false
21939 7456 Acked 8455 Read 5 Done New Done false
21940 7457 Acked 8456 Flush 0 Done New Done false
21941 7458 Acked 8457 Write 5 Done New Done false
21942 7459 Acked 8458 Read 6 Done New Done false
21943 7460 Acked 8459 Write 5 Done New Done false
21944 7461 Acked 8460 Flush 0 Done New Done false
21945 7462 Acked 8461 Read 7 Done New Done false
21946 7463 Acked 8462 Write 6 Done New Done false
21947 7464 Acked 8463 Write 3 Done New Done false
21948 7465 Acked 8464 Write 9 Done New Done false
21949 7466 Acked 8465 Read 5 Done New Done false
21950 7467 Acked 8466 Write 1 Done New Done false
21951 7468 Acked 8467 Write 4 Done New Done false
21952 7469 Acked 8468 Write 6 Done New Done false
21953 7470 Acked 8469 Flush 0 Done New Done false
21954 7471 Acked 8470 Read 6 Done New Done false
21955 7472 Acked 8471 Read 8 Done New Done false
21956 7473 Acked 8472 Read 8 Done New Done false
21957 7474 Acked 8473 Write 5 Done New Done false
21958 7475 Acked 8474 Read 9 Done New Done false
21959 7476 Acked 8475 Write 10 Done New Done false
21960 7477 Acked 8476 Write 10 Done New Done false
21961 7478 Acked 8477 Read 3 Done New Done false
21962 7479 Acked 8478 Write 8 Done New Done false
21963 7480 Acked 8479 Write 8 Done New Done false
21964 7481 Acked 8480 Write 9 Done New Done false
21965 7482 Acked 8481 Read 3 Done New Done false
21966 7483 Acked 8482 Read 3 Done New Done false
21967 7484 Acked 8483 Write 9 Done New Done false
21968 7485 Acked 8484 Write 5 Done New Done false
21969 7486 Acked 8485 Read 2 Done New Done false
21970 7487 Acked 8486 Read 7 Done New Done false
21971 7488 Acked 8487 Write 7 Done New Done false
21972 7489 Acked 8488 Write 3 Done New Done false
21973 7490 Acked 8489 Read 3 Done New Done false
21974 7491 Acked 8490 Read 5 Done New Done false
21975 7492 Acked 8491 Read 2 Done New Done false
21976 7493 Acked 8492 Read 1 Done New Done false
21977 7494 Acked 8493 Read 10 Done New Done false
21978 7495 Acked 8494 Read 3 Done New Done false
21979 7496 Acked 8495 Read 4 Done New Done false
21980 7497 Acked 8496 Read 7 Done New Done false
21981 7498 Acked 8497 Write 10 Done New Done false
21982 7499 Acked 8498 Write 9 Done New Done false
21983 7500 Acked 8499 Read 9 Done New Done false
21984 7501 Acked 8500 Write 7 Done New Done false
21985 7502 Acked 8501 Read 9 Done New Done false
21986 7503 Acked 8502 Read 1 Done New Done false
21987 7504 Acked 8503 Read 4 Done New Done false
21988 7505 Acked 8504 Write 7 Done New Done false
21989 7506 Acked 8505 Write 8 Done New Done false
21990 7507 Acked 8506 Flush 0 Done New Done false
21991 7508 Acked 8507 Flush 0 Done New Done false
21992 7509 Acked 8508 Write 1 Done New Done false
21993 7510 Acked 8509 Read 6 Done New Done false
21994 7511 Acked 8510 Read 6 Done New Done false
21995 7512 Acked 8511 Read 2 Done New Done false
21996 7513 Acked 8512 Read 4 Done New Done false
21997 7514 Acked 8513 Flush 0 Done New Done false
21998 7515 Acked 8514 Read 10 Done New Done false
21999 7516 Acked 8515 Read 3 Done New Done false
22000 7517 Acked 8516 Read 8 Done New Done false
22001 7518 Acked 8517 Write 1 Done New Done false
22002 7519 Acked 8518 Read 4 Done New Done false
22003 7520 Acked 8519 Write 8 Done New Done false
22004 7521 Acked 8520 Read 4 Done New Done false
22005 7522 Acked 8521 Read 5 Done New Done false
22006 7523 Acked 8522 Write 9 Done New Done false
22007 7524 Acked 8523 Write 5 Done New Done false
22008 7525 Acked 8524 Read 3 Done New Done false
22009 7526 Acked 8525 Read 5 Done New Done false
22010 7527 Acked 8526 Write 7 Done New Done false
22011 7528 Acked 8527 Read 9 Done New Done false
22012 7529 Acked 8528 Read 4 Done New Done false
22013 7530 Acked 8529 Read 5 Done New Done false
22014 7531 Acked 8530 Write 3 Done New Done false
22015 7532 Acked 8531 Read 8 Done New Done false
22016 7533 Acked 8532 Read 7 Done New Done false
22017 7534 Acked 8533 Write 10 Done New Done false
22018 7535 Acked 8534 Read 3 Done New Done false
22019 STATES DS:0 DS:1 DS:2 TOTAL
22020 New 0 301 0 301
22021 Sent 0 0 0 0
22022 Done 301 0 301 602
22023 Skipped 0 0 0 0
22024 Error 0 0 0 0
22025 Last Flush: 8513 8233 8513
22026 Downstairs last five completed: 8233 8232 8231 8230 8229
22027 Upstairs last five completed: 7535 7534 7533 7532 7531
22028 CLIENT: Up:0 ds:301 act:2
220292024-02-27T21:48:59.459ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
220302024-02-27T21:48:59.459ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
220312024-02-27T21:48:59.459ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
220322024-02-27T21:48:59.459ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
220332024-02-27T21:48:59.459ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
220342024-02-27T21:48:59.459ZINFOcrucible: [1] client re-new 302 jobs since flush 8233 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
220352024-02-27T21:48:59.459ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
220362024-02-27T21:49:00.460ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
220372024-02-27T21:49:00.460ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
220382024-02-27T21:49:00.460ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
220392024-02-27T21:49:00.460ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
220402024-02-27T21:49:00.460ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
220412024-02-27T21:49:00.460ZINFOcrucible: [1] client re-new 302 jobs since flush 8233 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
22042 [8820] got start action
220432024-02-27T21:49:00.460ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
22044 [8820] I am going to restart
22045 Starting downstairs at port 8820
22046 Make output file at "/tmp/dsc/downstairs-8820.txt"
22047 [8820][1] reports Starting
22048 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1192)
22049 [8820][1] reports Running
220502024-02-27T21:49:01.461ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
220512024-02-27T21:49:01.461ZINFOcrucible: ds_connection connected = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
220522024-02-27T21:49:01.463ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
220532024-02-27T21:49:01.463ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
220542024-02-27T21:49:01.463ZINFOcrucible: send last flush ID to this DS: 8233 = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
220552024-02-27T21:49:01.463ZINFOcrucible: Replied this last flush ID: 8233 = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
220562024-02-27T21:49:01.463ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
220572024-02-27T21:49:01.463ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
220582024-02-27T21:49:01.463ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
220592024-02-27T21:49:01.463ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
22060 ----------------------------------------------------------------
22061 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22062 Downstairs last five completed: 8535 8534 8533 8532 8531
22063 Upstairs last five completed: 7536 7535 7534 7533 7532
22064 CLIENT: Up:0 ds:0 act:3
22065 Replay: All jobs finished, all DS active.
22066 Main task has work to do, go find it
22067 got dsc Stop(2)
22068 stop 2
22069 Feb 27 21:49:02.805 INFO request completed, latency_us: 377, response_code: 204, uri: /stop/cid/2, method: POST, req_id: [8830] Got stop action so:false kr:false
22070 8905bb61-84fe-4c07-bc89-9b40be59c62a, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
220712024-02-27T21:49:02.805ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
22072 Feb 27 21:49:02.806{"msg":"client task is sending Done(Finished)","v":0,"name":"crucible","level":40,"time":"2024-02-27T21:49:02.80601654Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"io task","client":"2","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
220732024-02-27T21:49:02.806ZWARNcrucible: downstairs task for 2 stopped due to Finished session_id = e6a5a974-925a-494f-8845-facc441c2690
22074 INFO request completed, latency_us: 110, response_code{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:49:02.806075742Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"client":"2","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
22075 : 200, uri: /state/cid/2, method: GET, req_id: e76a41ff-efa8-4aee-a283-79ae435d4937, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
220762024-02-27T21:49:02.806ZINFOcrucible: [2] client re-new 0 jobs since flush 8535 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
220772024-02-27T21:49:02.806ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
22078 [8830] Exited with: ExitStatus(unix_wait_status(9))
22079 [8830][2] reports Exit
220802024-02-27T21:49:03.807ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
220812024-02-27T21:49:03.807ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
220822024-02-27T21:49:03.807ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
220832024-02-27T21:49:03.807ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
220842024-02-27T21:49:03.807ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
220852024-02-27T21:49:03.807ZINFOcrucible: [2] client re-new 0 jobs since flush 8535 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
220862024-02-27T21:49:03.807ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
220872024-02-27T21:49:04.808ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
220882024-02-27T21:49:04.808ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
220892024-02-27T21:49:04.808ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
220902024-02-27T21:49:04.808ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
220912024-02-27T21:49:04.809ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
220922024-02-27T21:49:04.809ZINFOcrucible: [2] client re-new 0 jobs since flush 8535 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
220932024-02-27T21:49:04.809ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
220942024-02-27T21:49:05.810ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
220952024-02-27T21:49:05.810ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
220962024-02-27T21:49:05.810ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
220972024-02-27T21:49:05.810ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
220982024-02-27T21:49:05.810ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
220992024-02-27T21:49:05.810ZINFOcrucible: [2] client re-new 0 jobs since flush 8535 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
221002024-02-27T21:49:05.810ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
22101 Feb 27 21:49:06.808 INFO request completed, latency_us: 228, response_code: 200, uri: /state/cid/2, method: GET, req_id: b44d9aa6-c4ef-4b8f-acce-9405c58fa4f5, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
22102 001/300 Read block 2625 len 16384
22103 002/300 Read block 2017 len 32768
221042024-02-27T21:49:06.811ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
221052024-02-27T21:49:06.811ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
221062024-02-27T21:49:06.811ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
221072024-02-27T21:49:06.811ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
221082024-02-27T21:49:06.811ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
221092024-02-27T21:49:06.811ZINFOcrucible: [2] client re-new 2 jobs since flush 8535 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
221102024-02-27T21:49:06.811ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
22111 003/300 Read block 1390 len 20480
22112 004/300 Flush
22113 005/300 Read block 776 len 4096
22114 006/300 Read block 306 len 28672
22115 007/300 Write block 701 len 24576 data: 4 4 3 3 5 6
22116 008/300 Read block 1019 len 16384
22117 009/300 Read block 1211 len 4096
22118 010/300 Read block 4071 len 4096
22119 011/300 Flush
22120 012/300 Read block 2324 len 12288
22121 013/300 Read block 2223 len 4096
22122 014/300 Write block 3020 len 12288 data: 7 7 8
22123 015/300 Write block 3457 len 20480 data: 5 4 4 4 4
22124 016/300 Read block 3308 len 12288
22125 017/300 Write block 654 len 32768 data: 3 4 3 2 2 2 3 3
22126 018/300 Write block 1165 len 16384 data: 5 5 5 4
22127 019/300 Write block 3078 len 4096 data: 2
22128 020/300 Write block 665 len 20480 data: 7 6 5 7 7
22129 021/300 Read block 2654 len 24576
22130 022/300 Write block 1039 len 16384 data: 8 8 7 5
22131 023/300 Read block 674 len 8192
22132 024/300 Write block 1690 len 24576 data: 4 3 3 2 4 4
22133 025/300 Read block 2297 len 16384
22134 026/300 Write block 4169 len 36864 data: 2 2 2 2 3 3 3 4 4
22135 027/300 Write block 1401 len 32768 data: 5 5 5 5 5 6 6 6
22136 028/300 Flush
22137 029/300 Write block 3976 len 4096 data: 4
22138 030/300 Read block 781 len 28672
22139 031/300 Write block 3234 len 36864 data: 7 7 6 6 8 10 8 9 6
22140 032/300 Write block 4763 len 40960 data: 5 4 6 6 5 4 5 6 7 6
22141 033/300 Write block 2843 len 4096 data: 6
22142 034/300 Write block 2493 len 16384 data: 7 7 6 5
22143 035/300 Read block 3129 len 16384
22144 036/300 Write block 807 len 4096 data: 5
22145 037/300 Write block 4853 len 28672 data: 3 4 4 6 3 3 3
22146 038/300 Write block 2372 len 32768 data: 5 6 5 6 6 5 5 5
22147 039/300 Read block 2267 len 8192
22148 040/300 Read block 4226 len 28672
22149 041/300 Read block 3740 len 36864
22150 042/300 Write block 3564 len 32768 data: 4 3 3 4 5 6 8 8
22151 043/300 Read block 3392 len 28672
22152 044/300 Write block 3815 len 24576 data: 6 7 6 7 8 8
22153 045/300 Read block 2148 len 32768
22154 046/300 Write block 396 len 24576 data: 7 7 6 8 9 8
22155 047/300 Read block 2372 len 12288
22156 048/300 Read block 6 len 28672
22157 049/300 Write block 4796 len 28672 data: 8 7 6 6 5 5 6
22158 050/300 Write block 1654 len 40960 data: 4 5 5 5 6 6 8 6 8 9
22159 051/300 Write block 4133 len 40960 data: 5 4 4 4 4 5 3 5 7 7
22160 052/300 Flush
22161 053/300 Write block 4947 len 4096 data: 5
22162 054/300 Read block 1889 len 28672
22163 055/300 Read block 3157 len 8192
22164 056/300 Read block 1705 len 12288
22165 057/300 Write block 2386 len 4096 data: 3
22166 058/300 Write block 3064 len 20480 data: 7 6 6 6 5
22167 059/300 Write block 1914 len 12288 data: 8 6 6
22168 060/300 Write block 4178 len 12288 data: 4 4 4
22169 061/300 Write block 969 len 36864 data: 6 6 6 6 7 5 6 6 7
22170 062/300 Read block 3053 len 24576
22171 063/300 Read block 3066 len 4096
22172 064/300 Read block 3816 len 28672
22173 065/300 Write block 1853 len 28672 data: 6 7 5 4 4 3 3
22174 066/300 Write block 2467 len 36864 data: 5 5 6 6 6 6 7 7 5
22175 067/300 Write block 2567 len 36864 data: 3 4 5 9 9 9 7 7 8
22176 068/300 Write block 2902 len 36864 data: 4 4 3 5 6 5 6 7 9
22177 069/300 Write block 1686 len 4096 data: 5
22178 070/300 Write block 634 len 32768 data: 5 6 4 4 4 3 3 3
22179 071/300 Read block 2398 len 16384
22180 072/300 Read block 135 len 40960
22181 073/300 Read block 985 len 24576
22182 074/300 Read block 3238 len 4096
22183 075/300 Write block 3944 len 16384 data: 5 7 8 7
22184 076/300 Read block 3787 len 4096
22185 077/300 Write block 1670 len 12288 data: 8 7 6
22186 078/300 Write block 1532 len 12288 data: 2 2 2
22187 079/300 Flush
22188 080/300 Write block 233 len 20480 data: 3 3 5 5 5
22189 081/300 Read block 1282 len 24576
22190 082/300 Write block 1904 len 28672 data: 4 3 3 3 3 3 5
22191 083/300 Read block 3008 len 32768
22192 084/300 Flush
22193 085/300 Read block 188 len 4096
22194 086/300 Read block 2114 len 32768
22195 087/300 Write block 4374 len 40960 data: 4 5 3 3 3 4 5 5 5 6
22196 088/300 Flush
22197 089/300 Write block 3023 len 24576 data: 8 8 6 6 6 7
22198 090/300 Write block 2114 len 8192 data: 3 3
22199 091/300 Read block 1289 len 24576
22200 092/300 Write block 1565 len 32768 data: 4 5 4 4 5 5 3 3
22201 093/300 Flush
22202 094/300 Read block 3860 len 12288
22203 095/300 Write block 3249 len 36864 data: 6 5 4 4 4 6 8 6 7
22204 096/300 Write block 3539 len 4096 data: 8
22205 097/300 Write block 62 len 4096 data: 3
22206 098/300 Write block 984 len 12288 data: 6 6 5
22207 099/300 Write block 1598 len 32768 data: 5 6 5 5 4 5 3 3
22208 100/300 Write block 4193 len 40960 data: 4 3 3 4 5 5 5 5 3 4
22209 101/300 Read block 37 len 28672
22210 102/300 Write block 1215 len 32768 data: 4 5 8 9 9 9 9 7
22211 103/300 Read block 4289 len 4096
22212 104/300 Read block 4913 len 32768
22213 105/300 Write block 931 len 4096 data: 5
22214 106/300 Read block 3466 len 40960
22215 107/300 Read block 1246 len 4096
22216 108/300 Read block 2502 len 4096
22217 109/300 Read block 2380 len 36864
22218 110/300 Read block 4812 len 12288
22219 111/300 Write block 1522 len 8192 data: 4 3
22220 112/300 Flush
22221 113/300 Write block 3227 len 8192 data: 4 5
22222 114/300 Read block 4207 len 32768
22223 115/300 Read block 2339 len 12288
22224 116/300 Read block 2060 len 16384
22225 117/300 Read block 3059 len 16384
22226 118/300 Read block 166 len 4096
22227 119/300 Read block 4134 len 32768
22228 120/300 Write block 3466 len 36864 data: 5 3 4 4 5 6 7 6 6
22229 121/300 Read block 4203 len 8192
22230 122/300 Read block 2847 len 40960
22231 123/300 Write block 3445 len 36864 data: 6 8 8 9 8 8 7 6 5
22232 124/300 Write block 4855 len 32768 data: 5 7 4 4 4 2 2 2
22233 125/300 Write block 4925 len 8192 data: 6 5
22234 126/300 Read block 2227 len 20480
22235 127/300 Read block 1922 len 40960
22236 128/300 Write block 3002 len 28672 data: 10 8 5 6 6 7 8
22237 129/300 Read block 2995 len 28672
22238 130/300 Read block 661 len 28672
22239 131/300 Flush
22240 132/300 Read block 4353 len 28672
22241 133/300 Write block 1298 len 20480 data: 2 3 3 4 5
22242 134/300 Flush
22243 135/300 Read block 3619 len 16384
22244 136/300 Read block 4990 len 12288
22245 137/300 Write block 99 len 24576 data: 5 5 5 6 5 4
22246 138/300 Read block 2532 len 20480
22247 139/300 Write block 3168 len 28672 data: 8 8 9 7 5 4 5
22248 140/300 Flush
22249 141/300 Read block 4600 len 12288
22250 142/300 Read block 2309 len 12288
22251 143/300 Write block 3358 len 8192 data: 4 4
22252 144/300 Flush
22253 145/300 Write block 81 len 16384 data: 2 2 2 2
22254 146/300 Flush
22255 147/300 Write block 105 len 20480 data: 3 2 3 4 4
22256 148/300 Read block 4074 len 24576
22257 149/300 Write block 2395 len 24576 data: 5 6 6 5 4 5
22258 150/300 Read block 1377 len 8192
22259 151/300 Write block 3076 len 20480 data: 2 2 3 2 2
22260 152/300 Write block 2846 len 12288 data: 8 8 6
22261 153/300 Read block 228 len 4096
22262 154/300 Read block 2024 len 16384
22263 155/300 Write block 3754 len 24576 data: 7 7 6 6 5 6
22264 156/300 Write block 603 len 28672 data: 6 5 4 5 8 9 8
22265 157/300 Flush
22266 158/300 Write block 2354 len 24576 data: 7 7 8 7 6 5
22267 159/300 Write block 4483 len 32768 data: 5 5 5 5 7 8 9 10
22268 160/300 Write block 1474 len 32768 data: 8 5 4 5 5 6 6 6
22269 161/300 Read block 818 len 36864
22270 162/300 Write block 3735 len 12288 data: 5 5 5
22271 163/300 Read block 2010 len 12288
22272 164/300 Read block 2389 len 12288
22273 165/300 Flush
22274 166/300 Read block 4140 len 12288
22275 167/300 Read block 3404 len 12288
22276 168/300 Write block 4558 len 24576 data: 5 4 3 4 4 5
22277 169/300 Write block 3184 len 4096 data: 4
22278 170/300 Read block 4539 len 40960
22279 171/300 Write block 3703 len 16384 data: 4 5 5 5
22280 172/300 Read block 1889 len 32768
22281 173/300 Read block 3734 len 12288
22282 174/300 Write block 2890 len 20480 data: 4 4 4 4 3
22283 175/300 Write block 516 len 36864 data: 6 8 8 7 7 7 8 9 7
22284 176/300 Read block 4624 len 20480
22285 177/300 Read block 3781 len 24576
22286 178/300 Write block 3626 len 4096 data: 3
22287 179/300 Read block 2162 len 36864
22288 180/300 Read block 1223 len 32768
22289 181/300 Read block 2852 len 4096
22290 182/300 Read block 781 len 36864
22291 183/300 Read block 2732 len 28672
22292 184/300 Read block 922 len 20480
22293 185/300 Read block 4514 len 24576
22294 186/300 Read block 3111 len 16384
22295 187/300 Read block 2413 len 32768
22296 188/300 Write block 2525 len 32768 data: 4 3 2 2 3 3 3 5
22297 189/300 Read block 1908 len 40960
22298 190/300 Write block 717 len 8192 data: 6 6
22299 191/300 Read block 3959 len 8192
22300 192/300 Write block 2851 len 8192 data: 3 2
22301 193/300 Read block 4244 len 32768
22302 194/300 Flush
22303 195/300 Write block 594 len 36864 data: 9 8 6 6 5 4 6 6 6
22304 196/300 Write block 3235 len 16384 data: 8 7 7 9
22305 197/300 Write block 3784 len 8192 data: 4 4
22306 198/300 Read block 3904 len 20480
22307 199/300 Write block 2594 len 4096 data: 7
22308 200/300 Write block 179 len 4096 data: 10
22309 201/300 Read block 1261 len 8192
22310 202/300 Read block 2347 len 32768
22311 203/300 Read block 4359 len 40960
22312 204/300 Write block 4147 len 12288 data: 6 7 7
22313 205/300 Read block 1226 len 40960
22314 206/300 Read block 583 len 36864
22315 207/300 Read block 775 len 8192
22316 208/300 Read block 4150 len 24576
22317 209/300 Write block 2325 len 8192 data: 8 8
22318 210/300 Flush
22319 211/300 Flush
22320 212/300 Write block 2059 len 20480 data: 4 4 4 5 2
22321 213/300 Read block 3896 len 28672
22322 214/300 Read block 638 len 20480
22323 215/300 Read block 3027 len 40960
22324 216/300 Read block 4894 len 28672
22325 217/300 Read block 2267 len 40960
22326 218/300 Write block 4922 len 28672 data: 5 6 7 7 6 5 6
22327 219/300 Read block 4058 len 12288
22328 220/300 Flush
22329 221/300 Write block 4178 len 32768 data: 5 5 5 4 3 3 3 4
22330 222/300 Write block 1567 len 16384 data: 5 5 6 6
22331 223/300 Write block 824 len 40960 data: 9 8 9 7 7 6 6 6 6 4
22332 224/300 Read block 3125 len 8192
22333 225/300 Read block 4881 len 20480
22334 226/300 Read block 4881 len 20480
22335 227/300 Read block 667 len 24576
22336 228/300 Write block 3250 len 36864 data: 6 5 5 5 7 9 7 8 7
22337 229/300 Read block 2135 len 8192
22338 230/300 Read block 2863 len 8192
22339 231/300 Write block 3440 len 32768 data: 3 3 5 6 6 7 9 9
22340 232/300 Read block 2687 len 16384
22341 233/300 Read block 4100 len 28672
22342 234/300 Read block 2917 len 4096
22343 235/300 Read block 4777 len 4096
22344 236/300 Write block 2890 len 12288 data: 5 5 5
22345 237/300 Write block 2601 len 24576 data: 7 7 7 4 4 4
22346 238/300 Read block 3684 len 16384
22347 239/300 Write block 1576 len 12288 data: 4 5 6
22348 240/300 Read block 3766 len 20480
22349 241/300 Write block 2758 len 40960 data: 4 5 5 6 6 6 5 4 4 3
22350 242/300 Read block 1035 len 8192
22351 243/300 Write block 967 len 12288 data: 4 4 7
22352 244/300 Read block 1287 len 40960
22353 245/300 Read block 4572 len 8192
22354 246/300 Write block 4923 len 16384 data: 7 8 8 7
22355 247/300 Read block 1 len 24576
22356 248/300 Write block 307 len 40960 data: 4 4 4 5 4 4 4 4 4 6
22357 249/300 Read block 4335 len 28672
22358 250/300 Read block 1548 len 32768
22359 251/300 Flush
22360 252/300 Read block 827 len 16384
22361 253/300 Read block 2194 len 24576
22362 254/300 Read block 4872 len 20480
22363 255/300 Read block 1879 len 8192
22364 256/300 Write block 3886 len 12288 data: 3 4 5
22365 257/300 Read block 2366 len 12288
22366 258/300 Write block 1716 len 24576 data: 8 7 7 5 3 4
22367 259/300 Write block 3090 len 36864 data: 5 3 4 5 6 5 5 6 7
22368 260/300 Write block 547 len 20480 data: 4 4 4 4 5
22369 261/300 Read block 4071 len 32768
22370 262/300 Write block 542 len 12288 data: 4 4 4
22371 263/300 Write block 4190 len 40960 data: 3 4 5 5 4 4 5 6 6 6
22372 264/300 Read block 3309 len 16384
22373 265/300 Read block 2762 len 40960
22374 266/300 Flush
22375 267/300 Write block 471 len 40960 data: 5 6 5 5 4 4 4 4 3 4
22376 268/300 Write block 1077 len 24576 data: 6 6 5 6 7 8
22377 269/300 Read block 2248 len 24576
22378 270/300 Read block 955 len 8192
22379 271/300 Read block 2290 len 28672
22380 272/300 Flush
22381 273/300 Flush
22382 274/300 Read block 2617 len 12288
22383 275/300 Flush
22384 276/300 Read block 2581 len 40960
22385 277/300 Write block 1300 len 12288 data: 4 5 6
22386 278/300 Flush
22387 279/300 Read block 1261 len 20480
22388 280/300 Write block 2199 len 8192 data: 11 11
22389 281/300 Flush
22390 282/300 Write block 3924 len 32768 data: 8 8 7 5 4 3 4 3
22391 283/300 Write block 1890 len 4096 data: 7
22392 284/300 Write block 2078 len 8192 data: 4 4
22393 285/300 Read block 2019 len 12288
22394 286/300 Write block 4411 len 16384 data: 7 7 6 6
22395 287/300 Flush
22396 288/300 Read block 2574 len 8192
22397 289/300 Read block 4308 len 40960
22398 290/300 Read block 404 len 28672
22399 291/300 Write block 4090 len 8192 data: 4 4
22400 292/300 Read block 2001 len 12288
22401 293/300 Write block 962 len 4096 data: 4
22402 294/300 Flush
22403 295/300 Flush
22404 296/300 Write block 2444 len 20480 data: 5 5 7 7 6
22405 297/300 Write block 3141 len 28672 data: 5 5 5 5 7 6 6
22406 298/300 Read block 4937 len 28672
22407 299/300 Read block 930 len 36864
22408 300/300 Write block 1917 len 36864 data: 7 6 6 6 6 4 4 5 5
22409 Main task has work to do, go find it
22410 got dsc Start(2)
22411 start 2
22412 Feb 27 21:49:07.138 INFO request completed, latency_us: 249, response_code: 204, uri: /start/cid/2, method: POST, req_id: 0bb03689-0f96-45f2-ac36-e6696dfb03a8, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
22413 Replay: started 2, returned:Ok(())
22414 ----------------------------------------------------------------
22415 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
22416 States: Active Active Offline
22417 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
22418 7537 Acked 8536 Read 4 Done Done New true
22419 7538 Acked 8537 Read 8 Done Done New true
22420 7539 Acked 8538 Read 5 Done Done New false
22421 7540 Acked 8539 Flush 0 Done Done New false
22422 7541 Acked 8540 Read 1 Done Done New false
22423 7542 Acked 8541 Read 7 Done Done New false
22424 7543 Acked 8542 Write 6 Done Done New false
22425 7544 Acked 8543 Read 4 Done Done New false
22426 7545 Acked 8544 Read 1 Done Done New false
22427 7546 Acked 8545 Read 1 Done Done New false
22428 7547 Acked 8546 Flush 0 Done Done New false
22429 7548 Acked 8547 Read 3 Done Done New false
22430 7549 Acked 8548 Read 1 Done Done New false
22431 7550 Acked 8549 Write 3 Done Done New false
22432 7551 Acked 8550 Write 5 Done Done New false
22433 7552 Acked 8551 Read 3 Done Done New false
22434 7553 Acked 8552 Write 8 Done Done New false
22435 7554 Acked 8553 Write 4 Done Done New false
22436 7555 Acked 8554 Write 1 Done Done New false
22437 7556 Acked 8555 Write 5 Done Done New false
22438 7557 Acked 8556 Read 6 Done Done New false
22439 7558 Acked 8557 Write 4 Done Done New false
22440 7559 Acked 8558 Read 2 Done Done New false
22441 7560 Acked 8559 Write 6 Done Done New false
22442 7561 Acked 8560 Read 4 Done Done New false
22443 7562 Acked 8561 Write 9 Done Done New false
22444 7563 Acked 8562 Write 8 Done Done New false
22445 7564 Acked 8563 Flush 0 Done Done New false
22446 7565 Acked 8564 Write 1 Done Done New false
22447 7566 Acked 8565 Read 7 Done Done New false
22448 7567 Acked 8566 Write 9 Done Done New false
22449 7568 Acked 8567 Write 10 Done Done New false
22450 7569 Acked 8568 Write 1 Done Done New false
22451 7570 Acked 8569 Write 4 Done Done New false
22452 7571 Acked 8570 Read 4 Done Done New false
22453 7572 Acked 8571 Write 1 Done Done New false
22454 7573 Acked 8572 Write 7 Done Done New false
22455 7574 Acked 8573 Write 8 Done Done New false
22456 7575 Acked 8574 Read 2 Done Done New false
22457 7576 Acked 8575 Read 7 Done Done New false
22458 7577 Acked 8576 Read 9 Done Done New false
22459 7578 Acked 8577 Write 8 Done Done New false
22460 7579 Acked 8578 Read 7 Done Done New false
22461 7580 Acked 8579 Write 6 Done Done New false
22462 7581 Acked 8580 Read 8 Done Done New false
22463 7582 Acked 8581 Write 6 Done Done New false
22464 7583 Acked 8582 Read 3 Done Done New false
22465 7584 Acked 8583 Read 7 Done Done New false
22466 7585 Acked 8584 Write 7 Done Done New false
22467 7586 Acked 8585 Write 10 Done Done New false
22468 7587 Acked 8586 Write 10 Done Done New false
22469 7588 Acked 8587 Flush 0 Done Done New false
22470 7589 Acked 8588 Write 1 Done Done New false
22471 7590 Acked 8589 Read 7 Done Done New false
22472 7591 Acked 8590 Read 2 Done Done New false
22473 7592 Acked 8591 Read 3 Done Done New false
22474 7593 Acked 8592 Write 1 Done Done New false
22475 7594 Acked 8593 Write 5 Done Done New false
22476 7595 Acked 8594 Write 3 Done Done New false
22477 7596 Acked 8595 Write 3 Done Done New false
22478 7597 Acked 8596 Write 9 Done Done New false
22479 7598 Acked 8597 Read 6 Done Done New false
22480 7599 Acked 8598 Read 1 Done Done New false
22481 7600 Acked 8599 Read 7 Done Done New false
22482 7601 Acked 8600 Write 7 Done Done New false
22483 7602 Acked 8601 Write 9 Done Done New false
22484 7603 Acked 8602 Write 9 Done Done New false
22485 7604 Acked 8603 Write 9 Done Done New false
22486 7605 Acked 8604 Write 1 Done Done New false
22487 7606 Acked 8605 Write 8 Done Done New false
22488 7607 Acked 8606 Read 4 Done Done New false
22489 7608 Acked 8607 Read 10 Done Done New false
22490 7609 Acked 8608 Read 6 Done Done New false
22491 7610 Acked 8609 Read 1 Done Done New false
22492 7611 Acked 8610 Write 4 Done Done New false
22493 7612 Acked 8611 Read 1 Done Done New false
22494 7613 Acked 8612 Write 3 Done Done New false
22495 7614 Acked 8613 Write 3 Done Done New false
22496 7615 Acked 8614 Flush 0 Done Done New false
22497 7616 Acked 8615 Write 5 Done Done New false
22498 7617 Acked 8616 Read 6 Done Done New false
22499 7618 Acked 8617 Write 7 Done Done New false
22500 7619 Acked 8618 Read 8 Done Done New false
22501 7620 Acked 8619 Flush 0 Done Done New false
22502 7621 Acked 8620 Read 1 Done Done New false
22503 7622 Acked 8621 Read 8 Done Done New false
22504 7623 Acked 8622 Write 10 Done Done New false
22505 7624 Acked 8623 Flush 0 Done Done New false
22506 7625 Acked 8624 Write 6 Done Done New false
22507 7626 Acked 8625 Write 2 Done Done New false
22508 7627 Acked 8626 Read 6 Done Done New false
22509 7628 Acked 8627 Write 8 Done Done New false
22510 7629 Acked 8628 Flush 0 Done Done New false
22511 7630 Acked 8629 Read 3 Done Done New false
22512 7631 Acked 8630 Write 9 Done Done New false
22513 7632 Acked 8631 Write 1 Done Done New false
22514 7633 Acked 8632 Write 1 Done Done New false
22515 7634 Acked 8633 Write 3 Done Done New false
22516 7635 Acked 8634 Write 8 Done Done New false
22517 7636 Acked 8635 Write 10 Done Done New false
22518 7637 Acked 8636 Read 7 Done Done New false
22519 7638 Acked 8637 Write 8 Done Done New false
22520 7639 Acked 8638 Read 1 Done Done New false
22521 7640 Acked 8639 Read 8 Done Done New false
22522 7641 Acked 8640 Write 1 Done Done New false
22523 7642 Acked 8641 Read 10 Done Done New false
22524 7643 Acked 8642 Read 1 Done Done New false
22525 7644 Acked 8643 Read 1 Done Done New false
22526 7645 Acked 8644 Read 9 Done Done New false
22527 7646 Acked 8645 Read 3 Done Done New false
22528 7647 Acked 8646 Write 2 Done Done New false
22529 7648 Acked 8647 Flush 0 Done Done New false
22530 7649 Acked 8648 Write 2 Done Done New false
22531 7650 Acked 8649 Read 8 Done Done New false
22532 7651 Acked 8650 Read 3 Done Done New false
22533 7652 Acked 8651 Read 4 Done Done New false
22534 7653 Acked 8652 Read 4 Done Done New false
22535 7654 Acked 8653 Read 1 Done Done New false
22536 7655 Acked 8654 Read 8 Done Done New false
22537 7656 Acked 8655 Write 9 Done Done New false
22538 7657 Acked 8656 Read 2 Done Done New false
22539 7658 Acked 8657 Read 10 Done Done New false
22540 7659 Acked 8658 Write 9 Done Done New false
22541 7660 Acked 8659 Write 8 Done Done New false
22542 7661 Acked 8660 Write 2 Done Done New false
22543 7662 Acked 8661 Read 5 Done Done New false
22544 7663 Acked 8662 Read 10 Done Done New false
22545 7664 Acked 8663 Write 7 Done Done New false
22546 7665 Acked 8664 Read 7 Done Done New false
22547 7666 Acked 8665 Read 7 Done Done New false
22548 7667 Acked 8666 Flush 0 Done Done New false
22549 7668 Acked 8667 Read 7 Done Done New false
22550 7669 Acked 8668 Write 5 Done Done New false
22551 7670 Acked 8669 Flush 0 Done Done New false
22552 7671 Acked 8670 Read 4 Done Done New false
22553 7672 Acked 8671 Read 3 Done Done New false
22554 7673 Acked 8672 Write 6 Done Done New false
22555 7674 Acked 8673 Read 5 Done Done New false
22556 7675 Acked 8674 Write 7 Done Done New false
22557 7676 Acked 8675 Flush 0 Done Done New false
22558 7677 Acked 8676 Read 3 Done Done New false
22559 7678 Acked 8677 Read 3 Done Done New false
22560 7679 Acked 8678 Write 2 Done Done New false
22561 7680 Acked 8679 Flush 0 Done Done New false
22562 7681 Acked 8680 Write 4 Done Done New false
22563 7682 Acked 8681 Flush 0 Done Done New false
22564 7683 Acked 8682 Write 5 Done Done New false
22565 7684 Acked 8683 Read 6 Done Done New false
22566 7685 Acked 8684 Write 6 Done Done New false
22567 7686 Acked 8685 Read 2 Done Done New false
22568 7687 Acked 8686 Write 5 Done Done New false
22569 7688 Acked 8687 Write 3 Done Done New false
22570 7689 Acked 8688 Read 1 Done Done New false
22571 7690 Acked 8689 Read 4 Done Done New false
22572 7691 Acked 8690 Write 6 Done Done New false
22573 7692 Acked 8691 Write 7 Done Done New false
22574 7693 Acked 8692 Flush 0 Done Done New false
22575 7694 Acked 8693 Write 6 Done Done New false
22576 7695 Acked 8694 Write 8 Done Done New false
22577 7696 Acked 8695 Write 8 Done Done New false
22578 7697 Acked 8696 Read 9 Done Done New false
22579 7698 Acked 8697 Flush 0 Done Done New false
22580 7699 Acked 8698 Write 3 Done Done New false
22581 7700 Acked 8699 Read 3 Done Done New false
22582 7701 Acked 8700 Read 3 Done Done New false
22583 7702 Acked 8701 Flush 0 Done Done New false
22584 7703 Acked 8702 Read 3 Done Done New false
22585 7704 Acked 8703 Read 3 Done Done New false
22586 7705 Acked 8704 Write 6 Done Done New false
22587 7706 Acked 8705 Write 1 Done Done New false
22588 7707 Acked 8706 Read 10 Done Done New false
22589 7708 Acked 8707 Write 4 Done Done New false
22590 7709 Acked 8708 Read 8 Done Done New false
22591 7710 Acked 8709 Read 3 Done Done New false
22592 7711 Acked 8710 Write 5 Done Done New false
22593 7712 Acked 8711 Write 9 Done Done New false
22594 7713 Acked 8712 Read 5 Done Done New false
22595 7714 Acked 8713 Read 6 Done Done New false
22596 7715 Acked 8714 Write 1 Done Done New false
22597 7716 Acked 8715 Read 9 Done Done New false
22598 7717 Acked 8716 Read 8 Done Done New false
22599 7718 Acked 8717 Read 1 Done Done New false
22600 7719 Acked 8718 Read 9 Done Done New false
22601 7720 Acked 8719 Read 7 Done Done New false
22602 7721 Acked 8720 Read 5 Done Done New false
22603 7722 Acked 8721 Read 6 Done Done New false
22604 7723 Acked 8722 Read 4 Done Done New false
22605 7724 Acked 8723 Read 8 Done Done New false
22606 7725 Acked 8724 Write 8 Done Done New false
22607 7726 Acked 8725 Read 10 Done Done New false
22608 7727 Acked 8726 Write 2 Done Done New false
22609 7728 Acked 8727 Read 2 Done Done New false
22610 7729 Acked 8728 Write 2 Done Done New false
22611 7730 Acked 8729 Read 8 Done Done New false
22612 7731 Acked 8730 Flush 0 Done Done New false
22613 7732 Acked 8731 Write 9 Done Done New false
22614 7733 Acked 8732 Write 4 Done Done New false
22615 7734 Acked 8733 Write 2 Done Done New false
22616 7735 Acked 8734 Read 5 Done Done New false
22617 7736 Acked 8735 Write 1 Done Done New false
22618 7737 Acked 8736 Write 1 Done Done New false
22619 7738 Acked 8737 Read 2 Done Done New false
22620 7739 Acked 8738 Read 8 Done Done New false
22621 7740 Acked 8739 Read 10 Done Done New false
22622 7741 Acked 8740 Write 3 Done Done New false
22623 7742 Acked 8741 Read 10 Done Done New false
22624 7743 Acked 8742 Read 9 Done Done New false
22625 7744 Acked 8743 Read 2 Done Done New false
22626 7745 Acked 8744 Read 6 Done Done New false
22627 7746 Acked 8745 Write 2 Done Done New false
22628 7747 Acked 8746 Flush 0 Done Done New false
22629 7748 Acked 8747 Flush 0 Done Done New false
22630 7749 Acked 8748 Write 5 Done Done New false
22631 7750 Acked 8749 Read 7 Done Done New false
22632 7751 Acked 8750 Read 5 Done Done New false
22633 7752 Acked 8751 Read 10 Done Done New false
22634 7753 Acked 8752 Read 7 Done Done New false
22635 7754 Acked 8753 Read 10 Done Done New false
22636 7755 Acked 8754 Write 7 Done Done New false
22637 7756 Acked 8755 Read 3 Done Done New false
22638 7757 Acked 8756 Flush 0 Done Done New false
22639 7758 Acked 8757 Write 8 Done Done New false
22640 7759 Acked 8758 Write 4 Done Done New false
22641 7760 Acked 8759 Write 10 Done Done New false
22642 7761 Acked 8760 Read 2 Done Done New false
22643 7762 Acked 8761 Read 5 Done Done New false
22644 7763 Acked 8762 Read 5 Done Done New false
22645 7764 Acked 8763 Read 6 Done Done New false
22646 7765 Acked 8764 Write 9 Done Done New false
22647 7766 Acked 8765 Read 2 Done Done New false
22648 7767 Acked 8766 Read 2 Done Done New false
22649 7768 Acked 8767 Write 8 Done Done New false
22650 7769 Acked 8768 Read 4 Done Done New false
22651 7770 Acked 8769 Read 7 Done Done New false
22652 7771 Acked 8770 Read 1 Done Done New false
22653 7772 Acked 8771 Read 1 Done Done New false
22654 7773 Acked 8772 Write 3 Done Done New false
22655 7774 Acked 8773 Write 6 Done Done New false
22656 7775 Acked 8774 Read 4 Done Done New false
22657 7776 Acked 8775 Write 3 Done Done New false
22658 7777 Acked 8776 Read 5 Done Done New false
22659 7778 Acked 8777 Write 10 Done Done New false
22660 7779 Acked 8778 Read 2 Done Done New false
22661 7780 Acked 8779 Write 3 Done Done New false
22662 7781 Acked 8780 Read 10 Done Done New false
22663 7782 Acked 8781 Read 2 Done Done New false
22664 7783 Acked 8782 Write 4 Done Done New false
22665 7784 Acked 8783 Read 6 Done Done New false
22666 7785 Acked 8784 Write 10 Done Done New false
22667 7786 Acked 8785 Read 7 Done Done New false
22668 7787 Acked 8786 Read 8 Done Done New false
22669 7788 Acked 8787 Flush 0 Done Done New false
22670 7789 Acked 8788 Read 4 Done Done New false
22671 7790 Acked 8789 Read 6 Done Done New false
22672 7791 Acked 8790 Read 5 Done Done New false
22673 7792 Acked 8791 Read 2 Done Done New false
22674 7793 Acked 8792 Write 3 Done Done New false
22675 7794 Acked 8793 Read 3 Done Done New false
22676 7795 Acked 8794 Write 6 Done Done New false
22677 7796 Acked 8795 Write 9 Done Done New false
22678 7797 Acked 8796 Write 5 Done Done New false
22679 7798 Acked 8797 Read 8 Done Done New false
22680 7799 Acked 8798 Write 3 Done Done New false
22681 7800 Acked 8799 Write 10 Done Done New false
22682 7801 Acked 8800 Read 4 Done Done New false
22683 7802 Acked 8801 Read 10 Done Done New false
22684 7803 Acked 8802 Flush 0 Done Done New false
22685 7804 Acked 8803 Write 10 Done Done New false
22686 7805 Acked 8804 Write 6 Done Done New false
22687 7806 Acked 8805 Read 6 Done Done New false
22688 7807 Acked 8806 Read 2 Done Done New false
22689 7808 Acked 8807 Read 7 Done Done New false
22690 7809 Acked 8808 Flush 0 Done Done New false
22691 7810 Acked 8809 Flush 0 Done Done New false
22692 7811 Acked 8810 Read 3 Done Done New false
22693 7812 Acked 8811 Flush 0 Done Done New false
22694 7813 Acked 8812 Read 10 Done Done New false
22695 7814 Acked 8813 Write 3 Done Done New false
22696 7815 Acked 8814 Flush 0 Done Done New false
22697 7816 Acked 8815 Read 5 Done Done New false
22698 7817 Acked 8816 Write 2 Done Done New false
22699 7818 Acked 8817 Flush 0 Done Done New false
22700 7819 Acked 8818 Write 8 Done Done New false
22701 7820 Acked 8819 Write 1 Done Done New false
22702 7821 Acked 8820 Write 2 Done Done New false
22703 7822 Acked 8821 Read 3 Done Done New false
22704 7823 Acked 8822 Write 4 Done Done New false
22705 7824 Acked 8823 Flush 0 Done Done New false
22706 7825 Acked 8824 Read 2 Done Done New false
22707 7826 Acked 8825 Read 10 Done Done New false
22708 7827 Acked 8826 Read 7 Done Done New false
22709 7828 Acked 8827 Write 2 Done Done New false
22710 7829 Acked 8828 Read 3 Done Done New false
22711 7830 Acked 8829 Write 1 Done Done New false
22712 7831 Acked 8830 Flush 0 Done Done New false
22713 7832 Acked 8831 Flush 0 Done Done New false
22714 7833 Acked 8832 Write 5 Done Done New false
22715 7834 Acked 8833 Write 7 Done Done New false
22716 7835 Acked 8834 Read 7 Done Done New false
22717 7836 Acked 8835 Read 9 Done Done New false
22718 7837 Acked 8836 Write 9 Done Done New false
22719 STATES DS:0 DS:1 DS:2 TOTAL
22720 New 0 0 301 301
22721 Sent 0 0 0 0
22722 Done 301 301 0 602
22723 Skipped 0 0 0 0
22724 Error 0 0 0 0
22725 Last Flush: 8831 8831 8535
22726 Downstairs last five completed: 8535 8534 8533 8532 8531
22727 Upstairs last five completed: 7837 7836 7835 7834 7833
22728 CLIENT: Up:0 ds:301 act:2
227292024-02-27T21:49:07.812ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
227302024-02-27T21:49:07.812ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
227312024-02-27T21:49:07.812ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
227322024-02-27T21:49:07.812ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
227332024-02-27T21:49:07.812ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
227342024-02-27T21:49:07.812ZINFOcrucible: [2] client re-new 302 jobs since flush 8535 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
227352024-02-27T21:49:07.812ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
22736 [8830] got start action
22737 [8830] I am going to restart
22738 Starting downstairs at port 8830
22739 Make output file at "/tmp/dsc/downstairs-8830.txt"
22740 [8830][2] reports Starting
227412024-02-27T21:49:08.813ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
227422024-02-27T21:49:08.813ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
227432024-02-27T21:49:08.814ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
227442024-02-27T21:49:08.814ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
227452024-02-27T21:49:08.814ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
227462024-02-27T21:49:08.814ZINFOcrucible: [2] client re-new 302 jobs since flush 8535 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
227472024-02-27T21:49:08.814ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
22748 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1193)
22749 [8830][2] reports Running
227502024-02-27T21:49:09.815ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
227512024-02-27T21:49:09.815ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
227522024-02-27T21:49:09.816ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
227532024-02-27T21:49:09.816ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
227542024-02-27T21:49:09.817ZINFOcrucible: send last flush ID to this DS: 8535 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
227552024-02-27T21:49:09.817ZINFOcrucible: Replied this last flush ID: 8535 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
227562024-02-27T21:49:09.817ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
227572024-02-27T21:49:09.817ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
227582024-02-27T21:49:09.817ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
227592024-02-27T21:49:09.817ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
22760 ----------------------------------------------------------------
22761 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22762 Downstairs last five completed: 8837 8836 8835 8834 8833
22763 Upstairs last five completed: 7838 7837 7836 7835 7834
22764 CLIENT: Up:0 ds:0 act:3
22765 Replay: All jobs finished, all DS active.
22766 Main task has work to do, go find it
22767 got dsc Stop(1)
22768 stop 1
22769 Feb 27 21:49:11.142 INFO request completed[8820] Got stop action so:false kr:false
22770 , latency_us: 366, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 99bd2acb-bbd3-4626-9ef2-7fffd43cbed6, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
227712024-02-27T21:49:11.142ZWARNcrucible: downstairs disconnected = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
227722024-02-27T21:49:11.142ZWARNcrucible: client task is sending Done(Finished) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
227732024-02-27T21:49:11.142ZWARNcrucible: downstairs task for 1 stopped due to Finished session_id = e6a5a974-925a-494f-8845-facc441c2690
22774 Feb 27 21:49:11.142 INFO request completed, latency_us{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:49:11.142979444Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"client":"1","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
22775 : 120, response_code: 200, uri: /state/cid/1, method: GET, req_id: 18e9080f-cf36-4359-b942-1111ea2dc057, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
227762024-02-27T21:49:11.143ZINFOcrucible: [1] client re-new 0 jobs since flush 8837 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
227772024-02-27T21:49:11.143ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
22778 [8820] Exited with: ExitStatus(unix_wait_status(9))
22779 [8820][1] reports Exit
227802024-02-27T21:49:12.144ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
227812024-02-27T21:49:12.144ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
227822024-02-27T21:49:12.144ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
227832024-02-27T21:49:12.144ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
227842024-02-27T21:49:12.144ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
227852024-02-27T21:49:12.144ZINFOcrucible: [1] client re-new 0 jobs since flush 8837 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
227862024-02-27T21:49:12.144ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
227872024-02-27T21:49:13.145ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
227882024-02-27T21:49:13.145ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
227892024-02-27T21:49:13.145ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
227902024-02-27T21:49:13.145ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
227912024-02-27T21:49:13.145ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
227922024-02-27T21:49:13.145ZINFOcrucible: [1] client re-new 0 jobs since flush 8837 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
227932024-02-27T21:49:13.145ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
227942024-02-27T21:49:14.146ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
227952024-02-27T21:49:14.146ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
227962024-02-27T21:49:14.146ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
227972024-02-27T21:49:14.146ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
227982024-02-27T21:49:14.146ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
227992024-02-27T21:49:14.146ZINFOcrucible: [1] client re-new 0 jobs since flush 8837 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
228002024-02-27T21:49:14.146ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
22801 Feb 27 21:49:15.145 INFO request completed, latency_us: 228, response_code: 200, uri: /state/cid/1, method: GET, req_id: ec553038-edbb-4116-8689-5db0f0352fa8, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
22802 001/300 Write block 3246 len 36864 data: 4 5 4 7 7 6 6 6 8
22803 002/300 Read block 2169 len 4096
228042024-02-27T21:49:15.147ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
228052024-02-27T21:49:15.147ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
228062024-02-27T21:49:15.147ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
228072024-02-27T21:49:15.147ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
228082024-02-27T21:49:15.147ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
228092024-02-27T21:49:15.147ZINFOcrucible: [1] client re-new 2 jobs since flush 8837 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
228102024-02-27T21:49:15.147ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
22811 003/300 Write block 3123 len 32768 data: 5 6 6 5 4 4 3 3
22812 004/300 Read block 187 len 40960
22813 005/300 Write block 3602 len 40960 data: 6 5 4 3 5 5 4 6 6 6
22814 006/300 Write block 2507 len 36864 data: 13 13 8 4 5 4 4 3 4
22815 007/300 Write block 4710 len 8192 data: 5 4
22816 008/300 Read block 4041 len 16384
22817 009/300 Write block 3643 len 40960 data: 6 6 4 4 4 4 4 4 4 5
22818 010/300 Read block 525 len 8192
22819 011/300 Read block 1161 len 16384
22820 012/300 Read block 2389 len 32768
22821 013/300 Write block 920 len 16384 data: 2 2 5 5
22822 014/300 Write block 1578 len 4096 data: 7
22823 015/300 Flush
22824 016/300 Write block 855 len 28672 data: 2 3 3 3 3 4 4
22825 017/300 Write block 1365 len 40960 data: 8 10 10 9 5 5 5 5 5 4
22826 018/300 Read block 2654 len 32768
22827 019/300 Flush
22828 020/300 Read block 3924 len 8192
22829 021/300 Read block 3568 len 28672
22830 022/300 Read block 3922 len 36864
22831 023/300 Write block 650 len 16384 data: 5 4 5 4
22832 024/300 Read block 4700 len 24576
22833 025/300 Write block 3202 len 4096 data: 5
22834 026/300 Read block 456 len 28672
22835 027/300 Write block 2939 len 24576 data: 5 4 4 6 7 6
22836 028/300 Write block 19 len 32768 data: 6 6 5 6 9 9 9 9
22837 029/300 Flush
22838 030/300 Write block 4330 len 40960 data: 5 4 4 4 4 4 5 5 5 5
22839 031/300 Write block 3826 len 40960 data: 4 3 3 3 4 4 4 3 3 4
22840 032/300 Write block 4116 len 20480 data: 3 3 3 4 5
22841 033/300 Read block 1061 len 28672
22842 034/300 Read block 2421 len 16384
22843 035/300 Write block 3441 len 16384 data: 4 6 7 7
22844 036/300 Read block 4286 len 12288
22845 037/300 Read block 740 len 8192
22846 038/300 Write block 2113 len 12288 data: 3 4 4
22847 039/300 Read block 565 len 28672
22848 040/300 Flush
22849 041/300 Read block 3197 len 24576
22850 042/300 Read block 1845 len 8192
22851 043/300 Write block 4088 len 28672 data: 4 5 5 5 7 6 4
22852 044/300 Read block 192 len 16384
22853 045/300 Read block 2756 len 36864
22854 046/300 Read block 827 len 28672
22855 047/300 Read block 4151 len 24576
22856 048/300 Read block 4870 len 20480
22857 049/300 Read block 2510 len 28672
22858 050/300 Write block 7 len 20480 data: 4 4 5 6 6
22859 051/300 Write block 1357 len 32768 data: 4 4 5 4 5 6 7 8
22860 052/300 Write block 4820 len 16384 data: 5 4 3 5
22861 053/300 Read block 1562 len 16384
22862 054/300 Write block 1683 len 16384 data: 5 6 6 6
22863 055/300 Write block 4304 len 16384 data: 5 4 5 3
22864 056/300 Flush
22865 057/300 Write block 1407 len 12288 data: 7 7 6
22866 058/300 Write block 1830 len 20480 data: 8 9 9 7 8
22867 059/300 Write block 2089 len 12288 data: 5 4 3
22868 060/300 Read block 1127 len 24576
22869 061/300 Flush
22870 062/300 Read block 2596 len 32768
22871 063/300 Write block 1594 len 40960 data: 8 7 6 5 6 7 6 6 5 6
22872 064/300 Read block 218 len 16384
22873 065/300 Read block 638 len 40960
22874 066/300 Read block 3623 len 8192
22875 067/300 Read block 818 len 12288
22876 068/300 Write block 378 len 20480 data: 4 5 4 3 3
22877 069/300 Read block 2886 len 16384
22878 070/300 Read block 4404 len 12288
22879 071/300 Read block 2395 len 4096
22880 072/300 Read block 3236 len 28672
22881 073/300 Read block 4567 len 20480
22882 074/300 Write block 3027 len 4096 data: 7
22883 075/300 Flush
22884 076/300 Write block 1240 len 8192 data: 4 3
22885 077/300 Read block 4802 len 16384
22886 078/300 Read block 3412 len 24576
22887 079/300 Flush
22888 080/300 Write block 4381 len 28672 data: 6 6 7 4 6 6 5
22889 081/300 Read block 4560 len 16384
22890 082/300 Write block 821 len 36864 data: 4 7 8 10 9 10 8 8 7
22891 083/300 Write block 731 len 16384 data: 6 7 6 6
22892 084/300 Read block 4963 len 40960
22893 085/300 Write block 4506 len 28672 data: 6 6 5 5 5 8 10
22894 086/300 Write block 2329 len 8192 data: 4 5
22895 087/300 Flush
22896 088/300 Read block 1626 len 4096
22897 089/300 Read block 1782 len 20480
22898 090/300 Read block 585 len 28672
22899 091/300 Write block 3549 len 4096 data: 5
22900 092/300 Flush
22901 093/300 Read block 890 len 20480
22902 094/300 Read block 1942 len 12288
22903 095/300 Read block 1556 len 32768
22904 096/300 Write block 2942 len 36864 data: 7 8 7 5 4 4 5 5 5
22905 097/300 Write block 4343 len 32768 data: 3 3 6 5 5 5 6 6
22906 098/300 Write block 203 len 36864 data: 5 5 5 7 8 7 7 6 6
22907 099/300 Read block 1058 len 12288
22908 100/300 Read block 583 len 20480
22909 101/300 Read block 4179 len 28672
22910 102/300 Read block 4180 len 8192
22911 103/300 Flush
22912 104/300 Write block 1924 len 28672 data: 6 6 6 6 8 8 8
22913 105/300 Read block 3915 len 24576
22914 106/300 Write block 1525 len 8192 data: 3 3
22915 107/300 Write block 1627 len 36864 data: 5 4 3 3 4 3 3 5 5
22916 108/300 Read block 1272 len 4096
22917 109/300 Read block 1804 len 24576
22918 110/300 Write block 2906 len 8192 data: 7 6
22919 111/300 Write block 2204 len 36864 data: 7 7 10 9 7 6 6 6 6
22920 112/300 Write block 1742 len 28672 data: 2 3 3 4 4 4 5
22921 113/300 Read block 3174 len 12288
22922 114/300 Read block 4048 len 20480
22923 115/300 Read block 3641 len 36864
22924 116/300 Flush
22925 117/300 Write block 4480 len 28672 data: 5 6 4 6 6 6 6
22926 118/300 Read block 3866 len 36864
22927 119/300 Read block 283 len 4096
22928 120/300 Read block 4283 len 40960
22929 121/300 Read block 1813 len 32768
22930 122/300 Write block 1326 len 12288 data: 9 7 6
22931 123/300 Write block 1579 len 28672 data: 7 6 6 5 6 7 7
22932 124/300 Write block 2179 len 16384 data: 4 3 4 6
22933 125/300 Read block 4443 len 36864
22934 126/300 Read block 40 len 4096
22935 127/300 Read block 3662 len 28672
22936 128/300 Read block 1684 len 32768
22937 129/300 Write block 703 len 40960 data: 4 4 6 7 6 5 6 5 7 6
22938 130/300 Write block 1148 len 24576 data: 7 6 6 5 3 3
22939 131/300 Flush
22940 132/300 Read block 2882 len 32768
22941 133/300 Read block 4849 len 40960
22942 134/300 Write block 772 len 28672 data: 4 4 3 3 3 2 2
22943 135/300 Write block 15 len 32768 data: 5 7 7 7 7 7 6 7
22944 136/300 Read block 4610 len 36864
22945 137/300 Write block 4764 len 8192 data: 5 7
22946 138/300 Read block 1871 len 12288
22947 139/300 Read block 4885 len 40960
22948 140/300 Read block 1023 len 4096
22949 141/300 Write block 4436 len 16384 data: 8 8 8 7
22950 142/300 Write block 4183 len 40960 data: 4 4 5 4 4 4 3 4 5 6
22951 143/300 Read block 2753 len 12288
22952 144/300 Write block 3834 len 36864 data: 4 5 5 5 7 7 8 7 8
22953 145/300 Flush
22954 146/300 Read block 1669 len 40960
22955 147/300 Write block 1284 len 36864 data: 2 3 4 5 6 6 5 5 4
22956 148/300 Read block 1856 len 32768
22957 149/300 Flush
22958 150/300 Read block 1474 len 20480
22959 151/300 Read block 2974 len 28672
22960 152/300 Write block 2819 len 8192 data: 10 10
22961 153/300 Write block 1941 len 20480 data: 7 5 5 4 6
22962 154/300 Read block 2540 len 20480
22963 155/300 Flush
22964 156/300 Write block 11 len 24576 data: 7 6 6 5 6 8
22965 157/300 Read block 3611 len 12288
22966 158/300 Write block 1150 len 28672 data: 7 6 4 4 7 7 6
22967 159/300 Write block 1068 len 16384 data: 5 5 5 4
22968 160/300 Read block 4342 len 20480
22969 161/300 Write block 1210 len 20480 data: 6 6 5 4 3
22970 162/300 Read block 3119 len 28672
22971 163/300 Read block 4170 len 20480
22972 164/300 Read block 631 len 24576
22973 165/300 Read block 3221 len 32768
22974 166/300 Read block 1678 len 12288
22975 167/300 Write block 2310 len 28672 data: 8 9 9 9 8 6 6
22976 168/300 Write block 1093 len 32768 data: 5 5 4 3 4 5 6 7
22977 169/300 Read block 4179 len 40960
22978 170/300 Write block 4840 len 40960 data: 4 4 3 4 3 4 5 5 4 3
22979 171/300 Read block 4411 len 32768
22980 172/300 Read block 4461 len 20480
22981 173/300 Write block 4907 len 36864 data: 4 4 4 4 4 4 4 3 3
22982 174/300 Read block 4565 len 8192
22983 175/300 Read block 3133 len 40960
22984 176/300 Read block 469 len 40960
22985 177/300 Flush
22986 178/300 Flush
22987 179/300 Read block 3368 len 20480
22988 180/300 Read block 2908 len 24576
22989 181/300 Read block 1697 len 4096
22990 182/300 Flush
22991 183/300 Write block 2858 len 20480 data: 6 6 5 5 6
22992 184/300 Write block 308 len 36864 data: 5 5 6 5 5 5 5 5 7
22993 185/300 Flush
22994 186/300 Write block 62 len 36864 data: 4 3 3 4 5 5 5 7 6
22995 187/300 Read block 84 len 8192
22996 188/300 Read block 4216 len 12288
22997 189/300 Write block 72 len 12288 data: 6 6 5
22998 190/300 Read block 1804 len 20480
22999 191/300 Write block 3887 len 36864 data: 5 6 5 5 6 3 3 4 6
23000 192/300 Write block 3034 len 36864 data: 4 4 4 5 4 6 6 6 5
23001 193/300 Read block 1795 len 24576
23002 194/300 Write block 2387 len 24576 data: 3 2 3 3 3 4
23003 195/300 Read block 1483 len 40960
23004 196/300 Read block 2870 len 8192
23005 197/300 Read block 4324 len 20480
23006 198/300 Write block 3677 len 24576 data: 7 9 11 7 6 7
23007 199/300 Write block 4503 len 40960 data: 6 6 6 7 7 6 6 6 9 11
23008 200/300 Write block 3175 len 28672 data: 8 8 9 7 5 6 7
23009 201/300 Write block 2356 len 4096 data: 9
23010 202/300 Flush
23011 203/300 Read block 1741 len 12288
23012 204/300 Write block 70 len 32768 data: 7 7 7 7 6 4 3 3
23013 205/300 Read block 836 len 4096
23014 206/300 Write block 300 len 4096 data: 2
23015 207/300 Flush
23016 208/300 Flush
23017 209/300 Write block 1482 len 8192 data: 7 4
23018 210/300 Read block 2628 len 36864
23019 211/300 Write block 1384 len 32768 data: 7 7 7 4 4 3 4 5
23020 212/300 Read block 3883 len 28672
23021 213/300 Read block 108 len 40960
23022 214/300 Read block 1469 len 8192
23023 215/300 Read block 1037 len 32768
23024 216/300 Read block 800 len 16384
23025 217/300 Write block 2163 len 12288 data: 6 5 5
23026 218/300 Write block 2800 len 36864 data: 5 4 6 6 6 6 4 5 4
23027 219/300 Write block 4318 len 28672 data: 7 8 8 8 6 8 8
23028 220/300 Read block 2080 len 32768
23029 221/300 Read block 4353 len 4096
23030 222/300 Write block 2684 len 12288 data: 4 4 4
23031 223/300 Read block 71 len 8192
23032 224/300 Flush
23033 225/300 Write block 3375 len 8192 data: 10 8
23034 226/300 Write block 2566 len 16384 data: 3 4 5 6
23035 227/300 Write block 978 len 20480 data: 6 6 6 5 5
23036 228/300 Read block 2176 len 36864
23037 229/300 Write block 1213 len 28672 data: 5 4 5 6 9 10 10
23038 230/300 Write block 3225 len 24576 data: 5 6 5 6 6 6
23039 231/300 Read block 2680 len 12288
23040 232/300 Write block 2127 len 20480 data: 6 5 5 5 5
23041 233/300 Read block 2791 len 36864
23042 234/300 Read block 1975 len 8192
23043 235/300 Write block 4300 len 32768 data: 7 6 6 7 6 5 6 4
23044 236/300 Read block 1771 len 36864
23045 237/300 Read block 1804 len 32768
23046 238/300 Read block 4039 len 28672
23047 239/300 Read block 523 len 36864
23048 240/300 Write block 2128 len 12288 data: 6 6 6
23049 241/300 Write block 4977 len 12288 data: 3 4 3
23050 242/300 Write block 2242 len 32768 data: 8 8 6 5 7 5 5 4
23051 243/300 Read block 3258 len 40960
23052 244/300 Write block 2866 len 32768 data: 3 3 3 4 6 7 7 5
23053 245/300 Read block 450 len 32768
23054 246/300 Read block 2837 len 32768
23055 247/300 Write block 465 len 28672 data: 3 3 3 4 4 5 6
23056 248/300 Write block 4550 len 24576 data: 13 11 9 7 6 4
23057 249/300 Read block 694 len 40960
23058 250/300 Read block 4306 len 24576
23059 251/300 Read block 587 len 40960
23060 252/300 Read block 343 len 40960
23061 253/300 Read block 1416 len 32768
23062 254/300 Read block 4803 len 4096
23063 255/300 Flush
23064 256/300 Read block 3694 len 20480
23065 257/300 Flush
23066 258/300 Write block 4927 len 4096 data: 6
23067 259/300 Flush
23068 260/300 Write block 3532 len 20480 data: 6 5 4 4 4
23069 261/300 Read block 2947 len 20480
23070 262/300 Read block 3829 len 36864
23071 263/300 Write block 689 len 36864 data: 5 4 4 4 5 4 6 5 5
23072 264/300 Read block 259 len 8192
23073 265/300 Write block 4962 len 12288 data: 4 4 6
23074 266/300 Write block 4752 len 40960 data: 7 6 6 4 2 2 4 4 4 3
23075 267/300 Write block 337 len 24576 data: 7 6 6 5 5 3
23076 268/300 Read block 1361 len 28672
23077 269/300 Write block 2696 len 36864 data: 4 4 3 4 3 4 4 4 3
23078 270/300 Write block 2066 len 32768 data: 4 5 4 4 4 4 5 5
23079 271/300 Write block 4249 len 16384 data: 7 7 6 7
23080 272/300 Read block 4074 len 4096
23081 273/300 Read block 4365 len 16384
23082 274/300 Read block 245 len 16384
23083 275/300 Read block 1305 len 36864
23084 276/300 Read block 2060 len 28672
23085 277/300 Flush
23086 278/300 Read block 1947 len 24576
23087 279/300 Read block 4366 len 36864
23088 280/300 Write block 1233 len 4096 data: 6
23089 281/300 Flush
23090 282/300 Read block 4226 len 24576
23091 283/300 Write block 1726 len 4096 data: 3
23092 284/300 Write block 2662 len 8192 data: 9 10
23093 285/300 Write block 2638 len 16384 data: 6 6 7 6
23094 286/300 Write block 1174 len 12288 data: 8 5 6
23095 287/300 Read block 2394 len 28672
23096 288/300 Write block 519 len 36864 data: 8 8 8 9 10 8 8 6 4
23097 289/300 Write block 2988 len 40960 data: 4 4 5 5 6 5 5 6 6 6
23098 290/300 Write block 4993 len 28672 data: 5 4 4 4 3 2 2
23099 291/300 Read block 4860 len 8192
23100 292/300 Read block 31 len 16384
23101 293/300 Write block 935 len 12288 data: 5 3 2
23102 294/300 Flush
23103 295/300 Read block 771 len 20480
23104 296/300 Write block 4112 len 24576 data: 4 4 2 2 4 4
23105 297/300 Read block 1757 len 24576
23106 298/300 Write block 105 len 40960 data: 4 3 4 5 5 3 3 4 5 6
23107 299/300 Read block 379 len 24576
23108 300/300 Read block 3341 len 28672
23109 Main task has work to do, go find it
23110 got dsc Start(1)
23111 start 1
23112 Feb 27 21:49:15.493 INFO request completed, latency_us: 223, response_code: 204, uri: /start/cid/1, method: POST, req_id: b3b38511-53e2-4982-a502-32a7e4256c8e, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
23113 Replay: started 1, returned:Ok(())
23114 ----------------------------------------------------------------
23115 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
23116 States: Active Offline Active
23117 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
23118 7839 Acked 8838 Write 9 Done New Done true
23119 7840 Acked 8839 Read 1 Done New Done true
23120 7841 Acked 8840 Write 8 Done New Done false
23121 7842 Acked 8841 Read 10 Done New Done false
23122 7843 Acked 8842 Write 10 Done New Done false
23123 7844 Acked 8843 Write 9 Done New Done false
23124 7845 Acked 8844 Write 2 Done New Done false
23125 7846 Acked 8845 Read 4 Done New Done false
23126 7847 Acked 8846 Write 10 Done New Done false
23127 7848 Acked 8847 Read 2 Done New Done false
23128 7849 Acked 8848 Read 4 Done New Done false
23129 7850 Acked 8849 Read 8 Done New Done false
23130 7851 Acked 8850 Write 4 Done New Done false
23131 7852 Acked 8851 Write 1 Done New Done false
23132 7853 Acked 8852 Flush 0 Done New Done false
23133 7854 Acked 8853 Write 7 Done New Done false
23134 7855 Acked 8854 Write 10 Done New Done false
23135 7856 Acked 8855 Read 8 Done New Done false
23136 7857 Acked 8856 Flush 0 Done New Done false
23137 7858 Acked 8857 Read 2 Done New Done false
23138 7859 Acked 8858 Read 7 Done New Done false
23139 7860 Acked 8859 Read 9 Done New Done false
23140 7861 Acked 8860 Write 4 Done New Done false
23141 7862 Acked 8861 Read 6 Done New Done false
23142 7863 Acked 8862 Write 1 Done New Done false
23143 7864 Acked 8863 Read 7 Done New Done false
23144 7865 Acked 8864 Write 6 Done New Done false
23145 7866 Acked 8865 Write 8 Done New Done false
23146 7867 Acked 8866 Flush 0 Done New Done false
23147 7868 Acked 8867 Write 10 Done New Done false
23148 7869 Acked 8868 Write 10 Done New Done false
23149 7870 Acked 8869 Write 5 Done New Done false
23150 7871 Acked 8870 Read 7 Done New Done false
23151 7872 Acked 8871 Read 4 Done New Done false
23152 7873 Acked 8872 Write 4 Done New Done false
23153 7874 Acked 8873 Read 3 Done New Done false
23154 7875 Acked 8874 Read 2 Done New Done false
23155 7876 Acked 8875 Write 3 Done New Done false
23156 7877 Acked 8876 Read 7 Done New Done false
23157 7878 Acked 8877 Flush 0 Done New Done false
23158 7879 Acked 8878 Read 6 Done New Done false
23159 7880 Acked 8879 Read 2 Done New Done false
23160 7881 Acked 8880 Write 7 Done New Done false
23161 7882 Acked 8881 Read 4 Done New Done false
23162 7883 Acked 8882 Read 9 Done New Done false
23163 7884 Acked 8883 Read 7 Done New Done false
23164 7885 Acked 8884 Read 6 Done New Done false
23165 7886 Acked 8885 Read 5 Done New Done false
23166 7887 Acked 8886 Read 7 Done New Done false
23167 7888 Acked 8887 Write 5 Done New Done false
23168 7889 Acked 8888 Write 8 Done New Done false
23169 7890 Acked 8889 Write 4 Done New Done false
23170 7891 Acked 8890 Read 4 Done New Done false
23171 7892 Acked 8891 Write 4 Done New Done false
23172 7893 Acked 8892 Write 4 Done New Done false
23173 7894 Acked 8893 Flush 0 Done New Done false
23174 7895 Acked 8894 Write 3 Done New Done false
23175 7896 Acked 8895 Write 5 Done New Done false
23176 7897 Acked 8896 Write 3 Done New Done false
23177 7898 Acked 8897 Read 6 Done New Done false
23178 7899 Acked 8898 Flush 0 Done New Done false
23179 7900 Acked 8899 Read 8 Done New Done false
23180 7901 Acked 8900 Write 10 Done New Done false
23181 7902 Acked 8901 Read 4 Done New Done false
23182 7903 Acked 8902 Read 10 Done New Done false
23183 7904 Acked 8903 Read 2 Done New Done false
23184 7905 Acked 8904 Read 3 Done New Done false
23185 7906 Acked 8905 Write 5 Done New Done false
23186 7907 Acked 8906 Read 4 Done New Done false
23187 7908 Acked 8907 Read 3 Done New Done false
23188 7909 Acked 8908 Read 1 Done New Done false
23189 7910 Acked 8909 Read 7 Done New Done false
23190 7911 Acked 8910 Read 5 Done New Done false
23191 7912 Acked 8911 Write 1 Done New Done false
23192 7913 Acked 8912 Flush 0 Done New Done false
23193 7914 Acked 8913 Write 2 Done New Done false
23194 7915 Acked 8914 Read 4 Done New Done false
23195 7916 Acked 8915 Read 6 Done New Done false
23196 7917 Acked 8916 Flush 0 Done New Done false
23197 7918 Acked 8917 Write 7 Done New Done false
23198 7919 Acked 8918 Read 4 Done New Done false
23199 7920 Acked 8919 Write 9 Done New Done false
23200 7921 Acked 8920 Write 4 Done New Done false
23201 7922 Acked 8921 Read 10 Done New Done false
23202 7923 Acked 8922 Write 7 Done New Done false
23203 7924 Acked 8923 Write 2 Done New Done false
23204 7925 Acked 8924 Flush 0 Done New Done false
23205 7926 Acked 8925 Read 1 Done New Done false
23206 7927 Acked 8926 Read 5 Done New Done false
23207 7928 Acked 8927 Read 7 Done New Done false
23208 7929 Acked 8928 Write 1 Done New Done false
23209 7930 Acked 8929 Flush 0 Done New Done false
23210 7931 Acked 8930 Read 5 Done New Done false
23211 7932 Acked 8931 Read 3 Done New Done false
23212 7933 Acked 8932 Read 8 Done New Done false
23213 7934 Acked 8933 Write 9 Done New Done false
23214 7935 Acked 8934 Write 8 Done New Done false
23215 7936 Acked 8935 Write 9 Done New Done false
23216 7937 Acked 8936 Read 3 Done New Done false
23217 7938 Acked 8937 Read 5 Done New Done false
23218 7939 Acked 8938 Read 7 Done New Done false
23219 7940 Acked 8939 Read 2 Done New Done false
23220 7941 Acked 8940 Flush 0 Done New Done false
23221 7942 Acked 8941 Write 7 Done New Done false
23222 7943 Acked 8942 Read 6 Done New Done false
23223 7944 Acked 8943 Write 2 Done New Done false
23224 7945 Acked 8944 Write 9 Done New Done false
23225 7946 Acked 8945 Read 1 Done New Done false
23226 7947 Acked 8946 Read 6 Done New Done false
23227 7948 Acked 8947 Write 2 Done New Done false
23228 7949 Acked 8948 Write 9 Done New Done false
23229 7950 Acked 8949 Write 7 Done New Done false
23230 7951 Acked 8950 Read 3 Done New Done false
23231 7952 Acked 8951 Read 5 Done New Done false
23232 7953 Acked 8952 Read 9 Done New Done false
23233 7954 Acked 8953 Flush 0 Done New Done false
23234 7955 Acked 8954 Write 7 Done New Done false
23235 7956 Acked 8955 Read 9 Done New Done false
23236 7957 Acked 8956 Read 1 Done New Done false
23237 7958 Acked 8957 Read 10 Done New Done false
23238 7959 Acked 8958 Read 8 Done New Done false
23239 7960 Acked 8959 Write 3 Done New Done false
23240 7961 Acked 8960 Write 7 Done New Done false
23241 7962 Acked 8961 Write 4 Done New Done false
23242 7963 Acked 8962 Read 9 Done New Done false
23243 7964 Acked 8963 Read 1 Done New Done false
23244 7965 Acked 8964 Read 7 Done New Done false
23245 7966 Acked 8965 Read 8 Done New Done false
23246 7967 Acked 8966 Write 10 Done New Done false
23247 7968 Acked 8967 Write 6 Done New Done false
23248 7969 Acked 8968 Flush 0 Done New Done false
23249 7970 Acked 8969 Read 8 Done New Done false
23250 7971 Acked 8970 Read 10 Done New Done false
23251 7972 Acked 8971 Write 7 Done New Done false
23252 7973 Acked 8972 Write 8 Done New Done false
23253 7974 Acked 8973 Read 9 Done New Done false
23254 7975 Acked 8974 Write 2 Done New Done false
23255 7976 Acked 8975 Read 3 Done New Done false
23256 7977 Acked 8976 Read 10 Done New Done false
23257 7978 Acked 8977 Read 1 Done New Done false
23258 7979 Acked 8978 Write 4 Done New Done false
23259 7980 Acked 8979 Write 10 Done New Done false
23260 7981 Acked 8980 Read 3 Done New Done false
23261 7982 Acked 8981 Write 9 Done New Done false
23262 7983 Acked 8982 Flush 0 Done New Done false
23263 7984 Acked 8983 Read 10 Done New Done false
23264 7985 Acked 8984 Write 9 Done New Done false
23265 7986 Acked 8985 Read 8 Done New Done false
23266 7987 Acked 8986 Flush 0 Done New Done false
23267 7988 Acked 8987 Read 5 Done New Done false
23268 7989 Acked 8988 Read 7 Done New Done false
23269 7990 Acked 8989 Write 2 Done New Done false
23270 7991 Acked 8990 Write 5 Done New Done false
23271 7992 Acked 8991 Read 5 Done New Done false
23272 7993 Acked 8992 Flush 0 Done New Done false
23273 7994 Acked 8993 Write 6 Done New Done false
23274 7995 Acked 8994 Read 3 Done New Done false
23275 7996 Acked 8995 Write 7 Done New Done false
23276 7997 Acked 8996 Write 4 Done New Done false
23277 7998 Acked 8997 Read 5 Done New Done false
23278 7999 Acked 8998 Write 5 Done New Done false
23279 8000 Acked 8999 Read 7 Done New Done false
23280 8001 Acked 9000 Read 5 Done New Done false
23281 8002 Acked 9001 Read 6 Done New Done false
23282 8003 Acked 9002 Read 8 Done New Done false
23283 8004 Acked 9003 Read 3 Done New Done false
23284 8005 Acked 9004 Write 7 Done New Done false
23285 8006 Acked 9005 Write 8 Done New Done false
23286 8007 Acked 9006 Read 10 Done New Done false
23287 8008 Acked 9007 Write 10 Done New Done false
23288 8009 Acked 9008 Read 8 Done New Done false
23289 8010 Acked 9009 Read 5 Done New Done false
23290 8011 Acked 9010 Write 9 Done New Done false
23291 8012 Acked 9011 Read 2 Done New Done false
23292 8013 Acked 9012 Read 10 Done New Done false
23293 8014 Acked 9013 Read 10 Done New Done false
23294 8015 Acked 9014 Flush 0 Done New Done false
23295 8016 Acked 9015 Flush 0 Done New Done false
23296 8017 Acked 9016 Read 5 Done New Done false
23297 8018 Acked 9017 Read 6 Done New Done false
23298 8019 Acked 9018 Read 1 Done New Done false
23299 8020 Acked 9019 Flush 0 Done New Done false
23300 8021 Acked 9020 Write 5 Done New Done false
23301 8022 Acked 9021 Write 9 Done New Done false
23302 8023 Acked 9022 Flush 0 Done New Done false
23303 8024 Acked 9023 Write 9 Done New Done false
23304 8025 Acked 9024 Read 2 Done New Done false
23305 8026 Acked 9025 Read 3 Done New Done false
23306 8027 Acked 9026 Write 3 Done New Done false
23307 8028 Acked 9027 Read 5 Done New Done false
23308 8029 Acked 9028 Write 9 Done New Done false
23309 8030 Acked 9029 Write 9 Done New Done false
23310 8031 Acked 9030 Read 6 Done New Done false
23311 8032 Acked 9031 Write 6 Done New Done false
23312 8033 Acked 9032 Read 10 Done New Done false
23313 8034 Acked 9033 Read 2 Done New Done false
23314 8035 Acked 9034 Read 5 Done New Done false
23315 8036 Acked 9035 Write 6 Done New Done false
23316 8037 Acked 9036 Write 10 Done New Done false
23317 8038 Acked 9037 Write 7 Done New Done false
23318 8039 Acked 9038 Write 1 Done New Done false
23319 8040 Acked 9039 Flush 0 Done New Done false
23320 8041 Acked 9040 Read 3 Done New Done false
23321 8042 Acked 9041 Write 8 Done New Done false
23322 8043 Acked 9042 Read 1 Done New Done false
23323 8044 Acked 9043 Write 1 Done New Done false
23324 8045 Acked 9044 Flush 0 Done New Done false
23325 8046 Acked 9045 Flush 0 Done New Done false
23326 8047 Acked 9046 Write 2 Done New Done false
23327 8048 Acked 9047 Read 9 Done New Done false
23328 8049 Acked 9048 Write 8 Done New Done false
23329 8050 Acked 9049 Read 7 Done New Done false
23330 8051 Acked 9050 Read 10 Done New Done false
23331 8052 Acked 9051 Read 2 Done New Done false
23332 8053 Acked 9052 Read 8 Done New Done false
23333 8054 Acked 9053 Read 4 Done New Done false
23334 8055 Acked 9054 Write 3 Done New Done false
23335 8056 Acked 9055 Write 9 Done New Done false
23336 8057 Acked 9056 Write 7 Done New Done false
23337 8058 Acked 9057 Read 8 Done New Done false
23338 8059 Acked 9058 Read 1 Done New Done false
23339 8060 Acked 9059 Write 3 Done New Done false
23340 8061 Acked 9060 Read 2 Done New Done false
23341 8062 Acked 9061 Flush 0 Done New Done false
23342 8063 Acked 9062 Write 2 Done New Done false
23343 8064 Acked 9063 Write 4 Done New Done false
23344 8065 Acked 9064 Write 5 Done New Done false
23345 8066 Acked 9065 Read 9 Done New Done false
23346 8067 Acked 9066 Write 7 Done New Done false
23347 8068 Acked 9067 Write 6 Done New Done false
23348 8069 Acked 9068 Read 3 Done New Done false
23349 8070 Acked 9069 Write 5 Done New Done false
23350 8071 Acked 9070 Read 9 Done New Done false
23351 8072 Acked 9071 Read 2 Done New Done false
23352 8073 Acked 9072 Write 8 Done New Done false
23353 8074 Acked 9073 Read 9 Done New Done false
23354 8075 Acked 9074 Read 8 Done New Done false
23355 8076 Acked 9075 Read 7 Done New Done false
23356 8077 Acked 9076 Read 9 Done New Done false
23357 8078 Acked 9077 Write 3 Done New Done false
23358 8079 Acked 9078 Write 3 Done New Done false
23359 8080 Acked 9079 Write 8 Done New Done false
23360 8081 Acked 9080 Read 10 Done New Done false
23361 8082 Acked 9081 Write 8 Done New Done false
23362 8083 Acked 9082 Read 8 Done New Done false
23363 8084 Acked 9083 Read 8 Done New Done false
23364 8085 Acked 9084 Write 7 Done New Done false
23365 8086 Acked 9085 Write 6 Done New Done false
23366 8087 Acked 9086 Read 10 Done New Done false
23367 8088 Acked 9087 Read 6 Done New Done false
23368 8089 Acked 9088 Read 10 Done New Done false
23369 8090 Acked 9089 Read 10 Done New Done false
23370 8091 Acked 9090 Read 8 Done New Done false
23371 8092 Acked 9091 Read 1 Done New Done false
23372 8093 Acked 9092 Flush 0 Done New Done false
23373 8094 Acked 9093 Read 5 Done New Done false
23374 8095 Acked 9094 Flush 0 Done New Done false
23375 8096 Acked 9095 Write 1 Done New Done false
23376 8097 Acked 9096 Flush 0 Done New Done false
23377 8098 Acked 9097 Write 5 Done New Done false
23378 8099 Acked 9098 Read 5 Done New Done false
23379 8100 Acked 9099 Read 9 Done New Done false
23380 8101 Acked 9100 Write 9 Done New Done false
23381 8102 Acked 9101 Read 2 Done New Done false
23382 8103 Acked 9102 Write 3 Done New Done false
23383 8104 Acked 9103 Write 10 Done New Done false
23384 8105 Acked 9104 Write 6 Done New Done false
23385 8106 Acked 9105 Read 7 Done New Done false
23386 8107 Acked 9106 Write 9 Done New Done false
23387 8108 Acked 9107 Write 8 Done New Done false
23388 8109 Acked 9108 Write 4 Done New Done false
23389 8110 Acked 9109 Read 1 Done New Done false
23390 8111 Acked 9110 Read 4 Done New Done false
23391 8112 Acked 9111 Read 4 Done New Done false
23392 8113 Acked 9112 Read 9 Done New Done false
23393 8114 Acked 9113 Read 7 Done New Done false
23394 8115 Acked 9114 Flush 0 Done New Done false
23395 8116 Acked 9115 Read 6 Done New Done false
23396 8117 Acked 9116 Read 9 Done New Done false
23397 8118 Acked 9117 Write 1 Done New Done false
23398 8119 Acked 9118 Flush 0 Done New Done false
23399 8120 Acked 9119 Read 6 Done New Done false
23400 8121 Acked 9120 Write 1 Done New Done false
23401 8122 Acked 9121 Write 2 Done New Done false
23402 8123 Acked 9122 Write 4 Done New Done false
23403 8124 Acked 9123 Write 3 Done New Done false
23404 8125 Acked 9124 Read 7 Done New Done false
23405 8126 Acked 9125 Write 9 Done New Done false
23406 8127 Acked 9126 Write 10 Done New Done false
23407 8128 Acked 9127 Write 7 Done New Done false
23408 8129 Acked 9128 Read 2 Done New Done false
23409 8130 Acked 9129 Read 4 Done New Done false
23410 8131 Acked 9130 Write 3 Done New Done false
23411 8132 Acked 9131 Flush 0 Done New Done false
23412 8133 Acked 9132 Read 5 Done New Done false
23413 8134 Acked 9133 Write 6 Done New Done false
23414 8135 Acked 9134 Read 6 Done New Done false
23415 8136 Acked 9135 Write 10 Done New Done false
23416 8137 Acked 9136 Read 6 Done New Done false
23417 8138 Acked 9137 Read 7 Done New Done false
23418 STATES DS:0 DS:1 DS:2 TOTAL
23419 New 0 300 0 300
23420 Sent 0 0 0 0
23421 Done 300 0 300 600
23422 Skipped 0 0 0 0
23423 Error 0 0 0 0
23424 Last Flush: 9131 8837 9131
23425 Downstairs last five completed: 8837 8836 8835 8834 8833
23426 Upstairs last five completed: 8138 8137 8136 8135 8134
23427 CLIENT: Up:0 ds:300 act:2
234282024-02-27T21:49:16.148ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
234292024-02-27T21:49:16.149ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
234302024-02-27T21:49:16.149ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
234312024-02-27T21:49:16.149ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
234322024-02-27T21:49:16.149ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
234332024-02-27T21:49:16.149ZINFOcrucible: [1] client re-new 301 jobs since flush 8837 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
234342024-02-27T21:49:16.149ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
234352024-02-27T21:49:17.150ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
234362024-02-27T21:49:17.150ZWARNcrucible: ds_connection connect to 127.0.0.1:8820 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
234372024-02-27T21:49:17.150ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
234382024-02-27T21:49:17.150ZWARNcrucible: downstairs task for 1 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
234392024-02-27T21:49:17.150ZINFOcrucible: Still missing, state is Offline = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
234402024-02-27T21:49:17.150ZINFOcrucible: [1] client re-new 301 jobs since flush 8837 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
234412024-02-27T21:49:17.150ZINFOcrucible: client task is exiting = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
23442 [8820] got start action
23443 [8820] I am going to restart
23444 Starting downstairs at port 8820
23445 Make output file at "/tmp/dsc/downstairs-8820.txt"
23446 [8820][1] reports Starting
23447 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1194)
23448 [8820][1] reports Running
234492024-02-27T21:49:18.151ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
234502024-02-27T21:49:18.151ZINFOcrucible: ds_connection connected = io task client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
234512024-02-27T21:49:18.153ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
234522024-02-27T21:49:18.153ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008820 matches = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
234532024-02-27T21:49:18.153ZINFOcrucible: send last flush ID to this DS: 8837 = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
234542024-02-27T21:49:18.153ZINFOcrucible: Replied this last flush ID: 8837 = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
234552024-02-27T21:49:18.153ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
234562024-02-27T21:49:18.153ZINFOcrucible: [1] Transition from Offline to Replay = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
234572024-02-27T21:49:18.153ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
234582024-02-27T21:49:18.153ZINFOcrucible: [1] Transition from Replay to Active = downstairs client = 1 session_id = e6a5a974-925a-494f-8845-facc441c2690
23459 ----------------------------------------------------------------
23460 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23461 Downstairs last five completed: 9138 9137 9136 9135 9134
23462 Upstairs last five completed: 8139 8138 8137 8136 8135
23463 CLIENT: Up:0 ds:0 act:3
23464 Replay: All jobs finished, all DS active.
23465 Main task has work to do, go find it
23466 got dsc Stop(2)
23467 stop 2
23468 [8830] Got stop action so:false kr:false
23469 Feb 27 21:49:19.497 INFO request completed, latency_us: 372, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 68054aab-d797-42f3-a705-61f2425d176b, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
234702024-02-27T21:49:19.498ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
234712024-02-27T21:49:19.498ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
23472 Feb 27 21:49:19.498 INFO request completed{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-27T21:49:19.498175052Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
23473 , latency_us: 172, response_code: 200, uri: /state/cid/2, method: GET, req_id: c26af7eb-3df9-41b2-85e6-2ac228bea0c7, remote_addr{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:49:19.498200489Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"client":"2","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
23474 : 127.0.0.1:42901{"msg":"[2] client re-new 0 jobs since flush 9138","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:49:19.498250782Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"downstairs","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
23475 , local_addr: 127.0.0.1:9998
234762024-02-27T21:49:19.498ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
23477 [8830] Exited with: ExitStatus(unix_wait_status(9))
23478 [8830][2] reports Exit
234792024-02-27T21:49:20.499ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
234802024-02-27T21:49:20.499ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
234812024-02-27T21:49:20.499ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
234822024-02-27T21:49:20.499ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
234832024-02-27T21:49:20.499ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
234842024-02-27T21:49:20.499ZINFOcrucible: [2] client re-new 0 jobs since flush 9138 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
234852024-02-27T21:49:20.499ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
234862024-02-27T21:49:21.500ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
234872024-02-27T21:49:21.500ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
234882024-02-27T21:49:21.500ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
234892024-02-27T21:49:21.500ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
234902024-02-27T21:49:21.500ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
234912024-02-27T21:49:21.500ZINFOcrucible: [2] client re-new 0 jobs since flush 9138 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
234922024-02-27T21:49:21.500ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
234932024-02-27T21:49:22.501ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
234942024-02-27T21:49:22.501ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
234952024-02-27T21:49:22.501ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
234962024-02-27T21:49:22.501ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
234972024-02-27T21:49:22.501ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
234982024-02-27T21:49:22.501ZINFOcrucible: [2] client re-new 0 jobs since flush 9138 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
234992024-02-27T21:49:22.501ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
23500 Feb 27 21:49:23.499 INFO request completed, latency_us: 205, response_code: 200, uri: /state/cid/2, method: GET, req_id: 5208e2db-ba66-475e-84ef-ac7e1e96c838, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
23501 001/300 Read block 2531 len 12288
23502 002/300 Flush
235032024-02-27T21:49:23.502ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
235042024-02-27T21:49:23.502ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
235052024-02-27T21:49:23.502ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
235062024-02-27T21:49:23.502ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
235072024-02-27T21:49:23.502ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
235082024-02-27T21:49:23.502ZINFOcrucible: [2] client re-new 2 jobs since flush 9138 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
235092024-02-27T21:49:23.502ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
23510 003/300 Write block 4615 len 4096 data: 5
23511 004/300 Read block 3479 len 4096
23512 005/300 Read block 1134 len 4096
23513 006/300 Read block 3207 len 4096
23514 007/300 Write block 3588 len 12288 data: 6 6 5
23515 008/300 Read block 2455 len 32768
23516 009/300 Read block 270 len 36864
23517 010/300 Read block 1048 len 16384
23518 011/300 Write block 724 len 8192 data: 2 2
23519 012/300 Write block 3390 len 20480 data: 6 5 6 5 4
23520 013/300 Write block 699 len 12288 data: 5 5 5
23521 014/300 Read block 3496 len 4096
23522 015/300 Read block 4090 len 32768
23523 016/300 Read block 4299 len 24576
23524 017/300 Flush
23525 018/300 Read block 4412 len 4096
23526 019/300 Read block 1328 len 36864
23527 020/300 Write block 4922 len 28672 data: 6 8 9 9 8 7 7
23528 021/300 Read block 1652 len 32768
23529 022/300 Write block 1145 len 12288 data: 8 8 7
23530 023/300 Read block 4162 len 40960
23531 024/300 Write block 2274 len 16384 data: 3 3 3 6
23532 025/300 Read block 4858 len 32768
23533 026/300 Read block 2310 len 20480
23534 027/300 Read block 526 len 8192
23535 028/300 Read block 2086 len 24576
23536 029/300 Read block 4813 len 32768
23537 030/300 Write block 600 len 24576 data: 7 7 7 7 6 5
23538 031/300 Read block 4489 len 40960
23539 032/300 Write block 3024 len 28672 data: 9 7 7 8 8 8 7
23540 033/300 Read block 1263 len 32768
23541 034/300 Read block 3359 len 36864
23542 035/300 Write block 3786 len 12288 data: 4 5 6
23543 036/300 Write block 3794 len 24576 data: 4 5 6 5 10 10
23544 037/300 Write block 4821 len 32768 data: 5 4 6 6 6 6 6 5
23545 038/300 Write block 1410 len 12288 data: 6 5 5
23546 039/300 Read block 3623 len 16384
23547 040/300 Write block 1288 len 32768 data: 7 7 6 6 5 4 4 4
23548 041/300 Read block 3591 len 24576
23549 042/300 Write block 1523 len 20480 data: 4 3 4 4 2
23550 043/300 Write block 1259 len 24576 data: 7 7 6 6 7 5
23551 044/300 Write block 1342 len 28672 data: 2 2 2 2 3 4 4
23552 045/300 Read block 3326 len 40960
23553 046/300 Flush
23554 047/300 Read block 4814 len 4096
23555 048/300 Write block 1330 len 32768 data: 6 5 5 5 4 4 4 4
23556 049/300 Write block 2010 len 20480 data: 4 4 3 3 5
23557 050/300 Read block 2203 len 20480
23558 051/300 Read block 3490 len 40960
23559 052/300 Read block 2130 len 28672
23560 053/300 Write block 1994 len 24576 data: 4 5 9 9 7 7
23561 054/300 Write block 562 len 8192 data: 6 6
23562 055/300 Write block 1747 len 16384 data: 5 6 5 5
23563 056/300 Flush
23564 057/300 Read block 4895 len 4096
23565 058/300 Write block 944 len 8192 data: 8 7
23566 059/300 Read block 699 len 12288
23567 060/300 Read block 4667 len 32768
23568 061/300 Read block 2666 len 40960
23569 062/300 Write block 3996 len 28672 data: 3 3 5 5 6 6 5
23570 063/300 Write block 4586 len 24576 data: 5 4 7 5 5 5
23571 064/300 Write block 298 len 24576 data: 2 2 3 2 3 3
23572 065/300 Read block 4827 len 36864
23573 066/300 Write block 1001 len 12288 data: 3 3 3
23574 067/300 Write block 3380 len 20480 data: 4 3 4 5 5
23575 068/300 Write block 564 len 20480 data: 6 6 7 6 6
23576 069/300 Read block 2748 len 20480
23577 070/300 Read block 3762 len 20480
23578 071/300 Read block 2828 len 28672
23579 072/300 Write block 1670 len 20480 data: 9 8 7 7 5
23580 073/300 Read block 1341 len 16384
23581 074/300 Write block 3859 len 20480 data: 6 7 7 8 8
23582 075/300 Read block 781 len 32768
23583 076/300 Read block 88 len 24576
23584 077/300 Read block 2649 len 40960
23585 078/300 Write block 1782 len 40960 data: 6 6 6 6 5 4 3 5 4 4
23586 079/300 Read block 3606 len 28672
23587 080/300 Read block 2083 len 4096
23588 081/300 Read block 2002 len 8192
23589 082/300 Flush
23590 083/300 Write block 149 len 16384 data: 5 5 7 7
23591 084/300 Read block 1983 len 4096
23592 085/300 Write block 2703 len 24576 data: 5 4 3 3 3 5
23593 086/300 Write block 4013 len 32768 data: 6 8 7 6 7 9 9 9
23594 087/300 Write block 925 len 36864 data: 5 4 5 5 5 5 6 4 5
23595 088/300 Flush
23596 089/300 Read block 1215 len 16384
23597 090/300 Write block 149 len 16384 data: 6 6 8 8
23598 091/300 Write block 1767 len 32768 data: 4 4 4 4 3 4 2 2
23599 092/300 Read block 1446 len 8192
23600 093/300 Read block 3864 len 32768
23601 094/300 Read block 2086 len 16384
23602 095/300 Read block 1777 len 4096
23603 096/300 Read block 3248 len 28672
23604 097/300 Write block 965 len 40960 data: 4 3 5 5 8 7 7 7 8 6
23605 098/300 Read block 946 len 8192
23606 099/300 Write block 1820 len 28672 data: 5 4 4 4 4 6 7
23607 100/300 Read block 4820 len 36864
23608 101/300 Read block 2215 len 24576
23609 102/300 Read block 2650 len 36864
23610 103/300 Read block 3147 len 28672
23611 104/300 Write block 1716 len 36864 data: 9 8 8 6 4 5 5 5 6
23612 105/300 Read block 2934 len 12288
23613 106/300 Read block 3881 len 24576
23614 107/300 Read block 424 len 16384
23615 108/300 Flush
23616 109/300 Read block 1468 len 32768
23617 110/300 Read block 4832 len 12288
23618 111/300 Write block 1618 len 24576 data: 6 7 7 7 7 5
23619 112/300 Write block 663 len 36864 data: 6 5 8 7 6 8 8 8 8
23620 113/300 Read block 1614 len 8192
23621 114/300 Write block 1376 len 12288 data: 4 4 6
23622 115/300 Write block 1880 len 20480 data: 3 4 5 5 4
23623 116/300 Write block 1115 len 4096 data: 5
23624 117/300 Write block 1974 len 20480 data: 6 9 10 9 8
23625 118/300 Read block 2234 len 36864
23626 119/300 Read block 858 len 28672
23627 120/300 Flush
23628 121/300 Read block 852 len 12288
23629 122/300 Read block 2114 len 16384
23630 123/300 Read block 4394 len 40960
23631 124/300 Read block 3890 len 12288
23632 125/300 Write block 3754 len 32768 data: 8 8 7 7 6 7 5 5
23633 126/300 Write block 1048 len 32768 data: 2 3 4 5 7 6 6 5
23634 127/300 Read block 1051 len 8192
23635 128/300 Read block 3660 len 40960
23636 129/300 Read block 327 len 36864
23637 130/300 Read block 3549 len 12288
23638 131/300 Read block 2926 len 28672
23639 132/300 Write block 996 len 24576 data: 3 4 5 5 5 4
23640 133/300 Flush
23641 134/300 Read block 331 len 28672
23642 135/300 Write block 2959 len 28672 data: 6 5 5 6 7 7 7
23643 136/300 Read block 2606 len 36864
23644 137/300 Read block 4584 len 8192
23645 138/300 Read block 4100 len 40960
23646 139/300 Read block 4720 len 36864
23647 140/300 Flush
23648 141/300 Read block 1694 len 28672
23649 142/300 Read block 208 len 4096
23650 143/300 Write block 4624 len 32768 data: 6 7 7 7 8 8 9 8
23651 144/300 Read block 3775 len 12288
23652 145/300 Write block 2068 len 36864 data: 5 5 5 5 6 6 5 6 5
23653 146/300 Write block 4445 len 24576 data: 8 5 5 4 6 6
23654 147/300 Read block 3864 len 4096
23655 148/300 Write block 4114 len 16384 data: 3 3 5 5
23656 149/300 Write block 2189 len 40960 data: 7 7 10 8 8 8 6 7 9 9
23657 150/300 Read block 3045 len 36864
23658 151/300 Write block 2832 len 28672 data: 6 6 4 5 6 5 4
23659 152/300 Write block 1283 len 16384 data: 2 3 4 5
23660 153/300 Write block 4430 len 16384 data: 7 8 8 7
23661 154/300 Write block 2401 len 36864 data: 5 4 6 6 6 6 6 5 5
23662 155/300 Read block 765 len 8192
23663 156/300 Read block 961 len 32768
23664 157/300 Read block 858 len 16384
23665 158/300 Read block 2886 len 40960
23666 159/300 Write block 3690 len 12288 data: 4 3 4
23667 160/300 Flush
23668 161/300 Write block 2868 len 16384 data: 4 5 7 8
23669 162/300 Flush
23670 163/300 Read block 4492 len 12288
23671 164/300 Read block 2575 len 4096
23672 165/300 Read block 3445 len 40960
23673 166/300 Write block 3492 len 40960 data: 9 8 9 6 5 6 5 5 5 5
23674 167/300 Read block 1522 len 12288
23675 168/300 Read block 3719 len 40960
23676 169/300 Write block 2949 len 8192 data: 6 6
23677 170/300 Read block 4435 len 32768
23678 171/300 Write block 3491 len 32768 data: 7 10 9 10 7 6 7 6
23679 172/300 Write block 4023 len 16384 data: 5 6 7 6
23680 173/300 Read block 1588 len 24576
23681 174/300 Write block 4664 len 24576 data: 7 8 8 8 9 7
23682 175/300 Read block 2161 len 4096
23683 176/300 Read block 3232 len 28672
23684 177/300 Read block 876 len 40960
23685 178/300 Read block 3159 len 40960
23686 179/300 Read block 2032 len 16384
23687 180/300 Write block 1116 len 12288 data: 5 5 7
23688 181/300 Write block 3546 len 8192 data: 5 5
23689 182/300 Read block 4177 len 16384
23690 183/300 Write block 3765 len 8192 data: 5 6
23691 184/300 Write block 1430 len 12288 data: 4 4 7
23692 185/300 Flush
23693 186/300 Read block 2848 len 32768
23694 187/300 Write block 4963 len 36864 data: 5 7 6 6 6 5 7 8 7
23695 188/300 Flush
23696 189/300 Read block 1801 len 4096
23697 190/300 Read block 4080 len 36864
23698 191/300 Read block 1634 len 12288
23699 192/300 Write block 567 len 28672 data: 7 7 4 3 6 5 5
23700 193/300 Read block 3706 len 36864
23701 194/300 Write block 2929 len 24576 data: 4 4 5 5 5 5
23702 195/300 Read block 3214 len 24576
23703 196/300 Write block 2891 len 12288 data: 6 6 5
23704 197/300 Read block 1276 len 36864
23705 198/300 Write block 3924 len 28672 data: 9 9 8 6 5 4 5
23706 199/300 Write block 244 len 20480 data: 6 7 7 6 6
23707 200/300 Read block 881 len 12288
23708 201/300 Write block 2739 len 4096 data: 3
23709 202/300 Write block 950 len 4096 data: 7
23710 203/300 Read block 1764 len 16384
23711 204/300 Read block 4464 len 24576
23712 205/300 Flush
23713 206/300 Flush
23714 207/300 Read block 3185 len 20480
23715 208/300 Read block 2331 len 36864
23716 209/300 Write block 2355 len 24576 data: 8 10 8 7 6 4
23717 210/300 Read block 414 len 8192
23718 211/300 Read block 3461 len 16384
23719 212/300 Read block 1812 len 20480
23720 213/300 Read block 1587 len 28672
23721 214/300 Write block 4846 len 24576 data: 6 6 5 4 3 3
23722 215/300 Read block 3804 len 28672
23723 216/300 Read block 443 len 24576
23724 217/300 Write block 751 len 16384 data: 4 5 5 6
23725 218/300 Flush
23726 219/300 Read block 3930 len 12288
23727 220/300 Read block 2952 len 40960
23728 221/300 Write block 1661 len 4096 data: 7
23729 222/300 Read block 3090 len 24576
23730 223/300 Read block 1742 len 12288
23731 224/300 Read block 1069 len 40960
23732 225/300 Write block 3930 len 4096 data: 6
23733 226/300 Read block 2906 len 12288
23734 227/300 Read block 3385 len 28672
23735 228/300 Read block 1628 len 8192
23736 229/300 Read block 3258 len 4096
23737 230/300 Write block 502 len 8192 data: 7 8
23738 231/300 Write block 3342 len 16384 data: 7 8 8 8
23739 232/300 Read block 1364 len 8192
23740 233/300 Write block 8 len 4096 data: 5
23741 234/300 Write block 3560 len 16384 data: 8 7 8 9
23742 235/300 Read block 3335 len 4096
23743 236/300 Read block 609 len 32768
23744 237/300 Write block 4777 len 4096 data: 7
23745 238/300 Write block 1285 len 40960 data: 5 6 6 8 8 7 7 6 5 5
23746 239/300 Write block 4320 len 12288 data: 9 9 7
23747 240/300 Read block 1794 len 16384
23748 241/300 Write block 3937 len 12288 data: 5 8 7
23749 242/300 Flush
23750 243/300 Read block 889 len 24576
23751 244/300 Read block 2971 len 40960
23752 245/300 Read block 4587 len 16384
23753 246/300 Flush
23754 247/300 Write block 2251 len 40960 data: 4 4 3 4 5 5 7 7 6 6
23755 248/300 Read block 1333 len 20480
23756 249/300 Write block 4435 len 24576 data: 8 9 9 9 8 8
23757 250/300 Read block 1953 len 24576
23758 251/300 Read block 1132 len 8192
23759 252/300 Read block 2810 len 8192
23760 253/300 Read block 1711 len 32768
23761 254/300 Write block 1502 len 28672 data: 7 5 5 6 5 5 5
23762 255/300 Read block 4611 len 40960
23763 256/300 Write block 109 len 36864 data: 6 4 4 5 6 7 6 5 6
23764 257/300 Flush
23765 258/300 Read block 935 len 32768
23766 259/300 Read block 1446 len 28672
23767 260/300 Read block 4165 len 8192
23768 261/300 Read block 1869 len 20480
23769 262/300 Read block 3243 len 24576
23770 263/300 Read block 3490 len 36864
23771 264/300 Read block 1323 len 32768
23772 265/300 Read block 4955 len 4096
23773 266/300 Write block 3009 len 36864 data: 8 9 8 8 6 5 5 6 5
23774 267/300 Write block 3596 len 16384 data: 9 8 7 6
23775 268/300 Read block 1912 len 4096
23776 269/300 Write block 3938 len 20480 data: 9 8 6 6 6
23777 270/300 Read block 4779 len 36864
23778 271/300 Read block 4981 len 16384
23779 272/300 Read block 900 len 32768
23780 273/300 Write block 4254 len 16384 data: 6 6 4 3
23781 274/300 Read block 82 len 28672
23782 275/300 Read block 1597 len 40960
23783 276/300 Read block 3550 len 16384
23784 277/300 Write block 2449 len 24576 data: 5 6 6 5 5 5
23785 278/300 Write block 4655 len 40960 data: 7 8 6 5 5 6 6 5 6 8
23786 279/300 Read block 1046 len 36864
23787 280/300 Write block 1993 len 20480 data: 4 5 6 10 10
23788 281/300 Flush
23789 282/300 Write block 1664 len 24576 data: 10 7 6 7 6 7
23790 283/300 Write block 730 len 12288 data: 6 7 8
23791 284/300 Write block 1967 len 40960 data: 5 7 6 6 7 6 6 7 10 11
23792 285/300 Write block 4194 len 32768 data: 5 5 6 7 7 7 6 4
23793 286/300 Write block 3788 len 32768 data: 7 6 6 5 5 4 5 6
23794 287/300 Read block 2524 len 20480
23795 288/300 Read block 1255 len 20480
23796 289/300 Flush
23797 290/300 Write block 515 len 8192 data: 7 7
23798 291/300 Read block 562 len 12288
23799 292/300 Write block 4205 len 12288 data: 3 3 4
23800 293/300 Read block 1547 len 4096
23801 294/300 Write block 2420 len 40960 data: 4 5 5 6 7 6 6 5 5 5
23802 295/300 Read block 1890 len 12288
23803 296/300 Write block 1265 len 8192 data: 6 7
23804 297/300 Read block 4730 len 28672
23805 298/300 Read block 4873 len 20480
23806 299/300 Read block 2365 len 28672
23807 300/300 Flush
23808 Main task has work to do, go find it
23809 got dsc Start(2)
23810 start 2
23811 Feb 27 21:49:23.843 INFO request completed, latency_us: 207, response_code: 204, uri: /start/cid/2, method: POST, req_id: b06bc61c-6b52-4f62-bd05-cc7358a4ddf7, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
23812 Replay: started 2, returned:Ok(())
23813 ----------------------------------------------------------------
23814 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23815 States: Active Active Offline
23816 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
23817 8140 Acked 9139 Read 3 Done Done New true
23818 8141 Acked 9140 Flush 0 Done Done New true
23819 8142 Acked 9141 Write 1 Done Done New false
23820 8143 Acked 9142 Read 1 Done Done New false
23821 8144 Acked 9143 Read 1 Done Done New false
23822 8145 Acked 9144 Read 1 Done Done New false
23823 8146 Acked 9145 Write 3 Done Done New false
23824 8147 Acked 9146 Read 8 Done Done New false
23825 8148 Acked 9147 Read 9 Done Done New false
23826 8149 Acked 9148 Read 4 Done Done New false
23827 8150 Acked 9149 Write 2 Done Done New false
23828 8151 Acked 9150 Write 5 Done Done New false
23829 8152 Acked 9151 Write 3 Done Done New false
23830 8153 Acked 9152 Read 1 Done Done New false
23831 8154 Acked 9153 Read 8 Done Done New false
23832 8155 Acked 9154 Read 6 Done Done New false
23833 8156 Acked 9155 Flush 0 Done Done New false
23834 8157 Acked 9156 Read 1 Done Done New false
23835 8158 Acked 9157 Read 9 Done Done New false
23836 8159 Acked 9158 Write 7 Done Done New false
23837 8160 Acked 9159 Read 8 Done Done New false
23838 8161 Acked 9160 Flush 0 Done Done New false
23839 8162 Acked 9161 Write 3 Done Done New false
23840 8163 Acked 9162 Read 10 Done Done New false
23841 8164 Acked 9163 Write 4 Done Done New false
23842 8165 Acked 9164 Read 8 Done Done New false
23843 8166 Acked 9165 Read 5 Done Done New false
23844 8167 Acked 9166 Read 2 Done Done New false
23845 8168 Acked 9167 Read 6 Done Done New false
23846 8169 Acked 9168 Read 8 Done Done New false
23847 8170 Acked 9169 Write 6 Done Done New false
23848 8171 Acked 9170 Read 10 Done Done New false
23849 8172 Acked 9171 Write 7 Done Done New false
23850 8173 Acked 9172 Read 8 Done Done New false
23851 8174 Acked 9173 Read 9 Done Done New false
23852 8175 Acked 9174 Write 3 Done Done New false
23853 8176 Acked 9175 Write 6 Done Done New false
23854 8177 Acked 9176 Write 8 Done Done New false
23855 8178 Acked 9177 Write 3 Done Done New false
23856 8179 Acked 9178 Read 4 Done Done New false
23857 8180 Acked 9179 Write 8 Done Done New false
23858 8181 Acked 9180 Read 6 Done Done New false
23859 8182 Acked 9181 Write 5 Done Done New false
23860 8183 Acked 9182 Write 6 Done Done New false
23861 8184 Acked 9183 Write 7 Done Done New false
23862 8185 Acked 9184 Read 10 Done Done New false
23863 8186 Acked 9185 Flush 0 Done Done New false
23864 8187 Acked 9186 Read 1 Done Done New false
23865 8188 Acked 9187 Write 8 Done Done New false
23866 8189 Acked 9188 Write 5 Done Done New false
23867 8190 Acked 9189 Read 5 Done Done New false
23868 8191 Acked 9190 Read 10 Done Done New false
23869 8192 Acked 9191 Read 7 Done Done New false
23870 8193 Acked 9192 Write 6 Done Done New false
23871 8194 Acked 9193 Write 2 Done Done New false
23872 8195 Acked 9194 Write 4 Done Done New false
23873 8196 Acked 9195 Flush 0 Done Done New false
23874 8197 Acked 9196 Read 1 Done Done New false
23875 8198 Acked 9197 Write 2 Done Done New false
23876 8199 Acked 9198 Read 3 Done Done New false
23877 8200 Acked 9199 Read 8 Done Done New false
23878 8201 Acked 9200 Read 10 Done Done New false
23879 8202 Acked 9201 Write 7 Done Done New false
23880 8203 Acked 9202 Write 6 Done Done New false
23881 8204 Acked 9203 Write 6 Done Done New false
23882 8205 Acked 9204 Read 9 Done Done New false
23883 8206 Acked 9205 Write 3 Done Done New false
23884 8207 Acked 9206 Write 5 Done Done New false
23885 8208 Acked 9207 Write 5 Done Done New false
23886 8209 Acked 9208 Read 5 Done Done New false
23887 8210 Acked 9209 Read 5 Done Done New false
23888 8211 Acked 9210 Read 7 Done Done New false
23889 8212 Acked 9211 Write 5 Done Done New false
23890 8213 Acked 9212 Read 4 Done Done New false
23891 8214 Acked 9213 Write 5 Done Done New false
23892 8215 Acked 9214 Read 8 Done Done New false
23893 8216 Acked 9215 Read 6 Done Done New false
23894 8217 Acked 9216 Read 10 Done Done New false
23895 8218 Acked 9217 Write 10 Done Done New false
23896 8219 Acked 9218 Read 7 Done Done New false
23897 8220 Acked 9219 Read 1 Done Done New false
23898 8221 Acked 9220 Read 2 Done Done New false
23899 8222 Acked 9221 Flush 0 Done Done New false
23900 8223 Acked 9222 Write 4 Done Done New false
23901 8224 Acked 9223 Read 1 Done Done New false
23902 8225 Acked 9224 Write 6 Done Done New false
23903 8226 Acked 9225 Write 8 Done Done New false
23904 8227 Acked 9226 Write 9 Done Done New false
23905 8228 Acked 9227 Flush 0 Done Done New false
23906 8229 Acked 9228 Read 4 Done Done New false
23907 8230 Acked 9229 Write 4 Done Done New false
23908 8231 Acked 9230 Write 8 Done Done New false
23909 8232 Acked 9231 Read 2 Done Done New false
23910 8233 Acked 9232 Read 8 Done Done New false
23911 8234 Acked 9233 Read 4 Done Done New false
23912 8235 Acked 9234 Read 1 Done Done New false
23913 8236 Acked 9235 Read 7 Done Done New false
23914 8237 Acked 9236 Write 10 Done Done New false
23915 8238 Acked 9237 Read 2 Done Done New false
23916 8239 Acked 9238 Write 7 Done Done New false
23917 8240 Acked 9239 Read 9 Done Done New false
23918 8241 Acked 9240 Read 6 Done Done New false
23919 8242 Acked 9241 Read 9 Done Done New false
23920 8243 Acked 9242 Read 7 Done Done New false
23921 8244 Acked 9243 Write 9 Done Done New false
23922 8245 Acked 9244 Read 3 Done Done New false
23923 8246 Acked 9245 Read 6 Done Done New false
23924 8247 Acked 9246 Read 4 Done Done New false
23925 8248 Acked 9247 Flush 0 Done Done New false
23926 8249 Acked 9248 Read 8 Done Done New false
23927 8250 Acked 9249 Read 3 Done Done New false
23928 8251 Acked 9250 Write 6 Done Done New false
23929 8252 Acked 9251 Write 9 Done Done New false
23930 8253 Acked 9252 Read 2 Done Done New false
23931 8254 Acked 9253 Write 3 Done Done New false
23932 8255 Acked 9254 Write 5 Done Done New false
23933 8256 Acked 9255 Write 1 Done Done New false
23934 8257 Acked 9256 Write 5 Done Done New false
23935 8258 Acked 9257 Read 9 Done Done New false
23936 8259 Acked 9258 Read 7 Done Done New false
23937 8260 Acked 9259 Flush 0 Done Done New false
23938 8261 Acked 9260 Read 3 Done Done New false
23939 8262 Acked 9261 Read 4 Done Done New false
23940 8263 Acked 9262 Read 10 Done Done New false
23941 8264 Acked 9263 Read 3 Done Done New false
23942 8265 Acked 9264 Write 8 Done Done New false
23943 8266 Acked 9265 Write 8 Done Done New false
23944 8267 Acked 9266 Read 2 Done Done New false
23945 8268 Acked 9267 Read 10 Done Done New false
23946 8269 Acked 9268 Read 9 Done Done New false
23947 8270 Acked 9269 Read 3 Done Done New false
23948 8271 Acked 9270 Read 7 Done Done New false
23949 8272 Acked 9271 Write 6 Done Done New false
23950 8273 Acked 9272 Flush 0 Done Done New false
23951 8274 Acked 9273 Read 7 Done Done New false
23952 8275 Acked 9274 Write 7 Done Done New false
23953 8276 Acked 9275 Read 9 Done Done New false
23954 8277 Acked 9276 Read 2 Done Done New false
23955 8278 Acked 9277 Read 10 Done Done New false
23956 8279 Acked 9278 Read 9 Done Done New false
23957 8280 Acked 9279 Flush 0 Done Done New false
23958 8281 Acked 9280 Read 7 Done Done New false
23959 8282 Acked 9281 Read 1 Done Done New false
23960 8283 Acked 9282 Write 8 Done Done New false
23961 8284 Acked 9283 Read 3 Done Done New false
23962 8285 Acked 9284 Write 9 Done Done New false
23963 8286 Acked 9285 Write 6 Done Done New false
23964 8287 Acked 9286 Read 1 Done Done New false
23965 8288 Acked 9287 Write 4 Done Done New false
23966 8289 Acked 9288 Write 10 Done Done New false
23967 8290 Acked 9289 Read 9 Done Done New false
23968 8291 Acked 9290 Write 7 Done Done New false
23969 8292 Acked 9291 Write 4 Done Done New false
23970 8293 Acked 9292 Write 4 Done Done New false
23971 8294 Acked 9293 Write 9 Done Done New false
23972 8295 Acked 9294 Read 2 Done Done New false
23973 8296 Acked 9295 Read 8 Done Done New false
23974 8297 Acked 9296 Read 4 Done Done New false
23975 8298 Acked 9297 Read 10 Done Done New false
23976 8299 Acked 9298 Write 3 Done Done New false
23977 8300 Acked 9299 Flush 0 Done Done New false
23978 8301 Acked 9300 Write 4 Done Done New false
23979 8302 Acked 9301 Flush 0 Done Done New false
23980 8303 Acked 9302 Read 3 Done Done New false
23981 8304 Acked 9303 Read 1 Done Done New false
23982 8305 Acked 9304 Read 10 Done Done New false
23983 8306 Acked 9305 Write 10 Done Done New false
23984 8307 Acked 9306 Read 3 Done Done New false
23985 8308 Acked 9307 Read 10 Done Done New false
23986 8309 Acked 9308 Write 2 Done Done New false
23987 8310 Acked 9309 Read 8 Done Done New false
23988 8311 Acked 9310 Write 8 Done Done New false
23989 8312 Acked 9311 Write 4 Done Done New false
23990 8313 Acked 9312 Read 6 Done Done New false
23991 8314 Acked 9313 Write 6 Done Done New false
23992 8315 Acked 9314 Read 1 Done Done New false
23993 8316 Acked 9315 Read 7 Done Done New false
23994 8317 Acked 9316 Read 10 Done Done New false
23995 8318 Acked 9317 Read 10 Done Done New false
23996 8319 Acked 9318 Read 4 Done Done New false
23997 8320 Acked 9319 Write 3 Done Done New false
23998 8321 Acked 9320 Write 2 Done Done New false
23999 8322 Acked 9321 Read 4 Done Done New false
24000 8323 Acked 9322 Write 2 Done Done New false
24001 8324 Acked 9323 Write 3 Done Done New false
24002 8325 Acked 9324 Flush 0 Done Done New false
24003 8326 Acked 9325 Read 8 Done Done New false
24004 8327 Acked 9326 Write 9 Done Done New false
24005 8328 Acked 9327 Flush 0 Done Done New false
24006 8329 Acked 9328 Read 1 Done Done New false
24007 8330 Acked 9329 Read 9 Done Done New false
24008 8331 Acked 9330 Read 3 Done Done New false
24009 8332 Acked 9331 Write 7 Done Done New false
24010 8333 Acked 9332 Read 9 Done Done New false
24011 8334 Acked 9333 Write 6 Done Done New false
24012 8335 Acked 9334 Read 6 Done Done New false
24013 8336 Acked 9335 Write 3 Done Done New false
24014 8337 Acked 9336 Read 9 Done Done New false
24015 8338 Acked 9337 Write 7 Done Done New false
24016 8339 Acked 9338 Write 5 Done Done New false
24017 8340 Acked 9339 Read 3 Done Done New false
24018 8341 Acked 9340 Write 1 Done Done New false
24019 8342 Acked 9341 Write 1 Done Done New false
24020 8343 Acked 9342 Read 4 Done Done New false
24021 8344 Acked 9343 Read 6 Done Done New false
24022 8345 Acked 9344 Flush 0 Done Done New false
24023 8346 Acked 9345 Flush 0 Done Done New false
24024 8347 Acked 9346 Read 5 Done Done New false
24025 8348 Acked 9347 Read 9 Done Done New false
24026 8349 Acked 9348 Write 6 Done Done New false
24027 8350 Acked 9349 Read 2 Done Done New false
24028 8351 Acked 9350 Read 4 Done Done New false
24029 8352 Acked 9351 Read 5 Done Done New false
24030 8353 Acked 9352 Read 7 Done Done New false
24031 8354 Acked 9353 Write 6 Done Done New false
24032 8355 Acked 9354 Read 7 Done Done New false
24033 8356 Acked 9355 Read 6 Done Done New false
24034 8357 Acked 9356 Write 4 Done Done New false
24035 8358 Acked 9357 Flush 0 Done Done New false
24036 8359 Acked 9358 Read 3 Done Done New false
24037 8360 Acked 9359 Read 10 Done Done New false
24038 8361 Acked 9360 Write 1 Done Done New false
24039 8362 Acked 9361 Read 6 Done Done New false
24040 8363 Acked 9362 Read 3 Done Done New false
24041 8364 Acked 9363 Read 10 Done Done New false
24042 8365 Acked 9364 Write 1 Done Done New false
24043 8366 Acked 9365 Read 3 Done Done New false
24044 8367 Acked 9366 Read 7 Done Done New false
24045 8368 Acked 9367 Read 2 Done Done New false
24046 8369 Acked 9368 Read 1 Done Done New false
24047 8370 Acked 9369 Write 2 Done Done New false
24048 8371 Acked 9370 Write 4 Done Done New false
24049 8372 Acked 9371 Read 2 Done Done New false
24050 8373 Acked 9372 Write 1 Done Done New false
24051 8374 Acked 9373 Write 4 Done Done New false
24052 8375 Acked 9374 Read 1 Done Done New false
24053 8376 Acked 9375 Read 8 Done Done New false
24054 8377 Acked 9376 Write 1 Done Done New false
24055 8378 Acked 9377 Write 10 Done Done New false
24056 8379 Acked 9378 Write 3 Done Done New false
24057 8380 Acked 9379 Read 4 Done Done New false
24058 8381 Acked 9380 Write 3 Done Done New false
24059 8382 Acked 9381 Flush 0 Done Done New false
24060 8383 Acked 9382 Read 6 Done Done New false
24061 8384 Acked 9383 Read 10 Done Done New false
24062 8385 Acked 9384 Read 4 Done Done New false
24063 8386 Acked 9385 Flush 0 Done Done New false
24064 8387 Acked 9386 Write 10 Done Done New false
24065 8388 Acked 9387 Read 5 Done Done New false
24066 8389 Acked 9388 Write 6 Done Done New false
24067 8390 Acked 9389 Read 6 Done Done New false
24068 8391 Acked 9390 Read 2 Done Done New false
24069 8392 Acked 9391 Read 2 Done Done New false
24070 8393 Acked 9392 Read 8 Done Done New false
24071 8394 Acked 9393 Write 7 Done Done New false
24072 8395 Acked 9394 Read 10 Done Done New false
24073 8396 Acked 9395 Write 9 Done Done New false
24074 8397 Acked 9396 Flush 0 Done Done New false
24075 8398 Acked 9397 Read 8 Done Done New false
24076 8399 Acked 9398 Read 7 Done Done New false
24077 8400 Acked 9399 Read 2 Done Done New false
24078 8401 Acked 9400 Read 5 Done Done New false
24079 8402 Acked 9401 Read 6 Done Done New false
24080 8403 Acked 9402 Read 9 Done Done New false
24081 8404 Acked 9403 Read 8 Done Done New false
24082 8405 Acked 9404 Read 1 Done Done New false
24083 8406 Acked 9405 Write 9 Done Done New false
24084 8407 Acked 9406 Write 4 Done Done New false
24085 8408 Acked 9407 Read 1 Done Done New false
24086 8409 Acked 9408 Write 5 Done Done New false
24087 8410 Acked 9409 Read 9 Done Done New false
24088 8411 Acked 9410 Read 4 Done Done New false
24089 8412 Acked 9411 Read 8 Done Done New false
24090 8413 Acked 9412 Write 4 Done Done New false
24091 8414 Acked 9413 Read 7 Done Done New false
24092 8415 Acked 9414 Read 10 Done Done New false
24093 8416 Acked 9415 Read 4 Done Done New false
24094 8417 Acked 9416 Write 6 Done Done New false
24095 8418 Acked 9417 Write 10 Done Done New false
24096 8419 Acked 9418 Read 9 Done Done New false
24097 8420 Acked 9419 Write 5 Done Done New false
24098 8421 Acked 9420 Flush 0 Done Done New false
24099 8422 Acked 9421 Write 6 Done Done New false
24100 8423 Acked 9422 Write 3 Done Done New false
24101 8424 Acked 9423 Write 10 Done Done New false
24102 8425 Acked 9424 Write 8 Done Done New false
24103 8426 Acked 9425 Write 8 Done Done New false
24104 8427 Acked 9426 Read 5 Done Done New false
24105 8428 Acked 9427 Read 5 Done Done New false
24106 8429 Acked 9428 Flush 0 Done Done New false
24107 8430 Acked 9429 Write 2 Done Done New false
24108 8431 Acked 9430 Read 3 Done Done New false
24109 8432 Acked 9431 Write 3 Done Done New false
24110 8433 Acked 9432 Read 1 Done Done New false
24111 8434 Acked 9433 Write 10 Done Done New false
24112 8435 Acked 9434 Read 3 Done Done New false
24113 8436 Acked 9435 Write 2 Done Done New false
24114 8437 Acked 9436 Read 7 Done Done New false
24115 8438 Acked 9437 Read 5 Done Done New false
24116 8439 Acked 9438 Read 7 Done Done New false
24117 8440 Acked 9439 Flush 0 Done Done New false
24118 STATES DS:0 DS:1 DS:2 TOTAL
24119 New 0 0 301 301
24120 Sent 0 0 0 0
24121 Done 301 301 0 602
24122 Skipped 0 0 0 0
24123 Error 0 0 0 0
24124 Last Flush: 9439 9439 9138
24125 Downstairs last five completed: 9138 9137 9136 9135 9134
24126 Upstairs last five completed: 8440 8439 8438 8437 8436
24127 CLIENT: Up:0 ds:301 act:2
241282024-02-27T21:49:24.503ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
241292024-02-27T21:49:24.503ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
241302024-02-27T21:49:24.503ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
241312024-02-27T21:49:24.503ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
241322024-02-27T21:49:24.504ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
241332024-02-27T21:49:24.504ZINFOcrucible: [2] client re-new 301 jobs since flush 9138 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
241342024-02-27T21:49:24.504ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
241352024-02-27T21:49:25.505ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
241362024-02-27T21:49:25.505ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
241372024-02-27T21:49:25.505ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
241382024-02-27T21:49:25.505ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
241392024-02-27T21:49:25.505ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
241402024-02-27T21:49:25.505ZINFOcrucible: [2] client re-new 301 jobs since flush 9138 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
241412024-02-27T21:49:25.505ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
241422024-02-27T21:49:26.506ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
241432024-02-27T21:49:26.506ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
241442024-02-27T21:49:26.506ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
241452024-02-27T21:49:26.506ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
241462024-02-27T21:49:26.506ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
241472024-02-27T21:49:26.506ZINFOcrucible: [2] client re-new 301 jobs since flush 9138 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
241482024-02-27T21:49:26.507ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
24149 [8830] got start action
24150 [8830] I am going to restart
24151 Starting downstairs at port 8830
24152 Make output file at "/tmp/dsc/downstairs-8830.txt"
24153 [8830][2] reports Starting
241542024-02-27T21:49:27.507ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
241552024-02-27T21:49:27.507ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
241562024-02-27T21:49:27.507ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
241572024-02-27T21:49:27.507ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
241582024-02-27T21:49:27.507ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
241592024-02-27T21:49:27.508ZINFOcrucible: [2] client re-new 301 jobs since flush 9138 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
241602024-02-27T21:49:27.508ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
24161 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1195)
24162 [8830][2] reports Running
24163 ----------------------------------------------------------------
24164 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24165 States: Active Active Offline
24166 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
24167 8140 Acked 9139 Read 3 Done Done New true
24168 8141 Acked 9140 Flush 0 Done Done New true
24169 8142 Acked 9141 Write 1 Done Done New true
24170 8143 Acked 9142 Read 1 Done Done New true
24171 8144 Acked 9143 Read 1 Done Done New true
24172 8145 Acked 9144 Read 1 Done Done New true
24173 8146 Acked 9145 Write 3 Done Done New true
24174 8147 Acked 9146 Read 8 Done Done New true
24175 8148 Acked 9147 Read 9 Done Done New true
24176 8149 Acked 9148 Read 4 Done Done New true
24177 8150 Acked 9149 Write 2 Done Done New true
24178 8151 Acked 9150 Write 5 Done Done New true
24179 8152 Acked 9151 Write 3 Done Done New true
24180 8153 Acked 9152 Read 1 Done Done New true
24181 8154 Acked 9153 Read 8 Done Done New true
24182 8155 Acked 9154 Read 6 Done Done New true
24183 8156 Acked 9155 Flush 0 Done Done New true
24184 8157 Acked 9156 Read 1 Done Done New true
24185 8158 Acked 9157 Read 9 Done Done New true
24186 8159 Acked 9158 Write 7 Done Done New true
24187 8160 Acked 9159 Read 8 Done Done New true
24188 8161 Acked 9160 Flush 0 Done Done New true
24189 8162 Acked 9161 Write 3 Done Done New true
24190 8163 Acked 9162 Read 10 Done Done New true
24191 8164 Acked 9163 Write 4 Done Done New true
24192 8165 Acked 9164 Read 8 Done Done New true
24193 8166 Acked 9165 Read 5 Done Done New true
24194 8167 Acked 9166 Read 2 Done Done New true
24195 8168 Acked 9167 Read 6 Done Done New true
24196 8169 Acked 9168 Read 8 Done Done New true
24197 8170 Acked 9169 Write 6 Done Done New true
24198 8171 Acked 9170 Read 10 Done Done New true
24199 8172 Acked 9171 Write 7 Done Done New true
24200 8173 Acked 9172 Read 8 Done Done New true
24201 8174 Acked 9173 Read 9 Done Done New true
24202 8175 Acked 9174 Write 3 Done Done New true
24203 8176 Acked 9175 Write 6 Done Done New true
24204 8177 Acked 9176 Write 8 Done Done New true
24205 8178 Acked 9177 Write 3 Done Done New true
24206 8179 Acked 9178 Read 4 Done Done New true
24207 8180 Acked 9179 Write 8 Done Done New true
24208 8181 Acked 9180 Read 6 Done Done New true
24209 8182 Acked 9181 Write 5 Done Done New true
24210 8183 Acked 9182 Write 6 Done Done New true
24211 8184 Acked 9183 Write 7 Done Done New true
24212 8185 Acked 9184 Read 10 Done Done New true
24213 8186 Acked 9185 Flush 0 Done Done New true
24214 8187 Acked 9186 Read 1 Done Done New true
24215 8188 Acked 9187 Write 8 Done Done New true
24216 8189 Acked 9188 Write 5 Done Done New true
24217 8190 Acked 9189 Read 5 Done Done New true
24218 8191 Acked 9190 Read 10 Done Done New true
24219 8192 Acked 9191 Read 7 Done Done New true
24220 8193 Acked 9192 Write 6 Done Done New true
24221 8194 Acked 9193 Write 2 Done Done New true
24222 8195 Acked 9194 Write 4 Done Done New true
24223 8196 Acked 9195 Flush 0 Done Done New true
24224 8197 Acked 9196 Read 1 Done Done New true
24225 8198 Acked 9197 Write 2 Done Done New true
24226 8199 Acked 9198 Read 3 Done Done New true
24227 8200 Acked 9199 Read 8 Done Done New true
24228 8201 Acked 9200 Read 10 Done Done New true
24229 8202 Acked 9201 Write 7 Done Done New true
24230 8203 Acked 9202 Write 6 Done Done New true
24231 8204 Acked 9203 Write 6 Done Done New true
24232 8205 Acked 9204 Read 9 Done Done New true
24233 8206 Acked 9205 Write 3 Done Done New true
24234 8207 Acked 9206 Write 5 Done Done New true
24235 8208 Acked 9207 Write 5 Done Done New true
24236 8209 Acked 9208 Read 5 Done Done New true
24237 8210 Acked 9209 Read 5 Done Done New true
24238 8211 Acked 9210 Read 7 Done Done New true
24239 8212 Acked 9211 Write 5 Done Done New true
24240 8213 Acked 9212 Read 4 Done Done New true
24241 8214 Acked 9213 Write 5 Done Done New true
24242 8215 Acked 9214 Read 8 Done Done New true
24243 8216 Acked 9215 Read 6 Done Done New true
24244 8217 Acked 9216 Read 10 Done Done New true
24245 8218 Acked 9217 Write 10 Done Done New true
24246 8219 Acked 9218 Read 7 Done Done New true
24247 8220 Acked 9219 Read 1 Done Done New true
24248 8221 Acked 9220 Read 2 Done Done New true
24249 8222 Acked 9221 Flush 0 Done Done New true
24250 8223 Acked 9222 Write 4 Done Done New true
24251 8224 Acked 9223 Read 1 Done Done New true
24252 8225 Acked 9224 Write 6 Done Done New true
24253 8226 Acked 9225 Write 8 Done Done New true
24254 8227 Acked 9226 Write 9 Done Done New true
24255 8228 Acked 9227 Flush 0 Done Done New true
24256 8229 Acked 9228 Read 4 Done Done New true
24257 8230 Acked 9229 Write 4 Done Done New true
24258 8231 Acked 9230 Write 8 Done Done New true
24259 8232 Acked 9231 Read 2 Done Done New true
24260 8233 Acked 9232 Read 8 Done Done New true
24261 8234 Acked 9233 Read 4 Done Done New true
24262 8235 Acked 9234 Read 1 Done Done New true
24263 8236 Acked 9235 Read 7 Done Done New true
24264 8237 Acked 9236 Write 10 Done Done New true
24265 8238 Acked 9237 Read 2 Done Done New true
24266 8239 Acked 9238 Write 7 Done Done New true
24267 8240 Acked 9239 Read 9 Done Done New true
24268 8241 Acked 9240 Read 6 Done Done New true
24269 8242 Acked 9241 Read 9 Done Done New true
24270 8243 Acked 9242 Read 7 Done Done New true
24271 8244 Acked 9243 Write 9 Done Done New true
24272 8245 Acked 9244 Read 3 Done Done New true
24273 8246 Acked 9245 Read 6 Done Done New true
24274 8247 Acked 9246 Read 4 Done Done New true
24275 8248 Acked 9247 Flush 0 Done Done New true
24276 8249 Acked 9248 Read 8 Done Done New true
24277 8250 Acked 9249 Read 3 Done Done New true
24278 8251 Acked 9250 Write 6 Done Done New true
24279 8252 Acked 9251 Write 9 Done Done New true
24280 8253 Acked 9252 Read 2 Done Done New true
24281 8254 Acked 9253 Write 3 Done Done New true
24282 8255 Acked 9254 Write 5 Done Done New true
24283 8256 Acked 9255 Write 1 Done Done New true
24284 8257 Acked 9256 Write 5 Done Done New true
24285 8258 Acked 9257 Read 9 Done Done New true
24286 8259 Acked 9258 Read 7 Done Done New true
24287 8260 Acked 9259 Flush 0 Done Done New true
24288 8261 Acked 9260 Read 3 Done Done New true
24289 8262 Acked 9261 Read 4 Done Done New true
24290 8263 Acked 9262 Read 10 Done Done New true
24291 8264 Acked 9263 Read 3 Done Done New true
24292 8265 Acked 9264 Write 8 Done Done New true
24293 8266 Acked 9265 Write 8 Done Done New true
24294 8267 Acked 9266 Read 2 Done Done New true
24295 8268 Acked 9267 Read 10 Done Done New true
24296 8269 Acked 9268 Read 9 Done Done New true
24297 8270 Acked 9269 Read 3 Done Done New true
24298 8271 Acked 9270 Read 7 Done Done New true
24299 8272 Acked 9271 Write 6 Done Done New true
24300 8273 Acked 9272 Flush 0 Done Done New true
24301 8274 Acked 9273 Read 7 Done Done New true
24302 8275 Acked 9274 Write 7 Done Done New true
24303 8276 Acked 9275 Read 9 Done Done New true
24304 8277 Acked 9276 Read 2 Done Done New true
24305 8278 Acked 9277 Read 10 Done Done New true
24306 8279 Acked 9278 Read 9 Done Done New true
24307 8280 Acked 9279 Flush 0 Done Done New true
24308 8281 Acked 9280 Read 7 Done Done New true
24309 8282 Acked 9281 Read 1 Done Done New true
24310 8283 Acked 9282 Write 8 Done Done New true
24311 8284 Acked 9283 Read 3 Done Done New true
24312 8285 Acked 9284 Write 9 Done Done New true
24313 8286 Acked 9285 Write 6 Done Done New true
24314 8287 Acked 9286 Read 1 Done Done New true
24315 8288 Acked 9287 Write 4 Done Done New true
24316 8289 Acked 9288 Write 10 Done Done New true
24317 8290 Acked 9289 Read 9 Done Done New true
24318 8291 Acked 9290 Write 7 Done Done New true
24319 8292 Acked 9291 Write 4 Done Done New true
24320 8293 Acked 9292 Write 4 Done Done New true
24321 8294 Acked 9293 Write 9 Done Done New true
24322 8295 Acked 9294 Read 2 Done Done New true
24323 8296 Acked 9295 Read 8 Done Done New true
24324 8297 Acked 9296 Read 4 Done Done New true
24325 8298 Acked 9297 Read 10 Done Done New true
24326 8299 Acked 9298 Write 3 Done Done New true
24327 8300 Acked 9299 Flush 0 Done Done New true
24328 8301 Acked 9300 Write 4 Done Done New true
24329 8302 Acked 9301 Flush 0 Done Done New true
24330 8303 Acked 9302 Read 3 Done Done New true
24331 8304 Acked 9303 Read 1 Done Done New true
24332 8305 Acked 9304 Read 10 Done Done New true
24333 8306 Acked 9305 Write 10 Done Done New true
24334 8307 Acked 9306 Read 3 Done Done New true
24335 8308 Acked 9307 Read 10 Done Done New true
24336 8309 Acked 9308 Write 2 Done Done New true
24337 8310 Acked 9309 Read 8 Done Done New true
24338 8311 Acked 9310 Write 8 Done Done New true
24339 8312 Acked 9311 Write 4 Done Done New true
24340 8313 Acked 9312 Read 6 Done Done New true
24341 8314 Acked 9313 Write 6 Done Done New true
24342 8315 Acked 9314 Read 1 Done Done New true
24343 8316 Acked 9315 Read 7 Done Done New true
24344 8317 Acked 9316 Read 10 Done Done New true
24345 8318 Acked 9317 Read 10 Done Done New true
24346 8319 Acked 9318 Read 4 Done Done New true
24347 8320 Acked 9319 Write 3 Done Done New true
24348 8321 Acked 9320 Write 2 Done Done New true
24349 8322 Acked 9321 Read 4 Done Done New true
24350 8323 Acked 9322 Write 2 Done Done New true
24351 8324 Acked 9323 Write 3 Done Done New true
24352 8325 Acked 9324 Flush 0 Done Done New true
24353 8326 Acked 9325 Read 8 Done Done New true
24354 8327 Acked 9326 Write 9 Done Done New true
24355 8328 Acked 9327 Flush 0 Done Done New true
24356 8329 Acked 9328 Read 1 Done Done New true
24357 8330 Acked 9329 Read 9 Done Done New true
24358 8331 Acked 9330 Read 3 Done Done New true
24359 8332 Acked 9331 Write 7 Done Done New true
24360 8333 Acked 9332 Read 9 Done Done New true
24361 8334 Acked 9333 Write 6 Done Done New true
24362 8335 Acked 9334 Read 6 Done Done New true
24363 8336 Acked 9335 Write 3 Done Done New true
24364 8337 Acked 9336 Read 9 Done Done New true
24365 8338 Acked 9337 Write 7 Done Done New true
24366 8339 Acked 9338 Write 5 Done Done New true
24367 8340 Acked 9339 Read 3 Done Done New true
24368 8341 Acked 9340 Write 1 Done Done New true
24369 8342 Acked 9341 Write 1 Done Done New true
24370 8343 Acked 9342 Read 4 Done Done New true
24371 8344 Acked 9343 Read 6 Done Done New true
24372 8345 Acked 9344 Flush 0 Done Done New true
24373 8346 Acked 9345 Flush 0 Done Done New true
24374 8347 Acked 9346 Read 5 Done Done New true
24375 8348 Acked 9347 Read 9 Done Done New true
24376 8349 Acked 9348 Write 6 Done Done New true
24377 8350 Acked 9349 Read 2 Done Done New true
24378 8351 Acked 9350 Read 4 Done Done New true
24379 8352 Acked 9351 Read 5 Done Done New true
24380 8353 Acked 9352 Read 7 Done Done New true
24381 8354 Acked 9353 Write 6 Done Done New true
24382 8355 Acked 9354 Read 7 Done Done New true
24383 8356 Acked 9355 Read 6 Done Done New true
24384 8357 Acked 9356 Write 4 Done Done New true
24385 8358 Acked 9357 Flush 0 Done Done New true
24386 8359 Acked 9358 Read 3 Done Done New true
24387 8360 Acked 9359 Read 10 Done Done New true
24388 8361 Acked 9360 Write 1 Done Done New true
24389 8362 Acked 9361 Read 6 Done Done New true
24390 8363 Acked 9362 Read 3 Done Done New true
24391 8364 Acked 9363 Read 10 Done Done New true
24392 8365 Acked 9364 Write 1 Done Done New true
24393 8366 Acked 9365 Read 3 Done Done New true
24394 8367 Acked 9366 Read 7 Done Done New true
24395 8368 Acked 9367 Read 2 Done Done New true
24396 8369 Acked 9368 Read 1 Done Done New true
24397 8370 Acked 9369 Write 2 Done Done New true
24398 8371 Acked 9370 Write 4 Done Done New true
24399 8372 Acked 9371 Read 2 Done Done New true
24400 8373 Acked 9372 Write 1 Done Done New true
24401 8374 Acked 9373 Write 4 Done Done New true
24402 8375 Acked 9374 Read 1 Done Done New true
24403 8376 Acked 9375 Read 8 Done Done New true
24404 8377 Acked 9376 Write 1 Done Done New true
24405 8378 Acked 9377 Write 10 Done Done New true
24406 8379 Acked 9378 Write 3 Done Done New true
24407 8380 Acked 9379 Read 4 Done Done New true
24408 8381 Acked 9380 Write 3 Done Done New true
24409 8382 Acked 9381 Flush 0 Done Done New true
24410 8383 Acked 9382 Read 6 Done Done New true
24411 8384 Acked 9383 Read 10 Done Done New true
24412 8385 Acked 9384 Read 4 Done Done New true
24413 8386 Acked 9385 Flush 0 Done Done New true
24414 8387 Acked 9386 Write 10 Done Done New true
24415 8388 Acked 9387 Read 5 Done Done New true
24416 8389 Acked 9388 Write 6 Done Done New true
24417 8390 Acked 9389 Read 6 Done Done New true
24418 8391 Acked 9390 Read 2 Done Done New true
24419 8392 Acked 9391 Read 2 Done Done New true
24420 8393 Acked 9392 Read 8 Done Done New true
24421 8394 Acked 9393 Write 7 Done Done New true
24422 8395 Acked 9394 Read 10 Done Done New true
24423 8396 Acked 9395 Write 9 Done Done New true
24424 8397 Acked 9396 Flush 0 Done Done New true
24425 8398 Acked 9397 Read 8 Done Done New true
24426 8399 Acked 9398 Read 7 Done Done New true
24427 8400 Acked 9399 Read 2 Done Done New true
24428 8401 Acked 9400 Read 5 Done Done New true
24429 8402 Acked 9401 Read 6 Done Done New true
24430 8403 Acked 9402 Read 9 Done Done New true
24431 8404 Acked 9403 Read 8 Done Done New true
24432 8405 Acked 9404 Read 1 Done Done New true
24433 8406 Acked 9405 Write 9 Done Done New true
24434 8407 Acked 9406 Write 4 Done Done New true
24435 8408 Acked 9407 Read 1 Done Done New true
24436 8409 Acked 9408 Write 5 Done Done New true
24437 8410 Acked 9409 Read 9 Done Done New true
24438 8411 Acked 9410 Read 4 Done Done New true
24439 8412 Acked 9411 Read 8 Done Done New true
24440 8413 Acked 9412 Write 4 Done Done New true
24441 8414 Acked 9413 Read 7 Done Done New true
24442 8415 Acked 9414 Read 10 Done Done New true
24443 8416 Acked 9415 Read 4 Done Done New true
24444 8417 Acked 9416 Write 6 Done Done New true
24445 8418 Acked 9417 Write 10 Done Done New true
24446 8419 Acked 9418 Read 9 Done Done New true
24447 8420 Acked 9419 Write 5 Done Done New true
24448 8421 Acked 9420 Flush 0 Done Done New true
24449 8422 Acked 9421 Write 6 Done Done New true
24450 8423 Acked 9422 Write 3 Done Done New true
24451 8424 Acked 9423 Write 10 Done Done New true
24452 8425 Acked 9424 Write 8 Done Done New true
24453 8426 Acked 9425 Write 8 Done Done New true
24454 8427 Acked 9426 Read 5 Done Done New true
24455 8428 Acked 9427 Read 5 Done Done New true
24456 8429 Acked 9428 Flush 0 Done Done New true
24457 8430 Acked 9429 Write 2 Done Done New true
24458 8431 Acked 9430 Read 3 Done Done New true
24459 8432 Acked 9431 Write 3 Done Done New true
24460 8433 Acked 9432 Read 1 Done Done New true
24461 8434 Acked 9433 Write 10 Done Done New true
24462 8435 Acked 9434 Read 3 Done Done New true
24463 8436 Acked 9435 Write 2 Done Done New true
24464 8437 Acked 9436 Read 7 Done Done New true
24465 8438 Acked 9437 Read 5 Done Done New true
24466 8439 Acked 9438 Read 7 Done Done New true
24467 8440 Acked 9439 Flush 0 Done Done New true
24468 STATES DS:0 DS:1 DS:2 TOTAL
24469 New 0 0 301 301
24470 Sent 0 0 0 0
24471 Done 301 301 0 602
24472 Skipped 0 0 0 0
24473 Error 0 0 0 0
24474 Last Flush: 9439 9439 9138
24475 Downstairs last five completed: 9138 9137 9136 9135 9134
24476 Upstairs last five completed: 8440 8439 8438 8437 8436
24477 CLIENT: Up:0 ds:301 act:2
244782024-02-27T21:49:28.509ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
244792024-02-27T21:49:28.509ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
244802024-02-27T21:49:28.511ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
244812024-02-27T21:49:28.511ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
244822024-02-27T21:49:28.511ZINFOcrucible: send last flush ID to this DS: 9138 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
244832024-02-27T21:49:28.511ZINFOcrucible: Replied this last flush ID: 9138 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
244842024-02-27T21:49:28.511ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
244852024-02-27T21:49:28.511ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
244862024-02-27T21:49:28.511ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
244872024-02-27T21:49:28.511ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
24488 ----------------------------------------------------------------
24489 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
24490 Downstairs last five completed: 9439 9438 9437 9436 9435
24491 Upstairs last five completed: 8440 8439 8438 8437 8436
24492 CLIENT: Up:0 ds:0 act:3
24493 Replay: All jobs finished, all DS active.
24494 Main task has work to do, go find it
24495 got dsc Stop(2)
24496 stop 2
24497 Feb 27 21:49:31.848 INFO[8830] Got stop action so:false kr:false
24498 request completed, latency_us: 323, response_code: 204, uri: /stop/cid/2, method: POST, req_id: f6b9c304-c332-426a-92ca-f145bd370571, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
244992024-02-27T21:49:31.849ZWARNcrucible: downstairs disconnected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
245002024-02-27T21:49:31.849ZWARNcrucible: client task is sending Done(Finished) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
24501 Feb 27 21:49:31.849 INFO request completed, latency_us: 141, response_code:{"msg":"downstairs task for 2 stopped due to Finished","v":0,"name":"crucible","level":40,"time":"2024-02-27T21:49:31.849544547Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
24502 200, uri: /state/cid/2, method: GET, req_id: 35ae4cc4-9043-4ef1-aa67-12efc91f34a7, remote_addr: 127.0.{"msg":"Gone missing, transition from Active to Offline","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:49:31.849585612Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"client":"2","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
24503 0{"msg":"[2] client re-new 0 jobs since flush 9439","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:49:31.849617988Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"downstairs","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
24504 .1:42901, local_addr: 127.0.0.1:9998
245052024-02-27T21:49:31.849ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
24506 [8830] Exited with: ExitStatus(unix_wait_status(9))
24507 [8830][2] reports Exit
245082024-02-27T21:49:32.850ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
245092024-02-27T21:49:32.850ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
245102024-02-27T21:49:32.850ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
245112024-02-27T21:49:32.850ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
245122024-02-27T21:49:32.850ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
245132024-02-27T21:49:32.850ZINFOcrucible: [2] client re-new 0 jobs since flush 9439 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
245142024-02-27T21:49:32.850ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
245152024-02-27T21:49:33.851ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
245162024-02-27T21:49:33.851ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
245172024-02-27T21:49:33.851ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
245182024-02-27T21:49:33.851ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
245192024-02-27T21:49:33.851ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
245202024-02-27T21:49:33.851ZINFOcrucible: [2] client re-new 0 jobs since flush 9439 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
245212024-02-27T21:49:33.851ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
245222024-02-27T21:49:34.852ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
245232024-02-27T21:49:34.852ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
245242024-02-27T21:49:34.852ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
245252024-02-27T21:49:34.852ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
245262024-02-27T21:49:34.852ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
245272024-02-27T21:49:34.852ZINFOcrucible: [2] client re-new 0 jobs since flush 9439 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
245282024-02-27T21:49:34.852ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
24529 Feb 27 21:49:35.850 INFO request completed, latency_us: 209, response_code: 200, uri: /state/cid/2, method: GET, req_id: fcca47f7-ad74-4a96-95c2-a8b1b5c78807, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
24530 001/300 Write block 1932 len 36864 data: 7 6 6 6 5 6 6 6 8
24531 002/300 Read block 3790 len 36864
245322024-02-27T21:49:35.853ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
245332024-02-27T21:49:35.853ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
245342024-02-27T21:49:35.853ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
245352024-02-27T21:49:35.853ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
245362024-02-27T21:49:35.853ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
245372024-02-27T21:49:35.853ZINFOcrucible: [2] client re-new 2 jobs since flush 9439 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
245382024-02-27T21:49:35.853ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
24539 003/300 Write block 3383 len 16384 data: 6 6 5 7
24540 004/300 Write block 4916 len 40960 data: 5 4 6 5 4 5 7 9 10 10
24541 005/300 Read block 4947 len 28672
24542 006/300 Write block 3910 len 28672 data: 3 4 4 4 6 6 7
24543 007/300 Read block 1969 len 36864
24544 008/300 Read block 3148 len 20480
24545 009/300 Read block 2654 len 36864
24546 010/300 Read block 721 len 16384
24547 011/300 Read block 225 len 16384
24548 012/300 Write block 1025 len 4096 data: 5
24549 013/300 Read block 2219 len 12288
24550 014/300 Read block 335 len 28672
24551 015/300 Read block 2102 len 28672
24552 016/300 Write block 298 len 40960 data: 3 3 4 3 4 4 4 4 4 5
24553 017/300 Write block 2532 len 20480 data: 6 5 5 4 5
24554 018/300 Read block 1951 len 36864
24555 019/300 Read block 1456 len 36864
24556 020/300 Read block 3672 len 28672
24557 021/300 Write block 525 len 24576 data: 9 7 5 5 4 4
24558 022/300 Write block 2759 len 20480 data: 6 6 7 7 7
24559 023/300 Write block 3562 len 28672 data: 9 10 5 4 4 5 6
24560 024/300 Flush
24561 025/300 Read block 2657 len 4096
24562 026/300 Read block 1383 len 24576
24563 027/300 Read block 1102 len 40960
24564 028/300 Flush
24565 029/300 Write block 1984 len 24576 data: 5 3 3 3 3 5
24566 030/300 Write block 3045 len 28672 data: 6 5 4 4 4 4 3
24567 031/300 Write block 3171 len 12288 data: 8 6 5
24568 032/300 Write block 1205 len 4096 data: 6
24569 033/300 Read block 1755 len 16384
24570 034/300 Read block 2090 len 16384
24571 035/300 Read block 174 len 40960
24572 036/300 Write block 3067 len 20480 data: 7 6 5 5 3
24573 037/300 Write block 2287 len 20480 data: 4 5 5 5 5
24574 038/300 Read block 4839 len 20480
24575 039/300 Read block 4471 len 32768
24576 040/300 Write block 2527 len 24576 data: 3 3 4 4 4 7
24577 041/300 Read block 3565 len 4096
24578 042/300 Read block 2281 len 20480
24579 043/300 Read block 4944 len 36864
24580 044/300 Read block 4249 len 24576
24581 045/300 Write block 269 len 40960 data: 8 8 8 7 5 4 6 5 4 3
24582 046/300 Read block 3378 len 4096
24583 047/300 Flush
24584 048/300 Write block 1344 len 8192 data: 3 3
24585 049/300 Read block 4977 len 36864
24586 050/300 Read block 540 len 20480
24587 051/300 Write block 3546 len 28672 data: 6 6 5 6 5 5 5
24588 052/300 Write block 1326 len 32768 data: 10 8 7 5 7 6 6 6
24589 053/300 Read block 1481 len 36864
24590 054/300 Write block 3972 len 12288 data: 3 3 3
24591 055/300 Read block 4596 len 4096
24592 056/300 Write block 3547 len 28672 data: 7 6 7 6 6 6 5
24593 057/300 Read block 318 len 4096
24594 058/300 Read block 1920 len 8192
24595 059/300 Read block 615 len 28672
24596 060/300 Read block 1141 len 16384
24597 061/300 Write block 556 len 24576 data: 7 6 6 6 5 5
24598 062/300 Read block 3063 len 4096
24599 063/300 Write block 568 len 32768 data: 8 5 4 7 6 6 4 5
24600 064/300 Read block 2910 len 16384
24601 065/300 Write block 1403 len 28672 data: 6 6 6 7 8 8 7
24602 066/300 Flush
24603 067/300 Flush
24604 068/300 Write block 3866 len 28672 data: 4 4 4 4 4 4 3
24605 069/300 Write block 611 len 24576 data: 7 7 6 6 5 4
24606 070/300 Read block 1775 len 16384
24607 071/300 Read block 3750 len 36864
24608 072/300 Read block 454 len 4096
24609 073/300 Write block 3341 len 36864 data: 6 8 9 9 9 8 7 6 7
24610 074/300 Read block 4632 len 12288
24611 075/300 Read block 3164 len 36864
24612 076/300 Flush
24613 077/300 Read block 1157 len 20480
24614 078/300 Read block 4182 len 20480
24615 079/300 Read block 1579 len 20480
24616 080/300 Read block 4412 len 16384
24617 081/300 Read block 1171 len 40960
24618 082/300 Flush
24619 083/300 Flush
24620 084/300 Read block 4989 len 4096
24621 085/300 Read block 1077 len 4096
24622 086/300 Read block 4683 len 4096
24623 087/300 Read block 4665 len 28672
24624 088/300 Flush
24625 089/300 Write block 4101 len 24576 data: 5 5 6 5 4 5
24626 090/300 Write block 3672 len 24576 data: 5 5 5 5 5 8
24627 091/300 Read block 3589 len 16384
24628 092/300 Read block 2401 len 16384
24629 093/300 Write block 1250 len 4096 data: 3
24630 094/300 Read block 4592 len 12288
24631 095/300 Write block 4360 len 24576 data: 5 7 7 8 8 8
24632 096/300 Read block 4936 len 4096
24633 097/300 Read block 1507 len 20480
24634 098/300 Write block 4119 len 8192 data: 5 6
24635 099/300 Read block 1094 len 36864
24636 100/300 Write block 2914 len 4096 data: 8
24637 101/300 Write block 2682 len 16384 data: 7 5 5 5
24638 102/300 Read block 3156 len 28672
24639 103/300 Write block 3101 len 36864 data: 5 7 5 5 7 7 8 7 10
24640 104/300 Read block 2925 len 28672
24641 105/300 Write block 3186 len 8192 data: 2 2
24642 106/300 Write block 1508 len 32768 data: 6 8 8 9 9 6 3 4
24643 107/300 Flush
24644 108/300 Read block 1017 len 24576
24645 109/300 Read block 4611 len 8192
24646 110/300 Flush
24647 111/300 Read block 3087 len 24576
24648 112/300 Read block 4281 len 28672
24649 113/300 Write block 3728 len 24576 data: 5 4 4 4 5 6
24650 114/300 Read block 1780 len 12288
24651 115/300 Write block 2684 len 32768 data: 6 6 5 3 2 3 3 4
24652 116/300 Read block 86 len 8192
24653 117/300 Write block 1570 len 36864 data: 7 4 4 3 3 4 5 6 8
24654 118/300 Read block 2619 len 8192
24655 119/300 Write block 2004 len 8192 data: 6 5
24656 120/300 Read block 2056 len 40960
24657 121/300 Write block 278 len 8192 data: 4 3
24658 122/300 Read block 759 len 4096
24659 123/300 Read block 89 len 4096
24660 124/300 Read block 4028 len 12288
24661 125/300 Read block 619 len 24576
24662 126/300 Read block 4244 len 36864
24663 127/300 Write block 3988 len 4096 data: 3
24664 128/300 Read block 4789 len 28672
24665 129/300 Read block 4951 len 24576
24666 130/300 Read block 949 len 20480
24667 131/300 Read block 4435 len 40960
24668 132/300 Write block 2115 len 36864 data: 5 4 4 4 5 5 6 7 7
24669 133/300 Write block 2727 len 40960 data: 2 3 3 3 4 5 6 7 7 5
24670 134/300 Write block 3371 len 8192 data: 5 8
24671 135/300 Write block 1009 len 36864 data: 3 4 5 5 7 8 6 5 5
24672 136/300 Read block 2831 len 16384
24673 137/300 Write block 1952 len 32768 data: 4 3 3 3 4 4 4 4
24674 138/300 Write block 4126 len 12288 data: 4 5 5
24675 139/300 Write block 2981 len 20480 data: 6 6 6 6 5
24676 140/300 Flush
24677 141/300 Read block 4719 len 36864
24678 142/300 Read block 4831 len 36864
24679 143/300 Read block 3451 len 20480
24680 144/300 Flush
24681 145/300 Write block 3010 len 24576 data: 10 9 9 7 6 6
24682 146/300 Write block 2261 len 4096 data: 6
24683 147/300 Flush
24684 148/300 Write block 3334 len 20480 data: 7 7 6 6 7
24685 149/300 Write block 1173 len 24576 data: 7 9 6 7 5 5
24686 150/300 Write block 4642 len 20480 data: 6 5 4 5 6
24687 151/300 Write block 2730 len 8192 data: 4 5
24688 152/300 Write block 3070 len 12288 data: 6 4 3
24689 153/300 Read block 4087 len 4096
24690 154/300 Write block 3092 len 32768 data: 5 6 7 6 6 7 8 9
24691 155/300 Read block 2651 len 40960
24692 156/300 Read block 1777 len 4096
24693 157/300 Flush
24694 158/300 Read block 2890 len 4096
24695 159/300 Read block 259 len 4096
24696 160/300 Read block 2979 len 8192
24697 161/300 Write block 1738 len 4096 data: 5
24698 162/300 Read block 57 len 20480
24699 163/300 Flush
24700 164/300 Flush
24701 165/300 Read block 1620 len 4096
24702 166/300 Read block 1443 len 8192
24703 167/300 Read block 833 len 12288
24704 168/300 Write block 3277 len 12288 data: 4 4 3
24705 169/300 Read block 4763 len 16384
24706 170/300 Read block 1542 len 20480
24707 171/300 Read block 576 len 8192
24708 172/300 Flush
24709 173/300 Write block 3417 len 32768 data: 7 8 8 8 7 8 7 5
24710 174/300 Read block 4279 len 24576
24711 175/300 Write block 3266 len 36864 data: 5 3 3 3 3 4 6 6 7
24712 176/300 Write block 1932 len 24576 data: 8 7 7 7 6 7
24713 177/300 Write block 1648 len 36864 data: 5 7 6 5 4 5 5 6 6
24714 178/300 Read block 4110 len 36864
24715 179/300 Write block 2177 len 28672 data: 2 3 5 4 5 7 7
24716 180/300 Write block 4684 len 32768 data: 5 4 6 6 5 5 7 7
24717 181/300 Write block 1145 len 16384 data: 9 9 8 8
24718 182/300 Write block 695 len 8192 data: 7 6
24719 183/300 Read block 1301 len 40960
24720 184/300 Write block 1245 len 36864 data: 2 3 4 4 4 4 3 3 3
24721 185/300 Write block 220 len 40960 data: 4 5 4 5 8 8 9 8 6 6
24722 186/300 Read block 2617 len 32768
24723 187/300 Write block 3273 len 20480 data: 7 8 5 4 5
24724 188/300 Read block 2517 len 12288
24725 189/300 Write block 1803 len 28672 data: 4 4 4 4 4 6 5
24726 190/300 Read block 3260 len 24576
24727 191/300 Read block 276 len 24576
24728 192/300 Write block 2120 len 32768 data: 6 7 8 8 6 6 6 7
24729 193/300 Read block 551 len 28672
24730 194/300 Read block 2650 len 32768
24731 195/300 Write block 4138 len 8192 data: 6 4
24732 196/300 Read block 1034 len 24576
24733 197/300 Read block 4592 len 36864
24734 198/300 Read block 2618 len 8192
24735 199/300 Write block 3750 len 24576 data: 5 5 5 8 9 9
24736 200/300 Write block 418 len 32768 data: 8 9 9 8 6 5 7 6
24737 201/300 Write block 3949 len 12288 data: 5 5 5
24738 202/300 Flush
24739 203/300 Read block 4251 len 28672
24740 204/300 Read block 904 len 40960
24741 205/300 Read block 1437 len 20480
24742 206/300 Write block 2723 len 28672 data: 2 3 3 3 3 4 4
24743 207/300 Read block 4215 len 32768
24744 208/300 Read block 3240 len 4096
24745 209/300 Write block 3821 len 20480 data: 5 5 5 5 6
24746 210/300 Read block 3202 len 4096
24747 211/300 Write block 124 len 16384 data: 6 4 4 3
24748 212/300 Write block 3873 len 24576 data: 4 3 3 3 3 4
24749 213/300 Write block 111 len 16384 data: 5 6 7 8
24750 214/300 Write block 1876 len 36864 data: 6 5 3 3 4 5 6 6 5
24751 215/300 Read block 2346 len 8192
24752 216/300 Write block 2315 len 20480 data: 7 7 5 6 6
24753 217/300 Read block 656 len 36864
24754 218/300 Read block 1583 len 36864
24755 219/300 Read block 2943 len 40960
24756 220/300 Read block 1852 len 32768
24757 221/300 Write block 3707 len 20480 data: 4 4 6 4 4
24758 222/300 Read block 4512 len 32768
24759 223/300 Write block 1153 len 4096 data: 5
24760 224/300 Write block 245 len 40960 data: 8 8 7 7 5 5 5 5 5 5
24761 225/300 Read block 449 len 28672
24762 226/300 Read block 886 len 4096
24763 227/300 Write block 3679 len 28672 data: 12 8 7 8 8 6 5
24764 228/300 Flush
24765 229/300 Write block 3938 len 36864 data: 10 9 7 7 7 7 6 8 9
24766 230/300 Read block 1103 len 32768
24767 231/300 Write block 904 len 20480 data: 4 5 6 7 7
24768 232/300 Write block 1825 len 24576 data: 7 8 9 8 7 9
24769 233/300 Read block 3338 len 32768
24770 234/300 Read block 4864 len 32768
24771 235/300 Write block 1024 len 4096 data: 6
24772 236/300 Read block 4428 len 28672
24773 237/300 Read block 1135 len 12288
24774 238/300 Write block 3583 len 36864 data: 3 3 4 4 6 7 7 6 6
24775 239/300 Read block 4940 len 20480
24776 240/300 Write block 659 len 4096 data: 3
24777 241/300 Read block 397 len 36864
24778 242/300 Write block 1684 len 20480 data: 7 7 7 4 4
24779 243/300 Write block 651 len 20480 data: 5 6 5 4 5
24780 244/300 Read block 2245 len 32768
24781 245/300 Read block 2217 len 32768
24782 246/300 Read block 2935 len 12288
24783 247/300 Write block 503 len 40960 data: 9 6 7 7 9 8 8 9 9 10
24784 248/300 Write block 3048 len 16384 data: 5 5 5 4
24785 249/300 Flush
24786 250/300 Read block 2427 len 12288
24787 251/300 Write block 1912 len 8192 data: 4 6
24788 252/300 Read block 312 len 40960
24789 253/300 Read block 1754 len 36864
24790 254/300 Read block 1482 len 32768
24791 255/300 Read block 4373 len 28672
24792 256/300 Read block 1158 len 8192
24793 257/300 Flush
24794 258/300 Read block 2057 len 28672
24795 259/300 Read block 2647 len 36864
24796 260/300 Flush
24797 261/300 Flush
24798 262/300 Write block 3349 len 16384 data: 8 6 4 3
24799 263/300 Write block 297 len 40960 data: 2 4 4 5 4 5 5 5 5 5
24800 264/300 Read block 4046 len 4096
24801 265/300 Write block 1629 len 8192 data: 4 4
24802 266/300 Read block 4089 len 24576
24803 267/300 Read block 616 len 8192
24804 268/300 Write block 4822 len 4096 data: 5
24805 269/300 Write block 1218 len 40960 data: 11 11 10 10 8 6 7 7 9 7
24806 270/300 Read block 1456 len 28672
24807 271/300 Write block 2537 len 16384 data: 6 6 6 7
24808 272/300 Write block 3121 len 4096 data: 5
24809 273/300 Read block 3945 len 24576
24810 274/300 Write block 4602 len 28672 data: 6 7 6 5 5 4 8
24811 275/300 Write block 4477 len 8192 data: 6 7
24812 276/300 Read block 4556 len 24576
24813 277/300 Write block 543 len 20480 data: 5 5 4 3 5
24814 278/300 Read block 3126 len 16384
24815 279/300 Write block 1339 len 4096 data: 3
24816 280/300 Read block 3714 len 36864
24817 281/300 Write block 2908 len 4096 data: 7
24818 282/300 Read block 2571 len 32768
24819 283/300 Write block 4832 len 32768 data: 7 6 7 7 6 6 6 5
24820 284/300 Read block 3046 len 32768
24821 285/300 Read block 3219 len 24576
24822 286/300 Flush
24823 287/300 Write block 1205 len 16384 data: 7 5 6 7
24824 288/300 Read block 1229 len 12288
24825 289/300 Read block 2728 len 12288
24826 290/300 Read block 4326 len 20480
24827 291/300 Read block 2889 len 32768
24828 292/300 Read block 4361 len 24576
24829 293/300 Write block 3160 len 36864 data: 4 5 6 6 8 8 7 7 9
24830 294/300 Read block 2045 len 32768
24831 295/300 Flush
24832 296/300 Write block 1628 len 40960 data: 5 5 5 5 4 4 6 6 5 5
24833 297/300 Flush
24834 298/300 Flush
24835 299/300 Flush
24836 300/300 Flush
24837 Main task has work to do, go find it
24838 got dsc Start(2)
24839 start 2
24840 Feb 27 21:49:36.184 INFO request completed, latency_us: 233, response_code: 204, uri: /start/cid/2, method: POST, req_id: ef2bbd94-202c-4429-a700-1b5e4e2ee36d, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
24841 Replay: started 2, returned:Ok(())
24842 ----------------------------------------------------------------
24843 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
24844 States: Active Active Offline
24845 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
24846 8441 Acked 9440 Write 9 Done Done New true
24847 8442 Acked 9441 Read 9 Done Done New true
24848 8443 Acked 9442 Write 4 Done Done New false
24849 8444 Acked 9443 Write 10 Done Done New false
24850 8445 Acked 9444 Read 7 Done Done New false
24851 8446 Acked 9445 Write 7 Done Done New false
24852 8447 Acked 9446 Read 9 Done Done New false
24853 8448 Acked 9447 Read 5 Done Done New false
24854 8449 Acked 9448 Read 9 Done Done New false
24855 8450 Acked 9449 Read 4 Done Done New false
24856 8451 Acked 9450 Read 4 Done Done New false
24857 8452 Acked 9451 Write 1 Done Done New false
24858 8453 Acked 9452 Read 3 Done Done New false
24859 8454 Acked 9453 Read 7 Done Done New false
24860 8455 Acked 9454 Read 7 Done Done New false
24861 8456 Acked 9455 Write 10 Done Done New false
24862 8457 Acked 9456 Write 5 Done Done New false
24863 8458 Acked 9457 Read 9 Done Done New false
24864 8459 Acked 9458 Read 9 Done Done New false
24865 8460 Acked 9459 Read 7 Done Done New false
24866 8461 Acked 9460 Write 6 Done Done New false
24867 8462 Acked 9461 Write 5 Done Done New false
24868 8463 Acked 9462 Write 7 Done Done New false
24869 8464 Acked 9463 Flush 0 Done Done New false
24870 8465 Acked 9464 Read 1 Done Done New false
24871 8466 Acked 9465 Read 6 Done Done New false
24872 8467 Acked 9466 Read 10 Done Done New false
24873 8468 Acked 9467 Flush 0 Done Done New false
24874 8469 Acked 9468 Write 6 Done Done New false
24875 8470 Acked 9469 Write 7 Done Done New false
24876 8471 Acked 9470 Write 3 Done Done New false
24877 8472 Acked 9471 Write 1 Done Done New false
24878 8473 Acked 9472 Read 4 Done Done New false
24879 8474 Acked 9473 Read 4 Done Done New false
24880 8475 Acked 9474 Read 10 Done Done New false
24881 8476 Acked 9475 Write 5 Done Done New false
24882 8477 Acked 9476 Write 5 Done Done New false
24883 8478 Acked 9477 Read 5 Done Done New false
24884 8479 Acked 9478 Read 8 Done Done New false
24885 8480 Acked 9479 Write 6 Done Done New false
24886 8481 Acked 9480 Read 1 Done Done New false
24887 8482 Acked 9481 Read 5 Done Done New false
24888 8483 Acked 9482 Read 9 Done Done New false
24889 8484 Acked 9483 Read 6 Done Done New false
24890 8485 Acked 9484 Write 10 Done Done New false
24891 8486 Acked 9485 Read 1 Done Done New false
24892 8487 Acked 9486 Flush 0 Done Done New false
24893 8488 Acked 9487 Write 2 Done Done New false
24894 8489 Acked 9488 Read 9 Done Done New false
24895 8490 Acked 9489 Read 5 Done Done New false
24896 8491 Acked 9490 Write 7 Done Done New false
24897 8492 Acked 9491 Write 8 Done Done New false
24898 8493 Acked 9492 Read 9 Done Done New false
24899 8494 Acked 9493 Write 3 Done Done New false
24900 8495 Acked 9494 Read 1 Done Done New false
24901 8496 Acked 9495 Write 7 Done Done New false
24902 8497 Acked 9496 Read 1 Done Done New false
24903 8498 Acked 9497 Read 2 Done Done New false
24904 8499 Acked 9498 Read 7 Done Done New false
24905 8500 Acked 9499 Read 4 Done Done New false
24906 8501 Acked 9500 Write 6 Done Done New false
24907 8502 Acked 9501 Read 1 Done Done New false
24908 8503 Acked 9502 Write 8 Done Done New false
24909 8504 Acked 9503 Read 4 Done Done New false
24910 8505 Acked 9504 Write 7 Done Done New false
24911 8506 Acked 9505 Flush 0 Done Done New false
24912 8507 Acked 9506 Flush 0 Done Done New false
24913 8508 Acked 9507 Write 7 Done Done New false
24914 8509 Acked 9508 Write 6 Done Done New false
24915 8510 Acked 9509 Read 4 Done Done New false
24916 8511 Acked 9510 Read 9 Done Done New false
24917 8512 Acked 9511 Read 1 Done Done New false
24918 8513 Acked 9512 Write 9 Done Done New false
24919 8514 Acked 9513 Read 3 Done Done New false
24920 8515 Acked 9514 Read 9 Done Done New false
24921 8516 Acked 9515 Flush 0 Done Done New false
24922 8517 Acked 9516 Read 5 Done Done New false
24923 8518 Acked 9517 Read 5 Done Done New false
24924 8519 Acked 9518 Read 5 Done Done New false
24925 8520 Acked 9519 Read 4 Done Done New false
24926 8521 Acked 9520 Read 10 Done Done New false
24927 8522 Acked 9521 Flush 0 Done Done New false
24928 8523 Acked 9522 Flush 0 Done Done New false
24929 8524 Acked 9523 Read 1 Done Done New false
24930 8525 Acked 9524 Read 1 Done Done New false
24931 8526 Acked 9525 Read 1 Done Done New false
24932 8527 Acked 9526 Read 7 Done Done New false
24933 8528 Acked 9527 Flush 0 Done Done New false
24934 8529 Acked 9528 Write 6 Done Done New false
24935 8530 Acked 9529 Write 6 Done Done New false
24936 8531 Acked 9530 Read 4 Done Done New false
24937 8532 Acked 9531 Read 4 Done Done New false
24938 8533 Acked 9532 Write 1 Done Done New false
24939 8534 Acked 9533 Read 3 Done Done New false
24940 8535 Acked 9534 Write 6 Done Done New false
24941 8536 Acked 9535 Read 1 Done Done New false
24942 8537 Acked 9536 Read 5 Done Done New false
24943 8538 Acked 9537 Write 2 Done Done New false
24944 8539 Acked 9538 Read 9 Done Done New false
24945 8540 Acked 9539 Write 1 Done Done New false
24946 8541 Acked 9540 Write 4 Done Done New false
24947 8542 Acked 9541 Read 7 Done Done New false
24948 8543 Acked 9542 Write 9 Done Done New false
24949 8544 Acked 9543 Read 7 Done Done New false
24950 8545 Acked 9544 Write 2 Done Done New false
24951 8546 Acked 9545 Write 8 Done Done New false
24952 8547 Acked 9546 Flush 0 Done Done New false
24953 8548 Acked 9547 Read 6 Done Done New false
24954 8549 Acked 9548 Read 2 Done Done New false
24955 8550 Acked 9549 Flush 0 Done Done New false
24956 8551 Acked 9550 Read 6 Done Done New false
24957 8552 Acked 9551 Read 7 Done Done New false
24958 8553 Acked 9552 Write 6 Done Done New false
24959 8554 Acked 9553 Read 3 Done Done New false
24960 8555 Acked 9554 Write 8 Done Done New false
24961 8556 Acked 9555 Read 2 Done Done New false
24962 8557 Acked 9556 Write 9 Done Done New false
24963 8558 Acked 9557 Read 2 Done Done New false
24964 8559 Acked 9558 Write 2 Done Done New false
24965 8560 Acked 9559 Read 10 Done Done New false
24966 8561 Acked 9560 Write 2 Done Done New false
24967 8562 Acked 9561 Read 1 Done Done New false
24968 8563 Acked 9562 Read 1 Done Done New false
24969 8564 Acked 9563 Read 3 Done Done New false
24970 8565 Acked 9564 Read 6 Done Done New false
24971 8566 Acked 9565 Read 9 Done Done New false
24972 8567 Acked 9566 Write 1 Done Done New false
24973 8568 Acked 9567 Read 7 Done Done New false
24974 8569 Acked 9568 Read 6 Done Done New false
24975 8570 Acked 9569 Read 5 Done Done New false
24976 8571 Acked 9570 Read 10 Done Done New false
24977 8572 Acked 9571 Write 9 Done Done New false
24978 8573 Acked 9572 Write 10 Done Done New false
24979 8574 Acked 9573 Write 2 Done Done New false
24980 8575 Acked 9574 Write 9 Done Done New false
24981 8576 Acked 9575 Read 4 Done Done New false
24982 8577 Acked 9576 Write 8 Done Done New false
24983 8578 Acked 9577 Write 3 Done Done New false
24984 8579 Acked 9578 Write 5 Done Done New false
24985 8580 Acked 9579 Flush 0 Done Done New false
24986 8581 Acked 9580 Read 9 Done Done New false
24987 8582 Acked 9581 Read 9 Done Done New false
24988 8583 Acked 9582 Read 5 Done Done New false
24989 8584 Acked 9583 Flush 0 Done Done New false
24990 8585 Acked 9584 Write 6 Done Done New false
24991 8586 Acked 9585 Write 1 Done Done New false
24992 8587 Acked 9586 Flush 0 Done Done New false
24993 8588 Acked 9587 Write 5 Done Done New false
24994 8589 Acked 9588 Write 6 Done Done New false
24995 8590 Acked 9589 Write 5 Done Done New false
24996 8591 Acked 9590 Write 2 Done Done New false
24997 8592 Acked 9591 Write 3 Done Done New false
24998 8593 Acked 9592 Read 1 Done Done New false
24999 8594 Acked 9593 Write 8 Done Done New false
25000 8595 Acked 9594 Read 10 Done Done New false
25001 8596 Acked 9595 Read 1 Done Done New false
25002 8597 Acked 9596 Flush 0 Done Done New false
25003 8598 Acked 9597 Read 1 Done Done New false
25004 8599 Acked 9598 Read 1 Done Done New false
25005 8600 Acked 9599 Read 2 Done Done New false
25006 8601 Acked 9600 Write 1 Done Done New false
25007 8602 Acked 9601 Read 5 Done Done New false
25008 8603 Acked 9602 Flush 0 Done Done New false
25009 8604 Acked 9603 Flush 0 Done Done New false
25010 8605 Acked 9604 Read 1 Done Done New false
25011 8606 Acked 9605 Read 2 Done Done New false
25012 8607 Acked 9606 Read 3 Done Done New false
25013 8608 Acked 9607 Write 3 Done Done New false
25014 8609 Acked 9608 Read 4 Done Done New false
25015 8610 Acked 9609 Read 5 Done Done New false
25016 8611 Acked 9610 Read 2 Done Done New false
25017 8612 Acked 9611 Flush 0 Done Done New false
25018 8613 Acked 9612 Write 8 Done Done New false
25019 8614 Acked 9613 Read 6 Done Done New false
25020 8615 Acked 9614 Write 9 Done Done New false
25021 8616 Acked 9615 Write 6 Done Done New false
25022 8617 Acked 9616 Write 9 Done Done New false
25023 8618 Acked 9617 Read 9 Done Done New false
25024 8619 Acked 9618 Write 7 Done Done New false
25025 8620 Acked 9619 Write 8 Done Done New false
25026 8621 Acked 9620 Write 4 Done Done New false
25027 8622 Acked 9621 Write 2 Done Done New false
25028 8623 Acked 9622 Read 10 Done Done New false
25029 8624 Acked 9623 Write 9 Done Done New false
25030 8625 Acked 9624 Write 10 Done Done New false
25031 8626 Acked 9625 Read 8 Done Done New false
25032 8627 Acked 9626 Write 5 Done Done New false
25033 8628 Acked 9627 Read 3 Done Done New false
25034 8629 Acked 9628 Write 7 Done Done New false
25035 8630 Acked 9629 Read 6 Done Done New false
25036 8631 Acked 9630 Flush 0 Done Done New false
25037 8632 Acked 9631 Read 6 Done Done New false
25038 8633 Acked 9632 Write 8 Done Done New false
25039 8634 Acked 9633 Read 7 Done Done New false
25040 8635 Acked 9634 Read 8 Done Done New false
25041 8636 Acked 9635 Write 2 Done Done New false
25042 8637 Acked 9636 Read 6 Done Done New false
25043 8638 Acked 9637 Read 9 Done Done New false
25044 8639 Acked 9638 Read 2 Done Done New false
25045 8640 Acked 9639 Write 6 Done Done New false
25046 8641 Acked 9640 Write 8 Done Done New false
25047 8642 Acked 9641 Write 3 Done Done New false
25048 8643 Acked 9642 Flush 0 Done Done New false
25049 8644 Acked 9643 Read 7 Done Done New false
25050 8645 Acked 9644 Read 10 Done Done New false
25051 8646 Acked 9645 Read 5 Done Done New false
25052 8647 Acked 9646 Write 7 Done Done New false
25053 8648 Acked 9647 Read 8 Done Done New false
25054 8649 Acked 9648 Read 1 Done Done New false
25055 8650 Acked 9649 Write 5 Done Done New false
25056 8651 Acked 9650 Read 1 Done Done New false
25057 8652 Acked 9651 Write 4 Done Done New false
25058 8653 Acked 9652 Write 6 Done Done New false
25059 8654 Acked 9653 Write 4 Done Done New false
25060 8655 Acked 9654 Write 9 Done Done New false
25061 8656 Acked 9655 Read 2 Done Done New false
25062 8657 Acked 9656 Write 5 Done Done New false
25063 8658 Acked 9657 Read 9 Done Done New false
25064 8659 Acked 9658 Read 9 Done Done New false
25065 8660 Acked 9659 Read 10 Done Done New false
25066 8661 Acked 9660 Read 8 Done Done New false
25067 8662 Acked 9661 Write 5 Done Done New false
25068 8663 Acked 9662 Read 8 Done Done New false
25069 8664 Acked 9663 Write 1 Done Done New false
25070 8665 Acked 9664 Write 10 Done Done New false
25071 8666 Acked 9665 Read 7 Done Done New false
25072 8667 Acked 9666 Read 1 Done Done New false
25073 8668 Acked 9667 Write 7 Done Done New false
25074 8669 Acked 9668 Flush 0 Done Done New false
25075 8670 Acked 9669 Write 9 Done Done New false
25076 8671 Acked 9670 Read 8 Done Done New false
25077 8672 Acked 9671 Write 5 Done Done New false
25078 8673 Acked 9672 Write 6 Done Done New false
25079 8674 Acked 9673 Read 8 Done Done New false
25080 8675 Acked 9674 Read 8 Done Done New false
25081 8676 Acked 9675 Write 1 Done Done New false
25082 8677 Acked 9676 Read 7 Done Done New false
25083 8678 Acked 9677 Read 3 Done Done New false
25084 8679 Acked 9678 Write 9 Done Done New false
25085 8680 Acked 9679 Read 5 Done Done New false
25086 8681 Acked 9680 Write 1 Done Done New false
25087 8682 Acked 9681 Read 9 Done Done New false
25088 8683 Acked 9682 Write 5 Done Done New false
25089 8684 Acked 9683 Write 5 Done Done New false
25090 8685 Acked 9684 Read 8 Done Done New false
25091 8686 Acked 9685 Read 8 Done Done New false
25092 8687 Acked 9686 Read 3 Done Done New false
25093 8688 Acked 9687 Write 10 Done Done New false
25094 8689 Acked 9688 Write 4 Done Done New false
25095 8690 Acked 9689 Flush 0 Done Done New false
25096 8691 Acked 9690 Read 3 Done Done New false
25097 8692 Acked 9691 Write 2 Done Done New false
25098 8693 Acked 9692 Read 10 Done Done New false
25099 8694 Acked 9693 Read 9 Done Done New false
25100 8695 Acked 9694 Read 8 Done Done New false
25101 8696 Acked 9695 Read 7 Done Done New false
25102 8697 Acked 9696 Read 2 Done Done New false
25103 8698 Acked 9697 Flush 0 Done Done New false
25104 8699 Acked 9698 Read 7 Done Done New false
25105 8700 Acked 9699 Read 9 Done Done New false
25106 8701 Acked 9700 Flush 0 Done Done New false
25107 8702 Acked 9701 Flush 0 Done Done New false
25108 8703 Acked 9702 Write 4 Done Done New false
25109 8704 Acked 9703 Write 10 Done Done New false
25110 8705 Acked 9704 Read 1 Done Done New false
25111 8706 Acked 9705 Write 2 Done Done New false
25112 8707 Acked 9706 Read 6 Done Done New false
25113 8708 Acked 9707 Read 2 Done Done New false
25114 8709 Acked 9708 Write 1 Done Done New false
25115 8710 Acked 9709 Write 10 Done Done New false
25116 8711 Acked 9710 Read 7 Done Done New false
25117 8712 Acked 9711 Write 4 Done Done New false
25118 8713 Acked 9712 Write 1 Done Done New false
25119 8714 Acked 9713 Read 6 Done Done New false
25120 8715 Acked 9714 Write 7 Done Done New false
25121 8716 Acked 9715 Write 2 Done Done New false
25122 8717 Acked 9716 Read 6 Done Done New false
25123 8718 Acked 9717 Write 5 Done Done New false
25124 8719 Acked 9718 Read 4 Done Done New false
25125 8720 Acked 9719 Write 1 Done Done New false
25126 8721 Acked 9720 Read 9 Done Done New false
25127 8722 Acked 9721 Write 1 Done Done New false
25128 8723 Acked 9722 Read 8 Done Done New false
25129 8724 Acked 9723 Write 8 Done Done New false
25130 8725 Acked 9724 Read 8 Done Done New false
25131 8726 Acked 9725 Read 6 Done Done New false
25132 8727 Acked 9726 Flush 0 Done Done New false
25133 8728 Acked 9727 Write 4 Done Done New false
25134 8729 Acked 9728 Read 3 Done Done New false
25135 8730 Acked 9729 Read 3 Done Done New false
25136 8731 Acked 9730 Read 5 Done Done New false
25137 8732 Acked 9731 Read 8 Done Done New false
25138 8733 Acked 9732 Read 6 Done Done New false
25139 8734 Acked 9733 Write 9 Done Done New false
25140 8735 Acked 9734 Read 8 Done Done New false
25141 8736 Acked 9735 Flush 0 Done Done New false
25142 8737 Acked 9736 Write 10 Done Done New false
25143 8738 Acked 9737 Flush 0 Done Done New false
25144 8739 Acked 9738 Flush 0 Done Done New false
25145 8740 Acked 9739 Flush 0 Done Done New false
25146 8741 Acked 9740 Flush 0 Done Done New false
25147 STATES DS:0 DS:1 DS:2 TOTAL
25148 New 0 0 301 301
25149 Sent 0 0 0 0
25150 Done 301 301 0 602
25151 Skipped 0 0 0 0
25152 Error 0 0 0 0
25153 Last Flush: 9740 9740 9439
25154 Downstairs last five completed: 9439 9438 9437 9436 9435
25155 Upstairs last five completed: 8741 8740 8739 8738 8737
25156 CLIENT: Up:0 ds:301 act:2
251572024-02-27T21:49:36.854ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
251582024-02-27T21:49:36.854ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
251592024-02-27T21:49:36.854ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
251602024-02-27T21:49:36.854ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
251612024-02-27T21:49:36.854ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
251622024-02-27T21:49:36.855ZINFOcrucible: [2] client re-new 301 jobs since flush 9439 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
251632024-02-27T21:49:36.855ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
251642024-02-27T21:49:37.857ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
251652024-02-27T21:49:37.857ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
251662024-02-27T21:49:37.857ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
251672024-02-27T21:49:37.857ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
251682024-02-27T21:49:37.857ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
251692024-02-27T21:49:37.857ZINFOcrucible: [2] client re-new 301 jobs since flush 9439 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
251702024-02-27T21:49:37.857ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
251712024-02-27T21:49:38.859ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
251722024-02-27T21:49:38.859ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
251732024-02-27T21:49:38.859ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
251742024-02-27T21:49:38.859ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
251752024-02-27T21:49:38.859ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
251762024-02-27T21:49:38.859ZINFOcrucible: [2] client re-new 301 jobs since flush 9439 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
251772024-02-27T21:49:38.859ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
251782024-02-27T21:49:39.860ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
251792024-02-27T21:49:39.860ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
251802024-02-27T21:49:39.860ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
251812024-02-27T21:49:39.860ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
251822024-02-27T21:49:39.860ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
251832024-02-27T21:49:39.860ZINFOcrucible: [2] client re-new 301 jobs since flush 9439 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
251842024-02-27T21:49:39.860ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
25185 ----------------------------------------------------------------
25186 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
25187 States: Active Active Offline
25188 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
25189 8441 Acked 9440 Write 9 Done Done New true
25190 8442 Acked 9441 Read 9 Done Done New true
25191 8443 Acked 9442 Write 4 Done Done New true
25192 8444 Acked 9443 Write 10 Done Done New true
25193 8445 Acked 9444 Read 7 Done Done New true
25194 8446 Acked 9445 Write 7 Done Done New true
25195 8447 Acked 9446 Read 9 Done Done New true
25196 8448 Acked 9447 Read 5 Done Done New true
25197 8449 Acked 9448 Read 9 Done Done New true
25198 8450 Acked 9449 Read 4 Done Done New true
25199 8451 Acked 9450 Read 4 Done Done New true
25200 8452 Acked 9451 Write 1 Done Done New true
25201 8453 Acked 9452 Read 3 Done Done New true
25202 8454 Acked 9453 Read 7 Done Done New true
25203 8455 Acked 9454 Read 7 Done Done New true
25204 8456 Acked 9455 Write 10 Done Done New true
25205 8457 Acked 9456 Write 5 Done Done New true
25206 8458 Acked 9457 Read 9 Done Done New true
25207 8459 Acked 9458 Read 9 Done Done New true
25208 8460 Acked 9459 Read 7 Done Done New true
25209 8461 Acked 9460 Write 6 Done Done New true
25210 8462 Acked 9461 Write 5 Done Done New true
25211 8463 Acked 9462 Write 7 Done Done New true
25212 8464 Acked 9463 Flush 0 Done Done New true
25213 8465 Acked 9464 Read 1 Done Done New true
25214 8466 Acked 9465 Read 6 Done Done New true
25215 8467 Acked 9466 Read 10 Done Done New true
25216 8468 Acked 9467 Flush 0 Done Done New true
25217 8469 Acked 9468 Write 6 Done Done New true
25218 8470 Acked 9469 Write 7 Done Done New true
25219 8471 Acked 9470 Write 3 Done Done New true
25220 8472 Acked 9471 Write 1 Done Done New true
25221 8473 Acked 9472 Read 4 Done Done New true
25222 8474 Acked 9473 Read 4 Done Done New true
25223 8475 Acked 9474 Read 10 Done Done New true
25224 8476 Acked 9475 Write 5 Done Done New true
25225 8477 Acked 9476 Write 5 Done Done New true
25226 8478 Acked 9477 Read 5 Done Done New true
25227 8479 Acked 9478 Read 8 Done Done New true
25228 8480 Acked 9479 Write 6 Done Done New true
25229 8481 Acked 9480 Read 1 Done Done New true
25230 8482 Acked 9481 Read 5 Done Done New true
25231 8483 Acked 9482 Read 9 Done Done New true
25232 8484 Acked 9483 Read 6 Done Done New true
25233 8485 Acked 9484 Write 10 Done Done New true
25234 8486 Acked 9485 Read 1 Done Done New true
25235 8487 Acked 9486 Flush 0 Done Done New true
25236 8488 Acked 9487 Write 2 Done Done New true
25237 8489 Acked 9488 Read 9 Done Done New true
25238 8490 Acked 9489 Read 5 Done Done New true
25239 8491 Acked 9490 Write 7 Done Done New true
25240 8492 Acked 9491 Write 8 Done Done New true
25241 8493 Acked 9492 Read 9 Done Done New true
25242 8494 Acked 9493 Write 3 Done Done New true
25243 8495 Acked 9494 Read 1 Done Done New true
25244 8496 Acked 9495 Write 7 Done Done New true
25245 8497 Acked 9496 Read 1 Done Done New true
25246 8498 Acked 9497 Read 2 Done Done New true
25247 8499 Acked 9498 Read 7 Done Done New true
25248 8500 Acked 9499 Read 4 Done Done New true
25249 8501 Acked 9500 Write 6 Done Done New true
25250 8502 Acked 9501 Read 1 Done Done New true
25251 8503 Acked 9502 Write 8 Done Done New true
25252 8504 Acked 9503 Read 4 Done Done New true
25253 8505 Acked 9504 Write 7 Done Done New true
25254 8506 Acked 9505 Flush 0 Done Done New true
25255 8507 Acked 9506 Flush 0 Done Done New true
25256 8508 Acked 9507 Write 7 Done Done New true
25257 8509 Acked 9508 Write 6 Done Done New true
25258 8510 Acked 9509 Read 4 Done Done New true
25259 8511 Acked 9510 Read 9 Done Done New true
25260 8512 Acked 9511 Read 1 Done Done New true
25261 8513 Acked 9512 Write 9 Done Done New true
25262 8514 Acked 9513 Read 3 Done Done New true
25263 8515 Acked 9514 Read 9 Done Done New true
25264 8516 Acked 9515 Flush 0 Done Done New true
25265 8517 Acked 9516 Read 5 Done Done New true
25266 8518 Acked 9517 Read 5 Done Done New true
25267 8519 Acked 9518 Read 5 Done Done New true
25268 8520 Acked 9519 Read 4 Done Done New true
25269 8521 Acked 9520 Read 10 Done Done New true
25270 8522 Acked 9521 Flush 0 Done Done New true
25271 8523 Acked 9522 Flush 0 Done Done New true
25272 8524 Acked 9523 Read 1 Done Done New true
25273 8525 Acked 9524 Read 1 Done Done New true
25274 8526 Acked 9525 Read 1 Done Done New true
25275 8527 Acked 9526 Read 7 Done Done New true
25276 8528 Acked 9527 Flush 0 Done Done New true
25277 8529 Acked 9528 Write 6 Done Done New true
25278 8530 Acked 9529 Write 6 Done Done New true
25279 8531 Acked 9530 Read 4 Done Done New true
25280 8532 Acked 9531 Read 4 Done Done New true
25281 8533 Acked 9532 Write 1 Done Done New true
25282 8534 Acked 9533 Read 3 Done Done New true
25283 8535 Acked 9534 Write 6 Done Done New true
25284 8536 Acked 9535 Read 1 Done Done New true
25285 8537 Acked 9536 Read 5 Done Done New true
25286 8538 Acked 9537 Write 2 Done Done New true
25287 8539 Acked 9538 Read 9 Done Done New true
25288 8540 Acked 9539 Write 1 Done Done New true
25289 8541 Acked 9540 Write 4 Done Done New true
25290 8542 Acked 9541 Read 7 Done Done New true
25291 8543 Acked 9542 Write 9 Done Done New true
25292 8544 Acked 9543 Read 7 Done Done New true
25293 8545 Acked 9544 Write 2 Done Done New true
25294 8546 Acked 9545 Write 8 Done Done New true
25295 8547 Acked 9546 Flush 0 Done Done New true
25296 8548 Acked 9547 Read 6 Done Done New true
25297 8549 Acked 9548 Read 2 Done Done New true
25298 8550 Acked 9549 Flush 0 Done Done New true
25299 8551 Acked 9550 Read 6 Done Done New true
25300 8552 Acked 9551 Read 7 Done Done New true
25301 8553 Acked 9552 Write 6 Done Done New true
25302 8554 Acked 9553 Read 3 Done Done New true
25303 8555 Acked 9554 Write 8 Done Done New true
25304 8556 Acked 9555 Read 2 Done Done New true
25305 8557 Acked 9556 Write 9 Done Done New true
25306 8558 Acked 9557 Read 2 Done Done New true
25307 8559 Acked 9558 Write 2 Done Done New true
25308 8560 Acked 9559 Read 10 Done Done New true
25309 8561 Acked 9560 Write 2 Done Done New true
25310 8562 Acked 9561 Read 1 Done Done New true
25311 8563 Acked 9562 Read 1 Done Done New true
25312 8564 Acked 9563 Read 3 Done Done New true
25313 8565 Acked 9564 Read 6 Done Done New true
25314 8566 Acked 9565 Read 9 Done Done New true
25315 8567 Acked 9566 Write 1 Done Done New true
25316 8568 Acked 9567 Read 7 Done Done New true
25317 8569 Acked 9568 Read 6 Done Done New true
25318 8570 Acked 9569 Read 5 Done Done New true
25319 8571 Acked 9570 Read 10 Done Done New true
25320 8572 Acked 9571 Write 9 Done Done New true
25321 8573 Acked 9572 Write 10 Done Done New true
25322 8574 Acked 9573 Write 2 Done Done New true
25323 8575 Acked 9574 Write 9 Done Done New true
25324 8576 Acked 9575 Read 4 Done Done New true
25325 8577 Acked 9576 Write 8 Done Done New true
25326 8578 Acked 9577 Write 3 Done Done New true
25327 8579 Acked 9578 Write 5 Done Done New true
25328 8580 Acked 9579 Flush 0 Done Done New true
25329 8581 Acked 9580 Read 9 Done Done New true
25330 8582 Acked 9581 Read 9 Done Done New true
25331 8583 Acked 9582 Read 5 Done Done New true
25332 8584 Acked 9583 Flush 0 Done Done New true
25333 8585 Acked 9584 Write 6 Done Done New true
25334 8586 Acked 9585 Write 1 Done Done New true
25335 8587 Acked 9586 Flush 0 Done Done New true
25336 8588 Acked 9587 Write 5 Done Done New true
25337 8589 Acked 9588 Write 6 Done Done New true
25338 8590 Acked 9589 Write 5 Done Done New true
25339 8591 Acked 9590 Write 2 Done Done New true
25340 8592 Acked 9591 Write 3 Done Done New true
25341 8593 Acked 9592 Read 1 Done Done New true
25342 8594 Acked 9593 Write 8 Done Done New true
25343 8595 Acked 9594 Read 10 Done Done New true
25344 8596 Acked 9595 Read 1 Done Done New true
25345 8597 Acked 9596 Flush 0 Done Done New true
25346 8598 Acked 9597 Read 1 Done Done New true
25347 8599 Acked 9598 Read 1 Done Done New true
25348 8600 Acked 9599 Read 2 Done Done New true
25349 8601 Acked 9600 Write 1 Done Done New true
25350 8602 Acked 9601 Read 5 Done Done New true
25351 8603 Acked 9602 Flush 0 Done Done New true
25352 8604 Acked 9603 Flush 0 Done Done New true
25353 8605 Acked 9604 Read 1 Done Done New true
25354 8606 Acked 9605 Read 2 Done Done New true
25355 8607 Acked 9606 Read 3 Done Done New true
25356 8608 Acked 9607 Write 3 Done Done New true
25357 8609 Acked 9608 Read 4 Done Done New true
25358 8610 Acked 9609 Read 5 Done Done New true
25359 8611 Acked 9610 Read 2 Done Done New true
25360 8612 Acked 9611 Flush 0 Done Done New true
25361 8613 Acked 9612 Write 8 Done Done New true
25362 8614 Acked 9613 Read 6 Done Done New true
25363 8615 Acked 9614 Write 9 Done Done New true
25364 8616 Acked 9615 Write 6 Done Done New true
25365 8617 Acked 9616 Write 9 Done Done New true
25366 8618 Acked 9617 Read 9 Done Done New true
25367 8619 Acked 9618 Write 7 Done Done New true
25368 8620 Acked 9619 Write 8 Done Done New true
25369 8621 Acked 9620 Write 4 Done Done New true
25370 8622 Acked 9621 Write 2 Done Done New true
25371 8623 Acked 9622 Read 10 Done Done New true
25372 8624 Acked 9623 Write 9 Done Done New true
25373 8625 Acked 9624 Write 10 Done Done New true
25374 8626 Acked 9625 Read 8 Done Done New true
25375 8627 Acked 9626 Write 5 Done Done New true
25376 8628 Acked 9627 Read 3 Done Done New true
25377 8629 Acked 9628 Write 7 Done Done New true
25378 8630 Acked 9629 Read 6 Done Done New true
25379 8631 Acked 9630 Flush 0 Done Done New true
25380 8632 Acked 9631 Read 6 Done Done New true
25381 8633 Acked 9632 Write 8 Done Done New true
25382 8634 Acked 9633 Read 7 Done Done New true
25383 8635 Acked 9634 Read 8 Done Done New true
25384 8636 Acked 9635 Write 2 Done Done New true
25385 8637 Acked 9636 Read 6 Done Done New true
25386 8638 Acked 9637 Read 9 Done Done New true
25387 8639 Acked 9638 Read 2 Done Done New true
25388 8640 Acked 9639 Write 6 Done Done New true
25389 8641 Acked 9640 Write 8 Done Done New true
25390 8642 Acked 9641 Write 3 Done Done New true
25391 8643 Acked 9642 Flush 0 Done Done New true
25392 8644 Acked 9643 Read 7 Done Done New true
25393 8645 Acked 9644 Read 10 Done Done New true
25394 8646 Acked 9645 Read 5 Done Done New true
25395 8647 Acked 9646 Write 7 Done Done New true
25396 8648 Acked 9647 Read 8 Done Done New true
25397 8649 Acked 9648 Read 1 Done Done New true
25398 8650 Acked 9649 Write 5 Done Done New true
25399 8651 Acked 9650 Read 1 Done Done New true
25400 8652 Acked 9651 Write 4 Done Done New true
25401 8653 Acked 9652 Write 6 Done Done New true
25402 8654 Acked 9653 Write 4 Done Done New true
25403 8655 Acked 9654 Write 9 Done Done New true
25404 8656 Acked 9655 Read 2 Done Done New true
25405 8657 Acked 9656 Write 5 Done Done New true
25406 8658 Acked 9657 Read 9 Done Done New true
25407 8659 Acked 9658 Read 9 Done Done New true
25408 8660 Acked 9659 Read 10 Done Done New true
25409 8661 Acked 9660 Read 8 Done Done New true
25410 8662 Acked 9661 Write 5 Done Done New true
25411 8663 Acked 9662 Read 8 Done Done New true
25412 8664 Acked 9663 Write 1 Done Done New true
25413 8665 Acked 9664 Write 10 Done Done New true
25414 8666 Acked 9665 Read 7 Done Done New true
25415 8667 Acked 9666 Read 1 Done Done New true
25416 8668 Acked 9667 Write 7 Done Done New true
25417 8669 Acked 9668 Flush 0 Done Done New true
25418 8670 Acked 9669 Write 9 Done Done New true
25419 8671 Acked 9670 Read 8 Done Done New true
25420 8672 Acked 9671 Write 5 Done Done New true
25421 8673 Acked 9672 Write 6 Done Done New true
25422 8674 Acked 9673 Read 8 Done Done New true
25423 8675 Acked 9674 Read 8 Done Done New true
25424 8676 Acked 9675 Write 1 Done Done New true
25425 8677 Acked 9676 Read 7 Done Done New true
25426 8678 Acked 9677 Read 3 Done Done New true
25427 8679 Acked 9678 Write 9 Done Done New true
25428 8680 Acked 9679 Read 5 Done Done New true
25429 8681 Acked 9680 Write 1 Done Done New true
25430 8682 Acked 9681 Read 9 Done Done New true
25431 8683 Acked 9682 Write 5 Done Done New true
25432 8684 Acked 9683 Write 5 Done Done New true
25433 8685 Acked 9684 Read 8 Done Done New true
25434 8686 Acked 9685 Read 8 Done Done New true
25435 8687 Acked 9686 Read 3 Done Done New true
25436 8688 Acked 9687 Write 10 Done Done New true
25437 8689 Acked 9688 Write 4 Done Done New true
25438 8690 Acked 9689 Flush 0 Done Done New true
25439 8691 Acked 9690 Read 3 Done Done New true
25440 8692 Acked 9691 Write 2 Done Done New true
25441 8693 Acked 9692 Read 10 Done Done New true
25442 8694 Acked 9693 Read 9 Done Done New true
25443 8695 Acked 9694 Read 8 Done Done New true
25444 8696 Acked 9695 Read 7 Done Done New true
25445 8697 Acked 9696 Read 2 Done Done New true
25446 8698 Acked 9697 Flush 0 Done Done New true
25447 8699 Acked 9698 Read 7 Done Done New true
25448 8700 Acked 9699 Read 9 Done Done New true
25449 8701 Acked 9700 Flush 0 Done Done New true
25450 8702 Acked 9701 Flush 0 Done Done New true
25451 8703 Acked 9702 Write 4 Done Done New true
25452 8704 Acked 9703 Write 10 Done Done New true
25453 8705 Acked 9704 Read 1 Done Done New true
25454 8706 Acked 9705 Write 2 Done Done New true
25455 8707 Acked 9706 Read 6 Done Done New true
25456 8708 Acked 9707 Read 2 Done Done New true
25457 8709 Acked 9708 Write 1 Done Done New true
25458 8710 Acked 9709 Write 10 Done Done New true
25459 8711 Acked 9710 Read 7 Done Done New true
25460 8712 Acked 9711 Write 4 Done Done New true
25461 8713 Acked 9712 Write 1 Done Done New true
25462 8714 Acked 9713 Read 6 Done Done New true
25463 8715 Acked 9714 Write 7 Done Done New true
25464 8716 Acked 9715 Write 2 Done Done New true
25465 8717 Acked 9716 Read 6 Done Done New true
25466 8718 Acked 9717 Write 5 Done Done New true
25467 8719 Acked 9718 Read 4 Done Done New true
25468 8720 Acked 9719 Write 1 Done Done New true
25469 8721 Acked 9720 Read 9 Done Done New true
25470 8722 Acked 9721 Write 1 Done Done New true
25471 8723 Acked 9722 Read 8 Done Done New true
25472 8724 Acked 9723 Write 8 Done Done New true
25473 8725 Acked 9724 Read 8 Done Done New true
25474 8726 Acked 9725 Read 6 Done Done New true
25475 8727 Acked 9726 Flush 0 Done Done New true
25476 8728 Acked 9727 Write 4 Done Done New true
25477 8729 Acked 9728 Read 3 Done Done New true
25478 8730 Acked 9729 Read 3 Done Done New true
25479 8731 Acked 9730 Read 5 Done Done New true
25480 8732 Acked 9731 Read 8 Done Done New true
25481 8733 Acked 9732 Read 6 Done Done New true
25482 8734 Acked 9733 Write 9 Done Done New true
25483 8735 Acked 9734 Read 8 Done Done New true
25484 8736 Acked 9735 Flush 0 Done Done New true
25485 8737 Acked 9736 Write 10 Done Done New true
25486 8738 Acked 9737 Flush 0 Done Done New true
25487 8739 Acked 9738 Flush 0 Done Done New true
25488 8740 Acked 9739 Flush 0 Done Done New true
25489 8741 Acked 9740 Flush 0 Done Done New true
25490 STATES DS:0 DS:1 DS:2 TOTAL
25491 New 0 0 301 301
25492 Sent 0 0 0 0
25493 Done 301 301 0 602
25494 Skipped 0 0 0 0
25495 Error 0 0 0 0
25496 Last Flush: 9740 9740 9439
25497 Downstairs last five completed: 9439 9438 9437 9436 9435
25498 Upstairs last five completed: 8741 8740 8739 8738 8737
25499 CLIENT: Up:0 ds:301 act:2
255002024-02-27T21:49:40.862ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
255012024-02-27T21:49:40.862ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
255022024-02-27T21:49:40.862ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
255032024-02-27T21:49:40.862ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
255042024-02-27T21:49:40.862ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
255052024-02-27T21:49:40.862ZINFOcrucible: [2] client re-new 301 jobs since flush 9439 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
255062024-02-27T21:49:40.862ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
25507 [8830] got start action
25508 [8830] I am going to restart
25509 Starting downstairs at port 8830
25510 Make output file at "/tmp/dsc/downstairs-8830.txt"
25511 [8830][2] reports Starting
25512 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1196)
25513 [8830][2] reports Running
255142024-02-27T21:49:41.863ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
255152024-02-27T21:49:41.863ZWARNcrucible: ds_connection connect to 127.0.0.1:8830 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
255162024-02-27T21:49:41.863ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
255172024-02-27T21:49:41.863ZWARNcrucible: downstairs task for 2 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
255182024-02-27T21:49:41.863ZINFOcrucible: Still missing, state is Offline = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
255192024-02-27T21:49:41.863ZINFOcrucible: [2] client re-new 301 jobs since flush 9439 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
255202024-02-27T21:49:41.863ZINFOcrucible: client task is exiting = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
255212024-02-27T21:49:42.864ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
255222024-02-27T21:49:42.864ZINFOcrucible: ds_connection connected = io task client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
255232024-02-27T21:49:42.866ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
255242024-02-27T21:49:42.866ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008830 matches = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
255252024-02-27T21:49:42.866ZINFOcrucible: send last flush ID to this DS: 9439 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
255262024-02-27T21:49:42.866ZINFOcrucible: Replied this last flush ID: 9439 = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
255272024-02-27T21:49:42.866ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
255282024-02-27T21:49:42.866ZINFOcrucible: [2] Transition from Offline to Replay = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
255292024-02-27T21:49:42.866ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
255302024-02-27T21:49:42.866ZINFOcrucible: [2] Transition from Replay to Active = downstairs client = 2 session_id = e6a5a974-925a-494f-8845-facc441c2690
25531 ----------------------------------------------------------------
25532 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25533 Downstairs last five completed: 9740 9739 9738 9737 9736
25534 Upstairs last five completed: 8741 8740 8739 8738 8737
25535 CLIENT: Up:0 ds:0 act:3
25536 Replay: All jobs finished, all DS active.
25537 Main task has work to do, go find it
25538 got dsc Stop(0)
25539 stop 0
25540 Feb 27 21:49:44.190 INFO request completed[8810] Got stop action so:false kr:false
25541 , latency_us: 352, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 3824fea7-b990-4118-86e0-d0bb4764248f, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
255422024-02-27T21:49:44.190ZWARNcrucible: downstairs disconnected = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
255432024-02-27T21:49:44.190ZWARNcrucible: client task is sending Done(Finished) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
255442024-02-27T21:49:44.190ZWARNcrucible: downstairs task for 0 stopped due to Finished session_id = e6a5a974-925a-494f-8845-facc441c2690
255452024-02-27T21:49:44.191ZINFOcrucible: Gone missing, transition from Active to Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
25546 Feb 27 21:49:44.191 {"msg":"[0] client re-new 0 jobs since flush 9740","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:49:44.191058239Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"downstairs","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
25547 INFO request completed, latency_us: 120, response_code: 200, uri: /state/cid/0, method: GET, req_id: c0f14ad0-60a9-4b98-808b-54ff2672fc3c, remote_addr:{"msg":"client task is exiting","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:49:44.191115312Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"io task","client":"0","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
25548 127.0.0.1:42901, local_addr: 127.0.0.1:9998
25549 [8810] Exited with: ExitStatus(unix_wait_status(9))
25550 [8810][0] reports Exit
255512024-02-27T21:49:45.192ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
255522024-02-27T21:49:45.192ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
255532024-02-27T21:49:45.192ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
255542024-02-27T21:49:45.192ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
255552024-02-27T21:49:45.192ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
255562024-02-27T21:49:45.192ZINFOcrucible: [0] client re-new 0 jobs since flush 9740 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
255572024-02-27T21:49:45.192ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
255582024-02-27T21:49:46.193ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
255592024-02-27T21:49:46.193ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
255602024-02-27T21:49:46.193ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
255612024-02-27T21:49:46.193ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
255622024-02-27T21:49:46.193ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
255632024-02-27T21:49:46.193ZINFOcrucible: [0] client re-new 0 jobs since flush 9740 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
255642024-02-27T21:49:46.193ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
255652024-02-27T21:49:47.194ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
255662024-02-27T21:49:47.194ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
255672024-02-27T21:49:47.194ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
255682024-02-27T21:49:47.194ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
255692024-02-27T21:49:47.194ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
255702024-02-27T21:49:47.194ZINFOcrucible: [0] client re-new 0 jobs since flush 9740 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
255712024-02-27T21:49:47.194ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
25572 Feb 27 21:49:48.193 INFO request completed, latency_us: 206, response_code: 200, uri: /state/cid/0, method: GET, req_id: a2ef244f-ff71-42a0-b0db-f785c60e6c61, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
25573 001/300 Read block 1288 len 28672
25574 {"msg":"connecting to 127.0.0.1:8810","v":0,"name":"crucible","level":30002/300 Write block 1389 len 16384 data: 4 5 6 6
25575 ,"time":"2024-02-27T21:49:48.195499811Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1167,"":"io task","client":"0","":"downstairs","session_id":"e6a5a974-925a-494f-8845-facc441c2690"}
255762024-02-27T21:49:48.195ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
255772024-02-27T21:49:48.195ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
255782024-02-27T21:49:48.195ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
255792024-02-27T21:49:48.195ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
255802024-02-27T21:49:48.195ZINFOcrucible: [0] client re-new 1 jobs since flush 9740 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
255812024-02-27T21:49:48.195ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
25582 003/300 Read block 3878 len 20480
25583 004/300 Read block 4479 len 32768
25584 005/300 Read block 852 len 16384
25585 006/300 Write block 1332 len 4096 data: 7
25586 007/300 Read block 710 len 24576
25587 008/300 Read block 2127 len 12288
25588 009/300 Read block 3182 len 24576
25589 010/300 Flush
25590 011/300 Write block 3825 len 32768 data: 7 5 4 4 4 5 5 5
25591 012/300 Read block 1616 len 40960
25592 013/300 Write block 269 len 8192 data: 9 9
25593 014/300 Write block 3548 len 36864 data: 7 8 7 7 7 6 6 8 9
25594 015/300 Read block 1046 len 4096
25595 016/300 Write block 2127 len 8192 data: 8 7
25596 017/300 Write block 4764 len 8192 data: 6 8
25597 018/300 Read block 2761 len 28672
25598 019/300 Read block 1980 len 24576
25599 020/300 Flush
25600 021/300 Read block 118 len 16384
25601 022/300 Read block 3646 len 4096
25602 023/300 Read block 345 len 16384
25603 024/300 Read block 2711 len 36864
25604 025/300 Read block 4216 len 20480
25605 026/300 Write block 803 len 28672 data: 4 6 5 4 6 5 5
25606 027/300 Write block 2938 len 8192 data: 7 6
25607 028/300 Read block 1514 len 16384
25608 029/300 Read block 2811 len 16384
25609 030/300 Write block 543 len 8192 data: 6 6
25610 031/300 Read block 4406 len 16384
25611 032/300 Read block 1757 len 12288
25612 033/300 Write block 752 len 16384 data: 6 6 7 6
25613 034/300 Read block 1261 len 20480
25614 035/300 Read block 777 len 24576
25615 036/300 Read block 549 len 8192
25616 037/300 Read block 2907 len 8192
25617 038/300 Write block 2040 len 12288 data: 3 3 3
25618 039/300 Read block 2361 len 24576
25619 040/300 Read block 3230 len 4096
25620 041/300 Write block 3411 len 8192 data: 11 10
25621 042/300 Read block 3336 len 4096
25622 043/300 Write block 2329 len 16384 data: 5 6 8 5
25623 044/300 Read block 3141 len 4096
25624 045/300 Read block 1648 len 12288
25625 046/300 Read block 4629 len 36864
25626 047/300 Read block 1540 len 24576
25627 048/300 Read block 1508 len 16384
25628 049/300 Read block 1341 len 24576
25629 050/300 Write block 2462 len 32768 data: 8 8 7 5 6 6 6 7
25630 051/300 Write block 1774 len 8192 data: 3 3
25631 052/300 Read block 4829 len 20480
25632 053/300 Read block 1969 len 12288
25633 054/300 Read block 4198 len 36864
25634 055/300 Flush
25635 056/300 Write block 634 len 28672 data: 6 7 5 5 5 4 4
25636 057/300 Read block 1532 len 32768
25637 058/300 Read block 887 len 28672
25638 059/300 Read block 368 len 12288
25639 060/300 Read block 1957 len 12288
25640 061/300 Write block 1058 len 36864 data: 3 6 6 6 7 8 8 7 7
25641 062/300 Read block 2893 len 36864
25642 063/300 Read block 4928 len 4096
25643 064/300 Write block 3793 len 8192 data: 5 6
25644 065/300 Flush
25645 066/300 Write block 2004 len 24576 data: 7 6 6 6 5 5
25646 067/300 Read block 3254 len 12288
25647 068/300 Write block 4471 len 32768 data: 3 3 5 4 6 6 7 8
25648 069/300 Write block 4959 len 20480 data: 5 5 3 5 6
25649 070/300 Read block 1894 len 8192
25650 071/300 Flush
25651 072/300 Read block 4338 len 4096
25652 073/300 Read block 3779 len 16384
25653 074/300 Write block 4691 len 32768 data: 8 8 9 9 9 8 7 7
25654 075/300 Write block 3890 len 32768 data: 6 7 4 4 5 7 6 5
25655 076/300 Read block 2543 len 32768
25656 077/300 Write block 561 len 8192 data: 6 7
25657 078/300 Read block 53 len 8192
25658 079/300 Read block 803 len 24576
25659 080/300 Write block 2599 len 16384 data: 5 5 8 8
25660 081/300 Read block 1763 len 24576
25661 082/300 Write block 40 len 12288 data: 2 2 4
25662 083/300 Write block 974 len 12288 data: 7 7 7
25663 084/300 Read block 3868 len 24576
25664 085/300 Read block 729 len 12288
25665 086/300 Read block 3598 len 16384
25666 087/300 Read block 156 len 32768
25667 088/300 Read block 484 len 4096
25668 089/300 Read block 1233 len 4096
25669 090/300 Write block 3580 len 4096 data: 3
25670 091/300 Read block 149 len 40960
25671 092/300 Write block 4459 len 16384 data: 5 5 3 3
25672 093/300 Flush
25673 094/300 Write block 3259 len 12288 data: 6 5 4
25674 095/300 Read block 2686 len 4096
25675 096/300 Read block 86 len 36864
25676 097/300 Read block 20 len 28672
25677 098/300 Write block 443 len 24576 data: 3 3 3 4 4 4
25678 099/300 Read block 389 len 16384
25679 100/300 Read block 4131 len 32768
25680 101/300 Write block 391 len 36864 data: 4 6 6 7 7 8 8 7 9
25681 102/300 Flush
25682 103/300 Write block 3304 len 4096 data: 6
25683 104/300 Read block 2166 len 40960
25684 105/300 Read block 1528 len 24576
25685 106/300 Write block 8 len 8192 data: 6 6
25686 107/300 Read block 1153 len 4096
25687 108/300 Read block 4542 len 12288
25688 109/300 Write block 317 len 28672 data: 5 5 5 5 4 5 4
25689 110/300 Write block 958 len 8192 data: 10 9
25690 111/300 Write block 1644 len 24576 data: 7 8 8 7 6 8
25691 112/300 Write block 1010 len 28672 data: 5 6 6 8 9 7 6
25692 113/300 Read block 2194 len 28672
25693 114/300 Read block 4278 len 12288
25694 115/300 Read block 4037 len 16384
25695 116/300 Write block 4187 len 32768 data: 5 5 4 5 6 7 6 6
25696 117/300 Read block 3301 len 8192
25697 118/300 Read block 1253 len 40960
25698 119/300 Write block 1529 len 24576 data: 3 2 2 3 3 3
25699 120/300 Read block 936 len 40960
25700 121/300 Write block 3160 len 20480 data: 5 6 7 7 9
25701 122/300 Read block 1700 len 24576
25702 123/300 Write block 1051 len 40960 data: 6 8 7 7 6 4 4 4 7 7
25703 124/300 Read block 4111 len 32768
25704 125/300 Write block 1290 len 28672 data: 8 8 7 6 6 5 4
25705 126/300 Write block 1396 len 28672 data: 6 4 6 5 5 6 6
25706 127/300 Write block 4669 len 32768 data: 8 8 8 8 9 8 6 6
25707 128/300 Read block 498 len 20480
25708 129/300 Write block 4609 len 24576 data: 7 6 7 6 6 4
25709 130/300 Write block 3542 len 12288 data: 7 7 8
25710 131/300 Read block 1198 len 12288
25711 132/300 Read block 4868 len 12288
25712 133/300 Read block 1236 len 8192
25713 134/300 Read block 1030 len 16384
25714 135/300 Read block 2058 len 12288
25715 136/300 Write block 2099 len 4096 data: 6
25716 137/300 Read block 4150 len 12288
25717 138/300 Write block 1363 len 24576 data: 8 9 9 11 11 10
25718 139/300 Write block 1650 len 24576 data: 7 6 5 6 6 7
25719 140/300 Read block 4505 len 8192
25720 141/300 Read block 2426 len 16384
25721 142/300 Flush
25722 143/300 Write block 4804 len 16384 data: 5 5 4 4
25723 144/300 Read block 3693 len 4096
25724 145/300 Read block 4660 len 28672
25725 146/300 Write block 4290 len 4096 data: 3
25726 147/300 Read block 141 len 32768
25727 148/300 Write block 1749 len 4096 data: 6
25728 149/300 Read block 4235 len 16384
25729 150/300 Read block 59 len 16384
25730 151/300 Read block 46 len 16384
25731 152/300 Write block 4838 len 8192 data: 7 6
25732 153/300 Write block 3307 len 16384 data: 4 4 5 6
25733 154/300 Write block 4424 len 16384 data: 4 5 4 6
25734 155/300 Write block 4489 len 16384 data: 10 11 7 8
25735 156/300 Read block 1547 len 4096
25736 157/300 Write block 2122 len 24576 data: 9 9 7 7 7 9
25737 158/300 Write block 3397 len 40960 data: 5 5 7 8 7 7 7 5 6 5
25738 159/300 Read block 3028 len 36864
25739 160/300 Read block 563 len 4096
25740 161/300 Read block 4892 len 20480
25741 162/300 Write block 150 len 36864 data: 7 9 9 6 6 5 5 6 7
25742 163/300 Flush
25743 164/300 Read block 3146 len 12288
25744 165/300 Write block 1489 len 28672 data: 5 6 6 6 7 8 5
25745 166/300 Read block 2500 len 8192
25746 167/300 Read block 4715 len 4096
25747 168/300 Read block 3366 len 4096
25748 169/300 Read block 476 len 8192
25749 170/300 Write block 2116 len 24576 data: 5 5 5 6 7 8
25750 171/300 Read block 4072 len 16384
25751 172/300 Write block 1962 len 40960 data: 4 5 6 5 4 6 8 7 7 8
25752 173/300 Read block 3386 len 36864
25753 174/300 Read block 3163 len 8192
25754 175/300 Read block 179 len 4096
25755 176/300 Write block 4994 len 16384 data: 5 5 5 4
25756 177/300 Write block 259 len 8192 data: 8 7
25757 178/300 Write block 1494 len 40960 data: 9 6 4 3 4 6 7 9 8 6
25758 179/300 Read block 2031 len 40960
25759 180/300 Write block 4691 len 32768 data: 9 9 10 10 10 9 8 8
25760 181/300 Flush
25761 182/300 Read block 4253 len 8192
25762 183/300 Read block 2703 len 16384
25763 184/300 Write block 3497 len 24576 data: 8 7 6 6 6 5
25764 185/300 Flush
25765 186/300 Read block 4073 len 20480
25766 187/300 Write block 2349 len 36864 data: 9 8 7 7 6 8 9 11 9
25767 188/300 Read block 2862 len 36864
25768 189/300 Flush
25769 190/300 Read block 1265 len 24576
25770 191/300 Write block 3726 len 40960 data: 6 5 6 5 5 5 6 7 5 6
25771 192/300 Write block 624 len 32768 data: 8 7 6 6 4 5 6 6
25772 193/300 Read block 114 len 8192
25773 194/300 Read block 4846 len 20480
25774 195/300 Write block 38 len 36864 data: 4 3 3 3 5 4 7 7 9
25775 196/300 Read block 4887 len 32768
25776 197/300 Flush
25777 198/300 Read block 2590 len 28672
25778 199/300 Write block 1915 len 32768 data: 7 7 8 7 7 7 7 5
25779 200/300 Read block 2300 len 4096
25780 201/300 Write block 1598 len 28672 data: 7 8 7 7 6 7 4
25781 202/300 Write block 2031 len 24576 data: 5 4 4 4 4 4
25782 203/300 Read block 3412 len 12288
25783 204/300 Read block 2246 len 20480
25784 205/300 Read block 4366 len 20480
25785 206/300 Flush
25786 207/300 Flush
25787 208/300 Write block 1152 len 20480 data: 5 6 8 8 7
25788 209/300 Read block 191 len 40960
25789 210/300 Read block 2031 len 16384
25790 211/300 Write block 1570 len 24576 data: 8 5 5 4 4 5
25791 212/300 Read block 1253 len 16384
25792 213/300 Read block 895 len 40960
25793 214/300 Read block 770 len 40960
25794 215/300 Flush
25795 216/300 Write block 1521 len 28672 data: 4 5 5 4 5 5 3
25796 217/300 Read block 3915 len 8192
25797 218/300 Read block 2799 len 20480
25798 219/300 Write block 815 len 28672 data: 4 4 4 3 5 4 5
25799 220/300 Read block 4177 len 4096
25800 221/300 Write block 569 len 16384 data: 6 5 8 7
25801 222/300 Write block 2452 len 40960 data: 6 6 6 4 4 5 5 5 7 8
25802 223/300 Read block 4038 len 8192
25803 224/300 Read block 4624 len 24576
25804 225/300 Flush
25805 226/300 Flush
25806 227/300 Write block 3676 len 4096 data: 6
25807 228/300 Read block 1100 len 32768
25808 229/300 Write block 2709 len 8192 data: 7 5
25809 230/300 Read block 906 len 12288
25810 231/300 Read block 885 len 40960
25811 232/300 Write block 262 len 8192 data: 4 4
25812 233/300 Write block 1224 len 40960 data: 8 8 10 8 8 7 7 6 5 7
25813 234/300 Write block 3694 len 36864 data: 6 4 5 5 6 6 5 5 4
25814 235/300 Flush
25815 236/300 Read block 2583 len 40960
25816 237/300 Read block 2050 len 12288
25817 238/300 Read block 4610 len 8192
25818 239/300 Flush
25819 240/300 Flush
25820 241/300 Read block 1174 len 36864
25821 242/300 Read block 311 len 24576
25822 243/300 Write block 3411 len 20480 data: 12 11 9 8 8
25823 244/300 Write block 31 len 24576 data: 6 5 5 4 4 4
25824 245/300 Read block 1727 len 28672
25825 246/300 Read block 2657 len 16384
25826 247/300 Read block 3982 len 32768
25827 248/300 Write block 1926 len 24576 data: 7 7 9 9 9 7
25828 249/300 Write block 1904 len 32768 data: 5 4 4 4 4 4 6 6
25829 250/300 Read block 2509 len 12288
25830 251/300 Write block 3236 len 36864 data: 8 8 10 11 9 10 7 5 4
25831 252/300 Read block 2793 len 16384
25832 253/300 Read block 2964 len 36864
25833 254/300 Read block 3783 len 4096
25834 255/300 Write block 3827 len 24576 data: 5 5 5 6 6 6
25835 256/300 Write block 3317 len 8192 data: 11 11
25836 257/300 Write block 3139 len 8192 data: 6 5
25837 258/300 Read block 1926 len 16384
25838 259/300 Write block 4753 len 16384 data: 7 7 5 3
25839 260/300 Flush
25840 261/300 Read block 1449 len 40960
25841 262/300 Read block 987 len 4096
25842 263/300 Write block 3555 len 32768 data: 9 10 7 8 8 9 8 10
25843 264/300 Flush
25844 265/300 Read block 4499 len 32768
25845 266/300 Write block 1161 len 16384 data: 7 5 5 4
25846 267/300 Read block 4628 len 24576
25847 268/300 Write block 4555 len 16384 data: 5 3 5 6
25848 269/300 Read block 545 len 40960
25849 270/300 Write block 1186 len 36864 data: 4 4 6 7 7 9 7 7 9
25850 271/300 Flush
25851 272/300 Write block 883 len 28672 data: 6 5 4 5 7 6 7
25852 273/300 Read block 457 len 4096
25853 274/300 Read block 1960 len 36864
25854 275/300 Read block 4942 len 28672
25855 276/300 Write block 1205 len 16384 data: 8 6 7 8
25856 277/300 Read block 546 len 4096
25857 278/300 Read block 3946 len 4096
25858 279/300 Read block 1943 len 4096
25859 280/300 Write block 1213 len 12288 data: 6 5 6
25860 281/300 Read block 904 len 28672
25861 282/300 Write block 1652 len 36864 data: 6 7 7 8 7 6 7 7 9
25862 283/300 Read block 681 len 24576
25863 284/300 Read block 657 len 40960
25864 285/300 Read block 3573 len 36864
25865 286/300 Read block 1926 len 32768
25866 287/300 Read block 4542 len 36864
25867 288/300 Read block 363 len 8192
25868 289/300 Read block 499 len 4096
25869 290/300 Write block 4010 len 40960 data: 5 5 5 7 9 8 7 8 10 10
25870 291/300 Read block 1921 len 8192
25871 292/300 Write block 3938 len 40960 data: 11 10 8 8 8 8 7 9 10 8
25872 293/300 Write block 1682 len 40960 data: 5 6 8 8 8 5 5 4 5 4
25873 294/300 Read block 477 len 36864
25874 295/300 Write block 2157 len 40960 data: 5 8 8 9 7 6 7 6 6 5
25875 296/300 Read block 2492 len 36864
25876 297/300 Read block 4799 len 32768
25877 298/300 Read block 324 len 28672
25878 299/300 Read block 2380 len 24576
25879 300/300 Read block 3739 len 40960
25880 Main task has work to do, go find it
25881 got dsc Start(0)
25882 start 0
25883 Feb 27 21:49:48.528 INFO request completed, latency_us: 229, response_code: 204, uri: /start/cid/0, method: POST, req_id: f86cbd5b-63f0-418d-8e51-666ca37fa3a5, remote_addr: 127.0.0.1:42901, local_addr: 127.0.0.1:9998
25884 Replay: started 0, returned:Ok(())
25885 ----------------------------------------------------------------
25886 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
25887 States: Offline Active Active
25888 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
25889 8742 Acked 9741 Read 7 New Done Done true
25890 8743 Acked 9742 Write 4 New Done Done false
25891 8744 Acked 9743 Read 5 New Done Done false
25892 8745 Acked 9744 Read 8 New Done Done false
25893 8746 Acked 9745 Read 4 New Done Done false
25894 8747 Acked 9746 Write 1 New Done Done false
25895 8748 Acked 9747 Read 6 New Done Done false
25896 8749 Acked 9748 Read 3 New Done Done false
25897 8750 Acked 9749 Read 6 New Done Done false
25898 8751 Acked 9750 Flush 0 New Done Done false
25899 8752 Acked 9751 Write 8 New Done Done false
25900 8753 Acked 9752 Read 10 New Done Done false
25901 8754 Acked 9753 Write 2 New Done Done false
25902 8755 Acked 9754 Write 9 New Done Done false
25903 8756 Acked 9755 Read 1 New Done Done false
25904 8757 Acked 9756 Write 2 New Done Done false
25905 8758 Acked 9757 Write 2 New Done Done false
25906 8759 Acked 9758 Read 7 New Done Done false
25907 8760 Acked 9759 Read 6 New Done Done false
25908 8761 Acked 9760 Flush 0 New Done Done false
25909 8762 Acked 9761 Read 4 New Done Done false
25910 8763 Acked 9762 Read 1 New Done Done false
25911 8764 Acked 9763 Read 4 New Done Done false
25912 8765 Acked 9764 Read 9 New Done Done false
25913 8766 Acked 9765 Read 5 New Done Done false
25914 8767 Acked 9766 Write 7 New Done Done false
25915 8768 Acked 9767 Write 2 New Done Done false
25916 8769 Acked 9768 Read 4 New Done Done false
25917 8770 Acked 9769 Read 4 New Done Done false
25918 8771 Acked 9770 Write 2 New Done Done false
25919 8772 Acked 9771 Read 4 New Done Done false
25920 8773 Acked 9772 Read 3 New Done Done false
25921 8774 Acked 9773 Write 4 New Done Done false
25922 8775 Acked 9774 Read 5 New Done Done false
25923 8776 Acked 9775 Read 6 New Done Done false
25924 8777 Acked 9776 Read 2 New Done Done false
25925 8778 Acked 9777 Read 2 New Done Done false
25926 8779 Acked 9778 Write 3 New Done Done false
25927 8780 Acked 9779 Read 6 New Done Done false
25928 8781 Acked 9780 Read 1 New Done Done false
25929 8782 Acked 9781 Write 2 New Done Done false
25930 8783 Acked 9782 Read 1 New Done Done false
25931 8784 Acked 9783 Write 4 New Done Done false
25932 8785 Acked 9784 Read 1 New Done Done false
25933 8786 Acked 9785 Read 3 New Done Done false
25934 8787 Acked 9786 Read 9 New Done Done false
25935 8788 Acked 9787 Read 6 New Done Done false
25936 8789 Acked 9788 Read 4 New Done Done false
25937 8790 Acked 9789 Read 6 New Done Done false
25938 8791 Acked 9790 Write 8 New Done Done false
25939 8792 Acked 9791 Write 2 New Done Done false
25940 8793 Acked 9792 Read 5 New Done Done false
25941 8794 Acked 9793 Read 3 New Done Done false
25942 8795 Acked 9794 Read 9 New Done Done false
25943 8796 Acked 9795 Flush 0 New Done Done false
25944 8797 Acked 9796 Write 7 New Done Done false
25945 8798 Acked 9797 Read 8 New Done Done false
25946 8799 Acked 9798 Read 7 New Done Done false
25947 8800 Acked 9799 Read 3 New Done Done false
25948 8801 Acked 9800 Read 3 New Done Done false
25949 8802 Acked 9801 Write 9 New Done Done false
25950 8803 Acked 9802 Read 9 New Done Done false
25951 8804 Acked 9803 Read 1 New Done Done false
25952 8805 Acked 9804 Write 2 New Done Done false
25953 8806 Acked 9805 Flush 0 New Done Done false
25954 8807 Acked 9806 Write 6 New Done Done false
25955 8808 Acked 9807 Read 3 New Done Done false
25956 8809 Acked 9808 Write 8 New Done Done false
25957 8810 Acked 9809 Write 5 New Done Done false
25958 8811 Acked 9810 Read 2 New Done Done false
25959 8812 Acked 9811 Flush 0 New Done Done false
25960 8813 Acked 9812 Read 1 New Done Done false
25961 8814 Acked 9813 Read 4 New Done Done false
25962 8815 Acked 9814 Write 8 New Done Done false
25963 8816 Acked 9815 Write 8 New Done Done false
25964 8817 Acked 9816 Read 8 New Done Done false
25965 8818 Acked 9817 Write 2 New Done Done false
25966 8819 Acked 9818 Read 2 New Done Done false
25967 8820 Acked 9819 Read 6 New Done Done false
25968 8821 Acked 9820 Write 4 New Done Done false
25969 8822 Acked 9821 Read 6 New Done Done false
25970 8823 Acked 9822 Write 3 New Done Done false
25971 8824 Acked 9823 Write 3 New Done Done false
25972 8825 Acked 9824 Read 6 New Done Done false
25973 8826 Acked 9825 Read 3 New Done Done false
25974 8827 Acked 9826 Read 4 New Done Done false
25975 8828 Acked 9827 Read 8 New Done Done false
25976 8829 Acked 9828 Read 1 New Done Done false
25977 8830 Acked 9829 Read 1 New Done Done false
25978 8831 Acked 9830 Write 1 New Done Done false
25979 8832 Acked 9831 Read 10 New Done Done false
25980 8833 Acked 9832 Write 4 New Done Done false
25981 8834 Acked 9833 Flush 0 New Done Done false
25982 8835 Acked 9834 Write 3 New Done Done false
25983 8836 Acked 9835 Read 1 New Done Done false
25984 8837 Acked 9836 Read 9 New Done Done false
25985 8838 Acked 9837 Read 7 New Done Done false
25986 8839 Acked 9838 Write 6 New Done Done false
25987 8840 Acked 9839 Read 4 New Done Done false
25988 8841 Acked 9840 Read 8 New Done Done false
25989 8842 Acked 9841 Write 9 New Done Done false
25990 8843 Acked 9842 Flush 0 New Done Done false
25991 8844 Acked 9843 Write 1 New Done Done false
25992 8845 Acked 9844 Read 10 New Done Done false
25993 8846 Acked 9845 Read 6 New Done Done false
25994 8847 Acked 9846 Write 2 New Done Done false
25995 8848 Acked 9847 Read 1 New Done Done false
25996 8849 Acked 9848 Read 3 New Done Done false
25997 8850 Acked 9849 Write 7 New Done Done false
25998 8851 Acked 9850 Write 2 New Done Done false
25999 8852 Acked 9851 Write 6 New Done Done false
26000 8853 Acked 9852 Write 7 New Done Done false
26001 8854 Acked 9853 Read 7 New Done Done false
26002 8855 Acked 9854 Read 3 New Done Done false
26003 8856 Acked 9855 Read 4 New Done Done false
26004 8857 Acked 9856 Write 8 New Done Done false
26005 8858 Acked 9857 Read 2 New Done Done false
26006 8859 Acked 9858 Read 10 New Done Done false
26007 8860 Acked 9859 Write 6 New Done Done false
26008 8861 Acked 9860 Read 10 New Done Done false
26009 8862 Acked 9861 Write 5 New Done Done false
26010 8863 Acked 9862 Read 6 New Done Done false
26011 8864 Acked 9863 Write 10 New Done Done false
26012 8865 Acked 9864 Read 8 New Done Done false
26013 8866 Acked 9865 Write 7 New Done Done false
26014 8867 Acked 9866 Write 7 New Done Done false
26015 8868 Acked 9867 Write 8 New Done Done false
26016 8869 Acked 9868 Read 5 New Done Done false
26017 8870 Acked 9869 Write 6 New Done Done false
26018 8871 Acked 9870 Write 3 New Done Done false
26019 8872 Acked 9871 Read 3 New Done Done false
26020 8873 Acked 9872 Read 3 New Done Done false
26021 8874 Acked 9873 Read 2 New Done Done false
26022 8875 Acked 9874 Read 4 New Done Done false
26023 8876 Acked 9875 Read 3 New Done Done false
26024 8877 Acked 9876 Write 1 New Done Done false
26025 8878 Acked 9877 Read 3 New Done Done false
26026 8879 Acked 9878 Write 6 New Done Done false
26027 8880 Acked 9879 Write 6 New Done Done false
26028 8881 Acked 9880 Read 2 New Done Done false
26029 8882 Acked 9881 Read 4 New Done Done false
26030 8883 Acked 9882 Flush 0 New Done Done false
26031 8884 Acked 9883 Write 4 New Done Done false
26032 8885 Acked 9884 Read 1 New Done Done false
26033 8886 Acked 9885 Read 7 New Done Done false
26034 8887 Acked 9886 Write 1 New Done Done false
26035 8888 Acked 9887 Read 8 New Done Done false
26036 8889 Acked 9888 Write 1 New Done Done false
26037 8890 Acked 9889 Read 4 New Done Done false
26038 8891 Acked 9890 Read 4 New Done Done false
26039 8892 Acked 9891 Read 4 New Done Done false
26040 8893 Acked 9892 Write 2 New Done Done false
26041 8894 Acked 9893 Write 4 New Done Done false
26042 8895 Acked 9894 Write 4 New Done Done false
26043 8896 Acked 9895 Write 4 New Done Done false
26044 8897 Acked 9896 Read 1 New Done Done false
26045 8898 Acked 9897 Write 6 New Done Done false
26046 8899 Acked 9898 Write 10 New Done Done false
26047 8900 Acked 9899 Read 9 New Done Done false
26048 8901 Acked 9900 Read 1 New Done Done false
26049 8902 Acked 9901 Read 5 New Done Done false
26050 8903 Acked 9902 Write 9 New Done Done false
26051 8904 Acked 9903 Flush 0 New Done Done false
26052 8905 Acked 9904 Read 3 New Done Done false
26053 8906 Acked 9905 Write 7 New Done Done false
26054 8907 Acked 9906 Read 2 New Done Done false
26055 8908 Acked 9907 Read 1 New Done Done false
26056 8909 Acked 9908 Read 1 New Done Done false
26057 8910 Acked 9909 Read 2 New Done Done false
26058 8911 Acked 9910 Write 6 New Done Done false
26059 8912 Acked 9911 Read 4 New Done Done false
26060 8913 Acked 9912 Write 10 New Done Done false
26061 8914 Acked 9913 Read 9 New Done Done false
26062 8915 Acked 9914 Read 2 New Done Done false
26063 8916 Acked 9915 Read 1 New Done Done false
26064 8917 Acked 9916 Write 4 New Done Done false
26065 8918 Acked 9917 Write 2 New Done Done false
26066 8919 Acked 9918 Write 10 New Done Done false
26067 8920 Acked 9919 Read 10 New Done Done false
26068 8921 Acked 9920 Write 8 New Done Done false
26069 8922 Acked 9921 Flush 0 New Done Done false
26070 8923 Acked 9922 Read 2 New Done Done false
26071 8924 Acked 9923 Read 4 New Done Done false
26072 8925 Acked 9924 Write 6 New Done Done false
26073 8926 Acked 9925 Flush 0 New Done Done false
26074 8927 Acked 9926 Read 5 New Done Done false
26075 8928 Acked 9927 Write 9 New Done Done false
26076 8929 Acked 9928 Read 9 New Done Done false
26077 8930 Acked 9929 Flush 0 New Done Done false
26078 8931 Acked 9930 Read 6 New Done Done false
26079 8932 Acked 9931 Write 10 New Done Done false
26080 8933 Acked 9932 Write 8 New Done Done false
26081 8934 Acked 9933 Read 2 New Done Done false
26082 8935 Acked 9934 Read 5 New Done Done false
26083 8936 Acked 9935 Write 9 New Done Done false
26084 8937 Acked 9936 Read 8 New Done Done false
26085 8938 Acked 9937 Flush 0 New Done Done false
26086 8939 Acked 9938 Read 7 New Done Done false
26087 8940 Acked 9939 Write 8 New Done Done false
26088 8941 Acked 9940 Read 1 New Done Done false
26089 8942 Acked 9941 Write 7 New Done Done false
26090 8943 Acked 9942 Write 6 New Done Done false
26091 8944 Acked 9943 Read 3 New Done Done false
26092 8945 Acked 9944 Read 5 New Done Done false
26093 8946 Acked 9945 Read 5 New Done Done false
26094 8947 Acked 9946 Flush 0 New Done Done false
26095 8948 Acked 9947 Flush 0 New Done Done false
26096 8949 Acked 9948 Write 5 New Done Done false
26097 8950 Acked 9949 Read 10 New Done Done false
26098 8951 Acked 9950 Read 4 New Done Done false
26099 8952 Acked 9951 Write 6 New Done Done false
26100 8953 Acked 9952 Read 4 New Done Done false
26101 8954 Acked 9953 Read 10 New Done Done false
26102 8955 Acked 9954 Read 10 New Done Done false
26103 8956 Acked 9955 Flush 0 New Done Done false
26104 8957 Acked 9956 Write 7 New Done Done false
26105 8958 Acked 9957 Read 2 New Done Done false
26106 8959 Acked 9958 Read 5 New Done Done false
26107 8960 Acked 9959 Write 7 New Done Done false
26108 8961 Acked 9960 Read 1 New Done Done false
26109 8962 Acked 9961 Write 4 New Done Done false
26110 8963 Acked 9962 Write 10 New Done Done false
26111 8964 Acked 9963 Read 2 New Done Done false
26112 8965 Acked 9964 Read 6 New Done Done false
26113 8966 Acked 9965 Flush 0 New Done Done false
26114 8967 Acked 9966 Flush 0 New Done Done false
26115 8968 Acked 9967 Write 1 New Done Done false
26116 8969 Acked 9968 Read 8 New Done Done false
26117 8970 Acked 9969 Write 2 New Done Done false
26118 8971 Acked 9970 Read 3 New Done Done false
26119 8972 Acked 9971 Read 10 New Done Done false
26120 8973 Acked 9972 Write 2 New Done Done false
26121 8974 Acked 9973 Write 10 New Done Done false
26122 8975 Acked 9974 Write 9 New Done Done false
26123 8976 Acked 9975 Flush 0 New Done Done false
26124 8977 Acked 9976 Read 10 New Done Done false
26125 8978 Acked 9977 Read 3 New Done Done false
26126 8979 Acked 9978 Read 2 New Done Done false
26127 8980 Acked 9979 Flush 0 New Done Done false
26128 8981 Acked 9980 Flush 0 New Done Done false
26129 8982 Acked 9981 Read 9 New Done Done false
26130 8983 Acked 9982 Read 6 New Done Done false
26131 8984 Acked 9983 Write 5 New Done Done false
26132 8985 Acked 9984 Write 6 New Done Done false
26133 8986 Acked 9985 Read 7 New Done Done false
26134 8987 Acked 9986 Read 4 New Done Done false
26135 8988 Acked 9987 Read 8 New Done Done false
26136 8989 Acked 9988 Write 6 New Done Done false
26137 8990 Acked 9989 Write 8 New Done Done false
26138 8991 Acked 9990 Read 3 New Done Done false
26139 8992 Acked 9991 Write 9 New Done Done false
26140 8993 Acked 9992 Read 4 New Done Done false
26141 8994 Acked 9993 Read 9 New Done Done false
26142 8995 Acked 9994 Read 1 New Done Done false
26143 8996 Acked 9995 Write 6 New Done Done false
26144 8997 Acked 9996 Write 2 New Done Done false
26145 8998 Acked 9997 Write 2 New Done Done false
26146 8999 Acked 9998 Read 4 New Done Done false
26147 9000 Acked 9999 Write 4 New Done Done false
26148 9001 Acked 10000 Flush 0 New Done Done false
26149 9002 Acked 10001 Read 10 New Done Done false
26150 9003 Acked 10002 Read 1 New Done Done false
26151 9004 Acked 10003 Write 8 New Done Done false
26152 9005 Acked 10004 Flush 0 New Done Done false
26153 9006 Acked 10005 Read 8 New Done Done false
26154 9007 Acked 10006 Write 4 New Done Done false
26155 9008 Acked 10007 Read 6 New Done Done false
26156 9009 Acked 10008 Write 4 New Done Done false
26157 9010 Acked 10009 Read 10 New Done Done false
26158 9011 Acked 10010 Write 9 New Done Done false
26159 9012 Acked 10011 Flush 0 New Done Done false
26160 9013 Acked 10012 Write 7 New Done Done false
26161 9014 Acked 10013 Read 1 New Done Done false
26162 9015 Acked 10014 Read 9 New Done Done false
26163 9016 Acked 10015 Read 7 New Done Done false
26164 9017 Acked 10016 Write 4 New Done Done false
26165 9018 Acked 10017 Read 1 New Done Done false
26166 9019 Acked 10018 Read 1 New Done Done false
26167 9020 Acked 10019 Read 1 New Done Done false
26168 9021 Acked 10020 Write 3 New Done Done false
26169 9022 Acked 10021 Read 7 New Done Done false
26170 9023 Acked 10022 Write 9 New Done Done false
26171 9024 Acked 10023 Read 6 New Done Done false
26172 9025 Acked 10024 Read 10 New Done Done false
26173 9026 Acked 10025 Read 9 New Done Done false
26174 9027 Acked 10026 Read 8 New Done Done false
26175 9028 Acked 10027 Read 9 New Done Done false
26176 9029 Acked 10028 Read 2 New Done Done false
26177 9030 Acked 10029 Read 1 New Done Done false
26178 9031 Acked 10030 Write 10 New Done Done false
26179 9032 Acked 10031 Read 2 New Done Done false
26180 9033 Acked 10032 Write 10 New Done Done false
26181 9034 Acked 10033 Write 10 New Done Done false
26182 9035 Acked 10034 Read 9 New Done Done false
26183 9036 Acked 10035 Write 10 New Done Done false
26184 9037 Acked 10036 Read 9 New Done Done false
26185 9038 Acked 10037 Read 8 New Done Done false
26186 9039 Acked 10038 Read 7 New Done Done false
26187 9040 Acked 10039 Read 6 New Done Done false
26188 9041 Acked 10040 Read 10 New Done Done false
26189 STATES DS:0 DS:1 DS:2 TOTAL
26190 New 300 0 0 300
26191 Sent 0 0 0 0
26192 Done 0 300 300 600
26193 Skipped 0 0 0 0
26194 Error 0 0 0 0
26195 Last Flush: 9740 10011 10011
26196 Downstairs last five completed: 9740 9739 9738 9737 9736
26197 Upstairs last five completed: 9041 9040 9039 9038 9037
26198 CLIENT: Up:0 ds:300 act:2
261992024-02-27T21:49:49.196ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
262002024-02-27T21:49:49.196ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
262012024-02-27T21:49:49.196ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
262022024-02-27T21:49:49.196ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
262032024-02-27T21:49:49.196ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
262042024-02-27T21:49:49.196ZINFOcrucible: [0] client re-new 301 jobs since flush 9740 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
262052024-02-27T21:49:49.197ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
262062024-02-27T21:49:50.199ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
262072024-02-27T21:49:50.199ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
262082024-02-27T21:49:50.199ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
262092024-02-27T21:49:50.199ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
262102024-02-27T21:49:50.199ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
262112024-02-27T21:49:50.199ZINFOcrucible: [0] client re-new 301 jobs since flush 9740 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
262122024-02-27T21:49:50.199ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
262132024-02-27T21:49:51.200ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
262142024-02-27T21:49:51.200ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
262152024-02-27T21:49:51.200ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
262162024-02-27T21:49:51.200ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
262172024-02-27T21:49:51.200ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
262182024-02-27T21:49:51.200ZINFOcrucible: [0] client re-new 301 jobs since flush 9740 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
262192024-02-27T21:49:51.200ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
262202024-02-27T21:49:52.201ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
262212024-02-27T21:49:52.201ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
262222024-02-27T21:49:52.201ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
262232024-02-27T21:49:52.201ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
262242024-02-27T21:49:52.201ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
262252024-02-27T21:49:52.201ZINFOcrucible: [0] client re-new 301 jobs since flush 9740 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
262262024-02-27T21:49:52.201ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
26227 ----------------------------------------------------------------
26228 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26229 States: Offline Active Active
26230 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
26231 8742 Acked 9741 Read 7 New Done Done true
26232 8743 Acked 9742 Write 4 New Done Done true
26233 8744 Acked 9743 Read 5 New Done Done true
26234 8745 Acked 9744 Read 8 New Done Done true
26235 8746 Acked 9745 Read 4 New Done Done true
26236 8747 Acked 9746 Write 1 New Done Done true
26237 8748 Acked 9747 Read 6 New Done Done true
26238 8749 Acked 9748 Read 3 New Done Done true
26239 8750 Acked 9749 Read 6 New Done Done true
26240 8751 Acked 9750 Flush 0 New Done Done true
26241 8752 Acked 9751 Write 8 New Done Done true
26242 8753 Acked 9752 Read 10 New Done Done true
26243 8754 Acked 9753 Write 2 New Done Done true
26244 8755 Acked 9754 Write 9 New Done Done true
26245 8756 Acked 9755 Read 1 New Done Done true
26246 8757 Acked 9756 Write 2 New Done Done true
26247 8758 Acked 9757 Write 2 New Done Done true
26248 8759 Acked 9758 Read 7 New Done Done true
26249 8760 Acked 9759 Read 6 New Done Done true
26250 8761 Acked 9760 Flush 0 New Done Done true
26251 8762 Acked 9761 Read 4 New Done Done true
26252 8763 Acked 9762 Read 1 New Done Done true
26253 8764 Acked 9763 Read 4 New Done Done true
26254 8765 Acked 9764 Read 9 New Done Done true
26255 8766 Acked 9765 Read 5 New Done Done true
26256 8767 Acked 9766 Write 7 New Done Done true
26257 8768 Acked 9767 Write 2 New Done Done true
26258 8769 Acked 9768 Read 4 New Done Done true
26259 8770 Acked 9769 Read 4 New Done Done true
26260 8771 Acked 9770 Write 2 New Done Done true
26261 8772 Acked 9771 Read 4 New Done Done true
26262 8773 Acked 9772 Read 3 New Done Done true
26263 8774 Acked 9773 Write 4 New Done Done true
26264 8775 Acked 9774 Read 5 New Done Done true
26265 8776 Acked 9775 Read 6 New Done Done true
26266 8777 Acked 9776 Read 2 New Done Done true
26267 8778 Acked 9777 Read 2 New Done Done true
26268 8779 Acked 9778 Write 3 New Done Done true
26269 8780 Acked 9779 Read 6 New Done Done true
26270 8781 Acked 9780 Read 1 New Done Done true
26271 8782 Acked 9781 Write 2 New Done Done true
26272 8783 Acked 9782 Read 1 New Done Done true
26273 8784 Acked 9783 Write 4 New Done Done true
26274 8785 Acked 9784 Read 1 New Done Done true
26275 8786 Acked 9785 Read 3 New Done Done true
26276 8787 Acked 9786 Read 9 New Done Done true
26277 8788 Acked 9787 Read 6 New Done Done true
26278 8789 Acked 9788 Read 4 New Done Done true
26279 8790 Acked 9789 Read 6 New Done Done true
26280 8791 Acked 9790 Write 8 New Done Done true
26281 8792 Acked 9791 Write 2 New Done Done true
26282 8793 Acked 9792 Read 5 New Done Done true
26283 8794 Acked 9793 Read 3 New Done Done true
26284 8795 Acked 9794 Read 9 New Done Done true
26285 8796 Acked 9795 Flush 0 New Done Done true
26286 8797 Acked 9796 Write 7 New Done Done true
26287 8798 Acked 9797 Read 8 New Done Done true
26288 8799 Acked 9798 Read 7 New Done Done true
26289 8800 Acked 9799 Read 3 New Done Done true
26290 8801 Acked 9800 Read 3 New Done Done true
26291 8802 Acked 9801 Write 9 New Done Done true
26292 8803 Acked 9802 Read 9 New Done Done true
26293 8804 Acked 9803 Read 1 New Done Done true
26294 8805 Acked 9804 Write 2 New Done Done true
26295 8806 Acked 9805 Flush 0 New Done Done true
26296 8807 Acked 9806 Write 6 New Done Done true
26297 8808 Acked 9807 Read 3 New Done Done true
26298 8809 Acked 9808 Write 8 New Done Done true
26299 8810 Acked 9809 Write 5 New Done Done true
26300 8811 Acked 9810 Read 2 New Done Done true
26301 8812 Acked 9811 Flush 0 New Done Done true
26302 8813 Acked 9812 Read 1 New Done Done true
26303 8814 Acked 9813 Read 4 New Done Done true
26304 8815 Acked 9814 Write 8 New Done Done true
26305 8816 Acked 9815 Write 8 New Done Done true
26306 8817 Acked 9816 Read 8 New Done Done true
26307 8818 Acked 9817 Write 2 New Done Done true
26308 8819 Acked 9818 Read 2 New Done Done true
26309 8820 Acked 9819 Read 6 New Done Done true
26310 8821 Acked 9820 Write 4 New Done Done true
26311 8822 Acked 9821 Read 6 New Done Done true
26312 8823 Acked 9822 Write 3 New Done Done true
26313 8824 Acked 9823 Write 3 New Done Done true
26314 8825 Acked 9824 Read 6 New Done Done true
26315 8826 Acked 9825 Read 3 New Done Done true
26316 8827 Acked 9826 Read 4 New Done Done true
26317 8828 Acked 9827 Read 8 New Done Done true
26318 8829 Acked 9828 Read 1 New Done Done true
26319 8830 Acked 9829 Read 1 New Done Done true
26320 8831 Acked 9830 Write 1 New Done Done true
26321 8832 Acked 9831 Read 10 New Done Done true
26322 8833 Acked 9832 Write 4 New Done Done true
26323 8834 Acked 9833 Flush 0 New Done Done true
26324 8835 Acked 9834 Write 3 New Done Done true
26325 8836 Acked 9835 Read 1 New Done Done true
26326 8837 Acked 9836 Read 9 New Done Done true
26327 8838 Acked 9837 Read 7 New Done Done true
26328 8839 Acked 9838 Write 6 New Done Done true
26329 8840 Acked 9839 Read 4 New Done Done true
26330 8841 Acked 9840 Read 8 New Done Done true
26331 8842 Acked 9841 Write 9 New Done Done true
26332 8843 Acked 9842 Flush 0 New Done Done true
26333 8844 Acked 9843 Write 1 New Done Done true
26334 8845 Acked 9844 Read 10 New Done Done true
26335 8846 Acked 9845 Read 6 New Done Done true
26336 8847 Acked 9846 Write 2 New Done Done true
26337 8848 Acked 9847 Read 1 New Done Done true
26338 8849 Acked 9848 Read 3 New Done Done true
26339 8850 Acked 9849 Write 7 New Done Done true
26340 8851 Acked 9850 Write 2 New Done Done true
26341 8852 Acked 9851 Write 6 New Done Done true
26342 8853 Acked 9852 Write 7 New Done Done true
26343 8854 Acked 9853 Read 7 New Done Done true
26344 8855 Acked 9854 Read 3 New Done Done true
26345 8856 Acked 9855 Read 4 New Done Done true
26346 8857 Acked 9856 Write 8 New Done Done true
26347 8858 Acked 9857 Read 2 New Done Done true
26348 8859 Acked 9858 Read 10 New Done Done true
26349 8860 Acked 9859 Write 6 New Done Done true
26350 8861 Acked 9860 Read 10 New Done Done true
26351 8862 Acked 9861 Write 5 New Done Done true
26352 8863 Acked 9862 Read 6 New Done Done true
26353 8864 Acked 9863 Write 10 New Done Done true
26354 8865 Acked 9864 Read 8 New Done Done true
26355 8866 Acked 9865 Write 7 New Done Done true
26356 8867 Acked 9866 Write 7 New Done Done true
26357 8868 Acked 9867 Write 8 New Done Done true
26358 8869 Acked 9868 Read 5 New Done Done true
26359 8870 Acked 9869 Write 6 New Done Done true
26360 8871 Acked 9870 Write 3 New Done Done true
26361 8872 Acked 9871 Read 3 New Done Done true
26362 8873 Acked 9872 Read 3 New Done Done true
26363 8874 Acked 9873 Read 2 New Done Done true
26364 8875 Acked 9874 Read 4 New Done Done true
26365 8876 Acked 9875 Read 3 New Done Done true
26366 8877 Acked 9876 Write 1 New Done Done true
26367 8878 Acked 9877 Read 3 New Done Done true
26368 8879 Acked 9878 Write 6 New Done Done true
26369 8880 Acked 9879 Write 6 New Done Done true
26370 8881 Acked 9880 Read 2 New Done Done true
26371 8882 Acked 9881 Read 4 New Done Done true
26372 8883 Acked 9882 Flush 0 New Done Done true
26373 8884 Acked 9883 Write 4 New Done Done true
26374 8885 Acked 9884 Read 1 New Done Done true
26375 8886 Acked 9885 Read 7 New Done Done true
26376 8887 Acked 9886 Write 1 New Done Done true
26377 8888 Acked 9887 Read 8 New Done Done true
26378 8889 Acked 9888 Write 1 New Done Done true
26379 8890 Acked 9889 Read 4 New Done Done true
26380 8891 Acked 9890 Read 4 New Done Done true
26381 8892 Acked 9891 Read 4 New Done Done true
26382 8893 Acked 9892 Write 2 New Done Done true
26383 8894 Acked 9893 Write 4 New Done Done true
26384 8895 Acked 9894 Write 4 New Done Done true
26385 8896 Acked 9895 Write 4 New Done Done true
26386 8897 Acked 9896 Read 1 New Done Done true
26387 8898 Acked 9897 Write 6 New Done Done true
26388 8899 Acked 9898 Write 10 New Done Done true
26389 8900 Acked 9899 Read 9 New Done Done true
26390 8901 Acked 9900 Read 1 New Done Done true
26391 8902 Acked 9901 Read 5 New Done Done true
26392 8903 Acked 9902 Write 9 New Done Done true
26393 8904 Acked 9903 Flush 0 New Done Done true
26394 8905 Acked 9904 Read 3 New Done Done true
26395 8906 Acked 9905 Write 7 New Done Done true
26396 8907 Acked 9906 Read 2 New Done Done true
26397 8908 Acked 9907 Read 1 New Done Done true
26398 8909 Acked 9908 Read 1 New Done Done true
26399 8910 Acked 9909 Read 2 New Done Done true
26400 8911 Acked 9910 Write 6 New Done Done true
26401 8912 Acked 9911 Read 4 New Done Done true
26402 8913 Acked 9912 Write 10 New Done Done true
26403 8914 Acked 9913 Read 9 New Done Done true
26404 8915 Acked 9914 Read 2 New Done Done true
26405 8916 Acked 9915 Read 1 New Done Done true
26406 8917 Acked 9916 Write 4 New Done Done true
26407 8918 Acked 9917 Write 2 New Done Done true
26408 8919 Acked 9918 Write 10 New Done Done true
26409 8920 Acked 9919 Read 10 New Done Done true
26410 8921 Acked 9920 Write 8 New Done Done true
26411 8922 Acked 9921 Flush 0 New Done Done true
26412 8923 Acked 9922 Read 2 New Done Done true
26413 8924 Acked 9923 Read 4 New Done Done true
26414 8925 Acked 9924 Write 6 New Done Done true
26415 8926 Acked 9925 Flush 0 New Done Done true
26416 8927 Acked 9926 Read 5 New Done Done true
26417 8928 Acked 9927 Write 9 New Done Done true
26418 8929 Acked 9928 Read 9 New Done Done true
26419 8930 Acked 9929 Flush 0 New Done Done true
26420 8931 Acked 9930 Read 6 New Done Done true
26421 8932 Acked 9931 Write 10 New Done Done true
26422 8933 Acked 9932 Write 8 New Done Done true
26423 8934 Acked 9933 Read 2 New Done Done true
26424 8935 Acked 9934 Read 5 New Done Done true
26425 8936 Acked 9935 Write 9 New Done Done true
26426 8937 Acked 9936 Read 8 New Done Done true
26427 8938 Acked 9937 Flush 0 New Done Done true
26428 8939 Acked 9938 Read 7 New Done Done true
26429 8940 Acked 9939 Write 8 New Done Done true
26430 8941 Acked 9940 Read 1 New Done Done true
26431 8942 Acked 9941 Write 7 New Done Done true
26432 8943 Acked 9942 Write 6 New Done Done true
26433 8944 Acked 9943 Read 3 New Done Done true
26434 8945 Acked 9944 Read 5 New Done Done true
26435 8946 Acked 9945 Read 5 New Done Done true
26436 8947 Acked 9946 Flush 0 New Done Done true
26437 8948 Acked 9947 Flush 0 New Done Done true
26438 8949 Acked 9948 Write 5 New Done Done true
26439 8950 Acked 9949 Read 10 New Done Done true
26440 8951 Acked 9950 Read 4 New Done Done true
26441 8952 Acked 9951 Write 6 New Done Done true
26442 8953 Acked 9952 Read 4 New Done Done true
26443 8954 Acked 9953 Read 10 New Done Done true
26444 8955 Acked 9954 Read 10 New Done Done true
26445 8956 Acked 9955 Flush 0 New Done Done true
26446 8957 Acked 9956 Write 7 New Done Done true
26447 8958 Acked 9957 Read 2 New Done Done true
26448 8959 Acked 9958 Read 5 New Done Done true
26449 8960 Acked 9959 Write 7 New Done Done true
26450 8961 Acked 9960 Read 1 New Done Done true
26451 8962 Acked 9961 Write 4 New Done Done true
26452 8963 Acked 9962 Write 10 New Done Done true
26453 8964 Acked 9963 Read 2 New Done Done true
26454 8965 Acked 9964 Read 6 New Done Done true
26455 8966 Acked 9965 Flush 0 New Done Done true
26456 8967 Acked 9966 Flush 0 New Done Done true
26457 8968 Acked 9967 Write 1 New Done Done true
26458 8969 Acked 9968 Read 8 New Done Done true
26459 8970 Acked 9969 Write 2 New Done Done true
26460 8971 Acked 9970 Read 3 New Done Done true
26461 8972 Acked 9971 Read 10 New Done Done true
26462 8973 Acked 9972 Write 2 New Done Done true
26463 8974 Acked 9973 Write 10 New Done Done true
26464 8975 Acked 9974 Write 9 New Done Done true
26465 8976 Acked 9975 Flush 0 New Done Done true
26466 8977 Acked 9976 Read 10 New Done Done true
26467 8978 Acked 9977 Read 3 New Done Done true
26468 8979 Acked 9978 Read 2 New Done Done true
26469 8980 Acked 9979 Flush 0 New Done Done true
26470 8981 Acked 9980 Flush 0 New Done Done true
26471 8982 Acked 9981 Read 9 New Done Done true
26472 8983 Acked 9982 Read 6 New Done Done true
26473 8984 Acked 9983 Write 5 New Done Done true
26474 8985 Acked 9984 Write 6 New Done Done true
26475 8986 Acked 9985 Read 7 New Done Done true
26476 8987 Acked 9986 Read 4 New Done Done true
26477 8988 Acked 9987 Read 8 New Done Done true
26478 8989 Acked 9988 Write 6 New Done Done true
26479 8990 Acked 9989 Write 8 New Done Done true
26480 8991 Acked 9990 Read 3 New Done Done true
26481 8992 Acked 9991 Write 9 New Done Done true
26482 8993 Acked 9992 Read 4 New Done Done true
26483 8994 Acked 9993 Read 9 New Done Done true
26484 8995 Acked 9994 Read 1 New Done Done true
26485 8996 Acked 9995 Write 6 New Done Done true
26486 8997 Acked 9996 Write 2 New Done Done true
26487 8998 Acked 9997 Write 2 New Done Done true
26488 8999 Acked 9998 Read 4 New Done Done true
26489 9000 Acked 9999 Write 4 New Done Done true
26490 9001 Acked 10000 Flush 0 New Done Done true
26491 9002 Acked 10001 Read 10 New Done Done true
26492 9003 Acked 10002 Read 1 New Done Done true
26493 9004 Acked 10003 Write 8 New Done Done true
26494 9005 Acked 10004 Flush 0 New Done Done true
26495 9006 Acked 10005 Read 8 New Done Done true
26496 9007 Acked 10006 Write 4 New Done Done true
26497 9008 Acked 10007 Read 6 New Done Done true
26498 9009 Acked 10008 Write 4 New Done Done true
26499 9010 Acked 10009 Read 10 New Done Done true
26500 9011 Acked 10010 Write 9 New Done Done true
26501 9012 Acked 10011 Flush 0 New Done Done true
26502 9013 Acked 10012 Write 7 New Done Done true
26503 9014 Acked 10013 Read 1 New Done Done true
26504 9015 Acked 10014 Read 9 New Done Done true
26505 9016 Acked 10015 Read 7 New Done Done true
26506 9017 Acked 10016 Write 4 New Done Done true
26507 9018 Acked 10017 Read 1 New Done Done true
26508 9019 Acked 10018 Read 1 New Done Done true
26509 9020 Acked 10019 Read 1 New Done Done true
26510 9021 Acked 10020 Write 3 New Done Done true
26511 9022 Acked 10021 Read 7 New Done Done true
26512 9023 Acked 10022 Write 9 New Done Done true
26513 9024 Acked 10023 Read 6 New Done Done true
26514 9025 Acked 10024 Read 10 New Done Done true
26515 9026 Acked 10025 Read 9 New Done Done true
26516 9027 Acked 10026 Read 8 New Done Done true
26517 9028 Acked 10027 Read 9 New Done Done true
26518 9029 Acked 10028 Read 2 New Done Done true
26519 9030 Acked 10029 Read 1 New Done Done true
26520 9031 Acked 10030 Write 10 New Done Done true
26521 9032 Acked 10031 Read 2 New Done Done true
26522 9033 Acked 10032 Write 10 New Done Done true
26523 9034 Acked 10033 Write 10 New Done Done true
26524 9035 Acked 10034 Read 9 New Done Done true
26525 9036 Acked 10035 Write 10 New Done Done true
26526 9037 Acked 10036 Read 9 New Done Done true
26527 9038 Acked 10037 Read 8 New Done Done true
26528 9039 Acked 10038 Read 7 New Done Done true
26529 9040 Acked 10039 Read 6 New Done Done true
26530 9041 Acked 10040 Read 10 New Done Done true
26531 9042 Acked 10041 Flush 0 New Done Done true
26532 STATES DS:0 DS:1 DS:2 TOTAL
26533 New 301 0 0 301
26534 Sent 0 0 0 0
26535 Done 0 301 301 602
26536 Skipped 0 0 0 0
26537 Error 0 0 0 0
26538 Last Flush: 9740 10041 10041
26539 Downstairs last five completed: 9740 9739 9738 9737 9736
26540 Upstairs last five completed: 9042 9041 9040 9039 9038
26541 CLIENT: Up:0 ds:301 act:2
265422024-02-27T21:49:53.202ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
265432024-02-27T21:49:53.202ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
265442024-02-27T21:49:53.202ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
265452024-02-27T21:49:53.202ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
265462024-02-27T21:49:53.202ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
265472024-02-27T21:49:53.202ZINFOcrucible: [0] client re-new 301 jobs since flush 9740 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
265482024-02-27T21:49:53.202ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
265492024-02-27T21:49:54.203ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
265502024-02-27T21:49:54.203ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
265512024-02-27T21:49:54.203ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
265522024-02-27T21:49:54.204ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
265532024-02-27T21:49:54.204ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
265542024-02-27T21:49:54.204ZINFOcrucible: [0] client re-new 301 jobs since flush 9740 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
265552024-02-27T21:49:54.204ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
265562024-02-27T21:49:55.206ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
265572024-02-27T21:49:55.206ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
265582024-02-27T21:49:55.206ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
265592024-02-27T21:49:55.206ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
265602024-02-27T21:49:55.206ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
265612024-02-27T21:49:55.206ZINFOcrucible: [0] client re-new 301 jobs since flush 9740 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
265622024-02-27T21:49:55.206ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
265632024-02-27T21:49:56.207ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
265642024-02-27T21:49:56.207ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
265652024-02-27T21:49:56.207ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
265662024-02-27T21:49:56.207ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
265672024-02-27T21:49:56.207ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
265682024-02-27T21:49:56.207ZINFOcrucible: [0] client re-new 301 jobs since flush 9740 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
265692024-02-27T21:49:56.207ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
26570 ----------------------------------------------------------------
26571 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
26572 States: Offline Active Active
26573 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
26574 8742 Acked 9741 Read 7 New Done Done true
26575 8743 Acked 9742 Write 4 New Done Done true
26576 8744 Acked 9743 Read 5 New Done Done true
26577 8745 Acked 9744 Read 8 New Done Done true
26578 8746 Acked 9745 Read 4 New Done Done true
26579 8747 Acked 9746 Write 1 New Done Done true
26580 8748 Acked 9747 Read 6 New Done Done true
26581 8749 Acked 9748 Read 3 New Done Done true
26582 8750 Acked 9749 Read 6 New Done Done true
26583 8751 Acked 9750 Flush 0 New Done Done true
26584 8752 Acked 9751 Write 8 New Done Done true
26585 8753 Acked 9752 Read 10 New Done Done true
26586 8754 Acked 9753 Write 2 New Done Done true
26587 8755 Acked 9754 Write 9 New Done Done true
26588 8756 Acked 9755 Read 1 New Done Done true
26589 8757 Acked 9756 Write 2 New Done Done true
26590 8758 Acked 9757 Write 2 New Done Done true
26591 8759 Acked 9758 Read 7 New Done Done true
26592 8760 Acked 9759 Read 6 New Done Done true
26593 8761 Acked 9760 Flush 0 New Done Done true
26594 8762 Acked 9761 Read 4 New Done Done true
26595 8763 Acked 9762 Read 1 New Done Done true
26596 8764 Acked 9763 Read 4 New Done Done true
26597 8765 Acked 9764 Read 9 New Done Done true
26598 8766 Acked 9765 Read 5 New Done Done true
26599 8767 Acked 9766 Write 7 New Done Done true
26600 8768 Acked 9767 Write 2 New Done Done true
26601 8769 Acked 9768 Read 4 New Done Done true
26602 8770 Acked 9769 Read 4 New Done Done true
26603 8771 Acked 9770 Write 2 New Done Done true
26604 8772 Acked 9771 Read 4 New Done Done true
26605 8773 Acked 9772 Read 3 New Done Done true
26606 8774 Acked 9773 Write 4 New Done Done true
26607 8775 Acked 9774 Read 5 New Done Done true
26608 8776 Acked 9775 Read 6 New Done Done true
26609 8777 Acked 9776 Read 2 New Done Done true
26610 8778 Acked 9777 Read 2 New Done Done true
26611 8779 Acked 9778 Write 3 New Done Done true
26612 8780 Acked 9779 Read 6 New Done Done true
26613 8781 Acked 9780 Read 1 New Done Done true
26614 8782 Acked 9781 Write 2 New Done Done true
26615 8783 Acked 9782 Read 1 New Done Done true
26616 8784 Acked 9783 Write 4 New Done Done true
26617 8785 Acked 9784 Read 1 New Done Done true
26618 8786 Acked 9785 Read 3 New Done Done true
26619 8787 Acked 9786 Read 9 New Done Done true
26620 8788 Acked 9787 Read 6 New Done Done true
26621 8789 Acked 9788 Read 4 New Done Done true
26622 8790 Acked 9789 Read 6 New Done Done true
26623 8791 Acked 9790 Write 8 New Done Done true
26624 8792 Acked 9791 Write 2 New Done Done true
26625 8793 Acked 9792 Read 5 New Done Done true
26626 8794 Acked 9793 Read 3 New Done Done true
26627 8795 Acked 9794 Read 9 New Done Done true
26628 8796 Acked 9795 Flush 0 New Done Done true
26629 8797 Acked 9796 Write 7 New Done Done true
26630 8798 Acked 9797 Read 8 New Done Done true
26631 8799 Acked 9798 Read 7 New Done Done true
26632 8800 Acked 9799 Read 3 New Done Done true
26633 8801 Acked 9800 Read 3 New Done Done true
26634 8802 Acked 9801 Write 9 New Done Done true
26635 8803 Acked 9802 Read 9 New Done Done true
26636 8804 Acked 9803 Read 1 New Done Done true
26637 8805 Acked 9804 Write 2 New Done Done true
26638 8806 Acked 9805 Flush 0 New Done Done true
26639 8807 Acked 9806 Write 6 New Done Done true
26640 8808 Acked 9807 Read 3 New Done Done true
26641 8809 Acked 9808 Write 8 New Done Done true
26642 8810 Acked 9809 Write 5 New Done Done true
26643 8811 Acked 9810 Read 2 New Done Done true
26644 8812 Acked 9811 Flush 0 New Done Done true
26645 8813 Acked 9812 Read 1 New Done Done true
26646 8814 Acked 9813 Read 4 New Done Done true
26647 8815 Acked 9814 Write 8 New Done Done true
26648 8816 Acked 9815 Write 8 New Done Done true
26649 8817 Acked 9816 Read 8 New Done Done true
26650 8818 Acked 9817 Write 2 New Done Done true
26651 8819 Acked 9818 Read 2 New Done Done true
26652 8820 Acked 9819 Read 6 New Done Done true
26653 8821 Acked 9820 Write 4 New Done Done true
26654 8822 Acked 9821 Read 6 New Done Done true
26655 8823 Acked 9822 Write 3 New Done Done true
26656 8824 Acked 9823 Write 3 New Done Done true
26657 8825 Acked 9824 Read 6 New Done Done true
26658 8826 Acked 9825 Read 3 New Done Done true
26659 8827 Acked 9826 Read 4 New Done Done true
26660 8828 Acked 9827 Read 8 New Done Done true
26661 8829 Acked 9828 Read 1 New Done Done true
26662 8830 Acked 9829 Read 1 New Done Done true
26663 8831 Acked 9830 Write 1 New Done Done true
26664 8832 Acked 9831 Read 10 New Done Done true
26665 8833 Acked 9832 Write 4 New Done Done true
26666 8834 Acked 9833 Flush 0 New Done Done true
26667 8835 Acked 9834 Write 3 New Done Done true
26668 8836 Acked 9835 Read 1 New Done Done true
26669 8837 Acked 9836 Read 9 New Done Done true
26670 8838 Acked 9837 Read 7 New Done Done true
26671 8839 Acked 9838 Write 6 New Done Done true
26672 8840 Acked 9839 Read 4 New Done Done true
26673 8841 Acked 9840 Read 8 New Done Done true
26674 8842 Acked 9841 Write 9 New Done Done true
26675 8843 Acked 9842 Flush 0 New Done Done true
26676 8844 Acked 9843 Write 1 New Done Done true
26677 8845 Acked 9844 Read 10 New Done Done true
26678 8846 Acked 9845 Read 6 New Done Done true
26679 8847 Acked 9846 Write 2 New Done Done true
26680 8848 Acked 9847 Read 1 New Done Done true
26681 8849 Acked 9848 Read 3 New Done Done true
26682 8850 Acked 9849 Write 7 New Done Done true
26683 8851 Acked 9850 Write 2 New Done Done true
26684 8852 Acked 9851 Write 6 New Done Done true
26685 8853 Acked 9852 Write 7 New Done Done true
26686 8854 Acked 9853 Read 7 New Done Done true
26687 8855 Acked 9854 Read 3 New Done Done true
26688 8856 Acked 9855 Read 4 New Done Done true
26689 8857 Acked 9856 Write 8 New Done Done true
26690 8858 Acked 9857 Read 2 New Done Done true
26691 8859 Acked 9858 Read 10 New Done Done true
26692 8860 Acked 9859 Write 6 New Done Done true
26693 8861 Acked 9860 Read 10 New Done Done true
26694 8862 Acked 9861 Write 5 New Done Done true
26695 8863 Acked 9862 Read 6 New Done Done true
26696 8864 Acked 9863 Write 10 New Done Done true
26697 8865 Acked 9864 Read 8 New Done Done true
26698 8866 Acked 9865 Write 7 New Done Done true
26699 8867 Acked 9866 Write 7 New Done Done true
26700 8868 Acked 9867 Write 8 New Done Done true
26701 8869 Acked 9868 Read 5 New Done Done true
26702 8870 Acked 9869 Write 6 New Done Done true
26703 8871 Acked 9870 Write 3 New Done Done true
26704 8872 Acked 9871 Read 3 New Done Done true
26705 8873 Acked 9872 Read 3 New Done Done true
26706 8874 Acked 9873 Read 2 New Done Done true
26707 8875 Acked 9874 Read 4 New Done Done true
26708 8876 Acked 9875 Read 3 New Done Done true
26709 8877 Acked 9876 Write 1 New Done Done true
26710 8878 Acked 9877 Read 3 New Done Done true
26711 8879 Acked 9878 Write 6 New Done Done true
26712 8880 Acked 9879 Write 6 New Done Done true
26713 8881 Acked 9880 Read 2 New Done Done true
26714 8882 Acked 9881 Read 4 New Done Done true
26715 8883 Acked 9882 Flush 0 New Done Done true
26716 8884 Acked 9883 Write 4 New Done Done true
26717 8885 Acked 9884 Read 1 New Done Done true
26718 8886 Acked 9885 Read 7 New Done Done true
26719 8887 Acked 9886 Write 1 New Done Done true
26720 8888 Acked 9887 Read 8 New Done Done true
26721 8889 Acked 9888 Write 1 New Done Done true
26722 8890 Acked 9889 Read 4 New Done Done true
26723 8891 Acked 9890 Read 4 New Done Done true
26724 8892 Acked 9891 Read 4 New Done Done true
26725 8893 Acked 9892 Write 2 New Done Done true
26726 8894 Acked 9893 Write 4 New Done Done true
26727 8895 Acked 9894 Write 4 New Done Done true
26728 8896 Acked 9895 Write 4 New Done Done true
26729 8897 Acked 9896 Read 1 New Done Done true
26730 8898 Acked 9897 Write 6 New Done Done true
26731 8899 Acked 9898 Write 10 New Done Done true
26732 8900 Acked 9899 Read 9 New Done Done true
26733 8901 Acked 9900 Read 1 New Done Done true
26734 8902 Acked 9901 Read 5 New Done Done true
26735 8903 Acked 9902 Write 9 New Done Done true
26736 8904 Acked 9903 Flush 0 New Done Done true
26737 8905 Acked 9904 Read 3 New Done Done true
26738 8906 Acked 9905 Write 7 New Done Done true
26739 8907 Acked 9906 Read 2 New Done Done true
26740 8908 Acked 9907 Read 1 New Done Done true
26741 8909 Acked 9908 Read 1 New Done Done true
26742 8910 Acked 9909 Read 2 New Done Done true
26743 8911 Acked 9910 Write 6 New Done Done true
26744 8912 Acked 9911 Read 4 New Done Done true
26745 8913 Acked 9912 Write 10 New Done Done true
26746 8914 Acked 9913 Read 9 New Done Done true
26747 8915 Acked 9914 Read 2 New Done Done true
26748 8916 Acked 9915 Read 1 New Done Done true
26749 8917 Acked 9916 Write 4 New Done Done true
26750 8918 Acked 9917 Write 2 New Done Done true
26751 8919 Acked 9918 Write 10 New Done Done true
26752 8920 Acked 9919 Read 10 New Done Done true
26753 8921 Acked 9920 Write 8 New Done Done true
26754 8922 Acked 9921 Flush 0 New Done Done true
26755 8923 Acked 9922 Read 2 New Done Done true
26756 8924 Acked 9923 Read 4 New Done Done true
26757 8925 Acked 9924 Write 6 New Done Done true
26758 8926 Acked 9925 Flush 0 New Done Done true
26759 8927 Acked 9926 Read 5 New Done Done true
26760 8928 Acked 9927 Write 9 New Done Done true
26761 8929 Acked 9928 Read 9 New Done Done true
26762 8930 Acked 9929 Flush 0 New Done Done true
26763 8931 Acked 9930 Read 6 New Done Done true
26764 8932 Acked 9931 Write 10 New Done Done true
26765 8933 Acked 9932 Write 8 New Done Done true
26766 8934 Acked 9933 Read 2 New Done Done true
26767 8935 Acked 9934 Read 5 New Done Done true
26768 8936 Acked 9935 Write 9 New Done Done true
26769 8937 Acked 9936 Read 8 New Done Done true
26770 8938 Acked 9937 Flush 0 New Done Done true
26771 8939 Acked 9938 Read 7 New Done Done true
26772 8940 Acked 9939 Write 8 New Done Done true
26773 8941 Acked 9940 Read 1 New Done Done true
26774 8942 Acked 9941 Write 7 New Done Done true
26775 8943 Acked 9942 Write 6 New Done Done true
26776 8944 Acked 9943 Read 3 New Done Done true
26777 8945 Acked 9944 Read 5 New Done Done true
26778 8946 Acked 9945 Read 5 New Done Done true
26779 8947 Acked 9946 Flush 0 New Done Done true
26780 8948 Acked 9947 Flush 0 New Done Done true
26781 8949 Acked 9948 Write 5 New Done Done true
26782 8950 Acked 9949 Read 10 New Done Done true
26783 8951 Acked 9950 Read 4 New Done Done true
26784 8952 Acked 9951 Write 6 New Done Done true
26785 8953 Acked 9952 Read 4 New Done Done true
26786 8954 Acked 9953 Read 10 New Done Done true
26787 8955 Acked 9954 Read 10 New Done Done true
26788 8956 Acked 9955 Flush 0 New Done Done true
26789 8957 Acked 9956 Write 7 New Done Done true
26790 8958 Acked 9957 Read 2 New Done Done true
26791 8959 Acked 9958 Read 5 New Done Done true
26792 8960 Acked 9959 Write 7 New Done Done true
26793 8961 Acked 9960 Read 1 New Done Done true
26794 8962 Acked 9961 Write 4 New Done Done true
26795 8963 Acked 9962 Write 10 New Done Done true
26796 8964 Acked 9963 Read 2 New Done Done true
26797 8965 Acked 9964 Read 6 New Done Done true
26798 8966 Acked 9965 Flush 0 New Done Done true
26799 8967 Acked 9966 Flush 0 New Done Done true
26800 8968 Acked 9967 Write 1 New Done Done true
26801 8969 Acked 9968 Read 8 New Done Done true
26802 8970 Acked 9969 Write 2 New Done Done true
26803 8971 Acked 9970 Read 3 New Done Done true
26804 8972 Acked 9971 Read 10 New Done Done true
26805 8973 Acked 9972 Write 2 New Done Done true
26806 8974 Acked 9973 Write 10 New Done Done true
26807 8975 Acked 9974 Write 9 New Done Done true
26808 8976 Acked 9975 Flush 0 New Done Done true
26809 8977 Acked 9976 Read 10 New Done Done true
26810 8978 Acked 9977 Read 3 New Done Done true
26811 8979 Acked 9978 Read 2 New Done Done true
26812 8980 Acked 9979 Flush 0 New Done Done true
26813 8981 Acked 9980 Flush 0 New Done Done true
26814 8982 Acked 9981 Read 9 New Done Done true
26815 8983 Acked 9982 Read 6 New Done Done true
26816 8984 Acked 9983 Write 5 New Done Done true
26817 8985 Acked 9984 Write 6 New Done Done true
26818 8986 Acked 9985 Read 7 New Done Done true
26819 8987 Acked 9986 Read 4 New Done Done true
26820 8988 Acked 9987 Read 8 New Done Done true
26821 8989 Acked 9988 Write 6 New Done Done true
26822 8990 Acked 9989 Write 8 New Done Done true
26823 8991 Acked 9990 Read 3 New Done Done true
26824 8992 Acked 9991 Write 9 New Done Done true
26825 8993 Acked 9992 Read 4 New Done Done true
26826 8994 Acked 9993 Read 9 New Done Done true
26827 8995 Acked 9994 Read 1 New Done Done true
26828 8996 Acked 9995 Write 6 New Done Done true
26829 8997 Acked 9996 Write 2 New Done Done true
26830 8998 Acked 9997 Write 2 New Done Done true
26831 8999 Acked 9998 Read 4 New Done Done true
26832 9000 Acked 9999 Write 4 New Done Done true
26833 9001 Acked 10000 Flush 0 New Done Done true
26834 9002 Acked 10001 Read 10 New Done Done true
26835 9003 Acked 10002 Read 1 New Done Done true
26836 9004 Acked 10003 Write 8 New Done Done true
26837 9005 Acked 10004 Flush 0 New Done Done true
26838 9006 Acked 10005 Read 8 New Done Done true
26839 9007 Acked 10006 Write 4 New Done Done true
26840 9008 Acked 10007 Read 6 New Done Done true
26841 9009 Acked 10008 Write 4 New Done Done true
26842 9010 Acked 10009 Read 10 New Done Done true
26843 9011 Acked 10010 Write 9 New Done Done true
26844 9012 Acked 10011 Flush 0 New Done Done true
26845 9013 Acked 10012 Write 7 New Done Done true
26846 9014 Acked 10013 Read 1 New Done Done true
26847 9015 Acked 10014 Read 9 New Done Done true
26848 9016 Acked 10015 Read 7 New Done Done true
26849 9017 Acked 10016 Write 4 New Done Done true
26850 9018 Acked 10017 Read 1 New Done Done true
26851 9019 Acked 10018 Read 1 New Done Done true
26852 9020 Acked 10019 Read 1 New Done Done true
26853 9021 Acked 10020 Write 3 New Done Done true
26854 9022 Acked 10021 Read 7 New Done Done true
26855 9023 Acked 10022 Write 9 New Done Done true
26856 9024 Acked 10023 Read 6 New Done Done true
26857 9025 Acked 10024 Read 10 New Done Done true
26858 9026 Acked 10025 Read 9 New Done Done true
26859 9027 Acked 10026 Read 8 New Done Done true
26860 9028 Acked 10027 Read 9 New Done Done true
26861 9029 Acked 10028 Read 2 New Done Done true
26862 9030 Acked 10029 Read 1 New Done Done true
26863 9031 Acked 10030 Write 10 New Done Done true
26864 9032 Acked 10031 Read 2 New Done Done true
26865 9033 Acked 10032 Write 10 New Done Done true
26866 9034 Acked 10033 Write 10 New Done Done true
26867 9035 Acked 10034 Read 9 New Done Done true
26868 9036 Acked 10035 Write 10 New Done Done true
26869 9037 Acked 10036 Read 9 New Done Done true
26870 9038 Acked 10037 Read 8 New Done Done true
26871 9039 Acked 10038 Read 7 New Done Done true
26872 9040 Acked 10039 Read 6 New Done Done true
26873 9041 Acked 10040 Read 10 New Done Done true
26874 9042 Acked 10041 Flush 0 New Done Done true
26875 STATES DS:0 DS:1 DS:2 TOTAL
26876 New 301 0 0 301
26877 Sent 0 0 0 0
26878 Done 0 301 301 602
26879 Skipped 0 0 0 0
26880 Error 0 0 0 0
26881 Last Flush: 9740 10041 10041
26882 Downstairs last five completed: 9740 9739 9738 9737 9736
26883 Upstairs last five completed: 9042 9041 9040 9039 9038
26884 CLIENT: Up:0 ds:301 act:2
268852024-02-27T21:49:57.208ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
268862024-02-27T21:49:57.209ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
268872024-02-27T21:49:57.209ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
268882024-02-27T21:49:57.209ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
268892024-02-27T21:49:57.209ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
268902024-02-27T21:49:57.209ZINFOcrucible: [0] client re-new 301 jobs since flush 9740 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
268912024-02-27T21:49:57.209ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
26892 [8810] got start action
26893 [8810] I am going to restart
26894 Starting downstairs at port 8810
26895 Make output file at "/tmp/dsc/downstairs-8810.txt"
26896 [8810][0] reports Starting
26897 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1197)
26898 [8810][0] reports Running
268992024-02-27T21:49:58.211ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
269002024-02-27T21:49:58.211ZWARNcrucible: ds_connection connect to 127.0.0.1:8810 failure: Os { code: 146, kind: ConnectionRefused, message: "Connection refused" } = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
269012024-02-27T21:49:58.211ZWARNcrucible: client task is sending Done(ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" })) = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
269022024-02-27T21:49:58.211ZWARNcrucible: downstairs task for 0 stopped due to ConnectionFailed(Os { code: 146, kind: ConnectionRefused, message: "Connection refused" }) session_id = e6a5a974-925a-494f-8845-facc441c2690
269032024-02-27T21:49:58.211ZINFOcrucible: Still missing, state is Offline = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
269042024-02-27T21:49:58.211ZINFOcrucible: [0] client re-new 301 jobs since flush 9740 = downstairs session_id = e6a5a974-925a-494f-8845-facc441c2690
269052024-02-27T21:49:58.211ZINFOcrucible: client task is exiting = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
269062024-02-27T21:49:59.212ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
269072024-02-27T21:49:59.212ZINFOcrucible: ds_connection connected = io task client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
269082024-02-27T21:49:59.214ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
269092024-02-27T21:49:59.214ZINFOcrucible: Returning UUID:12345678-0000-0000-0000-000000008810 matches = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
269102024-02-27T21:49:59.214ZINFOcrucible: send last flush ID to this DS: 9740 = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
269112024-02-27T21:49:59.214ZINFOcrucible: Replied this last flush ID: 9740 = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
269122024-02-27T21:49:59.214ZINFOcrucible: ds_transition from Offline to Replay = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
269132024-02-27T21:49:59.214ZINFOcrucible: [0] Transition from Offline to Replay = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
269142024-02-27T21:49:59.214ZINFOcrucible: ds_transition from Replay to Active = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
269152024-02-27T21:49:59.214ZINFOcrucible: [0] Transition from Replay to Active = downstairs client = 0 session_id = e6a5a974-925a-494f-8845-facc441c2690
26916 ----------------------------------------------------------------
26917 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
26918 Downstairs last five completed: 10041 10040 10039 10038 10037
26919 Upstairs last five completed: 9042 9041 9040 9039 9038
26920 CLIENT: Up:0 ds:0 act:3
26921 Replay: All jobs finished, all DS active.
26922 Test replay has completed
26923 Wrote out file "alan"
26924 CLIENT: Tests done. All submitted work has been ACK'd
26925 ----------------------------------------------------------------
26926 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
26927 Downstairs last five completed: 10041 10040 10039 10038 10037
26928 Upstairs last five completed: 9042 9041 9040 9039 9038
26929 CLIENT: Up:0 ds:0 act:3
26930 CLIENT: All jobs finished, all DS active.
26931 Replays:30 time: 5:19
26932 Do final verify
269332024-02-27T21:50:00.569ZINFOcrucible: Upstairs starts
269342024-02-27T21:50:00.570ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "f6c40444ff68361c835e88ee91c409e8deb48d7d", git_commit_timestamp: "2024-02-27T21:23:06.000000000Z", git_branch: "main", rustc_semver: "1.75.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "82e1608dfa6e0b5569232559e3d385fea5a93112", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
26935 {"msg":"Crucible runtime is spawned
26936 Upstairs <-> Downstairs Message Version: 5","v":0,"name":"crucible","level":30,"time":"2024-02-27T21:50:00.570311295Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1202}
269372024-02-27T21:50:00.570ZINFOcrucible: Crucible 7ecd8009-ec4e-4c96-a0f5-1e2eb36635a6 has session id: f176449c-621e-4a72-bdf2-beee47d61635 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269382024-02-27T21:50:00.570ZINFOcrucible: Upstairs opts: Upstairs UUID: 7ecd8009-ec4e-4c96-a0f5-1e2eb36635a6, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false session_id = f176449c-621e-4a72-bdf2-beee47d61635
269392024-02-27T21:50:00.570ZINFOcrucible: Crucible stats registered with UUID: 7ecd8009-ec4e-4c96-a0f5-1e2eb36635a6 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269402024-02-27T21:50:00.570ZINFOcrucible: The guest has requested activation with gen:3
269412024-02-27T21:50:00.570ZINFOcrucible: 7ecd8009-ec4e-4c96-a0f5-1e2eb36635a6 active request set session_id = f176449c-621e-4a72-bdf2-beee47d61635
269422024-02-27T21:50:00.570ZINFOcrucible: sending connect oneshot to client = downstairs client = 0 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269432024-02-27T21:50:00.570ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 0 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269442024-02-27T21:50:00.570ZINFOcrucible: sending connect oneshot to client = downstairs client = 1 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269452024-02-27T21:50:00.570ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 1 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269462024-02-27T21:50:00.570ZINFOcrucible: sending connect oneshot to client = downstairs client = 2 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269472024-02-27T21:50:00.570ZINFOcrucible: client set_active_request while in New; waiting... = downstairs client = 2 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269482024-02-27T21:50:00.570ZINFOcrucible: connecting to 127.0.0.1:8830 = io task client = 2 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269492024-02-27T21:50:00.570ZINFOcrucible: connecting to 127.0.0.1:8810 = io task client = 0 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269502024-02-27T21:50:00.570ZINFOcrucible: connecting to 127.0.0.1:8820 = io task client = 1 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269512024-02-27T21:50:00.571ZINFOcrucible: ds_connection connected = io task client = 0 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269522024-02-27T21:50:00.571ZINFOcrucible: ds_connection connected = io task client = 2 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269532024-02-27T21:50:00.571ZINFOcrucible: ds_connection connected = io task client = 1 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269542024-02-27T21:50:00.571ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 2 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269552024-02-27T21:50:00.571ZINFOcrucible: [2] Transition from New to WaitActive = downstairs client = 2 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269562024-02-27T21:50:00.571ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 1 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269572024-02-27T21:50:00.571ZINFOcrucible: [1] Transition from New to WaitActive = downstairs client = 1 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269582024-02-27T21:50:00.571ZINFOcrucible: ds_transition from New to WaitActive = downstairs client = 0 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269592024-02-27T21:50:00.571ZINFOcrucible: [0] Transition from New to WaitActive = downstairs client = 0 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269602024-02-27T21:50:00.572ZINFOcrucible: downstairs client at Some(127.0.0.1:8810) has region UUID 12345678-0000-0000-0000-000000008810 = downstairs client = 0 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269612024-02-27T21:50:00.572ZINFOcrucible: downstairs client at Some(127.0.0.1:8820) has region UUID 12345678-0000-0000-0000-000000008820 = downstairs client = 1 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269622024-02-27T21:50:00.572ZINFOcrucible: downstairs client at Some(127.0.0.1:8830) has region UUID 12345678-0000-0000-0000-000000008830 = downstairs client = 2 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269632024-02-27T21:50:00.573ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269642024-02-27T21:50:00.573ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum = downstairs client = 2 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269652024-02-27T21:50:00.573ZINFOcrucible: Waiting for true more clients to be ready session_id = f176449c-621e-4a72-bdf2-beee47d61635
269662024-02-27T21:50:00.573ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269672024-02-27T21:50:00.573ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum = downstairs client = 1 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269682024-02-27T21:50:00.573ZINFOcrucible: Waiting for true more clients to be ready session_id = f176449c-621e-4a72-bdf2-beee47d61635
269692024-02-27T21:50:00.573ZINFOcrucible: ds_transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269702024-02-27T21:50:00.573ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum = downstairs client = 0 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269712024-02-27T21:50:00.573ZINFOcrucible: [0]R flush_numbers[0..12]: [885, 872, 882, 871, 870, 880, 876, 866, 888, 871, 871, 887] = downstairs session_id = f176449c-621e-4a72-bdf2-beee47d61635
269722024-02-27T21:50:00.573ZINFOcrucible: [0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = f176449c-621e-4a72-bdf2-beee47d61635
269732024-02-27T21:50:00.573ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = f176449c-621e-4a72-bdf2-beee47d61635
269742024-02-27T21:50:00.573ZINFOcrucible: [1]R flush_numbers[0..12]: [885, 872, 882, 871, 870, 880, 876, 866, 888, 871, 871, 887] = downstairs session_id = f176449c-621e-4a72-bdf2-beee47d61635
269752024-02-27T21:50:00.573ZINFOcrucible: [1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = f176449c-621e-4a72-bdf2-beee47d61635
269762024-02-27T21:50:00.573ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false] = downstairs session_id = f176449c-621e-4a72-bdf2-beee47d61635
269772024-02-27T21:50:00.573ZINFOcrucible: [2]R flush_numbers[0..12]: [885, 872, 882, 871, 870, 880, 876, 866, 888, 871, 871, 887] = downstairs session_id = f176449c-621e-4a72-bdf2-beee47d61635
269782024-02-27T21:50:00.573ZINFOcrucible: [2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2] = downstairs session_id = f176449c-621e-4a72-bdf2-beee47d61635
26979 {"msg":"[2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":Wait for a query_work_queue command to finish before sending IO
26980 0,"name":"crucible","level":30,"time":"2024-02-27T21:50:00.573684275Z","hostname":"ip-10-150-1-114.us-west-2.compute.internal","pid":1202,"":"downstairs","":"downstairs","session_id":"f176449c-621e-4a72-bdf2-beee47d61635"}
269812024-02-27T21:50:00.573ZINFOcrucible: Max found gen is 3 = downstairs session_id = f176449c-621e-4a72-bdf2-beee47d61635
269822024-02-27T21:50:00.573ZINFOcrucible: Generation requested: 3 >= found:3 = downstairs session_id = f176449c-621e-4a72-bdf2-beee47d61635
269832024-02-27T21:50:00.573ZINFOcrucible: Next flush: 889 = downstairs session_id = f176449c-621e-4a72-bdf2-beee47d61635
269842024-02-27T21:50:00.573ZINFOcrucible: All extents match = downstairs session_id = f176449c-621e-4a72-bdf2-beee47d61635
269852024-02-27T21:50:00.573ZINFOcrucible: No downstairs reconciliation required session_id = f176449c-621e-4a72-bdf2-beee47d61635
269862024-02-27T21:50:00.573ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 0 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269872024-02-27T21:50:00.573ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 1 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269882024-02-27T21:50:00.573ZINFOcrucible: Transition from WaitQuorum to Active = downstairs client = 2 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269892024-02-27T21:50:00.573ZINFOcrucible: All required reconciliation work is completed session_id = f176449c-621e-4a72-bdf2-beee47d61635
269902024-02-27T21:50:00.573ZINFOcrucible: Set Downstairs and Upstairs active after reconciliation session_id = f176449c-621e-4a72-bdf2-beee47d61635
269912024-02-27T21:50:00.573ZINFOcrucible: 7ecd8009-ec4e-4c96-a0f5-1e2eb36635a6 is now active with session: f176449c-621e-4a72-bdf2-beee47d61635 session_id = f176449c-621e-4a72-bdf2-beee47d61635
269922024-02-27T21:50:00.573ZINFOcrucible: Set Active after no reconciliation session_id = f176449c-621e-4a72-bdf2-beee47d61635
269932024-02-27T21:50:00.573ZINFOcrucible: The guest has finished waiting for activation with:3
26994 Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
26995 Loading write count information from file "alan"
26996 Read and Verify all blocks (0..5000 range:false)
26997 Wrote out file "alan"
26998 Verify test completed
26999 Wrote out file "alan"
27000 CLIENT: Tests done. All submitted work has been ACK'd
27001 ----------------------------------------------------------------
27002 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:5
27003 States: Active Active Active
27004 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
27005 47 Acked 1046 Read 100 Done Done Done false
27006 48 Acked 1047 Read 100 Done Done Done false
27007 49 Acked 1048 Read 100 Done Done Done false
27008 50 Acked 1049 Read 100 Done Done Done false
27009 51 Acked 1050 Read 100 Done Done Done false
27010 STATES DS:0 DS:1 DS:2 TOTAL
27011 New 0 0 0 0
27012 Sent 0 0 0 0
27013 Done 5 5 5 15
27014 Skipped 0 0 0 0
27015 Error 0 0 0 0
27016 Last Flush: 1045 1045 1045
27017 Downstairs last five completed: 1045 1044 1043 1042 1041
27018 Upstairs last five completed: 51 50 49 48 47
27019 CLIENT: Up:0 ds:5 act:3
27020 ----------------------------------------------------------------
27021 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
27022 Downstairs last five completed: 1051 1050 1049 1048 1047
27023 Upstairs last five completed: 52 51 50 49 48
27024 CLIENT: Up:0 ds:0 act:3
27025 CLIENT: All crucible jobs finished, exiting program
27026 Feb 27 21:50:05.201 INFO accepted connection, remote_addr: 127.0.0.1:50024, local_addr: 127.0.0.1:9998
27027 Main task has work to do, go find it
27028 got dsc Shutdown
27029 Shutdown
27030 Shut it down
27031 Feb 27 21:50:05.201 INFO request completed, latency_us: 199, response_code: 204, uri: /shutdown, method: POST, req_id:[8810] Disable keep_running
27032 687ebdf4-b69a-4d25-bc56-bd9570466f4c, remote_addr: 127.0.0.1:50024[8830] Disable keep_running
27033 , local_addr: 127.0[8810] Got stop action so:false kr:false
27034 .[8830] Got stop action so:false kr:false
27035 0[8820] Disable keep_running
27036 .[8820] Got stop action so:false kr:false
27037 1:9998
27038 [8830] Exited with: ExitStatus(unix_wait_status(9))
27039 [8810] Exited with: ExitStatus(unix_wait_status(9))
27040 [8830][2] reports Exit
27041 [8820] Exited with: ExitStatus(unix_wait_status(9))
27042 [8810][0] reports Exit
27043 [8820][1] reports Exit
27044 February 27, 2024 at 09:50:10 PM UTC Test ends with 0