Max level shown:
1 starting October 13, 2023 at 04:41:36 AM UTC
2 Creating downstairs regions
3 Creating region directory at: /var/tmp/dsc/region
4 Creating dsc directory at: /tmp/dsc
5 Downstairs region 0 created at /var/tmp/dsc/region/8810 in 0.2862753
6 Downstairs region 1 created at /var/tmp/dsc/region/8820 in 0.15712406
7 Downstairs region 2 created at /var/tmp/dsc/region/8830 in 0.1549153
8 Region set with 3 regions was created
9 Starting downstairs
10 Using existing output directory "/tmp/dsc"
11 start ds: 8810
12 start ds: 8820
13 start access at:127.0.0.1:9998
14 Starting downstairs at port 8810
15 Starting downstairs at port 8820
16 start ds: 8830
17 Make output file at "/tmp/dsc/downstairs-8820.txt"
18 Make output file at "/tmp/dsc/downstairs-8810.txt"
19 Starting downstairs at port 8830
20 Make output file at "/tmp/dsc/downstairs-8830.txt"
21 [8820][1] reports Starting
22 [8810][0] reports Starting
23 [8830][2] reports Starting
24 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1166)
25 [8820][1] reports Running
26 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1167)
27 [8830][2] reports Running
28 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1168)
29 [8810][0] reports Running
30 Oct 13 04:41:37.051 INFO listening, local_addr: 127.0.0.1:9998
31 Control access at:127.0.0.1:9998
32 Running initial fill
332023-10-13T04:41:42.147ZINFOcrucible: Upstairs starts
342023-10-13T04:41:42.149ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "0a6b3facda3a9a8c75c70b591db2d34571e435e8", git_commit_timestamp: "2023-10-13T04:02:07.000000000Z", git_branch: "alan/unwrapless", rustc_semver: "1.70.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "90c541806f23a127002de5b4038be731ba1458ca", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
352023-10-13T04:41:42.149ZINFOcrucible: Upstairs <-> Downstairs Message Version: 4
362023-10-13T04:41:42.149ZINFOcrucible: Upstairs opts: Upstairs UUID: f1b5b3ef-a58a-487e-ab3b-dc4227e30424, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false
372023-10-13T04:41:42.149ZINFOcrucible: Crucible stats registered with UUID: f1b5b3ef-a58a-487e-ab3b-dc4227e30424
382023-10-13T04:41:42.149ZINFOcrucible: Crucible f1b5b3ef-a58a-487e-ab3b-dc4227e30424 has session id: ac1f3a08-cbb1-4257-b829-5d40422fceac
39 Crucible runtime is spawned
402023-10-13T04:41:42.151ZINFOcrucible: up_listen starts task = up_listen
412023-10-13T04:41:42.151ZINFOcrucible: Wait for all three downstairs to come online
42 {"msgThe guest has requested activation with gen:1
43 ":"Flush timeout: 0.5","v":0,"name":"crucible","level":30,"time":"2023-10-13T04:41:42.151971523Z","hostname":"ip-10-150-1-57.us-west-2.compute.internal","pid":1172}
442023-10-13T04:41:42.152ZINFOcrucible: [0] connecting to 127.0.0.1:8810 looper = 0
452023-10-13T04:41:42.153ZINFOcrucible: [0] f1b5b3ef-a58a-487e-ab3b-dc4227e30424 looper connected looper = 0
462023-10-13T04:41:42.153ZINFOcrucible: [1] connecting to 127.0.0.1:8820 looper = 1
472023-10-13T04:41:42.153ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
482023-10-13T04:41:42.153ZINFOcrucible: [1] f1b5b3ef-a58a-487e-ab3b-dc4227e30424 looper connected looper = 1
492023-10-13T04:41:42.153ZINFOcrucible: [2] f1b5b3ef-a58a-487e-ab3b-dc4227e30424 looper connected looper = 2
502023-10-13T04:41:42.154ZINFOcrucible: [0] Proc runs for 127.0.0.1:8810 in state New
512023-10-13T04:41:42.154ZINFOcrucible: f1b5b3ef-a58a-487e-ab3b-dc4227e30424 active request set
522023-10-13T04:41:42.154ZINFOcrucible: Set desired generation to :1
532023-10-13T04:41:42.155ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state New
542023-10-13T04:41:42.155ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state New
552023-10-13T04:41:42.157ZINFOcrucible: [2] f1b5b3ef-a58a-487e-ab3b-dc4227e30424 (5a8631c8-4d7b-4b1b-bac7-a178bee40f34) New New New ds_transition to WaitActive
562023-10-13T04:41:42.157ZINFOcrucible: [2] Transition from New to WaitActive
572023-10-13T04:41:42.157ZINFOcrucible: [1] f1b5b3ef-a58a-487e-ab3b-dc4227e30424 (5a8631c8-4d7b-4b1b-bac7-a178bee40f34) New New WaitActive ds_transition to WaitActive
582023-10-13T04:41:42.157ZINFOcrucible: [1] Transition from New to WaitActive
592023-10-13T04:41:42.157ZINFOcrucible: [0] f1b5b3ef-a58a-487e-ab3b-dc4227e30424 (5a8631c8-4d7b-4b1b-bac7-a178bee40f34) New WaitActive WaitActive ds_transition to WaitActive
602023-10-13T04:41:42.157ZINFOcrucible: [0] Transition from New to WaitActive
612023-10-13T04:41:42.157ZINFOcrucible: [2] client is_active_req TRUE, promote! session 5a8631c8-4d7b-4b1b-bac7-a178bee40f34
622023-10-13T04:41:42.157ZINFOcrucible: [1] client is_active_req TRUE, promote! session 5a8631c8-4d7b-4b1b-bac7-a178bee40f34
632023-10-13T04:41:42.157ZINFOcrucible: [0] client is_active_req TRUE, promote! session 5a8631c8-4d7b-4b1b-bac7-a178bee40f34
642023-10-13T04:41:42.160ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
652023-10-13T04:41:42.160ZINFOcrucible: [0] downstairs client at 127.0.0.1:8810 has UUID 12345678-0000-0000-0000-000000008810
662023-10-13T04:41:42.160ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
672023-10-13T04:41:42.160ZINFOcrucible: [0] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }
682023-10-13T04:41:42.160ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
692023-10-13T04:41:42.160ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
702023-10-13T04:41:42.161ZINFOcrucible: f1b5b3ef-a58a-487e-ab3b-dc4227e30424 WaitActive WaitActive WaitActive
712023-10-13T04:41:42.161ZINFOcrucible: f1b5b3ef-a58a-487e-ab3b-dc4227e30424 WaitActive WaitActive WaitActive
722023-10-13T04:41:42.161ZINFOcrucible: f1b5b3ef-a58a-487e-ab3b-dc4227e30424 WaitActive WaitActive WaitActive
732023-10-13T04:41:42.163ZINFOcrucible: [2] f1b5b3ef-a58a-487e-ab3b-dc4227e30424 (5a8631c8-4d7b-4b1b-bac7-a178bee40f34) WaitActive WaitActive WaitActive ds_transition to WaitQuorum
742023-10-13T04:41:42.163ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum
752023-10-13T04:41:42.164ZWARNcrucible: [2] new RM replaced this: None
762023-10-13T04:41:42.164ZINFOcrucible: [2] Starts reconcile loop
772023-10-13T04:41:42.164ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
782023-10-13T04:41:42.164ZINFOcrucible: f1b5b3ef-a58a-487e-ab3b-dc4227e30424 WaitActive WaitActive WaitQuorum
792023-10-13T04:41:42.164ZINFOcrucible: Waiting for 2 more clients to be ready
802023-10-13T04:41:42.164ZINFOcrucible: Set check for repair
812023-10-13T04:41:42.164ZINFOcrucible: [0] f1b5b3ef-a58a-487e-ab3b-dc4227e30424 (5a8631c8-4d7b-4b1b-bac7-a178bee40f34) WaitActive WaitActive WaitQuorum ds_transition to WaitQuorum
822023-10-13T04:41:42.164ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum
832023-10-13T04:41:42.164ZWARNcrucible: [0] new RM replaced this: None
842023-10-13T04:41:42.164ZINFOcrucible: [0] Starts reconcile loop
852023-10-13T04:41:42.164ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:true
862023-10-13T04:41:42.164ZINFOcrucible: f1b5b3ef-a58a-487e-ab3b-dc4227e30424 WaitQuorum WaitActive WaitQuorum
872023-10-13T04:41:42.164ZINFOcrucible: Waiting for 1 more clients to be ready
882023-10-13T04:41:42.164ZINFOcrucible: Set check for repair
892023-10-13T04:41:42.164ZINFOcrucible: [1] f1b5b3ef-a58a-487e-ab3b-dc4227e30424 (5a8631c8-4d7b-4b1b-bac7-a178bee40f34) WaitQuorum WaitActive WaitQuorum ds_transition to WaitQuorum
902023-10-13T04:41:42.164ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum
912023-10-13T04:41:42.164ZWARNcrucible: [1] new RM replaced this: None
922023-10-13T04:41:42.164ZINFOcrucible: [1] Starts reconcile loop
932023-10-13T04:41:42.164ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
942023-10-13T04:41:42.164ZINFOcrucible: f1b5b3ef-a58a-487e-ab3b-dc4227e30424 WaitQuorum WaitQuorum WaitQuorum
952023-10-13T04:41:42.165ZINFOcrucible: [0]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
962023-10-13T04:41:42.165ZINFOcrucible: [0]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
972023-10-13T04:41:42.165ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]
982023-10-13T04:41:42.165ZINFOcrucible: [1]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
992023-10-13T04:41:42.165ZINFOcrucible: [1]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
1002023-10-13T04:41:42.165ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]
1012023-10-13T04:41:42.165ZINFOcrucible: [2]R flush_numbers[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
1022023-10-13T04:41:42.165ZINFOcrucible: [2]R generation[0..12]: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
1032023-10-13T04:41:42.166ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]
1042023-10-13T04:41:42.166ZINFOcrucible: Max found gen is 1
1052023-10-13T04:41:42.166ZINFOcrucible: Generation requested: 1 >= found:1
1062023-10-13T04:41:42.166ZINFOcrucible: Next flush: 1
1072023-10-13T04:41:42.166ZINFOcrucible: All extents match
1082023-10-13T04:41:42.166ZINFOcrucible: No downstairs repair required
109 {"msg":"No initial repair work was required","v":0,"name":"crucible","level":30The guest has finished waiting for activation with:1
110 ,"time"Activate successful
111 :"Wait for a query_work_queue command to finish before sending IO
112 2023-10-13T04:41:42.166911036Z","hostname":"ip-10-150-1-57.us-west-2.compute.internal","pid":1172}
1132023-10-13T04:41:42.166ZINFOcrucible: Set Downstairs and Upstairs active
1142023-10-13T04:41:42.166ZINFOcrucible: f1b5b3ef-a58a-487e-ab3b-dc4227e30424 is now active with session: 5a8631c8-4d7b-4b1b-bac7-a178bee40f34
1152023-10-13T04:41:42.167ZINFOcrucible: f1b5b3ef-a58a-487e-ab3b-dc4227e30424 Set Active after no repair
1162023-10-13T04:41:42.167ZINFOcrucible: Notify all downstairs, region set compare is done.
1172023-10-13T04:41:42.167ZINFOcrucible: Set check for repair
1182023-10-13T04:41:42.167ZINFOcrucible: [0] received reconcile message
1192023-10-13T04:41:42.167ZINFOcrucible: [1] received reconcile message
1202023-10-13T04:41:42.167ZINFOcrucible: [2] received reconcile message
1212023-10-13T04:41:42.168ZINFOcrucible: [0] All repairs completed, exit
1222023-10-13T04:41:42.168ZINFOcrucible: [0] Starts cmd_loop
1232023-10-13T04:41:42.168ZINFOcrucible: [1] All repairs completed, exit
1242023-10-13T04:41:42.168ZINFOcrucible: [1] Starts cmd_loop
125 Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
1262023-10-13T04:41:42.168ZINFOcrucible: [2] All repairs completed, exit
1272023-10-13T04:41:42.168ZINFOcrucible: [2] Starts cmd_loop
128 Fill test
129 Read and Verify all blocks (0..5000 range:false)
1302023-10-13T04:41:43.186ZINFOcrucible: Checking if live repair is needed
1312023-10-13T04:41:43.186ZINFOcrucible: No Live Repair required at this time
132 Wrote out file "alan"
133 CLIENT: Tests done. All submitted work has been ACK'd
134 ----------------------------------------------------------------
135 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:8
136 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
137 99 Acked 1098 Read 100 Done Done Done false
138 100 Acked 1099 Read 100 Done Done Done false
139 101 Acked 1100 Read 100 Done Done Done false
140 102 Acked 1101 Read 100 Done Done Done false
141 103 Acked 1102 Read 100 Done Done Done false
142 104 Acked 1103 Read 100 Done Done Done false
143 105 Acked 1104 Read 100 Done Done Done false
144 106 Acked 1105 Read 100 Done Done Done false
145 STATES DS:0 DS:1 DS:2 TOTAL
146 New 0 0 0 0
147 Sent 0 0 0 0
148 Done 8 8 8 24
149 Skipped 0 0 0 0
150 Error 0 0 0 0
151 Last Flush: 1097 1097 1097
152 Downstairs last five completed: 1097 1096 1095 1094 1093 1092
153 Upstairs last five completed: 106 105 104 103 102 101
154 CLIENT: Up:0 ds:8 act:3
155 ----------------------------------------------------------------
156 Crucible gen:1 GIO:true work queues: Upstairs:0 downstairs:0
157 Downstairs last five completed: 1106 1105 1104 1103 1102 1101
158 Upstairs last five completed: 107 106 105 104 103 102
159 CLIENT: Up:0 ds:0 act:3
160 CLIENT: All crucible jobs finished, exiting program
161 Replay loop starts now October 13, 2023 at 04:41:49 AM UTC
1622023-10-13T04:41:49.090ZINFOcrucible: Upstairs starts
1632023-10-13T04:41:49.090ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "0a6b3facda3a9a8c75c70b591db2d34571e435e8", git_commit_timestamp: "2023-10-13T04:02:07.000000000Z", git_branch: "alan/unwrapless", rustc_semver: "1.70.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "90c541806f23a127002de5b4038be731ba1458ca", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
1642023-10-13T04:41:49.090ZINFOcrucible: Upstairs <-> Downstairs Message Version: 4
1652023-10-13T04:41:49.090ZINFOcrucible: Upstairs opts: Upstairs UUID: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false
1662023-10-13T04:41:49.090ZINFOcrucible: Crucible stats registered with UUID: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b
1672023-10-13T04:41:49.090ZINFOcrucible: Crucible 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b has session id: c0bd9bf7-14e1-440b-847f-3b6d58f8a714
168 Crucible runtime is spawned
169 {"The guest has requested activation with gen:2
170 msg":"up_listen starts","v":0,"name":"crucible","level":30,"time":"2023-10-13T04:41:49.091107556Z","hostname":"ip-10-150-1-57.us-west-2.compute.internal","pid":1176,"task":"up_listen"}
1712023-10-13T04:41:49.091ZINFOcrucible: Wait for all three downstairs to come online
1722023-10-13T04:41:49.091ZINFOcrucible: Flush timeout: 0.5
1732023-10-13T04:41:49.091ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
1742023-10-13T04:41:49.091ZINFOcrucible: [0] connecting to 127.0.0.1:8810 looper = 0
1752023-10-13T04:41:49.091ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 2
1762023-10-13T04:41:49.091ZINFOcrucible: [1] connecting to 127.0.0.1:8820 looper = 1
1772023-10-13T04:41:49.091ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 0
1782023-10-13T04:41:49.091ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 1
1792023-10-13T04:41:49.092ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b active request set
1802023-10-13T04:41:49.092ZINFOcrucible: Set desired generation to :2
1812023-10-13T04:41:49.092ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state New
1822023-10-13T04:41:49.092ZINFOcrucible: [0] Proc runs for 127.0.0.1:8810 in state New
1832023-10-13T04:41:49.092ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state New
1842023-10-13T04:41:49.093ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) New New New ds_transition to WaitActive
1852023-10-13T04:41:49.093ZINFOcrucible: [2] Transition from New to WaitActive
1862023-10-13T04:41:49.093ZINFOcrucible: [2] client is_active_req TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
1872023-10-13T04:41:49.093ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) New New WaitActive ds_transition to WaitActive
1882023-10-13T04:41:49.093ZINFOcrucible: [0] Transition from New to WaitActive
1892023-10-13T04:41:49.093ZINFOcrucible: [0] client is_active_req TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
1902023-10-13T04:41:49.093ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) WaitActive New WaitActive ds_transition to WaitActive
1912023-10-13T04:41:49.093ZINFOcrucible: [1] Transition from New to WaitActive
1922023-10-13T04:41:49.093ZINFOcrucible: [1] client is_active_req TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
1932023-10-13T04:41:49.093ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
1942023-10-13T04:41:49.093ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
1952023-10-13T04:41:49.093ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
1962023-10-13T04:41:49.094ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
1972023-10-13T04:41:49.094ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b WaitActive WaitActive WaitActive
1982023-10-13T04:41:49.094ZINFOcrucible: [0] downstairs client at 127.0.0.1:8810 has UUID 12345678-0000-0000-0000-000000008810
1992023-10-13T04:41:49.094ZINFOcrucible: [0] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }
2002023-10-13T04:41:49.094ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b WaitActive WaitActive WaitActive
2012023-10-13T04:41:49.094ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b WaitActive WaitActive WaitActive
2022023-10-13T04:41:49.095ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) WaitActive WaitActive WaitActive ds_transition to WaitQuorum
2032023-10-13T04:41:49.095ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum
2042023-10-13T04:41:49.095ZWARNcrucible: [0] new RM replaced this: None
2052023-10-13T04:41:49.095ZINFOcrucible: [0] Starts reconcile loop
2062023-10-13T04:41:49.095ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:true
2072023-10-13T04:41:49.095ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b WaitQuorum WaitActive WaitActive
2082023-10-13T04:41:49.096ZINFOcrucible: Waiting for 2 more clients to be ready
2092023-10-13T04:41:49.096ZINFOcrucible: Set check for repair
2102023-10-13T04:41:49.096ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) WaitQuorum WaitActive WaitActive ds_transition to WaitQuorum
2112023-10-13T04:41:49.096ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum
2122023-10-13T04:41:49.096ZWARNcrucible: [1] new RM replaced this: None
2132023-10-13T04:41:49.096ZINFOcrucible: [1] Starts reconcile loop
2142023-10-13T04:41:49.096ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
2152023-10-13T04:41:49.096ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) WaitQuorum WaitQuorum WaitActive ds_transition to WaitQuorum
2162023-10-13T04:41:49.096ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum
2172023-10-13T04:41:49.096ZWARNcrucible: [2] new RM replaced this: None
2182023-10-13T04:41:49.096ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b WaitQuorum WaitQuorum WaitQuorum
2192023-10-13T04:41:49.096ZINFOcrucible: [2] Starts reconcile loop
2202023-10-13T04:41:49.096ZINFOcrucible: [0]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
2212023-10-13T04:41:49.096ZINFOcrucible: [0]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
2222023-10-13T04:41:49.096ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]
2232023-10-13T04:41:49.096ZINFOcrucible: [1]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
2242023-10-13T04:41:49.096ZINFOcrucible: [1]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
225 {"msg":"[1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]","v":0,"name":"crucible","level":30The guest has finished waiting for activation with:2
226 Activate successful
227 Wait for a query_work_queue command to finish before sending IO
228 ,"time":"2023-10-13T04:41:49.096855978Z","hostname":"ip-10-150-1-57.us-west-2.compute.internal","pid":1176}
2292023-10-13T04:41:49.096ZINFOcrucible: [2]R flush_numbers[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
2302023-10-13T04:41:49.096ZINFOcrucible: [2]R generation[0..12]: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
2312023-10-13T04:41:49.096ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]
2322023-10-13T04:41:49.097ZINFOcrucible: Max found gen is 2
2332023-10-13T04:41:49.097ZINFOcrucible: Generation requested: 2 >= found:2
2342023-10-13T04:41:49.097ZINFOcrucible: Next flush: 3
2352023-10-13T04:41:49.097ZINFOcrucible: All extents match
2362023-10-13T04:41:49.097ZINFOcrucible: No downstairs repair required
2372023-10-13T04:41:49.097ZINFOcrucible: No initial repair work was required
2382023-10-13T04:41:49.097ZINFOcrucible: Set Downstairs and Upstairs active
2392023-10-13T04:41:49.097ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b is now active with session: 373d0eb1-b2be-4ec1-8e62-d92421d49954
2402023-10-13T04:41:49.097ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Set Active after no repair
2412023-10-13T04:41:49.097ZINFOcrucible: Notify all downstairs, region set compare is done.
2422023-10-13T04:41:49.097ZINFOcrucible: Set check for repair
2432023-10-13T04:41:49.097ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
2442023-10-13T04:41:49.097ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
2452023-10-13T04:41:49.097ZINFOcrucible: Set check for repair
2462023-10-13T04:41:49.097ZINFOcrucible: [1] received reconcile message
2472023-10-13T04:41:49.097ZINFOcrucible: [1] All repairs completed, exit
2482023-10-13T04:41:49.097ZINFOcrucible: [1] Starts cmd_loop
2492023-10-13T04:41:49.097ZINFOcrucible: [2] received reconcile message
2502023-10-13T04:41:49.097ZINFOcrucible: [2] All repairs completed, exit
2512023-10-13T04:41:49.097ZINFOcrucible: [2] Starts cmd_loop
2522023-10-13T04:41:49.097ZINFOcrucible: [0] received reconcile message
2532023-10-13T04:41:49.097ZINFOcrucible: [0] All repairs completed, exit
2542023-10-13T04:41:49.097ZINFOcrucible: [0] Starts cmd_loop
255 Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
256 Loading write count information from file "alan"
257 Oct 13 04:41:49.191 INFO accepted connection, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
258 Main task has work to do, go find it
259 got dsc Stop(1)
260 stop 1
261 Oct 13 04:41:49.196 INFO request completed, latency_us: 1425, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 7d3dd41e-58bb-4be4-ae45-fe496110bbd3, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
262 [8820] Got stop action so:false kr:false
2632023-10-13T04:41:49.198ZWARNcrucible: [1] downstairs disconnected
2642023-10-13T04:41:49.198ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
2652023-10-13T04:41:49.198ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8820 closed looper = 1
2662023-10-13T04:41:49.198ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:false
2672023-10-13T04:41:49.198ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
2682023-10-13T04:41:49.198ZINFOcrucible: [1] 127.0.0.1:8820 task reports offline
2692023-10-13T04:41:49.198ZWARNcrucible: [1] pm_task rx.recv() is None
270 Oct 13 04:41:49.199 INFO request completed, latency_us: 715, response_code: 200, uri: /state/cid/1, method: GET, req_id: 37e5da25-f681-42ed-893e-1046d1201494, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
271 [8820] Exited with: ExitStatus(unix_wait_status(9))
272 [8820][1] reports Exit
2732023-10-13T04:41:50.097ZINFOcrucible: Checking if live repair is needed
2742023-10-13T04:41:50.098ZINFOcrucible: No Live Repair required at this time
275 Oct 13 04:41:53.203 INFO request completed, latency_us: 265, response_code: 200, uri: /state/cid/1, method: GET, req_id: 2fc41391-99b7-4e1f-ac40-d3f55de23bef, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
276 001/300 Write block 3099 len 28672 data: 2 2 2 2 2 2 2
277 002/300 Write block 3709 len 28672 data: 2 2 2 2 2 2 2
278 003/300 Write block 728 len 12288 data: 2 2 2
279 004/300 Write block 4698 len 16384 data: 2 2 2 2
280 005/300 Write block 2515 len 32768 data: 2 2 2 2 2 2 2 2
281 006/300 Read block 39 len 8192
282 007/300 Read block 586 len 24576
283 008/300 Write block 3735 len 8192 data: 2 2
284 009/300 Read block 3015 len 16384
285 010/300 Read block 2112 len 8192
286 011/300 Read block 75 len 24576
287 012/300 Read block 1701 len 24576
288 013/300 Read block 543 len 40960
289 014/300 Write block 4680 len 36864 data: 2 2 2 2 2 2 2 2 2
290 015/300 Write block 77 len 24576 data: 2 2 2 2 2 2
291 016/300 Write block 1487 len 32768 data: 2 2 2 2 2 2 2 2
292 017/300 Write block 621 len 16384 data: 2 2 2 2
293 018/300 Read block 393 len 28672
294 019/300 Read block 3446 len 36864
295 020/300 Write block 990 len 16384 data: 2 2 2 2
296 021/300 Read block 1446 len 24576
297 022/300 Write block 544 len 36864 data: 2 2 2 2 2 2 2 2 2
298 023/300 Read block 962 len 28672
299 024/300 Write block 486 len 20480 data: 2 2 2 2 2
300 025/300 Read block 2127 len 8192
301 026/300 Read block 1749 len 4096
302 027/300 Read block 4485 len 36864
303 028/300 Read block 2351 len 36864
304 029/300 Read block 4335 len 20480
305 030/300 Read block 1085 len 40960
306 031/300 Flush
307 032/300 Write block 1992 len 40960 data: 2 2 2 2 2 2 2 2 2 2
308 033/300 Read block 3749 len 40960
309 034/300 Read block 4121 len 28672
310 035/300 Flush
311 036/300 Write block 4449 len 24576 data: 2 2 2 2 2 2
312 037/300 Flush
313 038/300 Read block 233 len 36864
314 039/300 Write block 4557 len 20480 data: 2 2 2 2 2
315 040/300 Write block 3135 len 24576 data: 2 2 2 2 2 2
316 041/300 Read block 2223 len 40960
317 042/300 Write block 2887 len 36864 data: 2 2 2 2 2 2 2 2 2
318 043/300 Read block 649 len 28672
319 044/300 Read block 2321 len 8192
320 045/300 Write block 2846 len 32768 data: 2 2 2 2 2 2 2 2
321 046/300 Read block 2737 len 12288
322 047/300 Write block 1550 len 24576 data: 2 2 2 2 2 2
323 048/300 Read block 2451 len 36864
324 049/300 Write block 1031 len 12288 data: 2 2 2
325 050/300 Write block 1288 len 40960 data: 2 2 2 2 2 2 2 2 2 2
326 051/300 Flush
327 052/300 Read block 801 len 40960
328 053/300 Write block 885 len 4096 data: 2
329 054/300 Write block 2813 len 24576 data: 2 2 2 2 2 2
330 055/300 Write block 3777 len 28672 data: 2 2 2 2 2 2 2
331 056/300 Read block 2876 len 28672
332 057/300 Flush
333 058/300 Write block 1824 len 28672 data: 2 2 2 2 2 2 2
334 059/300 Write block 981 len 32768 data: 2 2 2 2 2 2 2 2
335 060/300 Write block 1066 len 20480 data: 2 2 2 2 2
336 061/300 Write block 627 len 8192 data: 2 2
337 062/300 Write block 16 len 24576 data: 2 2 2 2 2 2
338 063/300 Flush
339 064/300 Flush
340 065/300 Read block 2694 len 24576
341 066/300 Read block 454 len 40960
342 067/300 Write block 1709 len 20480 data: 2 2 2 2 2
343 068/300 Read block 1304 len 36864
344 069/300 Read block 4040 len 4096
345 070/300 Read block 1420 len 32768
346 071/300 Read block 1592 len 40960
347 072/300 Write block 619 len 20480 data: 2 2 3 3 3
348 073/300 Write block 4454 len 36864 data: 3 2 2 2 2 2 2 2 2
349 074/300 Read block 2028 len 16384
350 075/300 Read block 3936 len 28672
351 076/300 Write block 1858 len 16384 data: 2 2 2 2
352 077/300 Write block 143 len 12288 data: 2 2 2
353 078/300 Flush
354 079/300 Write block 4940 len 40960 data: 2 2 2 2 2 2 2 2 2 2
355 080/300 Read block 1928 len 40960
356 081/300 Read block 2044 len 32768
357 082/300 Write block 3433 len 12288 data: 2 2 2
358 083/300 Read block 4709 len 36864
359 084/300 Read block 2092 len 40960
360 085/300 Write block 3256 len 28672 data: 2 2 2 2 2 2 2
361 086/300 Write block 1305 len 8192 data: 2 2
362 087/300 Write block 4260 len 28672 data: 2 2 2 2 2 2 2
363 088/300 Read block 2699 len 32768
364 089/300 Write block 4663 len 4096 data: 2
365 090/300 Write block 3552 len 20480 data: 2 2 2 2 2
366 091/300 Flush
367 092/300 Write block 3360 len 36864 data: 2 2 2 2 2 2 2 2 2
368 093/300 Read block 993 len 16384
369 094/300 Write block 2369 len 40960 data: 2 2 2 2 2 2 2 2 2 2
370 095/300 Write block 2783 len 32768 data: 2 2 2 2 2 2 2 2
371 096/300 Read block 3402 len 36864
372 097/300 Read block 98 len 32768
373 098/300 Write block 4004 len 16384 data: 2 2 2 2
374 099/300 Read block 928 len 12288
375 100/300 Read block 469 len 40960
376 101/300 Read block 949 len 40960
377 102/300 Read block 4746 len 24576
378 103/300 Write block 3405 len 36864 data: 2 2 2 2 2 2 2 2 2
379 104/300 Read block 2096 len 8192
380 105/300 Write block 214 len 24576 data: 2 2 2 2 2 2
381 106/300 Read block 1764 len 32768
382 107/300 Read block 573 len 36864
383 108/300 Read block 4991 len 12288
384 109/300 Write block 3492 len 24576 data: 2 2 2 2 2 2
385 110/300 Write block 171 len 32768 data: 2 2 2 2 2 2 2 2
386 111/300 Read block 1536 len 8192
387 112/300 Flush
388 113/300 Read block 272 len 8192
389 114/300 Flush
390 115/300 Read block 4869 len 8192
391 116/300 Read block 2582 len 4096
392 117/300 Flush
393 118/300 Write block 2538 len 36864 data: 2 2 2 2 2 2 2 2 2
394 119/300 Write block 1085 len 24576 data: 2 2 2 2 2 2
395 120/300 Write block 2523 len 12288 data: 2 2 2
396 121/300 Write block 4106 len 28672 data: 2 2 2 2 2 2 2
397 122/300 Read block 3147 len 40960
398 123/300 Write block 3127 len 32768 data: 2 2 2 2 2 2 2 2
399 124/300 Read block 3952 len 40960
400 125/300 Read block 3419 len 8192
401 126/300 Write block 1012 len 12288 data: 2 2 2
402 127/300 Write block 4278 len 28672 data: 2 2 2 2 2 2 2
403 128/300 Read block 2196 len 32768
404 129/300 Read block 600 len 8192
405 130/300 Read block 2790 len 28672
406 131/300 Write block 4081 len 32768 data: 2 2 2 2 2 2 2 2
407 132/300 Read block 4876 len 28672
408 133/300 Read block 3170 len 40960
409 134/300 Write block 2540 len 8192 data: 3 3
410 135/300 Write block 88 len 20480 data: 2 2 2 2 2
411 136/300 Write block 4602 len 24576 data: 2 2 2 2 2 2
412 137/300 Read block 2962 len 4096
413 138/300 Read block 3281 len 20480
414 139/300 Read block 137 len 4096
415 140/300 Write block 1284 len 40960 data: 2 2 2 2 3 3 3 3 3 3
416 141/300 Read block 4880 len 12288
417 142/300 Read block 2643 len 40960
418 143/300 Flush
419 144/300 Read block 4260 len 20480
420 145/300 Read block 514 len 12288
421 146/300 Read block 668 len 12288
422 147/300 Write block 1262 len 20480 data: 2 2 2 2 2
423 148/300 Write block 3854 len 4096 data: 2
424 149/300 Read block 3521 len 32768
425 150/300 Write block 436 len 36864 data: 2 2 2 2 2 2 2 2 2
426 151/300 Read block 3074 len 40960
427 152/300 Read block 3323 len 8192
428 153/300 Flush
429 154/300 Read block 923 len 32768
430 155/300 Write block 4732 len 16384 data: 2 2 2 2
431 156/300 Read block 1403 len 8192
432 157/300 Read block 19 len 24576
433 158/300 Flush
434 159/300 Read block 1029 len 12288
435 160/300 Read block 4156 len 16384
436 161/300 Flush
437 162/300 Read block 763 len 4096
438 163/300 Read block 3646 len 12288
439 164/300 Write block 523 len 28672 data: 2 2 2 2 2 2 2
440 165/300 Write block 4858 len 28672 data: 2 2 2 2 2 2 2
441 166/300 Read block 961 len 20480
442 167/300 Read block 2306 len 28672
443 168/300 Read block 2924 len 24576
444 169/300 Write block 3840 len 16384 data: 2 2 2 2
445 170/300 Read block 1977 len 20480
446 171/300 Flush
447 172/300 Read block 1078 len 32768
448 173/300 Write block 4886 len 12288 data: 2 2 2
449 174/300 Write block 2926 len 20480 data: 2 2 2 2 2
450 175/300 Read block 4222 len 16384
451 176/300 Read block 2034 len 36864
452 177/300 Write block 389 len 36864 data: 2 2 2 2 2 2 2 2 2
453 178/300 Read block 4954 len 40960
454 179/300 Write block 864 len 4096 data: 2
455 180/300 Write block 1005 len 8192 data: 2 2
456 181/300 Read block 2767 len 16384
457 182/300 Read block 1846 len 4096
458 183/300 Read block 3077 len 40960
459 184/300 Write block 2096 len 24576 data: 2 2 2 2 2 2
460 185/300 Write block 4838 len 4096 data: 2
461 186/300 Flush
462 187/300 Read block 1632 len 32768
463 188/300 Read block 3503 len 32768
464 189/300 Read block 597 len 12288
465 190/300 Flush
466 191/300 Write block 3396 len 16384 data: 2 2 2 2
467 192/300 Write block 2507 len 36864 data: 2 2 2 2 2 2 2 2 3
468 193/300 Read block 1418 len 20480
469 194/300 Write block 4827 len 24576 data: 2 2 2 2 2 2
470 195/300 Write block 3501 len 32768 data: 2 2 2 2 2 2 2 2
471 196/300 Read block 3009 len 20480
472 197/300 Read block 2991 len 40960
473 198/300 Write block 3477 len 20480 data: 2 2 2 2 2
474 199/300 Write block 2691 len 40960 data: 2 2 2 2 2 2 2 2 2 2
475 200/300 Write block 3425 len 20480 data: 2 2 2 2 2
476 201/300 Write block 253 len 32768 data: 2 2 2 2 2 2 2 2
477 202/300 Read block 3154 len 24576
478 203/300 Write block 3525 len 40960 data: 2 2 2 2 2 2 2 2 2 2
479 204/300 Write block 3025 len 24576 data: 2 2 2 2 2 2
480 205/300 Read block 884 len 8192
481 206/300 Write block 1679 len 32768 data: 2 2 2 2 2 2 2 2
482 207/300 Flush
483 208/300 Write block 1537 len 36864 data: 2 2 2 2 2 2 2 2 2
484 209/300 Read block 3687 len 28672
485 210/300 Write block 1785 len 36864 data: 2 2 2 2 2 2 2 2 2
486 211/300 Read block 3558 len 28672
487 212/300 Read block 468 len 24576
488 213/300 Read block 3328 len 32768
489 214/300 Read block 4749 len 24576
490 215/300 Read block 2294 len 12288
491 216/300 Write block 3341 len 8192 data: 2 2
492 217/300 Read block 1119 len 12288
493 218/300 Read block 412 len 40960
494 219/300 Read block 87 len 24576
495 220/300 Read block 2462 len 20480
496 221/300 Write block 594 len 28672 data: 2 2 2 2 2 2 2
497 222/300 Flush
498 223/300 Write block 2873 len 28672 data: 2 2 2 2 2 2 2
499 224/300 Read block 3157 len 24576
500 225/300 Write block 303 len 28672 data: 2 2 2 2 2 2 2
501 226/300 Write block 1209 len 36864 data: 2 2 2 2 2 2 2 2 2
502 227/300 Write block 1555 len 24576 data: 3 2 2 2 2 2
503 228/300 Write block 332 len 24576 data: 2 2 2 2 2 2
504 229/300 Write block 2384 len 20480 data: 2 2 2 2 2
505 230/300 Write block 3582 len 8192 data: 2 2
506 231/300 Write block 1422 len 32768 data: 2 2 2 2 2 2 2 2
507 232/300 Read block 3140 len 8192
508 233/300 Read block 3634 len 20480
509 234/300 Write block 4913 len 28672 data: 2 2 2 2 2 2 2
510 235/300 Write block 329 len 32768 data: 2 2 2 3 3 3 3 3
511 236/300 Write block 2588 len 8192 data: 2 2
512 237/300 Read block 2936 len 4096
513 238/300 Flush
514 239/300 Write block 587 len 4096 data: 2
515 240/300 Write block 2611 len 40960 data: 2 2 2 2 2 2 2 2 2 2
516 241/300 Read block 3453 len 20480
517 242/300 Write block 3558 len 24576 data: 2 2 2 2 2 2
518 243/300 Write block 851 len 16384 data: 2 2 2 2
519 244/300 Read block 2584 len 8192
520 245/300 Read block 3176 len 16384
521 246/300 Write block 3315 len 40960 data: 2 2 2 2 2 2 2 2 2 2
522 247/300 Flush
523 248/300 Read block 2689 len 16384
524 249/300 Read block 1663 len 8192
525 250/300 Write block 1436 len 40960 data: 2 2 2 2 2 2 2 2 2 2
526 251/300 Read block 1826 len 32768
527 252/300 Write block 4167 len 28672 data: 2 2 2 2 2 2 2
528 253/300 Read block 2859 len 36864
529 254/300 Write block 1150 len 4096 data: 2
530 255/300 Write block 3524 len 40960 data: 2 3 3 3 3 3 3 3 3 3
531 256/300 Read block 3637 len 4096
532 257/300 Write block 4135 len 4096 data: 2
533 258/300 Write block 1341 len 32768 data: 2 2 2 2 2 2 2 2
534 259/300 Read block 1135 len 24576
535 260/300 Write block 1530 len 28672 data: 2 2 2 2 2 2 2
536 261/300 Read block 3913 len 12288
537 262/300 Read block 1488 len 20480
538 263/300 Write block 2469 len 16384 data: 2 2 2 2
539 264/300 Read block 3520 len 4096
540 265/300 Read block 4141 len 36864
541 266/300 Write block 1094 len 36864 data: 2 2 2 2 2 2 2 2 2
542 267/300 Write block 1151 len 32768 data: 2 2 2 2 2 2 2 2
543 268/300 Read block 2785 len 40960
544 269/300 Write block 4000 len 12288 data: 2 2 2
545 270/300 Read block 416 len 40960
546 271/300 Write block 3775 len 40960 data: 2 2 3 3 3 3 3 3 3 2
547 272/300 Read block 261 len 20480
548 273/300 Read block 1653 len 8192
549 274/300 Write block 3459 len 8192 data: 2 2
550 275/300 Write block 4535 len 36864 data: 2 2 2 2 2 2 2 2 2
551 276/300 Read block 647 len 40960
552 277/300 Read block 647 len 8192
553 278/300 Read block 2373 len 4096
554 279/300 Write block 752 len 36864 data: 2 2 2 2 2 2 2 2 2
555 280/300 Read block 1908 len 20480
556 281/300 Flush
557 282/300 Write block 2643 len 32768 data: 2 2 2 2 2 2 2 2
558 283/300 Read block 2602 len 20480
559 284/300 Read block 1021 len 20480
560 285/300 Read block 1771 len 24576
561 286/300 Read block 3193 len 28672
562 287/300 Write block 320 len 36864 data: 2 2 2 2 2 2 2 2 2
563 288/300 Write block 1415 len 20480 data: 2 2 2 2 2
564 289/300 Read block 1714 len 4096
565 290/300 Read block 2642 len 40960
566 291/300 Read block 2586 len 20480
567 292/300 Read block 3053 len 20480
568 293/300 Write block 4002 len 12288 data: 3 2 3
569 294/300 Write block 3117 len 40960 data: 2 2 2 2 2 2 2 2 2 2
570 295/300 Read block 1825 len 28672
571 296/300 Write block 2805 len 8192 data: 2 2
572 297/300 Read block 861 len 12288
573 298/300 Write block 2690 len 24576 data: 2 3 3 3 3 3
574 299/300 Write block 662 len 4096 data: 2
575 300/300 Read block 2407 len 16384
576 Main task has work to do, go find it
577 got dsc Start(1)
578 start 1
579 Oct 13 04:41:54.123 INFO request completed, latency_us: 314, response_code: 204, uri: /start/cid/1, method: POST, req_id: c73c0d9e-80bc-49b2-a454-00437f60a37b, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
580 Replay: started 1, returned:Ok(())
581 ----------------------------------------------------------------
582 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
583 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
584 1 Acked 1000 Write 7 Done New Done false
585 2 Acked 1001 Write 7 Done New Done false
586 3 Acked 1002 Write 3 Done New Done false
587 4 Acked 1003 Write 4 Done New Done false
588 5 Acked 1004 Write 8 Done New Done false
589 6 Acked 1005 Read 2 Done New Done false
590 7 Acked 1006 Read 6 Done New Done false
591 8 Acked 1007 Write 2 Done New Done false
592 9 Acked 1008 Read 4 Done New Done false
593 10 Acked 1009 Read 2 Done New Done false
594 11 Acked 1010 Read 6 Done New Done false
595 12 Acked 1011 Read 6 Done New Done false
596 13 Acked 1012 Read 10 Done New Done false
597 14 Acked 1013 Write 9 Done New Done false
598 15 Acked 1014 Write 6 Done New Done false
599 16 Acked 1015 Write 8 Done New Done false
600 17 Acked 1016 Write 4 Done New Done false
601 18 Acked 1017 Read 7 Done New Done false
602 19 Acked 1018 Read 9 Done New Done false
603 20 Acked 1019 Write 4 Done New Done false
604 21 Acked 1020 Read 6 Done New Done false
605 22 Acked 1021 Write 9 Done New Done false
606 23 Acked 1022 Read 7 Done New Done false
607 24 Acked 1023 Write 5 Done New Done false
608 25 Acked 1024 Read 2 Done New Done false
609 26 Acked 1025 Read 1 Done New Done false
610 27 Acked 1026 Read 9 Done New Done false
611 28 Acked 1027 Read 9 Done New Done false
612 29 Acked 1028 Read 5 Done New Done false
613 30 Acked 1029 Read 10 Done New Done false
614 31 Acked 1030 Flush 0 Done New Done false
615 32 Acked 1031 Write 10 Done New Done false
616 33 Acked 1032 Read 10 Done New Done false
617 34 Acked 1033 Read 7 Done New Done false
618 35 Acked 1034 Flush 0 Done New Done false
619 36 Acked 1035 Write 6 Done New Done false
620 37 Acked 1036 Flush 0 Done New Done false
621 38 Acked 1037 Read 9 Done New Done false
622 39 Acked 1038 Write 5 Done New Done false
623 40 Acked 1039 Write 6 Done New Done false
624 41 Acked 1040 Read 10 Done New Done false
625 42 Acked 1041 Write 9 Done New Done false
626 43 Acked 1042 Read 7 Done New Done false
627 44 Acked 1043 Read 2 Done New Done false
628 45 Acked 1044 Write 8 Done New Done false
629 46 Acked 1045 Read 3 Done New Done false
630 47 Acked 1046 Write 6 Done New Done false
631 48 Acked 1047 Read 9 Done New Done false
632 49 Acked 1048 Write 3 Done New Done false
633 50 Acked 1049 Write 10 Done New Done false
634 51 Acked 1050 Flush 0 Done New Done false
635 52 Acked 1051 Read 10 Done New Done false
636 53 Acked 1052 Write 1 Done New Done false
637 54 Acked 1053 Write 6 Done New Done false
638 55 Acked 1054 Write 7 Done New Done false
639 56 Acked 1055 Read 7 Done New Done false
640 57 Acked 1056 Flush 0 Done New Done false
641 58 Acked 1057 Write 7 Done New Done false
642 59 Acked 1058 Write 8 Done New Done false
643 60 Acked 1059 Write 5 Done New Done false
644 61 Acked 1060 Write 2 Done New Done false
645 62 Acked 1061 Write 6 Done New Done false
646 63 Acked 1062 Flush 0 Done New Done false
647 64 Acked 1063 Flush 0 Done New Done false
648 65 Acked 1064 Read 6 Done New Done false
649 66 Acked 1065 Read 10 Done New Done false
650 67 Acked 1066 Write 5 Done New Done false
651 68 Acked 1067 Read 9 Done New Done false
652 69 Acked 1068 Read 1 Done New Done false
653 70 Acked 1069 Read 8 Done New Done false
654 71 Acked 1070 Read 10 Done New Done false
655 72 Acked 1071 Write 5 Done New Done false
656 73 Acked 1072 Write 9 Done New Done false
657 74 Acked 1073 Read 4 Done New Done false
658 75 Acked 1074 Read 7 Done New Done false
659 76 Acked 1075 Write 4 Done New Done false
660 77 Acked 1076 Write 3 Done New Done false
661 78 Acked 1077 Flush 0 Done New Done false
662 79 Acked 1078 Write 10 Done New Done false
663 80 Acked 1079 Read 10 Done New Done false
664 81 Acked 1080 Read 8 Done New Done false
665 82 Acked 1081 Write 3 Done New Done false
666 83 Acked 1082 Read 9 Done New Done false
667 84 Acked 1083 Read 10 Done New Done false
668 85 Acked 1084 Write 7 Done New Done false
669 86 Acked 1085 Write 2 Done New Done false
670 87 Acked 1086 Write 7 Done New Done false
671 88 Acked 1087 Read 8 Done New Done false
672 89 Acked 1088 Write 1 Done New Done false
673 90 Acked 1089 Write 5 Done New Done false
674 91 Acked 1090 Flush 0 Done New Done false
675 92 Acked 1091 Write 9 Done New Done false
676 93 Acked 1092 Read 4 Done New Done false
677 94 Acked 1093 Write 10 Done New Done false
678 95 Acked 1094 Write 8 Done New Done false
679 96 Acked 1095 Read 9 Done New Done false
680 97 Acked 1096 Read 8 Done New Done false
681 98 Acked 1097 Write 4 Done New Done false
682 99 Acked 1098 Read 3 Done New Done false
683 100 Acked 1099 Read 10 Done New Done false
684 101 Acked 1100 Read 10 Done New Done false
685 102 Acked 1101 Read 6 Done New Done false
686 103 Acked 1102 Write 9 Done New Done false
687 104 Acked 1103 Read 2 Done New Done false
688 105 Acked 1104 Write 6 Done New Done false
689 106 Acked 1105 Read 8 Done New Done false
690 107 Acked 1106 Read 9 Done New Done false
691 108 Acked 1107 Read 3 Done New Done false
692 109 Acked 1108 Write 6 Done New Done false
693 110 Acked 1109 Write 8 Done New Done false
694 111 Acked 1110 Read 2 Done New Done false
695 112 Acked 1111 Flush 0 Done New Done false
696 113 Acked 1112 Read 2 Done New Done false
697 114 Acked 1113 Flush 0 Done New Done false
698 115 Acked 1114 Read 2 Done New Done false
699 116 Acked 1115 Read 1 Done New Done false
700 117 Acked 1116 Flush 0 Done New Done false
701 118 Acked 1117 Write 9 Done New Done false
702 119 Acked 1118 Write 6 Done New Done false
703 120 Acked 1119 Write 3 Done New Done false
704 121 Acked 1120 Write 7 Done New Done false
705 122 Acked 1121 Read 10 Done New Done false
706 123 Acked 1122 Write 8 Done New Done false
707 124 Acked 1123 Read 10 Done New Done false
708 125 Acked 1124 Read 2 Done New Done false
709 126 Acked 1125 Write 3 Done New Done false
710 127 Acked 1126 Write 7 Done New Done false
711 128 Acked 1127 Read 8 Done New Done false
712 129 Acked 1128 Flush 0 Done New Done false
713 130 Acked 1129 Read 2 Done New Done false
714 131 Acked 1130 Read 7 Done New Done false
715 132 Acked 1131 Write 8 Done New Done false
716 133 Acked 1132 Read 7 Done New Done false
717 134 Acked 1133 Read 10 Done New Done false
718 135 Acked 1134 Write 2 Done New Done false
719 136 Acked 1135 Write 5 Done New Done false
720 137 Acked 1136 Write 6 Done New Done false
721 138 Acked 1137 Read 1 Done New Done false
722 139 Acked 1138 Read 5 Done New Done false
723 140 Acked 1139 Read 1 Done New Done false
724 141 Acked 1140 Write 10 Done New Done false
725 142 Acked 1141 Read 3 Done New Done false
726 143 Acked 1142 Read 10 Done New Done false
727 144 Acked 1143 Flush 0 Done New Done false
728 145 Acked 1144 Read 5 Done New Done false
729 146 Acked 1145 Read 3 Done New Done false
730 147 Acked 1146 Read 3 Done New Done false
731 148 Acked 1147 Write 5 Done New Done false
732 149 Acked 1148 Write 1 Done New Done false
733 150 Acked 1149 Read 8 Done New Done false
734 151 Acked 1150 Write 9 Done New Done false
735 152 Acked 1151 Read 10 Done New Done false
736 153 Acked 1152 Read 2 Done New Done false
737 154 Acked 1153 Flush 0 Done New Done false
738 155 Acked 1154 Read 8 Done New Done false
739 156 Acked 1155 Write 4 Done New Done false
740 157 Acked 1156 Read 2 Done New Done false
741 158 Acked 1157 Read 6 Done New Done false
742 159 Acked 1158 Flush 0 Done New Done false
743 160 Acked 1159 Read 3 Done New Done false
744 161 Acked 1160 Read 4 Done New Done false
745 162 Acked 1161 Flush 0 Done New Done false
746 163 Acked 1162 Read 1 Done New Done false
747 164 Acked 1163 Read 3 Done New Done false
748 165 Acked 1164 Write 7 Done New Done false
749 166 Acked 1165 Write 7 Done New Done false
750 167 Acked 1166 Read 5 Done New Done false
751 168 Acked 1167 Read 7 Done New Done false
752 169 Acked 1168 Read 6 Done New Done false
753 170 Acked 1169 Write 4 Done New Done false
754 171 Acked 1170 Read 5 Done New Done false
755 172 Acked 1171 Flush 0 Done New Done false
756 173 Acked 1172 Read 8 Done New Done false
757 174 Acked 1173 Write 3 Done New Done false
758 175 Acked 1174 Write 5 Done New Done false
759 176 Acked 1175 Read 4 Done New Done false
760 177 Acked 1176 Read 9 Done New Done false
761 178 Acked 1177 Write 9 Done New Done false
762 179 Acked 1178 Read 10 Done New Done false
763 180 Acked 1179 Write 1 Done New Done false
764 181 Acked 1180 Write 2 Done New Done false
765 182 Acked 1181 Read 4 Done New Done false
766 183 Acked 1182 Read 1 Done New Done false
767 184 Acked 1183 Read 10 Done New Done false
768 185 Acked 1184 Write 6 Done New Done false
769 186 Acked 1185 Write 1 Done New Done false
770 187 Acked 1186 Flush 0 Done New Done false
771 188 Acked 1187 Read 8 Done New Done false
772 189 Acked 1188 Read 8 Done New Done false
773 190 Acked 1189 Read 3 Done New Done false
774 191 Acked 1190 Flush 0 Done New Done false
775 192 Acked 1191 Write 4 Done New Done false
776 193 Acked 1192 Write 9 Done New Done false
777 194 Acked 1193 Read 5 Done New Done false
778 195 Acked 1194 Write 6 Done New Done false
779 196 Acked 1195 Write 8 Done New Done false
780 197 Acked 1196 Read 5 Done New Done false
781 198 Acked 1197 Read 10 Done New Done false
782 199 Acked 1198 Write 5 Done New Done false
783 200 Acked 1199 Write 10 Done New Done false
784 201 Acked 1200 Write 5 Done New Done false
785 202 Acked 1201 Write 8 Done New Done false
786 203 Acked 1202 Read 6 Done New Done false
787 204 Acked 1203 Write 10 Done New Done false
788 205 Acked 1204 Write 6 Done New Done false
789 206 Acked 1205 Read 2 Done New Done false
790 207 Acked 1206 Write 8 Done New Done false
791 208 Acked 1207 Flush 0 Done New Done false
792 209 Acked 1208 Write 9 Done New Done false
793 210 Acked 1209 Read 7 Done New Done false
794 211 Acked 1210 Write 9 Done New Done false
795 212 Acked 1211 Read 7 Done New Done false
796 213 Acked 1212 Read 6 Done New Done false
797 214 Acked 1213 Read 8 Done New Done false
798 215 Acked 1214 Read 6 Done New Done false
799 216 Acked 1215 Read 3 Done New Done false
800 217 Acked 1216 Write 2 Done New Done false
801 218 Acked 1217 Read 3 Done New Done false
802 219 Acked 1218 Read 10 Done New Done false
803 220 Acked 1219 Read 6 Done New Done false
804 221 Acked 1220 Read 5 Done New Done false
805 222 Acked 1221 Write 7 Done New Done false
806 223 Acked 1222 Flush 0 Done New Done false
807 224 Acked 1223 Write 7 Done New Done false
808 225 Acked 1224 Read 6 Done New Done false
809 226 Acked 1225 Write 7 Done New Done false
810 227 Acked 1226 Write 9 Done New Done false
811 228 Acked 1227 Write 6 Done New Done false
812 229 Acked 1228 Write 6 Done New Done false
813 230 Acked 1229 Write 5 Done New Done false
814 231 Acked 1230 Write 2 Done New Done false
815 232 Acked 1231 Write 8 Done New Done false
816 233 Acked 1232 Read 2 Done New Done false
817 234 Acked 1233 Read 5 Done New Done false
818 235 Acked 1234 Write 7 Done New Done false
819 236 Acked 1235 Write 8 Done New Done false
820 237 Acked 1236 Write 2 Done New Done false
821 238 Acked 1237 Read 1 Done New Done false
822 239 Acked 1238 Flush 0 Done New Done false
823 240 Acked 1239 Write 1 Done New Done false
824 241 Acked 1240 Write 10 Done New Done false
825 242 Acked 1241 Read 5 Done New Done false
826 243 Acked 1242 Write 6 Done New Done false
827 244 Acked 1243 Write 4 Done New Done false
828 245 Acked 1244 Read 2 Done New Done false
829 246 Acked 1245 Read 4 Done New Done false
830 247 Acked 1246 Write 10 Done New Done false
831 248 Acked 1247 Flush 0 Done New Done false
832 249 Acked 1248 Read 4 Done New Done false
833 250 Acked 1249 Read 2 Done New Done false
834 251 Acked 1250 Write 10 Done New Done false
835 252 Acked 1251 Read 8 Done New Done false
836 253 Acked 1252 Write 7 Done New Done false
837 254 Acked 1253 Read 9 Done New Done false
838 255 Acked 1254 Write 1 Done New Done false
839 256 Acked 1255 Write 10 Done New Done false
840 257 Acked 1256 Read 1 Done New Done false
841 258 Acked 1257 Write 1 Done New Done false
842 259 Acked 1258 Write 8 Done New Done false
843 260 Acked 1259 Read 6 Done New Done false
844 261 Acked 1260 Write 7 Done New Done false
845 262 Acked 1261 Read 3 Done New Done false
846 263 Acked 1262 Read 5 Done New Done false
847 264 Acked 1263 Write 4 Done New Done false
848 265 Acked 1264 Read 1 Done New Done false
849 266 Acked 1265 Read 9 Done New Done false
850 267 Acked 1266 Write 9 Done New Done false
851 268 Acked 1267 Write 8 Done New Done false
852 269 Acked 1268 Read 10 Done New Done false
853 270 Acked 1269 Write 3 Done New Done false
854 271 Acked 1270 Read 10 Done New Done false
855 272 Acked 1271 Write 10 Done New Done false
856 273 Acked 1272 Read 5 Done New Done false
857 274 Acked 1273 Read 2 Done New Done false
858 275 Acked 1274 Write 2 Done New Done false
859 276 Acked 1275 Write 9 Done New Done false
860 277 Acked 1276 Read 10 Done New Done false
861 278 Acked 1277 Read 2 Done New Done false
862 279 Acked 1278 Read 1 Done New Done false
863 280 Acked 1279 Write 9 Done New Done false
864 281 Acked 1280 Read 5 Done New Done false
865 282 Acked 1281 Flush 0 Done New Done false
866 283 Acked 1282 Write 8 Done New Done false
867 284 Acked 1283 Read 5 Done New Done false
868 285 Acked 1284 Read 5 Done New Done false
869 286 Acked 1285 Read 6 Done New Done false
870 287 Acked 1286 Read 7 Done New Done false
871 288 Acked 1287 Write 9 Done New Done false
872 289 Acked 1288 Write 5 Done New Done false
873 290 Acked 1289 Read 1 Done New Done false
874 291 Acked 1290 Read 10 Done New Done false
875 292 Acked 1291 Read 5 Done New Done false
876 293 Acked 1292 Read 5 Done New Done false
877 294 Acked 1293 Write 3 Done New Done false
878 295 Acked 1294 Write 10 Done New Done false
879 296 Acked 1295 Read 7 Done New Done false
880 297 Acked 1296 Flush 0 Done New Done false
881 298 Acked 1297 Write 2 Done New Done false
882 299 Acked 1298 Read 3 Done New Done false
883 300 Acked 1299 Write 6 Done New Done false
884 301 Acked 1300 Write 1 Done New Done false
885 302 Acked 1301 Read 4 Done New Done false
886 STATES DS:0 DS:1 DS:2 TOTAL
887 New 0 302 0 302
888 Sent 0 0 0 0
889 Done 302 0 302 604
890 Skipped 0 0 0 0
891 Error 0 0 0 0
892 Last Flush: 1296 0 1296
893 Downstairs last five completed:
894 Upstairs last five completed: 302 301 300 299 298 297
895 CLIENT: Up:0 ds:302 act:2
896 [8820] got start action
897 [8820] I am going to restart
898 Starting downstairs at port 8820
899 Make output file at "/tmp/dsc/downstairs-8820.txt"
900 [8820][1] reports Starting
901 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1177)
902 [8820][1] reports Running
9032023-10-13T04:41:56.206ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 1
9042023-10-13T04:41:56.206ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state Offline
9052023-10-13T04:41:56.206ZINFOcrucible: [1] client re-new 303 jobs since flush 0 = downstairs
9062023-10-13T04:41:56.207ZINFOcrucible: [1] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
9072023-10-13T04:41:56.207ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
9082023-10-13T04:41:56.207ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
9092023-10-13T04:41:56.207ZINFOcrucible: Returning client:1 UUID:12345678-0000-0000-0000-000000008820 matches
9102023-10-13T04:41:56.207ZINFOcrucible: [1] send last flush ID to this DS: 0
9112023-10-13T04:41:56.207ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
9122023-10-13T04:41:56.207ZINFOcrucible: [1] Replied this last flush ID: 0
9132023-10-13T04:41:56.208ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Offline Active ds_transition to Replay
9142023-10-13T04:41:56.208ZINFOcrucible: [1] Transition from Offline to Replay
9152023-10-13T04:41:56.208ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
9162023-10-13T04:41:56.208ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Replay Active ds_transition to Active
9172023-10-13T04:41:56.208ZINFOcrucible: [1] Transition from Replay to Active
9182023-10-13T04:41:56.208ZINFOcrucible: [1] Starts cmd_loop
9192023-10-13T04:41:56.208ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
9202023-10-13T04:41:56.208ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
9212023-10-13T04:41:56.208ZINFOcrucible: Set check for repair
9222023-10-13T04:41:57.209ZINFOcrucible: Checking if live repair is needed
9232023-10-13T04:41:57.209ZINFOcrucible: No Live Repair required at this time
9242023-10-13T04:41:57.209ZWARNcrucible: [1] flow control end
925 ----------------------------------------------------------------
926 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
927 Downstairs last five completed: 1302 1301 1300 1299 1298 1297
928 Upstairs last five completed: 303 302 301 300 299 298
929 CLIENT: Up:0 ds:0 act:3
930 Replay: All jobs finished, all DS active.
931 Main task has work to do, go find it
932 got dsc Stop(0)
933 stop 0
934 Oct 13 04:41:58.128 INFO request completed, latency_us: 489, [8810] Got stop action so:false kr:false
935 response_code: 204, uri: /stop/cid/0, method: POST, req_id: 1ab33b1b-a8f6-4fa5-b8d6-c122e8ffb855, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
936 Oct 13 04:41:58.129 INFO request completed, latency_us: 293, response_code: 200, uri: /state/cid/0, method: GET, req_id: e9486f3c-89ac-4356-813e-207d73f18382, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
9372023-10-13T04:41:58.129ZWARNcrucible: [0] downstairs disconnected
9382023-10-13T04:41:58.129ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
9392023-10-13T04:41:58.129ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8810 closed looper = 0
9402023-10-13T04:41:58.129ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:false
9412023-10-13T04:41:58.129ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Offline Active Active
9422023-10-13T04:41:58.129ZINFOcrucible: [0] 127.0.0.1:8810 task reports offline
9432023-10-13T04:41:58.129ZWARNcrucible: [0] pm_task rx.recv() is None
944 [8810] Exited with: ExitStatus(unix_wait_status(9))
945 [8810][0] reports Exit
946 Oct 13 04:42:02.131 INFO request completed, latency_us: 251, response_code: 200, uri: /state/cid/0, method: GET, req_id: 2b6c7290-f886-481a-b781-30ed2de1f69b, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
947 001/300 Flush
948 002/300 Read block 639 len 8192
949 003/300 Write block 2005 len 36864 data: 2 2 2 2 2 2 2 2 2
950 004/300 Read block 360 len 24576
951 005/300 Read block 4267 len 36864
952 006/300 Write block 898 len 12288 data: 2 2 2
953 007/300 Read block 3539 len 20480
954 008/300 Read block 1016 len 24576
955 009/300 Flush
956 010/300 Read block 1392 len 12288
957 011/300 Flush
958 012/300 Read block 4544 len 12288
959 013/300 Write block 3785 len 40960 data: 2 2 2 2 2 2 2 2 2 2
960 014/300 Write block 4915 len 8192 data: 3 3
961 015/300 Write block 2833 len 36864 data: 2 2 2 2 2 2 2 2 2
962 016/300 Read block 2253 len 4096
963 017/300 Write block 866 len 12288 data: 2 2 2
964 018/300 Read block 4980 len 28672
965 019/300 Write block 856 len 32768 data: 2 2 2 2 2 2 2 2
966 020/300 Write block 3206 len 16384 data: 2 2 2 2
967 021/300 Read block 1279 len 40960
968 022/300 Write block 1554 len 28672 data: 3 4 3 3 3 3 3
969 023/300 Write block 288 len 20480 data: 2 2 2 2 2
970 024/300 Read block 3122 len 16384
971 025/300 Write block 4909 len 28672 data: 2 2 2 2 3 3 4
972 026/300 Flush
973 027/300 Read block 1725 len 8192
974 028/300 Read block 2207 len 32768
975 029/300 Read block 1844 len 16384
976 030/300 Read block 4930 len 20480
977 031/300 Read block 4233 len 36864
978 032/300 Write block 452 len 16384 data: 2 2 2 2
979 033/300 Read block 2857 len 4096
980 034/300 Read block 378 len 40960
981 035/300 Read block 3704 len 4096
982 036/300 Write block 3884 len 40960 data: 2 2 2 2 2 2 2 2 2 2
983 037/300 Write block 4903 len 12288 data: 2 2 2
984 038/300 Read block 4888 len 8192
985 039/300 Read block 2214 len 24576
986 040/300 Write block 2358 len 36864 data: 2 2 2 2 2 2 2 2 2
987 041/300 Read block 3562 len 8192
988 042/300 Write block 4368 len 40960 data: 2 2 2 2 2 2 2 2 2 2
989 043/300 Read block 1498 len 32768
990 044/300 Write block 3653 len 4096 data: 2
991 045/300 Write block 4827 len 36864 data: 3 3 3 3 3 3 2 2 2
992 046/300 Write block 2322 len 36864 data: 2 2 2 2 2 2 2 2 2
993 047/300 Write block 4539 len 24576 data: 3 3 3 3 3 2
994 048/300 Read block 3998 len 24576
995 049/300 Write block 4180 len 20480 data: 2 2 2 2 2
996 050/300 Write block 1078 len 4096 data: 2
997 051/300 Read block 1368 len 16384
998 052/300 Read block 2240 len 32768
999 053/300 Write block 2425 len 32768 data: 2 2 2 2 2 2 2 2
1000 054/300 Write block 179 len 36864 data: 2 2 2 2 2 2 2 2 2
1001 055/300 Read block 3110 len 28672
1002 056/300 Read block 1864 len 36864
1003 057/300 Read block 3260 len 24576
1004 058/300 Write block 2413 len 24576 data: 2 2 2 2 2 2
1005 059/300 Write block 4694 len 40960 data: 2 2 2 2 3 3 3 3 2 2
1006 060/300 Read block 3334 len 24576
1007 061/300 Read block 2483 len 20480
1008 062/300 Read block 1621 len 36864
1009 063/300 Read block 2397 len 24576
1010 064/300 Read block 3961 len 16384
1011 065/300 Write block 2125 len 24576 data: 2 2 2 2 2 2
1012 066/300 Write block 4915 len 20480 data: 5 4 3 3 3
1013 067/300 Read block 3584 len 32768
1014 068/300 Write block 433 len 20480 data: 2 2 2 3 3
1015 069/300 Read block 3547 len 32768
1016 070/300 Read block 1543 len 20480
1017 071/300 Read block 1594 len 4096
1018 072/300 Read block 719 len 8192
1019 073/300 Read block 2621 len 8192
1020 074/300 Read block 4279 len 36864
1021 075/300 Read block 308 len 40960
1022 076/300 Read block 4563 len 40960
1023 077/300 Read block 1445 len 16384
1024 078/300 Write block 2195 len 20480 data: 2 2 2 2 2
1025 079/300 Read block 1795 len 16384
1026 080/300 Read block 2875 len 8192
1027 081/300 Write block 4086 len 20480 data: 3 3 3 2 2
1028 082/300 Read block 168 len 16384
1029 083/300 Flush
1030 084/300 Write block 1528 len 16384 data: 2 2 3 3
1031 085/300 Write block 1921 len 12288 data: 2 2 2
1032 086/300 Flush
1033 087/300 Read block 4151 len 8192
1034 088/300 Write block 1138 len 28672 data: 2 2 2 2 2 2 2
1035 089/300 Read block 4770 len 8192
1036 090/300 Write block 4019 len 36864 data: 2 2 2 2 2 2 2 2 2
1037 091/300 Read block 668 len 4096
1038 092/300 Write block 4810 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1039 093/300 Read block 4946 len 40960
1040 094/300 Read block 2359 len 20480
1041 095/300 Read block 3185 len 28672
1042 096/300 Read block 3088 len 8192
1043 097/300 Read block 4469 len 24576
1044 098/300 Read block 2887 len 20480
1045 099/300 Write block 44 len 28672 data: 2 2 2 2 2 2 2
1046 100/300 Read block 784 len 40960
1047 101/300 Read block 726 len 28672
1048 102/300 Write block 1158 len 40960 data: 3 2 2 2 2 2 2 2 2 2
1049 103/300 Read block 2631 len 4096
1050 104/300 Read block 2046 len 16384
1051 105/300 Write block 4974 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1052 106/300 Flush
1053 107/300 Flush
1054 108/300 Write block 3351 len 32768 data: 2 2 2 2 2 2 2 2
1055 109/300 Read block 3230 len 16384
1056 110/300 Write block 660 len 36864 data: 2 2 3 2 2 2 2 2 2
1057 111/300 Read block 739 len 12288
1058 112/300 Read block 2450 len 40960
1059 113/300 Flush
1060 114/300 Read block 2179 len 24576
1061 115/300 Write block 2190 len 28672 data: 2 2 2 2 2 3 3
1062 116/300 Write block 2581 len 16384 data: 2 2 2 2
1063 117/300 Flush
1064 118/300 Write block 511 len 32768 data: 2 2 2 2 2 2 2 2
1065 119/300 Read block 478 len 28672
1066 120/300 Write block 1803 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1067 121/300 Write block 2355 len 16384 data: 2 2 2 3
1068 122/300 Read block 2979 len 24576
1069 123/300 Read block 3376 len 28672
1070 124/300 Write block 3748 len 20480 data: 2 2 2 2 2
1071 125/300 Read block 2504 len 16384
1072 126/300 Write block 701 len 12288 data: 2 2 2
1073 127/300 Read block 2913 len 36864
1074 128/300 Write block 1591 len 16384 data: 2 2 2 2
1075 129/300 Write block 3492 len 12288 data: 3 3 3
1076 130/300 Read block 4837 len 40960
1077 131/300 Read block 2955 len 40960
1078 132/300 Flush
1079 133/300 Read block 1973 len 20480
1080 134/300 Read block 3440 len 8192
1081 135/300 Flush
1082 136/300 Read block 2809 len 32768
1083 137/300 Read block 2778 len 12288
1084 138/300 Write block 4614 len 8192 data: 2 2
1085 139/300 Read block 210 len 40960
1086 140/300 Read block 1101 len 32768
1087 141/300 Read block 2114 len 28672
1088 142/300 Flush
1089 143/300 Read block 4012 len 4096
1090 144/300 Read block 514 len 8192
1091 145/300 Read block 949 len 20480
1092 146/300 Write block 533 len 12288 data: 2 2 2
1093 147/300 Write block 4551 len 20480 data: 2 2 2 2 2
1094 148/300 Read block 3335 len 24576
1095 149/300 Write block 4939 len 20480 data: 2 3 3 3 3
1096 150/300 Flush
1097 151/300 Write block 972 len 8192 data: 2 2
1098 152/300 Read block 231 len 28672
1099 153/300 Read block 1409 len 32768
1100 154/300 Write block 2397 len 28672 data: 2 2 2 2 2 2 2
1101 155/300 Write block 3149 len 4096 data: 2
1102 156/300 Flush
1103 157/300 Read block 104 len 24576
1104 158/300 Write block 1770 len 32768 data: 2 2 2 2 2 2 2 2
1105 159/300 Write block 4658 len 36864 data: 2 2 2 2 2 3 2 2 2
1106 160/300 Read block 2482 len 8192
1107 161/300 Read block 4117 len 28672
1108 162/300 Write block 2272 len 20480 data: 2 2 2 2 2
1109 163/300 Read block 3864 len 32768
1110 164/300 Flush
1111 165/300 Flush
1112 166/300 Read block 3628 len 12288
1113 167/300 Write block 749 len 4096 data: 2
1114 168/300 Write block 3841 len 36864 data: 3 3 3 2 2 2 2 2 2
1115 169/300 Read block 2324 len 16384
1116 170/300 Read block 490 len 20480
1117 171/300 Read block 3888 len 40960
1118 172/300 Write block 1282 len 36864 data: 2 2 3 3 3 3 4 4 4
1119 173/300 Read block 327 len 8192
1120 174/300 Read block 471 len 36864
1121 175/300 Write block 2951 len 8192 data: 2 2
1122 176/300 Read block 4320 len 20480
1123 177/300 Write block 487 len 32768 data: 3 3 3 3 2 2 2 2
1124 178/300 Read block 4221 len 12288
1125 179/300 Write block 1514 len 12288 data: 2 2 2
1126 180/300 Read block 1057 len 28672
1127 181/300 Read block 2463 len 40960
1128 182/300 Read block 281 len 16384
1129 183/300 Write block 3952 len 24576 data: 2 2 2 2 2 2
1130 184/300 Write block 406 len 24576 data: 2 2 2 2 2 2
1131 185/300 Read block 187 len 4096
1132 186/300 Write block 2284 len 20480 data: 2 2 2 2 2
1133 187/300 Write block 4712 len 12288 data: 2 2 2
1134 188/300 Read block 4615 len 32768
1135 189/300 Read block 334 len 20480
1136 190/300 Read block 4111 len 28672
1137 191/300 Write block 2904 len 4096 data: 2
1138 192/300 Write block 3464 len 12288 data: 2 2 2
1139 193/300 Write block 4246 len 16384 data: 2 2 2 2
1140 194/300 Write block 4066 len 36864 data: 2 2 2 2 2 2 2 2 2
1141 195/300 Read block 2576 len 16384
1142 196/300 Read block 1706 len 12288
1143 197/300 Write block 2593 len 32768 data: 2 2 2 2 2 2 2 2
1144 198/300 Read block 1254 len 4096
1145 199/300 Write block 3077 len 12288 data: 2 2 2
1146 200/300 Write block 3580 len 32768 data: 2 2 3 3 2 2 2 2
1147 201/300 Read block 2276 len 8192
1148 202/300 Write block 4099 len 20480 data: 2 2 2 2 2
1149 203/300 Read block 2285 len 24576
1150 204/300 Read block 2390 len 24576
1151 205/300 Read block 3118 len 36864
1152 206/300 Write block 2260 len 32768 data: 2 2 2 2 2 2 2 2
1153 207/300 Read block 405 len 36864
1154 208/300 Read block 1506 len 36864
1155 209/300 Read block 3789 len 12288
1156 210/300 Read block 3579 len 12288
1157 211/300 Write block 1304 len 24576 data: 2 3 3 2 2 2
1158 212/300 Read block 357 len 24576
1159 213/300 Write block 1722 len 36864 data: 2 2 2 2 2 2 2 2 2
1160 214/300 Read block 3474 len 20480
1161 215/300 Read block 2391 len 40960
1162 216/300 Read block 2486 len 20480
1163 217/300 Read block 569 len 12288
1164 218/300 Read block 3752 len 16384
1165 219/300 Write block 3728 len 4096 data: 2
1166 220/300 Flush
1167 221/300 Read block 580 len 12288
1168 222/300 Read block 3739 len 28672
1169 223/300 Read block 4871 len 28672
1170 224/300 Flush
1171 225/300 Read block 4019 len 36864
1172 226/300 Flush
1173 227/300 Write block 871 len 32768 data: 2 2 2 2 2 2 2 2
1174 228/300 Write block 4345 len 20480 data: 2 2 2 2 2
1175 229/300 Flush
1176 230/300 Write block 587 len 36864 data: 3 2 2 2 2 2 2 3 3
1177 231/300 Read block 1970 len 28672
1178 232/300 Flush
1179 233/300 Read block 2580 len 40960
1180 234/300 Flush
1181 235/300 Read block 3007 len 32768
1182 236/300 Write block 3473 len 4096 data: 2
1183 237/300 Flush
1184 238/300 Read block 856 len 24576
1185 239/300 Read block 1813 len 40960
1186 240/300 Write block 715 len 16384 data: 2 2 2 2
1187 241/300 Write block 3306 len 16384 data: 2 2 2 2
1188 242/300 Write block 3828 len 12288 data: 2 2 2
1189 243/300 Write block 2736 len 28672 data: 2 2 2 2 2 2 2
1190 244/300 Read block 2777 len 16384
1191 245/300 Write block 217 len 12288 data: 3 3 3
1192 246/300 Read block 3085 len 4096
1193 247/300 Write block 2522 len 40960 data: 3 3 3 3 2 2 2 2 2 2
1194 248/300 Write block 2161 len 16384 data: 2 2 2 2
1195 249/300 Read block 542 len 8192
1196 250/300 Read block 3779 len 12288
1197 251/300 Read block 1744 len 8192
1198 252/300 Flush
1199 253/300 Read block 4091 len 24576
1200 254/300 Write block 1429 len 28672 data: 3 2 2 2 2 2 2
1201 255/300 Read block 787 len 8192
1202 256/300 Flush
1203 257/300 Read block 4267 len 12288
1204 258/300 Read block 2152 len 40960
1205 259/300 Write block 209 len 16384 data: 2 2 2 2
1206 260/300 Write block 2288 len 12288 data: 3 2 2
1207 261/300 Write block 4163 len 32768 data: 2 2 2 2 3 3 3 3
1208 262/300 Write block 4705 len 12288 data: 2 2 2
1209 263/300 Write block 3454 len 32768 data: 2 2 2 2 2 3 3 2
1210 264/300 Flush
1211 265/300 Write block 108 len 32768 data: 2 2 2 2 2 2 2 2
1212 266/300 Read block 560 len 28672
1213 267/300 Flush
1214 268/300 Read block 1844 len 40960
1215 269/300 Write block 1939 len 20480 data: 2 2 2 2 2
1216 270/300 Write block 509 len 28672 data: 2 2 3 3 3 3 3
1217 271/300 Write block 4417 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1218 272/300 Write block 304 len 32768 data: 3 3 3 3 3 3 2 2
1219 273/300 Read block 1940 len 8192
1220 274/300 Read block 4687 len 8192
1221 275/300 Write block 1859 len 12288 data: 3 3 3
1222 276/300 Read block 4012 len 16384
1223 277/300 Read block 236 len 24576
1224 278/300 Read block 4883 len 24576
1225 279/300 Read block 1824 len 12288
1226 280/300 Read block 4349 len 28672
1227 281/300 Read block 784 len 20480
1228 282/300 Write block 4273 len 12288 data: 2 2 2
1229 283/300 Write block 383 len 24576 data: 2 2 2 2 2 2
1230 284/300 Read block 889 len 28672
1231 285/300 Write block 292 len 36864 data: 3 2 2 2 2 2 2 2 2
1232 286/300 Read block 4772 len 32768
1233 287/300 Write block 3857 len 28672 data: 2 2 2 2 2 2 2
1234 288/300 Read block 3530 len 36864
1235 289/300 Read block 3691 len 4096
1236 290/300 Read block 4816 len 28672
1237 291/300 Write block 4792 len 16384 data: 2 2 2 2
1238 292/300 Write block 1401 len 24576 data: 2 2 2 2 2 2
1239 293/300 Write block 1187 len 20480 data: 2 2 2 2 2
1240 294/300 Write block 4909 len 4096 data: 3
1241 295/300 Read block 1691 len 16384
1242 296/300 Read block 4897 len 24576
1243 297/300 Write block 4355 len 32768 data: 2 2 2 2 2 2 2 2
1244 298/300 Read block 292 len 36864
1245 299/300 Write block 633 len 4096 data: 2
1246 300/300 Read block 1341 len 24576
1247 Main task has work to do, go find it
1248 got dsc Start(0)
1249 start 0
1250 Oct 13 04:42:03.042 INFO request completed, latency_us: 288, response_code: 204, uri: /start/cid/0, method: POST, req_id: 82b0513a-2925-4f52-8884-aea35bc90430, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
1251 Replay: started 0, returned:Ok(())
1252 ----------------------------------------------------------------
1253 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
1254 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
1255 304 Acked 1303 Flush 0 New Done Done false
1256 305 Acked 1304 Read 2 New Done Done false
1257 306 Acked 1305 Write 9 New Done Done false
1258 307 Acked 1306 Read 6 New Done Done false
1259 308 Acked 1307 Read 9 New Done Done false
1260 309 Acked 1308 Write 3 New Done Done false
1261 310 Acked 1309 Read 5 New Done Done false
1262 311 Acked 1310 Read 6 New Done Done false
1263 312 Acked 1311 Flush 0 New Done Done false
1264 313 Acked 1312 Read 3 New Done Done false
1265 314 Acked 1313 Flush 0 New Done Done false
1266 315 Acked 1314 Read 3 New Done Done false
1267 316 Acked 1315 Write 10 New Done Done false
1268 317 Acked 1316 Write 2 New Done Done false
1269 318 Acked 1317 Write 9 New Done Done false
1270 319 Acked 1318 Read 1 New Done Done false
1271 320 Acked 1319 Write 3 New Done Done false
1272 321 Acked 1320 Read 7 New Done Done false
1273 322 Acked 1321 Write 8 New Done Done false
1274 323 Acked 1322 Write 4 New Done Done false
1275 324 Acked 1323 Read 10 New Done Done false
1276 325 Acked 1324 Write 7 New Done Done false
1277 326 Acked 1325 Write 5 New Done Done false
1278 327 Acked 1326 Read 4 New Done Done false
1279 328 Acked 1327 Write 7 New Done Done false
1280 329 Acked 1328 Flush 0 New Done Done false
1281 330 Acked 1329 Read 2 New Done Done false
1282 331 Acked 1330 Read 8 New Done Done false
1283 332 Acked 1331 Read 4 New Done Done false
1284 333 Acked 1332 Read 5 New Done Done false
1285 334 Acked 1333 Read 9 New Done Done false
1286 335 Acked 1334 Write 4 New Done Done false
1287 336 Acked 1335 Read 1 New Done Done false
1288 337 Acked 1336 Read 10 New Done Done false
1289 338 Acked 1337 Read 1 New Done Done false
1290 339 Acked 1338 Write 10 New Done Done false
1291 340 Acked 1339 Write 3 New Done Done false
1292 341 Acked 1340 Read 2 New Done Done false
1293 342 Acked 1341 Read 6 New Done Done false
1294 343 Acked 1342 Write 9 New Done Done false
1295 344 Acked 1343 Read 2 New Done Done false
1296 345 Acked 1344 Write 10 New Done Done false
1297 346 Acked 1345 Read 8 New Done Done false
1298 347 Acked 1346 Write 1 New Done Done false
1299 348 Acked 1347 Write 9 New Done Done false
1300 349 Acked 1348 Write 9 New Done Done false
1301 350 Acked 1349 Write 6 New Done Done false
1302 351 Acked 1350 Read 6 New Done Done false
1303 352 Acked 1351 Write 5 New Done Done false
1304 353 Acked 1352 Write 1 New Done Done false
1305 354 Acked 1353 Read 4 New Done Done false
1306 355 Acked 1354 Read 8 New Done Done false
1307 356 Acked 1355 Write 8 New Done Done false
1308 357 Acked 1356 Write 9 New Done Done false
1309 358 Acked 1357 Read 7 New Done Done false
1310 359 Acked 1358 Read 9 New Done Done false
1311 360 Acked 1359 Read 6 New Done Done false
1312 361 Acked 1360 Write 6 New Done Done false
1313 362 Acked 1361 Write 10 New Done Done false
1314 363 Acked 1362 Read 6 New Done Done false
1315 364 Acked 1363 Read 5 New Done Done false
1316 365 Acked 1364 Read 9 New Done Done false
1317 366 Acked 1365 Read 6 New Done Done false
1318 367 Acked 1366 Read 4 New Done Done false
1319 368 Acked 1367 Write 6 New Done Done false
1320 369 Acked 1368 Write 5 New Done Done false
1321 370 Acked 1369 Read 8 New Done Done false
1322 371 Acked 1370 Write 5 New Done Done false
1323 372 Acked 1371 Read 8 New Done Done false
1324 373 Acked 1372 Read 5 New Done Done false
1325 374 Acked 1373 Read 1 New Done Done false
1326 375 Acked 1374 Read 2 New Done Done false
1327 376 Acked 1375 Read 2 New Done Done false
1328 377 Acked 1376 Read 9 New Done Done false
1329 378 Acked 1377 Read 10 New Done Done false
1330 379 Acked 1378 Read 10 New Done Done false
1331 380 Acked 1379 Read 4 New Done Done false
1332 381 Acked 1380 Write 5 New Done Done false
1333 382 Acked 1381 Read 4 New Done Done false
1334 383 Acked 1382 Read 2 New Done Done false
1335 384 Acked 1383 Write 5 New Done Done false
1336 385 Acked 1384 Read 4 New Done Done false
1337 386 Acked 1385 Flush 0 New Done Done false
1338 387 Acked 1386 Write 4 New Done Done false
1339 388 Acked 1387 Write 3 New Done Done false
1340 389 Acked 1388 Flush 0 New Done Done false
1341 390 Acked 1389 Read 2 New Done Done false
1342 391 Acked 1390 Write 7 New Done Done false
1343 392 Acked 1391 Read 2 New Done Done false
1344 393 Acked 1392 Write 9 New Done Done false
1345 394 Acked 1393 Read 1 New Done Done false
1346 395 Acked 1394 Write 10 New Done Done false
1347 396 Acked 1395 Read 10 New Done Done false
1348 397 Acked 1396 Read 5 New Done Done false
1349 398 Acked 1397 Read 7 New Done Done false
1350 399 Acked 1398 Read 2 New Done Done false
1351 400 Acked 1399 Read 6 New Done Done false
1352 401 Acked 1400 Read 5 New Done Done false
1353 402 Acked 1401 Write 7 New Done Done false
1354 403 Acked 1402 Read 10 New Done Done false
1355 404 Acked 1403 Read 7 New Done Done false
1356 405 Acked 1404 Write 10 New Done Done false
1357 406 Acked 1405 Read 1 New Done Done false
1358 407 Acked 1406 Read 4 New Done Done false
1359 408 Acked 1407 Write 10 New Done Done false
1360 409 Acked 1408 Flush 0 New Done Done false
1361 410 Acked 1409 Flush 0 New Done Done false
1362 411 Acked 1410 Write 8 New Done Done false
1363 412 Acked 1411 Read 4 New Done Done false
1364 413 Acked 1412 Write 9 New Done Done false
1365 414 Acked 1413 Read 3 New Done Done false
1366 415 Acked 1414 Read 10 New Done Done false
1367 416 Acked 1415 Flush 0 New Done Done false
1368 417 Acked 1416 Read 6 New Done Done false
1369 418 Acked 1417 Write 7 New Done Done false
1370 419 Acked 1418 Write 4 New Done Done false
1371 420 Acked 1419 Flush 0 New Done Done false
1372 421 Acked 1420 Write 8 New Done Done false
1373 422 Acked 1421 Read 7 New Done Done false
1374 423 Acked 1422 Write 10 New Done Done false
1375 424 Acked 1423 Write 4 New Done Done false
1376 425 Acked 1424 Read 6 New Done Done false
1377 426 Acked 1425 Read 7 New Done Done false
1378 427 Acked 1426 Write 5 New Done Done false
1379 428 Acked 1427 Read 4 New Done Done false
1380 429 Acked 1428 Write 3 New Done Done false
1381 430 Acked 1429 Read 9 New Done Done false
1382 431 Acked 1430 Write 4 New Done Done false
1383 432 Acked 1431 Write 3 New Done Done false
1384 433 Acked 1432 Read 10 New Done Done false
1385 434 Acked 1433 Read 10 New Done Done false
1386 435 Acked 1434 Flush 0 New Done Done false
1387 436 Acked 1435 Read 5 New Done Done false
1388 437 Acked 1436 Read 2 New Done Done false
1389 438 Acked 1437 Flush 0 New Done Done false
1390 439 Acked 1438 Read 8 New Done Done false
1391 440 Acked 1439 Read 3 New Done Done false
1392 441 Acked 1440 Write 2 New Done Done false
1393 442 Acked 1441 Read 10 New Done Done false
1394 443 Acked 1442 Read 8 New Done Done false
1395 444 Acked 1443 Read 7 New Done Done false
1396 445 Acked 1444 Flush 0 New Done Done false
1397 446 Acked 1445 Read 1 New Done Done false
1398 447 Acked 1446 Read 2 New Done Done false
1399 448 Acked 1447 Read 5 New Done Done false
1400 449 Acked 1448 Write 3 New Done Done false
1401 450 Acked 1449 Write 5 New Done Done false
1402 451 Acked 1450 Read 6 New Done Done false
1403 452 Acked 1451 Write 5 New Done Done false
1404 453 Acked 1452 Flush 0 New Done Done false
1405 454 Acked 1453 Write 2 New Done Done false
1406 455 Acked 1454 Read 7 New Done Done false
1407 456 Acked 1455 Read 8 New Done Done false
1408 457 Acked 1456 Write 7 New Done Done false
1409 458 Acked 1457 Write 1 New Done Done false
1410 459 Acked 1458 Flush 0 New Done Done false
1411 460 Acked 1459 Read 6 New Done Done false
1412 461 Acked 1460 Write 8 New Done Done false
1413 462 Acked 1461 Write 9 New Done Done false
1414 463 Acked 1462 Read 2 New Done Done false
1415 464 Acked 1463 Read 7 New Done Done false
1416 465 Acked 1464 Flush 0 New Done Done false
1417 466 Acked 1465 Write 5 New Done Done false
1418 467 Acked 1466 Read 8 New Done Done false
1419 468 Acked 1467 Flush 0 New Done Done false
1420 469 Acked 1468 Flush 0 New Done Done false
1421 470 Acked 1469 Read 3 New Done Done false
1422 471 Acked 1470 Write 1 New Done Done false
1423 472 Acked 1471 Write 9 New Done Done false
1424 473 Acked 1472 Read 4 New Done Done false
1425 474 Acked 1473 Read 5 New Done Done false
1426 475 Acked 1474 Read 10 New Done Done false
1427 476 Acked 1475 Write 9 New Done Done false
1428 477 Acked 1476 Read 2 New Done Done false
1429 478 Acked 1477 Read 9 New Done Done false
1430 479 Acked 1478 Write 2 New Done Done false
1431 480 Acked 1479 Read 5 New Done Done false
1432 481 Acked 1480 Write 8 New Done Done false
1433 482 Acked 1481 Read 3 New Done Done false
1434 483 Acked 1482 Write 3 New Done Done false
1435 484 Acked 1483 Read 7 New Done Done false
1436 485 Acked 1484 Read 10 New Done Done false
1437 486 Acked 1485 Read 4 New Done Done false
1438 487 Acked 1486 Write 6 New Done Done false
1439 488 Acked 1487 Write 6 New Done Done false
1440 489 Acked 1488 Read 1 New Done Done false
1441 490 Acked 1489 Write 5 New Done Done false
1442 491 Acked 1490 Write 3 New Done Done false
1443 492 Acked 1491 Read 8 New Done Done false
1444 493 Acked 1492 Read 5 New Done Done false
1445 494 Acked 1493 Read 7 New Done Done false
1446 495 Acked 1494 Write 1 New Done Done false
1447 496 Acked 1495 Write 3 New Done Done false
1448 497 Acked 1496 Write 4 New Done Done false
1449 498 Acked 1497 Write 9 New Done Done false
1450 499 Acked 1498 Read 4 New Done Done false
1451 500 Acked 1499 Read 3 New Done Done false
1452 501 Acked 1500 Write 8 New Done Done false
1453 502 Acked 1501 Read 1 New Done Done false
1454 503 Acked 1502 Write 3 New Done Done false
1455 504 Acked 1503 Write 8 New Done Done false
1456 505 Acked 1504 Read 2 New Done Done false
1457 506 Acked 1505 Write 5 New Done Done false
1458 507 Acked 1506 Read 6 New Done Done false
1459 508 Acked 1507 Read 6 New Done Done false
1460 509 Acked 1508 Read 9 New Done Done false
1461 510 Acked 1509 Write 8 New Done Done false
1462 511 Acked 1510 Read 9 New Done Done false
1463 512 Acked 1511 Read 9 New Done Done false
1464 513 Acked 1512 Read 3 New Done Done false
1465 514 Acked 1513 Read 3 New Done Done false
1466 515 Acked 1514 Write 6 New Done Done false
1467 516 Acked 1515 Read 6 New Done Done false
1468 517 Acked 1516 Write 9 New Done Done false
1469 518 Acked 1517 Read 5 New Done Done false
1470 519 Acked 1518 Read 10 New Done Done false
1471 520 Acked 1519 Read 5 New Done Done false
1472 521 Acked 1520 Read 3 New Done Done false
1473 522 Acked 1521 Read 4 New Done Done false
1474 523 Acked 1522 Write 1 New Done Done false
1475 524 Acked 1523 Flush 0 New Done Done false
1476 525 Acked 1524 Read 3 New Done Done false
1477 526 Acked 1525 Read 7 New Done Done false
1478 527 Acked 1526 Read 7 New Done Done false
1479 528 Acked 1527 Flush 0 New Done Done false
1480 529 Acked 1528 Read 9 New Done Done false
1481 530 Acked 1529 Flush 0 New Done Done false
1482 531 Acked 1530 Write 8 New Done Done false
1483 532 Acked 1531 Write 5 New Done Done false
1484 533 Acked 1532 Flush 0 New Done Done false
1485 534 Acked 1533 Write 9 New Done Done false
1486 535 Acked 1534 Read 7 New Done Done false
1487 536 Acked 1535 Flush 0 New Done Done false
1488 537 Acked 1536 Read 10 New Done Done false
1489 538 Acked 1537 Flush 0 New Done Done false
1490 539 Acked 1538 Read 8 New Done Done false
1491 540 Acked 1539 Write 1 New Done Done false
1492 541 Acked 1540 Flush 0 New Done Done false
1493 542 Acked 1541 Read 6 New Done Done false
1494 543 Acked 1542 Read 10 New Done Done false
1495 544 Acked 1543 Write 4 New Done Done false
1496 545 Acked 1544 Write 4 New Done Done false
1497 546 Acked 1545 Write 3 New Done Done false
1498 547 Acked 1546 Write 7 New Done Done false
1499 548 Acked 1547 Read 4 New Done Done false
1500 549 Acked 1548 Write 3 New Done Done false
1501 550 Acked 1549 Read 1 New Done Done false
1502 551 Acked 1550 Write 10 New Done Done false
1503 552 Acked 1551 Write 4 New Done Done false
1504 553 Acked 1552 Read 2 New Done Done false
1505 554 Acked 1553 Read 3 New Done Done false
1506 555 Acked 1554 Read 2 New Done Done false
1507 556 Acked 1555 Flush 0 New Done Done false
1508 557 Acked 1556 Read 6 New Done Done false
1509 558 Acked 1557 Write 7 New Done Done false
1510 559 Acked 1558 Read 2 New Done Done false
1511 560 Acked 1559 Flush 0 New Done Done false
1512 561 Acked 1560 Read 3 New Done Done false
1513 562 Acked 1561 Read 10 New Done Done false
1514 563 Acked 1562 Write 4 New Done Done false
1515 564 Acked 1563 Write 3 New Done Done false
1516 565 Acked 1564 Write 8 New Done Done false
1517 566 Acked 1565 Write 3 New Done Done false
1518 567 Acked 1566 Write 8 New Done Done false
1519 568 Acked 1567 Flush 0 New Done Done false
1520 569 Acked 1568 Write 8 New Done Done false
1521 570 Acked 1569 Read 7 New Done Done false
1522 571 Acked 1570 Flush 0 New Done Done false
1523 572 Acked 1571 Read 10 New Done Done false
1524 573 Acked 1572 Write 5 New Done Done false
1525 574 Acked 1573 Write 7 New Done Done false
1526 575 Acked 1574 Write 10 New Done Done false
1527 576 Acked 1575 Write 8 New Done Done false
1528 577 Acked 1576 Read 2 New Done Done false
1529 578 Acked 1577 Read 2 New Done Done false
1530 579 Acked 1578 Write 3 New Done Done false
1531 580 Acked 1579 Read 4 New Done Done false
1532 581 Acked 1580 Read 6 New Done Done false
1533 582 Acked 1581 Read 6 New Done Done false
1534 583 Acked 1582 Read 3 New Done Done false
1535 584 Acked 1583 Read 7 New Done Done false
1536 585 Acked 1584 Read 5 New Done Done false
1537 586 Acked 1585 Write 3 New Done Done false
1538 587 Acked 1586 Write 6 New Done Done false
1539 588 Acked 1587 Read 7 New Done Done false
1540 589 Acked 1588 Write 9 New Done Done false
1541 590 Acked 1589 Read 8 New Done Done false
1542 591 Acked 1590 Write 7 New Done Done false
1543 592 Acked 1591 Read 9 New Done Done false
1544 593 Acked 1592 Read 1 New Done Done false
1545 594 Acked 1593 Read 7 New Done Done false
1546 595 Acked 1594 Write 4 New Done Done false
1547 596 Acked 1595 Write 6 New Done Done false
1548 597 Acked 1596 Write 5 New Done Done false
1549 598 Acked 1597 Write 1 New Done Done false
1550 599 Acked 1598 Read 4 New Done Done false
1551 600 Acked 1599 Read 6 New Done Done false
1552 601 Acked 1600 Write 8 New Done Done false
1553 602 Acked 1601 Read 9 New Done Done false
1554 603 Acked 1602 Write 1 New Done Done false
1555 604 Acked 1603 Read 6 New Done Done false
1556 STATES DS:0 DS:1 DS:2 TOTAL
1557 New 301 0 0 301
1558 Sent 0 0 0 0
1559 Done 0 301 301 602
1560 Skipped 0 0 0 0
1561 Error 0 0 0 0
1562 Last Flush: 1302 1570 1570
1563 Downstairs last five completed: 1302 1301 1300 1299 1298 1297
1564 Upstairs last five completed: 604 603 602 601 600 599
1565 CLIENT: Up:0 ds:301 act:2
1566 [8810] got start action
1567 [8810] I am going to restart
1568 Starting downstairs at port 8810
1569 Make output file at "/tmp/dsc/downstairs-8810.txt"
1570 [8810][0] reports Starting
1571 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1178)
1572 [8810][0] reports Running
15732023-10-13T04:42:05.139ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 0
15742023-10-13T04:42:05.139ZINFOcrucible: [0] Proc runs for 127.0.0.1:8810 in state Offline
15752023-10-13T04:42:05.139ZINFOcrucible: [0] client re-new 302 jobs since flush 1302 = downstairs
15762023-10-13T04:42:05.140ZINFOcrucible: [0] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
15772023-10-13T04:42:05.140ZINFOcrucible: [0] downstairs client at 127.0.0.1:8810 has UUID 12345678-0000-0000-0000-000000008810
15782023-10-13T04:42:05.140ZINFOcrucible: [0] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }
15792023-10-13T04:42:05.140ZINFOcrucible: Returning client:0 UUID:12345678-0000-0000-0000-000000008810 matches
15802023-10-13T04:42:05.140ZINFOcrucible: [0] send last flush ID to this DS: 1302
15812023-10-13T04:42:05.140ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Offline Active Active
15822023-10-13T04:42:05.140ZINFOcrucible: [0] Replied this last flush ID: 1302
15832023-10-13T04:42:05.140ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Offline Active Active ds_transition to Replay
15842023-10-13T04:42:05.141ZINFOcrucible: [0] Transition from Offline to Replay
15852023-10-13T04:42:05.141ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
15862023-10-13T04:42:05.141ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Replay Active Active ds_transition to Active
15872023-10-13T04:42:05.141ZINFOcrucible: [0] Transition from Replay to Active
15882023-10-13T04:42:05.141ZINFOcrucible: [0] Starts cmd_loop
15892023-10-13T04:42:05.141ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:true
15902023-10-13T04:42:05.141ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
15912023-10-13T04:42:05.141ZINFOcrucible: Set check for repair
15922023-10-13T04:42:06.141ZINFOcrucible: Checking if live repair is needed
15932023-10-13T04:42:06.141ZINFOcrucible: No Live Repair required at this time
15942023-10-13T04:42:06.141ZWARNcrucible: [0] flow control end
1595 ----------------------------------------------------------------
1596 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
1597 Downstairs last five completed: 1604 1603 1602 1601 1600 1599
1598 Upstairs last five completed: 605 604 603 602 601 600
1599 CLIENT: Up:0 ds:0 act:3
1600 Replay: All jobs finished, all DS active.
1601 Main task has work to do, go find it
1602 got dsc Stop(1)
1603 stop 1
1604 Oct 13 04:42:07.047 INFO request completed, latency_us: 461, response_code: 204, uri[8820] Got stop action so:false kr:false
1605 : /stop/cid/1, method: POST, req_id: 8f6a85f2-82ad-4176-9c52-2b4dec0f5cff, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
1606 Oct 13 04:42:07.048 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/1, method: GET, req_id: 6879a47f-baf9-4184-919a-1e06ced5980e, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
16072023-10-13T04:42:07.048ZWARNcrucible: [1] downstairs disconnected
16082023-10-13T04:42:07.048ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
16092023-10-13T04:42:07.048ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8820 closed looper = 1
16102023-10-13T04:42:07.048ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:false
16112023-10-13T04:42:07.048ZWARNcrucible: [1] pm_task rx.recv() is None
16122023-10-13T04:42:07.048ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
16132023-10-13T04:42:07.048ZINFOcrucible: [1] 127.0.0.1:8820 task reports offline
1614 [8820] Exited with: ExitStatus(unix_wait_status(9))
1615 [8820][1] reports Exit
16162023-10-13T04:42:10.052ZINFOcrucible: [1] connecting to 127.0.0.1:8820 looper = 1
1617 Oct 13 04:42:11.050 INFO request completed, latency_us: 265, response_code: 200, uri: /state/cid/1, method: GET, req_id: b61ac8db-7b6f-4c45-bd0f-d3a0352eb146, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
1618 001/300 Read block 3760 len 36864
1619 002/300 Write block 2670 len 4096 data: 2
1620 003/300 Write block 3068 len 20480 data: 2 2 2 2 2
1621 004/300 Read block 1739 len 12288
1622 005/300 Flush
1623 006/300 Read block 4536 len 20480
1624 007/300 Read block 4705 len 20480
1625 008/300 Read block 2094 len 24576
1626 009/300 Read block 222 len 32768
1627 010/300 Read block 1686 len 24576
1628 011/300 Write block 1055 len 12288 data: 2 2 2
1629 012/300 Write block 124 len 8192 data: 2 2
1630 013/300 Write block 3187 len 20480 data: 2 2 2 2 2
1631 014/300 Flush
1632 015/300 Read block 3159 len 20480
1633 016/300 Write block 3932 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1634 017/300 Read block 1791 len 32768
1635 018/300 Read block 3782 len 4096
1636 019/300 Read block 895 len 20480
1637 020/300 Write block 3344 len 36864 data: 2 2 2 2 2 2 2 3 3
1638 021/300 Read block 738 len 20480
1639 022/300 Read block 3099 len 4096
1640 023/300 Read block 1081 len 4096
1641 024/300 Read block 1882 len 8192
1642 025/300 Read block 767 len 12288
1643 026/300 Write block 1876 len 8192 data: 2 2
1644 027/300 Write block 3469 len 8192 data: 2 2
1645 028/300 Write block 4460 len 16384 data: 3 3 3 2
1646 029/300 Read block 2309 len 8192
1647 030/300 Write block 348 len 36864 data: 2 2 2 2 2 2 2 2 2
1648 031/300 Write block 4052 len 32768 data: 2 2 2 2 2 2 2 2
1649 032/300 Flush
1650 033/300 Read block 1997 len 4096
1651 034/300 Flush
1652 035/300 Read block 3158 len 20480
1653 036/300 Read block 4754 len 16384
1654 037/300 Write block 3020 len 36864 data: 2 2 2 2 2 3 3 3 3
1655 038/300 Write block 1054 len 4096 data: 2
1656 039/300 Write block 4886 len 12288 data: 3 3 3
1657 040/300 Read block 764 len 32768
1658 041/300 Read block 3093 len 24576
1659 042/300 Write block 4101 len 28672 data: 3 3 3 2 2 3 3
1660 043/300 Read block 3300 len 20480
1661 044/300 Read block 458 len 8192
1662 045/300 Write block 1559 len 8192 data: 4 4
1663 046/300 Read block 1510 len 12288
1664 047/300 Write block 1435 len 8192 data: 3 3
1665 048/300 Read block 521 len 12288
1666 049/300 Write block 3378 len 36864 data: 2 2 2 2 2 2 2 2 2
1667 050/300 Read block 4109 len 20480
1668 051/300 Write block 1793 len 12288 data: 3 2 2
1669 052/300 Read block 3116 len 40960
1670 053/300 Read block 4781 len 24576
1671 054/300 Write block 47 len 16384 data: 3 3 3 3
1672 055/300 Read block 158 len 4096
1673 056/300 Read block 3473 len 32768
1674 057/300 Read block 3494 len 12288
1675 058/300 Read block 1106 len 4096
1676 059/300 Write block 4656 len 20480 data: 2 2 3 3 3
1677 060/300 Write block 3905 len 32768 data: 2 2 2 2 2 2 2 2
1678 061/300 Read block 1149 len 20480
1679 062/300 Read block 3005 len 20480
1680 063/300 Write block 3850 len 4096 data: 2
1681 064/300 Write block 4621 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1682 065/300 Write block 1738 len 4096 data: 2
1683 066/300 Flush
1684 067/300 Write block 1870 len 36864 data: 2 2 2 2 2 2 3 3 2
1685 068/300 Write block 569 len 40960 data: 2 2 2 2 2 2 2 2 2 2
1686 069/300 Read block 2277 len 12288
1687 070/300 Read block 316 len 28672
1688 071/300 Write block 4287 len 20480 data: 2 2 2 2 2
1689 072/300 Flush
1690 073/300 Read block 3338 len 20480
1691 074/300 Write block 1100 len 16384 data: 3 3 3 2
1692 075/300 Read block 2734 len 28672
1693 076/300 Read block 1512 len 4096
1694 077/300 Read block 3650 len 32768
1695 078/300 Write block 2110 len 32768 data: 2 2 2 2 2 2 2 2
1696 079/300 Write block 3345 len 4096 data: 3
1697 080/300 Read block 3131 len 24576
1698 081/300 Read block 1123 len 20480
1699 082/300 Read block 2128 len 36864
1700 083/300 Write block 610 len 12288 data: 2 2 2
1701 084/300 Read block 12 len 8192
1702 085/300 Write block 3203 len 40960 data: 2 2 2 3 3 3 3 2 2 2
1703 086/300 Read block 4894 len 4096
1704 087/300 Read block 4050 len 20480
1705 088/300 Write block 4751 len 16384 data: 2 2 2 2
1706 089/300 Write block 255 len 12288 data: 3 3 3
1707 090/300 Read block 1059 len 28672
1708 091/300 Flush
1709 092/300 Read block 3746 len 16384
1710 093/300 Read block 4975 len 4096
1711 094/300 Write block 1898 len 12288 data: 2 2 2
1712 095/300 Read block 901 len 8192
1713 096/300 Read block 4885 len 20480
1714 097/300 Write block 913 len 28672 data: 2 2 2 2 2 2 2
1715 098/300 Flush
1716 099/300 Read block 3581 len 32768
1717 100/300 Write block 4998 len 4096 data: 2
1718 101/300 Write block 3406 len 36864 data: 3 3 3 3 3 3 3 3 2
1719 102/300 Read block 1343 len 16384
1720 103/300 Read block 669 len 28672
1721 104/300 Flush
1722 105/300 Write block 3064 len 12288 data: 2 2 2
1723 106/300 Read block 4637 len 28672
1724 107/300 Read block 57 len 20480
1725 108/300 Read block 4560 len 28672
1726 109/300 Read block 4391 len 24576
1727 110/300 Read block 1057 len 20480
1728 111/300 Write block 2921 len 4096 data: 2
1729 112/300 Read block 371 len 12288
1730 113/300 Flush
1731 114/300 Write block 4162 len 16384 data: 2 3 3 3
1732 115/300 Write block 2238 len 16384 data: 2 2 2 2
1733 116/300 Read block 4928 len 40960
1734 117/300 Read block 1008 len 8192
1735 118/300 Read block 2142 len 12288
1736 119/300 Write block 4971 len 40960 data: 2 2 2 3 3 3 3 3 3 3
1737 120/300 Write block 986 len 16384 data: 3 3 3 2
1738 121/300 Write block 2021 len 16384 data: 2 2 2 2
1739 122/300 Read block 3781 len 28672
1740 123/300 Write block 2346 len 20480 data: 2 2 2 2 2
1741 124/300 Read block 1671 len 32768
1742 125/300 Write block 3761 len 32768 data: 2 2 2 2 2 2 2 2
1743 126/300 Read block 3387 len 4096
1744 127/300 Read block 1996 len 12288
1745 128/300 Read block 4924 len 20480
1746 129/300 Read block 294 len 8192
1747 130/300 Read block 55 len 24576
1748 131/300 Write block 4395 len 20480 data: 2 2 2 2 2
1749 132/300 Flush
1750 133/300 Write block 4021 len 36864 data: 3 3 3 3 3 3 3 2 2
1751 134/300 Write block 2402 len 40960 data: 3 3 2 2 2 2 2 2 2 2
1752 135/300 Read block 1963 len 28672
1753 136/300 Write block 1383 len 12288 data: 2 2 2
1754 137/300 Write block 1013 len 24576 data: 3 3 2 2 2 2
1755 138/300 Write block 4295 len 12288 data: 2 2 2
1756 139/300 Flush
1757 140/300 Flush
1758 141/300 Read block 4985 len 16384
1759 142/300 Write block 4526 len 28672 data: 2 2 2 2 2 2 2
1760 143/300 Flush
1761 144/300 Flush
1762 145/300 Read block 788 len 32768
1763 146/300 Read block 930 len 24576
1764 147/300 Write block 2259 len 28672 data: 2 3 3 3 3 3 3
1765 148/300 Flush
1766 149/300 Write block 4442 len 32768 data: 2 2 2 2 2 2 2 3
1767 150/300 Read block 731 len 8192
1768 151/300 Write block 192 len 28672 data: 2 2 2 2 2 2 2
1769 152/300 Read block 1706 len 8192
1770 153/300 Read block 376 len 8192
1771 154/300 Write block 3236 len 12288 data: 2 2 2
1772 155/300 Read block 1016 len 28672
1773 156/300 Read block 4765 len 4096
1774 157/300 Read block 1911 len 36864
1775 158/300 Flush
1776 159/300 Write block 1021 len 20480 data: 2 2 2 2 2
1777 160/300 Write block 873 len 40960 data: 3 3 3 3 3 3 2 2 2 2
1778 161/300 Read block 2968 len 8192
1779 162/300 Read block 3925 len 32768
1780 163/300 Read block 2181 len 40960
1781 164/300 Write block 1642 len 28672 data: 2 2 2 2 2 2 2
1782 165/300 Write block 51 len 28672 data: 2 2 2 2 2 2 2
1783 166/300 Read block 4952 len 40960
1784 167/300 Write block 2217 len 4096 data: 2
1785 168/300 Read block 715 len 16384
1786 169/300 Read block 1970 len 28672
1787 170/300 Write block 1362 len 4096 data: 2
1788 171/300 Read block 2650 len 16384
1789 172/300 Write block 2511 len 32768 data: 3 3 3 3 4 3 3 3
1790 173/300 Read block 410 len 16384
1791 174/300 Write block 692 len 4096 data: 2
1792 175/300 Read block 4793 len 36864
1793 176/300 Read block 121 len 32768
1794 177/300 Read block 940 len 28672
1795 178/300 Read block 1312 len 36864
1796 179/300 Read block 3841 len 40960
1797 180/300 Write block 1084 len 28672 data: 2 3 3 3 3 3 3
1798 181/300 Read block 2471 len 4096
1799 182/300 Write block 4860 len 20480 data: 3 3 3 3 3
1800 183/300 Read block 3570 len 8192
1801 184/300 Read block 4163 len 32768
1802 185/300 Read block 339 len 24576
1803 186/300 Write block 1007 len 4096 data: 2
1804 187/300 Write block 1650 len 4096 data: 2
1805 188/300 Write block 3421 len 16384 data: 2 2 2 2
1806 189/300 Write block 2747 len 28672 data: 2 2 2 2 2 2 2
1807 190/300 Read block 4983 len 28672
1808 191/300 Write block 3657 len 16384 data: 2 2 2 2
1809 192/300 Write block 2417 len 20480 data: 3 3 2 2 2
1810 193/300 Read block 1692 len 40960
1811 194/300 Read block 1942 len 32768
1812 195/300 Read block 4935 len 8192
1813 196/300 Read block 2935 len 28672
1814 197/300 Read block 4385 len 20480
1815 198/300 Read block 4174 len 16384
1816 199/300 Write block 440 len 16384 data: 3 3 3 3
1817 200/300 Read block 790 len 36864
1818 201/300 Read block 2249 len 8192
1819 202/300 Write block 1078 len 36864 data: 3 2 2 2 2 2 3 4 4
1820 203/300 Read block 3811 len 24576
1821 204/300 Write block 4427 len 8192 data: 2 2
1822 205/300 Read block 615 len 24576
1823 206/300 Write block 2695 len 32768 data: 4 3 3 3 3 3 2 2
1824 207/300 Read block 4271 len 36864
1825 208/300 Read block 2866 len 12288
1826 209/300 Write block 2852 len 24576 data: 3 3 2 2 2 2
1827 210/300 Write block 884 len 16384 data: 2 3 2 2
1828 211/300 Write block 2908 len 20480 data: 2 2 2 2 2
1829 212/300 Write block 2728 len 36864 data: 2 2 2 2 2 2 2 2 3
1830 213/300 Read block 3760 len 8192
1831 214/300 Flush
1832 215/300 Write block 3833 len 8192 data: 2 2
1833 216/300 Read block 4727 len 24576
1834 217/300 Write block 2060 len 8192 data: 2 2
1835 218/300 Write block 3751 len 12288 data: 3 3 2
1836 219/300 Write block 4057 len 32768 data: 3 3 3 2 2 2 2 2
1837 220/300 Write block 207 len 12288 data: 2 2 3
1838 221/300 Write block 1470 len 32768 data: 2 2 2 2 2 2 2 2
1839 222/300 Read block 2677 len 4096
1840 223/300 Write block 2863 len 8192 data: 2 2
1841 224/300 Read block 1734 len 4096
1842 225/300 Write block 793 len 36864 data: 2 2 2 2 2 2 2 2 2
1843 226/300 Read block 2558 len 16384
1844 227/300 Read block 4604 len 24576
1845 228/300 Write block 1085 len 20480 data: 5 5 4 4 4
1846 229/300 Write block 3414 len 8192 data: 3 2
1847 230/300 Read block 1341 len 12288
1848 231/300 Read block 2245 len 32768
1849 232/300 Read block 1347 len 12288
1850 233/300 Read block 4325 len 24576
1851 234/300 Write block 234 len 36864 data: 2 2 2 2 2 2 2 2 2
1852 235/300 Write block 3611 len 12288 data: 2 2 2
1853 236/300 Write block 1075 len 20480 data: 2 2 2 4 3
1854 237/300 Write block 4710 len 12288 data: 2 2 3
1855 238/300 Flush
1856 239/300 Read block 1420 len 40960
1857 240/300 Read block 4986 len 12288
1858 241/300 Read block 3429 len 40960
1859 242/300 Flush
1860 243/300 Write block 3078 len 16384 data: 3 3 2 2
1861 244/300 Write block 4317 len 8192 data: 2 2
1862 245/300 Flush
1863 246/300 Write block 3815 len 28672 data: 2 2 2 2 2 2 2
1864 247/300 Write block 4940 len 40960 data: 4 4 4 4 3 3 3 3 3 3
1865 248/300 Write block 4748 len 24576 data: 2 2 2 3 3 3
1866 249/300 Write block 1080 len 16384 data: 3 3 3 3
1867 250/300 Read block 4967 len 36864
1868 251/300 Write block 3858 len 16384 data: 3 3 3 3
1869 252/300 Write block 3805 len 12288 data: 2 2 2
1870 253/300 Read block 2273 len 20480
1871 254/300 Flush
1872 255/300 Read block 608 len 40960
1873 256/300 Read block 2767 len 4096
1874 257/300 Write block 420 len 24576 data: 2 2 2 2 2 2
1875 258/300 Write block 523 len 4096 data: 3
1876 259/300 Write block 4886 len 20480 data: 4 4 4 2 2
1877 260/300 Write block 4343 len 12288 data: 2 2 3
1878 261/300 Read block 3378 len 32768
1879 262/300 Read block 302 len 24576
1880 263/300 Read block 4068 len 4096
1881 264/300 Write block 2762 len 4096 data: 2
1882 265/300 Write block 3897 len 20480 data: 2 2 2 2 2
1883 266/300 Flush
1884 267/300 Write block 4316 len 28672 data: 2 3 3 2 2 2 2
1885 268/300 Write block 3602 len 32768 data: 2 2 2 2 2 2 2 2
1886 269/300 Read block 3318 len 36864
1887 270/300 Read block 1010 len 36864
1888 271/300 Read block 757 len 24576
1889 272/300 Read block 3369 len 28672
1890 273/300 Read block 678 len 28672
1891 274/300 Read block 4540 len 32768
1892 275/300 Write block 1234 len 20480 data: 2 2 2 2 2
1893 276/300 Read block 4734 len 20480
1894 277/300 Flush
1895 278/300 Read block 3474 len 12288
1896 279/300 Write block 3084 len 8192 data: 2 2
1897 280/300 Read block 2527 len 24576
1898 281/300 Flush
1899 282/300 Read block 4151 len 8192
1900 283/300 Write block 2459 len 4096 data: 2
1901 284/300 Write block 2212 len 16384 data: 2 2 2 2
1902 285/300 Read block 3597 len 32768
1903 286/300 Read block 4726 len 24576
1904 287/300 Flush
1905 288/300 Write block 4782 len 8192 data: 2 2
1906 289/300 Write block 2939 len 12288 data: 2 2 2
1907 290/300 Flush
1908 291/300 Write block 212 len 32768 data: 3 2 3 3 3 4 4 4
1909 292/300 Read block 1340 len 28672
1910 293/300 Read block 4939 len 28672
1911 294/300 Read block 1524 len 36864
1912 295/300 Read block 1118 len 16384
1913 296/300 Write block 2921 len 20480 data: 3 2 2 2 2
1914 297/300 Read block 4332 len 12288
1915 298/300 Read block 897 len 8192
1916 299/300 Write block 4733 len 20480 data: 3 3 3 2 2
1917 300/300 Read block 3120 len 40960
1918 Main task has work to do, go find it
1919 got dsc Start(1)
1920 start 1
1921 Oct 13 04:42:11.903 INFO request completed, latency_us: 285, response_code: 204, uri: /start/cid/1, method: POST, req_id: 83f1e603-5f08-436d-a975-f357a9b99aec, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
1922 Replay: started 1, returned:Ok(())
1923 ----------------------------------------------------------------
1924 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
1925 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
1926 606 Acked 1605 Read 9 Done New Done false
1927 607 Acked 1606 Write 1 Done New Done false
1928 608 Acked 1607 Write 5 Done New Done false
1929 609 Acked 1608 Read 3 Done New Done false
1930 610 Acked 1609 Flush 0 Done New Done false
1931 611 Acked 1610 Read 5 Done New Done false
1932 612 Acked 1611 Read 5 Done New Done false
1933 613 Acked 1612 Read 6 Done New Done false
1934 614 Acked 1613 Read 8 Done New Done false
1935 615 Acked 1614 Read 6 Done New Done false
1936 616 Acked 1615 Write 3 Done New Done false
1937 617 Acked 1616 Write 2 Done New Done false
1938 618 Acked 1617 Write 5 Done New Done false
1939 619 Acked 1618 Flush 0 Done New Done false
1940 620 Acked 1619 Read 5 Done New Done false
1941 621 Acked 1620 Write 10 Done New Done false
1942 622 Acked 1621 Read 8 Done New Done false
1943 623 Acked 1622 Read 1 Done New Done false
1944 624 Acked 1623 Read 5 Done New Done false
1945 625 Acked 1624 Write 9 Done New Done false
1946 626 Acked 1625 Read 5 Done New Done false
1947 627 Acked 1626 Read 1 Done New Done false
1948 628 Acked 1627 Read 1 Done New Done false
1949 629 Acked 1628 Read 2 Done New Done false
1950 630 Acked 1629 Read 3 Done New Done false
1951 631 Acked 1630 Write 2 Done New Done false
1952 632 Acked 1631 Write 2 Done New Done false
1953 633 Acked 1632 Write 4 Done New Done false
1954 634 Acked 1633 Read 2 Done New Done false
1955 635 Acked 1634 Write 9 Done New Done false
1956 636 Acked 1635 Write 8 Done New Done false
1957 637 Acked 1636 Flush 0 Done New Done false
1958 638 Acked 1637 Read 1 Done New Done false
1959 639 Acked 1638 Flush 0 Done New Done false
1960 640 Acked 1639 Read 5 Done New Done false
1961 641 Acked 1640 Read 4 Done New Done false
1962 642 Acked 1641 Write 9 Done New Done false
1963 643 Acked 1642 Write 1 Done New Done false
1964 644 Acked 1643 Write 3 Done New Done false
1965 645 Acked 1644 Read 8 Done New Done false
1966 646 Acked 1645 Flush 0 Done New Done false
1967 647 Acked 1646 Read 6 Done New Done false
1968 648 Acked 1647 Write 7 Done New Done false
1969 649 Acked 1648 Read 5 Done New Done false
1970 650 Acked 1649 Read 2 Done New Done false
1971 651 Acked 1650 Write 2 Done New Done false
1972 652 Acked 1651 Read 3 Done New Done false
1973 653 Acked 1652 Write 2 Done New Done false
1974 654 Acked 1653 Read 3 Done New Done false
1975 655 Acked 1654 Write 9 Done New Done false
1976 656 Acked 1655 Read 5 Done New Done false
1977 657 Acked 1656 Write 3 Done New Done false
1978 658 Acked 1657 Read 10 Done New Done false
1979 659 Acked 1658 Read 6 Done New Done false
1980 660 Acked 1659 Write 4 Done New Done false
1981 661 Acked 1660 Read 1 Done New Done false
1982 662 Acked 1661 Read 8 Done New Done false
1983 663 Acked 1662 Read 3 Done New Done false
1984 664 Acked 1663 Read 1 Done New Done false
1985 665 Acked 1664 Write 5 Done New Done false
1986 666 Acked 1665 Write 8 Done New Done false
1987 667 Acked 1666 Read 5 Done New Done false
1988 668 Acked 1667 Read 5 Done New Done false
1989 669 Acked 1668 Write 1 Done New Done false
1990 670 Acked 1669 Write 10 Done New Done false
1991 671 Acked 1670 Write 1 Done New Done false
1992 672 Acked 1671 Flush 0 Done New Done false
1993 673 Acked 1672 Write 9 Done New Done false
1994 674 Acked 1673 Write 10 Done New Done false
1995 675 Acked 1674 Read 3 Done New Done false
1996 676 Acked 1675 Read 7 Done New Done false
1997 677 Acked 1676 Write 5 Done New Done false
1998 678 Acked 1677 Flush 0 Done New Done false
1999 679 Acked 1678 Read 5 Done New Done false
2000 680 Acked 1679 Write 4 Done New Done false
2001 681 Acked 1680 Read 7 Done New Done false
2002 682 Acked 1681 Read 1 Done New Done false
2003 683 Acked 1682 Read 8 Done New Done false
2004 684 Acked 1683 Write 8 Done New Done false
2005 685 Acked 1684 Write 1 Done New Done false
2006 686 Acked 1685 Read 6 Done New Done false
2007 687 Acked 1686 Read 5 Done New Done false
2008 688 Acked 1687 Read 9 Done New Done false
2009 689 Acked 1688 Write 3 Done New Done false
2010 690 Acked 1689 Read 2 Done New Done false
2011 691 Acked 1690 Write 10 Done New Done false
2012 692 Acked 1691 Read 1 Done New Done false
2013 693 Acked 1692 Read 5 Done New Done false
2014 694 Acked 1693 Write 4 Done New Done false
2015 695 Acked 1694 Write 3 Done New Done false
2016 696 Acked 1695 Read 7 Done New Done false
2017 697 Acked 1696 Flush 0 Done New Done false
2018 698 Acked 1697 Read 4 Done New Done false
2019 699 Acked 1698 Read 1 Done New Done false
2020 700 Acked 1699 Write 3 Done New Done false
2021 701 Acked 1700 Read 2 Done New Done false
2022 702 Acked 1701 Read 5 Done New Done false
2023 703 Acked 1702 Write 7 Done New Done false
2024 704 Acked 1703 Flush 0 Done New Done false
2025 705 Acked 1704 Read 8 Done New Done false
2026 706 Acked 1705 Write 1 Done New Done false
2027 707 Acked 1706 Write 9 Done New Done false
2028 708 Acked 1707 Read 4 Done New Done false
2029 709 Acked 1708 Read 7 Done New Done false
2030 710 Acked 1709 Flush 0 Done New Done false
2031 711 Acked 1710 Write 3 Done New Done false
2032 712 Acked 1711 Read 7 Done New Done false
2033 713 Acked 1712 Read 5 Done New Done false
2034 714 Acked 1713 Read 7 Done New Done false
2035 715 Acked 1714 Read 6 Done New Done false
2036 716 Acked 1715 Read 5 Done New Done false
2037 717 Acked 1716 Write 1 Done New Done false
2038 718 Acked 1717 Read 3 Done New Done false
2039 719 Acked 1718 Flush 0 Done New Done false
2040 720 Acked 1719 Write 4 Done New Done false
2041 721 Acked 1720 Write 4 Done New Done false
2042 722 Acked 1721 Read 10 Done New Done false
2043 723 Acked 1722 Read 2 Done New Done false
2044 724 Acked 1723 Read 3 Done New Done false
2045 725 Acked 1724 Write 10 Done New Done false
2046 726 Acked 1725 Write 4 Done New Done false
2047 727 Acked 1726 Write 4 Done New Done false
2048 728 Acked 1727 Read 7 Done New Done false
2049 729 Acked 1728 Write 5 Done New Done false
2050 730 Acked 1729 Read 8 Done New Done false
2051 731 Acked 1730 Write 8 Done New Done false
2052 732 Acked 1731 Read 1 Done New Done false
2053 733 Acked 1732 Read 3 Done New Done false
2054 734 Acked 1733 Read 5 Done New Done false
2055 735 Acked 1734 Read 2 Done New Done false
2056 736 Acked 1735 Read 6 Done New Done false
2057 737 Acked 1736 Write 5 Done New Done false
2058 738 Acked 1737 Flush 0 Done New Done false
2059 739 Acked 1738 Write 9 Done New Done false
2060 740 Acked 1739 Write 10 Done New Done false
2061 741 Acked 1740 Read 7 Done New Done false
2062 742 Acked 1741 Write 3 Done New Done false
2063 743 Acked 1742 Write 6 Done New Done false
2064 744 Acked 1743 Write 3 Done New Done false
2065 745 Acked 1744 Flush 0 Done New Done false
2066 746 Acked 1745 Flush 0 Done New Done false
2067 747 Acked 1746 Read 4 Done New Done false
2068 748 Acked 1747 Write 7 Done New Done false
2069 749 Acked 1748 Flush 0 Done New Done false
2070 750 Acked 1749 Flush 0 Done New Done false
2071 751 Acked 1750 Read 8 Done New Done false
2072 752 Acked 1751 Read 6 Done New Done false
2073 753 Acked 1752 Write 7 Done New Done false
2074 754 Acked 1753 Flush 0 Done New Done false
2075 755 Acked 1754 Write 8 Done New Done false
2076 756 Acked 1755 Read 2 Done New Done false
2077 757 Acked 1756 Write 7 Done New Done false
2078 758 Acked 1757 Read 2 Done New Done false
2079 759 Acked 1758 Read 2 Done New Done false
2080 760 Acked 1759 Write 3 Done New Done false
2081 761 Acked 1760 Read 7 Done New Done false
2082 762 Acked 1761 Read 1 Done New Done false
2083 763 Acked 1762 Read 9 Done New Done false
2084 764 Acked 1763 Flush 0 Done New Done false
2085 765 Acked 1764 Write 5 Done New Done false
2086 766 Acked 1765 Write 10 Done New Done false
2087 767 Acked 1766 Read 2 Done New Done false
2088 768 Acked 1767 Read 8 Done New Done false
2089 769 Acked 1768 Read 10 Done New Done false
2090 770 Acked 1769 Write 7 Done New Done false
2091 771 Acked 1770 Write 7 Done New Done false
2092 772 Acked 1771 Read 10 Done New Done false
2093 773 Acked 1772 Write 1 Done New Done false
2094 774 Acked 1773 Read 4 Done New Done false
2095 775 Acked 1774 Read 7 Done New Done false
2096 776 Acked 1775 Write 1 Done New Done false
2097 777 Acked 1776 Read 4 Done New Done false
2098 778 Acked 1777 Write 8 Done New Done false
2099 779 Acked 1778 Read 4 Done New Done false
2100 780 Acked 1779 Write 1 Done New Done false
2101 781 Acked 1780 Read 9 Done New Done false
2102 782 Acked 1781 Read 8 Done New Done false
2103 783 Acked 1782 Read 7 Done New Done false
2104 784 Acked 1783 Read 9 Done New Done false
2105 785 Acked 1784 Read 10 Done New Done false
2106 786 Acked 1785 Write 7 Done New Done false
2107 787 Acked 1786 Read 1 Done New Done false
2108 788 Acked 1787 Write 5 Done New Done false
2109 789 Acked 1788 Read 2 Done New Done false
2110 790 Acked 1789 Read 8 Done New Done false
2111 791 Acked 1790 Read 6 Done New Done false
2112 792 Acked 1791 Write 1 Done New Done false
2113 793 Acked 1792 Write 1 Done New Done false
2114 794 Acked 1793 Write 4 Done New Done false
2115 795 Acked 1794 Write 7 Done New Done false
2116 796 Acked 1795 Read 7 Done New Done false
2117 797 Acked 1796 Write 4 Done New Done false
2118 798 Acked 1797 Write 5 Done New Done false
2119 799 Acked 1798 Read 10 Done New Done false
2120 800 Acked 1799 Read 8 Done New Done false
2121 801 Acked 1800 Read 2 Done New Done false
2122 802 Acked 1801 Read 7 Done New Done false
2123 803 Acked 1802 Read 5 Done New Done false
2124 804 Acked 1803 Read 4 Done New Done false
2125 805 Acked 1804 Write 4 Done New Done false
2126 806 Acked 1805 Read 9 Done New Done false
2127 807 Acked 1806 Read 2 Done New Done false
2128 808 Acked 1807 Write 9 Done New Done false
2129 809 Acked 1808 Read 6 Done New Done false
2130 810 Acked 1809 Write 2 Done New Done false
2131 811 Acked 1810 Read 6 Done New Done false
2132 812 Acked 1811 Write 8 Done New Done false
2133 813 Acked 1812 Read 9 Done New Done false
2134 814 Acked 1813 Read 3 Done New Done false
2135 815 Acked 1814 Write 6 Done New Done false
2136 816 Acked 1815 Write 4 Done New Done false
2137 817 Acked 1816 Write 5 Done New Done false
2138 818 Acked 1817 Write 9 Done New Done false
2139 819 Acked 1818 Read 2 Done New Done false
2140 820 Acked 1819 Flush 0 Done New Done false
2141 821 Acked 1820 Flush 0 Done New Done false
2142 822 Acked 1821 Write 2 Done New Done false
2143 823 Acked 1822 Read 6 Done New Done false
2144 824 Acked 1823 Write 2 Done New Done false
2145 825 Acked 1824 Write 3 Done New Done false
2146 826 Acked 1825 Write 8 Done New Done false
2147 827 Acked 1826 Write 3 Done New Done false
2148 828 Acked 1827 Write 8 Done New Done false
2149 829 Acked 1828 Read 1 Done New Done false
2150 830 Acked 1829 Write 2 Done New Done false
2151 831 Acked 1830 Read 1 Done New Done false
2152 832 Acked 1831 Write 9 Done New Done false
2153 833 Acked 1832 Read 4 Done New Done false
2154 834 Acked 1833 Read 6 Done New Done false
2155 835 Acked 1834 Write 5 Done New Done false
2156 836 Acked 1835 Write 2 Done New Done false
2157 837 Acked 1836 Read 3 Done New Done false
2158 838 Acked 1837 Read 8 Done New Done false
2159 839 Acked 1838 Read 3 Done New Done false
2160 840 Acked 1839 Read 6 Done New Done false
2161 841 Acked 1840 Write 9 Done New Done false
2162 842 Acked 1841 Write 3 Done New Done false
2163 843 Acked 1842 Write 5 Done New Done false
2164 844 Acked 1843 Write 3 Done New Done false
2165 845 Acked 1844 Flush 0 Done New Done false
2166 846 Acked 1845 Read 10 Done New Done false
2167 847 Acked 1846 Read 3 Done New Done false
2168 848 Acked 1847 Read 10 Done New Done false
2169 849 Acked 1848 Flush 0 Done New Done false
2170 850 Acked 1849 Write 4 Done New Done false
2171 851 Acked 1850 Write 2 Done New Done false
2172 852 Acked 1851 Flush 0 Done New Done false
2173 853 Acked 1852 Write 7 Done New Done false
2174 854 Acked 1853 Write 10 Done New Done false
2175 855 Acked 1854 Write 6 Done New Done false
2176 856 Acked 1855 Write 4 Done New Done false
2177 857 Acked 1856 Read 9 Done New Done false
2178 858 Acked 1857 Write 4 Done New Done false
2179 859 Acked 1858 Write 3 Done New Done false
2180 860 Acked 1859 Read 5 Done New Done false
2181 861 Acked 1860 Flush 0 Done New Done false
2182 862 Acked 1861 Read 10 Done New Done false
2183 863 Acked 1862 Read 1 Done New Done false
2184 864 Acked 1863 Write 6 Done New Done false
2185 865 Acked 1864 Write 1 Done New Done false
2186 866 Acked 1865 Write 5 Done New Done false
2187 867 Acked 1866 Write 3 Done New Done false
2188 868 Acked 1867 Read 8 Done New Done false
2189 869 Acked 1868 Read 6 Done New Done false
2190 870 Acked 1869 Read 1 Done New Done false
2191 871 Acked 1870 Write 1 Done New Done false
2192 872 Acked 1871 Write 5 Done New Done false
2193 873 Acked 1872 Flush 0 Done New Done false
2194 874 Acked 1873 Write 7 Done New Done false
2195 875 Acked 1874 Write 8 Done New Done false
2196 876 Acked 1875 Read 9 Done New Done false
2197 877 Acked 1876 Read 9 Done New Done false
2198 878 Acked 1877 Read 6 Done New Done false
2199 879 Acked 1878 Read 7 Done New Done false
2200 880 Acked 1879 Read 7 Done New Done false
2201 881 Acked 1880 Read 8 Done New Done false
2202 882 Acked 1881 Write 5 Done New Done false
2203 883 Acked 1882 Read 5 Done New Done false
2204 884 Acked 1883 Flush 0 Done New Done false
2205 885 Acked 1884 Read 3 Done New Done false
2206 886 Acked 1885 Write 2 Done New Done false
2207 887 Acked 1886 Read 6 Done New Done false
2208 888 Acked 1887 Flush 0 Done New Done false
2209 889 Acked 1888 Read 2 Done New Done false
2210 890 Acked 1889 Write 1 Done New Done false
2211 891 Acked 1890 Write 4 Done New Done false
2212 892 Acked 1891 Read 8 Done New Done false
2213 893 Acked 1892 Read 6 Done New Done false
2214 894 Acked 1893 Flush 0 Done New Done false
2215 895 Acked 1894 Write 2 Done New Done false
2216 896 Acked 1895 Write 3 Done New Done false
2217 897 Acked 1896 Flush 0 Done New Done false
2218 898 Acked 1897 Write 8 Done New Done false
2219 899 Acked 1898 Read 7 Done New Done false
2220 900 Acked 1899 Read 7 Done New Done false
2221 901 Acked 1900 Read 9 Done New Done false
2222 902 Acked 1901 Read 4 Done New Done false
2223 903 Acked 1902 Write 5 Done New Done false
2224 904 Acked 1903 Read 3 Done New Done false
2225 905 Acked 1904 Read 2 Done New Done false
2226 906 Acked 1905 Write 5 Done New Done false
2227 907 Acked 1906 Read 10 Done New Done false
2228 STATES DS:0 DS:1 DS:2 TOTAL
2229 New 0 302 0 302
2230 Sent 0 0 0 0
2231 Done 302 0 302 604
2232 Skipped 0 0 0 0
2233 Error 0 0 0 0
2234 Last Flush: 1896 1604 1896
2235 Downstairs last five completed: 1604 1603 1602 1601 1600 1599
2236 Upstairs last five completed: 907 906 905 904 903 902
2237 CLIENT: Up:0 ds:302 act:2
2238 [8820] got start action
2239 [8820] I am going to restart
2240 Starting downstairs at port 8820
2241 Make output file at "/tmp/dsc/downstairs-8820.txt"
2242 [8820][1] reports Starting
2243 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1179)
2244 [8820][1] reports Running
2245 ----------------------------------------------------------------
2246 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
2247 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
2248 606 Acked 1605 Read 9 Done New Done false
2249 607 Acked 1606 Write 1 Done New Done false
2250 608 Acked 1607 Write 5 Done New Done false
2251 609 Acked 1608 Read 3 Done New Done false
2252 610 Acked 1609 Flush 0 Done New Done false
2253 611 Acked 1610 Read 5 Done New Done false
2254 612 Acked 1611 Read 5 Done New Done false
2255 613 Acked 1612 Read 6 Done New Done false
2256 614 Acked 1613 Read 8 Done New Done false
2257 615 Acked 1614 Read 6 Done New Done false
2258 616 Acked 1615 Write 3 Done New Done false
2259 617 Acked 1616 Write 2 Done New Done false
2260 618 Acked 1617 Write 5 Done New Done false
2261 619 Acked 1618 Flush 0 Done New Done false
2262 620 Acked 1619 Read 5 Done New Done false
2263 621 Acked 1620 Write 10 Done New Done false
2264 622 Acked 1621 Read 8 Done New Done false
2265 623 Acked 1622 Read 1 Done New Done false
2266 624 Acked 1623 Read 5 Done New Done false
2267 625 Acked 1624 Write 9 Done New Done false
2268 626 Acked 1625 Read 5 Done New Done false
2269 627 Acked 1626 Read 1 Done New Done false
2270 628 Acked 1627 Read 1 Done New Done false
2271 629 Acked 1628 Read 2 Done New Done false
2272 630 Acked 1629 Read 3 Done New Done false
2273 631 Acked 1630 Write 2 Done New Done false
2274 632 Acked 1631 Write 2 Done New Done false
2275 633 Acked 1632 Write 4 Done New Done false
2276 634 Acked 1633 Read 2 Done New Done false
2277 635 Acked 1634 Write 9 Done New Done false
2278 636 Acked 1635 Write 8 Done New Done false
2279 637 Acked 1636 Flush 0 Done New Done false
2280 638 Acked 1637 Read 1 Done New Done false
2281 639 Acked 1638 Flush 0 Done New Done false
2282 640 Acked 1639 Read 5 Done New Done false
2283 641 Acked 1640 Read 4 Done New Done false
2284 642 Acked 1641 Write 9 Done New Done false
2285 643 Acked 1642 Write 1 Done New Done false
2286 644 Acked 1643 Write 3 Done New Done false
2287 645 Acked 1644 Read 8 Done New Done false
2288 646 Acked 1645 Flush 0 Done New Done false
2289 647 Acked 1646 Read 6 Done New Done false
2290 648 Acked 1647 Write 7 Done New Done false
2291 649 Acked 1648 Read 5 Done New Done false
2292 650 Acked 1649 Read 2 Done New Done false
2293 651 Acked 1650 Write 2 Done New Done false
2294 652 Acked 1651 Read 3 Done New Done false
2295 653 Acked 1652 Write 2 Done New Done false
2296 654 Acked 1653 Read 3 Done New Done false
2297 655 Acked 1654 Write 9 Done New Done false
2298 656 Acked 1655 Read 5 Done New Done false
2299 657 Acked 1656 Write 3 Done New Done false
2300 658 Acked 1657 Read 10 Done New Done false
2301 659 Acked 1658 Read 6 Done New Done false
2302 660 Acked 1659 Write 4 Done New Done false
2303 661 Acked 1660 Read 1 Done New Done false
2304 662 Acked 1661 Read 8 Done New Done false
2305 663 Acked 1662 Read 3 Done New Done false
2306 664 Acked 1663 Read 1 Done New Done false
2307 665 Acked 1664 Write 5 Done New Done false
2308 666 Acked 1665 Write 8 Done New Done false
2309 667 Acked 1666 Read 5 Done New Done false
2310 668 Acked 1667 Read 5 Done New Done false
2311 669 Acked 1668 Write 1 Done New Done false
2312 670 Acked 1669 Write 10 Done New Done false
2313 671 Acked 1670 Write 1 Done New Done false
2314 672 Acked 1671 Flush 0 Done New Done false
2315 673 Acked 1672 Write 9 Done New Done false
2316 674 Acked 1673 Write 10 Done New Done false
2317 675 Acked 1674 Read 3 Done New Done false
2318 676 Acked 1675 Read 7 Done New Done false
2319 677 Acked 1676 Write 5 Done New Done false
2320 678 Acked 1677 Flush 0 Done New Done false
2321 679 Acked 1678 Read 5 Done New Done false
2322 680 Acked 1679 Write 4 Done New Done false
2323 681 Acked 1680 Read 7 Done New Done false
2324 682 Acked 1681 Read 1 Done New Done false
2325 683 Acked 1682 Read 8 Done New Done false
2326 684 Acked 1683 Write 8 Done New Done false
2327 685 Acked 1684 Write 1 Done New Done false
2328 686 Acked 1685 Read 6 Done New Done false
2329 687 Acked 1686 Read 5 Done New Done false
2330 688 Acked 1687 Read 9 Done New Done false
2331 689 Acked 1688 Write 3 Done New Done false
2332 690 Acked 1689 Read 2 Done New Done false
2333 691 Acked 1690 Write 10 Done New Done false
2334 692 Acked 1691 Read 1 Done New Done false
2335 693 Acked 1692 Read 5 Done New Done false
2336 694 Acked 1693 Write 4 Done New Done false
2337 695 Acked 1694 Write 3 Done New Done false
2338 696 Acked 1695 Read 7 Done New Done false
2339 697 Acked 1696 Flush 0 Done New Done false
2340 698 Acked 1697 Read 4 Done New Done false
2341 699 Acked 1698 Read 1 Done New Done false
2342 700 Acked 1699 Write 3 Done New Done false
2343 701 Acked 1700 Read 2 Done New Done false
2344 702 Acked 1701 Read 5 Done New Done false
2345 703 Acked 1702 Write 7 Done New Done false
2346 704 Acked 1703 Flush 0 Done New Done false
2347 705 Acked 1704 Read 8 Done New Done false
2348 706 Acked 1705 Write 1 Done New Done false
2349 707 Acked 1706 Write 9 Done New Done false
2350 708 Acked 1707 Read 4 Done New Done false
2351 709 Acked 1708 Read 7 Done New Done false
2352 710 Acked 1709 Flush 0 Done New Done false
2353 711 Acked 1710 Write 3 Done New Done false
2354 712 Acked 1711 Read 7 Done New Done false
2355 713 Acked 1712 Read 5 Done New Done false
2356 714 Acked 1713 Read 7 Done New Done false
2357 715 Acked 1714 Read 6 Done New Done false
2358 716 Acked 1715 Read 5 Done New Done false
2359 717 Acked 1716 Write 1 Done New Done false
2360 718 Acked 1717 Read 3 Done New Done false
2361 719 Acked 1718 Flush 0 Done New Done false
2362 720 Acked 1719 Write 4 Done New Done false
2363 721 Acked 1720 Write 4 Done New Done false
2364 722 Acked 1721 Read 10 Done New Done false
2365 723 Acked 1722 Read 2 Done New Done false
2366 724 Acked 1723 Read 3 Done New Done false
2367 725 Acked 1724 Write 10 Done New Done false
2368 726 Acked 1725 Write 4 Done New Done false
2369 727 Acked 1726 Write 4 Done New Done false
2370 728 Acked 1727 Read 7 Done New Done false
2371 729 Acked 1728 Write 5 Done New Done false
2372 730 Acked 1729 Read 8 Done New Done false
2373 731 Acked 1730 Write 8 Done New Done false
2374 732 Acked 1731 Read 1 Done New Done false
2375 733 Acked 1732 Read 3 Done New Done false
2376 734 Acked 1733 Read 5 Done New Done false
2377 735 Acked 1734 Read 2 Done New Done false
2378 736 Acked 1735 Read 6 Done New Done false
2379 737 Acked 1736 Write 5 Done New Done false
2380 738 Acked 1737 Flush 0 Done New Done false
2381 739 Acked 1738 Write 9 Done New Done false
2382 740 Acked 1739 Write 10 Done New Done false
2383 741 Acked 1740 Read 7 Done New Done false
2384 742 Acked 1741 Write 3 Done New Done false
2385 743 Acked 1742 Write 6 Done New Done false
2386 744 Acked 1743 Write 3 Done New Done false
2387 745 Acked 1744 Flush 0 Done New Done false
2388 746 Acked 1745 Flush 0 Done New Done false
2389 747 Acked 1746 Read 4 Done New Done false
2390 748 Acked 1747 Write 7 Done New Done false
2391 749 Acked 1748 Flush 0 Done New Done false
2392 750 Acked 1749 Flush 0 Done New Done false
2393 751 Acked 1750 Read 8 Done New Done false
2394 752 Acked 1751 Read 6 Done New Done false
2395 753 Acked 1752 Write 7 Done New Done false
2396 754 Acked 1753 Flush 0 Done New Done false
2397 755 Acked 1754 Write 8 Done New Done false
2398 756 Acked 1755 Read 2 Done New Done false
2399 757 Acked 1756 Write 7 Done New Done false
2400 758 Acked 1757 Read 2 Done New Done false
2401 759 Acked 1758 Read 2 Done New Done false
2402 760 Acked 1759 Write 3 Done New Done false
2403 761 Acked 1760 Read 7 Done New Done false
2404 762 Acked 1761 Read 1 Done New Done false
2405 763 Acked 1762 Read 9 Done New Done false
2406 764 Acked 1763 Flush 0 Done New Done false
2407 765 Acked 1764 Write 5 Done New Done false
2408 766 Acked 1765 Write 10 Done New Done false
2409 767 Acked 1766 Read 2 Done New Done false
2410 768 Acked 1767 Read 8 Done New Done false
2411 769 Acked 1768 Read 10 Done New Done false
2412 770 Acked 1769 Write 7 Done New Done false
2413 771 Acked 1770 Write 7 Done New Done false
2414 772 Acked 1771 Read 10 Done New Done false
2415 773 Acked 1772 Write 1 Done New Done false
2416 774 Acked 1773 Read 4 Done New Done false
2417 775 Acked 1774 Read 7 Done New Done false
2418 776 Acked 1775 Write 1 Done New Done false
2419 777 Acked 1776 Read 4 Done New Done false
2420 778 Acked 1777 Write 8 Done New Done false
2421 779 Acked 1778 Read 4 Done New Done false
2422 780 Acked 1779 Write 1 Done New Done false
2423 781 Acked 1780 Read 9 Done New Done false
2424 782 Acked 1781 Read 8 Done New Done false
2425 783 Acked 1782 Read 7 Done New Done false
2426 784 Acked 1783 Read 9 Done New Done false
2427 785 Acked 1784 Read 10 Done New Done false
2428 786 Acked 1785 Write 7 Done New Done false
2429 787 Acked 1786 Read 1 Done New Done false
2430 788 Acked 1787 Write 5 Done New Done false
2431 789 Acked 1788 Read 2 Done New Done false
2432 790 Acked 1789 Read 8 Done New Done false
2433 791 Acked 1790 Read 6 Done New Done false
2434 792 Acked 1791 Write 1 Done New Done false
2435 793 Acked 1792 Write 1 Done New Done false
2436 794 Acked 1793 Write 4 Done New Done false
2437 795 Acked 1794 Write 7 Done New Done false
2438 796 Acked 1795 Read 7 Done New Done false
2439 797 Acked 1796 Write 4 Done New Done false
2440 798 Acked 1797 Write 5 Done New Done false
2441 799 Acked 1798 Read 10 Done New Done false
2442 800 Acked 1799 Read 8 Done New Done false
2443 801 Acked 1800 Read 2 Done New Done false
2444 802 Acked 1801 Read 7 Done New Done false
2445 803 Acked 1802 Read 5 Done New Done false
2446 804 Acked 1803 Read 4 Done New Done false
2447 805 Acked 1804 Write 4 Done New Done false
2448 806 Acked 1805 Read 9 Done New Done false
2449 807 Acked 1806 Read 2 Done New Done false
2450 808 Acked 1807 Write 9 Done New Done false
2451 809 Acked 1808 Read 6 Done New Done false
2452 810 Acked 1809 Write 2 Done New Done false
2453 811 Acked 1810 Read 6 Done New Done false
2454 812 Acked 1811 Write 8 Done New Done false
2455 813 Acked 1812 Read 9 Done New Done false
2456 814 Acked 1813 Read 3 Done New Done false
2457 815 Acked 1814 Write 6 Done New Done false
2458 816 Acked 1815 Write 4 Done New Done false
2459 817 Acked 1816 Write 5 Done New Done false
2460 818 Acked 1817 Write 9 Done New Done false
2461 819 Acked 1818 Read 2 Done New Done false
2462 820 Acked 1819 Flush 0 Done New Done false
2463 821 Acked 1820 Flush 0 Done New Done false
2464 822 Acked 1821 Write 2 Done New Done false
2465 823 Acked 1822 Read 6 Done New Done false
2466 824 Acked 1823 Write 2 Done New Done false
2467 825 Acked 1824 Write 3 Done New Done false
2468 826 Acked 1825 Write 8 Done New Done false
2469 827 Acked 1826 Write 3 Done New Done false
2470 828 Acked 1827 Write 8 Done New Done false
2471 829 Acked 1828 Read 1 Done New Done false
2472 830 Acked 1829 Write 2 Done New Done false
2473 831 Acked 1830 Read 1 Done New Done false
2474 832 Acked 1831 Write 9 Done New Done false
2475 833 Acked 1832 Read 4 Done New Done false
2476 834 Acked 1833 Read 6 Done New Done false
2477 835 Acked 1834 Write 5 Done New Done false
2478 836 Acked 1835 Write 2 Done New Done false
2479 837 Acked 1836 Read 3 Done New Done false
2480 838 Acked 1837 Read 8 Done New Done false
2481 839 Acked 1838 Read 3 Done New Done false
2482 840 Acked 1839 Read 6 Done New Done false
2483 841 Acked 1840 Write 9 Done New Done false
2484 842 Acked 1841 Write 3 Done New Done false
2485 843 Acked 1842 Write 5 Done New Done false
2486 844 Acked 1843 Write 3 Done New Done false
2487 845 Acked 1844 Flush 0 Done New Done false
2488 846 Acked 1845 Read 10 Done New Done false
2489 847 Acked 1846 Read 3 Done New Done false
2490 848 Acked 1847 Read 10 Done New Done false
2491 849 Acked 1848 Flush 0 Done New Done false
2492 850 Acked 1849 Write 4 Done New Done false
2493 851 Acked 1850 Write 2 Done New Done false
2494 852 Acked 1851 Flush 0 Done New Done false
2495 853 Acked 1852 Write 7 Done New Done false
2496 854 Acked 1853 Write 10 Done New Done false
2497 855 Acked 1854 Write 6 Done New Done false
2498 856 Acked 1855 Write 4 Done New Done false
2499 857 Acked 1856 Read 9 Done New Done false
2500 858 Acked 1857 Write 4 Done New Done false
2501 859 Acked 1858 Write 3 Done New Done false
2502 860 Acked 1859 Read 5 Done New Done false
2503 861 Acked 1860 Flush 0 Done New Done false
2504 862 Acked 1861 Read 10 Done New Done false
2505 863 Acked 1862 Read 1 Done New Done false
2506 864 Acked 1863 Write 6 Done New Done false
2507 865 Acked 1864 Write 1 Done New Done false
2508 866 Acked 1865 Write 5 Done New Done false
2509 867 Acked 1866 Write 3 Done New Done false
2510 868 Acked 1867 Read 8 Done New Done false
2511 869 Acked 1868 Read 6 Done New Done false
2512 870 Acked 1869 Read 1 Done New Done false
2513 871 Acked 1870 Write 1 Done New Done false
2514 872 Acked 1871 Write 5 Done New Done false
2515 873 Acked 1872 Flush 0 Done New Done false
2516 874 Acked 1873 Write 7 Done New Done false
2517 875 Acked 1874 Write 8 Done New Done false
2518 876 Acked 1875 Read 9 Done New Done false
2519 877 Acked 1876 Read 9 Done New Done false
2520 878 Acked 1877 Read 6 Done New Done false
2521 879 Acked 1878 Read 7 Done New Done false
2522 880 Acked 1879 Read 7 Done New Done false
2523 881 Acked 1880 Read 8 Done New Done false
2524 882 Acked 1881 Write 5 Done New Done false
2525 883 Acked 1882 Read 5 Done New Done false
2526 884 Acked 1883 Flush 0 Done New Done false
2527 885 Acked 1884 Read 3 Done New Done false
2528 886 Acked 1885 Write 2 Done New Done false
2529 887 Acked 1886 Read 6 Done New Done false
2530 888 Acked 1887 Flush 0 Done New Done false
2531 889 Acked 1888 Read 2 Done New Done false
2532 890 Acked 1889 Write 1 Done New Done false
2533 891 Acked 1890 Write 4 Done New Done false
2534 892 Acked 1891 Read 8 Done New Done false
2535 893 Acked 1892 Read 6 Done New Done false
2536 894 Acked 1893 Flush 0 Done New Done false
2537 895 Acked 1894 Write 2 Done New Done false
2538 896 Acked 1895 Write 3 Done New Done false
2539 897 Acked 1896 Flush 0 Done New Done false
2540 898 Acked 1897 Write 8 Done New Done false
2541 899 Acked 1898 Read 7 Done New Done false
2542 900 Acked 1899 Read 7 Done New Done false
2543 901 Acked 1900 Read 9 Done New Done false
2544 902 Acked 1901 Read 4 Done New Done false
2545 903 Acked 1902 Write 5 Done New Done false
2546 904 Acked 1903 Read 3 Done New Done false
2547 905 Acked 1904 Read 2 Done New Done false
2548 906 Acked 1905 Write 5 Done New Done false
2549 907 Acked 1906 Read 10 Done New Done false
2550 908 Acked 1907 Flush 0 Done New Done false
2551 STATES DS:0 DS:1 DS:2 TOTAL
2552 New 0 303 0 303
2553 Sent 0 0 0 0
2554 Done 303 0 303 606
2555 Skipped 0 0 0 0
2556 Error 0 0 0 0
2557 Last Flush: 1907 1604 1907
2558 Downstairs last five completed: 1604 1603 1602 1601 1600 1599
2559 Upstairs last five completed: 908 907 906 905 904 903
2560 CLIENT: Up:0 ds:303 act:2
25612023-10-13T04:42:16.060ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 1
25622023-10-13T04:42:16.060ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state Offline
25632023-10-13T04:42:16.060ZINFOcrucible: [1] client re-new 303 jobs since flush 1604 = downstairs
25642023-10-13T04:42:16.061ZINFOcrucible: [1] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
25652023-10-13T04:42:16.062ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
25662023-10-13T04:42:16.062ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
25672023-10-13T04:42:16.062ZINFOcrucible: Returning client:1 UUID:12345678-0000-0000-0000-000000008820 matches
25682023-10-13T04:42:16.062ZINFOcrucible: [1] send last flush ID to this DS: 1604
25692023-10-13T04:42:16.062ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
25702023-10-13T04:42:16.062ZINFOcrucible: [1] Replied this last flush ID: 1604
25712023-10-13T04:42:16.062ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Offline Active ds_transition to Replay
25722023-10-13T04:42:16.062ZINFOcrucible: [1] Transition from Offline to Replay
25732023-10-13T04:42:16.062ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
25742023-10-13T04:42:16.062ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Replay Active ds_transition to Active
25752023-10-13T04:42:16.062ZINFOcrucible: [1] Transition from Replay to Active
25762023-10-13T04:42:16.062ZINFOcrucible: [1] Starts cmd_loop
25772023-10-13T04:42:16.062ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
25782023-10-13T04:42:16.062ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
25792023-10-13T04:42:16.062ZINFOcrucible: Set check for repair
25802023-10-13T04:42:17.064ZINFOcrucible: Checking if live repair is needed
25812023-10-13T04:42:17.064ZINFOcrucible: No Live Repair required at this time
25822023-10-13T04:42:17.064ZWARNcrucible: [1] flow control end
2583 ----------------------------------------------------------------
2584 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
2585 Downstairs last five completed: 1907 1906 1905 1904 1903 1902
2586 Upstairs last five completed: 908 907 906 905 904 903
2587 CLIENT: Up:0 ds:0 act:3
2588 Replay: All jobs finished, all DS active.
2589 Main task has work to do, go find it
2590 got dsc Stop(1)
2591 stop 1
2592 Oct 13 04:42:19.912 INFO request completed, latency_us: 457, response_code: 204, uri: /stop/cid/1, method: [8820] Got stop action so:false kr:false
2593 POST, req_id: 7ab5c91b-7864-425e-a91c-3117452a3aff, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
2594 Oct 13 04:42:19.913 INFO request completed, latency_us: 193, response_code: 200, uri: /state/cid/1, method: GET, req_id: e72ed405-8449-4986-af41-be3d7863b385, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
25952023-10-13T04:42:19.913ZWARNcrucible: [1] downstairs disconnected
25962023-10-13T04:42:19.913ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
25972023-10-13T04:42:19.913ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8820 closed looper = 1
25982023-10-13T04:42:19.913ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:false
25992023-10-13T04:42:19.913ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
26002023-10-13T04:42:19.914ZWARNcrucible: [1] pm_task rx.recv() is None
26012023-10-13T04:42:19.914ZINFOcrucible: [1] 127.0.0.1:8820 task reports offline
2602 [8820] Exited with: ExitStatus(unix_wait_status(9))
2603 [8820][1] reports Exit
2604 Oct 13 04:42:23.916 INFO request completed, latency_us: 250, response_code: 200, uri: /state/cid/1, method: GET, req_id: 4cabb988-47ad-4a94-8729-c86a97cda20e, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
2605 001/300 Read block 502 len 40960
26062023-10-13T04:42:23.917ZINFOcrucible: [1] connecting to 127.0.0.1:8820 looper = 1
2607 002/300 Write block 2526 len 32768 data: 3 3 3 3 3 3 2 2
2608 003/300 Read block 846 len 32768
2609 004/300 Read block 2142 len 36864
2610 005/300 Read block 1316 len 16384
2611 006/300 Write block 169 len 24576 data: 2 2 3 3 3 3
2612 007/300 Write block 3711 len 20480 data: 3 3 3 3 3
2613 008/300 Read block 3736 len 24576
2614 009/300 Flush
2615 010/300 Read block 670 len 16384
2616 011/300 Write block 941 len 16384 data: 2 2 2 2
2617 012/300 Read block 3648 len 40960
2618 013/300 Read block 2878 len 4096
2619 014/300 Write block 3319 len 16384 data: 3 3 3 3
2620 015/300 Write block 4519 len 12288 data: 2 2 2
2621 016/300 Flush
2622 017/300 Flush
2623 018/300 Write block 2091 len 4096 data: 2
2624 019/300 Flush
2625 020/300 Write block 2967 len 20480 data: 2 2 2 2 2
2626 021/300 Read block 3629 len 4096
2627 022/300 Write block 1494 len 4096 data: 3
2628 023/300 Read block 2542 len 32768
2629 024/300 Read block 2663 len 20480
2630 025/300 Flush
2631 026/300 Read block 278 len 12288
2632 027/300 Read block 849 len 28672
2633 028/300 Flush
2634 029/300 Read block 764 len 24576
2635 030/300 Read block 779 len 24576
2636 031/300 Read block 2390 len 40960
2637 032/300 Write block 858 len 20480 data: 3 3 3 3 3
2638 033/300 Read block 4974 len 12288
2639 034/300 Write block 2781 len 4096 data: 2
2640 035/300 Flush
2641 036/300 Read block 2155 len 4096
2642 037/300 Write block 879 len 20480 data: 3 3 3 3 2
2643 038/300 Write block 535 len 24576 data: 3 2 2 2 2 2
2644 039/300 Write block 1125 len 20480 data: 2 2 2 2 2
2645 040/300 Read block 2771 len 36864
2646 041/300 Write block 1947 len 40960 data: 2 2 2 2 2 2 2 2 2 2
2647 042/300 Flush
2648 043/300 Read block 1112 len 4096
2649 044/300 Read block 2477 len 24576
2650 045/300 Write block 3624 len 32768 data: 2 2 2 2 2 2 2 2
2651 046/300 Read block 4036 len 4096
2652 047/300 Read block 2904 len 36864
2653 048/300 Read block 3349 len 36864
2654 049/300 Write block 1834 len 28672 data: 2 2 2 2 2 2 2
2655 050/300 Write block 4098 len 28672 data: 2 3 3 4 4 4 3
2656 051/300 Write block 3808 len 36864 data: 2 2 2 2 2 2 2 3 3
2657 052/300 Read block 1045 len 28672
2658 053/300 Read block 1966 len 24576
2659 054/300 Read block 2101 len 28672
2660 055/300 Read block 2839 len 28672
2661 056/300 Read block 4348 len 16384
2662 057/300 Read block 4033 len 8192
2663 058/300 Read block 4227 len 40960
2664 059/300 Read block 4172 len 32768
2665 060/300 Read block 4188 len 16384
2666 061/300 Write block 4078 len 20480 data: 2 2 2 3 3
2667 062/300 Flush
2668 063/300 Read block 373 len 4096
2669 064/300 Write block 2267 len 12288 data: 3 2 2
2670 065/300 Write block 968 len 8192 data: 2 2
2671 066/300 Read block 3678 len 40960
2672 067/300 Flush
2673 068/300 Read block 1855 len 8192
2674 069/300 Read block 1052 len 8192
2675 070/300 Read block 3143 len 12288
2676 071/300 Write block 1697 len 20480 data: 2 2 2 2 2
2677 072/300 Write block 398 len 36864 data: 2 2 2 2 2 2 2 2 3
2678 073/300 Read block 3464 len 36864
2679 074/300 Write block 881 len 12288 data: 4 4 3
2680 075/300 Read block 595 len 32768
2681 076/300 Read block 4989 len 20480
2682 077/300 Write block 4939 len 40960 data: 3 5 5 5 5 4 4 4 4 4
2683 078/300 Read block 2882 len 4096
2684 079/300 Read block 468 len 32768
2685 080/300 Write block 2794 len 24576 data: 2 2 2 2 2 2
2686 081/300 Read block 393 len 12288
2687 082/300 Write block 4670 len 12288 data: 2 2 2
2688 083/300 Read block 4502 len 12288
2689 084/300 Write block 4339 len 24576 data: 2 2 2 2 3 3
2690 085/300 Read block 1327 len 8192
2691 086/300 Write block 1086 len 40960 data: 6 5 5 5 4 2 2 2 3 3
2692 087/300 Write block 1777 len 12288 data: 3 2 2
2693 088/300 Write block 3144 len 8192 data: 2 2
2694 089/300 Write block 3086 len 16384 data: 2 2 2 2
2695 090/300 Write block 519 len 40960 data: 2 2 2 2 4 3 3 3 3 3
2696 091/300 Read block 1221 len 20480
2697 092/300 Read block 2360 len 4096
2698 093/300 Read block 119 len 16384
2699 094/300 Read block 3849 len 12288
2700 095/300 Read block 3352 len 40960
2701 096/300 Read block 376 len 36864
2702 097/300 Read block 2428 len 16384
2703 098/300 Write block 4353 len 32768 data: 2 2 3 3 3 3 3 3
2704 099/300 Read block 2723 len 20480
2705 100/300 Write block 738 len 4096 data: 2
2706 101/300 Read block 2790 len 20480
2707 102/300 Read block 2667 len 4096
2708 103/300 Read block 2126 len 32768
2709 104/300 Write block 3393 len 20480 data: 2 2 2 3 3
2710 105/300 Write block 1365 len 28672 data: 2 2 2 2 2 2 2
2711 106/300 Read block 3344 len 4096
2712 107/300 Read block 1845 len 4096
2713 108/300 Write block 4901 len 4096 data: 2
2714 109/300 Read block 914 len 8192
2715 110/300 Write block 4082 len 20480 data: 4 3 3 3 4
2716 111/300 Read block 1466 len 4096
2717 112/300 Flush
2718 113/300 Write block 4906 len 24576 data: 2 2 2 4 3 3
2719 114/300 Write block 4520 len 16384 data: 3 3 2 2
2720 115/300 Write block 425 len 12288 data: 3 2 2
2721 116/300 Write block 4071 len 24576 data: 3 3 3 3 2 2
2722 117/300 Read block 1589 len 28672
2723 118/300 Read block 2723 len 20480
2724 119/300 Read block 4768 len 32768
2725 120/300 Read block 4239 len 4096
2726 121/300 Write block 4160 len 16384 data: 2 2 3 4
2727 122/300 Read block 99 len 16384
2728 123/300 Write block 609 len 28672 data: 2 3 3 3 2 2 2
2729 124/300 Read block 1118 len 32768
2730 125/300 Write block 2487 len 16384 data: 2 2 2 2
2731 126/300 Read block 1261 len 4096
2732 127/300 Read block 2248 len 36864
2733 128/300 Write block 4777 len 28672 data: 2 2 2 2 2 3 3
2734 129/300 Read block 4046 len 20480
2735 130/300 Flush
2736 131/300 Read block 16 len 4096
2737 132/300 Read block 3997 len 32768
2738 133/300 Read block 1269 len 4096
2739 134/300 Read block 428 len 24576
2740 135/300 Write block 4640 len 16384 data: 2 2 2 2
2741 136/300 Read block 484 len 24576
2742 137/300 Read block 2903 len 4096
2743 138/300 Read block 3561 len 4096
2744 139/300 Flush
2745 140/300 Read block 176 len 4096
2746 141/300 Write block 3664 len 28672 data: 2 2 2 2 2 2 2
2747 142/300 Write block 4864 len 12288 data: 4 2 2
2748 143/300 Read block 3812 len 20480
2749 144/300 Write block 4451 len 20480 data: 3 3 3 4 3
2750 145/300 Write block 3448 len 32768 data: 2 2 2 2 2 2 3 3
2751 146/300 Read block 454 len 32768
2752 147/300 Write block 918 len 40960 data: 3 3 2 2 2 2 2 2 2 2
2753 148/300 Read block 4952 len 24576
2754 149/300 Read block 2251 len 28672
2755 150/300 Write block 2567 len 36864 data: 2 2 2 2 2 2 2 2 2
2756 151/300 Write block 2395 len 28672 data: 2 2 3 3 3 3 3
2757 152/300 Read block 229 len 4096
2758 153/300 Write block 830 len 12288 data: 2 2 2
2759 154/300 Read block 1758 len 40960
2760 155/300 Write block 2010 len 36864 data: 3 3 3 3 2 2 2 2 2
2761 156/300 Read block 3747 len 20480
2762 157/300 Write block 95 len 8192 data: 2 2
2763 158/300 Write block 2067 len 32768 data: 2 2 2 2 2 2 2 2
2764 159/300 Write block 4480 len 16384 data: 2 2 2 2
2765 160/300 Read block 4140 len 24576
2766 161/300 Read block 4875 len 24576
2767 162/300 Write block 4346 len 4096 data: 3
2768 163/300 Write block 4035 len 12288 data: 2 2 2
2769 164/300 Read block 3550 len 12288
2770 165/300 Write block 4958 len 8192 data: 2 2
2771 166/300 Write block 2374 len 4096 data: 3
2772 167/300 Read block 4250 len 24576
2773 168/300 Write block 205 len 24576 data: 2 2 3 3 4 3
2774 169/300 Read block 3527 len 16384
2775 170/300 Read block 3659 len 36864
2776 171/300 Read block 1437 len 16384
2777 172/300 Read block 433 len 20480
2778 173/300 Flush
2779 174/300 Write block 861 len 32768 data: 4 4 3 3 2 3 3 3
2780 175/300 Write block 2656 len 28672 data: 2 2 2 2 2 2 2
2781 176/300 Read block 3590 len 36864
2782 177/300 Read block 1922 len 32768
2783 178/300 Read block 1516 len 16384
2784 179/300 Write block 2465 len 32768 data: 2 2 2 2 3 3 3 3
2785 180/300 Write block 92 len 8192 data: 3 2
2786 181/300 Write block 3800 len 40960 data: 2 2 2 2 2 3 3 3 3 3
2787 182/300 Write block 1657 len 20480 data: 2 2 2 2 2
2788 183/300 Read block 211 len 20480
2789 184/300 Read block 4761 len 28672
2790 185/300 Read block 2956 len 40960
2791 186/300 Read block 2491 len 24576
2792 187/300 Read block 2847 len 24576
2793 188/300 Flush
2794 189/300 Read block 3068 len 20480
2795 190/300 Write block 4539 len 16384 data: 4 4 4 4
2796 191/300 Flush
2797 192/300 Read block 2037 len 24576
2798 193/300 Read block 1480 len 20480
2799 194/300 Read block 4165 len 16384
2800 195/300 Read block 2284 len 12288
2801 196/300 Read block 653 len 12288
2802 197/300 Flush
2803 198/300 Read block 4003 len 40960
2804 199/300 Read block 913 len 8192
2805 200/300 Flush
2806 201/300 Write block 1277 len 40960 data: 2 2 2 2 2 3 3 4 4 4
2807 202/300 Read block 3337 len 12288
2808 203/300 Write block 3866 len 12288 data: 2 2 2
2809 204/300 Read block 3216 len 4096
2810 205/300 Flush
2811 206/300 Read block 4529 len 24576
2812 207/300 Write block 2295 len 8192 data: 2 2
2813 208/300 Write block 4046 len 12288 data: 2 2 2
2814 209/300 Read block 4580 len 36864
2815 210/300 Write block 1775 len 40960 data: 3 3 4 3 3 2 2 2 2 2
2816 211/300 Read block 383 len 8192
2817 212/300 Write block 1601 len 32768 data: 2 2 2 2 2 2 2 2
2818 213/300 Write block 1132 len 12288 data: 2 2 2
2819 214/300 Read block 4915 len 40960
2820 215/300 Read block 3802 len 36864
2821 216/300 Read block 4960 len 8192
2822 217/300 Read block 4389 len 40960
2823 218/300 Flush
2824 219/300 Read block 1040 len 28672
2825 220/300 Read block 1879 len 4096
2826 221/300 Read block 8 len 28672
2827 222/300 Write block 106 len 12288 data: 2 2 3
2828 223/300 Write block 2004 len 36864 data: 2 3 3 3 3 3 4 4 4
2829 224/300 Write block 4114 len 32768 data: 2 2 2 2 2 2 2 2
2830 225/300 Read block 3684 len 12288
2831 226/300 Read block 2496 len 12288
2832 227/300 Read block 1804 len 8192
2833 228/300 Write block 1398 len 28672 data: 2 2 2 3 3 3 3
2834 229/300 Flush
2835 230/300 Read block 2621 len 36864
2836 231/300 Write block 1032 len 40960 data: 3 3 2 2 2 2 2 2 2 2
2837 232/300 Read block 1893 len 16384
2838 233/300 Read block 3617 len 36864
2839 234/300 Read block 3785 len 20480
2840 235/300 Read block 4827 len 12288
2841 236/300 Read block 4409 len 32768
2842 237/300 Read block 3227 len 12288
2843 238/300 Read block 3864 len 32768
2844 239/300 Read block 3398 len 12288
2845 240/300 Write block 2599 len 24576 data: 3 3 2 2 2 2
2846 241/300 Write block 3185 len 16384 data: 2 2 3 3
2847 242/300 Read block 3591 len 28672
2848 243/300 Flush
2849 244/300 Write block 1336 len 20480 data: 2 2 2 2 2
2850 245/300 Read block 3251 len 32768
2851 246/300 Write block 2657 len 4096 data: 3
2852 247/300 Write block 654 len 24576 data: 2 2 2 2 2 2
2853 248/300 Write block 4819 len 16384 data: 3 2 2 2
2854 249/300 Flush
2855 250/300 Read block 3591 len 8192
2856 251/300 Write block 2184 len 20480 data: 2 2 2 2 2
2857 252/300 Write block 109 len 20480 data: 3 3 3 3 3
2858 253/300 Read block 632 len 36864
2859 254/300 Read block 3944 len 28672
2860 255/300 Read block 1291 len 24576
2861 256/300 Write block 4785 len 16384 data: 2 2 2 2
2862 257/300 Read block 4162 len 12288
2863 258/300 Read block 886 len 8192
2864 259/300 Write block 3602 len 12288 data: 3 3 3
2865 260/300 Flush
2866 261/300 Read block 4270 len 4096
2867 262/300 Write block 2153 len 20480 data: 2 2 2 2 2
2868 263/300 Read block 4930 len 28672
2869 264/300 Write block 521 len 20480 data: 3 3 5 4 4
2870 265/300 Read block 2824 len 28672
2871 266/300 Write block 2578 len 36864 data: 2 2 2 3 3 3 3 2 2
2872 267/300 Read block 4041 len 16384
2873 268/300 Read block 2299 len 4096
2874 269/300 Write block 79 len 8192 data: 3 3
2875 270/300 Write block 1954 len 4096 data: 3
2876 271/300 Flush
2877 272/300 Flush
2878 273/300 Flush
2879 274/300 Write block 177 len 24576 data: 3 3 3 3 3 3
2880 275/300 Write block 4155 len 40960 data: 2 2 2 2 2 3 3 4 5 4
2881 276/300 Read block 384 len 28672
2882 277/300 Flush
2883 278/300 Write block 2699 len 36864 data: 4 4 3 3 2 2 2 2 2
2884 279/300 Write block 3428 len 24576 data: 3 3 2 2 2 3
2885 280/300 Read block 1898 len 16384
2886 281/300 Write block 4497 len 12288 data: 2 2 2
2887 282/300 Read block 1431 len 16384
2888 283/300 Write block 2564 len 16384 data: 2 2 2 3
2889 284/300 Read block 1070 len 36864
2890 285/300 Flush
2891 286/300 Write block 812 len 32768 data: 2 2 2 2 2 2 2 2
2892 287/300 Write block 1976 len 28672 data: 2 2 2 2 2 2 2
2893 288/300 Write block 1687 len 36864 data: 2 2 2 2 2 2 2 2 2
2894 289/300 Write block 3101 len 40960 data: 3 3 3 3 3 2 2 2 2 2
2895 290/300 Read block 505 len 12288
2896 291/300 Write block 3767 len 4096 data: 3
2897 292/300 Flush
2898 293/300 Write block 4495 len 24576 data: 2 2 3 3 3 2
2899 294/300 Read block 2527 len 40960
2900 295/300 Read block 2030 len 32768
2901 296/300 Write block 242 len 8192 data: 3 2
2902 297/300 Write block 3019 len 12288 data: 2 3 3
2903 298/300 Read block 4165 len 12288
2904 299/300 Read block 451 len 8192
2905 300/300 Write block 2686 len 20480 data: 2 2 2 2 3
2906 Main task has work to do, go find it
2907 got dsc Start(1)
2908 start 1
2909 Oct 13 04:42:24.816 INFO request completed, latency_us: 297, response_code: 204, uri: /start/cid/1, method: POST, req_id: 4782b4b7-d384-4483-95b5-c52ee0c60f7b, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
2910 Replay: started 1, returned:Ok(())
2911 ----------------------------------------------------------------
2912 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
2913 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
2914 909 Acked 1908 Read 10 Done New Done false
2915 910 Acked 1909 Write 8 Done New Done false
2916 911 Acked 1910 Read 8 Done New Done false
2917 912 Acked 1911 Read 9 Done New Done false
2918 913 Acked 1912 Read 4 Done New Done false
2919 914 Acked 1913 Write 6 Done New Done false
2920 915 Acked 1914 Write 5 Done New Done false
2921 916 Acked 1915 Read 6 Done New Done false
2922 917 Acked 1916 Flush 0 Done New Done false
2923 918 Acked 1917 Read 4 Done New Done false
2924 919 Acked 1918 Write 4 Done New Done false
2925 920 Acked 1919 Read 10 Done New Done false
2926 921 Acked 1920 Read 1 Done New Done false
2927 922 Acked 1921 Write 4 Done New Done false
2928 923 Acked 1922 Write 3 Done New Done false
2929 924 Acked 1923 Flush 0 Done New Done false
2930 925 Acked 1924 Flush 0 Done New Done false
2931 926 Acked 1925 Write 1 Done New Done false
2932 927 Acked 1926 Flush 0 Done New Done false
2933 928 Acked 1927 Write 5 Done New Done false
2934 929 Acked 1928 Read 1 Done New Done false
2935 930 Acked 1929 Write 1 Done New Done false
2936 931 Acked 1930 Read 8 Done New Done false
2937 932 Acked 1931 Read 5 Done New Done false
2938 933 Acked 1932 Flush 0 Done New Done false
2939 934 Acked 1933 Read 3 Done New Done false
2940 935 Acked 1934 Read 7 Done New Done false
2941 936 Acked 1935 Flush 0 Done New Done false
2942 937 Acked 1936 Read 6 Done New Done false
2943 938 Acked 1937 Read 6 Done New Done false
2944 939 Acked 1938 Read 10 Done New Done false
2945 940 Acked 1939 Write 5 Done New Done false
2946 941 Acked 1940 Read 3 Done New Done false
2947 942 Acked 1941 Write 1 Done New Done false
2948 943 Acked 1942 Flush 0 Done New Done false
2949 944 Acked 1943 Read 1 Done New Done false
2950 945 Acked 1944 Write 5 Done New Done false
2951 946 Acked 1945 Write 6 Done New Done false
2952 947 Acked 1946 Write 5 Done New Done false
2953 948 Acked 1947 Read 9 Done New Done false
2954 949 Acked 1948 Write 10 Done New Done false
2955 950 Acked 1949 Flush 0 Done New Done false
2956 951 Acked 1950 Read 1 Done New Done false
2957 952 Acked 1951 Read 6 Done New Done false
2958 953 Acked 1952 Write 8 Done New Done false
2959 954 Acked 1953 Read 1 Done New Done false
2960 955 Acked 1954 Read 9 Done New Done false
2961 956 Acked 1955 Read 9 Done New Done false
2962 957 Acked 1956 Write 7 Done New Done false
2963 958 Acked 1957 Write 7 Done New Done false
2964 959 Acked 1958 Write 9 Done New Done false
2965 960 Acked 1959 Read 7 Done New Done false
2966 961 Acked 1960 Read 6 Done New Done false
2967 962 Acked 1961 Read 7 Done New Done false
2968 963 Acked 1962 Read 7 Done New Done false
2969 964 Acked 1963 Read 4 Done New Done false
2970 965 Acked 1964 Read 2 Done New Done false
2971 966 Acked 1965 Read 10 Done New Done false
2972 967 Acked 1966 Read 8 Done New Done false
2973 968 Acked 1967 Read 4 Done New Done false
2974 969 Acked 1968 Write 5 Done New Done false
2975 970 Acked 1969 Flush 0 Done New Done false
2976 971 Acked 1970 Read 1 Done New Done false
2977 972 Acked 1971 Write 3 Done New Done false
2978 973 Acked 1972 Write 2 Done New Done false
2979 974 Acked 1973 Read 10 Done New Done false
2980 975 Acked 1974 Flush 0 Done New Done false
2981 976 Acked 1975 Read 2 Done New Done false
2982 977 Acked 1976 Read 2 Done New Done false
2983 978 Acked 1977 Read 3 Done New Done false
2984 979 Acked 1978 Write 5 Done New Done false
2985 980 Acked 1979 Write 9 Done New Done false
2986 981 Acked 1980 Read 9 Done New Done false
2987 982 Acked 1981 Write 3 Done New Done false
2988 983 Acked 1982 Read 8 Done New Done false
2989 984 Acked 1983 Read 5 Done New Done false
2990 985 Acked 1984 Write 10 Done New Done false
2991 986 Acked 1985 Read 1 Done New Done false
2992 987 Acked 1986 Read 8 Done New Done false
2993 988 Acked 1987 Write 6 Done New Done false
2994 989 Acked 1988 Read 3 Done New Done false
2995 990 Acked 1989 Write 3 Done New Done false
2996 991 Acked 1990 Read 3 Done New Done false
2997 992 Acked 1991 Write 6 Done New Done false
2998 993 Acked 1992 Read 2 Done New Done false
2999 994 Acked 1993 Write 10 Done New Done false
3000 995 Acked 1994 Flush 0 Done New Done false
3001 996 Acked 1995 Write 3 Done New Done false
3002 997 Acked 1996 Write 2 Done New Done false
3003 998 Acked 1997 Write 4 Done New Done false
3004 999 Acked 1998 Write 10 Done New Done false
3005 1000 Acked 1999 Read 5 Done New Done false
3006 1001 Acked 2000 Read 1 Done New Done false
3007 1002 Acked 2001 Read 4 Done New Done false
3008 1003 Acked 2002 Read 3 Done New Done false
3009 1004 Acked 2003 Read 10 Done New Done false
3010 1005 Acked 2004 Read 9 Done New Done false
3011 1006 Acked 2005 Read 4 Done New Done false
3012 1007 Acked 2006 Write 8 Done New Done false
3013 1008 Acked 2007 Read 5 Done New Done false
3014 1009 Acked 2008 Write 1 Done New Done false
3015 1010 Acked 2009 Read 5 Done New Done false
3016 1011 Acked 2010 Read 1 Done New Done false
3017 1012 Acked 2011 Read 8 Done New Done false
3018 1013 Acked 2012 Write 5 Done New Done false
3019 1014 Acked 2013 Write 7 Done New Done false
3020 1015 Acked 2014 Read 1 Done New Done false
3021 1016 Acked 2015 Read 1 Done New Done false
3022 1017 Acked 2016 Write 1 Done New Done false
3023 1018 Acked 2017 Read 2 Done New Done false
3024 1019 Acked 2018 Write 5 Done New Done false
3025 1020 Acked 2019 Read 1 Done New Done false
3026 1021 Acked 2020 Flush 0 Done New Done false
3027 1022 Acked 2021 Write 6 Done New Done false
3028 1023 Acked 2022 Write 4 Done New Done false
3029 1024 Acked 2023 Write 3 Done New Done false
3030 1025 Acked 2024 Write 6 Done New Done false
3031 1026 Acked 2025 Read 7 Done New Done false
3032 1027 Acked 2026 Read 5 Done New Done false
3033 1028 Acked 2027 Read 8 Done New Done false
3034 1029 Acked 2028 Read 1 Done New Done false
3035 1030 Acked 2029 Write 4 Done New Done false
3036 1031 Acked 2030 Read 4 Done New Done false
3037 1032 Acked 2031 Write 7 Done New Done false
3038 1033 Acked 2032 Read 8 Done New Done false
3039 1034 Acked 2033 Write 4 Done New Done false
3040 1035 Acked 2034 Read 1 Done New Done false
3041 1036 Acked 2035 Read 9 Done New Done false
3042 1037 Acked 2036 Write 7 Done New Done false
3043 1038 Acked 2037 Read 5 Done New Done false
3044 1039 Acked 2038 Flush 0 Done New Done false
3045 1040 Acked 2039 Read 1 Done New Done false
3046 1041 Acked 2040 Read 8 Done New Done false
3047 1042 Acked 2041 Read 1 Done New Done false
3048 1043 Acked 2042 Read 6 Done New Done false
3049 1044 Acked 2043 Write 4 Done New Done false
3050 1045 Acked 2044 Read 6 Done New Done false
3051 1046 Acked 2045 Read 1 Done New Done false
3052 1047 Acked 2046 Read 1 Done New Done false
3053 1048 Acked 2047 Flush 0 Done New Done false
3054 1049 Acked 2048 Read 1 Done New Done false
3055 1050 Acked 2049 Write 7 Done New Done false
3056 1051 Acked 2050 Write 3 Done New Done false
3057 1052 Acked 2051 Read 5 Done New Done false
3058 1053 Acked 2052 Write 5 Done New Done false
3059 1054 Acked 2053 Write 8 Done New Done false
3060 1055 Acked 2054 Read 8 Done New Done false
3061 1056 Acked 2055 Write 10 Done New Done false
3062 1057 Acked 2056 Read 6 Done New Done false
3063 1058 Acked 2057 Read 7 Done New Done false
3064 1059 Acked 2058 Write 9 Done New Done false
3065 1060 Acked 2059 Write 7 Done New Done false
3066 1061 Acked 2060 Read 1 Done New Done false
3067 1062 Acked 2061 Write 3 Done New Done false
3068 1063 Acked 2062 Read 10 Done New Done false
3069 1064 Acked 2063 Write 9 Done New Done false
3070 1065 Acked 2064 Read 5 Done New Done false
3071 1066 Acked 2065 Write 2 Done New Done false
3072 1067 Acked 2066 Write 8 Done New Done false
3073 1068 Acked 2067 Write 4 Done New Done false
3074 1069 Acked 2068 Read 6 Done New Done false
3075 1070 Acked 2069 Read 6 Done New Done false
3076 1071 Acked 2070 Write 1 Done New Done false
3077 1072 Acked 2071 Write 3 Done New Done false
3078 1073 Acked 2072 Read 3 Done New Done false
3079 1074 Acked 2073 Write 2 Done New Done false
3080 1075 Acked 2074 Write 1 Done New Done false
3081 1076 Acked 2075 Read 6 Done New Done false
3082 1077 Acked 2076 Write 6 Done New Done false
3083 1078 Acked 2077 Read 4 Done New Done false
3084 1079 Acked 2078 Read 9 Done New Done false
3085 1080 Acked 2079 Read 4 Done New Done false
3086 1081 Acked 2080 Read 5 Done New Done false
3087 1082 Acked 2081 Flush 0 Done New Done false
3088 1083 Acked 2082 Write 8 Done New Done false
3089 1084 Acked 2083 Write 7 Done New Done false
3090 1085 Acked 2084 Read 9 Done New Done false
3091 1086 Acked 2085 Read 8 Done New Done false
3092 1087 Acked 2086 Read 4 Done New Done false
3093 1088 Acked 2087 Write 8 Done New Done false
3094 1089 Acked 2088 Write 2 Done New Done false
3095 1090 Acked 2089 Write 10 Done New Done false
3096 1091 Acked 2090 Write 5 Done New Done false
3097 1092 Acked 2091 Read 5 Done New Done false
3098 1093 Acked 2092 Read 7 Done New Done false
3099 1094 Acked 2093 Read 10 Done New Done false
3100 1095 Acked 2094 Read 6 Done New Done false
3101 1096 Acked 2095 Read 6 Done New Done false
3102 1097 Acked 2096 Flush 0 Done New Done false
3103 1098 Acked 2097 Read 5 Done New Done false
3104 1099 Acked 2098 Write 4 Done New Done false
3105 1100 Acked 2099 Flush 0 Done New Done false
3106 1101 Acked 2100 Read 6 Done New Done false
3107 1102 Acked 2101 Read 5 Done New Done false
3108 1103 Acked 2102 Read 4 Done New Done false
3109 1104 Acked 2103 Read 3 Done New Done false
3110 1105 Acked 2104 Read 3 Done New Done false
3111 1106 Acked 2105 Flush 0 Done New Done false
3112 1107 Acked 2106 Read 10 Done New Done false
3113 1108 Acked 2107 Read 2 Done New Done false
3114 1109 Acked 2108 Flush 0 Done New Done false
3115 1110 Acked 2109 Write 10 Done New Done false
3116 1111 Acked 2110 Read 3 Done New Done false
3117 1112 Acked 2111 Write 3 Done New Done false
3118 1113 Acked 2112 Read 1 Done New Done false
3119 1114 Acked 2113 Flush 0 Done New Done false
3120 1115 Acked 2114 Read 6 Done New Done false
3121 1116 Acked 2115 Write 2 Done New Done false
3122 1117 Acked 2116 Write 3 Done New Done false
3123 1118 Acked 2117 Read 9 Done New Done false
3124 1119 Acked 2118 Write 10 Done New Done false
3125 1120 Acked 2119 Read 2 Done New Done false
3126 1121 Acked 2120 Write 8 Done New Done false
3127 1122 Acked 2121 Write 3 Done New Done false
3128 1123 Acked 2122 Read 10 Done New Done false
3129 1124 Acked 2123 Read 9 Done New Done false
3130 1125 Acked 2124 Read 2 Done New Done false
3131 1126 Acked 2125 Read 10 Done New Done false
3132 1127 Acked 2126 Flush 0 Done New Done false
3133 1128 Acked 2127 Read 7 Done New Done false
3134 1129 Acked 2128 Read 1 Done New Done false
3135 1130 Acked 2129 Read 7 Done New Done false
3136 1131 Acked 2130 Write 3 Done New Done false
3137 1132 Acked 2131 Write 9 Done New Done false
3138 1133 Acked 2132 Write 8 Done New Done false
3139 1134 Acked 2133 Read 3 Done New Done false
3140 1135 Acked 2134 Read 3 Done New Done false
3141 1136 Acked 2135 Read 2 Done New Done false
3142 1137 Acked 2136 Write 7 Done New Done false
3143 1138 Acked 2137 Flush 0 Done New Done false
3144 1139 Acked 2138 Read 9 Done New Done false
3145 1140 Acked 2139 Write 10 Done New Done false
3146 1141 Acked 2140 Read 4 Done New Done false
3147 1142 Acked 2141 Read 9 Done New Done false
3148 1143 Acked 2142 Read 5 Done New Done false
3149 1144 Acked 2143 Read 3 Done New Done false
3150 1145 Acked 2144 Read 8 Done New Done false
3151 1146 Acked 2145 Read 3 Done New Done false
3152 1147 Acked 2146 Read 8 Done New Done false
3153 1148 Acked 2147 Read 3 Done New Done false
3154 1149 Acked 2148 Write 6 Done New Done false
3155 1150 Acked 2149 Write 4 Done New Done false
3156 1151 Acked 2150 Read 7 Done New Done false
3157 1152 Acked 2151 Flush 0 Done New Done false
3158 1153 Acked 2152 Write 5 Done New Done false
3159 1154 Acked 2153 Read 8 Done New Done false
3160 1155 Acked 2154 Write 1 Done New Done false
3161 1156 Acked 2155 Write 6 Done New Done false
3162 1157 Acked 2156 Write 4 Done New Done false
3163 1158 Acked 2157 Flush 0 Done New Done false
3164 1159 Acked 2158 Read 2 Done New Done false
3165 1160 Acked 2159 Write 5 Done New Done false
3166 1161 Acked 2160 Write 5 Done New Done false
3167 1162 Acked 2161 Read 9 Done New Done false
3168 1163 Acked 2162 Flush 0 Done New Done false
3169 1164 Acked 2163 Read 7 Done New Done false
3170 1165 Acked 2164 Read 6 Done New Done false
3171 1166 Acked 2165 Write 4 Done New Done false
3172 1167 Acked 2166 Read 3 Done New Done false
3173 1168 Acked 2167 Read 2 Done New Done false
3174 1169 Acked 2168 Write 3 Done New Done false
3175 1170 Acked 2169 Flush 0 Done New Done false
3176 1171 Acked 2170 Read 1 Done New Done false
3177 1172 Acked 2171 Write 5 Done New Done false
3178 1173 Acked 2172 Read 7 Done New Done false
3179 1174 Acked 2173 Write 5 Done New Done false
3180 1175 Acked 2174 Read 7 Done New Done false
3181 1176 Acked 2175 Write 9 Done New Done false
3182 1177 Acked 2176 Read 4 Done New Done false
3183 1178 Acked 2177 Read 1 Done New Done false
3184 1179 Acked 2178 Write 2 Done New Done false
3185 1180 Acked 2179 Write 1 Done New Done false
3186 1181 Acked 2180 Flush 0 Done New Done false
3187 1182 Acked 2181 Flush 0 Done New Done false
3188 1183 Acked 2182 Flush 0 Done New Done false
3189 1184 Acked 2183 Write 6 Done New Done false
3190 1185 Acked 2184 Write 10 Done New Done false
3191 1186 Acked 2185 Read 7 Done New Done false
3192 1187 Acked 2186 Flush 0 Done New Done false
3193 1188 Acked 2187 Write 9 Done New Done false
3194 1189 Acked 2188 Write 6 Done New Done false
3195 1190 Acked 2189 Read 4 Done New Done false
3196 1191 Acked 2190 Write 3 Done New Done false
3197 1192 Acked 2191 Read 4 Done New Done false
3198 1193 Acked 2192 Write 4 Done New Done false
3199 1194 Acked 2193 Read 9 Done New Done false
3200 1195 Acked 2194 Flush 0 Done New Done false
3201 1196 Acked 2195 Write 8 Done New Done false
3202 1197 Acked 2196 Write 7 Done New Done false
3203 1198 Acked 2197 Write 9 Done New Done false
3204 1199 Acked 2198 Write 10 Done New Done false
3205 1200 Acked 2199 Read 3 Done New Done false
3206 1201 Acked 2200 Write 1 Done New Done false
3207 1202 Acked 2201 Flush 0 Done New Done false
3208 1203 Acked 2202 Write 6 Done New Done false
3209 1204 Acked 2203 Read 10 Done New Done false
3210 1205 Acked 2204 Read 8 Done New Done false
3211 1206 Acked 2205 Write 2 Done New Done false
3212 1207 Acked 2206 Write 3 Done New Done false
3213 1208 Acked 2207 Read 3 Done New Done false
3214 1209 Acked 2208 Read 2 Done New Done false
3215 1210 Acked 2209 Write 5 Sent New Sent false
3216 STATES DS:0 DS:1 DS:2 TOTAL
3217 New 0 302 0 302
3218 Sent 1 0 1 2
3219 Done 301 0 301 602
3220 Skipped 0 0 0 0
3221 Error 0 0 0 0
3222 Last Flush: 2201 1907 2201
3223 Downstairs last five completed: 1907 1906 1905 1904 1903 1902
3224 Upstairs last five completed: 1210 1209 1208 1207 1206 1205
3225 CLIENT: Up:0 ds:302 act:2
3226 [8820] got start action
3227 [8820] I am going to restart
3228 Starting downstairs at port 8820
3229 Make output file at "/tmp/dsc/downstairs-8820.txt"
3230 [8820][1] reports Starting
3231 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1180)
3232 [8820][1] reports Running
32332023-10-13T04:42:26.921ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 1
32342023-10-13T04:42:26.921ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state Offline
32352023-10-13T04:42:26.921ZINFOcrucible: [1] client re-new 303 jobs since flush 1907 = downstairs
32362023-10-13T04:42:26.922ZINFOcrucible: [1] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
32372023-10-13T04:42:26.923ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
32382023-10-13T04:42:26.923ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
32392023-10-13T04:42:26.923ZINFOcrucible: Returning client:1 UUID:12345678-0000-0000-0000-000000008820 matches
32402023-10-13T04:42:26.923ZINFOcrucible: [1] send last flush ID to this DS: 1907
32412023-10-13T04:42:26.923ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
32422023-10-13T04:42:26.923ZINFOcrucible: [1] Replied this last flush ID: 1907
32432023-10-13T04:42:26.923ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Offline Active ds_transition to Replay
32442023-10-13T04:42:26.923ZINFOcrucible: [1] Transition from Offline to Replay
32452023-10-13T04:42:26.923ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
32462023-10-13T04:42:26.923ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Replay Active ds_transition to Active
32472023-10-13T04:42:26.923ZINFOcrucible: [1] Transition from Replay to Active
32482023-10-13T04:42:26.923ZINFOcrucible: [1] Starts cmd_loop
32492023-10-13T04:42:26.923ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
32502023-10-13T04:42:26.923ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
32512023-10-13T04:42:26.923ZINFOcrucible: Set check for repair
32522023-10-13T04:42:27.925ZINFOcrucible: Checking if live repair is needed
32532023-10-13T04:42:27.925ZINFOcrucible: No Live Repair required at this time
32542023-10-13T04:42:27.925ZWARNcrucible: [1] flow control end
3255 ----------------------------------------------------------------
3256 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
3257 Downstairs last five completed: 2210 2209 2208 2207 2206 2205
3258 Upstairs last five completed: 1211 1210 1209 1208 1207 1206
3259 CLIENT: Up:0 ds:0 act:3
3260 Replay: All jobs finished, all DS active.
3261 Main task has work to do, go find it
3262 got dsc Stop(1)
3263 stop 1
3264 Oct 13 04:42:28.821 INFO request completed, latency_us: 474, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 9af876b0-766e-47a0-b465-08c4f8cf41ad, remote_addr: 127.0.0.1:52046, local_addr[8820] Got stop action so:false kr:false
3265 : 127.0.0.1:9998
3266 Oct 13 04:42:28.822 INFO request completed, latency_us: 196, response_code: 200, uri: /state/cid/1, method: GET, req_id: 92f6c405-2a23-49a5-a935-8b69411d911d, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
32672023-10-13T04:42:28.822ZWARNcrucible: [1] downstairs disconnected
32682023-10-13T04:42:28.822ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
32692023-10-13T04:42:28.823ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8820 closed looper = 1
32702023-10-13T04:42:28.823ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:false
32712023-10-13T04:42:28.823ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
32722023-10-13T04:42:28.823ZINFOcrucible: [1] 127.0.0.1:8820 task reports offline
32732023-10-13T04:42:28.823ZWARNcrucible: [1] pm_task rx.recv() is None
3274 [8820] Exited with: ExitStatus(unix_wait_status(9))
3275 [8820][1] reports Exit
3276 Oct 13 04:42:32.970 INFO request completed, latency_us: 255, response_code: 200, uri: /state/cid/1, method: GET, req_id: 750de7d8-184a-4f58-82f2-7ca87b4684cb, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
3277 001/300 Write block 3370 len 32768 data: 2 2 2 2 2 2 2 2
3278 002/300 Read block 3170 len 4096
3279 003/300 Write block 208 len 20480 data: 4 5 4 3 4
3280 004/300 Read block 333 len 36864
3281 005/300 Read block 862 len 24576
3282 006/300 Read block 3844 len 8192
3283 007/300 Read block 4254 len 28672
3284 008/300 Flush
3285 009/300 Write block 1357 len 16384 data: 2 2 2 2
3286 010/300 Write block 2952 len 28672 data: 3 2 2 2 2 2 2
3287 011/300 Read block 980 len 12288
3288 012/300 Write block 2748 len 8192 data: 3 3
3289 013/300 Read block 3241 len 36864
3290 014/300 Read block 3153 len 12288
3291 015/300 Write block 255 len 12288 data: 4 4 4
3292 016/300 Read block 2247 len 8192
3293 017/300 Read block 4490 len 36864
3294 018/300 Write block 103 len 40960 data: 2 2 2 3 3 4 4 4 4 4
3295 019/300 Read block 1478 len 36864
3296 020/300 Read block 2984 len 12288
3297 021/300 Write block 3200 len 8192 data: 2 2
3298 022/300 Read block 744 len 36864
3299 023/300 Read block 2661 len 36864
3300 024/300 Read block 938 len 24576
3301 025/300 Write block 2920 len 40960 data: 2 4 3 3 3 3 3 3 3 3
3302 026/300 Read block 3776 len 32768
3303 027/300 Read block 4744 len 28672
3304 028/300 Flush
3305 029/300 Flush
3306 030/300 Flush
3307 031/300 Read block 1609 len 12288
3308 032/300 Read block 2115 len 32768
3309 033/300 Write block 913 len 40960 data: 3 3 3 3 3 4 4 3 3 3
3310 034/300 Read block 228 len 4096
3311 035/300 Read block 4225 len 20480
3312 036/300 Read block 1461 len 12288
3313 037/300 Write block 4822 len 32768 data: 3 2 2 2 2 4 4 4
3314 038/300 Read block 4922 len 8192
3315 039/300 Read block 3641 len 4096
3316 040/300 Read block 178 len 8192
3317 041/300 Write block 2622 len 20480 data: 2 2 2 2 2
3318 042/300 Read block 4017 len 40960
3319 043/300 Read block 604 len 8192
3320 044/300 Write block 3620 len 16384 data: 2 2 2 2
3321 045/300 Flush
3322 046/300 Read block 2360 len 32768
3323 047/300 Read block 339 len 36864
3324 048/300 Flush
3325 049/300 Read block 4992 len 28672
3326 050/300 Write block 4735 len 20480 data: 4 3 3 2 2
3327 051/300 Read block 3157 len 20480
3328 052/300 Read block 1915 len 28672
3329 053/300 Write block 1900 len 36864 data: 3 2 2 2 2 2 2 2 2
3330 054/300 Read block 4000 len 24576
3331 055/300 Flush
3332 056/300 Write block 3428 len 16384 data: 4 4 3 3
3333 057/300 Read block 3501 len 36864
3334 058/300 Read block 3600 len 32768
3335 059/300 Read block 3314 len 12288
3336 060/300 Write block 884 len 12288 data: 3 4 3
3337 061/300 Read block 2800 len 36864
3338 062/300 Read block 3650 len 24576
3339 063/300 Read block 4145 len 32768
3340 064/300 Write block 1907 len 24576 data: 3 3 2 2 2 2
3341 065/300 Read block 1453 len 20480
3342 066/300 Write block 3611 len 4096 data: 3
3343 067/300 Read block 1079 len 4096
3344 068/300 Read block 2635 len 24576
3345 069/300 Read block 177 len 12288
3346 070/300 Write block 2980 len 20480 data: 2 2 2 2 2
3347 071/300 Read block 4492 len 28672
3348 072/300 Read block 2214 len 4096
3349 073/300 Write block 1693 len 36864 data: 3 3 3 2 3 3 3 3 3
3350 074/300 Read block 102 len 40960
3351 075/300 Write block 3778 len 16384 data: 4 4 4 4
3352 076/300 Write block 4021 len 8192 data: 4 4
3353 077/300 Read block 565 len 12288
3354 078/300 Write block 1219 len 4096 data: 2
3355 079/300 Write block 462 len 4096 data: 2
3356 080/300 Write block 1039 len 28672 data: 3 3 3 2 2 2 2
3357 081/300 Read block 3040 len 32768
3358 082/300 Read block 529 len 28672
3359 083/300 Read block 4038 len 16384
3360 084/300 Read block 1479 len 16384
3361 085/300 Write block 3248 len 12288 data: 2 2 2
3362 086/300 Write block 4032 len 24576 data: 2 2 2 3 3 3
3363 087/300 Write block 3187 len 16384 data: 4 4 3 3
3364 088/300 Read block 1184 len 24576
3365 089/300 Flush
3366 090/300 Write block 4167 len 16384 data: 4 4 4 4
3367 091/300 Read block 1603 len 4096
3368 092/300 Write block 4951 len 4096 data: 2
3369 093/300 Read block 2211 len 36864
3370 094/300 Read block 4367 len 36864
3371 095/300 Read block 115 len 4096
3372 096/300 Write block 520 len 16384 data: 3 4 4 6
3373 097/300 Write block 1601 len 28672 data: 3 3 3 3 3 3 3
3374 098/300 Read block 3266 len 4096
3375 099/300 Read block 3099 len 36864
3376 100/300 Read block 2005 len 40960
3377 101/300 Write block 3369 len 32768 data: 2 3 3 3 3 3 3 3
3378 102/300 Write block 3937 len 32768 data: 3 3 3 3 3 2 2 2
3379 103/300 Write block 4344 len 28672 data: 4 4 4 3 3 3 2
3380 104/300 Write block 3282 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3381 105/300 Read block 3469 len 8192
3382 106/300 Write block 989 len 36864 data: 3 3 3 3 3 2 2 2 2
3383 107/300 Write block 2490 len 20480 data: 3 2 2 2 2
3384 108/300 Write block 3798 len 20480 data: 2 2 3 3 3
3385 109/300 Write block 2986 len 16384 data: 2 2 2 2
3386 110/300 Write block 1856 len 32768 data: 2 2 3 4 4 4 2 2
3387 111/300 Read block 1002 len 16384
3388 112/300 Read block 2527 len 24576
3389 113/300 Read block 2796 len 12288
3390 114/300 Read block 2378 len 36864
3391 115/300 Flush
3392 116/300 Write block 4416 len 12288 data: 2 3 3
3393 117/300 Read block 1757 len 36864
3394 118/300 Read block 1125 len 24576
3395 119/300 Read block 2217 len 12288
3396 120/300 Read block 170 len 24576
3397 121/300 Write block 1088 len 28672 data: 6 6 5 3 3 3 4
3398 122/300 Read block 93 len 36864
3399 123/300 Write block 802 len 16384 data: 2 2 2 2
3400 124/300 Write block 3024 len 28672 data: 3 4 4 4 4 3 3
3401 125/300 Read block 3156 len 24576
3402 126/300 Flush
3403 127/300 Read block 2073 len 8192
3404 128/300 Read block 3503 len 40960
3405 129/300 Read block 3881 len 36864
3406 130/300 Read block 2245 len 20480
3407 131/300 Write block 2836 len 28672 data: 3 3 3 3 3 3 2
3408 132/300 Write block 2165 len 36864 data: 2 2 2 2 2 2 2 2 2
3409 133/300 Write block 4379 len 4096 data: 2
3410 134/300 Read block 3027 len 12288
3411 135/300 Write block 3754 len 20480 data: 2 2 2 2 2
3412 136/300 Write block 258 len 16384 data: 3 3 3 2
3413 137/300 Write block 2890 len 24576 data: 3 3 3 3 3 3
3414 138/300 Write block 2139 len 12288 data: 2 2 2
3415 139/300 Read block 4104 len 16384
3416 140/300 Write block 4721 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3417 141/300 Read block 685 len 36864
3418 142/300 Write block 2768 len 8192 data: 2 2
3419 143/300 Write block 4142 len 16384 data: 2 2 2 2
3420 144/300 Write block 4346 len 36864 data: 5 4 4 4 3 2 2 3 3
3421 145/300 Read block 1839 len 36864
3422 146/300 Write block 4086 len 12288 data: 5 4 4
3423 147/300 Flush
3424 148/300 Read block 3806 len 28672
3425 149/300 Read block 4384 len 4096
3426 150/300 Write block 581 len 28672 data: 2 2 2 2 2 2 4
3427 151/300 Write block 3599 len 32768 data: 2 2 2 4 4 4 3 3
3428 152/300 Read block 204 len 28672
3429 153/300 Read block 499 len 24576
3430 154/300 Read block 1181 len 8192
3431 155/300 Read block 2780 len 8192
3432 156/300 Flush
3433 157/300 Write block 1224 len 36864 data: 2 2 2 2 2 2 2 2 2
3434 158/300 Read block 1095 len 36864
3435 159/300 Write block 507 len 32768 data: 2 2 3 3 4 4 4 4
3436 160/300 Write block 1613 len 36864 data: 2 2 2 2 2 2 2 2 2
3437 161/300 Flush
3438 162/300 Read block 4796 len 36864
3439 163/300 Read block 3006 len 36864
3440 164/300 Write block 3284 len 20480 data: 3 3 3 3 3
3441 165/300 Read block 3188 len 8192
3442 166/300 Write block 4008 len 4096 data: 2
3443 167/300 Read block 693 len 20480
3444 168/300 Read block 3591 len 40960
3445 169/300 Write block 473 len 24576 data: 2 2 2 2 2 2
3446 170/300 Write block 3465 len 28672 data: 3 3 2 2 3 3 2
3447 171/300 Read block 2198 len 24576
3448 172/300 Read block 1364 len 28672
3449 173/300 Write block 3352 len 12288 data: 4 3 3
3450 174/300 Write block 3398 len 8192 data: 3 3
3451 175/300 Read block 844 len 24576
3452 176/300 Write block 3481 len 40960 data: 3 2 2 2 2 2 2 2 2 2
3453 177/300 Write block 3108 len 24576 data: 3 3 3 2 2 2
3454 178/300 Read block 1923 len 24576
3455 179/300 Write block 4210 len 36864 data: 2 2 2 2 2 2 2 2 2
3456 180/300 Write block 2060 len 20480 data: 3 3 2 2 2
3457 181/300 Read block 2988 len 20480
3458 182/300 Read block 2647 len 28672
3459 183/300 Read block 3310 len 4096
3460 184/300 Flush
3461 185/300 Read block 452 len 4096
3462 186/300 Read block 4419 len 20480
3463 187/300 Read block 476 len 16384
3464 188/300 Read block 4208 len 24576
3465 189/300 Write block 1775 len 8192 data: 4 4
3466 190/300 Write block 1408 len 36864 data: 2 2 2 2 2 2 2 3 3
3467 191/300 Read block 4685 len 40960
3468 192/300 Read block 111 len 12288
3469 193/300 Read block 3828 len 40960
3470 194/300 Write block 4577 len 12288 data: 2 2 2
3471 195/300 Read block 2228 len 32768
3472 196/300 Read block 1101 len 28672
3473 197/300 Write block 3877 len 20480 data: 2 2 2 2 2
3474 198/300 Write block 495 len 24576 data: 2 2 2 2 2 2
3475 199/300 Write block 4410 len 28672 data: 2 2 2 2 2 2 3
3476 200/300 Write block 2101 len 40960 data: 3 2 2 2 2 2 2 2 2 3
3477 201/300 Read block 4546 len 24576
3478 202/300 Read block 2074 len 32768
3479 203/300 Read block 2313 len 36864
3480 204/300 Read block 161 len 24576
3481 205/300 Write block 4268 len 4096 data: 2
3482 206/300 Write block 294 len 4096 data: 3
3483 207/300 Write block 1548 len 28672 data: 2 2 3 3 3 3 4
3484 208/300 Read block 4752 len 8192
3485 209/300 Read block 4960 len 24576
3486 210/300 Read block 4244 len 24576
3487 211/300 Flush
3488 212/300 Read block 3226 len 8192
3489 213/300 Flush
3490 214/300 Flush
3491 215/300 Write block 3356 len 20480 data: 3 3 3 2 3
3492 216/300 Write block 4224 len 8192 data: 2 2
3493 217/300 Flush
3494 218/300 Read block 4429 len 40960
3495 219/300 Read block 4930 len 8192
3496 220/300 Read block 4086 len 28672
3497 221/300 Flush
3498 222/300 Write block 3837 len 8192 data: 2 2
3499 223/300 Write block 2786 len 36864 data: 3 3 3 3 3 2 2 2 3
3500 224/300 Flush
3501 225/300 Flush
3502 226/300 Write block 2176 len 20480 data: 2 2 2 2 2
3503 227/300 Flush
3504 228/300 Read block 1939 len 28672
3505 229/300 Read block 4181 len 32768
3506 230/300 Read block 1783 len 32768
3507 231/300 Write block 2239 len 40960 data: 3 3 3 2 2 2 2 2 2 2
3508 232/300 Write block 2469 len 36864 data: 4 4 4 4 2 2 2 2 2
3509 233/300 Read block 2434 len 28672
3510 234/300 Write block 2169 len 12288 data: 3 3 3
3511 235/300 Read block 622 len 16384
3512 236/300 Write block 3003 len 32768 data: 2 2 2 2 2 2 2 2
3513 237/300 Read block 4835 len 8192
3514 238/300 Write block 2891 len 12288 data: 4 4 4
3515 239/300 Write block 619 len 16384 data: 3 3 4 4
3516 240/300 Flush
3517 241/300 Read block 3818 len 40960
3518 242/300 Write block 2893 len 16384 data: 5 4 4 2
3519 243/300 Write block 248 len 16384 data: 2 2 2 2
3520 244/300 Read block 2651 len 20480
3521 245/300 Read block 18 len 24576
3522 246/300 Write block 1450 len 36864 data: 2 2 2 2 2 2 2 2 2
3523 247/300 Read block 3183 len 4096
3524 248/300 Flush
3525 249/300 Write block 162 len 36864 data: 2 2 2 2 2 2 2 3 3
3526 250/300 Write block 4068 len 36864 data: 3 3 3 4 4 4 4 3 3
3527 251/300 Write block 1917 len 4096 data: 2
3528 252/300 Read block 251 len 16384
3529 253/300 Read block 273 len 8192
3530 254/300 Read block 1775 len 32768
3531 255/300 Flush
3532 256/300 Write block 770 len 36864 data: 2 2 2 2 2 2 2 2 2
3533 257/300 Flush
3534 258/300 Write block 4296 len 32768 data: 3 3 2 2 2 2 2 2
3535 259/300 Read block 3673 len 20480
3536 260/300 Write block 2332 len 36864 data: 2 2 2 2 2 2 2 2 2
3537 261/300 Read block 3411 len 32768
3538 262/300 Read block 2987 len 4096
3539 263/300 Write block 4679 len 40960 data: 2 3 3 3 3 3 3 3 3 3
3540 264/300 Write block 3423 len 32768 data: 3 3 3 3 3 5 5 4
3541 265/300 Write block 1935 len 20480 data: 2 2 2 2 3
3542 266/300 Read block 2900 len 12288
3543 267/300 Read block 4199 len 16384
3544 268/300 Read block 4302 len 28672
3545 269/300 Read block 4386 len 28672
3546 270/300 Read block 2252 len 8192
3547 271/300 Read block 246 len 40960
3548 272/300 Read block 2305 len 4096
3549 273/300 Flush
3550 274/300 Write block 2704 len 28672 data: 3 3 3 3 2 2 2
3551 275/300 Write block 288 len 24576 data: 3 3 3 3 4 3
3552 276/300 Read block 4825 len 40960
3553 277/300 Read block 4377 len 40960
3554 278/300 Read block 1848 len 36864
3555 279/300 Read block 1432 len 40960
3556 280/300 Read block 1948 len 36864
3557 281/300 Write block 2613 len 32768 data: 3 3 3 3 3 3 3 3
3558 282/300 Read block 3040 len 40960
3559 283/300 Read block 2213 len 28672
3560 284/300 Read block 1265 len 32768
3561 285/300 Write block 636 len 16384 data: 2 2 2 2
3562 286/300 Flush
3563 287/300 Write block 3612 len 24576 data: 3 3 2 2 2 2
3564 288/300 Read block 3489 len 16384
3565 289/300 Read block 4682 len 28672
3566 290/300 Write block 898 len 28672 data: 3 3 3 2 2 2 2
3567 291/300 Write block 3872 len 12288 data: 2 2 2
3568 292/300 Write block 1077 len 40960 data: 3 5 4 4 4 4 4 4 6 7
3569 293/300 Read block 2810 len 8192
3570 294/300 Write block 4279 len 20480 data: 3 3 3 3 3
3571 295/300 Read block 1607 len 12288
3572 296/300 Flush
3573 297/300 Read block 2884 len 4096
3574 298/300 Read block 3334 len 40960
3575 299/300 Read block 914 len 32768
3576 300/300 Read block 4644 len 24576
3577 Main task has work to do, go find it
3578 got dsc Start(1)
3579 start 1
3580 Oct 13 04:42:34.029 INFO request completed, latency_us: 283, response_code: 204, uri: /start/cid/1, method: POST, req_id: de6094ca-33b7-4542-a26e-135f651ef6ba, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
3581 Replay: started 1, returned:Ok(())
3582 ----------------------------------------------------------------
3583 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
3584 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
3585 1212 Acked 2211 Write 8 Done New Done false
3586 1213 Acked 2212 Read 1 Done New Done false
3587 1214 Acked 2213 Write 5 Done New Done false
3588 1215 Acked 2214 Read 9 Done New Done false
3589 1216 Acked 2215 Read 6 Done New Done false
3590 1217 Acked 2216 Read 2 Done New Done false
3591 1218 Acked 2217 Read 7 Done New Done false
3592 1219 Acked 2218 Flush 0 Done New Done false
3593 1220 Acked 2219 Write 4 Done New Done false
3594 1221 Acked 2220 Write 7 Done New Done false
3595 1222 Acked 2221 Read 3 Done New Done false
3596 1223 Acked 2222 Write 2 Done New Done false
3597 1224 Acked 2223 Read 9 Done New Done false
3598 1225 Acked 2224 Read 3 Done New Done false
3599 1226 Acked 2225 Write 3 Done New Done false
3600 1227 Acked 2226 Read 2 Done New Done false
3601 1228 Acked 2227 Read 9 Done New Done false
3602 1229 Acked 2228 Write 10 Done New Done false
3603 1230 Acked 2229 Read 9 Done New Done false
3604 1231 Acked 2230 Read 3 Done New Done false
3605 1232 Acked 2231 Write 2 Done New Done false
3606 1233 Acked 2232 Read 9 Done New Done false
3607 1234 Acked 2233 Read 9 Done New Done false
3608 1235 Acked 2234 Read 6 Done New Done false
3609 1236 Acked 2235 Write 10 Done New Done false
3610 1237 Acked 2236 Read 8 Done New Done false
3611 1238 Acked 2237 Read 7 Done New Done false
3612 1239 Acked 2238 Flush 0 Done New Done false
3613 1240 Acked 2239 Flush 0 Done New Done false
3614 1241 Acked 2240 Flush 0 Done New Done false
3615 1242 Acked 2241 Read 3 Done New Done false
3616 1243 Acked 2242 Read 8 Done New Done false
3617 1244 Acked 2243 Write 10 Done New Done false
3618 1245 Acked 2244 Read 1 Done New Done false
3619 1246 Acked 2245 Read 5 Done New Done false
3620 1247 Acked 2246 Read 3 Done New Done false
3621 1248 Acked 2247 Write 8 Done New Done false
3622 1249 Acked 2248 Read 2 Done New Done false
3623 1250 Acked 2249 Read 1 Done New Done false
3624 1251 Acked 2250 Read 2 Done New Done false
3625 1252 Acked 2251 Write 5 Done New Done false
3626 1253 Acked 2252 Read 10 Done New Done false
3627 1254 Acked 2253 Read 2 Done New Done false
3628 1255 Acked 2254 Write 4 Done New Done false
3629 1256 Acked 2255 Flush 0 Done New Done false
3630 1257 Acked 2256 Read 8 Done New Done false
3631 1258 Acked 2257 Read 9 Done New Done false
3632 1259 Acked 2258 Flush 0 Done New Done false
3633 1260 Acked 2259 Read 7 Done New Done false
3634 1261 Acked 2260 Write 5 Done New Done false
3635 1262 Acked 2261 Read 5 Done New Done false
3636 1263 Acked 2262 Read 7 Done New Done false
3637 1264 Acked 2263 Write 9 Done New Done false
3638 1265 Acked 2264 Read 6 Done New Done false
3639 1266 Acked 2265 Flush 0 Done New Done false
3640 1267 Acked 2266 Write 4 Done New Done false
3641 1268 Acked 2267 Read 9 Done New Done false
3642 1269 Acked 2268 Read 8 Done New Done false
3643 1270 Acked 2269 Read 3 Done New Done false
3644 1271 Acked 2270 Write 3 Done New Done false
3645 1272 Acked 2271 Read 9 Done New Done false
3646 1273 Acked 2272 Read 6 Done New Done false
3647 1274 Acked 2273 Read 8 Done New Done false
3648 1275 Acked 2274 Write 6 Done New Done false
3649 1276 Acked 2275 Read 5 Done New Done false
3650 1277 Acked 2276 Write 1 Done New Done false
3651 1278 Acked 2277 Read 1 Done New Done false
3652 1279 Acked 2278 Read 6 Done New Done false
3653 1280 Acked 2279 Read 3 Done New Done false
3654 1281 Acked 2280 Write 5 Done New Done false
3655 1282 Acked 2281 Read 7 Done New Done false
3656 1283 Acked 2282 Read 1 Done New Done false
3657 1284 Acked 2283 Write 9 Done New Done false
3658 1285 Acked 2284 Read 10 Done New Done false
3659 1286 Acked 2285 Write 4 Done New Done false
3660 1287 Acked 2286 Write 2 Done New Done false
3661 1288 Acked 2287 Read 3 Done New Done false
3662 1289 Acked 2288 Write 1 Done New Done false
3663 1290 Acked 2289 Write 1 Done New Done false
3664 1291 Acked 2290 Write 7 Done New Done false
3665 1292 Acked 2291 Read 8 Done New Done false
3666 1293 Acked 2292 Read 7 Done New Done false
3667 1294 Acked 2293 Read 4 Done New Done false
3668 1295 Acked 2294 Read 4 Done New Done false
3669 1296 Acked 2295 Write 3 Done New Done false
3670 1297 Acked 2296 Write 6 Done New Done false
3671 1298 Acked 2297 Write 4 Done New Done false
3672 1299 Acked 2298 Read 6 Done New Done false
3673 1300 Acked 2299 Flush 0 Done New Done false
3674 1301 Acked 2300 Write 4 Done New Done false
3675 1302 Acked 2301 Read 1 Done New Done false
3676 1303 Acked 2302 Write 1 Done New Done false
3677 1304 Acked 2303 Read 9 Done New Done false
3678 1305 Acked 2304 Read 9 Done New Done false
3679 1306 Acked 2305 Read 1 Done New Done false
3680 1307 Acked 2306 Write 4 Done New Done false
3681 1308 Acked 2307 Write 7 Done New Done false
3682 1309 Acked 2308 Read 1 Done New Done false
3683 1310 Acked 2309 Read 9 Done New Done false
3684 1311 Acked 2310 Read 10 Done New Done false
3685 1312 Acked 2311 Write 8 Done New Done false
3686 1313 Acked 2312 Write 8 Done New Done false
3687 1314 Acked 2313 Write 7 Done New Done false
3688 1315 Acked 2314 Write 10 Done New Done false
3689 1316 Acked 2315 Read 2 Done New Done false
3690 1317 Acked 2316 Write 9 Done New Done false
3691 1318 Acked 2317 Write 5 Done New Done false
3692 1319 Acked 2318 Write 5 Done New Done false
3693 1320 Acked 2319 Write 4 Done New Done false
3694 1321 Acked 2320 Write 8 Done New Done false
3695 1322 Acked 2321 Read 4 Done New Done false
3696 1323 Acked 2322 Read 6 Done New Done false
3697 1324 Acked 2323 Read 3 Done New Done false
3698 1325 Acked 2324 Read 9 Done New Done false
3699 1326 Acked 2325 Flush 0 Done New Done false
3700 1327 Acked 2326 Write 3 Done New Done false
3701 1328 Acked 2327 Read 9 Done New Done false
3702 1329 Acked 2328 Read 6 Done New Done false
3703 1330 Acked 2329 Read 3 Done New Done false
3704 1331 Acked 2330 Read 6 Done New Done false
3705 1332 Acked 2331 Write 7 Done New Done false
3706 1333 Acked 2332 Read 9 Done New Done false
3707 1334 Acked 2333 Write 4 Done New Done false
3708 1335 Acked 2334 Write 7 Done New Done false
3709 1336 Acked 2335 Flush 0 Done New Done false
3710 1337 Acked 2336 Read 6 Done New Done false
3711 1338 Acked 2337 Flush 0 Done New Done false
3712 1339 Acked 2338 Read 2 Done New Done false
3713 1340 Acked 2339 Read 10 Done New Done false
3714 1341 Acked 2340 Read 9 Done New Done false
3715 1342 Acked 2341 Read 5 Done New Done false
3716 1343 Acked 2342 Write 7 Done New Done false
3717 1344 Acked 2343 Write 9 Done New Done false
3718 1345 Acked 2344 Write 1 Done New Done false
3719 1346 Acked 2345 Read 3 Done New Done false
3720 1347 Acked 2346 Write 5 Done New Done false
3721 1348 Acked 2347 Write 4 Done New Done false
3722 1349 Acked 2348 Write 6 Done New Done false
3723 1350 Acked 2349 Write 3 Done New Done false
3724 1351 Acked 2350 Read 4 Done New Done false
3725 1352 Acked 2351 Write 10 Done New Done false
3726 1353 Acked 2352 Read 9 Done New Done false
3727 1354 Acked 2353 Write 2 Done New Done false
3728 1355 Acked 2354 Write 4 Done New Done false
3729 1356 Acked 2355 Write 9 Done New Done false
3730 1357 Acked 2356 Read 9 Done New Done false
3731 1358 Acked 2357 Write 3 Done New Done false
3732 1359 Acked 2358 Flush 0 Done New Done false
3733 1360 Acked 2359 Read 7 Done New Done false
3734 1361 Acked 2360 Read 1 Done New Done false
3735 1362 Acked 2361 Write 7 Done New Done false
3736 1363 Acked 2362 Write 8 Done New Done false
3737 1364 Acked 2363 Read 7 Done New Done false
3738 1365 Acked 2364 Read 6 Done New Done false
3739 1366 Acked 2365 Read 2 Done New Done false
3740 1367 Acked 2366 Read 2 Done New Done false
3741 1368 Acked 2367 Flush 0 Done New Done false
3742 1369 Acked 2368 Write 9 Done New Done false
3743 1370 Acked 2369 Read 9 Done New Done false
3744 1371 Acked 2370 Write 8 Done New Done false
3745 1372 Acked 2371 Write 9 Done New Done false
3746 1373 Acked 2372 Flush 0 Done New Done false
3747 1374 Acked 2373 Read 9 Done New Done false
3748 1375 Acked 2374 Read 9 Done New Done false
3749 1376 Acked 2375 Write 5 Done New Done false
3750 1377 Acked 2376 Read 2 Done New Done false
3751 1378 Acked 2377 Write 1 Done New Done false
3752 1379 Acked 2378 Read 5 Done New Done false
3753 1380 Acked 2379 Read 10 Done New Done false
3754 1381 Acked 2380 Write 6 Done New Done false
3755 1382 Acked 2381 Write 7 Done New Done false
3756 1383 Acked 2382 Read 6 Done New Done false
3757 1384 Acked 2383 Read 7 Done New Done false
3758 1385 Acked 2384 Write 3 Done New Done false
3759 1386 Acked 2385 Write 2 Done New Done false
3760 1387 Acked 2386 Read 6 Done New Done false
3761 1388 Acked 2387 Write 10 Done New Done false
3762 1389 Acked 2388 Write 6 Done New Done false
3763 1390 Acked 2389 Read 6 Done New Done false
3764 1391 Acked 2390 Write 9 Done New Done false
3765 1392 Acked 2391 Write 5 Done New Done false
3766 1393 Acked 2392 Read 5 Done New Done false
3767 1394 Acked 2393 Read 7 Done New Done false
3768 1395 Acked 2394 Read 1 Done New Done false
3769 1396 Acked 2395 Flush 0 Done New Done false
3770 1397 Acked 2396 Read 1 Done New Done false
3771 1398 Acked 2397 Read 5 Done New Done false
3772 1399 Acked 2398 Read 4 Done New Done false
3773 1400 Acked 2399 Read 6 Done New Done false
3774 1401 Acked 2400 Write 2 Done New Done false
3775 1402 Acked 2401 Write 9 Done New Done false
3776 1403 Acked 2402 Read 10 Done New Done false
3777 1404 Acked 2403 Read 3 Done New Done false
3778 1405 Acked 2404 Read 10 Done New Done false
3779 1406 Acked 2405 Write 3 Done New Done false
3780 1407 Acked 2406 Read 8 Done New Done false
3781 1408 Acked 2407 Read 7 Done New Done false
3782 1409 Acked 2408 Write 5 Done New Done false
3783 1410 Acked 2409 Write 6 Done New Done false
3784 1411 Acked 2410 Write 7 Done New Done false
3785 1412 Acked 2411 Write 10 Done New Done false
3786 1413 Acked 2412 Read 6 Done New Done false
3787 1414 Acked 2413 Read 8 Done New Done false
3788 1415 Acked 2414 Read 9 Done New Done false
3789 1416 Acked 2415 Read 6 Done New Done false
3790 1417 Acked 2416 Write 1 Done New Done false
3791 1418 Acked 2417 Write 1 Done New Done false
3792 1419 Acked 2418 Write 7 Done New Done false
3793 1420 Acked 2419 Read 2 Done New Done false
3794 1421 Acked 2420 Read 6 Done New Done false
3795 1422 Acked 2421 Read 6 Done New Done false
3796 1423 Acked 2422 Flush 0 Done New Done false
3797 1424 Acked 2423 Read 2 Done New Done false
3798 1425 Acked 2424 Flush 0 Done New Done false
3799 1426 Acked 2425 Flush 0 Done New Done false
3800 1427 Acked 2426 Write 5 Done New Done false
3801 1428 Acked 2427 Write 2 Done New Done false
3802 1429 Acked 2428 Flush 0 Done New Done false
3803 1430 Acked 2429 Read 10 Done New Done false
3804 1431 Acked 2430 Read 2 Done New Done false
3805 1432 Acked 2431 Read 7 Done New Done false
3806 1433 Acked 2432 Flush 0 Done New Done false
3807 1434 Acked 2433 Write 2 Done New Done false
3808 1435 Acked 2434 Write 9 Done New Done false
3809 1436 Acked 2435 Flush 0 Done New Done false
3810 1437 Acked 2436 Flush 0 Done New Done false
3811 1438 Acked 2437 Write 5 Done New Done false
3812 1439 Acked 2438 Flush 0 Done New Done false
3813 1440 Acked 2439 Read 7 Done New Done false
3814 1441 Acked 2440 Read 8 Done New Done false
3815 1442 Acked 2441 Read 8 Done New Done false
3816 1443 Acked 2442 Write 10 Done New Done false
3817 1444 Acked 2443 Write 9 Done New Done false
3818 1445 Acked 2444 Read 7 Done New Done false
3819 1446 Acked 2445 Write 3 Done New Done false
3820 1447 Acked 2446 Read 4 Done New Done false
3821 1448 Acked 2447 Write 8 Done New Done false
3822 1449 Acked 2448 Read 2 Done New Done false
3823 1450 Acked 2449 Write 3 Done New Done false
3824 1451 Acked 2450 Write 4 Done New Done false
3825 1452 Acked 2451 Flush 0 Done New Done false
3826 1453 Acked 2452 Read 10 Done New Done false
3827 1454 Acked 2453 Write 4 Done New Done false
3828 1455 Acked 2454 Write 4 Done New Done false
3829 1456 Acked 2455 Read 5 Done New Done false
3830 1457 Acked 2456 Flush 0 Done New Done false
3831 1458 Acked 2457 Read 6 Done New Done false
3832 1459 Acked 2458 Write 9 Done New Done false
3833 1460 Acked 2459 Read 1 Done New Done false
3834 1461 Acked 2460 Flush 0 Done New Done false
3835 1462 Acked 2461 Write 9 Done New Done false
3836 1463 Acked 2462 Write 9 Done New Done false
3837 1464 Acked 2463 Write 1 Done New Done false
3838 1465 Acked 2464 Read 4 Done New Done false
3839 1466 Acked 2465 Read 2 Done New Done false
3840 1467 Acked 2466 Read 8 Done New Done false
3841 1468 Acked 2467 Flush 0 Done New Done false
3842 1469 Acked 2468 Write 9 Done New Done false
3843 1470 Acked 2469 Flush 0 Done New Done false
3844 1471 Acked 2470 Write 8 Done New Done false
3845 1472 Acked 2471 Read 5 Done New Done false
3846 1473 Acked 2472 Write 9 Done New Done false
3847 1474 Acked 2473 Read 8 Done New Done false
3848 1475 Acked 2474 Read 1 Done New Done false
3849 1476 Acked 2475 Write 10 Done New Done false
3850 1477 Acked 2476 Write 8 Done New Done false
3851 1478 Acked 2477 Write 5 Done New Done false
3852 1479 Acked 2478 Read 3 Done New Done false
3853 1480 Acked 2479 Read 4 Done New Done false
3854 1481 Acked 2480 Read 7 Done New Done false
3855 1482 Acked 2481 Read 7 Done New Done false
3856 1483 Acked 2482 Read 2 Done New Done false
3857 1484 Acked 2483 Read 10 Done New Done false
3858 1485 Acked 2484 Read 1 Done New Done false
3859 1486 Acked 2485 Flush 0 Done New Done false
3860 1487 Acked 2486 Write 7 Done New Done false
3861 1488 Acked 2487 Write 6 Done New Done false
3862 1489 Acked 2488 Read 10 Done New Done false
3863 1490 Acked 2489 Read 10 Done New Done false
3864 1491 Acked 2490 Read 9 Done New Done false
3865 1492 Acked 2491 Read 10 Done New Done false
3866 1493 Acked 2492 Read 9 Done New Done false
3867 1494 Acked 2493 Write 8 Done New Done false
3868 1495 Acked 2494 Read 10 Done New Done false
3869 1496 Acked 2495 Read 7 Done New Done false
3870 1497 Acked 2496 Read 8 Done New Done false
3871 1498 Acked 2497 Write 4 Done New Done false
3872 1499 Acked 2498 Flush 0 Done New Done false
3873 1500 Acked 2499 Write 6 Done New Done false
3874 1501 Acked 2500 Read 4 Done New Done false
3875 1502 Acked 2501 Read 7 Done New Done false
3876 1503 Acked 2502 Write 7 Done New Done false
3877 1504 Acked 2503 Write 3 Done New Done false
3878 1505 Acked 2504 Write 10 Done New Done false
3879 1506 Acked 2505 Read 2 Done New Done false
3880 1507 Acked 2506 Write 5 Done New Done false
3881 1508 Acked 2507 Read 3 Done New Done false
3882 1509 Acked 2508 Flush 0 Done New Done false
3883 1510 Acked 2509 Read 1 Done New Done false
3884 1511 Acked 2510 Read 10 Done New Done false
3885 1512 Acked 2511 Read 8 Done New Done false
3886 1513 Acked 2512 Read 6 Done New Done false
3887 STATES DS:0 DS:1 DS:2 TOTAL
3888 New 0 302 0 302
3889 Sent 0 0 0 0
3890 Done 302 0 302 604
3891 Skipped 0 0 0 0
3892 Error 0 0 0 0
3893 Last Flush: 2508 2210 2508
3894 Downstairs last five completed: 2210 2209 2208 2207 2206 2205
3895 Upstairs last five completed: 1513 1512 1511 1510 1509 1508
3896 CLIENT: Up:0 ds:302 act:2
3897 [8820] got start action
3898 [8820] I am going to restart
3899 Starting downstairs at port 8820
3900 Make output file at "/tmp/dsc/downstairs-8820.txt"
3901 [8820][1] reports Starting
3902 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1181)
3903 [8820][1] reports Running
39042023-10-13T04:42:35.975ZINFOcrucible: [1] connecting to 127.0.0.1:8820 looper = 1
39052023-10-13T04:42:35.975ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 1
39062023-10-13T04:42:35.975ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state Offline
39072023-10-13T04:42:35.975ZINFOcrucible: [1] client re-new 303 jobs since flush 2210 = downstairs
39082023-10-13T04:42:35.976ZINFOcrucible: [1] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
39092023-10-13T04:42:35.977ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
39102023-10-13T04:42:35.977ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
39112023-10-13T04:42:35.977ZINFOcrucible: Returning client:1 UUID:12345678-0000-0000-0000-000000008820 matches
39122023-10-13T04:42:35.977ZINFOcrucible: [1] send last flush ID to this DS: 2210
39132023-10-13T04:42:35.977ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
39142023-10-13T04:42:35.977ZINFOcrucible: [1] Replied this last flush ID: 2210
39152023-10-13T04:42:35.977ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Offline Active ds_transition to Replay
39162023-10-13T04:42:35.977ZINFOcrucible: [1] Transition from Offline to Replay
39172023-10-13T04:42:35.977ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
39182023-10-13T04:42:35.977ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Replay Active ds_transition to Active
39192023-10-13T04:42:35.977ZINFOcrucible: [1] Transition from Replay to Active
39202023-10-13T04:42:35.977ZINFOcrucible: [1] Starts cmd_loop
39212023-10-13T04:42:35.977ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
39222023-10-13T04:42:35.977ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
39232023-10-13T04:42:35.977ZINFOcrucible: Set check for repair
39242023-10-13T04:42:36.978ZINFOcrucible: Checking if live repair is needed
39252023-10-13T04:42:36.978ZINFOcrucible: No Live Repair required at this time
39262023-10-13T04:42:36.978ZWARNcrucible: [1] flow control end
3927 ----------------------------------------------------------------
3928 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
3929 Downstairs last five completed: 2513 2512 2511 2510 2509 2508
3930 Upstairs last five completed: 1514 1513 1512 1511 1510 1509
3931 CLIENT: Up:0 ds:0 act:3
3932 Replay: All jobs finished, all DS active.
3933 Main task has work to do, go find it
3934 got dsc Stop(1)
3935 stop 1
3936 Oct 13 04:42:38.033 INFO request completed, latency_us: 465, response_code: 204, uri: /stop/cid/1, method: POST, req_id: [8820] Got stop action so:false kr:false
3937 3619aecd-e86c-4387-a424-433a27371dbe, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
3938 Oct 13 04:42:38.034 INFO request completed, latency_us: 181, response_code: 200, uri: /state/cid/1, method: GET, req_id: 6eb70464-8363-4e89-a9da-8baff39defc3, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
39392023-10-13T04:42:38.035ZWARNcrucible: [1] downstairs disconnected
39402023-10-13T04:42:38.035ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
39412023-10-13T04:42:38.035ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8820 closed looper = 1
39422023-10-13T04:42:38.035ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:false
39432023-10-13T04:42:38.035ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
39442023-10-13T04:42:38.035ZINFOcrucible: [1] 127.0.0.1:8820 task reports offline
39452023-10-13T04:42:38.035ZWARNcrucible: [1] pm_task rx.recv() is None
3946 [8820] Exited with: ExitStatus(unix_wait_status(9))
3947 [8820][1] reports Exit
3948 Oct 13 04:42:42.037 INFO request completed, latency_us: 246, response_code: 200, uri: /state/cid/1, method: GET, req_id: a7609455-1a54-4456-bffe-a4eb2f1a41b0, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
3949 001/300 Read block 2804 len 28672
3950 002/300 Write block 2570 len 32768 data: 3 3 3 3 3 3 2 2
3951 003/300 Write block 1630 len 40960 data: 2 2 2 2 2 2 2 2 2 2
3952 004/300 Write block 3956 len 28672 data: 3 3 2 2 2 2 2
3953 005/300 Read block 3679 len 4096
3954 006/300 Write block 2592 len 16384 data: 2 3 3 3
3955 007/300 Read block 384 len 12288
3956 008/300 Read block 2962 len 20480
3957 009/300 Read block 507 len 36864
3958 010/300 Read block 729 len 16384
3959 011/300 Read block 281 len 28672
3960 012/300 Read block 1802 len 16384
3961 013/300 Flush
3962 014/300 Write block 2366 len 32768 data: 3 2 2 3 3 3 3 3
3963 015/300 Read block 948 len 12288
3964 016/300 Read block 1753 len 32768
3965 017/300 Write block 3335 len 24576 data: 2 2 2 2 2 2
3966 018/300 Read block 925 len 12288
3967 019/300 Read block 1624 len 28672
3968 020/300 Flush
3969 021/300 Read block 2675 len 24576
3970 022/300 Flush
3971 023/300 Read block 483 len 36864
3972 024/300 Read block 4019 len 12288
3973 025/300 Write block 4204 len 36864 data: 2 2 2 2 2 2 3 3 3
3974 026/300 Write block 2828 len 16384 data: 2 2 2 2
3975 027/300 Read block 307 len 20480
3976 028/300 Read block 4774 len 36864
3977 029/300 Read block 740 len 32768
3978 030/300 Flush
3979 031/300 Read block 1750 len 36864
3980 032/300 Flush
3981 033/300 Read block 420 len 20480
3982 034/300 Read block 3343 len 28672
3983 035/300 Write block 751 len 28672 data: 2 3 3 3 3 3 3
3984 036/300 Flush
3985 037/300 Read block 9 len 36864
3986 038/300 Write block 4447 len 4096 data: 3
3987 039/300 Write block 1863 len 20480 data: 3 2 2 2 2
3988 040/300 Write block 4861 len 28672 data: 4 4 4 5 3 3 2
3989 041/300 Flush
3990 042/300 Write block 1496 len 16384 data: 2 2 2 2
3991 043/300 Write block 2906 len 16384 data: 2 2 3 3
3992 044/300 Write block 2741 len 4096 data: 3
3993 045/300 Write block 291 len 36864 data: 4 5 4 4 3 3 3 3 3
3994 046/300 Flush
3995 047/300 Read block 870 len 20480
3996 048/300 Flush
3997 049/300 Read block 4926 len 36864
3998 050/300 Read block 1790 len 24576
3999 051/300 Read block 4922 len 28672
4000 052/300 Write block 3449 len 36864 data: 3 3 3 3 3 4 4 3 3
4001 053/300 Write block 2194 len 12288 data: 3 4 4
4002 054/300 Write block 2040 len 16384 data: 2 2 2 2
4003 055/300 Read block 293 len 4096
4004 056/300 Write block 1014 len 32768 data: 4 3 3 3 3 2 2 3
4005 057/300 Flush
4006 058/300 Read block 61 len 16384
4007 059/300 Flush
4008 060/300 Read block 3627 len 12288
4009 061/300 Flush
4010 062/300 Write block 678 len 28672 data: 2 2 2 2 2 2 2
4011 063/300 Write block 375 len 8192 data: 2 2
4012 064/300 Flush
4013 065/300 Read block 4727 len 40960
4014 066/300 Read block 382 len 4096
4015 067/300 Write block 1530 len 36864 data: 4 4 3 3 3 3 3 3 3
4016 068/300 Read block 4207 len 8192
4017 069/300 Read block 3485 len 36864
4018 070/300 Read block 4942 len 28672
4019 071/300 Read block 1953 len 40960
4020 072/300 Read block 3449 len 20480
4021 073/300 Read block 4581 len 20480
4022 074/300 Write block 3274 len 12288 data: 2 2 2
4023 075/300 Read block 1430 len 24576
4024 076/300 Flush
4025 077/300 Read block 3896 len 28672
4026 078/300 Flush
4027 079/300 Read block 4596 len 24576
4028 080/300 Write block 72 len 16384 data: 2 2 2 2
4029 081/300 Read block 3432 len 4096
4030 082/300 Flush
4031 083/300 Write block 1824 len 24576 data: 3 3 3 3 3 3
4032 084/300 Write block 2155 len 40960 data: 3 3 3 2 2 2 3 3 3 3
4033 085/300 Read block 865 len 4096
4034 086/300 Read block 1617 len 32768
4035 087/300 Write block 3073 len 4096 data: 2
4036 088/300 Read block 135 len 28672
4037 089/300 Flush
4038 090/300 Read block 2723 len 20480
4039 091/300 Read block 4902 len 28672
4040 092/300 Read block 2558 len 12288
4041 093/300 Flush
4042 094/300 Read block 480 len 4096
4043 095/300 Write block 2209 len 4096 data: 2
4044 096/300 Flush
4045 097/300 Write block 395 len 36864 data: 3 3 3 3 3 3 3 3 3
4046 098/300 Write block 4699 len 20480 data: 4 4 4 3 3
4047 099/300 Read block 3570 len 24576
4048 100/300 Read block 2110 len 8192
4049 101/300 Read block 1441 len 32768
4050 102/300 Read block 2334 len 28672
4051 103/300 Read block 1096 len 32768
4052 104/300 Flush
4053 105/300 Write block 4390 len 4096 data: 2
4054 106/300 Write block 1009 len 20480 data: 2 2 2 3 4
4055 107/300 Read block 130 len 32768
4056 108/300 Write block 2003 len 32768 data: 2 3 4 4 4 4 4 5
4057 109/300 Read block 2455 len 12288
4058 110/300 Write block 872 len 20480 data: 3 4 4 4 4
4059 111/300 Read block 4956 len 28672
4060 112/300 Write block 44 len 12288 data: 3 3 3
4061 113/300 Write block 1911 len 24576 data: 3 3 2 2 2 2
4062 114/300 Read block 3401 len 12288
4063 115/300 Read block 4290 len 16384
4064 116/300 Write block 515 len 24576 data: 4 3 3 3 3 4
4065 117/300 Write block 609 len 16384 data: 3 4 4 4
4066 118/300 Read block 4523 len 28672
4067 119/300 Flush
4068 120/300 Write block 96 len 8192 data: 3 2
4069 121/300 Read block 2148 len 40960
4070 122/300 Write block 1688 len 4096 data: 3
4071 123/300 Flush
4072 124/300 Write block 4293 len 16384 data: 2 2 3 4
4073 125/300 Flush
4074 126/300 Read block 190 len 8192
4075 127/300 Read block 3565 len 28672
4076 128/300 Write block 1968 len 36864 data: 2 2 2 2 2 2 2 2 3
4077 129/300 Write block 1769 len 36864 data: 2 3 3 3 3 3 5 5 5
4078 130/300 Read block 816 len 12288
4079 131/300 Write block 2265 len 24576 data: 4 3 4 3 3 2
4080 132/300 Read block 1231 len 36864
4081 133/300 Read block 947 len 40960
4082 134/300 Read block 4502 len 16384
4083 135/300 Flush
4084 136/300 Read block 347 len 24576
4085 137/300 Read block 3516 len 20480
4086 138/300 Flush
4087 139/300 Write block 3042 len 8192 data: 2 2
4088 140/300 Read block 4786 len 24576
4089 141/300 Write block 3558 len 4096 data: 3
4090 142/300 Write block 3486 len 32768 data: 3 3 3 3 3 2 4 4
4091 143/300 Write block 3149 len 28672 data: 3 2 2 2 2 2 2
4092 144/300 Read block 814 len 40960
4093 145/300 Read block 4930 len 4096
4094 146/300 Read block 3775 len 12288
4095 147/300 Read block 4723 len 8192
4096 148/300 Write block 1122 len 36864 data: 2 2 2 3 3 3 3 3 2
4097 149/300 Read block 2624 len 24576
4098 150/300 Write block 820 len 16384 data: 2 2 2 2
4099 151/300 Read block 1815 len 8192
4100 152/300 Flush
4101 153/300 Write block 3652 len 32768 data: 2 3 2 2 2 3 3 3
4102 154/300 Write block 254 len 24576 data: 3 5 5 5 4 4
4103 155/300 Write block 1544 len 12288 data: 3 3 2
4104 156/300 Write block 1141 len 36864 data: 3 3 3 3 2 2 2 2 2
4105 157/300 Read block 3636 len 40960
4106 158/300 Write block 3747 len 4096 data: 2
4107 159/300 Write block 4243 len 28672 data: 2 2 2 3 3 3 3
4108 160/300 Read block 369 len 20480
4109 161/300 Read block 183 len 36864
4110 162/300 Read block 2220 len 24576
4111 163/300 Read block 4283 len 40960
4112 164/300 Read block 4786 len 40960
4113 165/300 Write block 1587 len 16384 data: 2 2 2 2
4114 166/300 Write block 1660 len 16384 data: 3 3 2 2
4115 167/300 Read block 3998 len 32768
4116 168/300 Write block 412 len 32768 data: 2 2 2 2 2 2 2 2
4117 169/300 Read block 3492 len 12288
4118 170/300 Write block 1069 len 36864 data: 3 3 2 2 2 2 3 3 4
4119 171/300 Read block 3362 len 24576
4120 172/300 Write block 1694 len 32768 data: 4 4 3 4 4 4 4 4
4121 173/300 Read block 4434 len 40960
4122 174/300 Read block 2255 len 24576
4123 175/300 Write block 2272 len 36864 data: 3 3 3 3 3 2 2 2 2
4124 176/300 Write block 307 len 36864 data: 4 4 4 3 3 2 2 2 2
4125 177/300 Write block 3507 len 28672 data: 3 3 2 2 2 2 2
4126 178/300 Write block 1531 len 20480 data: 5 4 4 4 4
4127 179/300 Write block 3714 len 40960 data: 4 4 2 2 2 2 2 2 2 2
4128 180/300 Flush
4129 181/300 Read block 1485 len 32768
4130 182/300 Write block 2585 len 12288 data: 3 3 2
4131 183/300 Write block 3739 len 36864 data: 2 2 2 2 2 2 2 2 3
4132 184/300 Read block 2418 len 36864
4133 185/300 Flush
4134 186/300 Read block 529 len 4096
4135 187/300 Read block 2669 len 36864
4136 188/300 Write block 2321 len 40960 data: 2 3 3 3 3 3 3 3 3 3
4137 189/300 Write block 4905 len 28672 data: 3 3 3 3 5 4 4
4138 190/300 Write block 2224 len 8192 data: 2 2
4139 191/300 Read block 4062 len 4096
4140 192/300 Flush
4141 193/300 Write block 3070 len 16384 data: 3 3 3 3
4142 194/300 Read block 3399 len 12288
4143 195/300 Read block 4039 len 4096
4144 196/300 Read block 2896 len 32768
4145 197/300 Write block 4160 len 40960 data: 4 4 5 6 5 4 3 5 5 5
4146 198/300 Write block 1735 len 20480 data: 2 2 2 3 2
4147 199/300 Read block 4853 len 28672
4148 200/300 Read block 718 len 16384
4149 201/300 Read block 4810 len 28672
4150 202/300 Write block 2366 len 8192 data: 4 3
4151 203/300 Write block 1148 len 4096 data: 3
4152 204/300 Write block 4833 len 4096 data: 3
4153 205/300 Read block 2497 len 16384
4154 206/300 Write block 2106 len 4096 data: 3
4155 207/300 Write block 1885 len 28672 data: 2 2 2 2 2 2 2
4156 208/300 Read block 411 len 12288
4157 209/300 Read block 1420 len 40960
4158 210/300 Write block 3874 len 20480 data: 3 2 2 3 3
4159 211/300 Read block 4139 len 16384
4160 212/300 Read block 4230 len 20480
4161 213/300 Read block 4628 len 32768
4162 214/300 Read block 4595 len 8192
4163 215/300 Read block 377 len 4096
4164 216/300 Read block 2235 len 24576
4165 217/300 Write block 4477 len 32768 data: 2 2 2 3 3 3 3 2
4166 218/300 Read block 4400 len 8192
4167 219/300 Read block 2622 len 12288
4168 220/300 Write block 1298 len 20480 data: 2 2 2 2 2
4169 221/300 Write block 4386 len 32768 data: 2 2 2 2 3 2 2 2
4170 222/300 Write block 4641 len 28672 data: 3 3 3 2 2 2 2
4171 223/300 Read block 1526 len 20480
4172 224/300 Write block 953 len 16384 data: 2 2 2 2
4173 225/300 Flush
4174 226/300 Read block 715 len 8192
4175 227/300 Flush
4176 228/300 Write block 3914 len 16384 data: 2 2 2 2
4177 229/300 Write block 4154 len 24576 data: 2 3 3 3 3 3
4178 230/300 Write block 4317 len 16384 data: 4 4 3 3
4179 231/300 Read block 1202 len 24576
4180 232/300 Write block 350 len 32768 data: 3 3 3 3 3 3 3 2
4181 233/300 Write block 2652 len 24576 data: 2 2 2 2 3 4
4182 234/300 Write block 4626 len 4096 data: 3
4183 235/300 Flush
4184 236/300 Read block 4751 len 4096
4185 237/300 Write block 3561 len 8192 data: 3 3
4186 238/300 Read block 4335 len 32768
4187 239/300 Write block 4409 len 28672 data: 2 3 3 3 3 3 3
4188 240/300 Write block 2497 len 32768 data: 2 2 2 2 2 2 2 2
4189 241/300 Read block 3488 len 16384
4190 242/300 Write block 2875 len 36864 data: 3 3 3 3 3 2 2 2 2
4191 243/300 Read block 3822 len 20480
4192 244/300 Write block 555 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4193 245/300 Write block 2377 len 32768 data: 3 3 2 2 2 2 2 3
4194 246/300 Read block 2091 len 8192
4195 247/300 Write block 2308 len 16384 data: 2 2 2 2
4196 248/300 Write block 2566 len 16384 data: 3 4 3 3
4197 249/300 Read block 4724 len 40960
4198 250/300 Write block 1149 len 32768 data: 3 3 3 3 3 3 3 3
4199 251/300 Write block 2162 len 12288 data: 4 4 4
4200 252/300 Read block 4375 len 4096
4201 253/300 Read block 2037 len 24576
4202 254/300 Write block 3234 len 8192 data: 2 2
4203 255/300 Write block 1610 len 12288 data: 2 2 2
4204 256/300 Read block 2603 len 28672
4205 257/300 Read block 3613 len 28672
4206 258/300 Write block 3430 len 32768 data: 5 4 3 4 3 3 2 2
4207 259/300 Write block 3581 len 28672 data: 3 4 4 3 3 3 3
4208 260/300 Read block 797 len 12288
4209 261/300 Write block 552 len 12288 data: 3 2 2
4210 262/300 Read block 4530 len 20480
4211 263/300 Read block 4415 len 40960
4212 264/300 Read block 2635 len 16384
4213 265/300 Read block 1399 len 20480
4214 266/300 Write block 3693 len 40960 data: 2 2 2 2 2 2 2 2 2 2
4215 267/300 Write block 2698 len 36864 data: 4 5 5 4 4 3 4 4 4
4216 268/300 Read block 3372 len 12288
4217 269/300 Read block 2509 len 16384
4218 270/300 Write block 4294 len 32768 data: 3 4 5 4 3 3 3 3
4219 271/300 Read block 3353 len 16384
4220 272/300 Read block 1067 len 36864
4221 273/300 Read block 1000 len 36864
4222 274/300 Read block 4431 len 32768
4223 275/300 Read block 2695 len 4096
4224 276/300 Read block 2329 len 8192
4225 277/300 Read block 3641 len 28672
4226 278/300 Write block 4015 len 40960 data: 2 2 2 2 3 3 5 5 4 4
4227 279/300 Read block 2453 len 32768
4228 280/300 Write block 492 len 32768 data: 3 3 3 3 3 3 3 3
4229 281/300 Read block 3645 len 28672
4230 282/300 Read block 2849 len 4096
4231 283/300 Read block 2832 len 12288
4232 284/300 Read block 1160 len 40960
4233 285/300 Write block 993 len 16384 data: 4 3 3 3
4234 286/300 Read block 2903 len 36864
4235 287/300 Flush
4236 288/300 Write block 2556 len 12288 data: 2 2 2
4237 289/300 Flush
4238 290/300 Read block 3068 len 40960
4239 291/300 Write block 2510 len 40960 data: 3 4 4 4 4 5 4 4 4 3
4240 292/300 Read block 653 len 24576
4241 293/300 Read block 4532 len 12288
4242 294/300 Read block 2112 len 8192
4243 295/300 Write block 2772 len 20480 data: 2 2 2 2 2
4244 296/300 Write block 4463 len 8192 data: 3 2
4245 297/300 Write block 4273 len 40960 data: 3 3 3 2 2 3 4 4 4 4
4246 298/300 Read block 279 len 36864
4247 299/300 Read block 1940 len 28672
4248 300/300 Write block 1026 len 20480 data: 2 2 2 2 2
4249 Main task has work to do, go find it
4250 got dsc Start(1)
4251 start 1
4252 Oct 13 04:42:42.942 INFO request completed, latency_us: 291, response_code: 204, uri: /start/cid/1, method: POST, req_id: 9b8da912-1660-42dc-a060-2378e9952a5e, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
4253 Replay: started 1, returned:Ok(())
4254 ----------------------------------------------------------------
4255 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
4256 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
4257 1515 Acked 2514 Read 7 Done New Done false
4258 1516 Acked 2515 Write 8 Done New Done false
4259 1517 Acked 2516 Write 10 Done New Done false
4260 1518 Acked 2517 Write 7 Done New Done false
4261 1519 Acked 2518 Read 1 Done New Done false
4262 1520 Acked 2519 Write 4 Done New Done false
4263 1521 Acked 2520 Read 3 Done New Done false
4264 1522 Acked 2521 Read 5 Done New Done false
4265 1523 Acked 2522 Read 9 Done New Done false
4266 1524 Acked 2523 Read 4 Done New Done false
4267 1525 Acked 2524 Read 7 Done New Done false
4268 1526 Acked 2525 Read 4 Done New Done false
4269 1527 Acked 2526 Flush 0 Done New Done false
4270 1528 Acked 2527 Write 8 Done New Done false
4271 1529 Acked 2528 Read 3 Done New Done false
4272 1530 Acked 2529 Read 8 Done New Done false
4273 1531 Acked 2530 Write 6 Done New Done false
4274 1532 Acked 2531 Read 3 Done New Done false
4275 1533 Acked 2532 Read 7 Done New Done false
4276 1534 Acked 2533 Flush 0 Done New Done false
4277 1535 Acked 2534 Read 6 Done New Done false
4278 1536 Acked 2535 Flush 0 Done New Done false
4279 1537 Acked 2536 Read 9 Done New Done false
4280 1538 Acked 2537 Read 3 Done New Done false
4281 1539 Acked 2538 Write 9 Done New Done false
4282 1540 Acked 2539 Write 4 Done New Done false
4283 1541 Acked 2540 Read 5 Done New Done false
4284 1542 Acked 2541 Read 9 Done New Done false
4285 1543 Acked 2542 Read 8 Done New Done false
4286 1544 Acked 2543 Flush 0 Done New Done false
4287 1545 Acked 2544 Read 9 Done New Done false
4288 1546 Acked 2545 Flush 0 Done New Done false
4289 1547 Acked 2546 Read 5 Done New Done false
4290 1548 Acked 2547 Read 7 Done New Done false
4291 1549 Acked 2548 Write 7 Done New Done false
4292 1550 Acked 2549 Flush 0 Done New Done false
4293 1551 Acked 2550 Read 9 Done New Done false
4294 1552 Acked 2551 Write 1 Done New Done false
4295 1553 Acked 2552 Write 5 Done New Done false
4296 1554 Acked 2553 Write 7 Done New Done false
4297 1555 Acked 2554 Flush 0 Done New Done false
4298 1556 Acked 2555 Write 4 Done New Done false
4299 1557 Acked 2556 Write 4 Done New Done false
4300 1558 Acked 2557 Write 1 Done New Done false
4301 1559 Acked 2558 Write 9 Done New Done false
4302 1560 Acked 2559 Flush 0 Done New Done false
4303 1561 Acked 2560 Read 5 Done New Done false
4304 1562 Acked 2561 Flush 0 Done New Done false
4305 1563 Acked 2562 Read 9 Done New Done false
4306 1564 Acked 2563 Read 6 Done New Done false
4307 1565 Acked 2564 Read 7 Done New Done false
4308 1566 Acked 2565 Write 9 Done New Done false
4309 1567 Acked 2566 Write 3 Done New Done false
4310 1568 Acked 2567 Write 4 Done New Done false
4311 1569 Acked 2568 Read 1 Done New Done false
4312 1570 Acked 2569 Write 8 Done New Done false
4313 1571 Acked 2570 Flush 0 Done New Done false
4314 1572 Acked 2571 Read 4 Done New Done false
4315 1573 Acked 2572 Flush 0 Done New Done false
4316 1574 Acked 2573 Read 3 Done New Done false
4317 1575 Acked 2574 Flush 0 Done New Done false
4318 1576 Acked 2575 Write 7 Done New Done false
4319 1577 Acked 2576 Write 2 Done New Done false
4320 1578 Acked 2577 Flush 0 Done New Done false
4321 1579 Acked 2578 Read 10 Done New Done false
4322 1580 Acked 2579 Read 1 Done New Done false
4323 1581 Acked 2580 Write 9 Done New Done false
4324 1582 Acked 2581 Read 2 Done New Done false
4325 1583 Acked 2582 Read 9 Done New Done false
4326 1584 Acked 2583 Read 7 Done New Done false
4327 1585 Acked 2584 Read 10 Done New Done false
4328 1586 Acked 2585 Read 5 Done New Done false
4329 1587 Acked 2586 Read 5 Done New Done false
4330 1588 Acked 2587 Write 3 Done New Done false
4331 1589 Acked 2588 Read 6 Done New Done false
4332 1590 Acked 2589 Flush 0 Done New Done false
4333 1591 Acked 2590 Read 7 Done New Done false
4334 1592 Acked 2591 Flush 0 Done New Done false
4335 1593 Acked 2592 Read 6 Done New Done false
4336 1594 Acked 2593 Write 4 Done New Done false
4337 1595 Acked 2594 Read 1 Done New Done false
4338 1596 Acked 2595 Flush 0 Done New Done false
4339 1597 Acked 2596 Write 6 Done New Done false
4340 1598 Acked 2597 Write 10 Done New Done false
4341 1599 Acked 2598 Read 1 Done New Done false
4342 1600 Acked 2599 Read 8 Done New Done false
4343 1601 Acked 2600 Write 1 Done New Done false
4344 1602 Acked 2601 Read 7 Done New Done false
4345 1603 Acked 2602 Flush 0 Done New Done false
4346 1604 Acked 2603 Read 5 Done New Done false
4347 1605 Acked 2604 Read 7 Done New Done false
4348 1606 Acked 2605 Read 3 Done New Done false
4349 1607 Acked 2606 Flush 0 Done New Done false
4350 1608 Acked 2607 Read 1 Done New Done false
4351 1609 Acked 2608 Write 1 Done New Done false
4352 1610 Acked 2609 Flush 0 Done New Done false
4353 1611 Acked 2610 Write 9 Done New Done false
4354 1612 Acked 2611 Write 5 Done New Done false
4355 1613 Acked 2612 Read 6 Done New Done false
4356 1614 Acked 2613 Read 2 Done New Done false
4357 1615 Acked 2614 Read 8 Done New Done false
4358 1616 Acked 2615 Read 7 Done New Done false
4359 1617 Acked 2616 Read 8 Done New Done false
4360 1618 Acked 2617 Flush 0 Done New Done false
4361 1619 Acked 2618 Write 1 Done New Done false
4362 1620 Acked 2619 Write 5 Done New Done false
4363 1621 Acked 2620 Read 8 Done New Done false
4364 1622 Acked 2621 Write 8 Done New Done false
4365 1623 Acked 2622 Read 3 Done New Done false
4366 1624 Acked 2623 Write 5 Done New Done false
4367 1625 Acked 2624 Read 7 Done New Done false
4368 1626 Acked 2625 Flush 0 Done New Done false
4369 1627 Acked 2626 Write 3 Done New Done false
4370 1628 Acked 2627 Write 6 Done New Done false
4371 1629 Acked 2628 Read 3 Done New Done false
4372 1630 Acked 2629 Read 4 Done New Done false
4373 1631 Acked 2630 Write 6 Done New Done false
4374 1632 Acked 2631 Write 4 Done New Done false
4375 1633 Acked 2632 Read 7 Done New Done false
4376 1634 Acked 2633 Flush 0 Done New Done false
4377 1635 Acked 2634 Write 2 Done New Done false
4378 1636 Acked 2635 Read 10 Done New Done false
4379 1637 Acked 2636 Write 1 Done New Done false
4380 1638 Acked 2637 Flush 0 Done New Done false
4381 1639 Acked 2638 Write 4 Done New Done false
4382 1640 Acked 2639 Flush 0 Done New Done false
4383 1641 Acked 2640 Read 2 Done New Done false
4384 1642 Acked 2641 Read 7 Done New Done false
4385 1643 Acked 2642 Write 9 Done New Done false
4386 1644 Acked 2643 Write 9 Done New Done false
4387 1645 Acked 2644 Read 3 Done New Done false
4388 1646 Acked 2645 Write 6 Done New Done false
4389 1647 Acked 2646 Read 9 Done New Done false
4390 1648 Acked 2647 Read 10 Done New Done false
4391 1649 Acked 2648 Read 4 Done New Done false
4392 1650 Acked 2649 Flush 0 Done New Done false
4393 1651 Acked 2650 Read 6 Done New Done false
4394 1652 Acked 2651 Read 5 Done New Done false
4395 1653 Acked 2652 Flush 0 Done New Done false
4396 1654 Acked 2653 Write 2 Done New Done false
4397 1655 Acked 2654 Read 6 Done New Done false
4398 1656 Acked 2655 Write 1 Done New Done false
4399 1657 Acked 2656 Write 8 Done New Done false
4400 1658 Acked 2657 Write 7 Done New Done false
4401 1659 Acked 2658 Read 10 Done New Done false
4402 1660 Acked 2659 Read 1 Done New Done false
4403 1661 Acked 2660 Read 3 Done New Done false
4404 1662 Acked 2661 Read 2 Done New Done false
4405 1663 Acked 2662 Write 9 Done New Done false
4406 1664 Acked 2663 Read 6 Done New Done false
4407 1665 Acked 2664 Write 4 Done New Done false
4408 1666 Acked 2665 Read 2 Done New Done false
4409 1667 Acked 2666 Flush 0 Done New Done false
4410 1668 Acked 2667 Write 8 Done New Done false
4411 1669 Acked 2668 Write 6 Done New Done false
4412 1670 Acked 2669 Write 3 Done New Done false
4413 1671 Acked 2670 Write 9 Done New Done false
4414 1672 Acked 2671 Read 10 Done New Done false
4415 1673 Acked 2672 Write 1 Done New Done false
4416 1674 Acked 2673 Write 7 Done New Done false
4417 1675 Acked 2674 Read 5 Done New Done false
4418 1676 Acked 2675 Read 9 Done New Done false
4419 1677 Acked 2676 Read 6 Done New Done false
4420 1678 Acked 2677 Read 10 Done New Done false
4421 1679 Acked 2678 Read 10 Done New Done false
4422 1680 Acked 2679 Write 4 Done New Done false
4423 1681 Acked 2680 Write 4 Done New Done false
4424 1682 Acked 2681 Read 8 Done New Done false
4425 1683 Acked 2682 Write 8 Done New Done false
4426 1684 Acked 2683 Read 3 Done New Done false
4427 1685 Acked 2684 Write 9 Done New Done false
4428 1686 Acked 2685 Read 6 Done New Done false
4429 1687 Acked 2686 Write 8 Done New Done false
4430 1688 Acked 2687 Read 10 Done New Done false
4431 1689 Acked 2688 Read 6 Done New Done false
4432 1690 Acked 2689 Write 9 Done New Done false
4433 1691 Acked 2690 Write 9 Done New Done false
4434 1692 Acked 2691 Write 7 Done New Done false
4435 1693 Acked 2692 Write 5 Done New Done false
4436 1694 Acked 2693 Write 10 Done New Done false
4437 1695 Acked 2694 Flush 0 Done New Done false
4438 1696 Acked 2695 Read 8 Done New Done false
4439 1697 Acked 2696 Write 3 Done New Done false
4440 1698 Acked 2697 Write 9 Done New Done false
4441 1699 Acked 2698 Read 9 Done New Done false
4442 1700 Acked 2699 Flush 0 Done New Done false
4443 1701 Acked 2700 Read 1 Done New Done false
4444 1702 Acked 2701 Read 9 Done New Done false
4445 1703 Acked 2702 Write 10 Done New Done false
4446 1704 Acked 2703 Write 7 Done New Done false
4447 1705 Acked 2704 Write 2 Done New Done false
4448 1706 Acked 2705 Read 1 Done New Done false
4449 1707 Acked 2706 Flush 0 Done New Done false
4450 1708 Acked 2707 Write 4 Done New Done false
4451 1709 Acked 2708 Read 3 Done New Done false
4452 1710 Acked 2709 Read 1 Done New Done false
4453 1711 Acked 2710 Read 8 Done New Done false
4454 1712 Acked 2711 Write 10 Done New Done false
4455 1713 Acked 2712 Write 5 Done New Done false
4456 1714 Acked 2713 Read 7 Done New Done false
4457 1715 Acked 2714 Read 4 Done New Done false
4458 1716 Acked 2715 Read 7 Done New Done false
4459 1717 Acked 2716 Write 2 Done New Done false
4460 1718 Acked 2717 Write 1 Done New Done false
4461 1719 Acked 2718 Write 1 Done New Done false
4462 1720 Acked 2719 Read 4 Done New Done false
4463 1721 Acked 2720 Write 1 Done New Done false
4464 1722 Acked 2721 Write 7 Done New Done false
4465 1723 Acked 2722 Read 3 Done New Done false
4466 1724 Acked 2723 Read 10 Done New Done false
4467 1725 Acked 2724 Write 5 Done New Done false
4468 1726 Acked 2725 Read 4 Done New Done false
4469 1727 Acked 2726 Read 5 Done New Done false
4470 1728 Acked 2727 Read 8 Done New Done false
4471 1729 Acked 2728 Read 2 Done New Done false
4472 1730 Acked 2729 Read 1 Done New Done false
4473 1731 Acked 2730 Read 6 Done New Done false
4474 1732 Acked 2731 Write 8 Done New Done false
4475 1733 Acked 2732 Read 2 Done New Done false
4476 1734 Acked 2733 Read 3 Done New Done false
4477 1735 Acked 2734 Write 5 Done New Done false
4478 1736 Acked 2735 Write 8 Done New Done false
4479 1737 Acked 2736 Write 7 Done New Done false
4480 1738 Acked 2737 Read 5 Done New Done false
4481 1739 Acked 2738 Write 4 Done New Done false
4482 1740 Acked 2739 Flush 0 Done New Done false
4483 1741 Acked 2740 Read 2 Done New Done false
4484 1742 Acked 2741 Flush 0 Done New Done false
4485 1743 Acked 2742 Write 4 Done New Done false
4486 1744 Acked 2743 Write 6 Done New Done false
4487 1745 Acked 2744 Write 4 Done New Done false
4488 1746 Acked 2745 Read 6 Done New Done false
4489 1747 Acked 2746 Write 8 Done New Done false
4490 1748 Acked 2747 Write 6 Done New Done false
4491 1749 Acked 2748 Write 1 Done New Done false
4492 1750 Acked 2749 Flush 0 Done New Done false
4493 1751 Acked 2750 Read 1 Done New Done false
4494 1752 Acked 2751 Write 2 Done New Done false
4495 1753 Acked 2752 Read 8 Done New Done false
4496 1754 Acked 2753 Write 7 Done New Done false
4497 1755 Acked 2754 Write 8 Done New Done false
4498 1756 Acked 2755 Read 4 Done New Done false
4499 1757 Acked 2756 Write 9 Done New Done false
4500 1758 Acked 2757 Read 5 Done New Done false
4501 1759 Acked 2758 Write 10 Done New Done false
4502 1760 Acked 2759 Write 8 Done New Done false
4503 1761 Acked 2760 Read 2 Done New Done false
4504 1762 Acked 2761 Write 4 Done New Done false
4505 1763 Acked 2762 Write 4 Done New Done false
4506 1764 Acked 2763 Read 10 Done New Done false
4507 1765 Acked 2764 Write 8 Done New Done false
4508 1766 Acked 2765 Write 3 Done New Done false
4509 1767 Acked 2766 Read 1 Done New Done false
4510 1768 Acked 2767 Read 6 Done New Done false
4511 1769 Acked 2768 Write 2 Done New Done false
4512 1770 Acked 2769 Write 3 Done New Done false
4513 1771 Acked 2770 Read 7 Done New Done false
4514 1772 Acked 2771 Read 7 Done New Done false
4515 1773 Acked 2772 Write 8 Done New Done false
4516 1774 Acked 2773 Write 7 Done New Done false
4517 1775 Acked 2774 Read 3 Done New Done false
4518 1776 Acked 2775 Write 3 Done New Done false
4519 1777 Acked 2776 Read 5 Done New Done false
4520 1778 Acked 2777 Read 10 Done New Done false
4521 1779 Acked 2778 Read 4 Done New Done false
4522 1780 Acked 2779 Read 5 Done New Done false
4523 1781 Acked 2780 Write 10 Done New Done false
4524 1782 Acked 2781 Write 9 Done New Done false
4525 1783 Acked 2782 Read 3 Done New Done false
4526 1784 Acked 2783 Read 4 Done New Done false
4527 1785 Acked 2784 Write 8 Done New Done false
4528 1786 Acked 2785 Read 4 Done New Done false
4529 1787 Acked 2786 Read 9 Done New Done false
4530 1788 Acked 2787 Read 9 Done New Done false
4531 1789 Acked 2788 Read 8 Done New Done false
4532 1790 Acked 2789 Read 1 Done New Done false
4533 1791 Acked 2790 Read 2 Done New Done false
4534 1792 Acked 2791 Read 7 Done New Done false
4535 1793 Acked 2792 Write 10 Done New Done false
4536 1794 Acked 2793 Flush 0 Done New Done false
4537 1795 Acked 2794 Read 8 Done New Done false
4538 1796 Acked 2795 Write 8 Done New Done false
4539 1797 Acked 2796 Read 7 Done New Done false
4540 1798 Acked 2797 Read 1 Done New Done false
4541 1799 Acked 2798 Read 3 Done New Done false
4542 1800 Acked 2799 Read 10 Done New Done false
4543 1801 Acked 2800 Write 4 Done New Done false
4544 1802 Acked 2801 Read 9 Done New Done false
4545 1803 Acked 2802 Flush 0 Done New Done false
4546 1804 Acked 2803 Write 3 Done New Done false
4547 1805 Acked 2804 Flush 0 Done New Done false
4548 1806 Acked 2805 Read 10 Done New Done false
4549 1807 Acked 2806 Write 10 Done New Done false
4550 1808 Acked 2807 Read 6 Done New Done false
4551 1809 Acked 2808 Read 3 Done New Done false
4552 1810 Acked 2809 Read 2 Done New Done false
4553 1811 Acked 2810 Write 5 Done New Done false
4554 1812 Acked 2811 Write 2 Done New Done false
4555 1813 Acked 2812 Write 10 Done New Done false
4556 1814 Acked 2813 Read 9 Done New Done false
4557 1815 Acked 2814 Read 7 Done New Done false
4558 1816 Acked 2815 Write 5 Sent New Sent false
4559 STATES DS:0 DS:1 DS:2 TOTAL
4560 New 0 302 0 302
4561 Sent 1 0 1 2
4562 Done 301 0 301 602
4563 Skipped 0 0 0 0
4564 Error 0 0 0 0
4565 Last Flush: 2804 2513 2804
4566 Downstairs last five completed: 2513 2512 2511 2510 2509 2508
4567 Upstairs last five completed: 1816 1815 1814 1813 1812 1811
4568 CLIENT: Up:0 ds:302 act:2
4569 [8820] got start action
4570 [8820] I am going to restart
4571 Starting downstairs at port 8820
4572 Make output file at "/tmp/dsc/downstairs-8820.txt"
4573 [8820][1] reports Starting
4574 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1182)
4575 [8820][1] reports Running
4576 ----------------------------------------------------------------
4577 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
4578 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
4579 1515 Acked 2514 Read 7 Done New Done false
4580 1516 Acked 2515 Write 8 Done New Done false
4581 1517 Acked 2516 Write 10 Done New Done false
4582 1518 Acked 2517 Write 7 Done New Done false
4583 1519 Acked 2518 Read 1 Done New Done false
4584 1520 Acked 2519 Write 4 Done New Done false
4585 1521 Acked 2520 Read 3 Done New Done false
4586 1522 Acked 2521 Read 5 Done New Done false
4587 1523 Acked 2522 Read 9 Done New Done false
4588 1524 Acked 2523 Read 4 Done New Done false
4589 1525 Acked 2524 Read 7 Done New Done false
4590 1526 Acked 2525 Read 4 Done New Done false
4591 1527 Acked 2526 Flush 0 Done New Done false
4592 1528 Acked 2527 Write 8 Done New Done false
4593 1529 Acked 2528 Read 3 Done New Done false
4594 1530 Acked 2529 Read 8 Done New Done false
4595 1531 Acked 2530 Write 6 Done New Done false
4596 1532 Acked 2531 Read 3 Done New Done false
4597 1533 Acked 2532 Read 7 Done New Done false
4598 1534 Acked 2533 Flush 0 Done New Done false
4599 1535 Acked 2534 Read 6 Done New Done false
4600 1536 Acked 2535 Flush 0 Done New Done false
4601 1537 Acked 2536 Read 9 Done New Done false
4602 1538 Acked 2537 Read 3 Done New Done false
4603 1539 Acked 2538 Write 9 Done New Done false
4604 1540 Acked 2539 Write 4 Done New Done false
4605 1541 Acked 2540 Read 5 Done New Done false
4606 1542 Acked 2541 Read 9 Done New Done false
4607 1543 Acked 2542 Read 8 Done New Done false
4608 1544 Acked 2543 Flush 0 Done New Done false
4609 1545 Acked 2544 Read 9 Done New Done false
4610 1546 Acked 2545 Flush 0 Done New Done false
4611 1547 Acked 2546 Read 5 Done New Done false
4612 1548 Acked 2547 Read 7 Done New Done false
4613 1549 Acked 2548 Write 7 Done New Done false
4614 1550 Acked 2549 Flush 0 Done New Done false
4615 1551 Acked 2550 Read 9 Done New Done false
4616 1552 Acked 2551 Write 1 Done New Done false
4617 1553 Acked 2552 Write 5 Done New Done false
4618 1554 Acked 2553 Write 7 Done New Done false
4619 1555 Acked 2554 Flush 0 Done New Done false
4620 1556 Acked 2555 Write 4 Done New Done false
4621 1557 Acked 2556 Write 4 Done New Done false
4622 1558 Acked 2557 Write 1 Done New Done false
4623 1559 Acked 2558 Write 9 Done New Done false
4624 1560 Acked 2559 Flush 0 Done New Done false
4625 1561 Acked 2560 Read 5 Done New Done false
4626 1562 Acked 2561 Flush 0 Done New Done false
4627 1563 Acked 2562 Read 9 Done New Done false
4628 1564 Acked 2563 Read 6 Done New Done false
4629 1565 Acked 2564 Read 7 Done New Done false
4630 1566 Acked 2565 Write 9 Done New Done false
4631 1567 Acked 2566 Write 3 Done New Done false
4632 1568 Acked 2567 Write 4 Done New Done false
4633 1569 Acked 2568 Read 1 Done New Done false
4634 1570 Acked 2569 Write 8 Done New Done false
4635 1571 Acked 2570 Flush 0 Done New Done false
4636 1572 Acked 2571 Read 4 Done New Done false
4637 1573 Acked 2572 Flush 0 Done New Done false
4638 1574 Acked 2573 Read 3 Done New Done false
4639 1575 Acked 2574 Flush 0 Done New Done false
4640 1576 Acked 2575 Write 7 Done New Done false
4641 1577 Acked 2576 Write 2 Done New Done false
4642 1578 Acked 2577 Flush 0 Done New Done false
4643 1579 Acked 2578 Read 10 Done New Done false
4644 1580 Acked 2579 Read 1 Done New Done false
4645 1581 Acked 2580 Write 9 Done New Done false
4646 1582 Acked 2581 Read 2 Done New Done false
4647 1583 Acked 2582 Read 9 Done New Done false
4648 1584 Acked 2583 Read 7 Done New Done false
4649 1585 Acked 2584 Read 10 Done New Done false
4650 1586 Acked 2585 Read 5 Done New Done false
4651 1587 Acked 2586 Read 5 Done New Done false
4652 1588 Acked 2587 Write 3 Done New Done false
4653 1589 Acked 2588 Read 6 Done New Done false
4654 1590 Acked 2589 Flush 0 Done New Done false
4655 1591 Acked 2590 Read 7 Done New Done false
4656 1592 Acked 2591 Flush 0 Done New Done false
4657 1593 Acked 2592 Read 6 Done New Done false
4658 1594 Acked 2593 Write 4 Done New Done false
4659 1595 Acked 2594 Read 1 Done New Done false
4660 1596 Acked 2595 Flush 0 Done New Done false
4661 1597 Acked 2596 Write 6 Done New Done false
4662 1598 Acked 2597 Write 10 Done New Done false
4663 1599 Acked 2598 Read 1 Done New Done false
4664 1600 Acked 2599 Read 8 Done New Done false
4665 1601 Acked 2600 Write 1 Done New Done false
4666 1602 Acked 2601 Read 7 Done New Done false
4667 1603 Acked 2602 Flush 0 Done New Done false
4668 1604 Acked 2603 Read 5 Done New Done false
4669 1605 Acked 2604 Read 7 Done New Done false
4670 1606 Acked 2605 Read 3 Done New Done false
4671 1607 Acked 2606 Flush 0 Done New Done false
4672 1608 Acked 2607 Read 1 Done New Done false
4673 1609 Acked 2608 Write 1 Done New Done false
4674 1610 Acked 2609 Flush 0 Done New Done false
4675 1611 Acked 2610 Write 9 Done New Done false
4676 1612 Acked 2611 Write 5 Done New Done false
4677 1613 Acked 2612 Read 6 Done New Done false
4678 1614 Acked 2613 Read 2 Done New Done false
4679 1615 Acked 2614 Read 8 Done New Done false
4680 1616 Acked 2615 Read 7 Done New Done false
4681 1617 Acked 2616 Read 8 Done New Done false
4682 1618 Acked 2617 Flush 0 Done New Done false
4683 1619 Acked 2618 Write 1 Done New Done false
4684 1620 Acked 2619 Write 5 Done New Done false
4685 1621 Acked 2620 Read 8 Done New Done false
4686 1622 Acked 2621 Write 8 Done New Done false
4687 1623 Acked 2622 Read 3 Done New Done false
4688 1624 Acked 2623 Write 5 Done New Done false
4689 1625 Acked 2624 Read 7 Done New Done false
4690 1626 Acked 2625 Flush 0 Done New Done false
4691 1627 Acked 2626 Write 3 Done New Done false
4692 1628 Acked 2627 Write 6 Done New Done false
4693 1629 Acked 2628 Read 3 Done New Done false
4694 1630 Acked 2629 Read 4 Done New Done false
4695 1631 Acked 2630 Write 6 Done New Done false
4696 1632 Acked 2631 Write 4 Done New Done false
4697 1633 Acked 2632 Read 7 Done New Done false
4698 1634 Acked 2633 Flush 0 Done New Done false
4699 1635 Acked 2634 Write 2 Done New Done false
4700 1636 Acked 2635 Read 10 Done New Done false
4701 1637 Acked 2636 Write 1 Done New Done false
4702 1638 Acked 2637 Flush 0 Done New Done false
4703 1639 Acked 2638 Write 4 Done New Done false
4704 1640 Acked 2639 Flush 0 Done New Done false
4705 1641 Acked 2640 Read 2 Done New Done false
4706 1642 Acked 2641 Read 7 Done New Done false
4707 1643 Acked 2642 Write 9 Done New Done false
4708 1644 Acked 2643 Write 9 Done New Done false
4709 1645 Acked 2644 Read 3 Done New Done false
4710 1646 Acked 2645 Write 6 Done New Done false
4711 1647 Acked 2646 Read 9 Done New Done false
4712 1648 Acked 2647 Read 10 Done New Done false
4713 1649 Acked 2648 Read 4 Done New Done false
4714 1650 Acked 2649 Flush 0 Done New Done false
4715 1651 Acked 2650 Read 6 Done New Done false
4716 1652 Acked 2651 Read 5 Done New Done false
4717 1653 Acked 2652 Flush 0 Done New Done false
4718 1654 Acked 2653 Write 2 Done New Done false
4719 1655 Acked 2654 Read 6 Done New Done false
4720 1656 Acked 2655 Write 1 Done New Done false
4721 1657 Acked 2656 Write 8 Done New Done false
4722 1658 Acked 2657 Write 7 Done New Done false
4723 1659 Acked 2658 Read 10 Done New Done false
4724 1660 Acked 2659 Read 1 Done New Done false
4725 1661 Acked 2660 Read 3 Done New Done false
4726 1662 Acked 2661 Read 2 Done New Done false
4727 1663 Acked 2662 Write 9 Done New Done false
4728 1664 Acked 2663 Read 6 Done New Done false
4729 1665 Acked 2664 Write 4 Done New Done false
4730 1666 Acked 2665 Read 2 Done New Done false
4731 1667 Acked 2666 Flush 0 Done New Done false
4732 1668 Acked 2667 Write 8 Done New Done false
4733 1669 Acked 2668 Write 6 Done New Done false
4734 1670 Acked 2669 Write 3 Done New Done false
4735 1671 Acked 2670 Write 9 Done New Done false
4736 1672 Acked 2671 Read 10 Done New Done false
4737 1673 Acked 2672 Write 1 Done New Done false
4738 1674 Acked 2673 Write 7 Done New Done false
4739 1675 Acked 2674 Read 5 Done New Done false
4740 1676 Acked 2675 Read 9 Done New Done false
4741 1677 Acked 2676 Read 6 Done New Done false
4742 1678 Acked 2677 Read 10 Done New Done false
4743 1679 Acked 2678 Read 10 Done New Done false
4744 1680 Acked 2679 Write 4 Done New Done false
4745 1681 Acked 2680 Write 4 Done New Done false
4746 1682 Acked 2681 Read 8 Done New Done false
4747 1683 Acked 2682 Write 8 Done New Done false
4748 1684 Acked 2683 Read 3 Done New Done false
4749 1685 Acked 2684 Write 9 Done New Done false
4750 1686 Acked 2685 Read 6 Done New Done false
4751 1687 Acked 2686 Write 8 Done New Done false
4752 1688 Acked 2687 Read 10 Done New Done false
4753 1689 Acked 2688 Read 6 Done New Done false
4754 1690 Acked 2689 Write 9 Done New Done false
4755 1691 Acked 2690 Write 9 Done New Done false
4756 1692 Acked 2691 Write 7 Done New Done false
4757 1693 Acked 2692 Write 5 Done New Done false
4758 1694 Acked 2693 Write 10 Done New Done false
4759 1695 Acked 2694 Flush 0 Done New Done false
4760 1696 Acked 2695 Read 8 Done New Done false
4761 1697 Acked 2696 Write 3 Done New Done false
4762 1698 Acked 2697 Write 9 Done New Done false
4763 1699 Acked 2698 Read 9 Done New Done false
4764 1700 Acked 2699 Flush 0 Done New Done false
4765 1701 Acked 2700 Read 1 Done New Done false
4766 1702 Acked 2701 Read 9 Done New Done false
4767 1703 Acked 2702 Write 10 Done New Done false
4768 1704 Acked 2703 Write 7 Done New Done false
4769 1705 Acked 2704 Write 2 Done New Done false
4770 1706 Acked 2705 Read 1 Done New Done false
4771 1707 Acked 2706 Flush 0 Done New Done false
4772 1708 Acked 2707 Write 4 Done New Done false
4773 1709 Acked 2708 Read 3 Done New Done false
4774 1710 Acked 2709 Read 1 Done New Done false
4775 1711 Acked 2710 Read 8 Done New Done false
4776 1712 Acked 2711 Write 10 Done New Done false
4777 1713 Acked 2712 Write 5 Done New Done false
4778 1714 Acked 2713 Read 7 Done New Done false
4779 1715 Acked 2714 Read 4 Done New Done false
4780 1716 Acked 2715 Read 7 Done New Done false
4781 1717 Acked 2716 Write 2 Done New Done false
4782 1718 Acked 2717 Write 1 Done New Done false
4783 1719 Acked 2718 Write 1 Done New Done false
4784 1720 Acked 2719 Read 4 Done New Done false
4785 1721 Acked 2720 Write 1 Done New Done false
4786 1722 Acked 2721 Write 7 Done New Done false
4787 1723 Acked 2722 Read 3 Done New Done false
4788 1724 Acked 2723 Read 10 Done New Done false
4789 1725 Acked 2724 Write 5 Done New Done false
4790 1726 Acked 2725 Read 4 Done New Done false
4791 1727 Acked 2726 Read 5 Done New Done false
4792 1728 Acked 2727 Read 8 Done New Done false
4793 1729 Acked 2728 Read 2 Done New Done false
4794 1730 Acked 2729 Read 1 Done New Done false
4795 1731 Acked 2730 Read 6 Done New Done false
4796 1732 Acked 2731 Write 8 Done New Done false
4797 1733 Acked 2732 Read 2 Done New Done false
4798 1734 Acked 2733 Read 3 Done New Done false
4799 1735 Acked 2734 Write 5 Done New Done false
4800 1736 Acked 2735 Write 8 Done New Done false
4801 1737 Acked 2736 Write 7 Done New Done false
4802 1738 Acked 2737 Read 5 Done New Done false
4803 1739 Acked 2738 Write 4 Done New Done false
4804 1740 Acked 2739 Flush 0 Done New Done false
4805 1741 Acked 2740 Read 2 Done New Done false
4806 1742 Acked 2741 Flush 0 Done New Done false
4807 1743 Acked 2742 Write 4 Done New Done false
4808 1744 Acked 2743 Write 6 Done New Done false
4809 1745 Acked 2744 Write 4 Done New Done false
4810 1746 Acked 2745 Read 6 Done New Done false
4811 1747 Acked 2746 Write 8 Done New Done false
4812 1748 Acked 2747 Write 6 Done New Done false
4813 1749 Acked 2748 Write 1 Done New Done false
4814 1750 Acked 2749 Flush 0 Done New Done false
4815 1751 Acked 2750 Read 1 Done New Done false
4816 1752 Acked 2751 Write 2 Done New Done false
4817 1753 Acked 2752 Read 8 Done New Done false
4818 1754 Acked 2753 Write 7 Done New Done false
4819 1755 Acked 2754 Write 8 Done New Done false
4820 1756 Acked 2755 Read 4 Done New Done false
4821 1757 Acked 2756 Write 9 Done New Done false
4822 1758 Acked 2757 Read 5 Done New Done false
4823 1759 Acked 2758 Write 10 Done New Done false
4824 1760 Acked 2759 Write 8 Done New Done false
4825 1761 Acked 2760 Read 2 Done New Done false
4826 1762 Acked 2761 Write 4 Done New Done false
4827 1763 Acked 2762 Write 4 Done New Done false
4828 1764 Acked 2763 Read 10 Done New Done false
4829 1765 Acked 2764 Write 8 Done New Done false
4830 1766 Acked 2765 Write 3 Done New Done false
4831 1767 Acked 2766 Read 1 Done New Done false
4832 1768 Acked 2767 Read 6 Done New Done false
4833 1769 Acked 2768 Write 2 Done New Done false
4834 1770 Acked 2769 Write 3 Done New Done false
4835 1771 Acked 2770 Read 7 Done New Done false
4836 1772 Acked 2771 Read 7 Done New Done false
4837 1773 Acked 2772 Write 8 Done New Done false
4838 1774 Acked 2773 Write 7 Done New Done false
4839 1775 Acked 2774 Read 3 Done New Done false
4840 1776 Acked 2775 Write 3 Done New Done false
4841 1777 Acked 2776 Read 5 Done New Done false
4842 1778 Acked 2777 Read 10 Done New Done false
4843 1779 Acked 2778 Read 4 Done New Done false
4844 1780 Acked 2779 Read 5 Done New Done false
4845 1781 Acked 2780 Write 10 Done New Done false
4846 1782 Acked 2781 Write 9 Done New Done false
4847 1783 Acked 2782 Read 3 Done New Done false
4848 1784 Acked 2783 Read 4 Done New Done false
4849 1785 Acked 2784 Write 8 Done New Done false
4850 1786 Acked 2785 Read 4 Done New Done false
4851 1787 Acked 2786 Read 9 Done New Done false
4852 1788 Acked 2787 Read 9 Done New Done false
4853 1789 Acked 2788 Read 8 Done New Done false
4854 1790 Acked 2789 Read 1 Done New Done false
4855 1791 Acked 2790 Read 2 Done New Done false
4856 1792 Acked 2791 Read 7 Done New Done false
4857 1793 Acked 2792 Write 10 Done New Done false
4858 1794 Acked 2793 Flush 0 Done New Done false
4859 1795 Acked 2794 Read 8 Done New Done false
4860 1796 Acked 2795 Write 8 Done New Done false
4861 1797 Acked 2796 Read 7 Done New Done false
4862 1798 Acked 2797 Read 1 Done New Done false
4863 1799 Acked 2798 Read 3 Done New Done false
4864 1800 Acked 2799 Read 10 Done New Done false
4865 1801 Acked 2800 Write 4 Done New Done false
4866 1802 Acked 2801 Read 9 Done New Done false
4867 1803 Acked 2802 Flush 0 Done New Done false
4868 1804 Acked 2803 Write 3 Done New Done false
4869 1805 Acked 2804 Flush 0 Done New Done false
4870 1806 Acked 2805 Read 10 Done New Done false
4871 1807 Acked 2806 Write 10 Done New Done false
4872 1808 Acked 2807 Read 6 Done New Done false
4873 1809 Acked 2808 Read 3 Done New Done false
4874 1810 Acked 2809 Read 2 Done New Done false
4875 1811 Acked 2810 Write 5 Done New Done false
4876 1812 Acked 2811 Write 2 Done New Done false
4877 1813 Acked 2812 Write 10 Done New Done false
4878 1814 Acked 2813 Read 9 Done New Done false
4879 1815 Acked 2814 Read 7 Done New Done false
4880 1816 Acked 2815 Write 5 Done New Done false
4881 1817 Acked 2816 Flush 0 Done New Done false
4882 STATES DS:0 DS:1 DS:2 TOTAL
4883 New 0 303 0 303
4884 Sent 0 0 0 0
4885 Done 303 0 303 606
4886 Skipped 0 0 0 0
4887 Error 0 0 0 0
4888 Last Flush: 2816 2513 2816
4889 Downstairs last five completed: 2513 2512 2511 2510 2509 2508
4890 Upstairs last five completed: 1817 1816 1815 1814 1813 1812
4891 CLIENT: Up:0 ds:303 act:2
48922023-10-13T04:42:47.048ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 1
48932023-10-13T04:42:47.048ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state Offline
48942023-10-13T04:42:47.048ZINFOcrucible: [1] client re-new 303 jobs since flush 2513 = downstairs
48952023-10-13T04:42:47.049ZINFOcrucible: [1] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
48962023-10-13T04:42:47.049ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
48972023-10-13T04:42:47.049ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
48982023-10-13T04:42:47.050ZINFOcrucible: Returning client:1 UUID:12345678-0000-0000-0000-000000008820 matches
48992023-10-13T04:42:47.050ZINFOcrucible: [1] send last flush ID to this DS: 2513
49002023-10-13T04:42:47.050ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
49012023-10-13T04:42:47.050ZINFOcrucible: [1] Replied this last flush ID: 2513
49022023-10-13T04:42:47.050ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Offline Active ds_transition to Replay
49032023-10-13T04:42:47.050ZINFOcrucible: [1] Transition from Offline to Replay
49042023-10-13T04:42:47.050ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
49052023-10-13T04:42:47.050ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Replay Active ds_transition to Active
49062023-10-13T04:42:47.050ZINFOcrucible: [1] Transition from Replay to Active
49072023-10-13T04:42:47.050ZINFOcrucible: [1] Starts cmd_loop
49082023-10-13T04:42:47.050ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
49092023-10-13T04:42:47.050ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
49102023-10-13T04:42:47.050ZINFOcrucible: Set check for repair
49112023-10-13T04:42:48.051ZWARNcrucible: [1] flow control end
49122023-10-13T04:42:48.052ZINFOcrucible: Checking if live repair is needed
49132023-10-13T04:42:48.052ZINFOcrucible: No Live Repair required at this time
4914 ----------------------------------------------------------------
4915 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
4916 Downstairs last five completed: 2816 2815 2814 2813 2812 2811
4917 Upstairs last five completed: 1817 1816 1815 1814 1813 1812
4918 CLIENT: Up:0 ds:0 act:3
4919 Replay: All jobs finished, all DS active.
4920 Main task has work to do, go find it
4921 got dsc Stop(2)
4922 stop 2
4923 Oct 13 04:42:50.948 INFO request completed, latency_us: 482, response_code: 204[8830] Got stop action so:false kr:false
4924 , uri: /stop/cid/2, method: POST, req_id: 0e51e209-dd11-4275-8c58-8d8909506ee8, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
4925 Oct 13 04:42:50.949 INFO request completed, latency_us: 179, response_code: 200, uri: /state/cid/2, method: GET, req_id: 272fb80a-4662-4ab1-adab-3a45add66f71, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
49262023-10-13T04:42:50.949ZWARNcrucible: [2] downstairs disconnected
49272023-10-13T04:42:50.949ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
49282023-10-13T04:42:50.949ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8830 closed looper = 2
49292023-10-13T04:42:50.949ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
49302023-10-13T04:42:50.949ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
49312023-10-13T04:42:50.949ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
49322023-10-13T04:42:50.949ZWARNcrucible: [2] pm_task rx.recv() is None
4933 [8830] Exited with: ExitStatus(unix_wait_status(9))
4934 [8830][2] reports Exit
4935 Oct 13 04:42:54.951 INFO request completed, latency_us: 252, response_code: 200, uri: /state/cid/2, method: GET, req_id: d1689101-31a1-4886-948c-6276291d9b32, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
4936 001/300 Read block 500 len 40960
4937 002/300 Flush
4938 003/300 Write block 1254 len 4096 data: 2
4939 004/300 Write block 3860 len 40960 data: 4 4 3 3 2 2 3 3 3 2
4940 005/300 Read block 2192 len 32768
4941 006/300 Read block 513 len 4096
4942 007/300 Flush
4943 008/300 Read block 2115 len 20480
4944 009/300 Read block 1211 len 36864
4945 010/300 Read block 330 len 24576
4946 011/300 Read block 1613 len 36864
4947 012/300 Read block 4388 len 20480
4948 013/300 Flush
4949 014/300 Write block 4758 len 20480 data: 2 2 2 2 2
4950 015/300 Read block 1858 len 4096
4951 016/300 Write block 118 len 24576 data: 2 2 2 2 2 2
4952 017/300 Read block 3716 len 8192
4953 018/300 Read block 32 len 28672
4954 019/300 Write block 1922 len 28672 data: 3 3 2 2 2 2 2
4955 020/300 Read block 3301 len 40960
4956 021/300 Write block 4697 len 24576 data: 3 4 5 5 5 4
4957 022/300 Read block 2144 len 40960
4958 023/300 Read block 339 len 4096
4959 024/300 Read block 4360 len 4096
4960 025/300 Write block 1071 len 8192 data: 3 3
4961 026/300 Write block 748 len 8192 data: 2 3
4962 027/300 Write block 4807 len 12288 data: 2 2 2
4963 028/300 Write block 71 len 8192 data: 2 3
4964 029/300 Write block 3983 len 28672 data: 2 2 2 2 2 2 2
4965 030/300 Write block 4713 len 12288 data: 3 3 2
4966 031/300 Read block 4855 len 12288
4967 032/300 Write block 1126 len 16384 data: 4 4 4 4
4968 033/300 Write block 175 len 12288 data: 3 3 4
4969 034/300 Read block 1790 len 8192
4970 035/300 Write block 4126 len 20480 data: 2 2 2 2 2
4971 036/300 Read block 1301 len 40960
4972 037/300 Read block 2447 len 4096
4973 038/300 Write block 166 len 16384 data: 3 3 3 4
4974 039/300 Read block 3017 len 12288
4975 040/300 Read block 1262 len 16384
4976 041/300 Flush
4977 042/300 Write block 2080 len 16384 data: 2 2 2 2
4978 043/300 Read block 999 len 12288
4979 044/300 Flush
4980 045/300 Flush
4981 046/300 Flush
4982 047/300 Write block 4424 len 28672 data: 3 3 3 3 3 2 2
4983 048/300 Write block 3115 len 12288 data: 2 2 3
4984 049/300 Read block 2673 len 40960
4985 050/300 Read block 4908 len 24576
4986 051/300 Write block 4289 len 36864 data: 3 3 3 2 3 4 5 6 5
4987 052/300 Read block 1057 len 12288
4988 053/300 Read block 1295 len 12288
4989 054/300 Read block 2041 len 40960
4990 055/300 Write block 2950 len 12288 data: 2 3 4
4991 056/300 Read block 4075 len 20480
4992 057/300 Read block 2733 len 24576
4993 058/300 Read block 276 len 8192
4994 059/300 Read block 510 len 12288
4995 060/300 Read block 24 len 12288
4996 061/300 Write block 4599 len 40960 data: 2 2 2 3 3 3 3 3 3 2
4997 062/300 Write block 4606 len 20480 data: 4 4 3 2 2
4998 063/300 Write block 3256 len 20480 data: 3 3 3 3 3
4999 064/300 Read block 969 len 36864
5000 065/300 Write block 2097 len 20480 data: 3 3 3 3 4
5001 066/300 Read block 2197 len 8192
5002 067/300 Write block 121 len 8192 data: 3 3
5003 068/300 Read block 4755 len 40960
5004 069/300 Read block 1664 len 24576
5005 070/300 Write block 3558 len 40960 data: 4 3 3 4 4 3 2 2 2 2
5006 071/300 Read block 4975 len 36864
5007 072/300 Write block 433 len 32768 data: 3 3 3 4 4 3 3 4
5008 073/300 Read block 495 len 24576
5009 074/300 Write block 2121 len 32768 data: 2 2 2 2 3 3 3 3
5010 075/300 Read block 3652 len 32768
5011 076/300 Read block 4517 len 8192
5012 077/300 Write block 498 len 40960 data: 4 4 3 2 2 2 2 2 2 3
5013 078/300 Read block 3189 len 40960
5014 079/300 Read block 870 len 36864
5015 080/300 Read block 4490 len 12288
5016 081/300 Write block 290 len 4096 data: 4
5017 082/300 Read block 2780 len 12288
5018 083/300 Read block 3168 len 24576
5019 084/300 Write block 1741 len 20480 data: 2 2 2 2 2
5020 085/300 Read block 4795 len 32768
5021 086/300 Read block 3587 len 20480
5022 087/300 Read block 96 len 16384
5023 088/300 Read block 4263 len 32768
5024 089/300 Read block 1744 len 32768
5025 090/300 Write block 571 len 20480 data: 3 3 3 3 3
5026 091/300 Read block 4224 len 28672
5027 092/300 Write block 1343 len 4096 data: 3
5028 093/300 Read block 4537 len 40960
5029 094/300 Read block 398 len 12288
5030 095/300 Read block 4962 len 8192
5031 096/300 Read block 2859 len 8192
5032 097/300 Read block 4270 len 8192
5033 098/300 Read block 3079 len 8192
5034 099/300 Write block 3855 len 8192 data: 2 2
5035 100/300 Read block 3951 len 12288
5036 101/300 Read block 3586 len 24576
5037 102/300 Write block 2792 len 28672 data: 3 3 4 3 3 3 3
5038 103/300 Write block 248 len 32768 data: 3 3 3 3 2 3 4 6
5039 104/300 Write block 4345 len 36864 data: 5 6 5 5 5 4 3 3 4
5040 105/300 Read block 1853 len 32768
5041 106/300 Read block 3099 len 40960
5042 107/300 Read block 954 len 16384
5043 108/300 Flush
5044 109/300 Flush
5045 110/300 Write block 3882 len 4096 data: 2
5046 111/300 Read block 4003 len 4096
5047 112/300 Read block 1021 len 40960
5048 113/300 Read block 2671 len 12288
5049 114/300 Write block 4067 len 32768 data: 3 4 4 4 5 5 5 5
5050 115/300 Write block 491 len 12288 data: 3 4 4
5051 116/300 Flush
5052 117/300 Read block 896 len 20480
5053 118/300 Read block 3102 len 28672
5054 119/300 Read block 1477 len 12288
5055 120/300 Write block 2294 len 32768 data: 2 3 3 2 2 2 2 2
5056 121/300 Read block 3086 len 8192
5057 122/300 Flush
5058 123/300 Read block 887 len 20480
5059 124/300 Read block 1783 len 12288
5060 125/300 Read block 436 len 40960
5061 126/300 Write block 3902 len 40960 data: 2 2 2 3 3 3 3 3 3 3
5062 127/300 Read block 4490 len 40960
5063 128/300 Read block 4897 len 4096
5064 129/300 Write block 4699 len 16384 data: 6 6 6 5
5065 130/300 Read block 4099 len 40960
5066 131/300 Write block 2593 len 20480 data: 4 4 4 3 3
5067 132/300 Write block 4518 len 28672 data: 2 3 4 4 3 3 2
5068 133/300 Write block 4084 len 32768 data: 4 4 6 5 5 3 3 2
5069 134/300 Flush
5070 135/300 Read block 1236 len 40960
5071 136/300 Write block 111 len 20480 data: 5 5 4 3 3
5072 137/300 Write block 4524 len 32768 data: 3 2 3 3 3 3 3 3
5073 138/300 Read block 1208 len 4096
5074 139/300 Read block 886 len 24576
5075 140/300 Write block 3682 len 20480 data: 2 2 2 2 2
5076 141/300 Read block 2325 len 12288
5077 142/300 Write block 2792 len 24576 data: 4 4 5 4 4 4
5078 143/300 Write block 4776 len 32768 data: 2 3 3 3 3 3 4 4
5079 144/300 Read block 2451 len 36864
5080 145/300 Write block 4497 len 16384 data: 4 4 4 3
5081 146/300 Write block 1880 len 4096 data: 2
5082 147/300 Write block 3613 len 40960 data: 4 3 3 3 3 2 2 3 3 3
5083 148/300 Read block 1473 len 32768
5084 149/300 Write block 4679 len 28672 data: 3 4 4 4 4 4 4
5085 150/300 Read block 3397 len 8192
5086 151/300 Read block 662 len 12288
5087 152/300 Read block 41 len 24576
5088 153/300 Read block 771 len 16384
5089 154/300 Read block 4545 len 20480
5090 155/300 Write block 1900 len 4096 data: 4
5091 156/300 Read block 3388 len 40960
5092 157/300 Write block 441 len 28672 data: 4 4 4 3 2 2 2
5093 158/300 Write block 1863 len 40960 data: 4 3 3 3 3 2 2 3 3 3
5094 159/300 Write block 3210 len 24576 data: 3 3 3 2 2 2
5095 160/300 Write block 380 len 4096 data: 2
5096 161/300 Write block 2091 len 32768 data: 3 2 2 2 2 3 4 4
5097 162/300 Read block 2814 len 32768
5098 163/300 Read block 4238 len 4096
5099 164/300 Read block 4853 len 12288
5100 165/300 Read block 1272 len 32768
5101 166/300 Write block 3524 len 8192 data: 3 4
5102 167/300 Write block 3468 len 4096 data: 3
5103 168/300 Read block 2859 len 36864
5104 169/300 Read block 1920 len 36864
5105 170/300 Read block 2062 len 28672
5106 171/300 Write block 695 len 36864 data: 2 2 2 2 2 2 3 3 3
5107 172/300 Write block 2322 len 12288 data: 4 4 4
5108 173/300 Write block 1548 len 40960 data: 3 3 4 4 4 4 5 5 4 4
5109 174/300 Read block 325 len 36864
5110 175/300 Read block 3743 len 40960
5111 176/300 Read block 3085 len 24576
5112 177/300 Write block 4413 len 32768 data: 4 4 4 4 4 4 3 3
5113 178/300 Read block 4230 len 40960
5114 179/300 Read block 3512 len 16384
5115 180/300 Write block 1687 len 24576 data: 3 4 3 3 3 3
5116 181/300 Write block 3569 len 36864 data: 2 2 2 2 2 2 2 2 2
5117 182/300 Read block 3896 len 8192
5118 183/300 Read block 4649 len 36864
5119 184/300 Write block 672 len 24576 data: 2 2 2 2 2 2
5120 185/300 Write block 912 len 32768 data: 2 4 4 4 4 4 5 5
5121 186/300 Read block 795 len 28672
5122 187/300 Flush
5123 188/300 Write block 3304 len 12288 data: 2 2 3
5124 189/300 Write block 4915 len 24576 data: 6 5 4 4 4 2
5125 190/300 Read block 1098 len 40960
5126 191/300 Write block 3804 len 16384 data: 3 4 4 4
5127 192/300 Read block 540 len 20480
5128 193/300 Write block 2197 len 24576 data: 3 3 3 2 2 2
5129 194/300 Read block 4408 len 40960
5130 195/300 Flush
5131 196/300 Write block 1150 len 20480 data: 4 4 4 4 4
5132 197/300 Read block 4353 len 12288
5133 198/300 Write block 2328 len 4096 data: 4
5134 199/300 Read block 833 len 28672
5135 200/300 Flush
5136 201/300 Read block 412 len 24576
5137 202/300 Read block 3857 len 28672
5138 203/300 Write block 4114 len 12288 data: 3 3 3
5139 204/300 Write block 2987 len 40960 data: 3 3 3 2 2 2 2 2 2 2
5140 205/300 Write block 4263 len 20480 data: 3 3 3 3 2
5141 206/300 Write block 1757 len 8192 data: 2 2
5142 207/300 Flush
5143 208/300 Write block 4465 len 24576 data: 2 2 2 2 2 2
5144 209/300 Read block 1889 len 12288
5145 210/300 Write block 2097 len 24576 data: 5 5 4 4 5 3
5146 211/300 Write block 3958 len 24576 data: 3 3 3 3 3 2
5147 212/300 Read block 4112 len 8192
5148 213/300 Write block 2795 len 40960 data: 5 5 5 4 3 2 2 2 2 2
5149 214/300 Write block 18 len 36864 data: 3 3 3 3 2 2 2 2 2
5150 215/300 Flush
5151 216/300 Write block 2483 len 16384 data: 2 2 2 2
5152 217/300 Read block 677 len 40960
5153 218/300 Read block 3969 len 32768
5154 219/300 Flush
5155 220/300 Read block 1265 len 28672
5156 221/300 Write block 1445 len 32768 data: 3 2 2 2 2 3 3 3
5157 222/300 Read block 3764 len 8192
5158 223/300 Write block 440 len 40960 data: 5 5 5 5 4 3 3 3 2 2
5159 224/300 Read block 1761 len 40960
5160 225/300 Read block 4825 len 28672
5161 226/300 Write block 1751 len 20480 data: 2 2 2 2 2
5162 227/300 Write block 3975 len 8192 data: 2 2
5163 228/300 Write block 1535 len 20480 data: 5 4 4 4 3
5164 229/300 Write block 1531 len 12288 data: 6 5 5
5165 230/300 Read block 3815 len 36864
5166 231/300 Read block 3748 len 16384
5167 232/300 Flush
5168 233/300 Write block 2372 len 40960 data: 4 4 4 3 3 4 4 3 3 3
5169 234/300 Write block 1399 len 36864 data: 3 3 4 4 4 4 3 3 2
5170 235/300 Write block 1252 len 8192 data: 2 2
5171 236/300 Write block 3530 len 28672 data: 4 4 4 4 3 2 2
5172 237/300 Read block 150 len 12288
5173 238/300 Read block 70 len 24576
5174 239/300 Read block 2163 len 40960
5175 240/300 Flush
5176 241/300 Read block 2529 len 28672
5177 242/300 Flush
5178 243/300 Flush
5179 244/300 Read block 3500 len 20480
5180 245/300 Read block 1622 len 12288
5181 246/300 Write block 2610 len 28672 data: 2 3 3 4 4 4 4
5182 247/300 Read block 3208 len 16384
5183 248/300 Read block 1261 len 4096
5184 249/300 Flush
5185 250/300 Write block 3426 len 4096 data: 4
5186 251/300 Flush
5187 252/300 Read block 646 len 24576
5188 253/300 Flush
5189 254/300 Write block 1061 len 16384 data: 2 2 2 2
5190 255/300 Write block 829 len 12288 data: 2 3 3
5191 256/300 Read block 1538 len 20480
5192 257/300 Read block 4932 len 40960
5193 258/300 Write block 3802 len 8192 data: 4 3
5194 259/300 Read block 868 len 24576
5195 260/300 Read block 3054 len 16384
5196 261/300 Read block 2953 len 16384
5197 262/300 Write block 3527 len 16384 data: 4 4 4 5
5198 263/300 Read block 2989 len 28672
5199 264/300 Read block 4925 len 4096
5200 265/300 Write block 2769 len 12288 data: 3 2 2
5201 266/300 Write block 4724 len 20480 data: 3 3 3 3 3
5202 267/300 Read block 4744 len 16384
5203 268/300 Flush
5204 269/300 Flush
5205 270/300 Write block 1452 len 4096 data: 4
5206 271/300 Read block 1621 len 28672
5207 272/300 Read block 2063 len 8192
5208 273/300 Read block 246 len 8192
5209 274/300 Write block 4679 len 24576 data: 4 5 5 5 5 5
5210 275/300 Read block 3037 len 4096
5211 276/300 Write block 718 len 32768 data: 3 2 2 2 2 2 2 2
5212 277/300 Write block 54 len 32768 data: 3 3 3 3 2 2 2 2
5213 278/300 Read block 2277 len 16384
5214 279/300 Read block 902 len 8192
5215 280/300 Read block 3079 len 36864
5216 281/300 Read block 2578 len 12288
5217 282/300 Read block 1353 len 32768
5218 283/300 Write block 4722 len 8192 data: 3 3
5219 284/300 Read block 1230 len 20480
5220 285/300 Flush
5221 286/300 Read block 2840 len 20480
5222 287/300 Flush
5223 288/300 Read block 1431 len 32768
5224 289/300 Read block 1250 len 32768
5225 290/300 Read block 1745 len 24576
5226 291/300 Write block 2091 len 20480 data: 4 3 3 3 3
5227 292/300 Write block 3682 len 36864 data: 3 3 3 3 3 2 2 2 2
5228 293/300 Read block 3148 len 24576
5229 294/300 Read block 623 len 8192
5230 295/300 Read block 1581 len 28672
5231 296/300 Flush
5232 297/300 Read block 4618 len 20480
5233 298/300 Write block 4494 len 24576 data: 2 3 3 5 5 5
5234 299/300 Read block 1473 len 4096
5235 300/300 Read block 995 len 12288
5236 Main task has work to do, go find it
5237 got dsc Start(2)
5238 start 2
5239 Oct 13 04:42:55.856 INFO request completed, latency_us: 286, response_code: 204, uri: /start/cid/2, method: POST, req_id: 9b9a8da9-54e8-42e9-b6d1-296a171328f4, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
5240 Replay: started 2, returned:Ok(())
5241 ----------------------------------------------------------------
5242 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
5243 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
5244 1818 Acked 2817 Read 10 Done Done New false
5245 1819 Acked 2818 Flush 0 Done Done New false
5246 1820 Acked 2819 Write 1 Done Done New false
5247 1821 Acked 2820 Write 10 Done Done New false
5248 1822 Acked 2821 Read 8 Done Done New false
5249 1823 Acked 2822 Read 1 Done Done New false
5250 1824 Acked 2823 Flush 0 Done Done New false
5251 1825 Acked 2824 Read 5 Done Done New false
5252 1826 Acked 2825 Read 9 Done Done New false
5253 1827 Acked 2826 Read 6 Done Done New false
5254 1828 Acked 2827 Read 9 Done Done New false
5255 1829 Acked 2828 Read 5 Done Done New false
5256 1830 Acked 2829 Flush 0 Done Done New false
5257 1831 Acked 2830 Write 5 Done Done New false
5258 1832 Acked 2831 Read 1 Done Done New false
5259 1833 Acked 2832 Write 6 Done Done New false
5260 1834 Acked 2833 Read 2 Done Done New false
5261 1835 Acked 2834 Read 7 Done Done New false
5262 1836 Acked 2835 Write 7 Done Done New false
5263 1837 Acked 2836 Read 10 Done Done New false
5264 1838 Acked 2837 Write 6 Done Done New false
5265 1839 Acked 2838 Read 10 Done Done New false
5266 1840 Acked 2839 Read 1 Done Done New false
5267 1841 Acked 2840 Read 1 Done Done New false
5268 1842 Acked 2841 Write 2 Done Done New false
5269 1843 Acked 2842 Write 2 Done Done New false
5270 1844 Acked 2843 Write 3 Done Done New false
5271 1845 Acked 2844 Write 2 Done Done New false
5272 1846 Acked 2845 Write 7 Done Done New false
5273 1847 Acked 2846 Write 3 Done Done New false
5274 1848 Acked 2847 Read 3 Done Done New false
5275 1849 Acked 2848 Write 4 Done Done New false
5276 1850 Acked 2849 Write 3 Done Done New false
5277 1851 Acked 2850 Read 2 Done Done New false
5278 1852 Acked 2851 Write 5 Done Done New false
5279 1853 Acked 2852 Read 10 Done Done New false
5280 1854 Acked 2853 Read 1 Done Done New false
5281 1855 Acked 2854 Write 4 Done Done New false
5282 1856 Acked 2855 Read 3 Done Done New false
5283 1857 Acked 2856 Read 4 Done Done New false
5284 1858 Acked 2857 Flush 0 Done Done New false
5285 1859 Acked 2858 Write 4 Done Done New false
5286 1860 Acked 2859 Read 3 Done Done New false
5287 1861 Acked 2860 Flush 0 Done Done New false
5288 1862 Acked 2861 Flush 0 Done Done New false
5289 1863 Acked 2862 Flush 0 Done Done New false
5290 1864 Acked 2863 Write 7 Done Done New false
5291 1865 Acked 2864 Write 3 Done Done New false
5292 1866 Acked 2865 Read 10 Done Done New false
5293 1867 Acked 2866 Read 6 Done Done New false
5294 1868 Acked 2867 Write 9 Done Done New false
5295 1869 Acked 2868 Read 3 Done Done New false
5296 1870 Acked 2869 Read 3 Done Done New false
5297 1871 Acked 2870 Read 10 Done Done New false
5298 1872 Acked 2871 Write 3 Done Done New false
5299 1873 Acked 2872 Read 5 Done Done New false
5300 1874 Acked 2873 Read 6 Done Done New false
5301 1875 Acked 2874 Read 2 Done Done New false
5302 1876 Acked 2875 Read 3 Done Done New false
5303 1877 Acked 2876 Read 3 Done Done New false
5304 1878 Acked 2877 Write 10 Done Done New false
5305 1879 Acked 2878 Write 5 Done Done New false
5306 1880 Acked 2879 Write 5 Done Done New false
5307 1881 Acked 2880 Read 9 Done Done New false
5308 1882 Acked 2881 Write 5 Done Done New false
5309 1883 Acked 2882 Read 2 Done Done New false
5310 1884 Acked 2883 Write 2 Done Done New false
5311 1885 Acked 2884 Read 10 Done Done New false
5312 1886 Acked 2885 Read 6 Done Done New false
5313 1887 Acked 2886 Write 10 Done Done New false
5314 1888 Acked 2887 Read 9 Done Done New false
5315 1889 Acked 2888 Write 8 Done Done New false
5316 1890 Acked 2889 Read 6 Done Done New false
5317 1891 Acked 2890 Write 8 Done Done New false
5318 1892 Acked 2891 Read 8 Done Done New false
5319 1893 Acked 2892 Read 2 Done Done New false
5320 1894 Acked 2893 Write 10 Done Done New false
5321 1895 Acked 2894 Read 10 Done Done New false
5322 1896 Acked 2895 Read 9 Done Done New false
5323 1897 Acked 2896 Read 3 Done Done New false
5324 1898 Acked 2897 Write 1 Done Done New false
5325 1899 Acked 2898 Read 3 Done Done New false
5326 1900 Acked 2899 Read 6 Done Done New false
5327 1901 Acked 2900 Write 5 Done Done New false
5328 1902 Acked 2901 Read 8 Done Done New false
5329 1903 Acked 2902 Read 5 Done Done New false
5330 1904 Acked 2903 Read 4 Done Done New false
5331 1905 Acked 2904 Read 8 Done Done New false
5332 1906 Acked 2905 Read 8 Done Done New false
5333 1907 Acked 2906 Write 5 Done Done New false
5334 1908 Acked 2907 Read 7 Done Done New false
5335 1909 Acked 2908 Write 1 Done Done New false
5336 1910 Acked 2909 Read 10 Done Done New false
5337 1911 Acked 2910 Read 3 Done Done New false
5338 1912 Acked 2911 Read 2 Done Done New false
5339 1913 Acked 2912 Read 2 Done Done New false
5340 1914 Acked 2913 Read 2 Done Done New false
5341 1915 Acked 2914 Read 2 Done Done New false
5342 1916 Acked 2915 Write 2 Done Done New false
5343 1917 Acked 2916 Read 3 Done Done New false
5344 1918 Acked 2917 Read 6 Done Done New false
5345 1919 Acked 2918 Write 7 Done Done New false
5346 1920 Acked 2919 Write 8 Done Done New false
5347 1921 Acked 2920 Write 9 Done Done New false
5348 1922 Acked 2921 Read 8 Done Done New false
5349 1923 Acked 2922 Read 10 Done Done New false
5350 1924 Acked 2923 Read 4 Done Done New false
5351 1925 Acked 2924 Flush 0 Done Done New false
5352 1926 Acked 2925 Flush 0 Done Done New false
5353 1927 Acked 2926 Write 1 Done Done New false
5354 1928 Acked 2927 Read 1 Done Done New false
5355 1929 Acked 2928 Read 10 Done Done New false
5356 1930 Acked 2929 Read 3 Done Done New false
5357 1931 Acked 2930 Write 8 Done Done New false
5358 1932 Acked 2931 Write 3 Done Done New false
5359 1933 Acked 2932 Flush 0 Done Done New false
5360 1934 Acked 2933 Read 5 Done Done New false
5361 1935 Acked 2934 Read 7 Done Done New false
5362 1936 Acked 2935 Read 3 Done Done New false
5363 1937 Acked 2936 Write 8 Done Done New false
5364 1938 Acked 2937 Read 2 Done Done New false
5365 1939 Acked 2938 Flush 0 Done Done New false
5366 1940 Acked 2939 Read 5 Done Done New false
5367 1941 Acked 2940 Read 3 Done Done New false
5368 1942 Acked 2941 Read 10 Done Done New false
5369 1943 Acked 2942 Write 10 Done Done New false
5370 1944 Acked 2943 Read 10 Done Done New false
5371 1945 Acked 2944 Read 1 Done Done New false
5372 1946 Acked 2945 Write 4 Done Done New false
5373 1947 Acked 2946 Read 10 Done Done New false
5374 1948 Acked 2947 Write 5 Done Done New false
5375 1949 Acked 2948 Write 7 Done Done New false
5376 1950 Acked 2949 Write 8 Done Done New false
5377 1951 Acked 2950 Flush 0 Done Done New false
5378 1952 Acked 2951 Read 10 Done Done New false
5379 1953 Acked 2952 Write 5 Done Done New false
5380 1954 Acked 2953 Write 8 Done Done New false
5381 1955 Acked 2954 Read 1 Done Done New false
5382 1956 Acked 2955 Read 6 Done Done New false
5383 1957 Acked 2956 Write 5 Done Done New false
5384 1958 Acked 2957 Read 3 Done Done New false
5385 1959 Acked 2958 Write 6 Done Done New false
5386 1960 Acked 2959 Write 8 Done Done New false
5387 1961 Acked 2960 Read 9 Done Done New false
5388 1962 Acked 2961 Write 4 Done Done New false
5389 1963 Acked 2962 Write 1 Done Done New false
5390 1964 Acked 2963 Write 10 Done Done New false
5391 1965 Acked 2964 Read 8 Done Done New false
5392 1966 Acked 2965 Flush 0 Done Done New false
5393 1967 Acked 2966 Write 7 Done Done New false
5394 1968 Acked 2967 Read 2 Done Done New false
5395 1969 Acked 2968 Read 3 Done Done New false
5396 1970 Acked 2969 Read 6 Done Done New false
5397 1971 Acked 2970 Read 4 Done Done New false
5398 1972 Acked 2971 Read 5 Done Done New false
5399 1973 Acked 2972 Write 1 Done Done New false
5400 1974 Acked 2973 Read 10 Done Done New false
5401 1975 Acked 2974 Write 7 Done Done New false
5402 1976 Acked 2975 Write 10 Done Done New false
5403 1977 Acked 2976 Write 6 Done Done New false
5404 1978 Acked 2977 Write 1 Done Done New false
5405 1979 Acked 2978 Write 8 Done Done New false
5406 1980 Acked 2979 Read 8 Done Done New false
5407 1981 Acked 2980 Read 1 Done Done New false
5408 1982 Acked 2981 Read 3 Done Done New false
5409 1983 Acked 2982 Read 8 Done Done New false
5410 1984 Acked 2983 Write 2 Done Done New false
5411 1985 Acked 2984 Write 1 Done Done New false
5412 1986 Acked 2985 Read 9 Done Done New false
5413 1987 Acked 2986 Read 9 Done Done New false
5414 1988 Acked 2987 Read 7 Done Done New false
5415 1989 Acked 2988 Write 9 Done Done New false
5416 1990 Acked 2989 Write 3 Done Done New false
5417 1991 Acked 2990 Write 10 Done Done New false
5418 1992 Acked 2991 Read 9 Done Done New false
5419 1993 Acked 2992 Read 10 Done Done New false
5420 1994 Acked 2993 Read 6 Done Done New false
5421 1995 Acked 2994 Write 8 Done Done New false
5422 1996 Acked 2995 Read 10 Done Done New false
5423 1997 Acked 2996 Read 4 Done Done New false
5424 1998 Acked 2997 Write 6 Done Done New false
5425 1999 Acked 2998 Write 9 Done Done New false
5426 2000 Acked 2999 Read 2 Done Done New false
5427 2001 Acked 3000 Read 9 Done Done New false
5428 2002 Acked 3001 Write 6 Done Done New false
5429 2003 Acked 3002 Write 8 Done Done New false
5430 2004 Acked 3003 Read 7 Done Done New false
5431 2005 Acked 3004 Flush 0 Done Done New false
5432 2006 Acked 3005 Write 3 Done Done New false
5433 2007 Acked 3006 Write 6 Done Done New false
5434 2008 Acked 3007 Read 10 Done Done New false
5435 2009 Acked 3008 Write 4 Done Done New false
5436 2010 Acked 3009 Read 5 Done Done New false
5437 2011 Acked 3010 Write 6 Done Done New false
5438 2012 Acked 3011 Read 10 Done Done New false
5439 2013 Acked 3012 Flush 0 Done Done New false
5440 2014 Acked 3013 Write 5 Done Done New false
5441 2015 Acked 3014 Read 3 Done Done New false
5442 2016 Acked 3015 Write 1 Done Done New false
5443 2017 Acked 3016 Read 7 Done Done New false
5444 2018 Acked 3017 Flush 0 Done Done New false
5445 2019 Acked 3018 Read 6 Done Done New false
5446 2020 Acked 3019 Read 7 Done Done New false
5447 2021 Acked 3020 Write 3 Done Done New false
5448 2022 Acked 3021 Write 10 Done Done New false
5449 2023 Acked 3022 Write 5 Done Done New false
5450 2024 Acked 3023 Write 2 Done Done New false
5451 2025 Acked 3024 Flush 0 Done Done New false
5452 2026 Acked 3025 Write 6 Done Done New false
5453 2027 Acked 3026 Read 3 Done Done New false
5454 2028 Acked 3027 Write 6 Done Done New false
5455 2029 Acked 3028 Write 6 Done Done New false
5456 2030 Acked 3029 Read 2 Done Done New false
5457 2031 Acked 3030 Write 10 Done Done New false
5458 2032 Acked 3031 Write 9 Done Done New false
5459 2033 Acked 3032 Flush 0 Done Done New false
5460 2034 Acked 3033 Write 4 Done Done New false
5461 2035 Acked 3034 Read 10 Done Done New false
5462 2036 Acked 3035 Read 8 Done Done New false
5463 2037 Acked 3036 Flush 0 Done Done New false
5464 2038 Acked 3037 Read 7 Done Done New false
5465 2039 Acked 3038 Write 8 Done Done New false
5466 2040 Acked 3039 Read 2 Done Done New false
5467 2041 Acked 3040 Write 10 Done Done New false
5468 2042 Acked 3041 Read 10 Done Done New false
5469 2043 Acked 3042 Read 7 Done Done New false
5470 2044 Acked 3043 Write 5 Done Done New false
5471 2045 Acked 3044 Write 2 Done Done New false
5472 2046 Acked 3045 Write 5 Done Done New false
5473 2047 Acked 3046 Write 3 Done Done New false
5474 2048 Acked 3047 Read 9 Done Done New false
5475 2049 Acked 3048 Read 4 Done Done New false
5476 2050 Acked 3049 Flush 0 Done Done New false
5477 2051 Acked 3050 Write 10 Done Done New false
5478 2052 Acked 3051 Write 9 Done Done New false
5479 2053 Acked 3052 Write 2 Done Done New false
5480 2054 Acked 3053 Write 7 Done Done New false
5481 2055 Acked 3054 Read 3 Done Done New false
5482 2056 Acked 3055 Read 6 Done Done New false
5483 2057 Acked 3056 Read 10 Done Done New false
5484 2058 Acked 3057 Flush 0 Done Done New false
5485 2059 Acked 3058 Read 7 Done Done New false
5486 2060 Acked 3059 Flush 0 Done Done New false
5487 2061 Acked 3060 Flush 0 Done Done New false
5488 2062 Acked 3061 Read 5 Done Done New false
5489 2063 Acked 3062 Read 3 Done Done New false
5490 2064 Acked 3063 Write 7 Done Done New false
5491 2065 Acked 3064 Read 4 Done Done New false
5492 2066 Acked 3065 Read 1 Done Done New false
5493 2067 Acked 3066 Flush 0 Done Done New false
5494 2068 Acked 3067 Write 1 Done Done New false
5495 2069 Acked 3068 Flush 0 Done Done New false
5496 2070 Acked 3069 Read 6 Done Done New false
5497 2071 Acked 3070 Flush 0 Done Done New false
5498 2072 Acked 3071 Write 4 Done Done New false
5499 2073 Acked 3072 Write 3 Done Done New false
5500 2074 Acked 3073 Read 5 Done Done New false
5501 2075 Acked 3074 Read 10 Done Done New false
5502 2076 Acked 3075 Write 2 Done Done New false
5503 2077 Acked 3076 Read 6 Done Done New false
5504 2078 Acked 3077 Read 4 Done Done New false
5505 2079 Acked 3078 Read 4 Done Done New false
5506 2080 Acked 3079 Write 4 Done Done New false
5507 2081 Acked 3080 Read 7 Done Done New false
5508 2082 Acked 3081 Read 1 Done Done New false
5509 2083 Acked 3082 Write 3 Done Done New false
5510 2084 Acked 3083 Write 5 Done Done New false
5511 2085 Acked 3084 Read 4 Done Done New false
5512 2086 Acked 3085 Flush 0 Done Done New false
5513 2087 Acked 3086 Flush 0 Done Done New false
5514 2088 Acked 3087 Write 1 Done Done New false
5515 2089 Acked 3088 Read 7 Done Done New false
5516 2090 Acked 3089 Read 2 Done Done New false
5517 2091 Acked 3090 Read 2 Done Done New false
5518 2092 Acked 3091 Write 6 Done Done New false
5519 2093 Acked 3092 Read 1 Done Done New false
5520 2094 Acked 3093 Write 8 Done Done New false
5521 2095 Acked 3094 Write 8 Done Done New false
5522 2096 Acked 3095 Read 4 Done Done New false
5523 2097 Acked 3096 Read 2 Done Done New false
5524 2098 Acked 3097 Read 9 Done Done New false
5525 2099 Acked 3098 Read 3 Done Done New false
5526 2100 Acked 3099 Read 8 Done Done New false
5527 2101 Acked 3100 Write 2 Done Done New false
5528 2102 Acked 3101 Read 5 Done Done New false
5529 2103 Acked 3102 Flush 0 Done Done New false
5530 2104 Acked 3103 Read 5 Done Done New false
5531 2105 Acked 3104 Flush 0 Done Done New false
5532 2106 Acked 3105 Read 8 Done Done New false
5533 2107 Acked 3106 Read 8 Done Done New false
5534 2108 Acked 3107 Read 6 Done Done New false
5535 2109 Acked 3108 Write 5 Done Done New false
5536 2110 Acked 3109 Write 9 Done Done New false
5537 2111 Acked 3110 Read 6 Done Done New false
5538 2112 Acked 3111 Read 2 Done Done New false
5539 2113 Acked 3112 Read 7 Done Done New false
5540 2114 Acked 3113 Flush 0 Done Done New false
5541 2115 Acked 3114 Read 5 Done Done New false
5542 2116 Acked 3115 Write 6 Done Done New false
5543 2117 Acked 3116 Read 1 Done Done New false
5544 2118 Acked 3117 Read 3 Done Done New false
5545 STATES DS:0 DS:1 DS:2 TOTAL
5546 New 0 0 301 301
5547 Sent 0 0 0 0
5548 Done 301 301 0 602
5549 Skipped 0 0 0 0
5550 Error 0 0 0 0
5551 Last Flush: 3113 3113 2816
5552 Downstairs last five completed: 2816 2815 2814 2813 2812 2811
5553 Upstairs last five completed: 2118 2117 2116 2115 2114 2113
5554 CLIENT: Up:0 ds:301 act:2
5555 [8830] got start action
5556 [8830] I am going to restart
5557 Starting downstairs at port 8830
5558 Make output file at "/tmp/dsc/downstairs-8830.txt"
5559 [8830][2] reports Starting
5560 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1183)
5561 [8830][2] reports Running
5562 ----------------------------------------------------------------
5563 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
5564 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
5565 1818 Acked 2817 Read 10 Done Done New false
5566 1819 Acked 2818 Flush 0 Done Done New false
5567 1820 Acked 2819 Write 1 Done Done New false
5568 1821 Acked 2820 Write 10 Done Done New false
5569 1822 Acked 2821 Read 8 Done Done New false
5570 1823 Acked 2822 Read 1 Done Done New false
5571 1824 Acked 2823 Flush 0 Done Done New false
5572 1825 Acked 2824 Read 5 Done Done New false
5573 1826 Acked 2825 Read 9 Done Done New false
5574 1827 Acked 2826 Read 6 Done Done New false
5575 1828 Acked 2827 Read 9 Done Done New false
5576 1829 Acked 2828 Read 5 Done Done New false
5577 1830 Acked 2829 Flush 0 Done Done New false
5578 1831 Acked 2830 Write 5 Done Done New false
5579 1832 Acked 2831 Read 1 Done Done New false
5580 1833 Acked 2832 Write 6 Done Done New false
5581 1834 Acked 2833 Read 2 Done Done New false
5582 1835 Acked 2834 Read 7 Done Done New false
5583 1836 Acked 2835 Write 7 Done Done New false
5584 1837 Acked 2836 Read 10 Done Done New false
5585 1838 Acked 2837 Write 6 Done Done New false
5586 1839 Acked 2838 Read 10 Done Done New false
5587 1840 Acked 2839 Read 1 Done Done New false
5588 1841 Acked 2840 Read 1 Done Done New false
5589 1842 Acked 2841 Write 2 Done Done New false
5590 1843 Acked 2842 Write 2 Done Done New false
5591 1844 Acked 2843 Write 3 Done Done New false
5592 1845 Acked 2844 Write 2 Done Done New false
5593 1846 Acked 2845 Write 7 Done Done New false
5594 1847 Acked 2846 Write 3 Done Done New false
5595 1848 Acked 2847 Read 3 Done Done New false
5596 1849 Acked 2848 Write 4 Done Done New false
5597 1850 Acked 2849 Write 3 Done Done New false
5598 1851 Acked 2850 Read 2 Done Done New false
5599 1852 Acked 2851 Write 5 Done Done New false
5600 1853 Acked 2852 Read 10 Done Done New false
5601 1854 Acked 2853 Read 1 Done Done New false
5602 1855 Acked 2854 Write 4 Done Done New false
5603 1856 Acked 2855 Read 3 Done Done New false
5604 1857 Acked 2856 Read 4 Done Done New false
5605 1858 Acked 2857 Flush 0 Done Done New false
5606 1859 Acked 2858 Write 4 Done Done New false
5607 1860 Acked 2859 Read 3 Done Done New false
5608 1861 Acked 2860 Flush 0 Done Done New false
5609 1862 Acked 2861 Flush 0 Done Done New false
5610 1863 Acked 2862 Flush 0 Done Done New false
5611 1864 Acked 2863 Write 7 Done Done New false
5612 1865 Acked 2864 Write 3 Done Done New false
5613 1866 Acked 2865 Read 10 Done Done New false
5614 1867 Acked 2866 Read 6 Done Done New false
5615 1868 Acked 2867 Write 9 Done Done New false
5616 1869 Acked 2868 Read 3 Done Done New false
5617 1870 Acked 2869 Read 3 Done Done New false
5618 1871 Acked 2870 Read 10 Done Done New false
5619 1872 Acked 2871 Write 3 Done Done New false
5620 1873 Acked 2872 Read 5 Done Done New false
5621 1874 Acked 2873 Read 6 Done Done New false
5622 1875 Acked 2874 Read 2 Done Done New false
5623 1876 Acked 2875 Read 3 Done Done New false
5624 1877 Acked 2876 Read 3 Done Done New false
5625 1878 Acked 2877 Write 10 Done Done New false
5626 1879 Acked 2878 Write 5 Done Done New false
5627 1880 Acked 2879 Write 5 Done Done New false
5628 1881 Acked 2880 Read 9 Done Done New false
5629 1882 Acked 2881 Write 5 Done Done New false
5630 1883 Acked 2882 Read 2 Done Done New false
5631 1884 Acked 2883 Write 2 Done Done New false
5632 1885 Acked 2884 Read 10 Done Done New false
5633 1886 Acked 2885 Read 6 Done Done New false
5634 1887 Acked 2886 Write 10 Done Done New false
5635 1888 Acked 2887 Read 9 Done Done New false
5636 1889 Acked 2888 Write 8 Done Done New false
5637 1890 Acked 2889 Read 6 Done Done New false
5638 1891 Acked 2890 Write 8 Done Done New false
5639 1892 Acked 2891 Read 8 Done Done New false
5640 1893 Acked 2892 Read 2 Done Done New false
5641 1894 Acked 2893 Write 10 Done Done New false
5642 1895 Acked 2894 Read 10 Done Done New false
5643 1896 Acked 2895 Read 9 Done Done New false
5644 1897 Acked 2896 Read 3 Done Done New false
5645 1898 Acked 2897 Write 1 Done Done New false
5646 1899 Acked 2898 Read 3 Done Done New false
5647 1900 Acked 2899 Read 6 Done Done New false
5648 1901 Acked 2900 Write 5 Done Done New false
5649 1902 Acked 2901 Read 8 Done Done New false
5650 1903 Acked 2902 Read 5 Done Done New false
5651 1904 Acked 2903 Read 4 Done Done New false
5652 1905 Acked 2904 Read 8 Done Done New false
5653 1906 Acked 2905 Read 8 Done Done New false
5654 1907 Acked 2906 Write 5 Done Done New false
5655 1908 Acked 2907 Read 7 Done Done New false
5656 1909 Acked 2908 Write 1 Done Done New false
5657 1910 Acked 2909 Read 10 Done Done New false
5658 1911 Acked 2910 Read 3 Done Done New false
5659 1912 Acked 2911 Read 2 Done Done New false
5660 1913 Acked 2912 Read 2 Done Done New false
5661 1914 Acked 2913 Read 2 Done Done New false
5662 1915 Acked 2914 Read 2 Done Done New false
5663 1916 Acked 2915 Write 2 Done Done New false
5664 1917 Acked 2916 Read 3 Done Done New false
5665 1918 Acked 2917 Read 6 Done Done New false
5666 1919 Acked 2918 Write 7 Done Done New false
5667 1920 Acked 2919 Write 8 Done Done New false
5668 1921 Acked 2920 Write 9 Done Done New false
5669 1922 Acked 2921 Read 8 Done Done New false
5670 1923 Acked 2922 Read 10 Done Done New false
5671 1924 Acked 2923 Read 4 Done Done New false
5672 1925 Acked 2924 Flush 0 Done Done New false
5673 1926 Acked 2925 Flush 0 Done Done New false
5674 1927 Acked 2926 Write 1 Done Done New false
5675 1928 Acked 2927 Read 1 Done Done New false
5676 1929 Acked 2928 Read 10 Done Done New false
5677 1930 Acked 2929 Read 3 Done Done New false
5678 1931 Acked 2930 Write 8 Done Done New false
5679 1932 Acked 2931 Write 3 Done Done New false
5680 1933 Acked 2932 Flush 0 Done Done New false
5681 1934 Acked 2933 Read 5 Done Done New false
5682 1935 Acked 2934 Read 7 Done Done New false
5683 1936 Acked 2935 Read 3 Done Done New false
5684 1937 Acked 2936 Write 8 Done Done New false
5685 1938 Acked 2937 Read 2 Done Done New false
5686 1939 Acked 2938 Flush 0 Done Done New false
5687 1940 Acked 2939 Read 5 Done Done New false
5688 1941 Acked 2940 Read 3 Done Done New false
5689 1942 Acked 2941 Read 10 Done Done New false
5690 1943 Acked 2942 Write 10 Done Done New false
5691 1944 Acked 2943 Read 10 Done Done New false
5692 1945 Acked 2944 Read 1 Done Done New false
5693 1946 Acked 2945 Write 4 Done Done New false
5694 1947 Acked 2946 Read 10 Done Done New false
5695 1948 Acked 2947 Write 5 Done Done New false
5696 1949 Acked 2948 Write 7 Done Done New false
5697 1950 Acked 2949 Write 8 Done Done New false
5698 1951 Acked 2950 Flush 0 Done Done New false
5699 1952 Acked 2951 Read 10 Done Done New false
5700 1953 Acked 2952 Write 5 Done Done New false
5701 1954 Acked 2953 Write 8 Done Done New false
5702 1955 Acked 2954 Read 1 Done Done New false
5703 1956 Acked 2955 Read 6 Done Done New false
5704 1957 Acked 2956 Write 5 Done Done New false
5705 1958 Acked 2957 Read 3 Done Done New false
5706 1959 Acked 2958 Write 6 Done Done New false
5707 1960 Acked 2959 Write 8 Done Done New false
5708 1961 Acked 2960 Read 9 Done Done New false
5709 1962 Acked 2961 Write 4 Done Done New false
5710 1963 Acked 2962 Write 1 Done Done New false
5711 1964 Acked 2963 Write 10 Done Done New false
5712 1965 Acked 2964 Read 8 Done Done New false
5713 1966 Acked 2965 Flush 0 Done Done New false
5714 1967 Acked 2966 Write 7 Done Done New false
5715 1968 Acked 2967 Read 2 Done Done New false
5716 1969 Acked 2968 Read 3 Done Done New false
5717 1970 Acked 2969 Read 6 Done Done New false
5718 1971 Acked 2970 Read 4 Done Done New false
5719 1972 Acked 2971 Read 5 Done Done New false
5720 1973 Acked 2972 Write 1 Done Done New false
5721 1974 Acked 2973 Read 10 Done Done New false
5722 1975 Acked 2974 Write 7 Done Done New false
5723 1976 Acked 2975 Write 10 Done Done New false
5724 1977 Acked 2976 Write 6 Done Done New false
5725 1978 Acked 2977 Write 1 Done Done New false
5726 1979 Acked 2978 Write 8 Done Done New false
5727 1980 Acked 2979 Read 8 Done Done New false
5728 1981 Acked 2980 Read 1 Done Done New false
5729 1982 Acked 2981 Read 3 Done Done New false
5730 1983 Acked 2982 Read 8 Done Done New false
5731 1984 Acked 2983 Write 2 Done Done New false
5732 1985 Acked 2984 Write 1 Done Done New false
5733 1986 Acked 2985 Read 9 Done Done New false
5734 1987 Acked 2986 Read 9 Done Done New false
5735 1988 Acked 2987 Read 7 Done Done New false
5736 1989 Acked 2988 Write 9 Done Done New false
5737 1990 Acked 2989 Write 3 Done Done New false
5738 1991 Acked 2990 Write 10 Done Done New false
5739 1992 Acked 2991 Read 9 Done Done New false
5740 1993 Acked 2992 Read 10 Done Done New false
5741 1994 Acked 2993 Read 6 Done Done New false
5742 1995 Acked 2994 Write 8 Done Done New false
5743 1996 Acked 2995 Read 10 Done Done New false
5744 1997 Acked 2996 Read 4 Done Done New false
5745 1998 Acked 2997 Write 6 Done Done New false
5746 1999 Acked 2998 Write 9 Done Done New false
5747 2000 Acked 2999 Read 2 Done Done New false
5748 2001 Acked 3000 Read 9 Done Done New false
5749 2002 Acked 3001 Write 6 Done Done New false
5750 2003 Acked 3002 Write 8 Done Done New false
5751 2004 Acked 3003 Read 7 Done Done New false
5752 2005 Acked 3004 Flush 0 Done Done New false
5753 2006 Acked 3005 Write 3 Done Done New false
5754 2007 Acked 3006 Write 6 Done Done New false
5755 2008 Acked 3007 Read 10 Done Done New false
5756 2009 Acked 3008 Write 4 Done Done New false
5757 2010 Acked 3009 Read 5 Done Done New false
5758 2011 Acked 3010 Write 6 Done Done New false
5759 2012 Acked 3011 Read 10 Done Done New false
5760 2013 Acked 3012 Flush 0 Done Done New false
5761 2014 Acked 3013 Write 5 Done Done New false
5762 2015 Acked 3014 Read 3 Done Done New false
5763 2016 Acked 3015 Write 1 Done Done New false
5764 2017 Acked 3016 Read 7 Done Done New false
5765 2018 Acked 3017 Flush 0 Done Done New false
5766 2019 Acked 3018 Read 6 Done Done New false
5767 2020 Acked 3019 Read 7 Done Done New false
5768 2021 Acked 3020 Write 3 Done Done New false
5769 2022 Acked 3021 Write 10 Done Done New false
5770 2023 Acked 3022 Write 5 Done Done New false
5771 2024 Acked 3023 Write 2 Done Done New false
5772 2025 Acked 3024 Flush 0 Done Done New false
5773 2026 Acked 3025 Write 6 Done Done New false
5774 2027 Acked 3026 Read 3 Done Done New false
5775 2028 Acked 3027 Write 6 Done Done New false
5776 2029 Acked 3028 Write 6 Done Done New false
5777 2030 Acked 3029 Read 2 Done Done New false
5778 2031 Acked 3030 Write 10 Done Done New false
5779 2032 Acked 3031 Write 9 Done Done New false
5780 2033 Acked 3032 Flush 0 Done Done New false
5781 2034 Acked 3033 Write 4 Done Done New false
5782 2035 Acked 3034 Read 10 Done Done New false
5783 2036 Acked 3035 Read 8 Done Done New false
5784 2037 Acked 3036 Flush 0 Done Done New false
5785 2038 Acked 3037 Read 7 Done Done New false
5786 2039 Acked 3038 Write 8 Done Done New false
5787 2040 Acked 3039 Read 2 Done Done New false
5788 2041 Acked 3040 Write 10 Done Done New false
5789 2042 Acked 3041 Read 10 Done Done New false
5790 2043 Acked 3042 Read 7 Done Done New false
5791 2044 Acked 3043 Write 5 Done Done New false
5792 2045 Acked 3044 Write 2 Done Done New false
5793 2046 Acked 3045 Write 5 Done Done New false
5794 2047 Acked 3046 Write 3 Done Done New false
5795 2048 Acked 3047 Read 9 Done Done New false
5796 2049 Acked 3048 Read 4 Done Done New false
5797 2050 Acked 3049 Flush 0 Done Done New false
5798 2051 Acked 3050 Write 10 Done Done New false
5799 2052 Acked 3051 Write 9 Done Done New false
5800 2053 Acked 3052 Write 2 Done Done New false
5801 2054 Acked 3053 Write 7 Done Done New false
5802 2055 Acked 3054 Read 3 Done Done New false
5803 2056 Acked 3055 Read 6 Done Done New false
5804 2057 Acked 3056 Read 10 Done Done New false
5805 2058 Acked 3057 Flush 0 Done Done New false
5806 2059 Acked 3058 Read 7 Done Done New false
5807 2060 Acked 3059 Flush 0 Done Done New false
5808 2061 Acked 3060 Flush 0 Done Done New false
5809 2062 Acked 3061 Read 5 Done Done New false
5810 2063 Acked 3062 Read 3 Done Done New false
5811 2064 Acked 3063 Write 7 Done Done New false
5812 2065 Acked 3064 Read 4 Done Done New false
5813 2066 Acked 3065 Read 1 Done Done New false
5814 2067 Acked 3066 Flush 0 Done Done New false
5815 2068 Acked 3067 Write 1 Done Done New false
5816 2069 Acked 3068 Flush 0 Done Done New false
5817 2070 Acked 3069 Read 6 Done Done New false
5818 2071 Acked 3070 Flush 0 Done Done New false
5819 2072 Acked 3071 Write 4 Done Done New false
5820 2073 Acked 3072 Write 3 Done Done New false
5821 2074 Acked 3073 Read 5 Done Done New false
5822 2075 Acked 3074 Read 10 Done Done New false
5823 2076 Acked 3075 Write 2 Done Done New false
5824 2077 Acked 3076 Read 6 Done Done New false
5825 2078 Acked 3077 Read 4 Done Done New false
5826 2079 Acked 3078 Read 4 Done Done New false
5827 2080 Acked 3079 Write 4 Done Done New false
5828 2081 Acked 3080 Read 7 Done Done New false
5829 2082 Acked 3081 Read 1 Done Done New false
5830 2083 Acked 3082 Write 3 Done Done New false
5831 2084 Acked 3083 Write 5 Done Done New false
5832 2085 Acked 3084 Read 4 Done Done New false
5833 2086 Acked 3085 Flush 0 Done Done New false
5834 2087 Acked 3086 Flush 0 Done Done New false
5835 2088 Acked 3087 Write 1 Done Done New false
5836 2089 Acked 3088 Read 7 Done Done New false
5837 2090 Acked 3089 Read 2 Done Done New false
5838 2091 Acked 3090 Read 2 Done Done New false
5839 2092 Acked 3091 Write 6 Done Done New false
5840 2093 Acked 3092 Read 1 Done Done New false
5841 2094 Acked 3093 Write 8 Done Done New false
5842 2095 Acked 3094 Write 8 Done Done New false
5843 2096 Acked 3095 Read 4 Done Done New false
5844 2097 Acked 3096 Read 2 Done Done New false
5845 2098 Acked 3097 Read 9 Done Done New false
5846 2099 Acked 3098 Read 3 Done Done New false
5847 2100 Acked 3099 Read 8 Done Done New false
5848 2101 Acked 3100 Write 2 Done Done New false
5849 2102 Acked 3101 Read 5 Done Done New false
5850 2103 Acked 3102 Flush 0 Done Done New false
5851 2104 Acked 3103 Read 5 Done Done New false
5852 2105 Acked 3104 Flush 0 Done Done New false
5853 2106 Acked 3105 Read 8 Done Done New false
5854 2107 Acked 3106 Read 8 Done Done New false
5855 2108 Acked 3107 Read 6 Done Done New false
5856 2109 Acked 3108 Write 5 Done Done New false
5857 2110 Acked 3109 Write 9 Done Done New false
5858 2111 Acked 3110 Read 6 Done Done New false
5859 2112 Acked 3111 Read 2 Done Done New false
5860 2113 Acked 3112 Read 7 Done Done New false
5861 2114 Acked 3113 Flush 0 Done Done New false
5862 2115 Acked 3114 Read 5 Done Done New false
5863 2116 Acked 3115 Write 6 Done Done New false
5864 2117 Acked 3116 Read 1 Done Done New false
5865 2118 Acked 3117 Read 3 Done Done New false
5866 2119 Acked 3118 Flush 0 Done Done New false
5867 STATES DS:0 DS:1 DS:2 TOTAL
5868 New 0 0 302 302
5869 Sent 0 0 0 0
5870 Done 302 302 0 604
5871 Skipped 0 0 0 0
5872 Error 0 0 0 0
5873 Last Flush: 3118 3118 2816
5874 Downstairs last five completed: 2816 2815 2814 2813 2812 2811
5875 Upstairs last five completed: 2119 2118 2117 2116 2115 2114
5876 CLIENT: Up:0 ds:302 act:2
58772023-10-13T04:42:59.961ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 2
58782023-10-13T04:42:59.961ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
58792023-10-13T04:42:59.961ZINFOcrucible: [2] client re-new 302 jobs since flush 2816 = downstairs
58802023-10-13T04:42:59.962ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
58812023-10-13T04:42:59.963ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
58822023-10-13T04:42:59.963ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
58832023-10-13T04:42:59.963ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
58842023-10-13T04:42:59.963ZINFOcrucible: [2] send last flush ID to this DS: 2816
58852023-10-13T04:42:59.963ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
58862023-10-13T04:42:59.963ZINFOcrucible: [2] Replied this last flush ID: 2816
58872023-10-13T04:42:59.963ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Offline ds_transition to Replay
58882023-10-13T04:42:59.963ZINFOcrucible: [2] Transition from Offline to Replay
58892023-10-13T04:42:59.963ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
58902023-10-13T04:42:59.963ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Replay ds_transition to Active
58912023-10-13T04:42:59.963ZINFOcrucible: [2] Transition from Replay to Active
58922023-10-13T04:42:59.963ZINFOcrucible: [2] Starts cmd_loop
58932023-10-13T04:42:59.963ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
58942023-10-13T04:42:59.963ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
58952023-10-13T04:42:59.963ZINFOcrucible: Set check for repair
58962023-10-13T04:43:00.964ZINFOcrucible: Checking if live repair is needed
58972023-10-13T04:43:00.964ZINFOcrucible: No Live Repair required at this time
58982023-10-13T04:43:00.964ZWARNcrucible: [2] flow control end
5899 ----------------------------------------------------------------
5900 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
5901 Downstairs last five completed: 3118 3117 3116 3115 3114 3113
5902 Upstairs last five completed: 2119 2118 2117 2116 2115 2114
5903 CLIENT: Up:0 ds:0 act:3
5904 Replay: All jobs finished, all DS active.
5905 Main task has work to do, go find it
5906 got dsc Stop(1)
5907 stop 1
5908 Oct 13 04:43:03.864 INFO request completed, latency_us: 484, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 7189f5ae-1065-42c4-b59b-9c5c0d98eab2, remote_addr: 127.0.0.1:[8820] Got stop action so:false kr:false
5909 52046, local_addr: 127.0.0.1:9998
5910 Oct 13 04:43:03.865 INFO accepted connection, remote_addr: 127.0.0.1:37069, local_addr: 127.0.0.1:9998
5911 Oct 13 04:43:03.865 INFO request completed, latency_us: 188, response_code: 200, uri: /state/cid/1, method: GET, req_id: f4232560-e919-4d98-814e-0a363cb86cbd, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
59122023-10-13T04:43:03.866ZWARNcrucible: [1] downstairs disconnected
59132023-10-13T04:43:03.866ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
59142023-10-13T04:43:03.866ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8820 closed looper = 1
59152023-10-13T04:43:03.866ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:false
59162023-10-13T04:43:03.866ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
59172023-10-13T04:43:03.866ZINFOcrucible: [1] 127.0.0.1:8820 task reports offline
59182023-10-13T04:43:03.866ZWARNcrucible: [1] pm_task rx.recv() is None
5919 [8820] Exited with: ExitStatus(unix_wait_status(9))
5920 [8820][1] reports Exit
59212023-10-13T04:43:04.867ZINFOcrucible: [1] connecting to 127.0.0.1:8820 looper = 1
5922 Oct 13 04:43:07.868 INFO request completed, latency_us: 248, response_code: 200, uri: /state/cid/1, method: GET, req_id: 69cd3d56-2c0b-4ae9-aef6-8277a81f56d2, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
5923 001/300 Write block 2106 len 36864 data: 4 3 3 3 4 3 3 3 3
5924 002/300 Write block 313 len 16384 data: 3 3 3 2
5925 003/300 Write block 1433 len 4096 data: 3
5926 004/300 Write block 2792 len 24576 data: 5 5 6 6 6 6
5927 005/300 Write block 3705 len 36864 data: 2 2 2 2 3 3 4 4 4
5928 006/300 Read block 1535 len 36864
5929 007/300 Read block 1940 len 32768
5930 008/300 Write block 2321 len 16384 data: 3 5 5 5
5931 009/300 Read block 4485 len 28672
5932 010/300 Read block 3037 len 8192
5933 011/300 Read block 852 len 32768
5934 012/300 Flush
5935 013/300 Read block 3751 len 28672
5936 014/300 Write block 1802 len 4096 data: 2
5937 015/300 Flush
5938 016/300 Read block 1902 len 12288
5939 017/300 Read block 4894 len 32768
5940 018/300 Read block 1802 len 28672
5941 019/300 Write block 4634 len 12288 data: 2 2 2
5942 020/300 Read block 2518 len 36864
5943 021/300 Read block 4877 len 16384
5944 022/300 Write block 845 len 36864 data: 2 2 2 2 2 2 3 3 3
5945 023/300 Write block 4866 len 12288 data: 4 3 2
5946 024/300 Write block 2414 len 40960 data: 3 3 3 4 4 3 3 3 2 2
5947 025/300 Read block 2681 len 12288
5948 026/300 Read block 4678 len 36864
5949 027/300 Write block 3607 len 20480 data: 3 3 3 2 4
5950 028/300 Read block 3037 len 32768
5951 029/300 Read block 1907 len 32768
5952 030/300 Write block 4645 len 12288 data: 3 3 3
5953 031/300 Read block 932 len 24576
5954 032/300 Read block 4687 len 24576
5955 033/300 Read block 438 len 32768
5956 034/300 Read block 4592 len 20480
5957 035/300 Write block 2288 len 36864 data: 4 3 3 2 2 2 3 4 4
5958 036/300 Read block 3184 len 16384
5959 037/300 Flush
5960 038/300 Flush
5961 039/300 Read block 4389 len 40960
5962 040/300 Read block 1025 len 32768
5963 041/300 Write block 3527 len 36864 data: 5 5 5 6 5 5 5 4 3
5964 042/300 Read block 3438 len 28672
5965 043/300 Write block 1239 len 28672 data: 2 2 2 2 2 2 2
5966 044/300 Read block 4641 len 24576
5967 045/300 Read block 2995 len 32768
5968 046/300 Write block 2766 len 28672 data: 2 2 3 4 3 3 3
5969 047/300 Write block 4135 len 16384 data: 3 2 2 2
5970 048/300 Write block 3742 len 20480 data: 3 3 3 3 3
5971 049/300 Write block 783 len 16384 data: 2 2 2 2
5972 050/300 Read block 2503 len 36864
5973 051/300 Write block 459 len 36864 data: 2 2 2 3 2 2 2 2 2
5974 052/300 Write block 4145 len 40960 data: 3 2 2 2 2 2 2 2 2 3
5975 053/300 Write block 4416 len 12288 data: 5 5 5
5976 054/300 Write block 1035 len 16384 data: 3 3 3 3
5977 055/300 Read block 1149 len 24576
5978 056/300 Read block 2265 len 20480
5979 057/300 Read block 4479 len 40960
5980 058/300 Read block 655 len 28672
5981 059/300 Read block 2076 len 4096
5982 060/300 Read block 4388 len 20480
5983 061/300 Read block 1184 len 8192
5984 062/300 Write block 2735 len 20480 data: 3 4 3 3 3
5985 063/300 Read block 1531 len 8192
5986 064/300 Read block 155 len 12288
5987 065/300 Read block 4811 len 36864
5988 066/300 Read block 2573 len 24576
5989 067/300 Write block 2049 len 40960 data: 2 2 2 2 2 2 2 2 2 2
5990 068/300 Read block 4141 len 12288
5991 069/300 Read block 2734 len 20480
5992 070/300 Write block 1783 len 4096 data: 3
5993 071/300 Read block 1349 len 8192
5994 072/300 Read block 4306 len 12288
5995 073/300 Write block 178 len 20480 data: 4 4 4 4 4
5996 074/300 Write block 2732 len 40960 data: 3 3 3 4 5 4 4 4 3 4
5997 075/300 Write block 638 len 4096 data: 3
5998 076/300 Read block 3226 len 20480
5999 077/300 Read block 3986 len 4096
6000 078/300 Write block 2907 len 36864 data: 3 4 4 3 3 3 2 2 2
6001 079/300 Read block 4580 len 36864
6002 080/300 Write block 1919 len 40960 data: 2 2 3 4 4 3 3 3 3 3
6003 081/300 Read block 4793 len 32768
6004 082/300 Read block 448 len 20480
6005 083/300 Read block 2881 len 40960
6006 084/300 Read block 828 len 40960
6007 085/300 Read block 2049 len 40960
6008 086/300 Read block 4180 len 8192
6009 087/300 Flush
6010 088/300 Read block 798 len 24576
6011 089/300 Read block 535 len 24576
6012 090/300 Read block 4730 len 20480
6013 091/300 Read block 150 len 4096
6014 092/300 Read block 2835 len 4096
6015 093/300 Read block 3301 len 8192
6016 094/300 Read block 3108 len 8192
6017 095/300 Write block 1397 len 36864 data: 2 3 4 4 5 5 5 5 4
6018 096/300 Read block 3908 len 4096
6019 097/300 Read block 1314 len 28672
6020 098/300 Read block 3472 len 20480
6021 099/300 Read block 1621 len 12288
6022 100/300 Write block 4487 len 4096 data: 2
6023 101/300 Read block 93 len 16384
6024 102/300 Read block 1108 len 28672
6025 103/300 Write block 4971 len 8192 data: 3 3
6026 104/300 Write block 1351 len 12288 data: 2 2 2
6027 105/300 Flush
6028 106/300 Read block 913 len 4096
6029 107/300 Write block 1870 len 16384 data: 4 4 4 3
6030 108/300 Write block 2566 len 36864 data: 4 5 4 4 4 4 4 4 4
6031 109/300 Write block 1506 len 36864 data: 2 2 2 2 2 2 2 2 3
6032 110/300 Read block 1684 len 32768
6033 111/300 Read block 1686 len 36864
6034 112/300 Flush
6035 113/300 Read block 4685 len 28672
6036 114/300 Flush
6037 115/300 Read block 2622 len 28672
6038 116/300 Read block 1464 len 12288
6039 117/300 Write block 3424 len 16384 data: 4 4 5 4
6040 118/300 Read block 198 len 28672
6041 119/300 Write block 1963 len 36864 data: 2 2 2 2 2 3 3 3 3
6042 120/300 Read block 926 len 16384
6043 121/300 Write block 3824 len 24576 data: 2 2 2 2 3 3
6044 122/300 Write block 841 len 40960 data: 2 2 2 2 3 3 3 3 3 3
6045 123/300 Read block 4559 len 4096
6046 124/300 Write block 4184 len 12288 data: 3 2 2
6047 125/300 Read block 2423 len 20480
6048 126/300 Read block 465 len 28672
6049 127/300 Write block 1515 len 4096 data: 3
6050 128/300 Read block 180 len 16384
6051 129/300 Write block 4573 len 36864 data: 2 2 2 2 3 3 3 2 2
6052 130/300 Read block 2970 len 32768
6053 131/300 Write block 1715 len 8192 data: 2 2
6054 132/300 Read block 2934 len 8192
6055 133/300 Write block 4499 len 40960 data: 6 4 2 2 2 2 2 2 2 2
6056 134/300 Read block 4982 len 12288
6057 135/300 Read block 4988 len 32768
6058 136/300 Read block 3512 len 36864
6059 137/300 Read block 2344 len 12288
6060 138/300 Read block 1234 len 24576
6061 139/300 Flush
6062 140/300 Write block 4413 len 12288 data: 5 5 5
6063 141/300 Read block 4182 len 36864
6064 142/300 Read block 2690 len 40960
6065 143/300 Write block 1622 len 4096 data: 2
6066 144/300 Read block 216 len 12288
6067 145/300 Read block 227 len 4096
6068 146/300 Read block 3088 len 28672
6069 147/300 Read block 4649 len 12288
6070 148/300 Read block 578 len 8192
6071 149/300 Write block 1264 len 4096 data: 3
6072 150/300 Flush
6073 151/300 Flush
6074 152/300 Read block 1896 len 8192
6075 153/300 Read block 3520 len 4096
6076 154/300 Write block 1747 len 12288 data: 2 2 2
6077 155/300 Read block 1729 len 36864
6078 156/300 Read block 60 len 8192
6079 157/300 Read block 2035 len 16384
6080 158/300 Read block 4956 len 12288
6081 159/300 Read block 569 len 20480
6082 160/300 Flush
6083 161/300 Write block 4272 len 24576 data: 2 4 4 4 3 3
6084 162/300 Read block 4833 len 20480
6085 163/300 Write block 424 len 12288 data: 3 4 3
6086 164/300 Write block 4883 len 28672 data: 2 2 2 5 5 5 3
6087 165/300 Read block 3059 len 4096
6088 166/300 Flush
6089 167/300 Read block 2840 len 36864
6090 168/300 Read block 3803 len 36864
6091 169/300 Write block 3037 len 28672 data: 2 2 2 2 2 3 3
6092 170/300 Read block 3507 len 4096
6093 171/300 Read block 4505 len 4096
6094 172/300 Write block 101 len 36864 data: 2 2 3 3 3 4 4 5 5
6095 173/300 Write block 102 len 4096 data: 3
6096 174/300 Read block 4760 len 40960
6097 175/300 Write block 3183 len 40960 data: 2 2 3 3 5 5 4 4 3 2
6098 176/300 Read block 1292 len 16384
6099 177/300 Write block 1973 len 40960 data: 3 3 3 4 3 3 3 3 3 3
6100 178/300 Write block 1107 len 20480 data: 2 2 2 2 2
6101 179/300 Write block 2628 len 4096 data: 2
6102 180/300 Read block 737 len 20480
6103 181/300 Write block 4773 len 36864 data: 2 2 2 3 4 4 4 4 4
6104 182/300 Write block 296 len 32768 data: 4 4 4 4 3 2 2 3
6105 183/300 Read block 4918 len 12288
6106 184/300 Read block 1710 len 20480
6107 185/300 Write block 853 len 32768 data: 4 3 2 3 3 4 4 4
6108 186/300 Read block 2002 len 8192
6109 187/300 Write block 1127 len 28672 data: 5 5 5 3 2 3 3
6110 188/300 Read block 1488 len 32768
6111 189/300 Read block 1150 len 4096
6112 190/300 Read block 156 len 24576
6113 191/300 Flush
6114 192/300 Write block 1445 len 8192 data: 4 3
6115 193/300 Write block 3220 len 8192 data: 2 2
6116 194/300 Read block 3770 len 40960
6117 195/300 Read block 1164 len 40960
6118 196/300 Write block 1584 len 24576 data: 2 2 2 3 3 3
6119 197/300 Read block 2296 len 40960
6120 198/300 Write block 292 len 16384 data: 6 5 5 4
6121 199/300 Write block 2700 len 40960 data: 6 5 5 4 5 5 5 4 3 3
6122 200/300 Write block 4860 len 12288 data: 4 5 5
6123 201/300 Read block 581 len 4096
6124 202/300 Flush
6125 203/300 Write block 203 len 8192 data: 2 2
6126 204/300 Write block 418 len 28672 data: 3 3 3 3 3 3 4
6127 205/300 Write block 2704 len 8192 data: 6 6
6128 206/300 Write block 1846 len 24576 data: 2 2 2 2 2 2
6129 207/300 Read block 1671 len 4096
6130 208/300 Read block 1995 len 40960
6131 209/300 Flush
6132 210/300 Read block 4688 len 8192
6133 211/300 Write block 4602 len 16384 data: 4 4 4 4
6134 212/300 Write block 4442 len 8192 data: 3 3
6135 213/300 Write block 3014 len 4096 data: 2
6136 214/300 Read block 3403 len 20480
6137 215/300 Write block 3966 len 20480 data: 2 2 2 2 2
6138 216/300 Flush
6139 217/300 Read block 2621 len 12288
6140 218/300 Read block 4412 len 28672
6141 219/300 Read block 4768 len 4096
6142 220/300 Read block 3336 len 20480
6143 221/300 Read block 1085 len 32768
6144 222/300 Read block 352 len 8192
6145 223/300 Read block 182 len 32768
6146 224/300 Read block 3299 len 40960
6147 225/300 Read block 3483 len 32768
6148 226/300 Read block 698 len 12288
6149 227/300 Read block 864 len 28672
6150 228/300 Flush
6151 229/300 Read block 3409 len 16384
6152 230/300 Write block 1080 len 20480 data: 5 5 5 5 5
6153 231/300 Write block 3184 len 36864 data: 3 4 4 6 6 5 5 4 3
6154 232/300 Read block 3887 len 32768
6155 233/300 Flush
6156 234/300 Read block 969 len 40960
6157 235/300 Read block 1218 len 16384
6158 236/300 Read block 1646 len 28672
6159 237/300 Read block 4410 len 8192
6160 238/300 Read block 4004 len 16384
6161 239/300 Read block 2161 len 28672
6162 240/300 Write block 2124 len 12288 data: 3 4 4
6163 241/300 Write block 1104 len 32768 data: 2 2 2 3 3 3 3 3
6164 242/300 Write block 1764 len 4096 data: 2
6165 243/300 Flush
6166 244/300 Read block 4768 len 36864
6167 245/300 Flush
6168 246/300 Write block 4121 len 4096 data: 3
6169 247/300 Read block 839 len 28672
6170 248/300 Write block 4652 len 8192 data: 2 2
6171 249/300 Read block 1614 len 20480
6172 250/300 Read block 855 len 12288
6173 251/300 Write block 3942 len 12288 data: 3 3 3
6174 252/300 Write block 2699 len 8192 data: 6 7
6175 253/300 Read block 4329 len 24576
6176 254/300 Read block 4798 len 20480
6177 255/300 Read block 544 len 28672
6178 256/300 Read block 2214 len 4096
6179 257/300 Write block 2654 len 16384 data: 3 3 4 5
6180 258/300 Read block 43 len 36864
6181 259/300 Flush
6182 260/300 Write block 2775 len 16384 data: 3 3 2 2
6183 261/300 Read block 4574 len 8192
6184 262/300 Flush
6185 263/300 Read block 1627 len 8192
6186 264/300 Read block 763 len 12288
6187 265/300 Read block 4773 len 28672
6188 266/300 Read block 3044 len 4096
6189 267/300 Read block 2535 len 12288
6190 268/300 Read block 920 len 32768
6191 269/300 Flush
6192 270/300 Read block 1140 len 12288
6193 271/300 Write block 1180 len 36864 data: 2 2 2 2 2 2 2 3 3
6194 272/300 Read block 3956 len 36864
6195 273/300 Read block 2570 len 12288
6196 274/300 Read block 588 len 24576
6197 275/300 Write block 3584 len 28672 data: 4 4 4 4 2 2 2
6198 276/300 Read block 4688 len 12288
6199 277/300 Write block 1921 len 16384 data: 4 5 5 4
6200 278/300 Flush
6201 279/300 Write block 3509 len 16384 data: 3 3 3 3
6202 280/300 Read block 2220 len 12288
6203 281/300 Flush
6204 282/300 Read block 1657 len 28672
6205 283/300 Read block 1166 len 32768
6206 284/300 Write block 1887 len 4096 data: 3
6207 285/300 Write block 3929 len 12288 data: 2 2 2
6208 286/300 Read block 3610 len 40960
6209 287/300 Write block 2670 len 8192 data: 3 2
6210 288/300 Read block 2079 len 36864
6211 289/300 Read block 1948 len 36864
6212 290/300 Write block 4583 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6213 291/300 Write block 4294 len 32768 data: 5 6 7 6 4 4 4 4
6214 292/300 Read block 2798 len 28672
6215 293/300 Read block 532 len 28672
6216 294/300 Write block 2691 len 8192 data: 4 4
6217 295/300 Write block 3728 len 20480 data: 3 2 2 2 2
6218 296/300 Write block 471 len 40960 data: 2 2 3 3 3 3 3 3 2 2
6219 297/300 Read block 2464 len 24576
6220 298/300 Read block 2921 len 20480
6221 299/300 Read block 1962 len 12288
6222 300/300 Read block 3391 len 28672
6223 Main task has work to do, go find it
6224 got dsc Start(1)
6225 start 1
6226 Oct 13 04:43:08.754 INFO request completed, latency_us: 284, response_code: 204, uri: /start/cid/1, method: POST, req_id: 47e12388-b938-4e4e-abf2-412870b8e8d8, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
6227 Replay: started 1, returned:Ok(())
6228 ----------------------------------------------------------------
6229 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
6230 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
6231 2120 Acked 3119 Write 9 Done New Done false
6232 2121 Acked 3120 Write 4 Done New Done false
6233 2122 Acked 3121 Write 1 Done New Done false
6234 2123 Acked 3122 Write 6 Done New Done false
6235 2124 Acked 3123 Write 9 Done New Done false
6236 2125 Acked 3124 Read 9 Done New Done false
6237 2126 Acked 3125 Read 8 Done New Done false
6238 2127 Acked 3126 Write 4 Done New Done false
6239 2128 Acked 3127 Read 7 Done New Done false
6240 2129 Acked 3128 Read 2 Done New Done false
6241 2130 Acked 3129 Read 8 Done New Done false
6242 2131 Acked 3130 Flush 0 Done New Done false
6243 2132 Acked 3131 Read 7 Done New Done false
6244 2133 Acked 3132 Write 1 Done New Done false
6245 2134 Acked 3133 Flush 0 Done New Done false
6246 2135 Acked 3134 Read 3 Done New Done false
6247 2136 Acked 3135 Read 8 Done New Done false
6248 2137 Acked 3136 Flush 0 Done New Done false
6249 2138 Acked 3137 Read 7 Done New Done false
6250 2139 Acked 3138 Write 3 Done New Done false
6251 2140 Acked 3139 Read 9 Done New Done false
6252 2141 Acked 3140 Read 4 Done New Done false
6253 2142 Acked 3141 Write 9 Done New Done false
6254 2143 Acked 3142 Write 3 Done New Done false
6255 2144 Acked 3143 Write 10 Done New Done false
6256 2145 Acked 3144 Read 3 Done New Done false
6257 2146 Acked 3145 Read 9 Done New Done false
6258 2147 Acked 3146 Write 5 Done New Done false
6259 2148 Acked 3147 Read 8 Done New Done false
6260 2149 Acked 3148 Read 8 Done New Done false
6261 2150 Acked 3149 Write 3 Done New Done false
6262 2151 Acked 3150 Read 6 Done New Done false
6263 2152 Acked 3151 Read 6 Done New Done false
6264 2153 Acked 3152 Read 8 Done New Done false
6265 2154 Acked 3153 Read 5 Done New Done false
6266 2155 Acked 3154 Write 9 Done New Done false
6267 2156 Acked 3155 Read 4 Done New Done false
6268 2157 Acked 3156 Flush 0 Done New Done false
6269 2158 Acked 3157 Flush 0 Done New Done false
6270 2159 Acked 3158 Read 10 Done New Done false
6271 2160 Acked 3159 Read 8 Done New Done false
6272 2161 Acked 3160 Write 9 Done New Done false
6273 2162 Acked 3161 Read 7 Done New Done false
6274 2163 Acked 3162 Write 7 Done New Done false
6275 2164 Acked 3163 Read 6 Done New Done false
6276 2165 Acked 3164 Read 8 Done New Done false
6277 2166 Acked 3165 Write 7 Done New Done false
6278 2167 Acked 3166 Write 4 Done New Done false
6279 2168 Acked 3167 Write 5 Done New Done false
6280 2169 Acked 3168 Write 4 Done New Done false
6281 2170 Acked 3169 Read 9 Done New Done false
6282 2171 Acked 3170 Write 9 Done New Done false
6283 2172 Acked 3171 Write 10 Done New Done false
6284 2173 Acked 3172 Write 3 Done New Done false
6285 2174 Acked 3173 Write 4 Done New Done false
6286 2175 Acked 3174 Read 6 Done New Done false
6287 2176 Acked 3175 Read 5 Done New Done false
6288 2177 Acked 3176 Read 10 Done New Done false
6289 2178 Acked 3177 Read 7 Done New Done false
6290 2179 Acked 3178 Read 1 Done New Done false
6291 2180 Acked 3179 Read 5 Done New Done false
6292 2181 Acked 3180 Read 2 Done New Done false
6293 2182 Acked 3181 Write 5 Done New Done false
6294 2183 Acked 3182 Read 2 Done New Done false
6295 2184 Acked 3183 Read 3 Done New Done false
6296 2185 Acked 3184 Read 9 Done New Done false
6297 2186 Acked 3185 Read 6 Done New Done false
6298 2187 Acked 3186 Write 10 Done New Done false
6299 2188 Acked 3187 Read 3 Done New Done false
6300 2189 Acked 3188 Read 5 Done New Done false
6301 2190 Acked 3189 Write 1 Done New Done false
6302 2191 Acked 3190 Read 2 Done New Done false
6303 2192 Acked 3191 Read 3 Done New Done false
6304 2193 Acked 3192 Write 5 Done New Done false
6305 2194 Acked 3193 Write 10 Done New Done false
6306 2195 Acked 3194 Write 1 Done New Done false
6307 2196 Acked 3195 Read 5 Done New Done false
6308 2197 Acked 3196 Read 1 Done New Done false
6309 2198 Acked 3197 Write 9 Done New Done false
6310 2199 Acked 3198 Read 9 Done New Done false
6311 2200 Acked 3199 Write 10 Done New Done false
6312 2201 Acked 3200 Read 8 Done New Done false
6313 2202 Acked 3201 Read 5 Done New Done false
6314 2203 Acked 3202 Read 10 Done New Done false
6315 2204 Acked 3203 Read 10 Done New Done false
6316 2205 Acked 3204 Read 10 Done New Done false
6317 2206 Acked 3205 Read 2 Done New Done false
6318 2207 Acked 3206 Flush 0 Done New Done false
6319 2208 Acked 3207 Read 6 Done New Done false
6320 2209 Acked 3208 Read 6 Done New Done false
6321 2210 Acked 3209 Read 5 Done New Done false
6322 2211 Acked 3210 Read 1 Done New Done false
6323 2212 Acked 3211 Read 1 Done New Done false
6324 2213 Acked 3212 Read 2 Done New Done false
6325 2214 Acked 3213 Read 2 Done New Done false
6326 2215 Acked 3214 Write 9 Done New Done false
6327 2216 Acked 3215 Read 1 Done New Done false
6328 2217 Acked 3216 Read 7 Done New Done false
6329 2218 Acked 3217 Read 5 Done New Done false
6330 2219 Acked 3218 Read 3 Done New Done false
6331 2220 Acked 3219 Write 1 Done New Done false
6332 2221 Acked 3220 Read 4 Done New Done false
6333 2222 Acked 3221 Read 7 Done New Done false
6334 2223 Acked 3222 Write 2 Done New Done false
6335 2224 Acked 3223 Write 3 Done New Done false
6336 2225 Acked 3224 Flush 0 Done New Done false
6337 2226 Acked 3225 Read 1 Done New Done false
6338 2227 Acked 3226 Write 4 Done New Done false
6339 2228 Acked 3227 Write 9 Done New Done false
6340 2229 Acked 3228 Write 9 Done New Done false
6341 2230 Acked 3229 Read 8 Done New Done false
6342 2231 Acked 3230 Read 9 Done New Done false
6343 2232 Acked 3231 Flush 0 Done New Done false
6344 2233 Acked 3232 Read 7 Done New Done false
6345 2234 Acked 3233 Flush 0 Done New Done false
6346 2235 Acked 3234 Read 7 Done New Done false
6347 2236 Acked 3235 Read 3 Done New Done false
6348 2237 Acked 3236 Write 4 Done New Done false
6349 2238 Acked 3237 Read 7 Done New Done false
6350 2239 Acked 3238 Write 9 Done New Done false
6351 2240 Acked 3239 Read 4 Done New Done false
6352 2241 Acked 3240 Write 6 Done New Done false
6353 2242 Acked 3241 Write 10 Done New Done false
6354 2243 Acked 3242 Read 1 Done New Done false
6355 2244 Acked 3243 Write 3 Done New Done false
6356 2245 Acked 3244 Read 5 Done New Done false
6357 2246 Acked 3245 Read 7 Done New Done false
6358 2247 Acked 3246 Write 1 Done New Done false
6359 2248 Acked 3247 Read 4 Done New Done false
6360 2249 Acked 3248 Write 9 Done New Done false
6361 2250 Acked 3249 Read 8 Done New Done false
6362 2251 Acked 3250 Write 2 Done New Done false
6363 2252 Acked 3251 Read 2 Done New Done false
6364 2253 Acked 3252 Write 10 Done New Done false
6365 2254 Acked 3253 Read 3 Done New Done false
6366 2255 Acked 3254 Read 8 Done New Done false
6367 2256 Acked 3255 Read 9 Done New Done false
6368 2257 Acked 3256 Read 3 Done New Done false
6369 2258 Acked 3257 Read 6 Done New Done false
6370 2259 Acked 3258 Flush 0 Done New Done false
6371 2260 Acked 3259 Write 3 Done New Done false
6372 2261 Acked 3260 Read 9 Done New Done false
6373 2262 Acked 3261 Read 10 Done New Done false
6374 2263 Acked 3262 Write 1 Done New Done false
6375 2264 Acked 3263 Read 3 Done New Done false
6376 2265 Acked 3264 Read 1 Done New Done false
6377 2266 Acked 3265 Read 7 Done New Done false
6378 2267 Acked 3266 Read 3 Done New Done false
6379 2268 Acked 3267 Read 2 Done New Done false
6380 2269 Acked 3268 Write 1 Done New Done false
6381 2270 Acked 3269 Flush 0 Done New Done false
6382 2271 Acked 3270 Flush 0 Done New Done false
6383 2272 Acked 3271 Read 2 Done New Done false
6384 2273 Acked 3272 Read 1 Done New Done false
6385 2274 Acked 3273 Write 3 Done New Done false
6386 2275 Acked 3274 Read 9 Done New Done false
6387 2276 Acked 3275 Read 2 Done New Done false
6388 2277 Acked 3276 Read 4 Done New Done false
6389 2278 Acked 3277 Read 3 Done New Done false
6390 2279 Acked 3278 Read 5 Done New Done false
6391 2280 Acked 3279 Flush 0 Done New Done false
6392 2281 Acked 3280 Write 6 Done New Done false
6393 2282 Acked 3281 Read 5 Done New Done false
6394 2283 Acked 3282 Write 3 Done New Done false
6395 2284 Acked 3283 Write 7 Done New Done false
6396 2285 Acked 3284 Read 1 Done New Done false
6397 2286 Acked 3285 Flush 0 Done New Done false
6398 2287 Acked 3286 Read 9 Done New Done false
6399 2288 Acked 3287 Read 9 Done New Done false
6400 2289 Acked 3288 Write 7 Done New Done false
6401 2290 Acked 3289 Read 1 Done New Done false
6402 2291 Acked 3290 Read 1 Done New Done false
6403 2292 Acked 3291 Write 9 Done New Done false
6404 2293 Acked 3292 Write 1 Done New Done false
6405 2294 Acked 3293 Read 10 Done New Done false
6406 2295 Acked 3294 Write 10 Done New Done false
6407 2296 Acked 3295 Read 4 Done New Done false
6408 2297 Acked 3296 Write 10 Done New Done false
6409 2298 Acked 3297 Write 5 Done New Done false
6410 2299 Acked 3298 Write 1 Done New Done false
6411 2300 Acked 3299 Read 5 Done New Done false
6412 2301 Acked 3300 Write 9 Done New Done false
6413 2302 Acked 3301 Write 8 Done New Done false
6414 2303 Acked 3302 Read 3 Done New Done false
6415 2304 Acked 3303 Read 5 Done New Done false
6416 2305 Acked 3304 Write 8 Done New Done false
6417 2306 Acked 3305 Read 2 Done New Done false
6418 2307 Acked 3306 Write 7 Done New Done false
6419 2308 Acked 3307 Flush 0 Done New Done false
6420 2309 Acked 3308 Read 8 Done New Done false
6421 2310 Acked 3309 Read 1 Done New Done false
6422 2311 Acked 3310 Read 6 Done New Done false
6423 2312 Acked 3311 Flush 0 Done New Done false
6424 2313 Acked 3312 Write 2 Done New Done false
6425 2314 Acked 3313 Write 2 Done New Done false
6426 2315 Acked 3314 Read 10 Done New Done false
6427 2316 Acked 3315 Read 10 Done New Done false
6428 2317 Acked 3316 Write 6 Done New Done false
6429 2318 Acked 3317 Read 10 Done New Done false
6430 2319 Acked 3318 Write 4 Done New Done false
6431 2320 Acked 3319 Write 10 Done New Done false
6432 2321 Acked 3320 Write 3 Done New Done false
6433 2322 Acked 3321 Read 1 Done New Done false
6434 2323 Acked 3322 Flush 0 Done New Done false
6435 2324 Acked 3323 Write 2 Done New Done false
6436 2325 Acked 3324 Write 7 Done New Done false
6437 2326 Acked 3325 Write 2 Done New Done false
6438 2327 Acked 3326 Write 6 Done New Done false
6439 2328 Acked 3327 Read 1 Done New Done false
6440 2329 Acked 3328 Read 10 Done New Done false
6441 2330 Acked 3329 Flush 0 Done New Done false
6442 2331 Acked 3330 Read 2 Done New Done false
6443 2332 Acked 3331 Write 4 Done New Done false
6444 2333 Acked 3332 Write 2 Done New Done false
6445 2334 Acked 3333 Write 1 Done New Done false
6446 2335 Acked 3334 Read 5 Done New Done false
6447 2336 Acked 3335 Write 5 Done New Done false
6448 2337 Acked 3336 Flush 0 Done New Done false
6449 2338 Acked 3337 Read 3 Done New Done false
6450 2339 Acked 3338 Read 7 Done New Done false
6451 2340 Acked 3339 Read 1 Done New Done false
6452 2341 Acked 3340 Read 5 Done New Done false
6453 2342 Acked 3341 Read 8 Done New Done false
6454 2343 Acked 3342 Read 2 Done New Done false
6455 2344 Acked 3343 Read 8 Done New Done false
6456 2345 Acked 3344 Read 10 Done New Done false
6457 2346 Acked 3345 Read 8 Done New Done false
6458 2347 Acked 3346 Read 3 Done New Done false
6459 2348 Acked 3347 Read 7 Done New Done false
6460 2349 Acked 3348 Flush 0 Done New Done false
6461 2350 Acked 3349 Read 4 Done New Done false
6462 2351 Acked 3350 Write 5 Done New Done false
6463 2352 Acked 3351 Write 9 Done New Done false
6464 2353 Acked 3352 Read 8 Done New Done false
6465 2354 Acked 3353 Flush 0 Done New Done false
6466 2355 Acked 3354 Read 10 Done New Done false
6467 2356 Acked 3355 Read 4 Done New Done false
6468 2357 Acked 3356 Read 7 Done New Done false
6469 2358 Acked 3357 Read 2 Done New Done false
6470 2359 Acked 3358 Read 4 Done New Done false
6471 2360 Acked 3359 Read 7 Done New Done false
6472 2361 Acked 3360 Write 3 Done New Done false
6473 2362 Acked 3361 Write 8 Done New Done false
6474 2363 Acked 3362 Write 1 Done New Done false
6475 2364 Acked 3363 Flush 0 Done New Done false
6476 2365 Acked 3364 Read 9 Done New Done false
6477 2366 Acked 3365 Flush 0 Done New Done false
6478 2367 Acked 3366 Write 1 Done New Done false
6479 2368 Acked 3367 Read 7 Done New Done false
6480 2369 Acked 3368 Write 2 Done New Done false
6481 2370 Acked 3369 Read 5 Done New Done false
6482 2371 Acked 3370 Read 3 Done New Done false
6483 2372 Acked 3371 Write 3 Done New Done false
6484 2373 Acked 3372 Write 2 Done New Done false
6485 2374 Acked 3373 Read 6 Done New Done false
6486 2375 Acked 3374 Read 5 Done New Done false
6487 2376 Acked 3375 Read 7 Done New Done false
6488 2377 Acked 3376 Read 1 Done New Done false
6489 2378 Acked 3377 Write 4 Done New Done false
6490 2379 Acked 3378 Read 9 Done New Done false
6491 2380 Acked 3379 Flush 0 Done New Done false
6492 2381 Acked 3380 Write 4 Done New Done false
6493 2382 Acked 3381 Read 2 Done New Done false
6494 2383 Acked 3382 Flush 0 Done New Done false
6495 2384 Acked 3383 Read 2 Done New Done false
6496 2385 Acked 3384 Read 3 Done New Done false
6497 2386 Acked 3385 Read 7 Done New Done false
6498 2387 Acked 3386 Read 1 Done New Done false
6499 2388 Acked 3387 Read 3 Done New Done false
6500 2389 Acked 3388 Read 8 Done New Done false
6501 2390 Acked 3389 Flush 0 Done New Done false
6502 2391 Acked 3390 Read 3 Done New Done false
6503 2392 Acked 3391 Write 9 Done New Done false
6504 2393 Acked 3392 Read 9 Done New Done false
6505 2394 Acked 3393 Read 3 Done New Done false
6506 2395 Acked 3394 Read 6 Done New Done false
6507 2396 Acked 3395 Write 7 Done New Done false
6508 2397 Acked 3396 Read 3 Done New Done false
6509 2398 Acked 3397 Write 4 Done New Done false
6510 2399 Acked 3398 Flush 0 Done New Done false
6511 2400 Acked 3399 Write 4 Done New Done false
6512 2401 Acked 3400 Read 3 Done New Done false
6513 2402 Acked 3401 Flush 0 Done New Done false
6514 2403 Acked 3402 Read 7 Done New Done false
6515 2404 Acked 3403 Read 8 Done New Done false
6516 2405 Acked 3404 Write 1 Done New Done false
6517 2406 Acked 3405 Write 3 Done New Done false
6518 2407 Acked 3406 Read 10 Done New Done false
6519 2408 Acked 3407 Write 2 Done New Done false
6520 2409 Acked 3408 Read 9 Done New Done false
6521 2410 Acked 3409 Read 9 Done New Done false
6522 2411 Acked 3410 Write 10 Done New Done false
6523 2412 Acked 3411 Write 8 Done New Done false
6524 2413 Acked 3412 Read 7 Done New Done false
6525 2414 Acked 3413 Read 7 Done New Done false
6526 2415 Acked 3414 Write 2 Done New Done false
6527 2416 Acked 3415 Write 5 Done New Done false
6528 2417 Acked 3416 Write 10 Done New Done false
6529 2418 Acked 3417 Read 6 Done New Done false
6530 2419 Acked 3418 Read 5 Done New Done false
6531 2420 Acked 3419 Read 3 Done New Done false
6532 2421 Acked 3420 Read 7 Done New Done false
6533 STATES DS:0 DS:1 DS:2 TOTAL
6534 New 0 302 0 302
6535 Sent 0 0 0 0
6536 Done 302 0 302 604
6537 Skipped 0 0 0 0
6538 Error 0 0 0 0
6539 Last Flush: 3401 3118 3401
6540 Downstairs last five completed: 3118 3117 3116 3115 3114 3113
6541 Upstairs last five completed: 2421 2420 2419 2418 2417 2416
6542 CLIENT: Up:0 ds:302 act:2
6543 [8820] got start action
6544 [8820] I am going to restart
6545 Starting downstairs at port 8820
6546 Make output file at "/tmp/dsc/downstairs-8820.txt"
6547 [8820][1] reports Starting
6548 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1184)
6549 [8820][1] reports Running
65502023-10-13T04:43:10.875ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 1
65512023-10-13T04:43:10.875ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state Offline
65522023-10-13T04:43:10.875ZINFOcrucible: [1] client re-new 303 jobs since flush 3118 = downstairs
65532023-10-13T04:43:10.876ZINFOcrucible: [1] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
65542023-10-13T04:43:10.876ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
65552023-10-13T04:43:10.876ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
65562023-10-13T04:43:10.876ZINFOcrucible: Returning client:1 UUID:12345678-0000-0000-0000-000000008820 matches
65572023-10-13T04:43:10.876ZINFOcrucible: [1] send last flush ID to this DS: 3118
65582023-10-13T04:43:10.876ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
65592023-10-13T04:43:10.877ZINFOcrucible: [1] Replied this last flush ID: 3118
65602023-10-13T04:43:10.877ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Offline Active ds_transition to Replay
65612023-10-13T04:43:10.877ZINFOcrucible: [1] Transition from Offline to Replay
65622023-10-13T04:43:10.877ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
65632023-10-13T04:43:10.877ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Replay Active ds_transition to Active
65642023-10-13T04:43:10.877ZINFOcrucible: [1] Transition from Replay to Active
65652023-10-13T04:43:10.877ZINFOcrucible: [1] Starts cmd_loop
65662023-10-13T04:43:10.877ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
65672023-10-13T04:43:10.877ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
65682023-10-13T04:43:10.877ZINFOcrucible: Set check for repair
65692023-10-13T04:43:11.878ZWARNcrucible: [1] flow control end
65702023-10-13T04:43:11.879ZINFOcrucible: Checking if live repair is needed
65712023-10-13T04:43:11.879ZINFOcrucible: No Live Repair required at this time
6572 ----------------------------------------------------------------
6573 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
6574 Downstairs last five completed: 3421 3420 3419 3418 3417 3416
6575 Upstairs last five completed: 2422 2421 2420 2419 2418 2417
6576 CLIENT: Up:0 ds:0 act:3
6577 Replay: All jobs finished, all DS active.
6578 Main task has work to do, go find it
6579 got dsc Stop(2)
6580 stop 2
6581 Oct 13 04:43:12.758 INFO request completed, latency_us: 462, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 0e6299cf-1b07-4ded-92f3-cf978937b846, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
6582 [8830] Got stop action so:false kr:false
6583 Oct 13 04:43:12.759 INFO request completed, latency_us: 186, response_code: 200, uri: /state/cid/2, method: GET, req_id: b596f244-9e99-44a6-b352-8e72ec1ac7ea, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
65842023-10-13T04:43:12.759ZWARNcrucible: [2] downstairs disconnected
65852023-10-13T04:43:12.759ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
65862023-10-13T04:43:12.759ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8830 closed looper = 2
65872023-10-13T04:43:12.759ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
65882023-10-13T04:43:12.759ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
65892023-10-13T04:43:12.759ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
65902023-10-13T04:43:12.759ZWARNcrucible: [2] pm_task rx.recv() is None
6591 [8830] Exited with: ExitStatus(unix_wait_status(9))
6592 [8830][2] reports Exit
65932023-10-13T04:43:13.761ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
6594 Oct 13 04:43:16.762 INFO request completed, latency_us: 249, response_code: 200, uri: /state/cid/2, method: GET, req_id: a337d7b8-70df-462e-8d03-7e586d04c52c, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
6595 001/300 Write block 3588 len 12288 data: 3 3 3
6596 002/300 Read block 1674 len 4096
6597 003/300 Write block 2044 len 16384 data: 2 2 2 2
6598 004/300 Read block 1008 len 36864
6599 005/300 Write block 1176 len 40960 data: 2 2 2 2 3 3 3 3 3 3
6600 006/300 Flush
6601 007/300 Read block 2561 len 40960
6602 008/300 Write block 3671 len 36864 data: 2 2 2 2 2 2 2 2 2
6603 009/300 Read block 457 len 24576
6604 010/300 Read block 3559 len 4096
6605 011/300 Read block 2705 len 36864
6606 012/300 Flush
6607 013/300 Flush
6608 014/300 Read block 4762 len 36864
6609 015/300 Read block 3258 len 40960
6610 016/300 Write block 4435 len 8192 data: 2 2
6611 017/300 Flush
6612 018/300 Read block 4664 len 40960
6613 019/300 Flush
6614 020/300 Read block 4204 len 36864
6615 021/300 Read block 2049 len 36864
6616 022/300 Flush
6617 023/300 Read block 717 len 32768
6618 024/300 Read block 4671 len 8192
6619 025/300 Flush
6620 026/300 Read block 3976 len 12288
6621 027/300 Read block 2208 len 28672
6622 028/300 Read block 4953 len 36864
6623 029/300 Write block 4424 len 24576 data: 4 4 4 4 4 3
6624 030/300 Read block 663 len 16384
6625 031/300 Write block 4214 len 12288 data: 3 3 3
6626 032/300 Write block 1425 len 40960 data: 3 3 3 3 4 3 3 3 4 3
6627 033/300 Write block 235 len 4096 data: 3
6628 034/300 Write block 3582 len 16384 data: 5 5 5 5
6629 035/300 Read block 1496 len 40960
6630 036/300 Write block 1875 len 8192 data: 3 4
6631 037/300 Read block 290 len 12288
6632 038/300 Read block 2431 len 8192
6633 039/300 Read block 2465 len 8192
6634 040/300 Write block 1098 len 20480 data: 3 3 4 4 4
6635 041/300 Read block 36 len 24576
6636 042/300 Read block 1768 len 12288
6637 043/300 Read block 3403 len 8192
6638 044/300 Write block 4349 len 20480 data: 6 5 4 4 5
6639 045/300 Read block 2230 len 16384
6640 046/300 Read block 2281 len 40960
6641 047/300 Read block 2294 len 24576
6642 048/300 Read block 2326 len 12288
6643 049/300 Read block 239 len 36864
6644 050/300 Read block 4060 len 40960
6645 051/300 Write block 3526 len 20480 data: 4 6 6 6 7
6646 052/300 Write block 3239 len 4096 data: 2
6647 053/300 Read block 4899 len 12288
6648 054/300 Read block 4062 len 4096
6649 055/300 Write block 4422 len 20480 data: 3 3 5 5 5
6650 056/300 Read block 1674 len 32768
6651 057/300 Write block 275 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6652 058/300 Read block 500 len 20480
6653 059/300 Write block 2940 len 8192 data: 3 3
6654 060/300 Flush
6655 061/300 Write block 1046 len 16384 data: 2 2 2 2
6656 062/300 Write block 873 len 28672 data: 5 5 5 5 4 4 4
6657 063/300 Read block 4155 len 36864
6658 064/300 Write block 1198 len 40960 data: 2 2 2 2 2 2 2 2 2 2
6659 065/300 Write block 3357 len 32768 data: 4 4 3 4 3 3 3 3
6660 066/300 Read block 1046 len 4096
6661 067/300 Flush
6662 068/300 Write block 13 len 36864 data: 2 2 2 3 3 4 4 4 4
6663 069/300 Read block 1279 len 40960
6664 070/300 Write block 3372 len 8192 data: 4 4
6665 071/300 Read block 3800 len 24576
6666 072/300 Write block 610 len 32768 data: 5 5 5 3 3 3 2 2
6667 073/300 Write block 361 len 24576 data: 2 2 2 2 2 2
6668 074/300 Flush
6669 075/300 Read block 430 len 20480
6670 076/300 Read block 684 len 16384
6671 077/300 Read block 3626 len 16384
6672 078/300 Read block 2172 len 40960
6673 079/300 Write block 3037 len 28672 data: 3 3 3 3 3 4 4
6674 080/300 Read block 4577 len 24576
6675 081/300 Read block 4119 len 16384
6676 082/300 Flush
6677 083/300 Write block 692 len 8192 data: 3 2
6678 084/300 Read block 3228 len 32768
6679 085/300 Write block 1488 len 40960 data: 3 3 3 3 3 3 4 2 3 3
6680 086/300 Write block 3578 len 12288 data: 2 2 3
6681 087/300 Write block 3825 len 40960 data: 3 3 3 4 4 3 2 2 3 3
6682 088/300 Write block 3172 len 20480 data: 2 2 2 2 2
6683 089/300 Write block 517 len 24576 data: 4 4 4 5 5 5
6684 090/300 Flush
6685 091/300 Write block 750 len 36864 data: 2 3 4 4 4 4 4 4 3
6686 092/300 Write block 2526 len 40960 data: 4 4 4 4 4 4 3 3 2 2
6687 093/300 Read block 4140 len 40960
6688 094/300 Read block 3898 len 12288
6689 095/300 Read block 3657 len 36864
6690 096/300 Read block 1984 len 36864
6691 097/300 Write block 2498 len 40960 data: 3 3 3 3 3 3 3 2 2 3
6692 098/300 Flush
6693 099/300 Write block 2216 len 40960 data: 2 3 2 2 2 2 2 2 3 3
6694 100/300 Read block 4586 len 36864
6695 101/300 Write block 3946 len 24576 data: 2 2 2 2 2 2
6696 102/300 Read block 3018 len 36864
6697 103/300 Read block 1975 len 28672
6698 104/300 Flush
6699 105/300 Read block 1217 len 32768
6700 106/300 Write block 2720 len 16384 data: 2 2 2 2
6701 107/300 Read block 4436 len 16384
6702 108/300 Read block 3459 len 24576
6703 109/300 Write block 282 len 16384 data: 3 3 3 2
6704 110/300 Write block 1231 len 28672 data: 3 3 2 3 3 3 3
6705 111/300 Write block 3170 len 4096 data: 2
6706 112/300 Write block 3136 len 28672 data: 3 3 3 3 3 2 2
6707 113/300 Read block 495 len 24576
6708 114/300 Read block 2762 len 4096
6709 115/300 Read block 1861 len 40960
6710 116/300 Write block 1491 len 4096 data: 4
6711 117/300 Read block 3334 len 4096
6712 118/300 Read block 2847 len 40960
6713 119/300 Write block 1667 len 28672 data: 2 2 2 2 2 2 2
6714 120/300 Read block 227 len 28672
6715 121/300 Read block 1886 len 40960
6716 122/300 Read block 3120 len 32768
6717 123/300 Read block 755 len 8192
6718 124/300 Read block 4844 len 8192
6719 125/300 Write block 234 len 12288 data: 3 4 3
6720 126/300 Write block 2626 len 32768 data: 3 2 3 2 2 2 2 2
6721 127/300 Write block 3698 len 36864 data: 3 3 3 3 3 2 2 3 3
6722 128/300 Read block 2820 len 32768
6723 129/300 Read block 3739 len 12288
6724 130/300 Read block 307 len 8192
6725 131/300 Write block 2030 len 20480 data: 2 2 2 2 2
6726 132/300 Write block 693 len 16384 data: 3 2 3 3
6727 133/300 Write block 4524 len 12288 data: 4 3 4
6728 134/300 Write block 1853 len 20480 data: 2 2 2 3 3
6729 135/300 Read block 2615 len 20480
6730 136/300 Write block 2612 len 20480 data: 4 5 5 5 5
6731 137/300 Read block 3850 len 24576
6732 138/300 Read block 708 len 8192
6733 139/300 Read block 913 len 4096
6734 140/300 Write block 1765 len 32768 data: 2 2 2 2 3 4 4 4
6735 141/300 Write block 1996 len 24576 data: 3 3 3 3 3 3
6736 142/300 Read block 471 len 28672
6737 143/300 Read block 2292 len 12288
6738 144/300 Read block 3867 len 36864
6739 145/300 Read block 649 len 36864
6740 146/300 Write block 2278 len 20480 data: 3 3 3 2 2
6741 147/300 Flush
6742 148/300 Write block 970 len 12288 data: 2 2 3
6743 149/300 Write block 695 len 24576 data: 4 4 3 3 3 3
6744 150/300 Write block 2070 len 4096 data: 3
6745 151/300 Read block 3625 len 20480
6746 152/300 Read block 110 len 40960
6747 153/300 Write block 3154 len 40960 data: 3 3 2 2 2 2 2 2 2 2
6748 154/300 Read block 4236 len 20480
6749 155/300 Write block 2300 len 24576 data: 3 3 2 2 2 2
6750 156/300 Write block 4599 len 32768 data: 3 3 3 5 5 5 5 5
6751 157/300 Read block 3788 len 40960
6752 158/300 Write block 4096 len 28672 data: 2 2 3 4 4 5 5
6753 159/300 Write block 4622 len 16384 data: 3 3 3 3
6754 160/300 Read block 4793 len 16384
6755 161/300 Read block 2781 len 40960
6756 162/300 Read block 636 len 24576
6757 163/300 Read block 1481 len 36864
6758 164/300 Read block 2534 len 8192
6759 165/300 Write block 4682 len 8192 data: 6 6
6760 166/300 Write block 948 len 12288 data: 2 2 2
6761 167/300 Flush
6762 168/300 Write block 517 len 24576 data: 5 5 5 6 6 6
6763 169/300 Read block 4097 len 20480
6764 170/300 Write block 1611 len 12288 data: 3 3 3
6765 171/300 Write block 3023 len 8192 data: 3 4
6766 172/300 Write block 4920 len 4096 data: 3
6767 173/300 Write block 921 len 28672 data: 4 4 3 3 3 3 3
6768 174/300 Flush
6769 175/300 Write block 3521 len 16384 data: 2 2 2 4
6770 176/300 Write block 1830 len 20480 data: 3 2 2 2 3
6771 177/300 Write block 638 len 20480 data: 4 3 2 2 2
6772 178/300 Write block 1081 len 8192 data: 6 6
6773 179/300 Write block 3766 len 24576 data: 3 4 3 2 2 2
6774 180/300 Write block 1023 len 36864 data: 3 3 3 3 3 3 3 3 3
6775 181/300 Read block 2772 len 4096
6776 182/300 Flush
6777 183/300 Write block 4940 len 8192 data: 6 6
6778 184/300 Read block 3665 len 24576
6779 185/300 Write block 567 len 28672 data: 2 2 3 3 4 4 4
6780 186/300 Read block 3233 len 20480
6781 187/300 Flush
6782 188/300 Read block 4127 len 36864
6783 189/300 Read block 4487 len 4096
6784 190/300 Read block 870 len 32768
6785 191/300 Flush
6786 192/300 Read block 1573 len 16384
6787 193/300 Read block 4683 len 12288
6788 194/300 Read block 3144 len 4096
6789 195/300 Read block 3157 len 36864
6790 196/300 Read block 3749 len 20480
6791 197/300 Write block 4988 len 36864 data: 2 2 2 2 2 2 2 2 2
6792 198/300 Write block 198 len 12288 data: 3 2 2
6793 199/300 Flush
6794 200/300 Flush
6795 201/300 Read block 4057 len 12288
6796 202/300 Read block 454 len 12288
6797 203/300 Flush
6798 204/300 Read block 1872 len 28672
6799 205/300 Read block 2624 len 36864
6800 206/300 Write block 212 len 4096 data: 5
6801 207/300 Read block 2004 len 16384
6802 208/300 Flush
6803 209/300 Write block 3250 len 32768 data: 3 2 2 2 2 2 4 4
6804 210/300 Read block 4381 len 8192
6805 211/300 Read block 2289 len 40960
6806 212/300 Write block 2381 len 20480 data: 4 3 3 4 3
6807 213/300 Read block 1334 len 20480
6808 214/300 Write block 605 len 36864 data: 2 2 2 2 4 6 6 6 4
6809 215/300 Read block 211 len 32768
6810 216/300 Write block 4023 len 12288 data: 5 5 4
6811 217/300 Read block 3865 len 16384
6812 218/300 Read block 3357 len 20480
6813 219/300 Read block 3109 len 12288
6814 220/300 Write block 4743 len 20480 data: 2 2 2 2 2
6815 221/300 Read block 3846 len 16384
6816 222/300 Write block 3030 len 28672 data: 4 2 2 2 2 2 2
6817 223/300 Write block 165 len 16384 data: 3 4 4 4
6818 224/300 Read block 1758 len 4096
6819 225/300 Write block 1468 len 24576 data: 2 2 3 3 3 3
6820 226/300 Read block 682 len 24576
6821 227/300 Write block 4642 len 28672 data: 4 4 3 4 4 4 2
6822 228/300 Read block 2634 len 12288
6823 229/300 Read block 2712 len 24576
6824 230/300 Read block 3344 len 40960
6825 231/300 Write block 4754 len 28672 data: 3 2 2 2 3 3 3
6826 232/300 Write block 727 len 4096 data: 2
6827 233/300 Write block 2757 len 32768 data: 2 2 2 2 2 3 2 2
6828 234/300 Read block 4309 len 4096
6829 235/300 Read block 2135 len 28672
6830 236/300 Write block 705 len 32768 data: 2 2 2 2 2 2 2 2
6831 237/300 Write block 4008 len 4096 data: 3
6832 238/300 Read block 4656 len 8192
6833 239/300 Write block 709 len 16384 data: 3 3 3 3
6834 240/300 Write block 1810 len 16384 data: 3 3 3 2
6835 241/300 Read block 3056 len 24576
6836 242/300 Read block 4636 len 24576
6837 243/300 Write block 4860 len 4096 data: 5
6838 244/300 Read block 4353 len 24576
6839 245/300 Write block 4890 len 24576 data: 3 2 2 2 2 2
6840 246/300 Write block 3313 len 40960 data: 2 2 3 3 3 3 4 4 4 4
6841 247/300 Write block 642 len 28672 data: 3 2 2 2 2 2 2
6842 248/300 Read block 52 len 40960
6843 249/300 Write block 2235 len 4096 data: 2
6844 250/300 Read block 197 len 8192
6845 251/300 Read block 4536 len 4096
6846 252/300 Read block 3545 len 12288
6847 253/300 Write block 2264 len 20480 data: 4 5 4 5 4
6848 254/300 Read block 1015 len 36864
6849 255/300 Read block 1612 len 4096
6850 256/300 Flush
6851 257/300 Flush
6852 258/300 Write block 2371 len 28672 data: 4 5 5 5 4 4 5
6853 259/300 Write block 775 len 4096 data: 3
6854 260/300 Flush
6855 261/300 Read block 3814 len 24576
6856 262/300 Write block 1658 len 28672 data: 3 3 4 4 3 3 2
6857 263/300 Write block 3728 len 8192 data: 4 3
6858 264/300 Read block 2645 len 8192
6859 265/300 Read block 3507 len 28672
6860 266/300 Write block 741 len 8192 data: 2 2
6861 267/300 Flush
6862 268/300 Flush
6863 269/300 Read block 1258 len 12288
6864 270/300 Read block 3905 len 20480
6865 271/300 Read block 3712 len 16384
6866 272/300 Write block 612 len 12288 data: 7 5 4
6867 273/300 Read block 2236 len 24576
6868 274/300 Write block 851 len 8192 data: 4 4
6869 275/300 Read block 159 len 24576
6870 276/300 Write block 2115 len 8192 data: 3 3
6871 277/300 Read block 971 len 40960
6872 278/300 Write block 725 len 36864 data: 3 2 3 3 3 3 2 2 2
6873 279/300 Read block 524 len 28672
6874 280/300 Read block 2870 len 28672
6875 281/300 Read block 4587 len 16384
6876 282/300 Read block 2120 len 36864
6877 283/300 Write block 3307 len 4096 data: 3
6878 284/300 Write block 3151 len 16384 data: 3 3 3 4
6879 285/300 Flush
6880 286/300 Flush
6881 287/300 Read block 1640 len 8192
6882 288/300 Read block 189 len 24576
6883 289/300 Read block 1671 len 12288
6884 290/300 Flush
6885 291/300 Write block 3493 len 40960 data: 5 4 3 3 3 2 2 2 3 3
6886 292/300 Read block 610 len 36864
6887 293/300 Write block 534 len 8192 data: 3 4
6888 294/300 Read block 1085 len 12288
6889 295/300 Read block 4236 len 32768
6890 296/300 Write block 2570 len 32768 data: 5 5 5 5 5 4 3 3
6891 297/300 Write block 578 len 12288 data: 3 2 2
6892 298/300 Read block 4649 len 8192
6893 299/300 Flush
6894 300/300 Read block 729 len 28672
6895 Main task has work to do, go find it
6896 got dsc Start(2)
6897 start 2
6898 Oct 13 04:43:17.669 INFO request completed, latency_us: 294, response_code: 204, uri: /start/cid/2, method: POST, req_id: b3f747e5-be57-4bf1-aad6-7ae04b2c7b02, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
6899 Replay: started 2, returned:Ok(())
6900 ----------------------------------------------------------------
6901 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
6902 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
6903 2423 Acked 3422 Write 3 Done Done New false
6904 2424 Acked 3423 Read 1 Done Done New false
6905 2425 Acked 3424 Write 4 Done Done New false
6906 2426 Acked 3425 Read 9 Done Done New false
6907 2427 Acked 3426 Write 10 Done Done New false
6908 2428 Acked 3427 Flush 0 Done Done New false
6909 2429 Acked 3428 Read 10 Done Done New false
6910 2430 Acked 3429 Write 9 Done Done New false
6911 2431 Acked 3430 Read 6 Done Done New false
6912 2432 Acked 3431 Read 1 Done Done New false
6913 2433 Acked 3432 Read 9 Done Done New false
6914 2434 Acked 3433 Flush 0 Done Done New false
6915 2435 Acked 3434 Flush 0 Done Done New false
6916 2436 Acked 3435 Read 9 Done Done New false
6917 2437 Acked 3436 Read 10 Done Done New false
6918 2438 Acked 3437 Write 2 Done Done New false
6919 2439 Acked 3438 Flush 0 Done Done New false
6920 2440 Acked 3439 Read 10 Done Done New false
6921 2441 Acked 3440 Flush 0 Done Done New false
6922 2442 Acked 3441 Read 9 Done Done New false
6923 2443 Acked 3442 Read 9 Done Done New false
6924 2444 Acked 3443 Flush 0 Done Done New false
6925 2445 Acked 3444 Read 8 Done Done New false
6926 2446 Acked 3445 Read 2 Done Done New false
6927 2447 Acked 3446 Flush 0 Done Done New false
6928 2448 Acked 3447 Read 3 Done Done New false
6929 2449 Acked 3448 Read 7 Done Done New false
6930 2450 Acked 3449 Read 9 Done Done New false
6931 2451 Acked 3450 Write 6 Done Done New false
6932 2452 Acked 3451 Read 4 Done Done New false
6933 2453 Acked 3452 Write 3 Done Done New false
6934 2454 Acked 3453 Write 10 Done Done New false
6935 2455 Acked 3454 Write 1 Done Done New false
6936 2456 Acked 3455 Write 4 Done Done New false
6937 2457 Acked 3456 Read 10 Done Done New false
6938 2458 Acked 3457 Write 2 Done Done New false
6939 2459 Acked 3458 Read 3 Done Done New false
6940 2460 Acked 3459 Read 2 Done Done New false
6941 2461 Acked 3460 Read 2 Done Done New false
6942 2462 Acked 3461 Write 5 Done Done New false
6943 2463 Acked 3462 Read 6 Done Done New false
6944 2464 Acked 3463 Read 3 Done Done New false
6945 2465 Acked 3464 Read 2 Done Done New false
6946 2466 Acked 3465 Write 5 Done Done New false
6947 2467 Acked 3466 Read 4 Done Done New false
6948 2468 Acked 3467 Read 10 Done Done New false
6949 2469 Acked 3468 Read 6 Done Done New false
6950 2470 Acked 3469 Read 3 Done Done New false
6951 2471 Acked 3470 Read 9 Done Done New false
6952 2472 Acked 3471 Read 10 Done Done New false
6953 2473 Acked 3472 Write 5 Done Done New false
6954 2474 Acked 3473 Write 1 Done Done New false
6955 2475 Acked 3474 Read 3 Done Done New false
6956 2476 Acked 3475 Read 1 Done Done New false
6957 2477 Acked 3476 Write 5 Done Done New false
6958 2478 Acked 3477 Read 8 Done Done New false
6959 2479 Acked 3478 Write 10 Done Done New false
6960 2480 Acked 3479 Read 5 Done Done New false
6961 2481 Acked 3480 Write 2 Done Done New false
6962 2482 Acked 3481 Flush 0 Done Done New false
6963 2483 Acked 3482 Write 4 Done Done New false
6964 2484 Acked 3483 Write 7 Done Done New false
6965 2485 Acked 3484 Read 9 Done Done New false
6966 2486 Acked 3485 Write 10 Done Done New false
6967 2487 Acked 3486 Write 8 Done Done New false
6968 2488 Acked 3487 Read 1 Done Done New false
6969 2489 Acked 3488 Flush 0 Done Done New false
6970 2490 Acked 3489 Write 9 Done Done New false
6971 2491 Acked 3490 Read 10 Done Done New false
6972 2492 Acked 3491 Write 2 Done Done New false
6973 2493 Acked 3492 Read 6 Done Done New false
6974 2494 Acked 3493 Write 8 Done Done New false
6975 2495 Acked 3494 Write 6 Done Done New false
6976 2496 Acked 3495 Flush 0 Done Done New false
6977 2497 Acked 3496 Read 5 Done Done New false
6978 2498 Acked 3497 Read 4 Done Done New false
6979 2499 Acked 3498 Read 4 Done Done New false
6980 2500 Acked 3499 Read 10 Done Done New false
6981 2501 Acked 3500 Write 7 Done Done New false
6982 2502 Acked 3501 Read 6 Done Done New false
6983 2503 Acked 3502 Read 4 Done Done New false
6984 2504 Acked 3503 Flush 0 Done Done New false
6985 2505 Acked 3504 Write 2 Done Done New false
6986 2506 Acked 3505 Read 8 Done Done New false
6987 2507 Acked 3506 Write 10 Done Done New false
6988 2508 Acked 3507 Write 3 Done Done New false
6989 2509 Acked 3508 Write 10 Done Done New false
6990 2510 Acked 3509 Write 5 Done Done New false
6991 2511 Acked 3510 Write 6 Done Done New false
6992 2512 Acked 3511 Flush 0 Done Done New false
6993 2513 Acked 3512 Write 9 Done Done New false
6994 2514 Acked 3513 Write 10 Done Done New false
6995 2515 Acked 3514 Read 10 Done Done New false
6996 2516 Acked 3515 Read 3 Done Done New false
6997 2517 Acked 3516 Read 9 Done Done New false
6998 2518 Acked 3517 Read 9 Done Done New false
6999 2519 Acked 3518 Write 10 Done Done New false
7000 2520 Acked 3519 Flush 0 Done Done New false
7001 2521 Acked 3520 Write 10 Done Done New false
7002 2522 Acked 3521 Read 9 Done Done New false
7003 2523 Acked 3522 Write 6 Done Done New false
7004 2524 Acked 3523 Read 9 Done Done New false
7005 2525 Acked 3524 Read 7 Done Done New false
7006 2526 Acked 3525 Flush 0 Done Done New false
7007 2527 Acked 3526 Read 8 Done Done New false
7008 2528 Acked 3527 Write 4 Done Done New false
7009 2529 Acked 3528 Read 4 Done Done New false
7010 2530 Acked 3529 Read 6 Done Done New false
7011 2531 Acked 3530 Write 4 Done Done New false
7012 2532 Acked 3531 Write 7 Done Done New false
7013 2533 Acked 3532 Write 1 Done Done New false
7014 2534 Acked 3533 Write 7 Done Done New false
7015 2535 Acked 3534 Read 6 Done Done New false
7016 2536 Acked 3535 Read 1 Done Done New false
7017 2537 Acked 3536 Read 10 Done Done New false
7018 2538 Acked 3537 Write 1 Done Done New false
7019 2539 Acked 3538 Read 1 Done Done New false
7020 2540 Acked 3539 Read 10 Done Done New false
7021 2541 Acked 3540 Write 7 Done Done New false
7022 2542 Acked 3541 Read 7 Done Done New false
7023 2543 Acked 3542 Read 10 Done Done New false
7024 2544 Acked 3543 Read 8 Done Done New false
7025 2545 Acked 3544 Read 2 Done Done New false
7026 2546 Acked 3545 Read 2 Done Done New false
7027 2547 Acked 3546 Write 3 Done Done New false
7028 2548 Acked 3547 Write 8 Done Done New false
7029 2549 Acked 3548 Write 9 Done Done New false
7030 2550 Acked 3549 Read 8 Done Done New false
7031 2551 Acked 3550 Read 3 Done Done New false
7032 2552 Acked 3551 Read 2 Done Done New false
7033 2553 Acked 3552 Write 5 Done Done New false
7034 2554 Acked 3553 Write 4 Done Done New false
7035 2555 Acked 3554 Write 3 Done Done New false
7036 2556 Acked 3555 Write 5 Done Done New false
7037 2557 Acked 3556 Read 5 Done Done New false
7038 2558 Acked 3557 Write 5 Done Done New false
7039 2559 Acked 3558 Read 6 Done Done New false
7040 2560 Acked 3559 Read 2 Done Done New false
7041 2561 Acked 3560 Read 1 Done Done New false
7042 2562 Acked 3561 Write 8 Done Done New false
7043 2563 Acked 3562 Write 6 Done Done New false
7044 2564 Acked 3563 Read 7 Done Done New false
7045 2565 Acked 3564 Read 3 Done Done New false
7046 2566 Acked 3565 Read 9 Done Done New false
7047 2567 Acked 3566 Read 9 Done Done New false
7048 2568 Acked 3567 Write 5 Done Done New false
7049 2569 Acked 3568 Flush 0 Done Done New false
7050 2570 Acked 3569 Write 3 Done Done New false
7051 2571 Acked 3570 Write 6 Done Done New false
7052 2572 Acked 3571 Write 1 Done Done New false
7053 2573 Acked 3572 Read 5 Done Done New false
7054 2574 Acked 3573 Read 10 Done Done New false
7055 2575 Acked 3574 Write 10 Done Done New false
7056 2576 Acked 3575 Read 5 Done Done New false
7057 2577 Acked 3576 Write 6 Done Done New false
7058 2578 Acked 3577 Write 8 Done Done New false
7059 2579 Acked 3578 Read 10 Done Done New false
7060 2580 Acked 3579 Write 7 Done Done New false
7061 2581 Acked 3580 Write 4 Done Done New false
7062 2582 Acked 3581 Read 4 Done Done New false
7063 2583 Acked 3582 Read 10 Done Done New false
7064 2584 Acked 3583 Read 6 Done Done New false
7065 2585 Acked 3584 Read 9 Done Done New false
7066 2586 Acked 3585 Read 2 Done Done New false
7067 2587 Acked 3586 Write 2 Done Done New false
7068 2588 Acked 3587 Write 3 Done Done New false
7069 2589 Acked 3588 Flush 0 Done Done New false
7070 2590 Acked 3589 Write 6 Done Done New false
7071 2591 Acked 3590 Read 5 Done Done New false
7072 2592 Acked 3591 Write 3 Done Done New false
7073 2593 Acked 3592 Write 2 Done Done New false
7074 2594 Acked 3593 Write 1 Done Done New false
7075 2595 Acked 3594 Write 7 Done Done New false
7076 2596 Acked 3595 Flush 0 Done Done New false
7077 2597 Acked 3596 Write 4 Done Done New false
7078 2598 Acked 3597 Write 5 Done Done New false
7079 2599 Acked 3598 Write 5 Done Done New false
7080 2600 Acked 3599 Write 2 Done Done New false
7081 2601 Acked 3600 Write 6 Done Done New false
7082 2602 Acked 3601 Write 9 Done Done New false
7083 2603 Acked 3602 Read 1 Done Done New false
7084 2604 Acked 3603 Flush 0 Done Done New false
7085 2605 Acked 3604 Write 2 Done Done New false
7086 2606 Acked 3605 Read 6 Done Done New false
7087 2607 Acked 3606 Write 7 Done Done New false
7088 2608 Acked 3607 Read 5 Done Done New false
7089 2609 Acked 3608 Flush 0 Done Done New false
7090 2610 Acked 3609 Read 9 Done Done New false
7091 2611 Acked 3610 Read 1 Done Done New false
7092 2612 Acked 3611 Read 8 Done Done New false
7093 2613 Acked 3612 Flush 0 Done Done New false
7094 2614 Acked 3613 Read 4 Done Done New false
7095 2615 Acked 3614 Read 3 Done Done New false
7096 2616 Acked 3615 Read 1 Done Done New false
7097 2617 Acked 3616 Read 9 Done Done New false
7098 2618 Acked 3617 Read 5 Done Done New false
7099 2619 Acked 3618 Write 9 Done Done New false
7100 2620 Acked 3619 Write 3 Done Done New false
7101 2621 Acked 3620 Flush 0 Done Done New false
7102 2622 Acked 3621 Flush 0 Done Done New false
7103 2623 Acked 3622 Read 3 Done Done New false
7104 2624 Acked 3623 Read 3 Done Done New false
7105 2625 Acked 3624 Flush 0 Done Done New false
7106 2626 Acked 3625 Read 7 Done Done New false
7107 2627 Acked 3626 Read 9 Done Done New false
7108 2628 Acked 3627 Write 1 Done Done New false
7109 2629 Acked 3628 Read 4 Done Done New false
7110 2630 Acked 3629 Flush 0 Done Done New false
7111 2631 Acked 3630 Write 8 Done Done New false
7112 2632 Acked 3631 Read 2 Done Done New false
7113 2633 Acked 3632 Read 10 Done Done New false
7114 2634 Acked 3633 Write 5 Done Done New false
7115 2635 Acked 3634 Read 5 Done Done New false
7116 2636 Acked 3635 Write 9 Done Done New false
7117 2637 Acked 3636 Read 8 Done Done New false
7118 2638 Acked 3637 Write 3 Done Done New false
7119 2639 Acked 3638 Read 4 Done Done New false
7120 2640 Acked 3639 Read 5 Done Done New false
7121 2641 Acked 3640 Read 3 Done Done New false
7122 2642 Acked 3641 Write 5 Done Done New false
7123 2643 Acked 3642 Flush 0 Done Done New false
7124 2644 Acked 3643 Read 4 Done Done New false
7125 2645 Acked 3644 Write 7 Done Done New false
7126 2646 Acked 3645 Write 4 Done Done New false
7127 2647 Acked 3646 Read 1 Done Done New false
7128 2648 Acked 3647 Write 6 Done Done New false
7129 2649 Acked 3648 Read 6 Done Done New false
7130 2650 Acked 3649 Write 7 Done Done New false
7131 2651 Acked 3650 Read 3 Done Done New false
7132 2652 Acked 3651 Read 6 Done Done New false
7133 2653 Acked 3652 Read 10 Done Done New false
7134 2654 Acked 3653 Write 7 Done Done New false
7135 2655 Acked 3654 Write 1 Done Done New false
7136 2656 Acked 3655 Write 8 Done Done New false
7137 2657 Acked 3656 Read 1 Done Done New false
7138 2658 Acked 3657 Read 7 Done Done New false
7139 2659 Acked 3658 Write 8 Done Done New false
7140 2660 Acked 3659 Write 1 Done Done New false
7141 2661 Acked 3660 Read 2 Done Done New false
7142 2662 Acked 3661 Write 4 Done Done New false
7143 2663 Acked 3662 Write 4 Done Done New false
7144 2664 Acked 3663 Read 6 Done Done New false
7145 2665 Acked 3664 Read 6 Done Done New false
7146 2666 Acked 3665 Write 1 Done Done New false
7147 2667 Acked 3666 Read 6 Done Done New false
7148 2668 Acked 3667 Write 6 Done Done New false
7149 2669 Acked 3668 Write 10 Done Done New false
7150 2670 Acked 3669 Write 7 Done Done New false
7151 2671 Acked 3670 Read 10 Done Done New false
7152 2672 Acked 3671 Write 1 Done Done New false
7153 2673 Acked 3672 Read 2 Done Done New false
7154 2674 Acked 3673 Read 1 Done Done New false
7155 2675 Acked 3674 Read 3 Done Done New false
7156 2676 Acked 3675 Write 5 Done Done New false
7157 2677 Acked 3676 Read 9 Done Done New false
7158 2678 Acked 3677 Read 1 Done Done New false
7159 2679 Acked 3678 Flush 0 Done Done New false
7160 2680 Acked 3679 Flush 0 Done Done New false
7161 2681 Acked 3680 Write 7 Done Done New false
7162 2682 Acked 3681 Write 1 Done Done New false
7163 2683 Acked 3682 Flush 0 Done Done New false
7164 2684 Acked 3683 Read 6 Done Done New false
7165 2685 Acked 3684 Write 7 Done Done New false
7166 2686 Acked 3685 Write 2 Done Done New false
7167 2687 Acked 3686 Read 2 Done Done New false
7168 2688 Acked 3687 Read 7 Done Done New false
7169 2689 Acked 3688 Write 2 Done Done New false
7170 2690 Acked 3689 Flush 0 Done Done New false
7171 2691 Acked 3690 Flush 0 Done Done New false
7172 2692 Acked 3691 Read 3 Done Done New false
7173 2693 Acked 3692 Read 5 Done Done New false
7174 2694 Acked 3693 Read 4 Done Done New false
7175 2695 Acked 3694 Write 3 Done Done New false
7176 2696 Acked 3695 Read 6 Done Done New false
7177 2697 Acked 3696 Write 2 Done Done New false
7178 2698 Acked 3697 Read 6 Done Done New false
7179 2699 Acked 3698 Write 2 Done Done New false
7180 2700 Acked 3699 Read 10 Done Done New false
7181 2701 Acked 3700 Write 9 Done Done New false
7182 2702 Acked 3701 Read 7 Done Done New false
7183 2703 Acked 3702 Read 7 Done Done New false
7184 2704 Acked 3703 Read 4 Done Done New false
7185 2705 Acked 3704 Read 9 Done Done New false
7186 2706 Acked 3705 Write 1 Done Done New false
7187 2707 Acked 3706 Write 4 Done Done New false
7188 2708 Acked 3707 Flush 0 Done Done New false
7189 2709 Acked 3708 Flush 0 Done Done New false
7190 2710 Acked 3709 Read 2 Done Done New false
7191 2711 Acked 3710 Read 6 Done Done New false
7192 2712 Acked 3711 Read 3 Done Done New false
7193 2713 Acked 3712 Flush 0 Done Done New false
7194 2714 Acked 3713 Write 10 Done Done New false
7195 2715 Acked 3714 Read 9 Done Done New false
7196 2716 Acked 3715 Write 2 Done Done New false
7197 2717 Acked 3716 Read 3 Done Done New false
7198 2718 Acked 3717 Read 8 Done Done New false
7199 2719 Acked 3718 Write 8 Done Done New false
7200 2720 Acked 3719 Write 3 Done Done New false
7201 2721 Acked 3720 Read 2 Done Done New false
7202 2722 Acked 3721 Flush 0 Done Done New false
7203 2723 Acked 3722 Read 7 Done Done New false
7204 STATES DS:0 DS:1 DS:2 TOTAL
7205 New 0 0 301 301
7206 Sent 0 0 0 0
7207 Done 301 301 0 602
7208 Skipped 0 0 0 0
7209 Error 0 0 0 0
7210 Last Flush: 3721 3721 3421
7211 Downstairs last five completed: 3421 3420 3419 3418 3417 3416
7212 Upstairs last five completed: 2723 2722 2721 2720 2719 2718
7213 CLIENT: Up:0 ds:301 act:2
7214 [8830] got start action
7215 [8830] I am going to restart
7216 Starting downstairs at port 8830
7217 Make output file at "/tmp/dsc/downstairs-8830.txt"
7218 [8830][2] reports Starting
7219 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1185)
7220 [8830][2] reports Running
7221 ----------------------------------------------------------------
7222 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
7223 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
7224 2423 Acked 3422 Write 3 Done Done New false
7225 2424 Acked 3423 Read 1 Done Done New false
7226 2425 Acked 3424 Write 4 Done Done New false
7227 2426 Acked 3425 Read 9 Done Done New false
7228 2427 Acked 3426 Write 10 Done Done New false
7229 2428 Acked 3427 Flush 0 Done Done New false
7230 2429 Acked 3428 Read 10 Done Done New false
7231 2430 Acked 3429 Write 9 Done Done New false
7232 2431 Acked 3430 Read 6 Done Done New false
7233 2432 Acked 3431 Read 1 Done Done New false
7234 2433 Acked 3432 Read 9 Done Done New false
7235 2434 Acked 3433 Flush 0 Done Done New false
7236 2435 Acked 3434 Flush 0 Done Done New false
7237 2436 Acked 3435 Read 9 Done Done New false
7238 2437 Acked 3436 Read 10 Done Done New false
7239 2438 Acked 3437 Write 2 Done Done New false
7240 2439 Acked 3438 Flush 0 Done Done New false
7241 2440 Acked 3439 Read 10 Done Done New false
7242 2441 Acked 3440 Flush 0 Done Done New false
7243 2442 Acked 3441 Read 9 Done Done New false
7244 2443 Acked 3442 Read 9 Done Done New false
7245 2444 Acked 3443 Flush 0 Done Done New false
7246 2445 Acked 3444 Read 8 Done Done New false
7247 2446 Acked 3445 Read 2 Done Done New false
7248 2447 Acked 3446 Flush 0 Done Done New false
7249 2448 Acked 3447 Read 3 Done Done New false
7250 2449 Acked 3448 Read 7 Done Done New false
7251 2450 Acked 3449 Read 9 Done Done New false
7252 2451 Acked 3450 Write 6 Done Done New false
7253 2452 Acked 3451 Read 4 Done Done New false
7254 2453 Acked 3452 Write 3 Done Done New false
7255 2454 Acked 3453 Write 10 Done Done New false
7256 2455 Acked 3454 Write 1 Done Done New false
7257 2456 Acked 3455 Write 4 Done Done New false
7258 2457 Acked 3456 Read 10 Done Done New false
7259 2458 Acked 3457 Write 2 Done Done New false
7260 2459 Acked 3458 Read 3 Done Done New false
7261 2460 Acked 3459 Read 2 Done Done New false
7262 2461 Acked 3460 Read 2 Done Done New false
7263 2462 Acked 3461 Write 5 Done Done New false
7264 2463 Acked 3462 Read 6 Done Done New false
7265 2464 Acked 3463 Read 3 Done Done New false
7266 2465 Acked 3464 Read 2 Done Done New false
7267 2466 Acked 3465 Write 5 Done Done New false
7268 2467 Acked 3466 Read 4 Done Done New false
7269 2468 Acked 3467 Read 10 Done Done New false
7270 2469 Acked 3468 Read 6 Done Done New false
7271 2470 Acked 3469 Read 3 Done Done New false
7272 2471 Acked 3470 Read 9 Done Done New false
7273 2472 Acked 3471 Read 10 Done Done New false
7274 2473 Acked 3472 Write 5 Done Done New false
7275 2474 Acked 3473 Write 1 Done Done New false
7276 2475 Acked 3474 Read 3 Done Done New false
7277 2476 Acked 3475 Read 1 Done Done New false
7278 2477 Acked 3476 Write 5 Done Done New false
7279 2478 Acked 3477 Read 8 Done Done New false
7280 2479 Acked 3478 Write 10 Done Done New false
7281 2480 Acked 3479 Read 5 Done Done New false
7282 2481 Acked 3480 Write 2 Done Done New false
7283 2482 Acked 3481 Flush 0 Done Done New false
7284 2483 Acked 3482 Write 4 Done Done New false
7285 2484 Acked 3483 Write 7 Done Done New false
7286 2485 Acked 3484 Read 9 Done Done New false
7287 2486 Acked 3485 Write 10 Done Done New false
7288 2487 Acked 3486 Write 8 Done Done New false
7289 2488 Acked 3487 Read 1 Done Done New false
7290 2489 Acked 3488 Flush 0 Done Done New false
7291 2490 Acked 3489 Write 9 Done Done New false
7292 2491 Acked 3490 Read 10 Done Done New false
7293 2492 Acked 3491 Write 2 Done Done New false
7294 2493 Acked 3492 Read 6 Done Done New false
7295 2494 Acked 3493 Write 8 Done Done New false
7296 2495 Acked 3494 Write 6 Done Done New false
7297 2496 Acked 3495 Flush 0 Done Done New false
7298 2497 Acked 3496 Read 5 Done Done New false
7299 2498 Acked 3497 Read 4 Done Done New false
7300 2499 Acked 3498 Read 4 Done Done New false
7301 2500 Acked 3499 Read 10 Done Done New false
7302 2501 Acked 3500 Write 7 Done Done New false
7303 2502 Acked 3501 Read 6 Done Done New false
7304 2503 Acked 3502 Read 4 Done Done New false
7305 2504 Acked 3503 Flush 0 Done Done New false
7306 2505 Acked 3504 Write 2 Done Done New false
7307 2506 Acked 3505 Read 8 Done Done New false
7308 2507 Acked 3506 Write 10 Done Done New false
7309 2508 Acked 3507 Write 3 Done Done New false
7310 2509 Acked 3508 Write 10 Done Done New false
7311 2510 Acked 3509 Write 5 Done Done New false
7312 2511 Acked 3510 Write 6 Done Done New false
7313 2512 Acked 3511 Flush 0 Done Done New false
7314 2513 Acked 3512 Write 9 Done Done New false
7315 2514 Acked 3513 Write 10 Done Done New false
7316 2515 Acked 3514 Read 10 Done Done New false
7317 2516 Acked 3515 Read 3 Done Done New false
7318 2517 Acked 3516 Read 9 Done Done New false
7319 2518 Acked 3517 Read 9 Done Done New false
7320 2519 Acked 3518 Write 10 Done Done New false
7321 2520 Acked 3519 Flush 0 Done Done New false
7322 2521 Acked 3520 Write 10 Done Done New false
7323 2522 Acked 3521 Read 9 Done Done New false
7324 2523 Acked 3522 Write 6 Done Done New false
7325 2524 Acked 3523 Read 9 Done Done New false
7326 2525 Acked 3524 Read 7 Done Done New false
7327 2526 Acked 3525 Flush 0 Done Done New false
7328 2527 Acked 3526 Read 8 Done Done New false
7329 2528 Acked 3527 Write 4 Done Done New false
7330 2529 Acked 3528 Read 4 Done Done New false
7331 2530 Acked 3529 Read 6 Done Done New false
7332 2531 Acked 3530 Write 4 Done Done New false
7333 2532 Acked 3531 Write 7 Done Done New false
7334 2533 Acked 3532 Write 1 Done Done New false
7335 2534 Acked 3533 Write 7 Done Done New false
7336 2535 Acked 3534 Read 6 Done Done New false
7337 2536 Acked 3535 Read 1 Done Done New false
7338 2537 Acked 3536 Read 10 Done Done New false
7339 2538 Acked 3537 Write 1 Done Done New false
7340 2539 Acked 3538 Read 1 Done Done New false
7341 2540 Acked 3539 Read 10 Done Done New false
7342 2541 Acked 3540 Write 7 Done Done New false
7343 2542 Acked 3541 Read 7 Done Done New false
7344 2543 Acked 3542 Read 10 Done Done New false
7345 2544 Acked 3543 Read 8 Done Done New false
7346 2545 Acked 3544 Read 2 Done Done New false
7347 2546 Acked 3545 Read 2 Done Done New false
7348 2547 Acked 3546 Write 3 Done Done New false
7349 2548 Acked 3547 Write 8 Done Done New false
7350 2549 Acked 3548 Write 9 Done Done New false
7351 2550 Acked 3549 Read 8 Done Done New false
7352 2551 Acked 3550 Read 3 Done Done New false
7353 2552 Acked 3551 Read 2 Done Done New false
7354 2553 Acked 3552 Write 5 Done Done New false
7355 2554 Acked 3553 Write 4 Done Done New false
7356 2555 Acked 3554 Write 3 Done Done New false
7357 2556 Acked 3555 Write 5 Done Done New false
7358 2557 Acked 3556 Read 5 Done Done New false
7359 2558 Acked 3557 Write 5 Done Done New false
7360 2559 Acked 3558 Read 6 Done Done New false
7361 2560 Acked 3559 Read 2 Done Done New false
7362 2561 Acked 3560 Read 1 Done Done New false
7363 2562 Acked 3561 Write 8 Done Done New false
7364 2563 Acked 3562 Write 6 Done Done New false
7365 2564 Acked 3563 Read 7 Done Done New false
7366 2565 Acked 3564 Read 3 Done Done New false
7367 2566 Acked 3565 Read 9 Done Done New false
7368 2567 Acked 3566 Read 9 Done Done New false
7369 2568 Acked 3567 Write 5 Done Done New false
7370 2569 Acked 3568 Flush 0 Done Done New false
7371 2570 Acked 3569 Write 3 Done Done New false
7372 2571 Acked 3570 Write 6 Done Done New false
7373 2572 Acked 3571 Write 1 Done Done New false
7374 2573 Acked 3572 Read 5 Done Done New false
7375 2574 Acked 3573 Read 10 Done Done New false
7376 2575 Acked 3574 Write 10 Done Done New false
7377 2576 Acked 3575 Read 5 Done Done New false
7378 2577 Acked 3576 Write 6 Done Done New false
7379 2578 Acked 3577 Write 8 Done Done New false
7380 2579 Acked 3578 Read 10 Done Done New false
7381 2580 Acked 3579 Write 7 Done Done New false
7382 2581 Acked 3580 Write 4 Done Done New false
7383 2582 Acked 3581 Read 4 Done Done New false
7384 2583 Acked 3582 Read 10 Done Done New false
7385 2584 Acked 3583 Read 6 Done Done New false
7386 2585 Acked 3584 Read 9 Done Done New false
7387 2586 Acked 3585 Read 2 Done Done New false
7388 2587 Acked 3586 Write 2 Done Done New false
7389 2588 Acked 3587 Write 3 Done Done New false
7390 2589 Acked 3588 Flush 0 Done Done New false
7391 2590 Acked 3589 Write 6 Done Done New false
7392 2591 Acked 3590 Read 5 Done Done New false
7393 2592 Acked 3591 Write 3 Done Done New false
7394 2593 Acked 3592 Write 2 Done Done New false
7395 2594 Acked 3593 Write 1 Done Done New false
7396 2595 Acked 3594 Write 7 Done Done New false
7397 2596 Acked 3595 Flush 0 Done Done New false
7398 2597 Acked 3596 Write 4 Done Done New false
7399 2598 Acked 3597 Write 5 Done Done New false
7400 2599 Acked 3598 Write 5 Done Done New false
7401 2600 Acked 3599 Write 2 Done Done New false
7402 2601 Acked 3600 Write 6 Done Done New false
7403 2602 Acked 3601 Write 9 Done Done New false
7404 2603 Acked 3602 Read 1 Done Done New false
7405 2604 Acked 3603 Flush 0 Done Done New false
7406 2605 Acked 3604 Write 2 Done Done New false
7407 2606 Acked 3605 Read 6 Done Done New false
7408 2607 Acked 3606 Write 7 Done Done New false
7409 2608 Acked 3607 Read 5 Done Done New false
7410 2609 Acked 3608 Flush 0 Done Done New false
7411 2610 Acked 3609 Read 9 Done Done New false
7412 2611 Acked 3610 Read 1 Done Done New false
7413 2612 Acked 3611 Read 8 Done Done New false
7414 2613 Acked 3612 Flush 0 Done Done New false
7415 2614 Acked 3613 Read 4 Done Done New false
7416 2615 Acked 3614 Read 3 Done Done New false
7417 2616 Acked 3615 Read 1 Done Done New false
7418 2617 Acked 3616 Read 9 Done Done New false
7419 2618 Acked 3617 Read 5 Done Done New false
7420 2619 Acked 3618 Write 9 Done Done New false
7421 2620 Acked 3619 Write 3 Done Done New false
7422 2621 Acked 3620 Flush 0 Done Done New false
7423 2622 Acked 3621 Flush 0 Done Done New false
7424 2623 Acked 3622 Read 3 Done Done New false
7425 2624 Acked 3623 Read 3 Done Done New false
7426 2625 Acked 3624 Flush 0 Done Done New false
7427 2626 Acked 3625 Read 7 Done Done New false
7428 2627 Acked 3626 Read 9 Done Done New false
7429 2628 Acked 3627 Write 1 Done Done New false
7430 2629 Acked 3628 Read 4 Done Done New false
7431 2630 Acked 3629 Flush 0 Done Done New false
7432 2631 Acked 3630 Write 8 Done Done New false
7433 2632 Acked 3631 Read 2 Done Done New false
7434 2633 Acked 3632 Read 10 Done Done New false
7435 2634 Acked 3633 Write 5 Done Done New false
7436 2635 Acked 3634 Read 5 Done Done New false
7437 2636 Acked 3635 Write 9 Done Done New false
7438 2637 Acked 3636 Read 8 Done Done New false
7439 2638 Acked 3637 Write 3 Done Done New false
7440 2639 Acked 3638 Read 4 Done Done New false
7441 2640 Acked 3639 Read 5 Done Done New false
7442 2641 Acked 3640 Read 3 Done Done New false
7443 2642 Acked 3641 Write 5 Done Done New false
7444 2643 Acked 3642 Flush 0 Done Done New false
7445 2644 Acked 3643 Read 4 Done Done New false
7446 2645 Acked 3644 Write 7 Done Done New false
7447 2646 Acked 3645 Write 4 Done Done New false
7448 2647 Acked 3646 Read 1 Done Done New false
7449 2648 Acked 3647 Write 6 Done Done New false
7450 2649 Acked 3648 Read 6 Done Done New false
7451 2650 Acked 3649 Write 7 Done Done New false
7452 2651 Acked 3650 Read 3 Done Done New false
7453 2652 Acked 3651 Read 6 Done Done New false
7454 2653 Acked 3652 Read 10 Done Done New false
7455 2654 Acked 3653 Write 7 Done Done New false
7456 2655 Acked 3654 Write 1 Done Done New false
7457 2656 Acked 3655 Write 8 Done Done New false
7458 2657 Acked 3656 Read 1 Done Done New false
7459 2658 Acked 3657 Read 7 Done Done New false
7460 2659 Acked 3658 Write 8 Done Done New false
7461 2660 Acked 3659 Write 1 Done Done New false
7462 2661 Acked 3660 Read 2 Done Done New false
7463 2662 Acked 3661 Write 4 Done Done New false
7464 2663 Acked 3662 Write 4 Done Done New false
7465 2664 Acked 3663 Read 6 Done Done New false
7466 2665 Acked 3664 Read 6 Done Done New false
7467 2666 Acked 3665 Write 1 Done Done New false
7468 2667 Acked 3666 Read 6 Done Done New false
7469 2668 Acked 3667 Write 6 Done Done New false
7470 2669 Acked 3668 Write 10 Done Done New false
7471 2670 Acked 3669 Write 7 Done Done New false
7472 2671 Acked 3670 Read 10 Done Done New false
7473 2672 Acked 3671 Write 1 Done Done New false
7474 2673 Acked 3672 Read 2 Done Done New false
7475 2674 Acked 3673 Read 1 Done Done New false
7476 2675 Acked 3674 Read 3 Done Done New false
7477 2676 Acked 3675 Write 5 Done Done New false
7478 2677 Acked 3676 Read 9 Done Done New false
7479 2678 Acked 3677 Read 1 Done Done New false
7480 2679 Acked 3678 Flush 0 Done Done New false
7481 2680 Acked 3679 Flush 0 Done Done New false
7482 2681 Acked 3680 Write 7 Done Done New false
7483 2682 Acked 3681 Write 1 Done Done New false
7484 2683 Acked 3682 Flush 0 Done Done New false
7485 2684 Acked 3683 Read 6 Done Done New false
7486 2685 Acked 3684 Write 7 Done Done New false
7487 2686 Acked 3685 Write 2 Done Done New false
7488 2687 Acked 3686 Read 2 Done Done New false
7489 2688 Acked 3687 Read 7 Done Done New false
7490 2689 Acked 3688 Write 2 Done Done New false
7491 2690 Acked 3689 Flush 0 Done Done New false
7492 2691 Acked 3690 Flush 0 Done Done New false
7493 2692 Acked 3691 Read 3 Done Done New false
7494 2693 Acked 3692 Read 5 Done Done New false
7495 2694 Acked 3693 Read 4 Done Done New false
7496 2695 Acked 3694 Write 3 Done Done New false
7497 2696 Acked 3695 Read 6 Done Done New false
7498 2697 Acked 3696 Write 2 Done Done New false
7499 2698 Acked 3697 Read 6 Done Done New false
7500 2699 Acked 3698 Write 2 Done Done New false
7501 2700 Acked 3699 Read 10 Done Done New false
7502 2701 Acked 3700 Write 9 Done Done New false
7503 2702 Acked 3701 Read 7 Done Done New false
7504 2703 Acked 3702 Read 7 Done Done New false
7505 2704 Acked 3703 Read 4 Done Done New false
7506 2705 Acked 3704 Read 9 Done Done New false
7507 2706 Acked 3705 Write 1 Done Done New false
7508 2707 Acked 3706 Write 4 Done Done New false
7509 2708 Acked 3707 Flush 0 Done Done New false
7510 2709 Acked 3708 Flush 0 Done Done New false
7511 2710 Acked 3709 Read 2 Done Done New false
7512 2711 Acked 3710 Read 6 Done Done New false
7513 2712 Acked 3711 Read 3 Done Done New false
7514 2713 Acked 3712 Flush 0 Done Done New false
7515 2714 Acked 3713 Write 10 Done Done New false
7516 2715 Acked 3714 Read 9 Done Done New false
7517 2716 Acked 3715 Write 2 Done Done New false
7518 2717 Acked 3716 Read 3 Done Done New false
7519 2718 Acked 3717 Read 8 Done Done New false
7520 2719 Acked 3718 Write 8 Done Done New false
7521 2720 Acked 3719 Write 3 Done Done New false
7522 2721 Acked 3720 Read 2 Done Done New false
7523 2722 Acked 3721 Flush 0 Done Done New false
7524 2723 Acked 3722 Read 7 Done Done New false
7525 2724 Acked 3723 Flush 0 Done Done New false
7526 STATES DS:0 DS:1 DS:2 TOTAL
7527 New 0 0 302 302
7528 Sent 0 0 0 0
7529 Done 302 302 0 604
7530 Skipped 0 0 0 0
7531 Error 0 0 0 0
7532 Last Flush: 3723 3723 3421
7533 Downstairs last five completed: 3421 3420 3419 3418 3417 3416
7534 Upstairs last five completed: 2724 2723 2722 2721 2720 2719
7535 CLIENT: Up:0 ds:302 act:2
75362023-10-13T04:43:21.770ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 2
75372023-10-13T04:43:21.770ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
75382023-10-13T04:43:21.770ZINFOcrucible: [2] client re-new 302 jobs since flush 3421 = downstairs
75392023-10-13T04:43:21.771ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
75402023-10-13T04:43:21.771ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
75412023-10-13T04:43:21.771ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
75422023-10-13T04:43:21.771ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
75432023-10-13T04:43:21.771ZINFOcrucible: [2] send last flush ID to this DS: 3421
75442023-10-13T04:43:21.771ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
75452023-10-13T04:43:21.771ZINFOcrucible: [2] Replied this last flush ID: 3421
75462023-10-13T04:43:21.771ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Offline ds_transition to Replay
75472023-10-13T04:43:21.772ZINFOcrucible: [2] Transition from Offline to Replay
75482023-10-13T04:43:21.772ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
75492023-10-13T04:43:21.772ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Replay ds_transition to Active
75502023-10-13T04:43:21.772ZINFOcrucible: [2] Transition from Replay to Active
75512023-10-13T04:43:21.772ZINFOcrucible: [2] Starts cmd_loop
75522023-10-13T04:43:21.772ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
75532023-10-13T04:43:21.772ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
75542023-10-13T04:43:21.772ZINFOcrucible: Set check for repair
75552023-10-13T04:43:22.772ZINFOcrucible: Checking if live repair is needed
75562023-10-13T04:43:22.772ZINFOcrucible: No Live Repair required at this time
75572023-10-13T04:43:22.772ZWARNcrucible: [2] flow control end
7558 ----------------------------------------------------------------
7559 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
7560 Downstairs last five completed: 3723 3722 3721 3720 3719 3718
7561 Upstairs last five completed: 2724 2723 2722 2721 2720 2719
7562 CLIENT: Up:0 ds:0 act:3
7563 Replay: All jobs finished, all DS active.
7564 Main task has work to do, go find it
7565 got dsc Stop(2)
7566 stop 2
7567 Oct 13 04:43:25.676 INFO request completed, latency_us: 442, response_code[8830] Got stop action so:false kr:false
7568 : 204, uri: /stop/cid/2, method: POST, req_id: f58b8bbe-a450-4566-8cc0-ef73699f5c5e, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
7569 Oct 13 04:43:25.678 INFO request completed, latency_us: 187, response_code: 200, uri: /state/cid/2, method: GET, req_id: c3b813eb-f3d5-40ce-a20a-66ec2ed5f2fa, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
75702023-10-13T04:43:25.678ZWARNcrucible: [2] downstairs disconnected
75712023-10-13T04:43:25.678ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
75722023-10-13T04:43:25.678ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8830 closed looper = 2
75732023-10-13T04:43:25.679ZWARNcrucible: [2] pm_task rx.recv() is None
75742023-10-13T04:43:25.679ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
75752023-10-13T04:43:25.679ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
75762023-10-13T04:43:25.679ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
7577 [8830] Exited with: ExitStatus(unix_wait_status(9))
7578 [8830][2] reports Exit
75792023-10-13T04:43:27.681ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
7580 Oct 13 04:43:29.681 INFO request completed, latency_us: 254, response_code: 200, uri: /state/cid/2, method: GET, req_id: 47e101c7-4654-43a4-80de-669ac3b39443, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
7581 001/300 Read block 3785 len 36864
7582 002/300 Write block 4178 len 28672 data: 2 2 3 3 3 3 4
7583 003/300 Read block 3593 len 12288
7584 004/300 Write block 1376 len 36864 data: 2 2 2 2 2 2 2 3 3
7585 005/300 Write block 2842 len 40960 data: 3 2 2 2 3 3 3 3 3 3
7586 006/300 Flush
7587 007/300 Flush
7588 008/300 Write block 3342 len 20480 data: 3 2 3 4 3
7589 009/300 Flush
7590 010/300 Read block 4073 len 20480
7591 011/300 Write block 3601 len 12288 data: 3 5 5
7592 012/300 Write block 4963 len 12288 data: 2 2 2
7593 013/300 Flush
7594 014/300 Write block 1707 len 40960 data: 2 2 3 3 3 3 3 2 3 3
7595 015/300 Read block 4421 len 28672
7596 016/300 Write block 3605 len 32768 data: 4 4 4 4 4 3 5 4
7597 017/300 Read block 1086 len 4096
7598 018/300 Read block 35 len 24576
7599 019/300 Write block 1713 len 20480 data: 4 3 4 4 2
7600 020/300 Flush
7601 021/300 Read block 3982 len 32768
7602 022/300 Read block 3007 len 20480
7603 023/300 Read block 4615 len 16384
7604 024/300 Flush
7605 025/300 Read block 2461 len 20480
7606 026/300 Read block 4416 len 8192
7607 027/300 Write block 3975 len 36864 data: 3 3 2 2 2 2 2 2 3
7608 028/300 Read block 4926 len 24576
7609 029/300 Read block 3960 len 24576
7610 030/300 Read block 3293 len 4096
7611 031/300 Write block 4283 len 32768 data: 4 3 2 2 3 3 4 4
7612 032/300 Write block 27 len 8192 data: 2 2
7613 033/300 Write block 4953 len 4096 data: 2
7614 034/300 Write block 4652 len 32768 data: 3 3 2 2 3 3 4 4
7615 035/300 Flush
7616 036/300 Write block 3041 len 20480 data: 4 5 5 2 2
7617 037/300 Write block 4401 len 24576 data: 2 2 2 2 2 2
7618 038/300 Write block 1823 len 40960 data: 2 4 4 4 4 4 4 4 3 3
7619 039/300 Flush
7620 040/300 Read block 3401 len 40960
7621 041/300 Flush
7622 042/300 Write block 2715 len 28672 data: 2 2 2 2 2 3 3
7623 043/300 Read block 4155 len 24576
7624 044/300 Write block 1749 len 28672 data: 3 2 3 3 3 3 3
7625 045/300 Read block 2005 len 40960
7626 046/300 Read block 2461 len 24576
7627 047/300 Read block 172 len 8192
7628 048/300 Write block 4631 len 32768 data: 2 2 2 3 3 3 2 2
7629 049/300 Read block 4922 len 32768
7630 050/300 Read block 855 len 4096
7631 051/300 Flush
7632 052/300 Read block 3566 len 32768
7633 053/300 Flush
7634 054/300 Write block 3849 len 8192 data: 3 3
7635 055/300 Read block 4753 len 20480
7636 056/300 Write block 4062 len 16384 data: 3 3 3 2
7637 057/300 Read block 347 len 36864
7638 058/300 Write block 1313 len 36864 data: 2 2 2 2 2 2 2 2 2
7639 059/300 Read block 1463 len 8192
7640 060/300 Read block 542 len 40960
7641 061/300 Write block 1595 len 8192 data: 2 2
7642 062/300 Write block 3140 len 32768 data: 4 3 3 2 3 3 2 2
7643 063/300 Flush
7644 064/300 Flush
7645 065/300 Write block 3174 len 16384 data: 3 3 3 2
7646 066/300 Write block 2025 len 28672 data: 2 2 2 2 2 3 3
7647 067/300 Read block 3063 len 36864
7648 068/300 Write block 1356 len 8192 data: 2 3
7649 069/300 Read block 541 len 12288
7650 070/300 Read block 3147 len 16384
7651 071/300 Read block 3140 len 24576
7652 072/300 Write block 2081 len 32768 data: 3 3 3 2 2 2 2 2
7653 073/300 Read block 3985 len 20480
7654 074/300 Write block 2491 len 20480 data: 3 3 3 3 2
7655 075/300 Read block 4447 len 8192
7656 076/300 Read block 2347 len 24576
7657 077/300 Read block 4432 len 4096
7658 078/300 Write block 3553 len 40960 data: 3 3 3 3 2 5 4 4 5 5
7659 079/300 Write block 974 len 16384 data: 2 2 2 2
7660 080/300 Read block 2378 len 8192
7661 081/300 Read block 197 len 20480
7662 082/300 Read block 1942 len 24576
7663 083/300 Read block 4082 len 36864
7664 084/300 Write block 2864 len 16384 data: 3 2 2 2
7665 085/300 Write block 1128 len 32768 data: 6 6 4 3 4 4 3 2
7666 086/300 Write block 1918 len 4096 data: 2
7667 087/300 Write block 3787 len 8192 data: 3 3
7668 088/300 Read block 2262 len 36864
7669 089/300 Read block 1488 len 40960
7670 090/300 Read block 3062 len 36864
7671 091/300 Read block 762 len 24576
7672 092/300 Write block 1101 len 36864 data: 5 5 3 3 3 3 4 4 4
7673 093/300 Read block 4330 len 24576
7674 094/300 Read block 2142 len 36864
7675 095/300 Flush
7676 096/300 Write block 196 len 12288 data: 3 3 4
7677 097/300 Flush
7678 098/300 Write block 4498 len 32768 data: 6 7 5 3 3 3 3 3
7679 099/300 Read block 2662 len 40960
7680 100/300 Write block 1751 len 24576 data: 4 4 4 4 4 2
7681 101/300 Read block 2424 len 36864
7682 102/300 Write block 3365 len 24576 data: 3 3 3 3 3 4
7683 103/300 Read block 1196 len 4096
7684 104/300 Read block 1102 len 28672
7685 105/300 Read block 4051 len 28672
7686 106/300 Write block 1497 len 8192 data: 4 3
7687 107/300 Write block 1831 len 40960 data: 4 4 3 4 3 3 3 3 3 3
7688 108/300 Read block 1025 len 32768
7689 109/300 Write block 1924 len 40960 data: 5 4 4 4 4 2 2 2 2 2
7690 110/300 Read block 1558 len 40960
7691 111/300 Write block 117 len 28672 data: 2 3 3 3 4 4 3
7692 112/300 Flush
7693 113/300 Read block 3768 len 24576
7694 114/300 Write block 4920 len 40960 data: 4 2 2 2 2 2 2 2 2 2
7695 115/300 Read block 2389 len 24576
7696 116/300 Read block 1300 len 8192
7697 117/300 Read block 3494 len 36864
7698 118/300 Read block 2134 len 12288
7699 119/300 Read block 1635 len 8192
7700 120/300 Write block 1505 len 36864 data: 2 3 3 3 3 3 3 3 3
7701 121/300 Read block 3150 len 32768
7702 122/300 Write block 1889 len 24576 data: 3 3 3 2 2 2
7703 123/300 Write block 4747 len 32768 data: 3 3 3 3 4 4 4 4
7704 124/300 Read block 4731 len 16384
7705 125/300 Write block 2775 len 16384 data: 4 4 3 3
7706 126/300 Read block 558 len 16384
7707 127/300 Write block 1945 len 8192 data: 2 2
7708 128/300 Read block 3477 len 20480
7709 129/300 Write block 4622 len 24576 data: 4 4 4 4 4 3
7710 130/300 Write block 3380 len 28672 data: 3 3 3 3 3 3 3
7711 131/300 Flush
7712 132/300 Read block 2090 len 20480
7713 133/300 Write block 2071 len 8192 data: 3 3
7714 134/300 Read block 4015 len 16384
7715 135/300 Write block 3206 len 20480 data: 4 4 4 4 4
7716 136/300 Flush
7717 137/300 Write block 4662 len 16384 data: 3 4 3 3
7718 138/300 Write block 3638 len 28672 data: 2 2 2 2 2 2 2
7719 139/300 Read block 2067 len 8192
7720 140/300 Read block 2526 len 8192
7721 141/300 Write block 457 len 36864 data: 2 2 3 3 3 4 3 3 3
7722 142/300 Flush
7723 143/300 Write block 2027 len 16384 data: 3 3 3 4
7724 144/300 Write block 487 len 36864 data: 4 4 4 4 4 5 5 4 4
7725 145/300 Write block 967 len 16384 data: 2 3 3 3
7726 146/300 Write block 2534 len 8192 data: 3 3
7727 147/300 Read block 2480 len 8192
7728 148/300 Write block 2862 len 4096 data: 2
7729 149/300 Write block 446 len 32768 data: 4 4 3 3 2 2 3 3
7730 150/300 Read block 4248 len 20480
7731 151/300 Write block 4982 len 36864 data: 3 3 2 2 2 2 3 3 3
7732 152/300 Write block 700 len 4096 data: 4
7733 153/300 Flush
7734 154/300 Write block 311 len 16384 data: 4 3 4 4
7735 155/300 Read block 967 len 24576
7736 156/300 Read block 2921 len 8192
7737 157/300 Read block 2187 len 24576
7738 158/300 Read block 3514 len 28672
7739 159/300 Read block 1013 len 4096
7740 160/300 Write block 52 len 40960 data: 3 3 4 4 4 4 3 3 3 3
7741 161/300 Read block 4966 len 12288
7742 162/300 Write block 3725 len 12288 data: 2 2 2
7743 163/300 Flush
7744 164/300 Write block 3846 len 32768 data: 3 3 3 4 4 2 2 2
7745 165/300 Write block 4753 len 8192 data: 5 5
7746 166/300 Read block 2523 len 20480
7747 167/300 Read block 518 len 8192
7748 168/300 Write block 3977 len 24576 data: 3 3 3 3 3 3
7749 169/300 Write block 1663 len 24576 data: 4 3 2 2 3 3
7750 170/300 Read block 370 len 4096
7751 171/300 Flush
7752 172/300 Write block 3574 len 16384 data: 3 3 3 3
7753 173/300 Write block 1490 len 16384 data: 4 5 4 4
7754 174/300 Read block 4674 len 40960
7755 175/300 Write block 3 len 24576 data: 2 2 2 2 2 2
7756 176/300 Write block 1115 len 28672 data: 2 2 2 2 2 2 2
7757 177/300 Read block 3510 len 36864
7758 178/300 Write block 2683 len 4096 data: 2
7759 179/300 Write block 4441 len 4096 data: 2
7760 180/300 Flush
7761 181/300 Read block 671 len 12288
7762 182/300 Read block 469 len 8192
7763 183/300 Read block 2775 len 8192
7764 184/300 Flush
7765 185/300 Write block 2355 len 8192 data: 3 3
7766 186/300 Write block 4851 len 8192 data: 2 2
7767 187/300 Read block 3435 len 40960
7768 188/300 Read block 529 len 28672
7769 189/300 Write block 3181 len 16384 data: 2 2 3 4
7770 190/300 Write block 3205 len 32768 data: 3 5 5 5 5 5 4 4
7771 191/300 Read block 1585 len 40960
7772 192/300 Read block 4638 len 40960
7773 193/300 Read block 4904 len 28672
7774 194/300 Read block 4319 len 8192
7775 195/300 Read block 4267 len 12288
7776 196/300 Read block 4402 len 12288
7777 197/300 Read block 1505 len 36864
7778 198/300 Write block 4454 len 32768 data: 5 4 3 3 3 3 4 4
7779 199/300 Read block 2088 len 4096
7780 200/300 Read block 494 len 28672
7781 201/300 Write block 4080 len 24576 data: 3 4 5 4 5 5
7782 202/300 Write block 1786 len 8192 data: 3 3
7783 203/300 Write block 2506 len 12288 data: 3 4 3
7784 204/300 Write block 2218 len 20480 data: 3 3 3 3 3
7785 205/300 Read block 3325 len 12288
7786 206/300 Read block 703 len 36864
7787 207/300 Read block 192 len 4096
7788 208/300 Read block 1236 len 16384
7789 209/300 Read block 57 len 24576
7790 210/300 Read block 3970 len 24576
7791 211/300 Read block 1909 len 20480
7792 212/300 Read block 2503 len 12288
7793 213/300 Read block 1615 len 12288
7794 214/300 Read block 2565 len 40960
7795 215/300 Write block 735 len 8192 data: 2 2
7796 216/300 Read block 3690 len 16384
7797 217/300 Write block 4572 len 4096 data: 2
7798 218/300 Write block 6 len 40960 data: 3 3 3 2 2 2 2 3 3 3
7799 219/300 Read block 590 len 4096
7800 220/300 Flush
7801 221/300 Read block 694 len 12288
7802 222/300 Read block 1723 len 28672
7803 223/300 Write block 4615 len 32768 data: 3 2 2 2 2 2 3 5
7804 224/300 Read block 2543 len 40960
7805 225/300 Write block 2471 len 16384 data: 5 5 3 3
7806 226/300 Read block 4062 len 16384
7807 227/300 Write block 1738 len 40960 data: 4 3 2 3 3 3 3 3 2 3
7808 228/300 Write block 4797 len 32768 data: 2 2 2 2 2 2 2 2
7809 229/300 Write block 4367 len 32768 data: 2 3 3 3 3 3 3 3
7810 230/300 Write block 4348 len 16384 data: 6 7 6 5
7811 231/300 Read block 3732 len 4096
7812 232/300 Write block 2868 len 4096 data: 2
7813 233/300 Read block 1731 len 40960
7814 234/300 Write block 3237 len 40960 data: 3 3 3 2 2 2 2 2 2 2
7815 235/300 Write block 900 len 32768 data: 4 3 3 3 3 2 2 2
7816 236/300 Write block 749 len 36864 data: 4 3 4 5 5 5 5 5 5
7817 237/300 Read block 2055 len 36864
7818 238/300 Read block 3165 len 8192
7819 239/300 Flush
7820 240/300 Read block 4857 len 4096
7821 241/300 Flush
7822 242/300 Flush
7823 243/300 Write block 2184 len 12288 data: 3 3 3
7824 244/300 Write block 4152 len 16384 data: 3 3 4 4
7825 245/300 Read block 513 len 12288
7826 246/300 Write block 2292 len 24576 data: 3 3 4 5 5 3
7827 247/300 Flush
7828 248/300 Read block 627 len 36864
7829 249/300 Read block 718 len 36864
7830 250/300 Read block 1948 len 28672
7831 251/300 Read block 1064 len 24576
7832 252/300 Read block 2534 len 8192
7833 253/300 Write block 4429 len 12288 data: 4 3 2
7834 254/300 Flush
7835 255/300 Write block 3706 len 32768 data: 4 3 3 4 4 5 5 5
7836 256/300 Read block 2866 len 4096
7837 257/300 Read block 2139 len 4096
7838 258/300 Write block 3825 len 20480 data: 4 4 4 5 5
7839 259/300 Read block 510 len 12288
7840 260/300 Write block 1342 len 16384 data: 3 4 3 3
7841 261/300 Read block 4054 len 12288
7842 262/300 Read block 1696 len 28672
7843 263/300 Write block 2826 len 20480 data: 2 2 3 3 3
7844 264/300 Write block 1640 len 40960 data: 2 2 3 3 3 3 3 3 3 2
7845 265/300 Read block 4062 len 8192
7846 266/300 Write block 2679 len 4096 data: 2
7847 267/300 Read block 3031 len 20480
7848 268/300 Write block 4386 len 40960 data: 3 3 3 3 4 3 3 3 2 3
7849 269/300 Read block 1270 len 16384
7850 270/300 Write block 1809 len 12288 data: 3 4 4
7851 271/300 Flush
7852 272/300 Read block 3188 len 8192
7853 273/300 Read block 2170 len 32768
7854 274/300 Write block 3646 len 20480 data: 2 2 2 2 2
7855 275/300 Read block 2128 len 24576
7856 276/300 Write block 4197 len 32768 data: 2 2 2 2 2 2 2 3
7857 277/300 Read block 4718 len 4096
7858 278/300 Read block 3848 len 8192
7859 279/300 Read block 1893 len 12288
7860 280/300 Read block 269 len 16384
7861 281/300 Read block 3326 len 4096
7862 282/300 Write block 2763 len 28672 data: 3 3 2 3 3 4 5
7863 283/300 Read block 2655 len 40960
7864 284/300 Read block 251 len 4096
7865 285/300 Write block 2532 len 32768 data: 4 4 4 4 2 2 3 3
7866 286/300 Write block 3753 len 12288 data: 3 3 3
7867 287/300 Read block 2157 len 28672
7868 288/300 Write block 4127 len 40960 data: 3 3 3 3 2 2 2 2 4 3
7869 289/300 Write block 238 len 28672 data: 3 3 3 3 4 3 2
7870 290/300 Read block 1887 len 28672
7871 291/300 Read block 2663 len 36864
7872 292/300 Write block 914 len 4096 data: 5
7873 293/300 Read block 908 len 20480
7874 294/300 Write block 747 len 4096 data: 2
7875 295/300 Read block 3160 len 40960
7876 296/300 Read block 2455 len 40960
7877 297/300 Read block 4059 len 4096
7878 298/300 Write block 2792 len 40960 data: 6 6 7 7 7 7 5 4 3 3
7879 299/300 Read block 125 len 32768
7880 300/300 Write block 4997 len 8192 data: 2 3
7881 Main task has work to do, go find it
7882 got dsc Start(2)
7883 start 2
7884 Oct 13 04:43:30.566 INFO request completed, latency_us: 294, response_code: 204, uri: /start/cid/2, method: POST, req_id: d8e29d2c-a678-4235-8c20-6d1059c203c2, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
7885 Replay: started 2, returned:Ok(())
7886 ----------------------------------------------------------------
7887 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
7888 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
7889 2725 Acked 3724 Read 9 Done Done New false
7890 2726 Acked 3725 Write 7 Done Done New false
7891 2727 Acked 3726 Read 3 Done Done New false
7892 2728 Acked 3727 Write 9 Done Done New false
7893 2729 Acked 3728 Write 10 Done Done New false
7894 2730 Acked 3729 Flush 0 Done Done New false
7895 2731 Acked 3730 Flush 0 Done Done New false
7896 2732 Acked 3731 Write 5 Done Done New false
7897 2733 Acked 3732 Flush 0 Done Done New false
7898 2734 Acked 3733 Read 5 Done Done New false
7899 2735 Acked 3734 Write 3 Done Done New false
7900 2736 Acked 3735 Write 3 Done Done New false
7901 2737 Acked 3736 Flush 0 Done Done New false
7902 2738 Acked 3737 Write 10 Done Done New false
7903 2739 Acked 3738 Read 7 Done Done New false
7904 2740 Acked 3739 Write 8 Done Done New false
7905 2741 Acked 3740 Read 1 Done Done New false
7906 2742 Acked 3741 Read 6 Done Done New false
7907 2743 Acked 3742 Write 5 Done Done New false
7908 2744 Acked 3743 Flush 0 Done Done New false
7909 2745 Acked 3744 Read 8 Done Done New false
7910 2746 Acked 3745 Read 5 Done Done New false
7911 2747 Acked 3746 Read 4 Done Done New false
7912 2748 Acked 3747 Flush 0 Done Done New false
7913 2749 Acked 3748 Read 5 Done Done New false
7914 2750 Acked 3749 Read 2 Done Done New false
7915 2751 Acked 3750 Write 9 Done Done New false
7916 2752 Acked 3751 Read 6 Done Done New false
7917 2753 Acked 3752 Read 6 Done Done New false
7918 2754 Acked 3753 Read 1 Done Done New false
7919 2755 Acked 3754 Write 8 Done Done New false
7920 2756 Acked 3755 Write 2 Done Done New false
7921 2757 Acked 3756 Write 1 Done Done New false
7922 2758 Acked 3757 Write 8 Done Done New false
7923 2759 Acked 3758 Flush 0 Done Done New false
7924 2760 Acked 3759 Write 5 Done Done New false
7925 2761 Acked 3760 Write 6 Done Done New false
7926 2762 Acked 3761 Write 10 Done Done New false
7927 2763 Acked 3762 Flush 0 Done Done New false
7928 2764 Acked 3763 Read 10 Done Done New false
7929 2765 Acked 3764 Flush 0 Done Done New false
7930 2766 Acked 3765 Write 7 Done Done New false
7931 2767 Acked 3766 Read 6 Done Done New false
7932 2768 Acked 3767 Write 7 Done Done New false
7933 2769 Acked 3768 Read 10 Done Done New false
7934 2770 Acked 3769 Read 6 Done Done New false
7935 2771 Acked 3770 Read 2 Done Done New false
7936 2772 Acked 3771 Write 8 Done Done New false
7937 2773 Acked 3772 Read 8 Done Done New false
7938 2774 Acked 3773 Read 1 Done Done New false
7939 2775 Acked 3774 Flush 0 Done Done New false
7940 2776 Acked 3775 Read 8 Done Done New false
7941 2777 Acked 3776 Flush 0 Done Done New false
7942 2778 Acked 3777 Write 2 Done Done New false
7943 2779 Acked 3778 Read 5 Done Done New false
7944 2780 Acked 3779 Write 4 Done Done New false
7945 2781 Acked 3780 Read 9 Done Done New false
7946 2782 Acked 3781 Write 9 Done Done New false
7947 2783 Acked 3782 Read 2 Done Done New false
7948 2784 Acked 3783 Read 10 Done Done New false
7949 2785 Acked 3784 Write 2 Done Done New false
7950 2786 Acked 3785 Write 8 Done Done New false
7951 2787 Acked 3786 Flush 0 Done Done New false
7952 2788 Acked 3787 Flush 0 Done Done New false
7953 2789 Acked 3788 Write 4 Done Done New false
7954 2790 Acked 3789 Write 7 Done Done New false
7955 2791 Acked 3790 Read 9 Done Done New false
7956 2792 Acked 3791 Write 2 Done Done New false
7957 2793 Acked 3792 Read 3 Done Done New false
7958 2794 Acked 3793 Read 4 Done Done New false
7959 2795 Acked 3794 Read 6 Done Done New false
7960 2796 Acked 3795 Write 8 Done Done New false
7961 2797 Acked 3796 Read 5 Done Done New false
7962 2798 Acked 3797 Write 5 Done Done New false
7963 2799 Acked 3798 Read 2 Done Done New false
7964 2800 Acked 3799 Read 6 Done Done New false
7965 2801 Acked 3800 Read 1 Done Done New false
7966 2802 Acked 3801 Write 10 Done Done New false
7967 2803 Acked 3802 Write 4 Done Done New false
7968 2804 Acked 3803 Read 2 Done Done New false
7969 2805 Acked 3804 Read 5 Done Done New false
7970 2806 Acked 3805 Read 6 Done Done New false
7971 2807 Acked 3806 Read 9 Done Done New false
7972 2808 Acked 3807 Write 4 Done Done New false
7973 2809 Acked 3808 Write 8 Done Done New false
7974 2810 Acked 3809 Write 1 Done Done New false
7975 2811 Acked 3810 Write 2 Done Done New false
7976 2812 Acked 3811 Read 9 Done Done New false
7977 2813 Acked 3812 Read 10 Done Done New false
7978 2814 Acked 3813 Read 9 Done Done New false
7979 2815 Acked 3814 Read 6 Done Done New false
7980 2816 Acked 3815 Write 9 Done Done New false
7981 2817 Acked 3816 Read 6 Done Done New false
7982 2818 Acked 3817 Read 9 Done Done New false
7983 2819 Acked 3818 Flush 0 Done Done New false
7984 2820 Acked 3819 Write 3 Done Done New false
7985 2821 Acked 3820 Flush 0 Done Done New false
7986 2822 Acked 3821 Write 8 Done Done New false
7987 2823 Acked 3822 Read 10 Done Done New false
7988 2824 Acked 3823 Write 6 Done Done New false
7989 2825 Acked 3824 Read 9 Done Done New false
7990 2826 Acked 3825 Write 6 Done Done New false
7991 2827 Acked 3826 Read 1 Done Done New false
7992 2828 Acked 3827 Read 7 Done Done New false
7993 2829 Acked 3828 Read 7 Done Done New false
7994 2830 Acked 3829 Write 2 Done Done New false
7995 2831 Acked 3830 Write 10 Done Done New false
7996 2832 Acked 3831 Read 8 Done Done New false
7997 2833 Acked 3832 Write 10 Done Done New false
7998 2834 Acked 3833 Read 10 Done Done New false
7999 2835 Acked 3834 Write 7 Done Done New false
8000 2836 Acked 3835 Flush 0 Done Done New false
8001 2837 Acked 3836 Read 6 Done Done New false
8002 2838 Acked 3837 Write 10 Done Done New false
8003 2839 Acked 3838 Read 6 Done Done New false
8004 2840 Acked 3839 Read 2 Done Done New false
8005 2841 Acked 3840 Read 9 Done Done New false
8006 2842 Acked 3841 Read 3 Done Done New false
8007 2843 Acked 3842 Read 2 Done Done New false
8008 2844 Acked 3843 Write 9 Done Done New false
8009 2845 Acked 3844 Read 8 Done Done New false
8010 2846 Acked 3845 Write 6 Done Done New false
8011 2847 Acked 3846 Write 8 Done Done New false
8012 2848 Acked 3847 Read 4 Done Done New false
8013 2849 Acked 3848 Write 4 Done Done New false
8014 2850 Acked 3849 Read 4 Done Done New false
8015 2851 Acked 3850 Write 2 Done Done New false
8016 2852 Acked 3851 Read 5 Done Done New false
8017 2853 Acked 3852 Write 6 Done Done New false
8018 2854 Acked 3853 Write 7 Done Done New false
8019 2855 Acked 3854 Flush 0 Done Done New false
8020 2856 Acked 3855 Read 5 Done Done New false
8021 2857 Acked 3856 Write 2 Done Done New false
8022 2858 Acked 3857 Read 4 Done Done New false
8023 2859 Acked 3858 Write 5 Done Done New false
8024 2860 Acked 3859 Flush 0 Done Done New false
8025 2861 Acked 3860 Write 4 Done Done New false
8026 2862 Acked 3861 Write 7 Done Done New false
8027 2863 Acked 3862 Read 2 Done Done New false
8028 2864 Acked 3863 Read 2 Done Done New false
8029 2865 Acked 3864 Write 9 Done Done New false
8030 2866 Acked 3865 Flush 0 Done Done New false
8031 2867 Acked 3866 Write 4 Done Done New false
8032 2868 Acked 3867 Write 9 Done Done New false
8033 2869 Acked 3868 Write 4 Done Done New false
8034 2870 Acked 3869 Write 2 Done Done New false
8035 2871 Acked 3870 Read 2 Done Done New false
8036 2872 Acked 3871 Write 1 Done Done New false
8037 2873 Acked 3872 Write 8 Done Done New false
8038 2874 Acked 3873 Read 5 Done Done New false
8039 2875 Acked 3874 Write 9 Done Done New false
8040 2876 Acked 3875 Write 1 Done Done New false
8041 2877 Acked 3876 Flush 0 Done Done New false
8042 2878 Acked 3877 Write 4 Done Done New false
8043 2879 Acked 3878 Read 6 Done Done New false
8044 2880 Acked 3879 Read 2 Done Done New false
8045 2881 Acked 3880 Read 6 Done Done New false
8046 2882 Acked 3881 Read 7 Done Done New false
8047 2883 Acked 3882 Read 1 Done Done New false
8048 2884 Acked 3883 Write 10 Done Done New false
8049 2885 Acked 3884 Read 3 Done Done New false
8050 2886 Acked 3885 Write 3 Done Done New false
8051 2887 Acked 3886 Flush 0 Done Done New false
8052 2888 Acked 3887 Write 8 Done Done New false
8053 2889 Acked 3888 Write 2 Done Done New false
8054 2890 Acked 3889 Read 5 Done Done New false
8055 2891 Acked 3890 Read 2 Done Done New false
8056 2892 Acked 3891 Write 6 Done Done New false
8057 2893 Acked 3892 Write 6 Done Done New false
8058 2894 Acked 3893 Read 1 Done Done New false
8059 2895 Acked 3894 Flush 0 Done Done New false
8060 2896 Acked 3895 Write 4 Done Done New false
8061 2897 Acked 3896 Write 4 Done Done New false
8062 2898 Acked 3897 Read 10 Done Done New false
8063 2899 Acked 3898 Write 6 Done Done New false
8064 2900 Acked 3899 Write 7 Done Done New false
8065 2901 Acked 3900 Read 9 Done Done New false
8066 2902 Acked 3901 Write 1 Done Done New false
8067 2903 Acked 3902 Write 1 Done Done New false
8068 2904 Acked 3903 Flush 0 Done Done New false
8069 2905 Acked 3904 Read 3 Done Done New false
8070 2906 Acked 3905 Read 2 Done Done New false
8071 2907 Acked 3906 Read 2 Done Done New false
8072 2908 Acked 3907 Flush 0 Done Done New false
8073 2909 Acked 3908 Write 2 Done Done New false
8074 2910 Acked 3909 Write 2 Done Done New false
8075 2911 Acked 3910 Read 10 Done Done New false
8076 2912 Acked 3911 Read 7 Done Done New false
8077 2913 Acked 3912 Write 4 Done Done New false
8078 2914 Acked 3913 Write 8 Done Done New false
8079 2915 Acked 3914 Read 10 Done Done New false
8080 2916 Acked 3915 Read 10 Done Done New false
8081 2917 Acked 3916 Read 7 Done Done New false
8082 2918 Acked 3917 Read 2 Done Done New false
8083 2919 Acked 3918 Read 3 Done Done New false
8084 2920 Acked 3919 Read 3 Done Done New false
8085 2921 Acked 3920 Read 9 Done Done New false
8086 2922 Acked 3921 Write 8 Done Done New false
8087 2923 Acked 3922 Read 1 Done Done New false
8088 2924 Acked 3923 Read 7 Done Done New false
8089 2925 Acked 3924 Write 6 Done Done New false
8090 2926 Acked 3925 Write 2 Done Done New false
8091 2927 Acked 3926 Write 3 Done Done New false
8092 2928 Acked 3927 Write 5 Done Done New false
8093 2929 Acked 3928 Read 3 Done Done New false
8094 2930 Acked 3929 Read 9 Done Done New false
8095 2931 Acked 3930 Read 1 Done Done New false
8096 2932 Acked 3931 Read 4 Done Done New false
8097 2933 Acked 3932 Read 6 Done Done New false
8098 2934 Acked 3933 Read 6 Done Done New false
8099 2935 Acked 3934 Read 5 Done Done New false
8100 2936 Acked 3935 Read 3 Done Done New false
8101 2937 Acked 3936 Read 3 Done Done New false
8102 2938 Acked 3937 Read 10 Done Done New false
8103 2939 Acked 3938 Write 2 Done Done New false
8104 2940 Acked 3939 Read 4 Done Done New false
8105 2941 Acked 3940 Write 1 Done Done New false
8106 2942 Acked 3941 Write 10 Done Done New false
8107 2943 Acked 3942 Read 1 Done Done New false
8108 2944 Acked 3943 Flush 0 Done Done New false
8109 2945 Acked 3944 Read 3 Done Done New false
8110 2946 Acked 3945 Read 7 Done Done New false
8111 2947 Acked 3946 Write 8 Done Done New false
8112 2948 Acked 3947 Read 10 Done Done New false
8113 2949 Acked 3948 Write 4 Done Done New false
8114 2950 Acked 3949 Read 4 Done Done New false
8115 2951 Acked 3950 Write 10 Done Done New false
8116 2952 Acked 3951 Write 8 Done Done New false
8117 2953 Acked 3952 Write 8 Done Done New false
8118 2954 Acked 3953 Write 4 Done Done New false
8119 2955 Acked 3954 Read 1 Done Done New false
8120 2956 Acked 3955 Write 1 Done Done New false
8121 2957 Acked 3956 Read 10 Done Done New false
8122 2958 Acked 3957 Write 10 Done Done New false
8123 2959 Acked 3958 Write 8 Done Done New false
8124 2960 Acked 3959 Write 9 Done Done New false
8125 2961 Acked 3960 Read 9 Done Done New false
8126 2962 Acked 3961 Read 2 Done Done New false
8127 2963 Acked 3962 Flush 0 Done Done New false
8128 2964 Acked 3963 Read 1 Done Done New false
8129 2965 Acked 3964 Flush 0 Done Done New false
8130 2966 Acked 3965 Flush 0 Done Done New false
8131 2967 Acked 3966 Write 3 Done Done New false
8132 2968 Acked 3967 Write 4 Done Done New false
8133 2969 Acked 3968 Read 3 Done Done New false
8134 2970 Acked 3969 Write 6 Done Done New false
8135 2971 Acked 3970 Flush 0 Done Done New false
8136 2972 Acked 3971 Read 9 Done Done New false
8137 2973 Acked 3972 Read 9 Done Done New false
8138 2974 Acked 3973 Read 7 Done Done New false
8139 2975 Acked 3974 Read 6 Done Done New false
8140 2976 Acked 3975 Read 2 Done Done New false
8141 2977 Acked 3976 Write 3 Done Done New false
8142 2978 Acked 3977 Flush 0 Done Done New false
8143 2979 Acked 3978 Write 8 Done Done New false
8144 2980 Acked 3979 Read 1 Done Done New false
8145 2981 Acked 3980 Read 1 Done Done New false
8146 2982 Acked 3981 Write 5 Done Done New false
8147 2983 Acked 3982 Read 3 Done Done New false
8148 2984 Acked 3983 Write 4 Done Done New false
8149 2985 Acked 3984 Read 3 Done Done New false
8150 2986 Acked 3985 Read 7 Done Done New false
8151 2987 Acked 3986 Write 5 Done Done New false
8152 2988 Acked 3987 Write 10 Done Done New false
8153 2989 Acked 3988 Read 2 Done Done New false
8154 2990 Acked 3989 Write 1 Done Done New false
8155 2991 Acked 3990 Read 5 Done Done New false
8156 2992 Acked 3991 Write 10 Done Done New false
8157 2993 Acked 3992 Read 4 Done Done New false
8158 2994 Acked 3993 Flush 0 Done Done New false
8159 2995 Acked 3994 Write 3 Done Done New false
8160 2996 Acked 3995 Flush 0 Done Done New false
8161 2997 Acked 3996 Read 2 Done Done New false
8162 2998 Acked 3997 Read 8 Done Done New false
8163 2999 Acked 3998 Write 5 Done Done New false
8164 3000 Acked 3999 Read 6 Done Done New false
8165 3001 Acked 4000 Write 8 Done Done New false
8166 3002 Acked 4001 Read 1 Done Done New false
8167 3003 Acked 4002 Read 2 Done Done New false
8168 3004 Acked 4003 Read 3 Done Done New false
8169 3005 Acked 4004 Read 4 Done Done New false
8170 3006 Acked 4005 Read 1 Done Done New false
8171 3007 Acked 4006 Write 7 Done Done New false
8172 3008 Acked 4007 Read 10 Done Done New false
8173 3009 Acked 4008 Read 1 Done Done New false
8174 3010 Acked 4009 Write 8 Done Done New false
8175 3011 Acked 4010 Write 3 Done Done New false
8176 3012 Acked 4011 Read 7 Done Done New false
8177 3013 Acked 4012 Write 10 Done Done New false
8178 3014 Acked 4013 Write 7 Done Done New false
8179 3015 Acked 4014 Read 7 Done Done New false
8180 3016 Acked 4015 Read 9 Done Done New false
8181 3017 Acked 4016 Write 1 Done Done New false
8182 3018 Acked 4017 Read 5 Done Done New false
8183 3019 Acked 4018 Write 1 Done Done New false
8184 3020 Acked 4019 Read 10 Done Done New false
8185 3021 Acked 4020 Read 10 Done Done New false
8186 3022 Acked 4021 Read 1 Done Done New false
8187 3023 Acked 4022 Write 10 Done Done New false
8188 3024 Acked 4023 Read 8 Done Done New false
8189 3025 Acked 4024 Write 2 Sent Sent New false
8190 STATES DS:0 DS:1 DS:2 TOTAL
8191 New 0 0 301 301
8192 Sent 1 1 0 2
8193 Done 300 300 0 600
8194 Skipped 0 0 0 0
8195 Error 0 0 0 0
8196 Last Flush: 3995 3995 3723
8197 Downstairs last five completed: 3723 3722 3721 3720 3719 3718
8198 Upstairs last five completed: 3025 3024 3023 3022 3021 3020
8199 CLIENT: Up:0 ds:301 act:2
8200 ----------------------------------------------------------------
8201 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
8202 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
8203 2725 Acked 3724 Read 9 Done Done New false
8204 2726 Acked 3725 Write 7 Done Done New false
8205 2727 Acked 3726 Read 3 Done Done New false
8206 2728 Acked 3727 Write 9 Done Done New false
8207 2729 Acked 3728 Write 10 Done Done New false
8208 2730 Acked 3729 Flush 0 Done Done New false
8209 2731 Acked 3730 Flush 0 Done Done New false
8210 2732 Acked 3731 Write 5 Done Done New false
8211 2733 Acked 3732 Flush 0 Done Done New false
8212 2734 Acked 3733 Read 5 Done Done New false
8213 2735 Acked 3734 Write 3 Done Done New false
8214 2736 Acked 3735 Write 3 Done Done New false
8215 2737 Acked 3736 Flush 0 Done Done New false
8216 2738 Acked 3737 Write 10 Done Done New false
8217 2739 Acked 3738 Read 7 Done Done New false
8218 2740 Acked 3739 Write 8 Done Done New false
8219 2741 Acked 3740 Read 1 Done Done New false
8220 2742 Acked 3741 Read 6 Done Done New false
8221 2743 Acked 3742 Write 5 Done Done New false
8222 2744 Acked 3743 Flush 0 Done Done New false
8223 2745 Acked 3744 Read 8 Done Done New false
8224 2746 Acked 3745 Read 5 Done Done New false
8225 2747 Acked 3746 Read 4 Done Done New false
8226 2748 Acked 3747 Flush 0 Done Done New false
8227 2749 Acked 3748 Read 5 Done Done New false
8228 2750 Acked 3749 Read 2 Done Done New false
8229 2751 Acked 3750 Write 9 Done Done New false
8230 2752 Acked 3751 Read 6 Done Done New false
8231 2753 Acked 3752 Read 6 Done Done New false
8232 2754 Acked 3753 Read 1 Done Done New false
8233 2755 Acked 3754 Write 8 Done Done New false
8234 2756 Acked 3755 Write 2 Done Done New false
8235 2757 Acked 3756 Write 1 Done Done New false
8236 2758 Acked 3757 Write 8 Done Done New false
8237 2759 Acked 3758 Flush 0 Done Done New false
8238 2760 Acked 3759 Write 5 Done Done New false
8239 2761 Acked 3760 Write 6 Done Done New false
8240 2762 Acked 3761 Write 10 Done Done New false
8241 2763 Acked 3762 Flush 0 Done Done New false
8242 2764 Acked 3763 Read 10 Done Done New false
8243 2765 Acked 3764 Flush 0 Done Done New false
8244 2766 Acked 3765 Write 7 Done Done New false
8245 2767 Acked 3766 Read 6 Done Done New false
8246 2768 Acked 3767 Write 7 Done Done New false
8247 2769 Acked 3768 Read 10 Done Done New false
8248 2770 Acked 3769 Read 6 Done Done New false
8249 2771 Acked 3770 Read 2 Done Done New false
8250 2772 Acked 3771 Write 8 Done Done New false
8251 2773 Acked 3772 Read 8 Done Done New false
8252 2774 Acked 3773 Read 1 Done Done New false
8253 2775 Acked 3774 Flush 0 Done Done New false
8254 2776 Acked 3775 Read 8 Done Done New false
8255 2777 Acked 3776 Flush 0 Done Done New false
8256 2778 Acked 3777 Write 2 Done Done New false
8257 2779 Acked 3778 Read 5 Done Done New false
8258 2780 Acked 3779 Write 4 Done Done New false
8259 2781 Acked 3780 Read 9 Done Done New false
8260 2782 Acked 3781 Write 9 Done Done New false
8261 2783 Acked 3782 Read 2 Done Done New false
8262 2784 Acked 3783 Read 10 Done Done New false
8263 2785 Acked 3784 Write 2 Done Done New false
8264 2786 Acked 3785 Write 8 Done Done New false
8265 2787 Acked 3786 Flush 0 Done Done New false
8266 2788 Acked 3787 Flush 0 Done Done New false
8267 2789 Acked 3788 Write 4 Done Done New false
8268 2790 Acked 3789 Write 7 Done Done New false
8269 2791 Acked 3790 Read 9 Done Done New false
8270 2792 Acked 3791 Write 2 Done Done New false
8271 2793 Acked 3792 Read 3 Done Done New false
8272 2794 Acked 3793 Read 4 Done Done New false
8273 2795 Acked 3794 Read 6 Done Done New false
8274 2796 Acked 3795 Write 8 Done Done New false
8275 2797 Acked 3796 Read 5 Done Done New false
8276 2798 Acked 3797 Write 5 Done Done New false
8277 2799 Acked 3798 Read 2 Done Done New false
8278 2800 Acked 3799 Read 6 Done Done New false
8279 2801 Acked 3800 Read 1 Done Done New false
8280 2802 Acked 3801 Write 10 Done Done New false
8281 2803 Acked 3802 Write 4 Done Done New false
8282 2804 Acked 3803 Read 2 Done Done New false
8283 2805 Acked 3804 Read 5 Done Done New false
8284 2806 Acked 3805 Read 6 Done Done New false
8285 2807 Acked 3806 Read 9 Done Done New false
8286 2808 Acked 3807 Write 4 Done Done New false
8287 2809 Acked 3808 Write 8 Done Done New false
8288 2810 Acked 3809 Write 1 Done Done New false
8289 2811 Acked 3810 Write 2 Done Done New false
8290 2812 Acked 3811 Read 9 Done Done New false
8291 2813 Acked 3812 Read 10 Done Done New false
8292 2814 Acked 3813 Read 9 Done Done New false
8293 2815 Acked 3814 Read 6 Done Done New false
8294 2816 Acked 3815 Write 9 Done Done New false
8295 2817 Acked 3816 Read 6 Done Done New false
8296 2818 Acked 3817 Read 9 Done Done New false
8297 2819 Acked 3818 Flush 0 Done Done New false
8298 2820 Acked 3819 Write 3 Done Done New false
8299 2821 Acked 3820 Flush 0 Done Done New false
8300 2822 Acked 3821 Write 8 Done Done New false
8301 2823 Acked 3822 Read 10 Done Done New false
8302 2824 Acked 3823 Write 6 Done Done New false
8303 2825 Acked 3824 Read 9 Done Done New false
8304 2826 Acked 3825 Write 6 Done Done New false
8305 2827 Acked 3826 Read 1 Done Done New false
8306 2828 Acked 3827 Read 7 Done Done New false
8307 2829 Acked 3828 Read 7 Done Done New false
8308 2830 Acked 3829 Write 2 Done Done New false
8309 2831 Acked 3830 Write 10 Done Done New false
8310 2832 Acked 3831 Read 8 Done Done New false
8311 2833 Acked 3832 Write 10 Done Done New false
8312 2834 Acked 3833 Read 10 Done Done New false
8313 2835 Acked 3834 Write 7 Done Done New false
8314 2836 Acked 3835 Flush 0 Done Done New false
8315 2837 Acked 3836 Read 6 Done Done New false
8316 2838 Acked 3837 Write 10 Done Done New false
8317 2839 Acked 3838 Read 6 Done Done New false
8318 2840 Acked 3839 Read 2 Done Done New false
8319 2841 Acked 3840 Read 9 Done Done New false
8320 2842 Acked 3841 Read 3 Done Done New false
8321 2843 Acked 3842 Read 2 Done Done New false
8322 2844 Acked 3843 Write 9 Done Done New false
8323 2845 Acked 3844 Read 8 Done Done New false
8324 2846 Acked 3845 Write 6 Done Done New false
8325 2847 Acked 3846 Write 8 Done Done New false
8326 2848 Acked 3847 Read 4 Done Done New false
8327 2849 Acked 3848 Write 4 Done Done New false
8328 2850 Acked 3849 Read 4 Done Done New false
8329 2851 Acked 3850 Write 2 Done Done New false
8330 2852 Acked 3851 Read 5 Done Done New false
8331 2853 Acked 3852 Write 6 Done Done New false
8332 2854 Acked 3853 Write 7 Done Done New false
8333 2855 Acked 3854 Flush 0 Done Done New false
8334 2856 Acked 3855 Read 5 Done Done New false
8335 2857 Acked 3856 Write 2 Done Done New false
8336 2858 Acked 3857 Read 4 Done Done New false
8337 2859 Acked 3858 Write 5 Done Done New false
8338 2860 Acked 3859 Flush 0 Done Done New false
8339 2861 Acked 3860 Write 4 Done Done New false
8340 2862 Acked 3861 Write 7 Done Done New false
8341 2863 Acked 3862 Read 2 Done Done New false
8342 2864 Acked 3863 Read 2 Done Done New false
8343 2865 Acked 3864 Write 9 Done Done New false
8344 2866 Acked 3865 Flush 0 Done Done New false
8345 2867 Acked 3866 Write 4 Done Done New false
8346 2868 Acked 3867 Write 9 Done Done New false
8347 2869 Acked 3868 Write 4 Done Done New false
8348 2870 Acked 3869 Write 2 Done Done New false
8349 2871 Acked 3870 Read 2 Done Done New false
8350 2872 Acked 3871 Write 1 Done Done New false
8351 2873 Acked 3872 Write 8 Done Done New false
8352 2874 Acked 3873 Read 5 Done Done New false
8353 2875 Acked 3874 Write 9 Done Done New false
8354 2876 Acked 3875 Write 1 Done Done New false
8355 2877 Acked 3876 Flush 0 Done Done New false
8356 2878 Acked 3877 Write 4 Done Done New false
8357 2879 Acked 3878 Read 6 Done Done New false
8358 2880 Acked 3879 Read 2 Done Done New false
8359 2881 Acked 3880 Read 6 Done Done New false
8360 2882 Acked 3881 Read 7 Done Done New false
8361 2883 Acked 3882 Read 1 Done Done New false
8362 2884 Acked 3883 Write 10 Done Done New false
8363 2885 Acked 3884 Read 3 Done Done New false
8364 2886 Acked 3885 Write 3 Done Done New false
8365 2887 Acked 3886 Flush 0 Done Done New false
8366 2888 Acked 3887 Write 8 Done Done New false
8367 2889 Acked 3888 Write 2 Done Done New false
8368 2890 Acked 3889 Read 5 Done Done New false
8369 2891 Acked 3890 Read 2 Done Done New false
8370 2892 Acked 3891 Write 6 Done Done New false
8371 2893 Acked 3892 Write 6 Done Done New false
8372 2894 Acked 3893 Read 1 Done Done New false
8373 2895 Acked 3894 Flush 0 Done Done New false
8374 2896 Acked 3895 Write 4 Done Done New false
8375 2897 Acked 3896 Write 4 Done Done New false
8376 2898 Acked 3897 Read 10 Done Done New false
8377 2899 Acked 3898 Write 6 Done Done New false
8378 2900 Acked 3899 Write 7 Done Done New false
8379 2901 Acked 3900 Read 9 Done Done New false
8380 2902 Acked 3901 Write 1 Done Done New false
8381 2903 Acked 3902 Write 1 Done Done New false
8382 2904 Acked 3903 Flush 0 Done Done New false
8383 2905 Acked 3904 Read 3 Done Done New false
8384 2906 Acked 3905 Read 2 Done Done New false
8385 2907 Acked 3906 Read 2 Done Done New false
8386 2908 Acked 3907 Flush 0 Done Done New false
8387 2909 Acked 3908 Write 2 Done Done New false
8388 2910 Acked 3909 Write 2 Done Done New false
8389 2911 Acked 3910 Read 10 Done Done New false
8390 2912 Acked 3911 Read 7 Done Done New false
8391 2913 Acked 3912 Write 4 Done Done New false
8392 2914 Acked 3913 Write 8 Done Done New false
8393 2915 Acked 3914 Read 10 Done Done New false
8394 2916 Acked 3915 Read 10 Done Done New false
8395 2917 Acked 3916 Read 7 Done Done New false
8396 2918 Acked 3917 Read 2 Done Done New false
8397 2919 Acked 3918 Read 3 Done Done New false
8398 2920 Acked 3919 Read 3 Done Done New false
8399 2921 Acked 3920 Read 9 Done Done New false
8400 2922 Acked 3921 Write 8 Done Done New false
8401 2923 Acked 3922 Read 1 Done Done New false
8402 2924 Acked 3923 Read 7 Done Done New false
8403 2925 Acked 3924 Write 6 Done Done New false
8404 2926 Acked 3925 Write 2 Done Done New false
8405 2927 Acked 3926 Write 3 Done Done New false
8406 2928 Acked 3927 Write 5 Done Done New false
8407 2929 Acked 3928 Read 3 Done Done New false
8408 2930 Acked 3929 Read 9 Done Done New false
8409 2931 Acked 3930 Read 1 Done Done New false
8410 2932 Acked 3931 Read 4 Done Done New false
8411 2933 Acked 3932 Read 6 Done Done New false
8412 2934 Acked 3933 Read 6 Done Done New false
8413 2935 Acked 3934 Read 5 Done Done New false
8414 2936 Acked 3935 Read 3 Done Done New false
8415 2937 Acked 3936 Read 3 Done Done New false
8416 2938 Acked 3937 Read 10 Done Done New false
8417 2939 Acked 3938 Write 2 Done Done New false
8418 2940 Acked 3939 Read 4 Done Done New false
8419 2941 Acked 3940 Write 1 Done Done New false
8420 2942 Acked 3941 Write 10 Done Done New false
8421 2943 Acked 3942 Read 1 Done Done New false
8422 2944 Acked 3943 Flush 0 Done Done New false
8423 2945 Acked 3944 Read 3 Done Done New false
8424 2946 Acked 3945 Read 7 Done Done New false
8425 2947 Acked 3946 Write 8 Done Done New false
8426 2948 Acked 3947 Read 10 Done Done New false
8427 2949 Acked 3948 Write 4 Done Done New false
8428 2950 Acked 3949 Read 4 Done Done New false
8429 2951 Acked 3950 Write 10 Done Done New false
8430 2952 Acked 3951 Write 8 Done Done New false
8431 2953 Acked 3952 Write 8 Done Done New false
8432 2954 Acked 3953 Write 4 Done Done New false
8433 2955 Acked 3954 Read 1 Done Done New false
8434 2956 Acked 3955 Write 1 Done Done New false
8435 2957 Acked 3956 Read 10 Done Done New false
8436 2958 Acked 3957 Write 10 Done Done New false
8437 2959 Acked 3958 Write 8 Done Done New false
8438 2960 Acked 3959 Write 9 Done Done New false
8439 2961 Acked 3960 Read 9 Done Done New false
8440 2962 Acked 3961 Read 2 Done Done New false
8441 2963 Acked 3962 Flush 0 Done Done New false
8442 2964 Acked 3963 Read 1 Done Done New false
8443 2965 Acked 3964 Flush 0 Done Done New false
8444 2966 Acked 3965 Flush 0 Done Done New false
8445 2967 Acked 3966 Write 3 Done Done New false
8446 2968 Acked 3967 Write 4 Done Done New false
8447 2969 Acked 3968 Read 3 Done Done New false
8448 2970 Acked 3969 Write 6 Done Done New false
8449 2971 Acked 3970 Flush 0 Done Done New false
8450 2972 Acked 3971 Read 9 Done Done New false
8451 2973 Acked 3972 Read 9 Done Done New false
8452 2974 Acked 3973 Read 7 Done Done New false
8453 2975 Acked 3974 Read 6 Done Done New false
8454 2976 Acked 3975 Read 2 Done Done New false
8455 2977 Acked 3976 Write 3 Done Done New false
8456 2978 Acked 3977 Flush 0 Done Done New false
8457 2979 Acked 3978 Write 8 Done Done New false
8458 2980 Acked 3979 Read 1 Done Done New false
8459 2981 Acked 3980 Read 1 Done Done New false
8460 2982 Acked 3981 Write 5 Done Done New false
8461 2983 Acked 3982 Read 3 Done Done New false
8462 2984 Acked 3983 Write 4 Done Done New false
8463 2985 Acked 3984 Read 3 Done Done New false
8464 2986 Acked 3985 Read 7 Done Done New false
8465 2987 Acked 3986 Write 5 Done Done New false
8466 2988 Acked 3987 Write 10 Done Done New false
8467 2989 Acked 3988 Read 2 Done Done New false
8468 2990 Acked 3989 Write 1 Done Done New false
8469 2991 Acked 3990 Read 5 Done Done New false
8470 2992 Acked 3991 Write 10 Done Done New false
8471 2993 Acked 3992 Read 4 Done Done New false
8472 2994 Acked 3993 Flush 0 Done Done New false
8473 2995 Acked 3994 Write 3 Done Done New false
8474 2996 Acked 3995 Flush 0 Done Done New false
8475 2997 Acked 3996 Read 2 Done Done New false
8476 2998 Acked 3997 Read 8 Done Done New false
8477 2999 Acked 3998 Write 5 Done Done New false
8478 3000 Acked 3999 Read 6 Done Done New false
8479 3001 Acked 4000 Write 8 Done Done New false
8480 3002 Acked 4001 Read 1 Done Done New false
8481 3003 Acked 4002 Read 2 Done Done New false
8482 3004 Acked 4003 Read 3 Done Done New false
8483 3005 Acked 4004 Read 4 Done Done New false
8484 3006 Acked 4005 Read 1 Done Done New false
8485 3007 Acked 4006 Write 7 Done Done New false
8486 3008 Acked 4007 Read 10 Done Done New false
8487 3009 Acked 4008 Read 1 Done Done New false
8488 3010 Acked 4009 Write 8 Done Done New false
8489 3011 Acked 4010 Write 3 Done Done New false
8490 3012 Acked 4011 Read 7 Done Done New false
8491 3013 Acked 4012 Write 10 Done Done New false
8492 3014 Acked 4013 Write 7 Done Done New false
8493 3015 Acked 4014 Read 7 Done Done New false
8494 3016 Acked 4015 Read 9 Done Done New false
8495 3017 Acked 4016 Write 1 Done Done New false
8496 3018 Acked 4017 Read 5 Done Done New false
8497 3019 Acked 4018 Write 1 Done Done New false
8498 3020 Acked 4019 Read 10 Done Done New false
8499 3021 Acked 4020 Read 10 Done Done New false
8500 3022 Acked 4021 Read 1 Done Done New false
8501 3023 Acked 4022 Write 10 Done Done New false
8502 3024 Acked 4023 Read 8 Done Done New false
8503 3025 Acked 4024 Write 2 Done Done New false
8504 3026 Acked 4025 Flush 0 Done Done New false
8505 STATES DS:0 DS:1 DS:2 TOTAL
8506 New 0 0 302 302
8507 Sent 0 0 0 0
8508 Done 302 302 0 604
8509 Skipped 0 0 0 0
8510 Error 0 0 0 0
8511 Last Flush: 4025 4025 3723
8512 Downstairs last five completed: 3723 3722 3721 3720 3719 3718
8513 Upstairs last five completed: 3026 3025 3024 3023 3022 3021
8514 CLIENT: Up:0 ds:302 act:2
8515 [8830] got start action
8516 [8830] I am going to restart
8517 Starting downstairs at port 8830
8518 Make output file at "/tmp/dsc/downstairs-8830.txt"
8519 [8830][2] reports Starting
8520 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1186)
8521 [8830][2] reports Running
85222023-10-13T04:43:36.692ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 2
85232023-10-13T04:43:36.692ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
85242023-10-13T04:43:36.692ZINFOcrucible: [2] client re-new 302 jobs since flush 3723 = downstairs
85252023-10-13T04:43:36.693ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
85262023-10-13T04:43:36.693ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
85272023-10-13T04:43:36.693ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
85282023-10-13T04:43:36.693ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
85292023-10-13T04:43:36.693ZINFOcrucible: [2] send last flush ID to this DS: 3723
85302023-10-13T04:43:36.693ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
85312023-10-13T04:43:36.694ZINFOcrucible: [2] Replied this last flush ID: 3723
85322023-10-13T04:43:36.694ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Offline ds_transition to Replay
85332023-10-13T04:43:36.694ZINFOcrucible: [2] Transition from Offline to Replay
85342023-10-13T04:43:36.694ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
85352023-10-13T04:43:36.694ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Replay ds_transition to Active
85362023-10-13T04:43:36.694ZINFOcrucible: [2] Transition from Replay to Active
85372023-10-13T04:43:36.694ZINFOcrucible: [2] Starts cmd_loop
85382023-10-13T04:43:36.694ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
85392023-10-13T04:43:36.694ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
85402023-10-13T04:43:36.694ZINFOcrucible: Set check for repair
85412023-10-13T04:43:37.695ZWARNcrucible: [2] flow control end
85422023-10-13T04:43:37.695ZINFOcrucible: Checking if live repair is needed
85432023-10-13T04:43:37.695ZINFOcrucible: No Live Repair required at this time
8544 ----------------------------------------------------------------
8545 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
8546 Downstairs last five completed: 4025 4024 4023 4022 4021 4020
8547 Upstairs last five completed: 3026 3025 3024 3023 3022 3021
8548 CLIENT: Up:0 ds:0 act:3
8549 Replay: All jobs finished, all DS active.
8550 Main task has work to do, go find it
8551 got dsc Stop(2)
8552 stop 2
8553 Oct 13 04:43:38.573 INFO request completed, latency_us: 485, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 3e0a8689-6dfe-4038-b4ad-8a764361df73, remote_addr: 127[8830] Got stop action so:false kr:false
8554 .0.0.1:52046, local_addr: 127.0.0.1:9998
85552023-10-13T04:43:38.574ZWARNcrucible: [2] downstairs disconnected
85562023-10-13T04:43:38.575ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
85572023-10-13T04:43:38.575ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8830 closed looper = 2
8558 Oct 13 04:43:38.575 INFO request completed, latency_us: 189, response_code: 200, uri: /state/cid/2, method: GET, req_id: 72332fbd-c142-4663-a68a-8f0ec963d044, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
85592023-10-13T04:43:38.575ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
85602023-10-13T04:43:38.575ZWARNcrucible: [2] pm_task rx.recv() is None
85612023-10-13T04:43:38.575ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
85622023-10-13T04:43:38.575ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
8563 [8830] Exited with: ExitStatus(unix_wait_status(9))
8564 [8830][2] reports Exit
85652023-10-13T04:43:39.576ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
8566 Oct 13 04:43:42.577 INFO request completed, latency_us: 251, response_code: 200, uri: /state/cid/2, method: GET, req_id: dc4e707f-60e1-4ffb-b231-efc88b9dfceb, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
8567 001/300 Read block 3230 len 24576
8568 002/300 Write block 4253 len 16384 data: 2 2 2 2
8569 003/300 Write block 2368 len 4096 data: 3
8570 004/300 Read block 1379 len 4096
8571 005/300 Read block 2109 len 36864
8572 006/300 Flush
8573 007/300 Read block 1483 len 28672
8574 008/300 Write block 2145 len 20480 data: 2 2 2 2 2
8575 009/300 Read block 2363 len 8192
8576 010/300 Write block 910 len 24576 data: 2 2 3 5 6 5
8577 011/300 Read block 1901 len 12288
8578 012/300 Flush
8579 013/300 Read block 1060 len 32768
8580 014/300 Read block 1489 len 36864
8581 015/300 Write block 522 len 20480 data: 7 7 5 5 4
8582 016/300 Read block 4493 len 28672
8583 017/300 Write block 2833 len 4096 data: 3
8584 018/300 Flush
8585 019/300 Read block 1306 len 16384
8586 020/300 Write block 3391 len 24576 data: 2 2 3 3 3 4
8587 021/300 Read block 2556 len 4096
8588 022/300 Write block 4150 len 36864 data: 3 3 4 4 5 5 4 4 4
8589 023/300 Write block 3066 len 32768 data: 3 2 3 3 4 4 4 4
8590 024/300 Write block 4542 len 32768 data: 5 4 3 2 2 2 2 2
8591 025/300 Read block 2341 len 28672
8592 026/300 Write block 3802 len 16384 data: 5 4 4 5
8593 027/300 Read block 2465 len 16384
8594 028/300 Write block 3857 len 40960 data: 3 4 4 5 5 4 4 3 3 4
8595 029/300 Read block 2796 len 36864
8596 030/300 Write block 3483 len 16384 data: 3 3 3 4
8597 031/300 Read block 3951 len 24576
8598 032/300 Write block 1357 len 4096 data: 4
8599 033/300 Write block 991 len 24576 data: 4 4 5 4 4 4
8600 034/300 Write block 2318 len 36864 data: 2 2 2 4 6 6 6 4 4
8601 035/300 Read block 1568 len 28672
8602 036/300 Flush
8603 037/300 Read block 2045 len 12288
8604 038/300 Read block 1089 len 24576
8605 039/300 Write block 2421 len 28672 data: 4 3 3 2 3 3 3
8606 040/300 Read block 4704 len 4096
8607 041/300 Read block 3294 len 4096
8608 042/300 Read block 1793 len 32768
8609 043/300 Write block 3767 len 28672 data: 5 4 3 3 3 2 2
8610 044/300 Read block 2283 len 24576
8611 045/300 Read block 3778 len 32768
8612 046/300 Read block 1603 len 20480
8613 047/300 Read block 1251 len 40960
8614 048/300 Flush
8615 049/300 Write block 46 len 16384 data: 4 4 4 4
8616 050/300 Flush
8617 051/300 Read block 2392 len 16384
8618 052/300 Write block 1692 len 16384 data: 4 4 5 5
8619 053/300 Read block 1298 len 28672
8620 054/300 Read block 3859 len 16384
8621 055/300 Read block 3198 len 12288
8622 056/300 Read block 3978 len 4096
8623 057/300 Read block 1847 len 36864
8624 058/300 Flush
8625 059/300 Flush
8626 060/300 Read block 2511 len 40960
8627 061/300 Write block 1925 len 24576 data: 5 5 5 5 3 3
8628 062/300 Write block 1956 len 28672 data: 3 2 2 2 2 2 2
8629 063/300 Read block 3256 len 12288
8630 064/300 Read block 290 len 8192
8631 065/300 Read block 592 len 36864
8632 066/300 Write block 1941 len 8192 data: 3 3
8633 067/300 Write block 2032 len 24576 data: 3 3 3 2 2 2
8634 068/300 Read block 304 len 4096
8635 069/300 Read block 1981 len 24576
8636 070/300 Read block 491 len 16384
8637 071/300 Read block 2196 len 4096
8638 072/300 Read block 4359 len 24576
8639 073/300 Read block 4580 len 32768
8640 074/300 Read block 455 len 4096
8641 075/300 Write block 98 len 8192 data: 2 2
8642 076/300 Write block 3714 len 8192 data: 5 5
8643 077/300 Read block 4665 len 36864
8644 078/300 Write block 975 len 8192 data: 3 3
8645 079/300 Write block 3184 len 24576 data: 5 5 5 7 7 6
8646 080/300 Write block 2103 len 12288 data: 3 3 3
8647 081/300 Read block 2642 len 12288
8648 082/300 Read block 4049 len 32768
8649 083/300 Write block 4571 len 36864 data: 2 3 3 3 3 3 4 4 4
8650 084/300 Read block 1 len 12288
8651 085/300 Flush
8652 086/300 Read block 3963 len 12288
8653 087/300 Read block 558 len 16384
8654 088/300 Read block 3953 len 24576
8655 089/300 Read block 404 len 4096
8656 090/300 Read block 2170 len 36864
8657 091/300 Read block 883 len 8192
8658 092/300 Read block 4056 len 32768
8659 093/300 Write block 2941 len 40960 data: 4 2 2 2 2 2 2 2 2 3
8660 094/300 Read block 177 len 16384
8661 095/300 Write block 1014 len 12288 data: 5 4 4
8662 096/300 Read block 1669 len 24576
8663 097/300 Read block 1719 len 32768
8664 098/300 Write block 4630 len 24576 data: 3 3 3 3 4 4
8665 099/300 Read block 3474 len 4096
8666 100/300 Read block 2487 len 36864
8667 101/300 Write block 4834 len 24576 data: 3 3 2 2 3 2
8668 102/300 Write block 3841 len 36864 data: 4 4 4 3 3 4 4 4 5
8669 103/300 Read block 1889 len 20480
8670 104/300 Read block 2431 len 20480
8671 105/300 Read block 4449 len 40960
8672 106/300 Write block 988 len 40960 data: 4 4 4 5 5 6 5 5 5 3
8673 107/300 Write block 4262 len 8192 data: 3 4
8674 108/300 Write block 3440 len 28672 data: 2 2 2 2 2 2 2
8675 109/300 Read block 4114 len 40960
8676 110/300 Write block 613 len 36864 data: 6 5 4 3 3 2 4 4 5
8677 111/300 Read block 4899 len 32768
8678 112/300 Read block 3 len 12288
8679 113/300 Flush
8680 114/300 Write block 3992 len 4096 data: 2
8681 115/300 Write block 4247 len 28672 data: 4 4 4 2 2 2 3
8682 116/300 Read block 4125 len 36864
8683 117/300 Write block 3527 len 40960 data: 7 7 7 8 6 6 6 5 4 3
8684 118/300 Read block 4681 len 24576
8685 119/300 Read block 4053 len 12288
8686 120/300 Flush
8687 121/300 Read block 4536 len 20480
8688 122/300 Write block 4456 len 36864 data: 4 4 4 4 5 5 4 4 3
8689 123/300 Read block 267 len 24576
8690 124/300 Read block 4286 len 16384
8691 125/300 Read block 2857 len 4096
8692 126/300 Flush
8693 127/300 Read block 1646 len 16384
8694 128/300 Read block 793 len 40960
8695 129/300 Write block 27 len 36864 data: 3 3 2 2 2 2 2 2 2
8696 130/300 Read block 3473 len 28672
8697 131/300 Write block 1611 len 36864 data: 4 4 4 3 3 3 3 3 3
8698 132/300 Write block 2983 len 40960 data: 3 3 2 3 4 4 4 3 3 3
8699 133/300 Write block 2810 len 16384 data: 2 2 2 3
8700 134/300 Read block 3050 len 32768
8701 135/300 Read block 813 len 28672
8702 136/300 Write block 3524 len 36864 data: 5 5 5 8 8 8 9 7 7
8703 137/300 Read block 3598 len 32768
8704 138/300 Flush
8705 139/300 Read block 4111 len 8192
8706 140/300 Read block 1887 len 8192
8707 141/300 Read block 569 len 36864
8708 142/300 Read block 3199 len 28672
8709 143/300 Read block 4177 len 8192
8710 144/300 Read block 1389 len 4096
8711 145/300 Read block 4575 len 4096
8712 146/300 Write block 3955 len 40960 data: 3 4 4 4 4 4 4 4 3 2
8713 147/300 Write block 1669 len 12288 data: 3 3 3
8714 148/300 Read block 3055 len 24576
8715 149/300 Write block 2968 len 20480 data: 3 3 3 3 2
8716 150/300 Write block 2677 len 12288 data: 2 2 3
8717 151/300 Write block 534 len 36864 data: 4 5 3 3 3 3 3 2 2
8718 152/300 Flush
8719 153/300 Read block 1365 len 28672
8720 154/300 Read block 2438 len 4096
8721 155/300 Read block 4580 len 4096
8722 156/300 Write block 196 len 24576 data: 4 4 5 3 3 2
8723 157/300 Write block 3979 len 24576 data: 4 4 4 4 4 3
8724 158/300 Read block 4652 len 36864
8725 159/300 Write block 4121 len 28672 data: 4 2 2 2 2 3 4
8726 160/300 Read block 2828 len 12288
8727 161/300 Read block 1110 len 4096
8728 162/300 Write block 147 len 40960 data: 2 2 2 2 2 2 2 2 2 2
8729 163/300 Flush
8730 164/300 Read block 3613 len 28672
8731 165/300 Write block 5 len 8192 data: 3 4
8732 166/300 Write block 1204 len 24576 data: 3 3 3 3 2 3
8733 167/300 Write block 3048 len 8192 data: 2 2
8734 168/300 Write block 4539 len 24576 data: 5 5 5 6 5 4
8735 169/300 Flush
8736 170/300 Write block 4018 len 12288 data: 3 4 4
8737 171/300 Read block 1430 len 8192
8738 172/300 Read block 3793 len 28672
8739 173/300 Write block 4722 len 32768 data: 4 4 4 4 4 4 4 3
8740 174/300 Write block 1606 len 20480 data: 4 4 3 2 3
8741 175/300 Read block 3218 len 32768
8742 176/300 Read block 753 len 40960
8743 177/300 Write block 3039 len 40960 data: 4 4 5 6 6 3 3 2 2 3
8744 178/300 Read block 4808 len 40960
8745 179/300 Write block 4653 len 4096 data: 4
8746 180/300 Write block 3464 len 12288 data: 3 4 4
8747 181/300 Read block 1591 len 32768
8748 182/300 Read block 2048 len 36864
8749 183/300 Read block 3891 len 36864
8750 184/300 Flush
8751 185/300 Read block 3000 len 12288
8752 186/300 Read block 1651 len 36864
8753 187/300 Write block 3386 len 28672 data: 4 2 2 2 2 3 3
8754 188/300 Read block 2595 len 32768
8755 189/300 Read block 3053 len 36864
8756 190/300 Read block 2426 len 36864
8757 191/300 Flush
8758 192/300 Read block 3354 len 16384
8759 193/300 Read block 4434 len 32768
8760 194/300 Write block 800 len 16384 data: 3 3 3 3
8761 195/300 Read block 508 len 4096
8762 196/300 Write block 4438 len 4096 data: 2
8763 197/300 Flush
8764 198/300 Read block 604 len 36864
8765 199/300 Read block 1602 len 28672
8766 200/300 Read block 2591 len 36864
8767 201/300 Write block 3951 len 8192 data: 3 3
8768 202/300 Write block 3322 len 20480 data: 5 3 3 2 2
8769 203/300 Write block 1989 len 24576 data: 2 2 2 3 3 3
8770 204/300 Write block 2054 len 8192 data: 3 3
8771 205/300 Write block 1874 len 20480 data: 3 4 5 4 3
8772 206/300 Read block 4931 len 24576
8773 207/300 Write block 210 len 36864 data: 5 4 6 3 4 4 4 5 5
8774 208/300 Write block 4918 len 16384 data: 5 5 5 3
8775 209/300 Write block 4072 len 4096 data: 6
8776 210/300 Read block 2652 len 28672
8777 211/300 Read block 4246 len 40960
8778 212/300 Read block 4844 len 36864
8779 213/300 Read block 1781 len 28672
8780 214/300 Write block 2602 len 24576 data: 3 3 3 2 2 2
8781 215/300 Write block 2287 len 32768 data: 3 5 4 4 3 4 4 5
8782 216/300 Read block 327 len 36864
8783 217/300 Read block 782 len 4096
8784 218/300 Read block 2182 len 4096
8785 219/300 Read block 2260 len 32768
8786 220/300 Flush
8787 221/300 Read block 4883 len 20480
8788 222/300 Write block 2446 len 28672 data: 2 2 2 2 2 2 2
8789 223/300 Read block 239 len 4096
8790 224/300 Read block 3928 len 16384
8791 225/300 Read block 1224 len 12288
8792 226/300 Read block 1809 len 20480
8793 227/300 Flush
8794 228/300 Read block 503 len 24576
8795 229/300 Write block 1498 len 40960 data: 4 3 2 2 2 2 2 3 4 4
8796 230/300 Read block 516 len 40960
8797 231/300 Write block 1857 len 36864 data: 4 4 5 5 5 3 5 4 4
8798 232/300 Write block 3802 len 12288 data: 6 5 5
8799 233/300 Flush
8800 234/300 Write block 4256 len 20480 data: 3 2 2 2 3
8801 235/300 Read block 727 len 40960
8802 236/300 Read block 2467 len 16384
8803 237/300 Read block 1994 len 24576
8804 238/300 Read block 2157 len 12288
8805 239/300 Read block 4862 len 16384
8806 240/300 Flush
8807 241/300 Read block 715 len 20480
8808 242/300 Read block 454 len 36864
8809 243/300 Read block 115 len 8192
8810 244/300 Read block 3201 len 12288
8811 245/300 Write block 641 len 40960 data: 3 4 3 3 3 3 3 3 2 2
8812 246/300 Write block 2349 len 8192 data: 3 3
8813 247/300 Read block 976 len 36864
8814 248/300 Write block 3002 len 24576 data: 2 3 3 3 3 3
8815 249/300 Write block 321 len 36864 data: 3 3 3 3 3 3 3 3 3
8816 250/300 Read block 3482 len 12288
8817 251/300 Flush
8818 252/300 Write block 3476 len 40960 data: 2 3 3 3 3 4 3 4 4 4
8819 253/300 Write block 4409 len 24576 data: 3 4 4 4 6 6
8820 254/300 Write block 2708 len 28672 data: 4 4 3 2 2 2 2
8821 255/300 Read block 522 len 12288
8822 256/300 Read block 554 len 28672
8823 257/300 Write block 2406 len 24576 data: 3 3 3 3 3 3
8824 258/300 Read block 4872 len 24576
8825 259/300 Read block 3011 len 16384
8826 260/300 Read block 250 len 28672
8827 261/300 Read block 3807 len 20480
8828 262/300 Write block 2169 len 36864 data: 4 4 4 3 3 2 2 3 3
8829 263/300 Write block 2568 len 28672 data: 5 5 6 6 6 6 6
8830 264/300 Write block 2487 len 24576 data: 3 3 3 4 4 4
8831 265/300 Write block 375 len 20480 data: 3 3 2 2 2
8832 266/300 Read block 4793 len 4096
8833 267/300 Read block 4419 len 20480
8834 268/300 Read block 3469 len 12288
8835 269/300 Write block 3553 len 36864 data: 4 4 4 4 3 6 5 5 6
8836 270/300 Read block 4042 len 32768
8837 271/300 Read block 1529 len 28672
8838 272/300 Write block 2924 len 36864 data: 4 4 4 4 4 4 3 2 2
8839 273/300 Read block 3282 len 4096
8840 274/300 Read block 4217 len 36864
8841 275/300 Write block 64 len 16384 data: 2 2 2 2
8842 276/300 Flush
8843 277/300 Read block 2190 len 20480
8844 278/300 Flush
8845 279/300 Write block 4200 len 28672 data: 3 3 3 3 4 3 3
8846 280/300 Write block 4875 len 4096 data: 2
8847 281/300 Write block 3540 len 40960 data: 2 2 2 2 2 2 2 2 2 2
8848 282/300 Read block 3411 len 20480
8849 283/300 Write block 92 len 16384 data: 4 3 2 3
8850 284/300 Write block 2124 len 40960 data: 4 5 5 4 4 3 3 2 2 2
8851 285/300 Flush
8852 286/300 Write block 1653 len 8192 data: 2 2
8853 287/300 Read block 387 len 40960
8854 288/300 Write block 4439 len 36864 data: 2 2 3 4 4 3 3 3 4
8855 289/300 Flush
8856 290/300 Write block 3786 len 20480 data: 3 4 4 3 3
8857 291/300 Write block 3575 len 32768 data: 4 4 4 3 3 4 4 6
8858 292/300 Write block 3088 len 36864 data: 3 3 2 2 2 2 2 2 2
8859 293/300 Write block 4191 len 32768 data: 2 2 2 2 2 2 3 3
8860 294/300 Read block 4403 len 12288
8861 295/300 Write block 3017 len 20480 data: 2 2 3 4 4
8862 296/300 Read block 3314 len 16384
8863 297/300 Read block 2395 len 8192
8864 298/300 Read block 3663 len 24576
8865 299/300 Write block 4817 len 40960 data: 3 3 4 3 3 4 3 3 3 3
8866 300/300 Write block 4752 len 24576 data: 5 6 6 3 3 3
8867 Main task has work to do, go find it
8868 got dsc Start(2)
8869 start 2
8870 Oct 13 04:43:43.480 INFO request completed, latency_us: 284, response_code: 204, uri: /start/cid/2, method: POST, req_id: ff306be2-afd0-4305-8135-01cc58d598e9, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
8871 Replay: started 2, returned:Ok(())
8872 ----------------------------------------------------------------
8873 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
8874 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
8875 3027 Acked 4026 Read 6 Done Done New false
8876 3028 Acked 4027 Write 4 Done Done New false
8877 3029 Acked 4028 Write 1 Done Done New false
8878 3030 Acked 4029 Read 1 Done Done New false
8879 3031 Acked 4030 Read 9 Done Done New false
8880 3032 Acked 4031 Flush 0 Done Done New false
8881 3033 Acked 4032 Read 7 Done Done New false
8882 3034 Acked 4033 Write 5 Done Done New false
8883 3035 Acked 4034 Read 2 Done Done New false
8884 3036 Acked 4035 Write 6 Done Done New false
8885 3037 Acked 4036 Read 3 Done Done New false
8886 3038 Acked 4037 Flush 0 Done Done New false
8887 3039 Acked 4038 Read 8 Done Done New false
8888 3040 Acked 4039 Read 9 Done Done New false
8889 3041 Acked 4040 Write 5 Done Done New false
8890 3042 Acked 4041 Read 7 Done Done New false
8891 3043 Acked 4042 Write 1 Done Done New false
8892 3044 Acked 4043 Flush 0 Done Done New false
8893 3045 Acked 4044 Read 4 Done Done New false
8894 3046 Acked 4045 Write 6 Done Done New false
8895 3047 Acked 4046 Read 1 Done Done New false
8896 3048 Acked 4047 Write 9 Done Done New false
8897 3049 Acked 4048 Write 8 Done Done New false
8898 3050 Acked 4049 Write 8 Done Done New false
8899 3051 Acked 4050 Read 7 Done Done New false
8900 3052 Acked 4051 Write 4 Done Done New false
8901 3053 Acked 4052 Read 4 Done Done New false
8902 3054 Acked 4053 Write 10 Done Done New false
8903 3055 Acked 4054 Read 9 Done Done New false
8904 3056 Acked 4055 Write 4 Done Done New false
8905 3057 Acked 4056 Read 6 Done Done New false
8906 3058 Acked 4057 Write 1 Done Done New false
8907 3059 Acked 4058 Write 6 Done Done New false
8908 3060 Acked 4059 Write 9 Done Done New false
8909 3061 Acked 4060 Read 7 Done Done New false
8910 3062 Acked 4061 Flush 0 Done Done New false
8911 3063 Acked 4062 Read 3 Done Done New false
8912 3064 Acked 4063 Read 6 Done Done New false
8913 3065 Acked 4064 Write 7 Done Done New false
8914 3066 Acked 4065 Read 1 Done Done New false
8915 3067 Acked 4066 Read 1 Done Done New false
8916 3068 Acked 4067 Read 8 Done Done New false
8917 3069 Acked 4068 Write 7 Done Done New false
8918 3070 Acked 4069 Read 6 Done Done New false
8919 3071 Acked 4070 Read 8 Done Done New false
8920 3072 Acked 4071 Read 5 Done Done New false
8921 3073 Acked 4072 Read 10 Done Done New false
8922 3074 Acked 4073 Flush 0 Done Done New false
8923 3075 Acked 4074 Write 4 Done Done New false
8924 3076 Acked 4075 Flush 0 Done Done New false
8925 3077 Acked 4076 Read 4 Done Done New false
8926 3078 Acked 4077 Write 4 Done Done New false
8927 3079 Acked 4078 Read 7 Done Done New false
8928 3080 Acked 4079 Read 4 Done Done New false
8929 3081 Acked 4080 Read 3 Done Done New false
8930 3082 Acked 4081 Read 1 Done Done New false
8931 3083 Acked 4082 Read 9 Done Done New false
8932 3084 Acked 4083 Flush 0 Done Done New false
8933 3085 Acked 4084 Flush 0 Done Done New false
8934 3086 Acked 4085 Read 10 Done Done New false
8935 3087 Acked 4086 Write 6 Done Done New false
8936 3088 Acked 4087 Write 7 Done Done New false
8937 3089 Acked 4088 Read 3 Done Done New false
8938 3090 Acked 4089 Read 2 Done Done New false
8939 3091 Acked 4090 Read 9 Done Done New false
8940 3092 Acked 4091 Write 2 Done Done New false
8941 3093 Acked 4092 Write 6 Done Done New false
8942 3094 Acked 4093 Read 1 Done Done New false
8943 3095 Acked 4094 Read 6 Done Done New false
8944 3096 Acked 4095 Read 4 Done Done New false
8945 3097 Acked 4096 Read 1 Done Done New false
8946 3098 Acked 4097 Read 6 Done Done New false
8947 3099 Acked 4098 Read 8 Done Done New false
8948 3100 Acked 4099 Read 1 Done Done New false
8949 3101 Acked 4100 Write 2 Done Done New false
8950 3102 Acked 4101 Write 2 Done Done New false
8951 3103 Acked 4102 Read 9 Done Done New false
8952 3104 Acked 4103 Write 2 Done Done New false
8953 3105 Acked 4104 Write 6 Done Done New false
8954 3106 Acked 4105 Write 3 Done Done New false
8955 3107 Acked 4106 Read 3 Done Done New false
8956 3108 Acked 4107 Read 8 Done Done New false
8957 3109 Acked 4108 Write 9 Done Done New false
8958 3110 Acked 4109 Read 3 Done Done New false
8959 3111 Acked 4110 Flush 0 Done Done New false
8960 3112 Acked 4111 Read 3 Done Done New false
8961 3113 Acked 4112 Read 4 Done Done New false
8962 3114 Acked 4113 Read 6 Done Done New false
8963 3115 Acked 4114 Read 1 Done Done New false
8964 3116 Acked 4115 Read 9 Done Done New false
8965 3117 Acked 4116 Read 2 Done Done New false
8966 3118 Acked 4117 Read 8 Done Done New false
8967 3119 Acked 4118 Write 10 Done Done New false
8968 3120 Acked 4119 Read 4 Done Done New false
8969 3121 Acked 4120 Write 3 Done Done New false
8970 3122 Acked 4121 Read 6 Done Done New false
8971 3123 Acked 4122 Read 8 Done Done New false
8972 3124 Acked 4123 Write 6 Done Done New false
8973 3125 Acked 4124 Read 1 Done Done New false
8974 3126 Acked 4125 Read 9 Done Done New false
8975 3127 Acked 4126 Write 6 Done Done New false
8976 3128 Acked 4127 Write 9 Done Done New false
8977 3129 Acked 4128 Read 5 Done Done New false
8978 3130 Acked 4129 Read 5 Done Done New false
8979 3131 Acked 4130 Read 10 Done Done New false
8980 3132 Acked 4131 Write 10 Done Done New false
8981 3133 Acked 4132 Write 2 Done Done New false
8982 3134 Acked 4133 Write 7 Done Done New false
8983 3135 Acked 4134 Read 10 Done Done New false
8984 3136 Acked 4135 Write 9 Done Done New false
8985 3137 Acked 4136 Read 8 Done Done New false
8986 3138 Acked 4137 Read 3 Done Done New false
8987 3139 Acked 4138 Flush 0 Done Done New false
8988 3140 Acked 4139 Write 1 Done Done New false
8989 3141 Acked 4140 Write 7 Done Done New false
8990 3142 Acked 4141 Read 9 Done Done New false
8991 3143 Acked 4142 Write 10 Done Done New false
8992 3144 Acked 4143 Read 6 Done Done New false
8993 3145 Acked 4144 Read 3 Done Done New false
8994 3146 Acked 4145 Flush 0 Done Done New false
8995 3147 Acked 4146 Read 5 Done Done New false
8996 3148 Acked 4147 Write 9 Done Done New false
8997 3149 Acked 4148 Read 6 Done Done New false
8998 3150 Acked 4149 Read 4 Done Done New false
8999 3151 Acked 4150 Read 1 Done Done New false
9000 3152 Acked 4151 Flush 0 Done Done New false
9001 3153 Acked 4152 Read 4 Done Done New false
9002 3154 Acked 4153 Read 10 Done Done New false
9003 3155 Acked 4154 Write 9 Done Done New false
9004 3156 Acked 4155 Read 7 Done Done New false
9005 3157 Acked 4156 Write 9 Done Done New false
9006 3158 Acked 4157 Write 10 Done Done New false
9007 3159 Acked 4158 Write 4 Done Done New false
9008 3160 Acked 4159 Read 8 Done Done New false
9009 3161 Acked 4160 Read 7 Done Done New false
9010 3162 Acked 4161 Write 9 Done Done New false
9011 3163 Acked 4162 Read 8 Done Done New false
9012 3164 Acked 4163 Flush 0 Done Done New false
9013 3165 Acked 4164 Read 2 Done Done New false
9014 3166 Acked 4165 Read 2 Done Done New false
9015 3167 Acked 4166 Read 9 Done Done New false
9016 3168 Acked 4167 Flush 0 Done Done New false
9017 3169 Acked 4168 Read 7 Done Done New false
9018 3170 Acked 4169 Read 2 Done Done New false
9019 3171 Acked 4170 Read 1 Done Done New false
9020 3172 Acked 4171 Read 1 Done Done New false
9021 3173 Acked 4172 Write 10 Done Done New false
9022 3174 Acked 4173 Write 3 Done Done New false
9023 3175 Acked 4174 Read 6 Done Done New false
9024 3176 Acked 4175 Write 5 Done Done New false
9025 3177 Acked 4176 Write 3 Done Done New false
9026 3178 Acked 4177 Write 9 Done Done New false
9027 3179 Acked 4178 Flush 0 Done Done New false
9028 3180 Acked 4179 Read 7 Done Done New false
9029 3181 Acked 4180 Read 1 Done Done New false
9030 3182 Acked 4181 Read 1 Done Done New false
9031 3183 Acked 4182 Write 6 Done Done New false
9032 3184 Acked 4183 Write 6 Done Done New false
9033 3185 Acked 4184 Read 9 Done Done New false
9034 3186 Acked 4185 Write 7 Done Done New false
9035 3187 Acked 4186 Read 3 Done Done New false
9036 3188 Acked 4187 Read 1 Done Done New false
9037 3189 Acked 4188 Write 10 Done Done New false
9038 3190 Acked 4189 Flush 0 Done Done New false
9039 3191 Acked 4190 Read 7 Done Done New false
9040 3192 Acked 4191 Write 2 Done Done New false
9041 3193 Acked 4192 Write 6 Done Done New false
9042 3194 Acked 4193 Write 2 Done Done New false
9043 3195 Acked 4194 Write 6 Done Done New false
9044 3196 Acked 4195 Flush 0 Done Done New false
9045 3197 Acked 4196 Write 3 Done Done New false
9046 3198 Acked 4197 Read 2 Done Done New false
9047 3199 Acked 4198 Read 7 Done Done New false
9048 3200 Acked 4199 Write 8 Done Done New false
9049 3201 Acked 4200 Write 5 Done Done New false
9050 3202 Acked 4201 Read 8 Done Done New false
9051 3203 Acked 4202 Read 10 Done Done New false
9052 3204 Acked 4203 Write 10 Done Done New false
9053 3205 Acked 4204 Read 10 Done Done New false
9054 3206 Acked 4205 Write 1 Done Done New false
9055 3207 Acked 4206 Write 3 Done Done New false
9056 3208 Acked 4207 Read 8 Done Done New false
9057 3209 Acked 4208 Read 9 Done Done New false
9058 3210 Acked 4209 Read 9 Done Done New false
9059 3211 Acked 4210 Flush 0 Done Done New false
9060 3212 Acked 4211 Read 3 Done Done New false
9061 3213 Acked 4212 Read 9 Done Done New false
9062 3214 Acked 4213 Write 7 Done Done New false
9063 3215 Acked 4214 Read 8 Done Done New false
9064 3216 Acked 4215 Read 9 Done Done New false
9065 3217 Acked 4216 Read 9 Done Done New false
9066 3218 Acked 4217 Flush 0 Done Done New false
9067 3219 Acked 4218 Read 4 Done Done New false
9068 3220 Acked 4219 Read 8 Done Done New false
9069 3221 Acked 4220 Write 4 Done Done New false
9070 3222 Acked 4221 Read 1 Done Done New false
9071 3223 Acked 4222 Write 1 Done Done New false
9072 3224 Acked 4223 Flush 0 Done Done New false
9073 3225 Acked 4224 Read 9 Done Done New false
9074 3226 Acked 4225 Read 7 Done Done New false
9075 3227 Acked 4226 Read 9 Done Done New false
9076 3228 Acked 4227 Write 2 Done Done New false
9077 3229 Acked 4228 Write 5 Done Done New false
9078 3230 Acked 4229 Write 6 Done Done New false
9079 3231 Acked 4230 Write 2 Done Done New false
9080 3232 Acked 4231 Write 5 Done Done New false
9081 3233 Acked 4232 Read 6 Done Done New false
9082 3234 Acked 4233 Write 9 Done Done New false
9083 3235 Acked 4234 Write 4 Done Done New false
9084 3236 Acked 4235 Write 1 Done Done New false
9085 3237 Acked 4236 Read 7 Done Done New false
9086 3238 Acked 4237 Read 10 Done Done New false
9087 3239 Acked 4238 Read 9 Done Done New false
9088 3240 Acked 4239 Read 7 Done Done New false
9089 3241 Acked 4240 Write 6 Done Done New false
9090 3242 Acked 4241 Write 8 Done Done New false
9091 3243 Acked 4242 Read 9 Done Done New false
9092 3244 Acked 4243 Read 1 Done Done New false
9093 3245 Acked 4244 Read 1 Done Done New false
9094 3246 Acked 4245 Read 8 Done Done New false
9095 3247 Acked 4246 Flush 0 Done Done New false
9096 3248 Acked 4247 Read 5 Done Done New false
9097 3249 Acked 4248 Write 7 Done Done New false
9098 3250 Acked 4249 Read 1 Done Done New false
9099 3251 Acked 4250 Read 4 Done Done New false
9100 3252 Acked 4251 Read 3 Done Done New false
9101 3253 Acked 4252 Read 5 Done Done New false
9102 3254 Acked 4253 Flush 0 Done Done New false
9103 3255 Acked 4254 Read 6 Done Done New false
9104 3256 Acked 4255 Write 10 Done Done New false
9105 3257 Acked 4256 Read 10 Done Done New false
9106 3258 Acked 4257 Write 9 Done Done New false
9107 3259 Acked 4258 Write 3 Done Done New false
9108 3260 Acked 4259 Flush 0 Done Done New false
9109 3261 Acked 4260 Write 5 Done Done New false
9110 3262 Acked 4261 Read 10 Done Done New false
9111 3263 Acked 4262 Read 4 Done Done New false
9112 3264 Acked 4263 Read 6 Done Done New false
9113 3265 Acked 4264 Read 3 Done Done New false
9114 3266 Acked 4265 Read 4 Done Done New false
9115 3267 Acked 4266 Flush 0 Done Done New false
9116 3268 Acked 4267 Read 5 Done Done New false
9117 3269 Acked 4268 Read 9 Done Done New false
9118 3270 Acked 4269 Read 2 Done Done New false
9119 3271 Acked 4270 Read 3 Done Done New false
9120 3272 Acked 4271 Write 10 Done Done New false
9121 3273 Acked 4272 Write 2 Done Done New false
9122 3274 Acked 4273 Read 9 Done Done New false
9123 3275 Acked 4274 Write 6 Done Done New false
9124 3276 Acked 4275 Write 9 Done Done New false
9125 3277 Acked 4276 Read 3 Done Done New false
9126 3278 Acked 4277 Flush 0 Done Done New false
9127 3279 Acked 4278 Write 10 Done Done New false
9128 3280 Acked 4279 Write 6 Done Done New false
9129 3281 Acked 4280 Write 7 Done Done New false
9130 3282 Acked 4281 Read 3 Done Done New false
9131 3283 Acked 4282 Read 7 Done Done New false
9132 3284 Acked 4283 Write 6 Done Done New false
9133 3285 Acked 4284 Read 6 Done Done New false
9134 3286 Acked 4285 Read 4 Done Done New false
9135 3287 Acked 4286 Read 7 Done Done New false
9136 3288 Acked 4287 Read 5 Done Done New false
9137 3289 Acked 4288 Write 9 Done Done New false
9138 3290 Acked 4289 Write 7 Done Done New false
9139 3291 Acked 4290 Write 6 Done Done New false
9140 3292 Acked 4291 Write 5 Done Done New false
9141 3293 Acked 4292 Read 1 Done Done New false
9142 3294 Acked 4293 Read 5 Done Done New false
9143 3295 Acked 4294 Read 3 Done Done New false
9144 3296 Acked 4295 Write 9 Done Done New false
9145 3297 Acked 4296 Read 8 Done Done New false
9146 3298 Acked 4297 Read 7 Done Done New false
9147 3299 Acked 4298 Write 9 Done Done New false
9148 3300 Acked 4299 Read 1 Done Done New false
9149 3301 Acked 4300 Read 9 Done Done New false
9150 3302 Acked 4301 Write 4 Done Done New false
9151 3303 Acked 4302 Flush 0 Done Done New false
9152 3304 Acked 4303 Read 5 Done Done New false
9153 3305 Acked 4304 Flush 0 Done Done New false
9154 3306 Acked 4305 Write 7 Done Done New false
9155 3307 Acked 4306 Write 1 Done Done New false
9156 3308 Acked 4307 Write 10 Done Done New false
9157 3309 Acked 4308 Read 5 Done Done New false
9158 3310 Acked 4309 Write 4 Done Done New false
9159 3311 Acked 4310 Write 10 Done Done New false
9160 3312 Acked 4311 Flush 0 Done Done New false
9161 3313 Acked 4312 Write 2 Done Done New false
9162 3314 Acked 4313 Read 10 Done Done New false
9163 3315 Acked 4314 Write 9 Done Done New false
9164 3316 Acked 4315 Flush 0 Done Done New false
9165 3317 Acked 4316 Write 5 Done Done New false
9166 3318 Acked 4317 Write 8 Done Done New false
9167 3319 Acked 4318 Write 9 Done Done New false
9168 3320 Acked 4319 Write 8 Done Done New false
9169 3321 Acked 4320 Read 3 Done Done New false
9170 3322 Acked 4321 Write 5 Done Done New false
9171 3323 Acked 4322 Read 4 Done Done New false
9172 3324 Acked 4323 Read 2 Done Done New false
9173 3325 Acked 4324 Read 6 Done Done New false
9174 3326 Acked 4325 Write 10 Sent Sent New false
9175 3327 Acked 4326 Write 6 Sent Sent New false
9176 STATES DS:0 DS:1 DS:2 TOTAL
9177 New 0 0 301 301
9178 Sent 2 2 0 4
9179 Done 299 299 0 598
9180 Skipped 0 0 0 0
9181 Error 0 0 0 0
9182 Last Flush: 4315 4315 4025
9183 Downstairs last five completed: 4025 4024 4023 4022 4021 4020
9184 Upstairs last five completed: 3327 3326 3325 3324 3323 3322
9185 CLIENT: Up:0 ds:301 act:2
9186 ----------------------------------------------------------------
9187 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
9188 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
9189 3027 Acked 4026 Read 6 Done Done New false
9190 3028 Acked 4027 Write 4 Done Done New false
9191 3029 Acked 4028 Write 1 Done Done New false
9192 3030 Acked 4029 Read 1 Done Done New false
9193 3031 Acked 4030 Read 9 Done Done New false
9194 3032 Acked 4031 Flush 0 Done Done New false
9195 3033 Acked 4032 Read 7 Done Done New false
9196 3034 Acked 4033 Write 5 Done Done New false
9197 3035 Acked 4034 Read 2 Done Done New false
9198 3036 Acked 4035 Write 6 Done Done New false
9199 3037 Acked 4036 Read 3 Done Done New false
9200 3038 Acked 4037 Flush 0 Done Done New false
9201 3039 Acked 4038 Read 8 Done Done New false
9202 3040 Acked 4039 Read 9 Done Done New false
9203 3041 Acked 4040 Write 5 Done Done New false
9204 3042 Acked 4041 Read 7 Done Done New false
9205 3043 Acked 4042 Write 1 Done Done New false
9206 3044 Acked 4043 Flush 0 Done Done New false
9207 3045 Acked 4044 Read 4 Done Done New false
9208 3046 Acked 4045 Write 6 Done Done New false
9209 3047 Acked 4046 Read 1 Done Done New false
9210 3048 Acked 4047 Write 9 Done Done New false
9211 3049 Acked 4048 Write 8 Done Done New false
9212 3050 Acked 4049 Write 8 Done Done New false
9213 3051 Acked 4050 Read 7 Done Done New false
9214 3052 Acked 4051 Write 4 Done Done New false
9215 3053 Acked 4052 Read 4 Done Done New false
9216 3054 Acked 4053 Write 10 Done Done New false
9217 3055 Acked 4054 Read 9 Done Done New false
9218 3056 Acked 4055 Write 4 Done Done New false
9219 3057 Acked 4056 Read 6 Done Done New false
9220 3058 Acked 4057 Write 1 Done Done New false
9221 3059 Acked 4058 Write 6 Done Done New false
9222 3060 Acked 4059 Write 9 Done Done New false
9223 3061 Acked 4060 Read 7 Done Done New false
9224 3062 Acked 4061 Flush 0 Done Done New false
9225 3063 Acked 4062 Read 3 Done Done New false
9226 3064 Acked 4063 Read 6 Done Done New false
9227 3065 Acked 4064 Write 7 Done Done New false
9228 3066 Acked 4065 Read 1 Done Done New false
9229 3067 Acked 4066 Read 1 Done Done New false
9230 3068 Acked 4067 Read 8 Done Done New false
9231 3069 Acked 4068 Write 7 Done Done New false
9232 3070 Acked 4069 Read 6 Done Done New false
9233 3071 Acked 4070 Read 8 Done Done New false
9234 3072 Acked 4071 Read 5 Done Done New false
9235 3073 Acked 4072 Read 10 Done Done New false
9236 3074 Acked 4073 Flush 0 Done Done New false
9237 3075 Acked 4074 Write 4 Done Done New false
9238 3076 Acked 4075 Flush 0 Done Done New false
9239 3077 Acked 4076 Read 4 Done Done New false
9240 3078 Acked 4077 Write 4 Done Done New false
9241 3079 Acked 4078 Read 7 Done Done New false
9242 3080 Acked 4079 Read 4 Done Done New false
9243 3081 Acked 4080 Read 3 Done Done New false
9244 3082 Acked 4081 Read 1 Done Done New false
9245 3083 Acked 4082 Read 9 Done Done New false
9246 3084 Acked 4083 Flush 0 Done Done New false
9247 3085 Acked 4084 Flush 0 Done Done New false
9248 3086 Acked 4085 Read 10 Done Done New false
9249 3087 Acked 4086 Write 6 Done Done New false
9250 3088 Acked 4087 Write 7 Done Done New false
9251 3089 Acked 4088 Read 3 Done Done New false
9252 3090 Acked 4089 Read 2 Done Done New false
9253 3091 Acked 4090 Read 9 Done Done New false
9254 3092 Acked 4091 Write 2 Done Done New false
9255 3093 Acked 4092 Write 6 Done Done New false
9256 3094 Acked 4093 Read 1 Done Done New false
9257 3095 Acked 4094 Read 6 Done Done New false
9258 3096 Acked 4095 Read 4 Done Done New false
9259 3097 Acked 4096 Read 1 Done Done New false
9260 3098 Acked 4097 Read 6 Done Done New false
9261 3099 Acked 4098 Read 8 Done Done New false
9262 3100 Acked 4099 Read 1 Done Done New false
9263 3101 Acked 4100 Write 2 Done Done New false
9264 3102 Acked 4101 Write 2 Done Done New false
9265 3103 Acked 4102 Read 9 Done Done New false
9266 3104 Acked 4103 Write 2 Done Done New false
9267 3105 Acked 4104 Write 6 Done Done New false
9268 3106 Acked 4105 Write 3 Done Done New false
9269 3107 Acked 4106 Read 3 Done Done New false
9270 3108 Acked 4107 Read 8 Done Done New false
9271 3109 Acked 4108 Write 9 Done Done New false
9272 3110 Acked 4109 Read 3 Done Done New false
9273 3111 Acked 4110 Flush 0 Done Done New false
9274 3112 Acked 4111 Read 3 Done Done New false
9275 3113 Acked 4112 Read 4 Done Done New false
9276 3114 Acked 4113 Read 6 Done Done New false
9277 3115 Acked 4114 Read 1 Done Done New false
9278 3116 Acked 4115 Read 9 Done Done New false
9279 3117 Acked 4116 Read 2 Done Done New false
9280 3118 Acked 4117 Read 8 Done Done New false
9281 3119 Acked 4118 Write 10 Done Done New false
9282 3120 Acked 4119 Read 4 Done Done New false
9283 3121 Acked 4120 Write 3 Done Done New false
9284 3122 Acked 4121 Read 6 Done Done New false
9285 3123 Acked 4122 Read 8 Done Done New false
9286 3124 Acked 4123 Write 6 Done Done New false
9287 3125 Acked 4124 Read 1 Done Done New false
9288 3126 Acked 4125 Read 9 Done Done New false
9289 3127 Acked 4126 Write 6 Done Done New false
9290 3128 Acked 4127 Write 9 Done Done New false
9291 3129 Acked 4128 Read 5 Done Done New false
9292 3130 Acked 4129 Read 5 Done Done New false
9293 3131 Acked 4130 Read 10 Done Done New false
9294 3132 Acked 4131 Write 10 Done Done New false
9295 3133 Acked 4132 Write 2 Done Done New false
9296 3134 Acked 4133 Write 7 Done Done New false
9297 3135 Acked 4134 Read 10 Done Done New false
9298 3136 Acked 4135 Write 9 Done Done New false
9299 3137 Acked 4136 Read 8 Done Done New false
9300 3138 Acked 4137 Read 3 Done Done New false
9301 3139 Acked 4138 Flush 0 Done Done New false
9302 3140 Acked 4139 Write 1 Done Done New false
9303 3141 Acked 4140 Write 7 Done Done New false
9304 3142 Acked 4141 Read 9 Done Done New false
9305 3143 Acked 4142 Write 10 Done Done New false
9306 3144 Acked 4143 Read 6 Done Done New false
9307 3145 Acked 4144 Read 3 Done Done New false
9308 3146 Acked 4145 Flush 0 Done Done New false
9309 3147 Acked 4146 Read 5 Done Done New false
9310 3148 Acked 4147 Write 9 Done Done New false
9311 3149 Acked 4148 Read 6 Done Done New false
9312 3150 Acked 4149 Read 4 Done Done New false
9313 3151 Acked 4150 Read 1 Done Done New false
9314 3152 Acked 4151 Flush 0 Done Done New false
9315 3153 Acked 4152 Read 4 Done Done New false
9316 3154 Acked 4153 Read 10 Done Done New false
9317 3155 Acked 4154 Write 9 Done Done New false
9318 3156 Acked 4155 Read 7 Done Done New false
9319 3157 Acked 4156 Write 9 Done Done New false
9320 3158 Acked 4157 Write 10 Done Done New false
9321 3159 Acked 4158 Write 4 Done Done New false
9322 3160 Acked 4159 Read 8 Done Done New false
9323 3161 Acked 4160 Read 7 Done Done New false
9324 3162 Acked 4161 Write 9 Done Done New false
9325 3163 Acked 4162 Read 8 Done Done New false
9326 3164 Acked 4163 Flush 0 Done Done New false
9327 3165 Acked 4164 Read 2 Done Done New false
9328 3166 Acked 4165 Read 2 Done Done New false
9329 3167 Acked 4166 Read 9 Done Done New false
9330 3168 Acked 4167 Flush 0 Done Done New false
9331 3169 Acked 4168 Read 7 Done Done New false
9332 3170 Acked 4169 Read 2 Done Done New false
9333 3171 Acked 4170 Read 1 Done Done New false
9334 3172 Acked 4171 Read 1 Done Done New false
9335 3173 Acked 4172 Write 10 Done Done New false
9336 3174 Acked 4173 Write 3 Done Done New false
9337 3175 Acked 4174 Read 6 Done Done New false
9338 3176 Acked 4175 Write 5 Done Done New false
9339 3177 Acked 4176 Write 3 Done Done New false
9340 3178 Acked 4177 Write 9 Done Done New false
9341 3179 Acked 4178 Flush 0 Done Done New false
9342 3180 Acked 4179 Read 7 Done Done New false
9343 3181 Acked 4180 Read 1 Done Done New false
9344 3182 Acked 4181 Read 1 Done Done New false
9345 3183 Acked 4182 Write 6 Done Done New false
9346 3184 Acked 4183 Write 6 Done Done New false
9347 3185 Acked 4184 Read 9 Done Done New false
9348 3186 Acked 4185 Write 7 Done Done New false
9349 3187 Acked 4186 Read 3 Done Done New false
9350 3188 Acked 4187 Read 1 Done Done New false
9351 3189 Acked 4188 Write 10 Done Done New false
9352 3190 Acked 4189 Flush 0 Done Done New false
9353 3191 Acked 4190 Read 7 Done Done New false
9354 3192 Acked 4191 Write 2 Done Done New false
9355 3193 Acked 4192 Write 6 Done Done New false
9356 3194 Acked 4193 Write 2 Done Done New false
9357 3195 Acked 4194 Write 6 Done Done New false
9358 3196 Acked 4195 Flush 0 Done Done New false
9359 3197 Acked 4196 Write 3 Done Done New false
9360 3198 Acked 4197 Read 2 Done Done New false
9361 3199 Acked 4198 Read 7 Done Done New false
9362 3200 Acked 4199 Write 8 Done Done New false
9363 3201 Acked 4200 Write 5 Done Done New false
9364 3202 Acked 4201 Read 8 Done Done New false
9365 3203 Acked 4202 Read 10 Done Done New false
9366 3204 Acked 4203 Write 10 Done Done New false
9367 3205 Acked 4204 Read 10 Done Done New false
9368 3206 Acked 4205 Write 1 Done Done New false
9369 3207 Acked 4206 Write 3 Done Done New false
9370 3208 Acked 4207 Read 8 Done Done New false
9371 3209 Acked 4208 Read 9 Done Done New false
9372 3210 Acked 4209 Read 9 Done Done New false
9373 3211 Acked 4210 Flush 0 Done Done New false
9374 3212 Acked 4211 Read 3 Done Done New false
9375 3213 Acked 4212 Read 9 Done Done New false
9376 3214 Acked 4213 Write 7 Done Done New false
9377 3215 Acked 4214 Read 8 Done Done New false
9378 3216 Acked 4215 Read 9 Done Done New false
9379 3217 Acked 4216 Read 9 Done Done New false
9380 3218 Acked 4217 Flush 0 Done Done New false
9381 3219 Acked 4218 Read 4 Done Done New false
9382 3220 Acked 4219 Read 8 Done Done New false
9383 3221 Acked 4220 Write 4 Done Done New false
9384 3222 Acked 4221 Read 1 Done Done New false
9385 3223 Acked 4222 Write 1 Done Done New false
9386 3224 Acked 4223 Flush 0 Done Done New false
9387 3225 Acked 4224 Read 9 Done Done New false
9388 3226 Acked 4225 Read 7 Done Done New false
9389 3227 Acked 4226 Read 9 Done Done New false
9390 3228 Acked 4227 Write 2 Done Done New false
9391 3229 Acked 4228 Write 5 Done Done New false
9392 3230 Acked 4229 Write 6 Done Done New false
9393 3231 Acked 4230 Write 2 Done Done New false
9394 3232 Acked 4231 Write 5 Done Done New false
9395 3233 Acked 4232 Read 6 Done Done New false
9396 3234 Acked 4233 Write 9 Done Done New false
9397 3235 Acked 4234 Write 4 Done Done New false
9398 3236 Acked 4235 Write 1 Done Done New false
9399 3237 Acked 4236 Read 7 Done Done New false
9400 3238 Acked 4237 Read 10 Done Done New false
9401 3239 Acked 4238 Read 9 Done Done New false
9402 3240 Acked 4239 Read 7 Done Done New false
9403 3241 Acked 4240 Write 6 Done Done New false
9404 3242 Acked 4241 Write 8 Done Done New false
9405 3243 Acked 4242 Read 9 Done Done New false
9406 3244 Acked 4243 Read 1 Done Done New false
9407 3245 Acked 4244 Read 1 Done Done New false
9408 3246 Acked 4245 Read 8 Done Done New false
9409 3247 Acked 4246 Flush 0 Done Done New false
9410 3248 Acked 4247 Read 5 Done Done New false
9411 3249 Acked 4248 Write 7 Done Done New false
9412 3250 Acked 4249 Read 1 Done Done New false
9413 3251 Acked 4250 Read 4 Done Done New false
9414 3252 Acked 4251 Read 3 Done Done New false
9415 3253 Acked 4252 Read 5 Done Done New false
9416 3254 Acked 4253 Flush 0 Done Done New false
9417 3255 Acked 4254 Read 6 Done Done New false
9418 3256 Acked 4255 Write 10 Done Done New false
9419 3257 Acked 4256 Read 10 Done Done New false
9420 3258 Acked 4257 Write 9 Done Done New false
9421 3259 Acked 4258 Write 3 Done Done New false
9422 3260 Acked 4259 Flush 0 Done Done New false
9423 3261 Acked 4260 Write 5 Done Done New false
9424 3262 Acked 4261 Read 10 Done Done New false
9425 3263 Acked 4262 Read 4 Done Done New false
9426 3264 Acked 4263 Read 6 Done Done New false
9427 3265 Acked 4264 Read 3 Done Done New false
9428 3266 Acked 4265 Read 4 Done Done New false
9429 3267 Acked 4266 Flush 0 Done Done New false
9430 3268 Acked 4267 Read 5 Done Done New false
9431 3269 Acked 4268 Read 9 Done Done New false
9432 3270 Acked 4269 Read 2 Done Done New false
9433 3271 Acked 4270 Read 3 Done Done New false
9434 3272 Acked 4271 Write 10 Done Done New false
9435 3273 Acked 4272 Write 2 Done Done New false
9436 3274 Acked 4273 Read 9 Done Done New false
9437 3275 Acked 4274 Write 6 Done Done New false
9438 3276 Acked 4275 Write 9 Done Done New false
9439 3277 Acked 4276 Read 3 Done Done New false
9440 3278 Acked 4277 Flush 0 Done Done New false
9441 3279 Acked 4278 Write 10 Done Done New false
9442 3280 Acked 4279 Write 6 Done Done New false
9443 3281 Acked 4280 Write 7 Done Done New false
9444 3282 Acked 4281 Read 3 Done Done New false
9445 3283 Acked 4282 Read 7 Done Done New false
9446 3284 Acked 4283 Write 6 Done Done New false
9447 3285 Acked 4284 Read 6 Done Done New false
9448 3286 Acked 4285 Read 4 Done Done New false
9449 3287 Acked 4286 Read 7 Done Done New false
9450 3288 Acked 4287 Read 5 Done Done New false
9451 3289 Acked 4288 Write 9 Done Done New false
9452 3290 Acked 4289 Write 7 Done Done New false
9453 3291 Acked 4290 Write 6 Done Done New false
9454 3292 Acked 4291 Write 5 Done Done New false
9455 3293 Acked 4292 Read 1 Done Done New false
9456 3294 Acked 4293 Read 5 Done Done New false
9457 3295 Acked 4294 Read 3 Done Done New false
9458 3296 Acked 4295 Write 9 Done Done New false
9459 3297 Acked 4296 Read 8 Done Done New false
9460 3298 Acked 4297 Read 7 Done Done New false
9461 3299 Acked 4298 Write 9 Done Done New false
9462 3300 Acked 4299 Read 1 Done Done New false
9463 3301 Acked 4300 Read 9 Done Done New false
9464 3302 Acked 4301 Write 4 Done Done New false
9465 3303 Acked 4302 Flush 0 Done Done New false
9466 3304 Acked 4303 Read 5 Done Done New false
9467 3305 Acked 4304 Flush 0 Done Done New false
9468 3306 Acked 4305 Write 7 Done Done New false
9469 3307 Acked 4306 Write 1 Done Done New false
9470 3308 Acked 4307 Write 10 Done Done New false
9471 3309 Acked 4308 Read 5 Done Done New false
9472 3310 Acked 4309 Write 4 Done Done New false
9473 3311 Acked 4310 Write 10 Done Done New false
9474 3312 Acked 4311 Flush 0 Done Done New false
9475 3313 Acked 4312 Write 2 Done Done New false
9476 3314 Acked 4313 Read 10 Done Done New false
9477 3315 Acked 4314 Write 9 Done Done New false
9478 3316 Acked 4315 Flush 0 Done Done New false
9479 3317 Acked 4316 Write 5 Done Done New false
9480 3318 Acked 4317 Write 8 Done Done New false
9481 3319 Acked 4318 Write 9 Done Done New false
9482 3320 Acked 4319 Write 8 Done Done New false
9483 3321 Acked 4320 Read 3 Done Done New false
9484 3322 Acked 4321 Write 5 Done Done New false
9485 3323 Acked 4322 Read 4 Done Done New false
9486 3324 Acked 4323 Read 2 Done Done New false
9487 3325 Acked 4324 Read 6 Done Done New false
9488 3326 Acked 4325 Write 10 Done Done New false
9489 3327 Acked 4326 Write 6 Done Done New false
9490 3328 Acked 4327 Flush 0 Done Done New false
9491 STATES DS:0 DS:1 DS:2 TOTAL
9492 New 0 0 302 302
9493 Sent 0 0 0 0
9494 Done 302 302 0 604
9495 Skipped 0 0 0 0
9496 Error 0 0 0 0
9497 Last Flush: 4327 4327 4025
9498 Downstairs last five completed: 4025 4024 4023 4022 4021 4020
9499 Upstairs last five completed: 3328 3327 3326 3325 3324 3323
9500 CLIENT: Up:0 ds:302 act:2
9501 [8830] got start action
9502 [8830] I am going to restart
9503 Starting downstairs at port 8830
9504 Make output file at "/tmp/dsc/downstairs-8830.txt"
9505 [8830][2] reports Starting
9506 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1187)
9507 [8830][2] reports Running
95082023-10-13T04:43:49.590ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
95092023-10-13T04:43:49.590ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 2
95102023-10-13T04:43:49.590ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
95112023-10-13T04:43:49.590ZINFOcrucible: [2] client re-new 302 jobs since flush 4025 = downstairs
95122023-10-13T04:43:49.591ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
95132023-10-13T04:43:49.591ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
95142023-10-13T04:43:49.591ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
95152023-10-13T04:43:49.591ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
95162023-10-13T04:43:49.591ZINFOcrucible: [2] send last flush ID to this DS: 4025
95172023-10-13T04:43:49.591ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
95182023-10-13T04:43:49.592ZINFOcrucible: [2] Replied this last flush ID: 4025
95192023-10-13T04:43:49.592ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Offline ds_transition to Replay
95202023-10-13T04:43:49.592ZINFOcrucible: [2] Transition from Offline to Replay
95212023-10-13T04:43:49.592ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
95222023-10-13T04:43:49.592ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Replay ds_transition to Active
95232023-10-13T04:43:49.592ZINFOcrucible: [2] Transition from Replay to Active
95242023-10-13T04:43:49.592ZINFOcrucible: [2] Starts cmd_loop
95252023-10-13T04:43:49.592ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
95262023-10-13T04:43:49.592ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
95272023-10-13T04:43:49.592ZINFOcrucible: Set check for repair
95282023-10-13T04:43:50.593ZWARNcrucible: [2] flow control end
95292023-10-13T04:43:50.593ZINFOcrucible: Checking if live repair is needed
95302023-10-13T04:43:50.593ZINFOcrucible: No Live Repair required at this time
9531 ----------------------------------------------------------------
9532 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
9533 Downstairs last five completed: 4327 4326 4325 4324 4323 4322
9534 Upstairs last five completed: 3328 3327 3326 3325 3324 3323
9535 CLIENT: Up:0 ds:0 act:3
9536 Replay: All jobs finished, all DS active.
9537 Main task has work to do, go find it
9538 got dsc Stop(2)
9539 stop 2
9540 Oct 13 04:43:51.488 INFO request completed, latency_us: 459, response_code: 204, uri: /stop/cid/2, [8830] Got stop action so:false kr:false
9541 method: POST, req_id: 3b1593c4-14d0-40b5-8780-756291216aac, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
9542 Oct 13 04:43:51.489 INFO request completed, latency_us: 170, response_code: 200, uri: /state/cid/2, method: GET, req_id: 435763a0-0b8c-46fe-aaf1-10936ebc282e, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
95432023-10-13T04:43:51.489ZWARNcrucible: [2] downstairs disconnected
95442023-10-13T04:43:51.489ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
95452023-10-13T04:43:51.489ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8830 closed looper = 2
95462023-10-13T04:43:51.489ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
95472023-10-13T04:43:51.490ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
95482023-10-13T04:43:51.490ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
95492023-10-13T04:43:51.490ZWARNcrucible: [2] pm_task rx.recv() is None
9550 [8830] Exited with: ExitStatus(unix_wait_status(9))
9551 [8830][2] reports Exit
9552 Oct 13 04:43:55.492 INFO request completed, latency_us: 246, response_code: 200, uri: /state/cid/2, method: GET, req_id: 2d1ed553-fc61-4675-a393-b8fc0b395525, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
9553 001/300 Read block 4815 len 8192
9554 002/300 Read block 236 len 40960
9555 003/300 Write block 4302 len 4096 data: 3
9556 004/300 Read block 3405 len 8192
9557 005/300 Read block 1439 len 32768
9558 006/300 Write block 3559 len 16384 data: 6 6 7 6
9559 007/300 Read block 4486 len 12288
9560 008/300 Read block 416 len 32768
9561 009/300 Write block 2016 len 40960 data: 3 3 3 2 2 3 3 3 3 3
9562 010/300 Read block 2302 len 32768
9563 011/300 Write block 661 len 20480 data: 3 4 3 3 3
9564 012/300 Read block 4582 len 24576
9565 013/300 Write block 4137 len 12288 data: 3 3 2
9566 014/300 Read block 970 len 20480
9567 015/300 Write block 601 len 24576 data: 2 2 2 2 3 3
9568 016/300 Read block 1669 len 8192
9569 017/300 Read block 1849 len 32768
9570 018/300 Write block 1883 len 28672 data: 2 2 3 3 4 3 4
9571 019/300 Write block 4307 len 12288 data: 2 2 2
9572 020/300 Read block 230 len 12288
9573 021/300 Read block 1011 len 32768
9574 022/300 Read block 570 len 24576
9575 023/300 Read block 2561 len 40960
9576 024/300 Write block 1165 len 36864 data: 3 3 3 2 2 2 2 2 2
9577 025/300 Write block 788 len 28672 data: 2 2 2 2 2 3 3
9578 026/300 Write block 692 len 32768 data: 4 4 3 5 5 4 4 4
9579 027/300 Flush
9580 028/300 Read block 2652 len 24576
9581 029/300 Write block 3199 len 36864 data: 2 3 3 2 3 3 4 6 6
9582 030/300 Write block 4950 len 36864 data: 2 3 2 3 2 2 2 2 3
9583 031/300 Read block 3669 len 28672
9584 032/300 Write block 1563 len 16384 data: 2 2 2 2
9585 033/300 Read block 863 len 16384
9586 034/300 Read block 4147 len 28672
9587 035/300 Read block 620 len 4096
9588 036/300 Write block 2087 len 12288 data: 3 3 2
9589 037/300 Write block 3444 len 20480 data: 3 3 3 2 3
9590 038/300 Write block 969 len 8192 data: 4 4
9591 039/300 Write block 1844 len 20480 data: 2 2 3 3 3
9592 040/300 Read block 3422 len 28672
9593 041/300 Read block 3788 len 12288
9594 042/300 Read block 1270 len 36864
9595 043/300 Write block 1244 len 32768 data: 3 3 2 2 2 2 2 2
9596 044/300 Write block 2235 len 4096 data: 3
9597 045/300 Write block 370 len 36864 data: 2 2 2 2 2 4 4 3 3
9598 046/300 Read block 1334 len 24576
9599 047/300 Read block 3552 len 8192
9600 048/300 Read block 4259 len 28672
9601 049/300 Flush
9602 050/300 Write block 1811 len 20480 data: 5 4 3 2 2
9603 051/300 Write block 2524 len 20480 data: 4 4 5 5 5
9604 052/300 Read block 2065 len 16384
9605 053/300 Read block 4398 len 20480
9606 054/300 Write block 157 len 16384 data: 2 2 2 2
9607 055/300 Flush
9608 056/300 Write block 2242 len 40960 data: 3 3 3 3 3 3 3 2 2 2
9609 057/300 Read block 4485 len 16384
9610 058/300 Read block 2181 len 16384
9611 059/300 Write block 3321 len 24576 data: 5 6 4 4 3 3
9612 060/300 Write block 2000 len 12288 data: 4 4 2
9613 061/300 Write block 3889 len 28672 data: 3 3 3 3 3 2 2
9614 062/300 Read block 2546 len 36864
9615 063/300 Write block 159 len 20480 data: 3 3 2 3 3
9616 064/300 Read block 4553 len 36864
9617 065/300 Write block 3163 len 32768 data: 3 2 2 2 2 2 2 3
9618 066/300 Read block 2031 len 40960
9619 067/300 Write block 3199 len 20480 data: 3 4 4 3 4
9620 068/300 Read block 2964 len 24576
9621 069/300 Write block 1900 len 16384 data: 5 3 3 3
9622 070/300 Flush
9623 071/300 Write block 3158 len 24576 data: 3 3 3 3 3 4
9624 072/300 Read block 48 len 20480
9625 073/300 Flush
9626 074/300 Read block 428 len 16384
9627 075/300 Read block 707 len 16384
9628 076/300 Read block 4278 len 24576
9629 077/300 Write block 4713 len 4096 data: 4
9630 078/300 Read block 4635 len 20480
9631 079/300 Write block 1254 len 8192 data: 3 2
9632 080/300 Read block 3384 len 4096
9633 081/300 Write block 2634 len 8192 data: 2 2
9634 082/300 Write block 1891 len 36864 data: 4 3 3 3 2 2 2 3 3
9635 083/300 Write block 400 len 40960 data: 4 4 4 4 3 3 4 3 3 3
9636 084/300 Read block 3959 len 16384
9637 085/300 Flush
9638 086/300 Flush
9639 087/300 Read block 2905 len 28672
9640 088/300 Read block 2624 len 20480
9641 089/300 Write block 2237 len 4096 data: 2
9642 090/300 Write block 983 len 20480 data: 3 3 3 4 4
9643 091/300 Write block 3839 len 20480 data: 2 3 5 5 5
9644 092/300 Read block 264 len 8192
9645 093/300 Read block 1420 len 40960
9646 094/300 Read block 4113 len 24576
9647 095/300 Flush
9648 096/300 Write block 4524 len 32768 data: 5 4 5 4 4 4 4 4
9649 097/300 Read block 3657 len 32768
9650 098/300 Write block 3852 len 40960 data: 3 3 3 3 3 4 5 5 6 6
9651 099/300 Read block 3575 len 36864
9652 100/300 Read block 4723 len 40960
9653 101/300 Flush
9654 102/300 Read block 3497 len 4096
9655 103/300 Read block 1257 len 20480
9656 104/300 Read block 1000 len 4096
9657 105/300 Write block 3942 len 32768 data: 4 4 4 2 3 3 3 3
9658 106/300 Write block 3310 len 32768 data: 2 2 2 3 3 4 4 4
9659 107/300 Write block 2553 len 36864 data: 2 2 2 3 3 3 2 2 2
9660 108/300 Read block 1095 len 12288
9661 109/300 Read block 4695 len 12288
9662 110/300 Read block 469 len 40960
9663 111/300 Write block 3001 len 8192 data: 2 3
9664 112/300 Read block 2438 len 32768
9665 113/300 Write block 4833 len 4096 data: 4
9666 114/300 Write block 970 len 40960 data: 5 3 4 3 3 4 4 3 2 2
9667 115/300 Read block 3721 len 36864
9668 116/300 Flush
9669 117/300 Read block 3729 len 32768
9670 118/300 Write block 1289 len 28672 data: 5 5 4 4 4 3 3
9671 119/300 Read block 871 len 36864
9672 120/300 Read block 1019 len 24576
9673 121/300 Write block 1232 len 36864 data: 4 3 4 4 4 4 3 3 3
9674 122/300 Read block 63 len 32768
9675 123/300 Write block 851 len 28672 data: 5 5 5 4 3 4 4
9676 124/300 Read block 92 len 40960
9677 125/300 Write block 4488 len 12288 data: 2 2 2
9678 126/300 Read block 4933 len 12288
9679 127/300 Read block 4019 len 4096
9680 128/300 Read block 4679 len 20480
9681 129/300 Read block 3981 len 16384
9682 130/300 Read block 2169 len 32768
9683 131/300 Read block 4215 len 12288
9684 132/300 Read block 1976 len 32768
9685 133/300 Read block 2219 len 4096
9686 134/300 Write block 2809 len 8192 data: 2 3
9687 135/300 Flush
9688 136/300 Read block 1117 len 24576
9689 137/300 Write block 638 len 40960 data: 5 4 3 4 5 4 4 4 4 4
9690 138/300 Read block 4047 len 32768
9691 139/300 Read block 3301 len 12288
9692 140/300 Read block 4296 len 4096
9693 141/300 Write block 723 len 40960 data: 3 3 4 3 4 4 4 4 3 3
9694 142/300 Write block 2848 len 8192 data: 4 4
9695 143/300 Write block 2763 len 32768 data: 4 4 3 4 4 5 6 4
9696 144/300 Write block 3522 len 12288 data: 3 3 6
9697 145/300 Write block 297 len 4096 data: 5
9698 146/300 Flush
9699 147/300 Write block 3486 len 16384 data: 5 4 4 4
9700 148/300 Read block 58 len 8192
9701 149/300 Read block 4293 len 16384
9702 150/300 Read block 797 len 20480
9703 151/300 Flush
9704 152/300 Read block 3258 len 20480
9705 153/300 Flush
9706 154/300 Read block 425 len 28672
9707 155/300 Read block 4049 len 16384
9708 156/300 Read block 2660 len 40960
9709 157/300 Write block 901 len 16384 data: 4 4 4 4
9710 158/300 Read block 504 len 40960
9711 159/300 Read block 860 len 12288
9712 160/300 Write block 2411 len 8192 data: 4 2
9713 161/300 Write block 1745 len 36864 data: 4 3 4 3 4 3 5 5 5
9714 162/300 Write block 398 len 12288 data: 4 4 5
9715 163/300 Read block 742 len 24576
9716 164/300 Read block 847 len 24576
9717 165/300 Flush
9718 166/300 Read block 4694 len 24576
9719 167/300 Read block 626 len 24576
9720 168/300 Write block 1810 len 16384 data: 5 6 5 4
9721 169/300 Write block 4468 len 12288 data: 3 3 3
9722 170/300 Write block 4486 len 16384 data: 2 3 3 3
9723 171/300 Read block 434 len 24576
9724 172/300 Write block 1310 len 16384 data: 2 2 2 3
9725 173/300 Read block 704 len 8192
9726 174/300 Read block 914 len 40960
9727 175/300 Read block 1280 len 20480
9728 176/300 Write block 4132 len 16384 data: 3 3 3 5
9729 177/300 Read block 513 len 36864
9730 178/300 Read block 995 len 20480
9731 179/300 Read block 4367 len 4096
9732 180/300 Write block 4536 len 28672 data: 3 3 3 6 6 6 7
9733 181/300 Read block 831 len 40960
9734 182/300 Read block 3826 len 40960
9735 183/300 Flush
9736 184/300 Read block 3744 len 28672
9737 185/300 Read block 3466 len 12288
9738 186/300 Flush
9739 187/300 Read block 1996 len 36864
9740 188/300 Write block 3064 len 12288 data: 3 3 4
9741 189/300 Read block 2447 len 36864
9742 190/300 Read block 3757 len 24576
9743 191/300 Read block 1673 len 12288
9744 192/300 Read block 4128 len 8192
9745 193/300 Read block 2112 len 12288
9746 194/300 Read block 3048 len 36864
9747 195/300 Write block 1506 len 40960 data: 5 5 4 4 4 4 4 4 4 4
9748 196/300 Read block 3665 len 16384
9749 197/300 Read block 4505 len 24576
9750 198/300 Write block 509 len 36864 data: 4 4 5 5 5 5 5 4 6
9751 199/300 Read block 4093 len 12288
9752 200/300 Flush
9753 201/300 Write block 3323 len 12288 data: 5 5 4
9754 202/300 Flush
9755 203/300 Read block 4491 len 20480
9756 204/300 Write block 3453 len 8192 data: 4 5
9757 205/300 Flush
9758 206/300 Read block 2130 len 28672
9759 207/300 Read block 1885 len 4096
9760 208/300 Read block 1109 len 16384
9761 209/300 Write block 4807 len 32768 data: 3 3 3 3 3 3 3 3
9762 210/300 Write block 2219 len 32768 data: 4 4 4 4 3 4 4 2
9763 211/300 Write block 4205 len 32768 data: 4 4 3 3 3 4 4 4
9764 212/300 Write block 4968 len 16384 data: 2 2 2 4
9765 213/300 Read block 2591 len 12288
9766 214/300 Read block 2808 len 16384
9767 215/300 Read block 671 len 32768
9768 216/300 Read block 1332 len 8192
9769 217/300 Write block 1298 len 4096 data: 3
9770 218/300 Write block 1874 len 4096 data: 4
9771 219/300 Read block 136 len 36864
9772 220/300 Write block 708 len 4096 data: 3
9773 221/300 Read block 757 len 32768
9774 222/300 Flush
9775 223/300 Read block 2085 len 36864
9776 224/300 Read block 3465 len 28672
9777 225/300 Read block 817 len 24576
9778 226/300 Read block 3704 len 28672
9779 227/300 Read block 3845 len 8192
9780 228/300 Read block 1614 len 20480
9781 229/300 Read block 1991 len 32768
9782 230/300 Read block 1384 len 40960
9783 231/300 Flush
9784 232/300 Flush
9785 233/300 Write block 1025 len 12288 data: 4 4 4
9786 234/300 Read block 4158 len 40960
9787 235/300 Write block 2037 len 32768 data: 3 2 2 3 3 3 3 3
9788 236/300 Write block 4181 len 36864 data: 4 4 4 5 3 3 2 2 2
9789 237/300 Write block 1384 len 20480 data: 4 3 2 2 2
9790 238/300 Write block 211 len 40960 data: 5 7 4 5 5 5 6 6 5 2
9791 239/300 Read block 853 len 16384
9792 240/300 Write block 2239 len 16384 data: 4 4 4 4
9793 241/300 Read block 2692 len 40960
9794 242/300 Write block 3834 len 24576 data: 4 2 2 3 3 3
9795 243/300 Read block 524 len 40960
9796 244/300 Write block 3061 len 8192 data: 2 2
9797 245/300 Write block 572 len 24576 data: 5 5 4 4 3 3
9798 246/300 Write block 2388 len 4096 data: 3
9799 247/300 Write block 2884 len 12288 data: 2 2 2
9800 248/300 Read block 2960 len 4096
9801 249/300 Flush
9802 250/300 Read block 4741 len 28672
9803 251/300 Write block 4022 len 24576 data: 6 6 6 5 4 4
9804 252/300 Write block 1185 len 4096 data: 4
9805 253/300 Read block 4183 len 36864
9806 254/300 Read block 462 len 12288
9807 255/300 Read block 3528 len 32768
9808 256/300 Read block 59 len 36864
9809 257/300 Write block 1515 len 16384 data: 5 3 2 2
9810 258/300 Read block 3014 len 20480
9811 259/300 Write block 321 len 4096 data: 4
9812 260/300 Read block 4539 len 8192
9813 261/300 Read block 3088 len 40960
9814 262/300 Flush
9815 263/300 Read block 872 len 4096
9816 264/300 Write block 4862 len 20480 data: 6 5 6 4 5
9817 265/300 Read block 2836 len 16384
9818 266/300 Read block 2984 len 24576
9819 267/300 Write block 3552 len 8192 data: 3 5
9820 268/300 Flush
9821 269/300 Write block 3355 len 40960 data: 3 4 5 5 4 5 4 4 4 4
9822 270/300 Write block 1285 len 4096 data: 5
9823 271/300 Flush
9824 272/300 Read block 2628 len 4096
9825 273/300 Write block 4570 len 28672 data: 2 3 4 4 4 4 4
9826 274/300 Read block 3909 len 16384
9827 275/300 Read block 76 len 4096
9828 276/300 Write block 702 len 28672 data: 4 4 2 3 3 3 4
9829 277/300 Write block 793 len 24576 data: 4 4 3 3 3 3
9830 278/300 Flush
9831 279/300 Read block 333 len 28672
9832 280/300 Write block 3939 len 20480 data: 4 4 4 5 5
9833 281/300 Write block 3594 len 28672 data: 2 2 2 2 2 3 3
9834 282/300 Write block 3148 len 24576 data: 2 4 3 4 4 4
9835 283/300 Flush
9836 284/300 Write block 3722 len 16384 data: 3 3 2 3
9837 285/300 Read block 4532 len 36864
9838 286/300 Read block 2302 len 20480
9839 287/300 Read block 2286 len 24576
9840 288/300 Read block 885 len 8192
9841 289/300 Read block 2378 len 36864
9842 290/300 Write block 427 len 12288 data: 3 2 2
9843 291/300 Write block 1770 len 12288 data: 5 5 5
9844 292/300 Write block 3780 len 32768 data: 5 5 4 4 3 3 4 5
9845 293/300 Write block 173 len 40960 data: 4 4 4 4 5 5 5 5 5 5
9846 294/300 Read block 504 len 8192
9847 295/300 Read block 4656 len 4096
9848 296/300 Flush
9849 297/300 Write block 4422 len 32768 data: 4 4 6 6 6 5 5 5
9850 298/300 Write block 2669 len 12288 data: 2 4 3
9851 299/300 Read block 1832 len 28672
9852 300/300 Read block 2436 len 12288
9853 Main task has work to do, go find it
9854 got dsc Start(2)
9855 start 2
9856 Oct 13 04:43:56.384 INFO request completed, latency_us: 318, response_code: 204, uri: /start/cid/2, method: POST, req_id: 71ee58d8-d5e4-406d-86e2-5e0e7d47453c, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
9857 Replay: started 2, returned:Ok(())
9858 ----------------------------------------------------------------
9859 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
9860 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
9861 3329 Acked 4328 Read 2 Done Done New false
9862 3330 Acked 4329 Read 10 Done Done New false
9863 3331 Acked 4330 Write 1 Done Done New false
9864 3332 Acked 4331 Read 2 Done Done New false
9865 3333 Acked 4332 Read 8 Done Done New false
9866 3334 Acked 4333 Write 4 Done Done New false
9867 3335 Acked 4334 Read 3 Done Done New false
9868 3336 Acked 4335 Read 8 Done Done New false
9869 3337 Acked 4336 Write 10 Done Done New false
9870 3338 Acked 4337 Read 8 Done Done New false
9871 3339 Acked 4338 Write 5 Done Done New false
9872 3340 Acked 4339 Read 6 Done Done New false
9873 3341 Acked 4340 Flush 0 Done Done New false
9874 3342 Acked 4341 Write 3 Done Done New false
9875 3343 Acked 4342 Read 5 Done Done New false
9876 3344 Acked 4343 Write 6 Done Done New false
9877 3345 Acked 4344 Read 2 Done Done New false
9878 3346 Acked 4345 Read 8 Done Done New false
9879 3347 Acked 4346 Write 7 Done Done New false
9880 3348 Acked 4347 Write 3 Done Done New false
9881 3349 Acked 4348 Read 3 Done Done New false
9882 3350 Acked 4349 Read 8 Done Done New false
9883 3351 Acked 4350 Read 6 Done Done New false
9884 3352 Acked 4351 Read 10 Done Done New false
9885 3353 Acked 4352 Write 9 Done Done New false
9886 3354 Acked 4353 Write 7 Done Done New false
9887 3355 Acked 4354 Write 8 Done Done New false
9888 3356 Acked 4355 Flush 0 Done Done New false
9889 3357 Acked 4356 Read 6 Done Done New false
9890 3358 Acked 4357 Write 9 Done Done New false
9891 3359 Acked 4358 Write 9 Done Done New false
9892 3360 Acked 4359 Read 7 Done Done New false
9893 3361 Acked 4360 Write 4 Done Done New false
9894 3362 Acked 4361 Read 4 Done Done New false
9895 3363 Acked 4362 Read 7 Done Done New false
9896 3364 Acked 4363 Read 1 Done Done New false
9897 3365 Acked 4364 Write 3 Done Done New false
9898 3366 Acked 4365 Write 5 Done Done New false
9899 3367 Acked 4366 Write 2 Done Done New false
9900 3368 Acked 4367 Write 5 Done Done New false
9901 3369 Acked 4368 Read 7 Done Done New false
9902 3370 Acked 4369 Read 3 Done Done New false
9903 3371 Acked 4370 Read 9 Done Done New false
9904 3372 Acked 4371 Write 8 Done Done New false
9905 3373 Acked 4372 Write 1 Done Done New false
9906 3374 Acked 4373 Write 9 Done Done New false
9907 3375 Acked 4374 Read 6 Done Done New false
9908 3376 Acked 4375 Read 2 Done Done New false
9909 3377 Acked 4376 Read 7 Done Done New false
9910 3378 Acked 4377 Flush 0 Done Done New false
9911 3379 Acked 4378 Write 5 Done Done New false
9912 3380 Acked 4379 Write 5 Done Done New false
9913 3381 Acked 4380 Read 4 Done Done New false
9914 3382 Acked 4381 Read 5 Done Done New false
9915 3383 Acked 4382 Write 4 Done Done New false
9916 3384 Acked 4383 Flush 0 Done Done New false
9917 3385 Acked 4384 Write 10 Done Done New false
9918 3386 Acked 4385 Read 4 Done Done New false
9919 3387 Acked 4386 Read 4 Done Done New false
9920 3388 Acked 4387 Write 6 Done Done New false
9921 3389 Acked 4388 Write 3 Done Done New false
9922 3390 Acked 4389 Write 7 Done Done New false
9923 3391 Acked 4390 Read 9 Done Done New false
9924 3392 Acked 4391 Write 5 Done Done New false
9925 3393 Acked 4392 Read 9 Done Done New false
9926 3394 Acked 4393 Write 8 Done Done New false
9927 3395 Acked 4394 Read 10 Done Done New false
9928 3396 Acked 4395 Write 5 Done Done New false
9929 3397 Acked 4396 Read 6 Done Done New false
9930 3398 Acked 4397 Write 4 Done Done New false
9931 3399 Acked 4398 Flush 0 Done Done New false
9932 3400 Acked 4399 Write 6 Done Done New false
9933 3401 Acked 4400 Read 5 Done Done New false
9934 3402 Acked 4401 Flush 0 Done Done New false
9935 3403 Acked 4402 Read 4 Done Done New false
9936 3404 Acked 4403 Read 4 Done Done New false
9937 3405 Acked 4404 Read 6 Done Done New false
9938 3406 Acked 4405 Write 1 Done Done New false
9939 3407 Acked 4406 Read 5 Done Done New false
9940 3408 Acked 4407 Write 2 Done Done New false
9941 3409 Acked 4408 Read 1 Done Done New false
9942 3410 Acked 4409 Write 2 Done Done New false
9943 3411 Acked 4410 Write 9 Done Done New false
9944 3412 Acked 4411 Write 10 Done Done New false
9945 3413 Acked 4412 Read 4 Done Done New false
9946 3414 Acked 4413 Flush 0 Done Done New false
9947 3415 Acked 4414 Flush 0 Done Done New false
9948 3416 Acked 4415 Read 7 Done Done New false
9949 3417 Acked 4416 Read 5 Done Done New false
9950 3418 Acked 4417 Write 1 Done Done New false
9951 3419 Acked 4418 Write 5 Done Done New false
9952 3420 Acked 4419 Write 5 Done Done New false
9953 3421 Acked 4420 Read 2 Done Done New false
9954 3422 Acked 4421 Read 10 Done Done New false
9955 3423 Acked 4422 Read 6 Done Done New false
9956 3424 Acked 4423 Flush 0 Done Done New false
9957 3425 Acked 4424 Write 8 Done Done New false
9958 3426 Acked 4425 Read 8 Done Done New false
9959 3427 Acked 4426 Write 10 Done Done New false
9960 3428 Acked 4427 Read 9 Done Done New false
9961 3429 Acked 4428 Read 10 Done Done New false
9962 3430 Acked 4429 Flush 0 Done Done New false
9963 3431 Acked 4430 Read 1 Done Done New false
9964 3432 Acked 4431 Read 5 Done Done New false
9965 3433 Acked 4432 Read 1 Done Done New false
9966 3434 Acked 4433 Write 8 Done Done New false
9967 3435 Acked 4434 Write 8 Done Done New false
9968 3436 Acked 4435 Write 9 Done Done New false
9969 3437 Acked 4436 Read 3 Done Done New false
9970 3438 Acked 4437 Read 3 Done Done New false
9971 3439 Acked 4438 Read 10 Done Done New false
9972 3440 Acked 4439 Write 2 Done Done New false
9973 3441 Acked 4440 Read 8 Done Done New false
9974 3442 Acked 4441 Write 1 Done Done New false
9975 3443 Acked 4442 Write 10 Done Done New false
9976 3444 Acked 4443 Read 9 Done Done New false
9977 3445 Acked 4444 Flush 0 Done Done New false
9978 3446 Acked 4445 Read 8 Done Done New false
9979 3447 Acked 4446 Write 7 Done Done New false
9980 3448 Acked 4447 Read 9 Done Done New false
9981 3449 Acked 4448 Read 6 Done Done New false
9982 3450 Acked 4449 Write 9 Done Done New false
9983 3451 Acked 4450 Read 8 Done Done New false
9984 3452 Acked 4451 Write 7 Done Done New false
9985 3453 Acked 4452 Read 10 Done Done New false
9986 3454 Acked 4453 Write 3 Done Done New false
9987 3455 Acked 4454 Read 3 Done Done New false
9988 3456 Acked 4455 Read 1 Done Done New false
9989 3457 Acked 4456 Read 5 Done Done New false
9990 3458 Acked 4457 Read 4 Done Done New false
9991 3459 Acked 4458 Read 8 Done Done New false
9992 3460 Acked 4459 Read 3 Done Done New false
9993 3461 Acked 4460 Read 8 Done Done New false
9994 3462 Acked 4461 Read 1 Done Done New false
9995 3463 Acked 4462 Write 2 Done Done New false
9996 3464 Acked 4463 Flush 0 Done Done New false
9997 3465 Acked 4464 Read 6 Done Done New false
9998 3466 Acked 4465 Write 10 Done Done New false
9999 3467 Acked 4466 Read 8 Done Done New false
10000 3468 Acked 4467 Read 3 Done Done New false
10001 3469 Acked 4468 Read 1 Done Done New false
10002 3470 Acked 4469 Write 10 Done Done New false
10003 3471 Acked 4470 Write 2 Done Done New false
10004 3472 Acked 4471 Write 8 Done Done New false
10005 3473 Acked 4472 Write 3 Done Done New false
10006 3474 Acked 4473 Write 1 Done Done New false
10007 3475 Acked 4474 Flush 0 Done Done New false
10008 3476 Acked 4475 Write 4 Done Done New false
10009 3477 Acked 4476 Read 2 Done Done New false
10010 3478 Acked 4477 Read 4 Done Done New false
10011 3479 Acked 4478 Read 5 Done Done New false
10012 3480 Acked 4479 Flush 0 Done Done New false
10013 3481 Acked 4480 Read 5 Done Done New false
10014 3482 Acked 4481 Flush 0 Done Done New false
10015 3483 Acked 4482 Read 7 Done Done New false
10016 3484 Acked 4483 Read 4 Done Done New false
10017 3485 Acked 4484 Read 10 Done Done New false
10018 3486 Acked 4485 Write 4 Done Done New false
10019 3487 Acked 4486 Read 10 Done Done New false
10020 3488 Acked 4487 Read 3 Done Done New false
10021 3489 Acked 4488 Write 2 Done Done New false
10022 3490 Acked 4489 Write 9 Done Done New false
10023 3491 Acked 4490 Write 3 Done Done New false
10024 3492 Acked 4491 Read 6 Done Done New false
10025 3493 Acked 4492 Read 6 Done Done New false
10026 3494 Acked 4493 Flush 0 Done Done New false
10027 3495 Acked 4494 Read 6 Done Done New false
10028 3496 Acked 4495 Read 6 Done Done New false
10029 3497 Acked 4496 Write 4 Done Done New false
10030 3498 Acked 4497 Write 3 Done Done New false
10031 3499 Acked 4498 Write 4 Done Done New false
10032 3500 Acked 4499 Read 6 Done Done New false
10033 3501 Acked 4500 Write 4 Done Done New false
10034 3502 Acked 4501 Read 2 Done Done New false
10035 3503 Acked 4502 Read 10 Done Done New false
10036 3504 Acked 4503 Read 5 Done Done New false
10037 3505 Acked 4504 Write 4 Done Done New false
10038 3506 Acked 4505 Read 9 Done Done New false
10039 3507 Acked 4506 Read 5 Done Done New false
10040 3508 Acked 4507 Read 1 Done Done New false
10041 3509 Acked 4508 Flush 0 Done Done New false
10042 3510 Acked 4509 Write 7 Done Done New false
10043 3511 Acked 4510 Read 10 Done Done New false
10044 3512 Acked 4511 Read 10 Done Done New false
10045 3513 Acked 4512 Flush 0 Done Done New false
10046 3514 Acked 4513 Read 7 Done Done New false
10047 3515 Acked 4514 Read 3 Done Done New false
10048 3516 Acked 4515 Flush 0 Done Done New false
10049 3517 Acked 4516 Read 9 Done Done New false
10050 3518 Acked 4517 Write 3 Done Done New false
10051 3519 Acked 4518 Read 9 Done Done New false
10052 3520 Acked 4519 Read 6 Done Done New false
10053 3521 Acked 4520 Read 3 Done Done New false
10054 3522 Acked 4521 Read 2 Done Done New false
10055 3523 Acked 4522 Read 3 Done Done New false
10056 3524 Acked 4523 Read 9 Done Done New false
10057 3525 Acked 4524 Write 10 Done Done New false
10058 3526 Acked 4525 Read 4 Done Done New false
10059 3527 Acked 4526 Read 6 Done Done New false
10060 3528 Acked 4527 Write 9 Done Done New false
10061 3529 Acked 4528 Read 3 Done Done New false
10062 3530 Acked 4529 Flush 0 Done Done New false
10063 3531 Acked 4530 Write 3 Done Done New false
10064 3532 Acked 4531 Flush 0 Done Done New false
10065 3533 Acked 4532 Read 5 Done Done New false
10066 3534 Acked 4533 Write 2 Done Done New false
10067 3535 Acked 4534 Flush 0 Done Done New false
10068 3536 Acked 4535 Read 7 Done Done New false
10069 3537 Acked 4536 Read 1 Done Done New false
10070 3538 Acked 4537 Read 4 Done Done New false
10071 3539 Acked 4538 Write 8 Done Done New false
10072 3540 Acked 4539 Write 8 Done Done New false
10073 3541 Acked 4540 Write 8 Done Done New false
10074 3542 Acked 4541 Write 4 Done Done New false
10075 3543 Acked 4542 Read 3 Done Done New false
10076 3544 Acked 4543 Read 4 Done Done New false
10077 3545 Acked 4544 Read 8 Done Done New false
10078 3546 Acked 4545 Read 2 Done Done New false
10079 3547 Acked 4546 Write 1 Done Done New false
10080 3548 Acked 4547 Write 1 Done Done New false
10081 3549 Acked 4548 Read 9 Done Done New false
10082 3550 Acked 4549 Write 1 Done Done New false
10083 3551 Acked 4550 Read 8 Done Done New false
10084 3552 Acked 4551 Flush 0 Done Done New false
10085 3553 Acked 4552 Read 9 Done Done New false
10086 3554 Acked 4553 Read 7 Done Done New false
10087 3555 Acked 4554 Read 6 Done Done New false
10088 3556 Acked 4555 Read 7 Done Done New false
10089 3557 Acked 4556 Read 2 Done Done New false
10090 3558 Acked 4557 Read 5 Done Done New false
10091 3559 Acked 4558 Read 8 Done Done New false
10092 3560 Acked 4559 Read 10 Done Done New false
10093 3561 Acked 4560 Flush 0 Done Done New false
10094 3562 Acked 4561 Flush 0 Done Done New false
10095 3563 Acked 4562 Write 3 Done Done New false
10096 3564 Acked 4563 Read 10 Done Done New false
10097 3565 Acked 4564 Write 8 Done Done New false
10098 3566 Acked 4565 Write 9 Done Done New false
10099 3567 Acked 4566 Write 5 Done Done New false
10100 3568 Acked 4567 Write 10 Done Done New false
10101 3569 Acked 4568 Read 4 Done Done New false
10102 3570 Acked 4569 Write 4 Done Done New false
10103 3571 Acked 4570 Read 10 Done Done New false
10104 3572 Acked 4571 Write 6 Done Done New false
10105 3573 Acked 4572 Read 10 Done Done New false
10106 3574 Acked 4573 Write 2 Done Done New false
10107 3575 Acked 4574 Write 6 Done Done New false
10108 3576 Acked 4575 Write 1 Done Done New false
10109 3577 Acked 4576 Write 3 Done Done New false
10110 3578 Acked 4577 Read 1 Done Done New false
10111 3579 Acked 4578 Flush 0 Done Done New false
10112 3580 Acked 4579 Read 7 Done Done New false
10113 3581 Acked 4580 Write 6 Done Done New false
10114 3582 Acked 4581 Write 1 Done Done New false
10115 3583 Acked 4582 Read 9 Done Done New false
10116 3584 Acked 4583 Read 3 Done Done New false
10117 3585 Acked 4584 Read 8 Done Done New false
10118 3586 Acked 4585 Read 9 Done Done New false
10119 3587 Acked 4586 Write 4 Done Done New false
10120 3588 Acked 4587 Read 5 Done Done New false
10121 3589 Acked 4588 Write 1 Done Done New false
10122 3590 Acked 4589 Read 2 Done Done New false
10123 3591 Acked 4590 Read 10 Done Done New false
10124 3592 Acked 4591 Flush 0 Done Done New false
10125 3593 Acked 4592 Read 1 Done Done New false
10126 3594 Acked 4593 Write 5 Done Done New false
10127 3595 Acked 4594 Read 4 Done Done New false
10128 3596 Acked 4595 Read 6 Done Done New false
10129 3597 Acked 4596 Write 2 Done Done New false
10130 3598 Acked 4597 Flush 0 Done Done New false
10131 3599 Acked 4598 Write 10 Done Done New false
10132 3600 Acked 4599 Write 1 Done Done New false
10133 3601 Acked 4600 Flush 0 Done Done New false
10134 3602 Acked 4601 Read 1 Done Done New false
10135 3603 Acked 4602 Write 7 Done Done New false
10136 3604 Acked 4603 Read 4 Done Done New false
10137 3605 Acked 4604 Read 1 Done Done New false
10138 3606 Acked 4605 Write 7 Done Done New false
10139 3607 Acked 4606 Write 6 Done Done New false
10140 3608 Acked 4607 Flush 0 Done Done New false
10141 3609 Acked 4608 Read 7 Done Done New false
10142 3610 Acked 4609 Write 5 Done Done New false
10143 3611 Acked 4610 Write 7 Done Done New false
10144 3612 Acked 4611 Write 6 Done Done New false
10145 3613 Acked 4612 Flush 0 Done Done New false
10146 3614 Acked 4613 Write 4 Done Done New false
10147 3615 Acked 4614 Read 9 Done Done New false
10148 3616 Acked 4615 Read 5 Done Done New false
10149 3617 Acked 4616 Read 6 Done Done New false
10150 3618 Acked 4617 Read 2 Done Done New false
10151 3619 Acked 4618 Read 9 Done Done New false
10152 3620 Acked 4619 Write 3 Done Done New false
10153 3621 Acked 4620 Write 3 Done Done New false
10154 3622 Acked 4621 Write 8 Done Done New false
10155 3623 Acked 4622 Write 10 Done Done New false
10156 3624 Acked 4623 Read 2 Done Done New false
10157 3625 Acked 4624 Read 1 Done Done New false
10158 3626 Acked 4625 Flush 0 Done Done New false
10159 3627 Acked 4626 Write 8 Done Done New false
10160 3628 Acked 4627 Write 3 Done Done New false
10161 3629 Acked 4628 Read 7 Done Done New false
10162 3630 Acked 4629 Read 3 Done Done New false
10163 STATES DS:0 DS:1 DS:2 TOTAL
10164 New 0 0 302 302
10165 Sent 0 0 0 0
10166 Done 302 302 0 604
10167 Skipped 0 0 0 0
10168 Error 0 0 0 0
10169 Last Flush: 4625 4625 4327
10170 Downstairs last five completed: 4327 4326 4325 4324 4323 4322
10171 Upstairs last five completed: 3630 3629 3628 3627 3626 3625
10172 CLIENT: Up:0 ds:302 act:2
10173 [8830] got start action
10174 [8830] I am going to restart
10175 Starting downstairs at port 8830
10176 Make output file at "/tmp/dsc/downstairs-8830.txt"
10177 [8830][2] reports Starting
10178 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1188)
10179 [8830][2] reports Running
101802023-10-13T04:43:58.497ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 2
101812023-10-13T04:43:58.497ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
101822023-10-13T04:43:58.497ZINFOcrucible: [2] client re-new 303 jobs since flush 4327 = downstairs
101832023-10-13T04:43:58.498ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
101842023-10-13T04:43:58.499ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
101852023-10-13T04:43:58.499ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
101862023-10-13T04:43:58.499ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
101872023-10-13T04:43:58.499ZINFOcrucible: [2] send last flush ID to this DS: 4327
101882023-10-13T04:43:58.499ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
101892023-10-13T04:43:58.499ZINFOcrucible: [2] Replied this last flush ID: 4327
101902023-10-13T04:43:58.499ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Offline ds_transition to Replay
101912023-10-13T04:43:58.499ZINFOcrucible: [2] Transition from Offline to Replay
101922023-10-13T04:43:58.499ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
101932023-10-13T04:43:58.499ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Replay ds_transition to Active
101942023-10-13T04:43:58.499ZINFOcrucible: [2] Transition from Replay to Active
101952023-10-13T04:43:58.499ZINFOcrucible: [2] Starts cmd_loop
101962023-10-13T04:43:58.499ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
101972023-10-13T04:43:58.499ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
101982023-10-13T04:43:58.499ZINFOcrucible: Set check for repair
101992023-10-13T04:43:59.501ZINFOcrucible: Checking if live repair is needed
102002023-10-13T04:43:59.501ZINFOcrucible: No Live Repair required at this time
102012023-10-13T04:43:59.501ZWARNcrucible: [2] flow control end
10202 ----------------------------------------------------------------
10203 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10204 Downstairs last five completed: 4630 4629 4628 4627 4626 4625
10205 Upstairs last five completed: 3631 3630 3629 3628 3627 3626
10206 CLIENT: Up:0 ds:0 act:3
10207 Replay: All jobs finished, all DS active.
10208 Main task has work to do, go find it
10209 got dsc Stop(0)
10210 stop 0
10211 Oct 13 04:44:00.388 INFO request completed, latency_us: 474, response_code: 204, uri: /stop/cid/0, method: POST, req_id: [8810] Got stop action so:false kr:false
10212 0bf21467-745f-4b73-a833-5bee2b7f702b, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
10213 Oct 13 04:44:00.390 INFO request completed, latency_us: 176, response_code: 200, uri: /state/cid/0, method: GET, req_id: 40dd79e9-a092-43c5-8b01-790ab32453b6, remote_addr:{"msg":"[0] downstairs disconnected","v":0,"name":"crucible","level":40,"time":"2023-10-13T04:44:00.390248072Z","hostname":"ip-10-150-1-57.us-west-2.compute.internal","pid":1176}
10214 127.0.0.1:52046, local_addr: 127.0.0.1:9998
102152023-10-13T04:44:00.390ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
102162023-10-13T04:44:00.390ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8810 closed looper = 0
102172023-10-13T04:44:00.390ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:false
102182023-10-13T04:44:00.390ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Offline Active Active
102192023-10-13T04:44:00.390ZINFOcrucible: [0] 127.0.0.1:8810 task reports offline
102202023-10-13T04:44:00.390ZWARNcrucible: [0] pm_task rx.recv() is None
10221 [8810] Exited with: ExitStatus(unix_wait_status(9))
10222 [8810][0] reports Exit
102232023-10-13T04:44:03.393ZINFOcrucible: [0] connecting to 127.0.0.1:8810 looper = 0
10224 Oct 13 04:44:04.393 INFO request completed, latency_us: 256, response_code: 200, uri: /state/cid/0, method: GET, req_id: 15912e63-f7c8-49dc-8495-6acea2961bb5, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
10225 001/300 Read block 4361 len 20480
10226 002/300 Read block 1039 len 28672
10227 003/300 Read block 2145 len 24576
10228 004/300 Write block 3800 len 40960 data: 4 4 7 6 6 6 5 5 4 4
10229 005/300 Read block 4521 len 4096
10230 006/300 Read block 2055 len 20480
10231 007/300 Write block 2544 len 40960 data: 3 3 3 2 2 2 2 2 2 3
10232 008/300 Write block 4401 len 32768 data: 3 3 3 3 3 3 2 2
10233 009/300 Write block 1361 len 24576 data: 2 3 2 2 3 3
10234 010/300 Write block 4120 len 40960 data: 3 5 3 3 3 3 4 5 4 4
10235 011/300 Write block 2852 len 36864 data: 4 4 3 3 3 3 2 2 2
10236 012/300 Read block 3745 len 8192
10237 013/300 Flush
10238 014/300 Write block 3040 len 16384 data: 5 6 7 7
10239 015/300 Write block 1821 len 40960 data: 2 2 3 5 5 5 5 5 5 5
10240 016/300 Write block 3203 len 8192 data: 5 4
10241 017/300 Write block 1681 len 28672 data: 3 3 3 3 3 3 4
10242 018/300 Read block 1646 len 28672
10243 019/300 Write block 253 len 12288 data: 4 5 7
10244 020/300 Write block 3195 len 20480 data: 2 2 2 2 4
10245 021/300 Read block 1639 len 4096
10246 022/300 Read block 3403 len 24576
10247 023/300 Write block 1948 len 28672 data: 3 3 3 3 3 3 4
10248 024/300 Read block 941 len 40960
10249 025/300 Read block 2286 len 8192
10250 026/300 Read block 1762 len 24576
10251 027/300 Read block 3086 len 8192
10252 028/300 Read block 4603 len 16384
10253 029/300 Read block 2614 len 36864
10254 030/300 Read block 4145 len 32768
10255 031/300 Read block 3225 len 4096
10256 032/300 Read block 356 len 32768
10257 033/300 Write block 1629 len 28672 data: 2 3 3 3 3 3 3
10258 034/300 Write block 514 len 8192 data: 6 6
10259 035/300 Read block 4464 len 24576
10260 036/300 Read block 1485 len 4096
10261 037/300 Read block 3201 len 36864
10262 038/300 Write block 3242 len 8192 data: 3 3
10263 039/300 Flush
10264 040/300 Write block 4825 len 8192 data: 4 4
10265 041/300 Write block 1487 len 20480 data: 3 4 4 5 6
10266 042/300 Write block 3069 len 12288 data: 4 5 5
10267 043/300 Write block 3488 len 12288 data: 5 5 4
10268 044/300 Read block 4293 len 32768
10269 045/300 Write block 3713 len 28672 data: 6 6 6 3 3 3 3
10270 046/300 Read block 2280 len 24576
10271 047/300 Write block 4538 len 32768 data: 4 7 7 7 8 6 5 3
10272 048/300 Read block 2437 len 28672
10273 049/300 Flush
10274 050/300 Write block 20 len 28672 data: 5 5 3 3 3 3 3
10275 051/300 Flush
10276 052/300 Read block 1551 len 24576
10277 053/300 Write block 307 len 4096 data: 5
10278 054/300 Write block 4099 len 28672 data: 5 5 6 6 5 4 3
10279 055/300 Write block 297 len 12288 data: 6 5 5
10280 056/300 Read block 3671 len 36864
10281 057/300 Write block 393 len 24576 data: 3 3 4 4 4 5
10282 058/300 Read block 3951 len 40960
10283 059/300 Write block 665 len 16384 data: 4 3 3 3
10284 060/300 Read block 3134 len 16384
10285 061/300 Read block 4842 len 4096
10286 062/300 Write block 1261 len 12288 data: 2 3 3
10287 063/300 Read block 1897 len 16384
10288 064/300 Flush
10289 065/300 Read block 4426 len 16384
10290 066/300 Read block 2036 len 36864
10291 067/300 Read block 1460 len 32768
10292 068/300 Read block 1946 len 12288
10293 069/300 Read block 2566 len 8192
10294 070/300 Read block 177 len 12288
10295 071/300 Read block 4042 len 24576
10296 072/300 Read block 3819 len 24576
10297 073/300 Write block 1390 len 16384 data: 2 2 2 2
10298 074/300 Read block 637 len 20480
10299 075/300 Flush
10300 076/300 Read block 1503 len 40960
10301 077/300 Read block 2045 len 32768
10302 078/300 Read block 3934 len 16384
10303 079/300 Read block 3622 len 28672
10304 080/300 Read block 968 len 4096
10305 081/300 Write block 1705 len 36864 data: 2 2 3 3 4 4 4 4 5
10306 082/300 Write block 966 len 20480 data: 2 3 4 5 6
10307 083/300 Read block 3982 len 36864
10308 084/300 Write block 4906 len 8192 data: 4 4
10309 085/300 Read block 3957 len 32768
10310 086/300 Flush
10311 087/300 Write block 4622 len 16384 data: 6 5 5 5
10312 088/300 Flush
10313 089/300 Read block 1121 len 28672
10314 090/300 Read block 2235 len 8192
10315 091/300 Flush
10316 092/300 Read block 3154 len 32768
10317 093/300 Write block 2961 len 32768 data: 2 2 2 2 2 2 3 4
10318 094/300 Write block 4801 len 36864 data: 3 3 3 3 2 2 4 4 4
10319 095/300 Read block 2284 len 28672
10320 096/300 Read block 1154 len 4096
10321 097/300 Flush
10322 098/300 Read block 1857 len 24576
10323 099/300 Write block 2597 len 12288 data: 4 3 4
10324 100/300 Read block 2926 len 12288
10325 101/300 Write block 3643 len 32768 data: 3 3 2 3 3 3 3 3
10326 102/300 Write block 4000 len 24576 data: 3 3 4 3 4 3
10327 103/300 Read block 2273 len 28672
10328 104/300 Read block 3967 len 36864
10329 105/300 Read block 4990 len 32768
10330 106/300 Write block 3055 len 20480 data: 2 2 2 2 2
10331 107/300 Read block 4453 len 24576
10332 108/300 Flush
10333 109/300 Write block 3429 len 16384 data: 6 6 5 4
10334 110/300 Write block 3791 len 12288 data: 3 3 3
10335 111/300 Write block 3380 len 20480 data: 4 4 4 4 4
10336 112/300 Read block 1986 len 28672
10337 113/300 Read block 2403 len 4096
10338 114/300 Read block 4934 len 24576
10339 115/300 Read block 316 len 28672
10340 116/300 Write block 1451 len 20480 data: 4 5 3 3 3
10341 117/300 Read block 258 len 24576
10342 118/300 Write block 3838 len 20480 data: 4 4 4 6 6
10343 119/300 Write block 898 len 32768 data: 4 4 5 5 5 5 5 3
10344 120/300 Read block 385 len 36864
10345 121/300 Flush
10346 122/300 Read block 2990 len 40960
10347 123/300 Read block 3952 len 32768
10348 124/300 Read block 2281 len 8192
10349 125/300 Read block 2174 len 8192
10350 126/300 Flush
10351 127/300 Read block 3334 len 36864
10352 128/300 Write block 4001 len 40960 data: 4 5 4 5 4 3 3 4 2 2
10353 129/300 Read block 4267 len 24576
10354 130/300 Read block 522 len 40960
10355 131/300 Write block 1097 len 40960 data: 3 4 4 5 6 6 4 4 4 4
10356 132/300 Write block 2006 len 4096 data: 5
10357 133/300 Write block 1788 len 40960 data: 3 3 3 3 3 4 3 3 2 2
10358 134/300 Read block 2056 len 36864
10359 135/300 Write block 2334 len 32768 data: 3 3 3 3 3 3 3 2
10360 136/300 Read block 3147 len 28672
10361 137/300 Read block 2912 len 12288
10362 138/300 Write block 64 len 40960 data: 3 3 3 3 2 2 2 3 4 3
10363 139/300 Write block 1945 len 12288 data: 3 3 3
10364 140/300 Write block 4069 len 12288 data: 5 5 6
10365 141/300 Write block 1036 len 4096 data: 4
10366 142/300 Read block 3878 len 4096
10367 143/300 Write block 2285 len 8192 data: 3 3
10368 144/300 Read block 4007 len 40960
10369 145/300 Read block 1143 len 16384
10370 146/300 Read block 462 len 12288
10371 147/300 Write block 1994 len 32768 data: 4 3 4 4 4 4 5 5
10372 148/300 Read block 1725 len 4096
10373 149/300 Read block 1145 len 40960
10374 150/300 Read block 4273 len 16384
10375 151/300 Write block 847 len 8192 data: 4 4
10376 152/300 Write block 982 len 32768 data: 3 4 4 4 5 5 5 5
10377 153/300 Write block 3136 len 40960 data: 4 4 4 4 5 4 4 3 4 4
10378 154/300 Write block 4433 len 40960 data: 2 2 3 3 2 3 3 3 4 5
10379 155/300 Read block 4957 len 36864
10380 156/300 Read block 4162 len 24576
10381 157/300 Flush
10382 158/300 Write block 289 len 32768 data: 4 5 5 7 6 6 5 5
10383 159/300 Write block 2134 len 32768 data: 2 2 2 2 2 3 3 3
10384 160/300 Read block 838 len 36864
10385 161/300 Read block 84 len 24576
10386 162/300 Read block 2771 len 28672
10387 163/300 Flush
10388 164/300 Write block 865 len 40960 data: 3 4 4 4 2 2 3 4 6 6
10389 165/300 Write block 1940 len 8192 data: 3 4
10390 166/300 Write block 1743 len 40960 data: 4 4 5 4 5 4 5 4 6 6
10391 167/300 Write block 1072 len 4096 data: 4
10392 168/300 Read block 3870 len 24576
10393 169/300 Write block 1493 len 32768 data: 5 5 3 4 5 5 4 3
10394 170/300 Read block 4660 len 16384
10395 171/300 Read block 2136 len 36864
10396 172/300 Flush
10397 173/300 Read block 3455 len 40960
10398 174/300 Read block 3249 len 12288
10399 175/300 Read block 273 len 8192
10400 176/300 Write block 4482 len 28672 data: 4 4 3 2 3 4 4
10401 177/300 Read block 3998 len 16384
10402 178/300 Read block 229 len 32768
10403 179/300 Flush
10404 180/300 Read block 1811 len 16384
10405 181/300 Read block 3886 len 28672
10406 182/300 Read block 836 len 36864
10407 183/300 Write block 3513 len 4096 data: 3
10408 184/300 Write block 4762 len 36864 data: 3 2 2 2 2 2 2 2 2
10409 185/300 Write block 3350 len 36864 data: 3 4 5 4 4 4 5 6 6
10410 186/300 Read block 3859 len 8192
10411 187/300 Flush
10412 188/300 Write block 1683 len 20480 data: 4 4 4 4 5
10413 189/300 Write block 3922 len 32768 data: 2 2 2 2 2 2 2 3
10414 190/300 Flush
10415 191/300 Write block 2540 len 4096 data: 4
10416 192/300 Write block 4573 len 24576 data: 5 5 5 5 5 5
10417 193/300 Read block 3396 len 16384
10418 194/300 Write block 1640 len 4096 data: 3
10419 195/300 Write block 1269 len 32768 data: 2 2 2 2 2 2 2 2
10420 196/300 Write block 1004 len 16384 data: 2 3 3 3
10421 197/300 Write block 1287 len 24576 data: 4 5 6 6 5 5
10422 198/300 Write block 2922 len 40960 data: 4 4 5 5 5 5 5 5 4 3
10423 199/300 Write block 3079 len 40960 data: 4 3 3 2 2 3 3 3 3 4
10424 200/300 Read block 1756 len 8192
10425 201/300 Write block 4200 len 16384 data: 4 4 4 4
10426 202/300 Flush
10427 203/300 Read block 3985 len 24576
10428 204/300 Write block 4711 len 32768 data: 3 4 5 4 3 2 2 2
10429 205/300 Write block 1175 len 20480 data: 2 3 3 3 3
10430 206/300 Flush
10431 207/300 Read block 4260 len 12288
10432 208/300 Read block 692 len 36864
10433 209/300 Write block 978 len 40960 data: 3 3 2 3 4 5 5 5 6 6
10434 210/300 Read block 3523 len 20480
10435 211/300 Read block 4082 len 4096
10436 212/300 Write block 3971 len 4096 data: 2
10437 213/300 Write block 4768 len 8192 data: 3 3
10438 214/300 Write block 3532 len 20480 data: 8 7 6 5 4
10439 215/300 Read block 1216 len 12288
10440 216/300 Read block 384 len 40960
10441 217/300 Read block 4901 len 16384
10442 218/300 Read block 3645 len 16384
10443 219/300 Read block 3674 len 24576
10444 220/300 Read block 2021 len 8192
10445 221/300 Read block 4836 len 28672
10446 222/300 Write block 2096 len 20480 data: 4 6 6 5 5
10447 223/300 Flush
10448 224/300 Write block 4139 len 16384 data: 3 2 2 3
10449 225/300 Flush
10450 226/300 Read block 2598 len 36864
10451 227/300 Flush
10452 228/300 Read block 1161 len 12288
10453 229/300 Read block 531 len 4096
10454 230/300 Flush
10455 231/300 Read block 3333 len 12288
10456 232/300 Read block 247 len 28672
10457 233/300 Write block 982 len 4096 data: 5
10458 234/300 Read block 2872 len 40960
10459 235/300 Read block 4596 len 40960
10460 236/300 Read block 2353 len 40960
10461 237/300 Read block 2208 len 16384
10462 238/300 Write block 188 len 16384 data: 2 2 2 2
10463 239/300 Read block 77 len 40960
10464 240/300 Read block 503 len 28672
10465 241/300 Read block 2509 len 24576
10466 242/300 Write block 1022 len 16384 data: 3 4 4 5
10467 243/300 Read block 4314 len 20480
10468 244/300 Write block 1091 len 32768 data: 4 4 4 5 4 3 4 5
10469 245/300 Write block 1720 len 12288 data: 2 2 3
10470 246/300 Flush
10471 247/300 Write block 754 len 40960 data: 6 6 6 6 4 3 3 2 2 2
10472 248/300 Read block 3044 len 40960
10473 249/300 Write block 512 len 4096 data: 6
10474 250/300 Read block 4591 len 32768
10475 251/300 Write block 4601 len 4096 data: 4
10476 252/300 Write block 1074 len 36864 data: 3 4 4 5 6 5 6 7 7
10477 253/300 Write block 1441 len 24576 data: 3 3 3 3 5 4
10478 254/300 Write block 217 len 12288 data: 7 7 6
10479 255/300 Read block 1145 len 8192
10480 256/300 Write block 25 len 8192 data: 4 4
10481 257/300 Write block 3757 len 12288 data: 3 3 2
10482 258/300 Read block 4690 len 8192
10483 259/300 Read block 727 len 20480
10484 260/300 Write block 1668 len 12288 data: 4 4 4
10485 261/300 Write block 3737 len 12288 data: 2 2 3
10486 262/300 Flush
10487 263/300 Read block 1001 len 4096
10488 264/300 Flush
10489 265/300 Read block 3642 len 4096
10490 266/300 Read block 739 len 4096
10491 267/300 Flush
10492 268/300 Read block 3277 len 32768
10493 269/300 Write block 579 len 12288 data: 3 3 3
10494 270/300 Read block 4104 len 24576
10495 271/300 Write block 4435 len 12288 data: 4 4 3
10496 272/300 Write block 2950 len 4096 data: 4
10497 273/300 Write block 1113 len 4096 data: 2
10498 274/300 Read block 4856 len 16384
10499 275/300 Read block 4103 len 28672
10500 276/300 Flush
10501 277/300 Read block 1448 len 16384
10502 278/300 Write block 1223 len 40960 data: 2 3 3 3 3 3 3 3 4 5
10503 279/300 Write block 4428 len 8192 data: 6 6
10504 280/300 Read block 103 len 8192
10505 281/300 Read block 328 len 4096
10506 282/300 Read block 4396 len 16384
10507 283/300 Write block 775 len 12288 data: 4 3 3
10508 284/300 Read block 4959 len 4096
10509 285/300 Write block 2748 len 28672 data: 4 4 3 3 3 3 2
10510 286/300 Read block 4071 len 24576
10511 287/300 Write block 2160 len 40960 data: 3 4 5 5 5 3 3 3 3 5
10512 288/300 Read block 1434 len 28672
10513 289/300 Write block 835 len 36864 data: 2 2 2 2 2 2 3 3 3
10514 290/300 Write block 4477 len 4096 data: 3
10515 291/300 Write block 3384 len 4096 data: 5
10516 292/300 Flush
10517 293/300 Read block 4228 len 8192
10518 294/300 Flush
10519 295/300 Read block 3148 len 8192
10520 296/300 Read block 4542 len 16384
10521 297/300 Read block 3219 len 12288
10522 298/300 Flush
10523 299/300 Write block 4745 len 36864 data: 3 3 4 4 4 4 5 6 7
10524 300/300 Write block 3120 len 32768 data: 3 3 3 3 3 3 3 3
10525 Main task has work to do, go find it
10526 got dsc Start(0)
10527 start 0
10528 Oct 13 04:44:05.258 INFO request completed, latency_us: 294, response_code: 204, uri: /start/cid/0, method: POST, req_id: f824b9cf-5d5a-446d-a120-c9d82e73c612, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
10529 Replay: started 0, returned:Ok(())
10530 ----------------------------------------------------------------
10531 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
10532 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
10533 3632 Acked 4631 Read 5 New Done Done false
10534 3633 Acked 4632 Read 7 New Done Done false
10535 3634 Acked 4633 Read 6 New Done Done false
10536 3635 Acked 4634 Write 10 New Done Done false
10537 3636 Acked 4635 Read 1 New Done Done false
10538 3637 Acked 4636 Read 5 New Done Done false
10539 3638 Acked 4637 Write 10 New Done Done false
10540 3639 Acked 4638 Write 8 New Done Done false
10541 3640 Acked 4639 Write 6 New Done Done false
10542 3641 Acked 4640 Write 10 New Done Done false
10543 3642 Acked 4641 Write 9 New Done Done false
10544 3643 Acked 4642 Read 2 New Done Done false
10545 3644 Acked 4643 Flush 0 New Done Done false
10546 3645 Acked 4644 Write 4 New Done Done false
10547 3646 Acked 4645 Write 10 New Done Done false
10548 3647 Acked 4646 Write 2 New Done Done false
10549 3648 Acked 4647 Write 7 New Done Done false
10550 3649 Acked 4648 Read 7 New Done Done false
10551 3650 Acked 4649 Write 3 New Done Done false
10552 3651 Acked 4650 Write 5 New Done Done false
10553 3652 Acked 4651 Read 1 New Done Done false
10554 3653 Acked 4652 Read 6 New Done Done false
10555 3654 Acked 4653 Write 7 New Done Done false
10556 3655 Acked 4654 Read 10 New Done Done false
10557 3656 Acked 4655 Read 2 New Done Done false
10558 3657 Acked 4656 Read 6 New Done Done false
10559 3658 Acked 4657 Read 2 New Done Done false
10560 3659 Acked 4658 Read 4 New Done Done false
10561 3660 Acked 4659 Read 9 New Done Done false
10562 3661 Acked 4660 Read 8 New Done Done false
10563 3662 Acked 4661 Read 1 New Done Done false
10564 3663 Acked 4662 Read 8 New Done Done false
10565 3664 Acked 4663 Write 7 New Done Done false
10566 3665 Acked 4664 Write 2 New Done Done false
10567 3666 Acked 4665 Read 6 New Done Done false
10568 3667 Acked 4666 Read 1 New Done Done false
10569 3668 Acked 4667 Read 9 New Done Done false
10570 3669 Acked 4668 Write 2 New Done Done false
10571 3670 Acked 4669 Flush 0 New Done Done false
10572 3671 Acked 4670 Write 2 New Done Done false
10573 3672 Acked 4671 Write 5 New Done Done false
10574 3673 Acked 4672 Write 3 New Done Done false
10575 3674 Acked 4673 Write 3 New Done Done false
10576 3675 Acked 4674 Read 8 New Done Done false
10577 3676 Acked 4675 Write 7 New Done Done false
10578 3677 Acked 4676 Read 6 New Done Done false
10579 3678 Acked 4677 Write 8 New Done Done false
10580 3679 Acked 4678 Read 7 New Done Done false
10581 3680 Acked 4679 Flush 0 New Done Done false
10582 3681 Acked 4680 Write 7 New Done Done false
10583 3682 Acked 4681 Flush 0 New Done Done false
10584 3683 Acked 4682 Read 6 New Done Done false
10585 3684 Acked 4683 Write 1 New Done Done false
10586 3685 Acked 4684 Write 7 New Done Done false
10587 3686 Acked 4685 Write 3 New Done Done false
10588 3687 Acked 4686 Read 9 New Done Done false
10589 3688 Acked 4687 Flush 0 New Done Done false
10590 3689 Acked 4688 Write 6 New Done Done false
10591 3690 Acked 4689 Read 10 New Done Done false
10592 3691 Acked 4690 Write 4 New Done Done false
10593 3692 Acked 4691 Read 4 New Done Done false
10594 3693 Acked 4692 Read 1 New Done Done false
10595 3694 Acked 4693 Write 3 New Done Done false
10596 3695 Acked 4694 Read 4 New Done Done false
10597 3696 Acked 4695 Flush 0 New Done Done false
10598 3697 Acked 4696 Read 4 New Done Done false
10599 3698 Acked 4697 Read 9 New Done Done false
10600 3699 Acked 4698 Read 8 New Done Done false
10601 3700 Acked 4699 Read 3 New Done Done false
10602 3701 Acked 4700 Read 2 New Done Done false
10603 3702 Acked 4701 Read 3 New Done Done false
10604 3703 Acked 4702 Read 6 New Done Done false
10605 3704 Acked 4703 Read 6 New Done Done false
10606 3705 Acked 4704 Write 4 New Done Done false
10607 3706 Acked 4705 Read 5 New Done Done false
10608 3707 Acked 4706 Flush 0 New Done Done false
10609 3708 Acked 4707 Read 10 New Done Done false
10610 3709 Acked 4708 Read 8 New Done Done false
10611 3710 Acked 4709 Read 4 New Done Done false
10612 3711 Acked 4710 Read 7 New Done Done false
10613 3712 Acked 4711 Read 1 New Done Done false
10614 3713 Acked 4712 Write 9 New Done Done false
10615 3714 Acked 4713 Write 5 New Done Done false
10616 3715 Acked 4714 Read 9 New Done Done false
10617 3716 Acked 4715 Write 2 New Done Done false
10618 3717 Acked 4716 Read 8 New Done Done false
10619 3718 Acked 4717 Flush 0 New Done Done false
10620 3719 Acked 4718 Write 4 New Done Done false
10621 3720 Acked 4719 Flush 0 New Done Done false
10622 3721 Acked 4720 Read 7 New Done Done false
10623 3722 Acked 4721 Read 2 New Done Done false
10624 3723 Acked 4722 Flush 0 New Done Done false
10625 3724 Acked 4723 Read 8 New Done Done false
10626 3725 Acked 4724 Write 8 New Done Done false
10627 3726 Acked 4725 Write 9 New Done Done false
10628 3727 Acked 4726 Read 7 New Done Done false
10629 3728 Acked 4727 Read 1 New Done Done false
10630 3729 Acked 4728 Flush 0 New Done Done false
10631 3730 Acked 4729 Read 6 New Done Done false
10632 3731 Acked 4730 Write 3 New Done Done false
10633 3732 Acked 4731 Read 3 New Done Done false
10634 3733 Acked 4732 Write 8 New Done Done false
10635 3734 Acked 4733 Write 6 New Done Done false
10636 3735 Acked 4734 Read 7 New Done Done false
10637 3736 Acked 4735 Read 9 New Done Done false
10638 3737 Acked 4736 Read 8 New Done Done false
10639 3738 Acked 4737 Write 5 New Done Done false
10640 3739 Acked 4738 Read 6 New Done Done false
10641 3740 Acked 4739 Flush 0 New Done Done false
10642 3741 Acked 4740 Write 4 New Done Done false
10643 3742 Acked 4741 Write 3 New Done Done false
10644 3743 Acked 4742 Write 5 New Done Done false
10645 3744 Acked 4743 Read 7 New Done Done false
10646 3745 Acked 4744 Read 1 New Done Done false
10647 3746 Acked 4745 Read 6 New Done Done false
10648 3747 Acked 4746 Read 7 New Done Done false
10649 3748 Acked 4747 Write 5 New Done Done false
10650 3749 Acked 4748 Read 6 New Done Done false
10651 3750 Acked 4749 Write 5 New Done Done false
10652 3751 Acked 4750 Write 8 New Done Done false
10653 3752 Acked 4751 Read 9 New Done Done false
10654 3753 Acked 4752 Flush 0 New Done Done false
10655 3754 Acked 4753 Read 10 New Done Done false
10656 3755 Acked 4754 Read 8 New Done Done false
10657 3756 Acked 4755 Read 2 New Done Done false
10658 3757 Acked 4756 Read 2 New Done Done false
10659 3758 Acked 4757 Flush 0 New Done Done false
10660 3759 Acked 4758 Read 9 New Done Done false
10661 3760 Acked 4759 Write 10 New Done Done false
10662 3761 Acked 4760 Read 6 New Done Done false
10663 3762 Acked 4761 Read 10 New Done Done false
10664 3763 Acked 4762 Write 10 New Done Done false
10665 3764 Acked 4763 Write 1 New Done Done false
10666 3765 Acked 4764 Write 10 New Done Done false
10667 3766 Acked 4765 Read 9 New Done Done false
10668 3767 Acked 4766 Write 8 New Done Done false
10669 3768 Acked 4767 Read 7 New Done Done false
10670 3769 Acked 4768 Read 3 New Done Done false
10671 3770 Acked 4769 Write 10 New Done Done false
10672 3771 Acked 4770 Write 3 New Done Done false
10673 3772 Acked 4771 Write 3 New Done Done false
10674 3773 Acked 4772 Write 1 New Done Done false
10675 3774 Acked 4773 Read 1 New Done Done false
10676 3775 Acked 4774 Write 2 New Done Done false
10677 3776 Acked 4775 Read 10 New Done Done false
10678 3777 Acked 4776 Read 4 New Done Done false
10679 3778 Acked 4777 Read 3 New Done Done false
10680 3779 Acked 4778 Write 8 New Done Done false
10681 3780 Acked 4779 Read 1 New Done Done false
10682 3781 Acked 4780 Read 10 New Done Done false
10683 3782 Acked 4781 Read 4 New Done Done false
10684 3783 Acked 4782 Write 2 New Done Done false
10685 3784 Acked 4783 Write 8 New Done Done false
10686 3785 Acked 4784 Write 10 New Done Done false
10687 3786 Acked 4785 Write 10 New Done Done false
10688 3787 Acked 4786 Read 9 New Done Done false
10689 3788 Acked 4787 Read 6 New Done Done false
10690 3789 Acked 4788 Flush 0 New Done Done false
10691 3790 Acked 4789 Write 8 New Done Done false
10692 3791 Acked 4790 Write 8 New Done Done false
10693 3792 Acked 4791 Read 9 New Done Done false
10694 3793 Acked 4792 Read 6 New Done Done false
10695 3794 Acked 4793 Read 7 New Done Done false
10696 3795 Acked 4794 Flush 0 New Done Done false
10697 3796 Acked 4795 Write 10 New Done Done false
10698 3797 Acked 4796 Write 2 New Done Done false
10699 3798 Acked 4797 Write 10 New Done Done false
10700 3799 Acked 4798 Write 1 New Done Done false
10701 3800 Acked 4799 Read 6 New Done Done false
10702 3801 Acked 4800 Write 8 New Done Done false
10703 3802 Acked 4801 Read 4 New Done Done false
10704 3803 Acked 4802 Read 9 New Done Done false
10705 3804 Acked 4803 Flush 0 New Done Done false
10706 3805 Acked 4804 Read 10 New Done Done false
10707 3806 Acked 4805 Read 3 New Done Done false
10708 3807 Acked 4806 Read 2 New Done Done false
10709 3808 Acked 4807 Write 7 New Done Done false
10710 3809 Acked 4808 Read 4 New Done Done false
10711 3810 Acked 4809 Read 8 New Done Done false
10712 3811 Acked 4810 Flush 0 New Done Done false
10713 3812 Acked 4811 Read 4 New Done Done false
10714 3813 Acked 4812 Read 7 New Done Done false
10715 3814 Acked 4813 Read 9 New Done Done false
10716 3815 Acked 4814 Write 1 New Done Done false
10717 3816 Acked 4815 Write 9 New Done Done false
10718 3817 Acked 4816 Write 9 New Done Done false
10719 3818 Acked 4817 Read 2 New Done Done false
10720 3819 Acked 4818 Flush 0 New Done Done false
10721 3820 Acked 4819 Write 5 New Done Done false
10722 3821 Acked 4820 Write 8 New Done Done false
10723 3822 Acked 4821 Flush 0 New Done Done false
10724 3823 Acked 4822 Write 1 New Done Done false
10725 3824 Acked 4823 Write 6 New Done Done false
10726 3825 Acked 4824 Read 4 New Done Done false
10727 3826 Acked 4825 Write 1 New Done Done false
10728 3827 Acked 4826 Write 8 New Done Done false
10729 3828 Acked 4827 Write 4 New Done Done false
10730 3829 Acked 4828 Write 6 New Done Done false
10731 3830 Acked 4829 Write 10 New Done Done false
10732 3831 Acked 4830 Write 10 New Done Done false
10733 3832 Acked 4831 Read 2 New Done Done false
10734 3833 Acked 4832 Write 4 New Done Done false
10735 3834 Acked 4833 Flush 0 New Done Done false
10736 3835 Acked 4834 Read 6 New Done Done false
10737 3836 Acked 4835 Write 8 New Done Done false
10738 3837 Acked 4836 Write 5 New Done Done false
10739 3838 Acked 4837 Flush 0 New Done Done false
10740 3839 Acked 4838 Read 3 New Done Done false
10741 3840 Acked 4839 Read 9 New Done Done false
10742 3841 Acked 4840 Write 10 New Done Done false
10743 3842 Acked 4841 Read 5 New Done Done false
10744 3843 Acked 4842 Read 1 New Done Done false
10745 3844 Acked 4843 Write 1 New Done Done false
10746 3845 Acked 4844 Write 2 New Done Done false
10747 3846 Acked 4845 Write 5 New Done Done false
10748 3847 Acked 4846 Read 3 New Done Done false
10749 3848 Acked 4847 Read 10 New Done Done false
10750 3849 Acked 4848 Read 4 New Done Done false
10751 3850 Acked 4849 Read 4 New Done Done false
10752 3851 Acked 4850 Read 6 New Done Done false
10753 3852 Acked 4851 Read 2 New Done Done false
10754 3853 Acked 4852 Read 7 New Done Done false
10755 3854 Acked 4853 Write 5 New Done Done false
10756 3855 Acked 4854 Flush 0 New Done Done false
10757 3856 Acked 4855 Flush 0 New Done Done false
10758 3857 Acked 4856 Write 4 New Done Done false
10759 3858 Acked 4857 Flush 0 New Done Done false
10760 3859 Acked 4858 Read 9 New Done Done false
10761 3860 Acked 4859 Flush 0 New Done Done false
10762 3861 Acked 4860 Read 3 New Done Done false
10763 3862 Acked 4861 Read 1 New Done Done false
10764 3863 Acked 4862 Flush 0 New Done Done false
10765 3864 Acked 4863 Read 3 New Done Done false
10766 3865 Acked 4864 Read 7 New Done Done false
10767 3866 Acked 4865 Write 1 New Done Done false
10768 3867 Acked 4866 Read 10 New Done Done false
10769 3868 Acked 4867 Read 10 New Done Done false
10770 3869 Acked 4868 Read 10 New Done Done false
10771 3870 Acked 4869 Read 4 New Done Done false
10772 3871 Acked 4870 Write 4 New Done Done false
10773 3872 Acked 4871 Read 10 New Done Done false
10774 3873 Acked 4872 Read 7 New Done Done false
10775 3874 Acked 4873 Read 6 New Done Done false
10776 3875 Acked 4874 Write 4 New Done Done false
10777 3876 Acked 4875 Read 5 New Done Done false
10778 3877 Acked 4876 Write 8 New Done Done false
10779 3878 Acked 4877 Write 3 New Done Done false
10780 3879 Acked 4878 Flush 0 New Done Done false
10781 3880 Acked 4879 Write 10 New Done Done false
10782 3881 Acked 4880 Read 10 New Done Done false
10783 3882 Acked 4881 Write 1 New Done Done false
10784 3883 Acked 4882 Read 8 New Done Done false
10785 3884 Acked 4883 Write 1 New Done Done false
10786 3885 Acked 4884 Write 9 New Done Done false
10787 3886 Acked 4885 Write 6 New Done Done false
10788 3887 Acked 4886 Write 3 New Done Done false
10789 3888 Acked 4887 Read 2 New Done Done false
10790 3889 Acked 4888 Write 2 New Done Done false
10791 3890 Acked 4889 Write 3 New Done Done false
10792 3891 Acked 4890 Read 2 New Done Done false
10793 3892 Acked 4891 Read 5 New Done Done false
10794 3893 Acked 4892 Write 3 New Done Done false
10795 3894 Acked 4893 Write 3 New Done Done false
10796 3895 Acked 4894 Flush 0 New Done Done false
10797 3896 Acked 4895 Read 1 New Done Done false
10798 3897 Acked 4896 Flush 0 New Done Done false
10799 3898 Acked 4897 Read 1 New Done Done false
10800 3899 Acked 4898 Read 1 New Done Done false
10801 3900 Acked 4899 Flush 0 New Done Done false
10802 3901 Acked 4900 Read 8 New Done Done false
10803 3902 Acked 4901 Write 3 New Done Done false
10804 3903 Acked 4902 Read 6 New Done Done false
10805 3904 Acked 4903 Write 3 New Done Done false
10806 3905 Acked 4904 Write 1 New Done Done false
10807 3906 Acked 4905 Write 1 New Done Done false
10808 3907 Acked 4906 Read 4 New Done Done false
10809 3908 Acked 4907 Read 7 New Done Done false
10810 3909 Acked 4908 Flush 0 New Done Done false
10811 3910 Acked 4909 Read 4 New Done Done false
10812 3911 Acked 4910 Write 10 New Done Done false
10813 3912 Acked 4911 Write 2 New Done Done false
10814 3913 Acked 4912 Read 2 New Done Done false
10815 3914 Acked 4913 Read 1 New Done Done false
10816 3915 Acked 4914 Read 4 New Done Done false
10817 3916 Acked 4915 Write 3 New Done Done false
10818 3917 Acked 4916 Read 1 New Done Done false
10819 3918 Acked 4917 Write 7 New Done Done false
10820 3919 Acked 4918 Read 6 New Done Done false
10821 3920 Acked 4919 Write 10 New Done Done false
10822 3921 Acked 4920 Read 7 New Done Done false
10823 3922 Acked 4921 Write 9 New Done Done false
10824 3923 Acked 4922 Write 1 New Done Done false
10825 3924 Acked 4923 Write 1 New Done Done false
10826 3925 Acked 4924 Flush 0 New Done Done false
10827 3926 Acked 4925 Read 2 New Done Done false
10828 3927 Acked 4926 Flush 0 New Done Done false
10829 3928 Acked 4927 Read 2 New Done Done false
10830 3929 Acked 4928 Read 4 New Done Done false
10831 3930 Acked 4929 Read 3 New Done Done false
10832 3931 Acked 4930 Flush 0 New Done Done false
10833 3932 Acked 4931 Write 9 New Done Sent false
10834 3933 Acked 4932 Write 8 New Sent Sent false
10835 STATES DS:0 DS:1 DS:2 TOTAL
10836 New 302 0 0 302
10837 Sent 0 1 2 3
10838 Done 0 301 300 601
10839 Skipped 0 0 0 0
10840 Error 0 0 0 0
10841 Last Flush: 4630 4930 4930
10842 Downstairs last five completed: 4630 4629 4628 4627 4626 4625
10843 Upstairs last five completed: 3933 3932 3931 3930 3929 3928
10844 CLIENT: Up:0 ds:302 act:2
10845 [8810] got start action
10846 [8810] I am going to restart
10847 Starting downstairs at port 8810
10848 Make output file at "/tmp/dsc/downstairs-8810.txt"
10849 [8810][0] reports Starting
10850 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1189)
10851 [8810][0] reports Running
108522023-10-13T04:44:07.399ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 0
108532023-10-13T04:44:07.399ZINFOcrucible: [0] Proc runs for 127.0.0.1:8810 in state Offline
108542023-10-13T04:44:07.399ZINFOcrucible: [0] client re-new 303 jobs since flush 4630 = downstairs
108552023-10-13T04:44:07.400ZINFOcrucible: [0] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
108562023-10-13T04:44:07.400ZINFOcrucible: [0] downstairs client at 127.0.0.1:8810 has UUID 12345678-0000-0000-0000-000000008810
108572023-10-13T04:44:07.401ZINFOcrucible: [0] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }
108582023-10-13T04:44:07.401ZINFOcrucible: Returning client:0 UUID:12345678-0000-0000-0000-000000008810 matches
108592023-10-13T04:44:07.401ZINFOcrucible: [0] send last flush ID to this DS: 4630
108602023-10-13T04:44:07.401ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Offline Active Active
108612023-10-13T04:44:07.401ZINFOcrucible: [0] Replied this last flush ID: 4630
108622023-10-13T04:44:07.401ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Offline Active Active ds_transition to Replay
108632023-10-13T04:44:07.401ZINFOcrucible: [0] Transition from Offline to Replay
108642023-10-13T04:44:07.401ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
108652023-10-13T04:44:07.401ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Replay Active Active ds_transition to Active
108662023-10-13T04:44:07.401ZINFOcrucible: [0] Transition from Replay to Active
108672023-10-13T04:44:07.401ZINFOcrucible: [0] Starts cmd_loop
108682023-10-13T04:44:07.401ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:true
108692023-10-13T04:44:07.401ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
108702023-10-13T04:44:07.401ZINFOcrucible: Set check for repair
108712023-10-13T04:44:08.402ZWARNcrucible: [0] flow control end
108722023-10-13T04:44:08.403ZINFOcrucible: Checking if live repair is needed
108732023-10-13T04:44:08.403ZINFOcrucible: No Live Repair required at this time
10874 ----------------------------------------------------------------
10875 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
10876 Downstairs last five completed: 4933 4932 4931 4930 4929 4928
10877 Upstairs last five completed: 3934 3933 3932 3931 3930 3929
10878 CLIENT: Up:0 ds:0 act:3
10879 Replay: All jobs finished, all DS active.
10880 Main task has work to do, go find it
10881 got dsc Stop(0)
10882 stop 0
10883 Oct 13 04:44:09.262 INFO request completed, latency_us: 478, response_code: 204, uri: /stop/cid/0, method: POST, req_id: d8540ccc-c78c-44e3-85bf-b8dced4ecd52, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
10884 [8810] Got stop action so:false kr:false
10885 Oct 13 04:44:09.263 INFO request completed, latency_us: 170, response_code: 200, uri: /state/cid/0, method: GET, req_id: ca35dc49-5a56-400d-9d74-c4f82cc34055, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
108862023-10-13T04:44:09.263ZWARNcrucible: [0] downstairs disconnected
108872023-10-13T04:44:09.264ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
108882023-10-13T04:44:09.264ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8810 closed looper = 0
108892023-10-13T04:44:09.264ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:false
108902023-10-13T04:44:09.264ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Offline Active Active
108912023-10-13T04:44:09.264ZINFOcrucible: [0] 127.0.0.1:8810 task reports offline
108922023-10-13T04:44:09.264ZWARNcrucible: [0] pm_task rx.recv() is None
10893 [8810] Exited with: ExitStatus(unix_wait_status(9))
10894 [8810][0] reports Exit
10895 Oct 13 04:44:13.266 INFO request completed, latency_us: 279, response_code: 200, uri: /state/cid/0, method: GET, req_id: 1e678cbc-72fa-44f0-b52b-5611ddeaf242, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
10896 001/300 Write block 1288 len 24576 data: 6 7 7 6 6 5
10897 002/300 Read block 4741 len 4096
10898 003/300 Read block 4635 len 28672
10899 004/300 Read block 709 len 8192
10900 005/300 Read block 1348 len 28672
10901 006/300 Write block 255 len 16384 data: 8 6 6 5
10902 007/300 Flush
10903 008/300 Read block 4894 len 16384
10904 009/300 Read block 2816 len 24576
10905 010/300 Write block 4297 len 40960 data: 7 5 5 5 5 4 3 2 2 2
10906 011/300 Write block 202 len 32768 data: 2 3 3 3 3 4 5 6
10907 012/300 Read block 4132 len 36864
10908 013/300 Read block 2901 len 24576
10909 014/300 Read block 1341 len 16384
10910 015/300 Write block 1835 len 20480 data: 4 4 4 4 4
10911 016/300 Read block 4223 len 8192
10912 017/300 Write block 4887 len 4096 data: 6
10913 018/300 Write block 4133 len 4096 data: 4
10914 019/300 Flush
10915 020/300 Read block 3519 len 40960
10916 021/300 Read block 1072 len 24576
10917 022/300 Write block 3304 len 4096 data: 3
10918 023/300 Read block 3306 len 16384
10919 024/300 Write block 3571 len 20480 data: 3 3 3 4 5
10920 025/300 Read block 1284 len 12288
10921 026/300 Write block 1569 len 40960 data: 2 2 2 2 2 2 2 2 2 2
10922 027/300 Write block 741 len 28672 data: 3 3 2 2 2 2 3
10923 028/300 Write block 706 len 24576 data: 4 4 5 4 4 4
10924 029/300 Read block 3663 len 32768
10925 030/300 Read block 214 len 40960
10926 031/300 Read block 609 len 4096
10927 032/300 Read block 1154 len 40960
10928 033/300 Write block 1434 len 28672 data: 4 4 4 3 3 3 3
10929 034/300 Write block 341 len 4096 data: 2
10930 035/300 Write block 3792 len 24576 data: 4 4 3 2 2 2
10931 036/300 Read block 3445 len 24576
10932 037/300 Write block 1900 len 8192 data: 6 4
10933 038/300 Write block 2086 len 28672 data: 3 4 4 3 2 5 4
10934 039/300 Flush
10935 040/300 Read block 4698 len 24576
10936 041/300 Read block 2384 len 8192
10937 042/300 Read block 4901 len 24576
10938 043/300 Write block 3496 len 12288 data: 4 4 3
10939 044/300 Flush
10940 045/300 Read block 949 len 4096
10941 046/300 Read block 2259 len 36864
10942 047/300 Read block 1771 len 28672
10943 048/300 Flush
10944 049/300 Read block 3726 len 28672
10945 050/300 Read block 2116 len 16384
10946 051/300 Write block 4825 len 28672 data: 5 5 5 5 5 4 4
10947 052/300 Read block 4611 len 20480
10948 053/300 Write block 720 len 28672 data: 3 3 3 4 4 5 4
10949 054/300 Read block 1445 len 28672
10950 055/300 Flush
10951 056/300 Read block 4517 len 36864
10952 057/300 Write block 4875 len 8192 data: 3 2
10953 058/300 Write block 195 len 16384 data: 3 5 5 6
10954 059/300 Write block 3860 len 32768 data: 7 7 5 5 4 4 5 4
10955 060/300 Read block 2409 len 28672
10956 061/300 Read block 3067 len 32768
10957 062/300 Read block 645 len 8192
10958 063/300 Write block 401 len 20480 data: 5 5 5 4 4
10959 064/300 Read block 2461 len 32768
10960 065/300 Read block 2106 len 12288
10961 066/300 Write block 523 len 40960 data: 8 6 6 5 4 4 3 2 2 2
10962 067/300 Write block 3067 len 4096 data: 3
10963 068/300 Write block 1138 len 4096 data: 3
10964 069/300 Read block 998 len 36864
10965 070/300 Write block 4584 len 12288 data: 3 3 3
10966 071/300 Read block 3200 len 36864
10967 072/300 Flush
10968 073/300 Read block 1002 len 8192
10969 074/300 Read block 2100 len 12288
10970 075/300 Flush
10971 076/300 Write block 2443 len 16384 data: 2 2 2 3
10972 077/300 Read block 3934 len 24576
10973 078/300 Read block 230 len 28672
10974 079/300 Write block 3284 len 16384 data: 4 4 4 4
10975 080/300 Flush
10976 081/300 Write block 4050 len 20480 data: 2 2 3 3 3
10977 082/300 Read block 1794 len 28672
10978 083/300 Write block 4979 len 16384 data: 4 4 3 4
10979 084/300 Read block 3325 len 12288
10980 085/300 Write block 2888 len 4096 data: 3
10981 086/300 Read block 2420 len 28672
10982 087/300 Write block 2160 len 12288 data: 4 5 6
10983 088/300 Read block 3840 len 40960
10984 089/300 Read block 4435 len 32768
10985 090/300 Read block 3662 len 40960
10986 091/300 Flush
10987 092/300 Read block 3116 len 20480
10988 093/300 Write block 1349 len 28672 data: 2 2 3 3 3 2 2
10989 094/300 Write block 4902 len 32768 data: 2 3 3 4 5 5 4 6
10990 095/300 Write block 3196 len 40960 data: 3 3 3 5 5 5 4 6 5 5
10991 096/300 Flush
10992 097/300 Write block 1277 len 8192 data: 3 3
10993 098/300 Flush
10994 099/300 Flush
10995 100/300 Write block 2001 len 24576 data: 6 3 3 4 5 6
10996 101/300 Write block 4718 len 40960 data: 3 2 2 3 5 5 5 5 5 5
10997 102/300 Flush
10998 103/300 Write block 2343 len 8192 data: 2 2
10999 104/300 Write block 187 len 32768 data: 3 3 3 3 3 3 3 3
11000 105/300 Read block 3505 len 24576
11001 106/300 Read block 2140 len 8192
11002 107/300 Read block 288 len 24576
11003 108/300 Read block 2095 len 36864
11004 109/300 Flush
11005 110/300 Write block 266 len 20480 data: 2 2 2 2 2
11006 111/300 Read block 3370 len 16384
11007 112/300 Read block 202 len 20480
11008 113/300 Write block 2985 len 40960 data: 3 4 5 5 5 4 4 4 3 3
11009 114/300 Write block 4181 len 4096 data: 5
11010 115/300 Read block 65 len 32768
11011 116/300 Write block 2323 len 36864 data: 7 7 5 5 4 5 4 4 2
11012 117/300 Read block 4804 len 28672
11013 118/300 Flush
11014 119/300 Read block 2381 len 40960
11015 120/300 Read block 4505 len 24576
11016 121/300 Write block 2119 len 4096 data: 2
11017 122/300 Read block 1092 len 36864
11018 123/300 Write block 1629 len 4096 data: 3
11019 124/300 Read block 4846 len 24576
11020 125/300 Write block 3893 len 12288 data: 4 3 3
11021 126/300 Write block 3830 len 28672 data: 4 3 3 4 5 3 3
11022 127/300 Write block 4452 len 16384 data: 4 4 6 5
11023 128/300 Write block 4540 len 24576 data: 8 8 9 7 6 4
11024 129/300 Read block 997 len 24576
11025 130/300 Write block 2421 len 4096 data: 5
11026 131/300 Write block 1027 len 28672 data: 5 4 4 4 4 4 4
11027 132/300 Write block 1085 len 8192 data: 7 8
11028 133/300 Flush
11029 134/300 Read block 4925 len 16384
11030 135/300 Read block 3587 len 16384
11031 136/300 Write block 2161 len 20480 data: 6 7 6 6 4
11032 137/300 Write block 1678 len 16384 data: 2 3 3 4
11033 138/300 Read block 4587 len 32768
11034 139/300 Flush
11035 140/300 Read block 291 len 36864
11036 141/300 Read block 1026 len 4096
11037 142/300 Flush
11038 143/300 Read block 4061 len 36864
11039 144/300 Read block 1063 len 20480
11040 145/300 Read block 878 len 28672
11041 146/300 Read block 2920 len 40960
11042 147/300 Read block 275 len 20480
11043 148/300 Read block 3569 len 16384
11044 149/300 Read block 2448 len 40960
11045 150/300 Write block 4889 len 32768 data: 4 4 3 3 3 3 3 2
11046 151/300 Write block 2084 len 36864 data: 3 3 4 5 5 4 3 6 5
11047 152/300 Read block 934 len 28672
11048 153/300 Write block 2705 len 32768 data: 7 6 5 5 5 4 3 3
11049 154/300 Write block 4369 len 16384 data: 4 4 4 4
11050 155/300 Read block 4811 len 40960
11051 156/300 Write block 4344 len 32768 data: 5 6 7 6 7 8 7 6
11052 157/300 Write block 692 len 28672 data: 5 5 4 6 6 5 5
11053 158/300 Read block 4886 len 8192
11054 159/300 Write block 1035 len 16384 data: 4 5 4 4
11055 160/300 Read block 93 len 8192
11056 161/300 Write block 4107 len 16384 data: 4 3 3 3
11057 162/300 Write block 801 len 20480 data: 4 4 4 3 3
11058 163/300 Write block 3632 len 20480 data: 2 2 2 2 2
11059 164/300 Read block 2665 len 40960
11060 165/300 Read block 985 len 20480
11061 166/300 Read block 2171 len 24576
11062 167/300 Write block 2607 len 40960 data: 3 2 2 3 4 5 6 6 6 6
11063 168/300 Read block 4713 len 32768
11064 169/300 Read block 3270 len 8192
11065 170/300 Read block 2456 len 16384
11066 171/300 Read block 64 len 4096
11067 172/300 Read block 3704 len 28672
11068 173/300 Write block 409 len 12288 data: 4 3 3
11069 174/300 Flush
11070 175/300 Read block 969 len 24576
11071 176/300 Flush
11072 177/300 Read block 907 len 12288
11073 178/300 Read block 4888 len 16384
11074 179/300 Write block 521 len 8192 data: 7 8
11075 180/300 Write block 2271 len 4096 data: 2
11076 181/300 Read block 3011 len 32768
11077 182/300 Write block 2571 len 20480 data: 7 7 7 7 5
11078 183/300 Write block 2142 len 24576 data: 2 2 2 3 3 3
11079 184/300 Write block 1295 len 12288 data: 4 3 3
11080 185/300 Read block 498 len 24576
11081 186/300 Read block 4407 len 8192
11082 187/300 Read block 3181 len 4096
11083 188/300 Write block 3852 len 12288 data: 4 4 4
11084 189/300 Read block 276 len 4096
11085 190/300 Read block 2518 len 12288
11086 191/300 Flush
11087 192/300 Read block 4770 len 36864
11088 193/300 Read block 4683 len 12288
11089 194/300 Write block 3004 len 24576 data: 4 4 4 4 3 3
11090 195/300 Read block 2234 len 4096
11091 196/300 Flush
11092 197/300 Read block 1593 len 12288
11093 198/300 Read block 1435 len 20480
11094 199/300 Read block 3983 len 8192
11095 200/300 Read block 1673 len 4096
11096 201/300 Read block 3534 len 8192
11097 202/300 Read block 1067 len 4096
11098 203/300 Write block 296 len 20480 data: 6 7 6 6 4
11099 204/300 Write block 2850 len 20480 data: 4 4 5 5 4
11100 205/300 Write block 3428 len 24576 data: 6 7 7 6 5 5
11101 206/300 Read block 768 len 28672
11102 207/300 Flush
11103 208/300 Read block 1184 len 8192
11104 209/300 Read block 3041 len 40960
11105 210/300 Write block 4777 len 16384 data: 5 5 5 5
11106 211/300 Write block 4526 len 4096 data: 6
11107 212/300 Read block 1244 len 32768
11108 213/300 Read block 3943 len 36864
11109 214/300 Read block 3158 len 4096
11110 215/300 Read block 4413 len 8192
11111 216/300 Write block 2139 len 24576 data: 4 4 4 3 3 3
11112 217/300 Read block 2078 len 20480
11113 218/300 Write block 4869 len 36864 data: 2 2 2 2 2 2 4 3 2
11114 219/300 Flush
11115 220/300 Read block 4155 len 16384
11116 221/300 Write block 4437 len 24576 data: 4 4 4 4 5 6
11117 222/300 Read block 1516 len 28672
11118 223/300 Write block 1518 len 32768 data: 3 2 2 2 2 2 2 2
11119 224/300 Read block 4865 len 24576
11120 225/300 Read block 3925 len 24576
11121 226/300 Read block 4875 len 32768
11122 227/300 Read block 2824 len 12288
11123 228/300 Write block 4215 len 24576 data: 4 4 3 3 2 2
11124 229/300 Write block 2897 len 8192 data: 2 2
11125 230/300 Write block 1655 len 28672 data: 2 2 3 4 4 5 5
11126 231/300 Read block 2542 len 32768
11127 232/300 Write block 665 len 16384 data: 5 4 4 4
11128 233/300 Flush
11129 234/300 Read block 1224 len 16384
11130 235/300 Write block 4312 len 8192 data: 2 2
11131 236/300 Write block 1495 len 20480 data: 4 5 6 6 5
11132 237/300 Read block 2675 len 32768
11133 238/300 Read block 4005 len 16384
11134 239/300 Read block 4819 len 16384
11135 240/300 Read block 4196 len 8192
11136 241/300 Write block 3720 len 4096 data: 3
11137 242/300 Read block 4733 len 24576
11138 243/300 Write block 1093 len 8192 data: 5 6
11139 244/300 Write block 3163 len 20480 data: 5 3 3 3 3
11140 245/300 Read block 2182 len 24576
11141 246/300 Write block 3935 len 4096 data: 3
11142 247/300 Write block 1081 len 8192 data: 8 8
11143 248/300 Write block 1636 len 20480 data: 3 3 3 3 4
11144 249/300 Flush
11145 250/300 Write block 396 len 40960 data: 5 5 6 5 6 6 6 6 5 5
11146 251/300 Write block 4821 len 20480 data: 4 5 4 4 6
11147 252/300 Write block 4211 len 36864 data: 5 5 3 4 5 5 4 4 3
11148 253/300 Write block 353 len 24576 data: 4 4 4 4 3 2
11149 254/300 Read block 4781 len 16384
11150 255/300 Read block 1197 len 16384
11151 256/300 Write block 380 len 20480 data: 3 2 2 3 3
11152 257/300 Write block 4039 len 8192 data: 2 2
11153 258/300 Write block 1414 len 24576 data: 3 4 4 3 3 3
11154 259/300 Read block 4342 len 12288
11155 260/300 Read block 1683 len 36864
11156 261/300 Write block 3118 len 32768 data: 3 3 4 4 4 4 4 4
11157 262/300 Read block 79 len 28672
11158 263/300 Read block 3935 len 28672
11159 264/300 Write block 3031 len 8192 data: 3 3
11160 265/300 Read block 4575 len 32768
11161 266/300 Read block 3243 len 12288
11162 267/300 Write block 160 len 20480 data: 4 3 4 4 3
11163 268/300 Read block 289 len 36864
11164 269/300 Read block 1933 len 8192
11165 270/300 Write block 2455 len 36864 data: 2 2 2 2 3 2 2 2 2
11166 271/300 Write block 1940 len 16384 data: 4 5 4 3
11167 272/300 Read block 883 len 16384
11168 273/300 Read block 2864 len 20480
11169 274/300 Read block 3626 len 36864
11170 275/300 Read block 2794 len 4096
11171 276/300 Write block 3909 len 4096 data: 4
11172 277/300 Read block 795 len 32768
11173 278/300 Write block 1823 len 20480 data: 4 6 6 6 6
11174 279/300 Read block 1514 len 20480
11175 280/300 Read block 181 len 8192
11176 281/300 Write block 4452 len 12288 data: 5 5 7
11177 282/300 Read block 3421 len 40960
11178 283/300 Write block 3419 len 16384 data: 2 2 3 3
11179 284/300 Read block 3273 len 16384
11180 285/300 Write block 1547 len 4096 data: 2
11181 286/300 Read block 4080 len 28672
11182 287/300 Write block 1619 len 16384 data: 4 3 3 3
11183 288/300 Flush
11184 289/300 Read block 1207 len 36864
11185 290/300 Write block 1428 len 12288 data: 4 5 4
11186 291/300 Write block 1918 len 20480 data: 3 3 3 5 6
11187 292/300 Write block 4245 len 36864 data: 3 4 5 5 5 3 3 3 4
11188 293/300 Read block 1281 len 12288
11189 294/300 Write block 3815 len 28672 data: 4 4 3 3 3 3 3
11190 295/300 Read block 738 len 24576
11191 296/300 Write block 37 len 16384 data: 2 2 2 2
11192 297/300 Read block 865 len 8192
11193 298/300 Write block 1009 len 36864 data: 3 3 3 4 5 6 5 5 4
11194 299/300 Flush
11195 300/300 Write block 115 len 8192 data: 4 2
11196 Main task has work to do, go find it
11197 got dsc Start(0)
11198 start 0
11199 Oct 13 04:44:14.143 INFO request completed, latency_us: 287, response_code: 204, uri: /start/cid/0, method: POST, req_id: 250c058b-0a15-42ba-b6fa-e1367bf6ba7e, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
11200 Replay: started 0, returned:Ok(())
11201 ----------------------------------------------------------------
11202 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
11203 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
11204 3935 Acked 4934 Write 6 New Done Done false
11205 3936 Acked 4935 Read 1 New Done Done false
11206 3937 Acked 4936 Read 7 New Done Done false
11207 3938 Acked 4937 Read 2 New Done Done false
11208 3939 Acked 4938 Read 7 New Done Done false
11209 3940 Acked 4939 Write 4 New Done Done false
11210 3941 Acked 4940 Flush 0 New Done Done false
11211 3942 Acked 4941 Read 4 New Done Done false
11212 3943 Acked 4942 Read 6 New Done Done false
11213 3944 Acked 4943 Write 10 New Done Done false
11214 3945 Acked 4944 Write 8 New Done Done false
11215 3946 Acked 4945 Read 9 New Done Done false
11216 3947 Acked 4946 Read 6 New Done Done false
11217 3948 Acked 4947 Read 4 New Done Done false
11218 3949 Acked 4948 Write 5 New Done Done false
11219 3950 Acked 4949 Read 2 New Done Done false
11220 3951 Acked 4950 Write 1 New Done Done false
11221 3952 Acked 4951 Write 1 New Done Done false
11222 3953 Acked 4952 Flush 0 New Done Done false
11223 3954 Acked 4953 Read 10 New Done Done false
11224 3955 Acked 4954 Read 6 New Done Done false
11225 3956 Acked 4955 Write 1 New Done Done false
11226 3957 Acked 4956 Read 4 New Done Done false
11227 3958 Acked 4957 Write 5 New Done Done false
11228 3959 Acked 4958 Read 3 New Done Done false
11229 3960 Acked 4959 Write 10 New Done Done false
11230 3961 Acked 4960 Write 7 New Done Done false
11231 3962 Acked 4961 Write 6 New Done Done false
11232 3963 Acked 4962 Read 8 New Done Done false
11233 3964 Acked 4963 Read 10 New Done Done false
11234 3965 Acked 4964 Read 1 New Done Done false
11235 3966 Acked 4965 Read 10 New Done Done false
11236 3967 Acked 4966 Write 7 New Done Done false
11237 3968 Acked 4967 Write 1 New Done Done false
11238 3969 Acked 4968 Write 6 New Done Done false
11239 3970 Acked 4969 Read 6 New Done Done false
11240 3971 Acked 4970 Write 2 New Done Done false
11241 3972 Acked 4971 Write 7 New Done Done false
11242 3973 Acked 4972 Flush 0 New Done Done false
11243 3974 Acked 4973 Read 6 New Done Done false
11244 3975 Acked 4974 Read 2 New Done Done false
11245 3976 Acked 4975 Read 6 New Done Done false
11246 3977 Acked 4976 Write 3 New Done Done false
11247 3978 Acked 4977 Flush 0 New Done Done false
11248 3979 Acked 4978 Read 1 New Done Done false
11249 3980 Acked 4979 Read 9 New Done Done false
11250 3981 Acked 4980 Read 7 New Done Done false
11251 3982 Acked 4981 Flush 0 New Done Done false
11252 3983 Acked 4982 Read 7 New Done Done false
11253 3984 Acked 4983 Read 4 New Done Done false
11254 3985 Acked 4984 Write 7 New Done Done false
11255 3986 Acked 4985 Read 5 New Done Done false
11256 3987 Acked 4986 Write 7 New Done Done false
11257 3988 Acked 4987 Read 7 New Done Done false
11258 3989 Acked 4988 Flush 0 New Done Done false
11259 3990 Acked 4989 Read 9 New Done Done false
11260 3991 Acked 4990 Write 2 New Done Done false
11261 3992 Acked 4991 Write 4 New Done Done false
11262 3993 Acked 4992 Write 8 New Done Done false
11263 3994 Acked 4993 Read 7 New Done Done false
11264 3995 Acked 4994 Read 8 New Done Done false
11265 3996 Acked 4995 Read 2 New Done Done false
11266 3997 Acked 4996 Write 5 New Done Done false
11267 3998 Acked 4997 Read 8 New Done Done false
11268 3999 Acked 4998 Read 3 New Done Done false
11269 4000 Acked 4999 Write 10 New Done Done false
11270 4001 Acked 5000 Write 1 New Done Done false
11271 4002 Acked 5001 Write 1 New Done Done false
11272 4003 Acked 5002 Read 9 New Done Done false
11273 4004 Acked 5003 Write 3 New Done Done false
11274 4005 Acked 5004 Read 9 New Done Done false
11275 4006 Acked 5005 Flush 0 New Done Done false
11276 4007 Acked 5006 Read 2 New Done Done false
11277 4008 Acked 5007 Read 3 New Done Done false
11278 4009 Acked 5008 Flush 0 New Done Done false
11279 4010 Acked 5009 Write 4 New Done Done false
11280 4011 Acked 5010 Read 6 New Done Done false
11281 4012 Acked 5011 Read 7 New Done Done false
11282 4013 Acked 5012 Write 4 New Done Done false
11283 4014 Acked 5013 Flush 0 New Done Done false
11284 4015 Acked 5014 Write 5 New Done Done false
11285 4016 Acked 5015 Read 7 New Done Done false
11286 4017 Acked 5016 Write 4 New Done Done false
11287 4018 Acked 5017 Read 3 New Done Done false
11288 4019 Acked 5018 Write 1 New Done Done false
11289 4020 Acked 5019 Read 7 New Done Done false
11290 4021 Acked 5020 Write 3 New Done Done false
11291 4022 Acked 5021 Read 10 New Done Done false
11292 4023 Acked 5022 Read 8 New Done Done false
11293 4024 Acked 5023 Read 10 New Done Done false
11294 4025 Acked 5024 Flush 0 New Done Done false
11295 4026 Acked 5025 Read 5 New Done Done false
11296 4027 Acked 5026 Write 7 New Done Done false
11297 4028 Acked 5027 Write 8 New Done Done false
11298 4029 Acked 5028 Write 10 New Done Done false
11299 4030 Acked 5029 Flush 0 New Done Done false
11300 4031 Acked 5030 Write 2 New Done Done false
11301 4032 Acked 5031 Flush 0 New Done Done false
11302 4033 Acked 5032 Flush 0 New Done Done false
11303 4034 Acked 5033 Write 6 New Done Done false
11304 4035 Acked 5034 Write 10 New Done Done false
11305 4036 Acked 5035 Flush 0 New Done Done false
11306 4037 Acked 5036 Write 2 New Done Done false
11307 4038 Acked 5037 Write 8 New Done Done false
11308 4039 Acked 5038 Read 6 New Done Done false
11309 4040 Acked 5039 Read 2 New Done Done false
11310 4041 Acked 5040 Read 6 New Done Done false
11311 4042 Acked 5041 Read 9 New Done Done false
11312 4043 Acked 5042 Flush 0 New Done Done false
11313 4044 Acked 5043 Write 5 New Done Done false
11314 4045 Acked 5044 Read 4 New Done Done false
11315 4046 Acked 5045 Read 5 New Done Done false
11316 4047 Acked 5046 Write 10 New Done Done false
11317 4048 Acked 5047 Write 1 New Done Done false
11318 4049 Acked 5048 Read 8 New Done Done false
11319 4050 Acked 5049 Write 9 New Done Done false
11320 4051 Acked 5050 Read 7 New Done Done false
11321 4052 Acked 5051 Flush 0 New Done Done false
11322 4053 Acked 5052 Read 10 New Done Done false
11323 4054 Acked 5053 Read 6 New Done Done false
11324 4055 Acked 5054 Write 1 New Done Done false
11325 4056 Acked 5055 Read 9 New Done Done false
11326 4057 Acked 5056 Write 1 New Done Done false
11327 4058 Acked 5057 Read 6 New Done Done false
11328 4059 Acked 5058 Write 3 New Done Done false
11329 4060 Acked 5059 Write 7 New Done Done false
11330 4061 Acked 5060 Write 4 New Done Done false
11331 4062 Acked 5061 Write 6 New Done Done false
11332 4063 Acked 5062 Read 6 New Done Done false
11333 4064 Acked 5063 Write 1 New Done Done false
11334 4065 Acked 5064 Write 7 New Done Done false
11335 4066 Acked 5065 Write 2 New Done Done false
11336 4067 Acked 5066 Flush 0 New Done Done false
11337 4068 Acked 5067 Read 4 New Done Done false
11338 4069 Acked 5068 Read 4 New Done Done false
11339 4070 Acked 5069 Write 5 New Done Done false
11340 4071 Acked 5070 Write 4 New Done Done false
11341 4072 Acked 5071 Read 8 New Done Done false
11342 4073 Acked 5072 Flush 0 New Done Done false
11343 4074 Acked 5073 Read 9 New Done Done false
11344 4075 Acked 5074 Read 1 New Done Done false
11345 4076 Acked 5075 Flush 0 New Done Done false
11346 4077 Acked 5076 Read 9 New Done Done false
11347 4078 Acked 5077 Read 5 New Done Done false
11348 4079 Acked 5078 Read 7 New Done Done false
11349 4080 Acked 5079 Read 10 New Done Done false
11350 4081 Acked 5080 Read 5 New Done Done false
11351 4082 Acked 5081 Read 4 New Done Done false
11352 4083 Acked 5082 Read 10 New Done Done false
11353 4084 Acked 5083 Write 8 New Done Done false
11354 4085 Acked 5084 Write 9 New Done Done false
11355 4086 Acked 5085 Read 7 New Done Done false
11356 4087 Acked 5086 Write 8 New Done Done false
11357 4088 Acked 5087 Write 4 New Done Done false
11358 4089 Acked 5088 Read 10 New Done Done false
11359 4090 Acked 5089 Write 8 New Done Done false
11360 4091 Acked 5090 Write 7 New Done Done false
11361 4092 Acked 5091 Read 2 New Done Done false
11362 4093 Acked 5092 Write 4 New Done Done false
11363 4094 Acked 5093 Read 2 New Done Done false
11364 4095 Acked 5094 Write 4 New Done Done false
11365 4096 Acked 5095 Write 5 New Done Done false
11366 4097 Acked 5096 Write 5 New Done Done false
11367 4098 Acked 5097 Read 10 New Done Done false
11368 4099 Acked 5098 Read 5 New Done Done false
11369 4100 Acked 5099 Read 6 New Done Done false
11370 4101 Acked 5100 Write 10 New Done Done false
11371 4102 Acked 5101 Read 8 New Done Done false
11372 4103 Acked 5102 Read 2 New Done Done false
11373 4104 Acked 5103 Read 4 New Done Done false
11374 4105 Acked 5104 Read 1 New Done Done false
11375 4106 Acked 5105 Read 7 New Done Done false
11376 4107 Acked 5106 Write 3 New Done Done false
11377 4108 Acked 5107 Flush 0 New Done Done false
11378 4109 Acked 5108 Read 6 New Done Done false
11379 4110 Acked 5109 Flush 0 New Done Done false
11380 4111 Acked 5110 Read 3 New Done Done false
11381 4112 Acked 5111 Read 4 New Done Done false
11382 4113 Acked 5112 Write 2 New Done Done false
11383 4114 Acked 5113 Write 1 New Done Done false
11384 4115 Acked 5114 Read 8 New Done Done false
11385 4116 Acked 5115 Write 5 New Done Done false
11386 4117 Acked 5116 Write 6 New Done Done false
11387 4118 Acked 5117 Write 3 New Done Done false
11388 4119 Acked 5118 Read 6 New Done Done false
11389 4120 Acked 5119 Read 2 New Done Done false
11390 4121 Acked 5120 Read 1 New Done Done false
11391 4122 Acked 5121 Write 3 New Done Done false
11392 4123 Acked 5122 Read 1 New Done Done false
11393 4124 Acked 5123 Read 3 New Done Done false
11394 4125 Acked 5124 Flush 0 New Done Done false
11395 4126 Acked 5125 Read 9 New Done Done false
11396 4127 Acked 5126 Read 3 New Done Done false
11397 4128 Acked 5127 Write 6 New Done Done false
11398 4129 Acked 5128 Read 1 New Done Done false
11399 4130 Acked 5129 Flush 0 New Done Done false
11400 4131 Acked 5130 Read 3 New Done Done false
11401 4132 Acked 5131 Read 5 New Done Done false
11402 4133 Acked 5132 Read 2 New Done Done false
11403 4134 Acked 5133 Read 1 New Done Done false
11404 4135 Acked 5134 Read 2 New Done Done false
11405 4136 Acked 5135 Read 1 New Done Done false
11406 4137 Acked 5136 Write 5 New Done Done false
11407 4138 Acked 5137 Write 5 New Done Done false
11408 4139 Acked 5138 Write 6 New Done Done false
11409 4140 Acked 5139 Read 7 New Done Done false
11410 4141 Acked 5140 Flush 0 New Done Done false
11411 4142 Acked 5141 Read 2 New Done Done false
11412 4143 Acked 5142 Read 10 New Done Done false
11413 4144 Acked 5143 Write 4 New Done Done false
11414 4145 Acked 5144 Write 1 New Done Done false
11415 4146 Acked 5145 Read 8 New Done Done false
11416 4147 Acked 5146 Read 9 New Done Done false
11417 4148 Acked 5147 Read 1 New Done Done false
11418 4149 Acked 5148 Read 2 New Done Done false
11419 4150 Acked 5149 Write 6 New Done Done false
11420 4151 Acked 5150 Read 5 New Done Done false
11421 4152 Acked 5151 Write 9 New Done Done false
11422 4153 Acked 5152 Flush 0 New Done Done false
11423 4154 Acked 5153 Read 4 New Done Done false
11424 4155 Acked 5154 Write 6 New Done Done false
11425 4156 Acked 5155 Read 7 New Done Done false
11426 4157 Acked 5156 Write 8 New Done Done false
11427 4158 Acked 5157 Read 6 New Done Done false
11428 4159 Acked 5158 Read 6 New Done Done false
11429 4160 Acked 5159 Read 8 New Done Done false
11430 4161 Acked 5160 Read 3 New Done Done false
11431 4162 Acked 5161 Write 6 New Done Done false
11432 4163 Acked 5162 Write 2 New Done Done false
11433 4164 Acked 5163 Write 7 New Done Done false
11434 4165 Acked 5164 Read 8 New Done Done false
11435 4166 Acked 5165 Write 4 New Done Done false
11436 4167 Acked 5166 Flush 0 New Done Done false
11437 4168 Acked 5167 Read 4 New Done Done false
11438 4169 Acked 5168 Write 2 New Done Done false
11439 4170 Acked 5169 Write 5 New Done Done false
11440 4171 Acked 5170 Read 8 New Done Done false
11441 4172 Acked 5171 Read 4 New Done Done false
11442 4173 Acked 5172 Read 4 New Done Done false
11443 4174 Acked 5173 Read 2 New Done Done false
11444 4175 Acked 5174 Write 1 New Done Done false
11445 4176 Acked 5175 Read 6 New Done Done false
11446 4177 Acked 5176 Write 2 New Done Done false
11447 4178 Acked 5177 Write 5 New Done Done false
11448 4179 Acked 5178 Read 6 New Done Done false
11449 4180 Acked 5179 Write 1 New Done Done false
11450 4181 Acked 5180 Write 2 New Done Done false
11451 4182 Acked 5181 Write 5 New Done Done false
11452 4183 Acked 5182 Flush 0 New Done Done false
11453 4184 Acked 5183 Write 10 New Done Done false
11454 4185 Acked 5184 Write 5 New Done Done false
11455 4186 Acked 5185 Write 9 New Done Done false
11456 4187 Acked 5186 Write 6 New Done Done false
11457 4188 Acked 5187 Read 4 New Done Done false
11458 4189 Acked 5188 Read 4 New Done Done false
11459 4190 Acked 5189 Write 5 New Done Done false
11460 4191 Acked 5190 Write 2 New Done Done false
11461 4192 Acked 5191 Write 6 New Done Done false
11462 4193 Acked 5192 Read 3 New Done Done false
11463 4194 Acked 5193 Read 9 New Done Done false
11464 4195 Acked 5194 Write 8 New Done Done false
11465 4196 Acked 5195 Read 7 New Done Done false
11466 4197 Acked 5196 Read 7 New Done Done false
11467 4198 Acked 5197 Write 2 New Done Done false
11468 4199 Acked 5198 Read 8 New Done Done false
11469 4200 Acked 5199 Read 3 New Done Done false
11470 4201 Acked 5200 Write 5 New Done Done false
11471 4202 Acked 5201 Read 9 New Done Done false
11472 4203 Acked 5202 Read 2 New Done Done false
11473 4204 Acked 5203 Write 9 New Done Done false
11474 4205 Acked 5204 Write 4 New Done Done false
11475 4206 Acked 5205 Read 4 New Done Done false
11476 4207 Acked 5206 Read 5 New Done Done false
11477 4208 Acked 5207 Read 9 New Done Done false
11478 4209 Acked 5208 Read 1 New Done Done false
11479 4210 Acked 5209 Write 1 New Done Done false
11480 4211 Acked 5210 Read 8 New Done Done false
11481 4212 Acked 5211 Flush 0 New Done Done false
11482 4213 Acked 5212 Write 5 New Done Done false
11483 4214 Acked 5213 Read 5 New Done Done false
11484 4215 Acked 5214 Read 2 New Done Done false
11485 4216 Acked 5215 Write 3 New Done Done false
11486 4217 Acked 5216 Read 10 New Done Done false
11487 4218 Acked 5217 Write 4 New Done Done false
11488 4219 Acked 5218 Read 4 New Done Done false
11489 4220 Acked 5219 Write 1 New Done Done false
11490 4221 Acked 5220 Read 7 New Done Done false
11491 4222 Acked 5221 Write 4 New Done Done false
11492 4223 Acked 5222 Flush 0 New Done Done false
11493 4224 Acked 5223 Read 9 New Done Done false
11494 4225 Acked 5224 Write 3 New Done Done false
11495 4226 Acked 5225 Write 5 New Done Done false
11496 4227 Acked 5226 Write 9 New Done Done false
11497 4228 Acked 5227 Read 3 New Done Done false
11498 4229 Acked 5228 Write 7 New Done Done false
11499 4230 Acked 5229 Read 6 New Done Done false
11500 4231 Acked 5230 Write 4 New Done Done false
11501 4232 Acked 5231 Read 2 New Done Done false
11502 4233 Acked 5232 Write 9 New Done Done false
11503 4234 Acked 5233 Flush 0 New Done Done false
11504 4235 Acked 5234 Write 2 New Sent Sent false
11505 STATES DS:0 DS:1 DS:2 TOTAL
11506 New 301 0 0 301
11507 Sent 0 1 1 2
11508 Done 0 300 300 600
11509 Skipped 0 0 0 0
11510 Error 0 0 0 0
11511 Last Flush: 4933 5233 5233
11512 Downstairs last five completed: 4933 4932 4931 4930 4929 4928
11513 Upstairs last five completed: 4235 4234 4233 4232 4231 4230
11514 CLIENT: Up:0 ds:301 act:2
115152023-10-13T04:44:15.271ZINFOcrucible: [0] connecting to 127.0.0.1:8810 looper = 0
11516 ----------------------------------------------------------------
11517 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
11518 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
11519 3935 Acked 4934 Write 6 New Done Done false
11520 3936 Acked 4935 Read 1 New Done Done false
11521 3937 Acked 4936 Read 7 New Done Done false
11522 3938 Acked 4937 Read 2 New Done Done false
11523 3939 Acked 4938 Read 7 New Done Done false
11524 3940 Acked 4939 Write 4 New Done Done false
11525 3941 Acked 4940 Flush 0 New Done Done false
11526 3942 Acked 4941 Read 4 New Done Done false
11527 3943 Acked 4942 Read 6 New Done Done false
11528 3944 Acked 4943 Write 10 New Done Done false
11529 3945 Acked 4944 Write 8 New Done Done false
11530 3946 Acked 4945 Read 9 New Done Done false
11531 3947 Acked 4946 Read 6 New Done Done false
11532 3948 Acked 4947 Read 4 New Done Done false
11533 3949 Acked 4948 Write 5 New Done Done false
11534 3950 Acked 4949 Read 2 New Done Done false
11535 3951 Acked 4950 Write 1 New Done Done false
11536 3952 Acked 4951 Write 1 New Done Done false
11537 3953 Acked 4952 Flush 0 New Done Done false
11538 3954 Acked 4953 Read 10 New Done Done false
11539 3955 Acked 4954 Read 6 New Done Done false
11540 3956 Acked 4955 Write 1 New Done Done false
11541 3957 Acked 4956 Read 4 New Done Done false
11542 3958 Acked 4957 Write 5 New Done Done false
11543 3959 Acked 4958 Read 3 New Done Done false
11544 3960 Acked 4959 Write 10 New Done Done false
11545 3961 Acked 4960 Write 7 New Done Done false
11546 3962 Acked 4961 Write 6 New Done Done false
11547 3963 Acked 4962 Read 8 New Done Done false
11548 3964 Acked 4963 Read 10 New Done Done false
11549 3965 Acked 4964 Read 1 New Done Done false
11550 3966 Acked 4965 Read 10 New Done Done false
11551 3967 Acked 4966 Write 7 New Done Done false
11552 3968 Acked 4967 Write 1 New Done Done false
11553 3969 Acked 4968 Write 6 New Done Done false
11554 3970 Acked 4969 Read 6 New Done Done false
11555 3971 Acked 4970 Write 2 New Done Done false
11556 3972 Acked 4971 Write 7 New Done Done false
11557 3973 Acked 4972 Flush 0 New Done Done false
11558 3974 Acked 4973 Read 6 New Done Done false
11559 3975 Acked 4974 Read 2 New Done Done false
11560 3976 Acked 4975 Read 6 New Done Done false
11561 3977 Acked 4976 Write 3 New Done Done false
11562 3978 Acked 4977 Flush 0 New Done Done false
11563 3979 Acked 4978 Read 1 New Done Done false
11564 3980 Acked 4979 Read 9 New Done Done false
11565 3981 Acked 4980 Read 7 New Done Done false
11566 3982 Acked 4981 Flush 0 New Done Done false
11567 3983 Acked 4982 Read 7 New Done Done false
11568 3984 Acked 4983 Read 4 New Done Done false
11569 3985 Acked 4984 Write 7 New Done Done false
11570 3986 Acked 4985 Read 5 New Done Done false
11571 3987 Acked 4986 Write 7 New Done Done false
11572 3988 Acked 4987 Read 7 New Done Done false
11573 3989 Acked 4988 Flush 0 New Done Done false
11574 3990 Acked 4989 Read 9 New Done Done false
11575 3991 Acked 4990 Write 2 New Done Done false
11576 3992 Acked 4991 Write 4 New Done Done false
11577 3993 Acked 4992 Write 8 New Done Done false
11578 3994 Acked 4993 Read 7 New Done Done false
11579 3995 Acked 4994 Read 8 New Done Done false
11580 3996 Acked 4995 Read 2 New Done Done false
11581 3997 Acked 4996 Write 5 New Done Done false
11582 3998 Acked 4997 Read 8 New Done Done false
11583 3999 Acked 4998 Read 3 New Done Done false
11584 4000 Acked 4999 Write 10 New Done Done false
11585 4001 Acked 5000 Write 1 New Done Done false
11586 4002 Acked 5001 Write 1 New Done Done false
11587 4003 Acked 5002 Read 9 New Done Done false
11588 4004 Acked 5003 Write 3 New Done Done false
11589 4005 Acked 5004 Read 9 New Done Done false
11590 4006 Acked 5005 Flush 0 New Done Done false
11591 4007 Acked 5006 Read 2 New Done Done false
11592 4008 Acked 5007 Read 3 New Done Done false
11593 4009 Acked 5008 Flush 0 New Done Done false
11594 4010 Acked 5009 Write 4 New Done Done false
11595 4011 Acked 5010 Read 6 New Done Done false
11596 4012 Acked 5011 Read 7 New Done Done false
11597 4013 Acked 5012 Write 4 New Done Done false
11598 4014 Acked 5013 Flush 0 New Done Done false
11599 4015 Acked 5014 Write 5 New Done Done false
11600 4016 Acked 5015 Read 7 New Done Done false
11601 4017 Acked 5016 Write 4 New Done Done false
11602 4018 Acked 5017 Read 3 New Done Done false
11603 4019 Acked 5018 Write 1 New Done Done false
11604 4020 Acked 5019 Read 7 New Done Done false
11605 4021 Acked 5020 Write 3 New Done Done false
11606 4022 Acked 5021 Read 10 New Done Done false
11607 4023 Acked 5022 Read 8 New Done Done false
11608 4024 Acked 5023 Read 10 New Done Done false
11609 4025 Acked 5024 Flush 0 New Done Done false
11610 4026 Acked 5025 Read 5 New Done Done false
11611 4027 Acked 5026 Write 7 New Done Done false
11612 4028 Acked 5027 Write 8 New Done Done false
11613 4029 Acked 5028 Write 10 New Done Done false
11614 4030 Acked 5029 Flush 0 New Done Done false
11615 4031 Acked 5030 Write 2 New Done Done false
11616 4032 Acked 5031 Flush 0 New Done Done false
11617 4033 Acked 5032 Flush 0 New Done Done false
11618 4034 Acked 5033 Write 6 New Done Done false
11619 4035 Acked 5034 Write 10 New Done Done false
11620 4036 Acked 5035 Flush 0 New Done Done false
11621 4037 Acked 5036 Write 2 New Done Done false
11622 4038 Acked 5037 Write 8 New Done Done false
11623 4039 Acked 5038 Read 6 New Done Done false
11624 4040 Acked 5039 Read 2 New Done Done false
11625 4041 Acked 5040 Read 6 New Done Done false
11626 4042 Acked 5041 Read 9 New Done Done false
11627 4043 Acked 5042 Flush 0 New Done Done false
11628 4044 Acked 5043 Write 5 New Done Done false
11629 4045 Acked 5044 Read 4 New Done Done false
11630 4046 Acked 5045 Read 5 New Done Done false
11631 4047 Acked 5046 Write 10 New Done Done false
11632 4048 Acked 5047 Write 1 New Done Done false
11633 4049 Acked 5048 Read 8 New Done Done false
11634 4050 Acked 5049 Write 9 New Done Done false
11635 4051 Acked 5050 Read 7 New Done Done false
11636 4052 Acked 5051 Flush 0 New Done Done false
11637 4053 Acked 5052 Read 10 New Done Done false
11638 4054 Acked 5053 Read 6 New Done Done false
11639 4055 Acked 5054 Write 1 New Done Done false
11640 4056 Acked 5055 Read 9 New Done Done false
11641 4057 Acked 5056 Write 1 New Done Done false
11642 4058 Acked 5057 Read 6 New Done Done false
11643 4059 Acked 5058 Write 3 New Done Done false
11644 4060 Acked 5059 Write 7 New Done Done false
11645 4061 Acked 5060 Write 4 New Done Done false
11646 4062 Acked 5061 Write 6 New Done Done false
11647 4063 Acked 5062 Read 6 New Done Done false
11648 4064 Acked 5063 Write 1 New Done Done false
11649 4065 Acked 5064 Write 7 New Done Done false
11650 4066 Acked 5065 Write 2 New Done Done false
11651 4067 Acked 5066 Flush 0 New Done Done false
11652 4068 Acked 5067 Read 4 New Done Done false
11653 4069 Acked 5068 Read 4 New Done Done false
11654 4070 Acked 5069 Write 5 New Done Done false
11655 4071 Acked 5070 Write 4 New Done Done false
11656 4072 Acked 5071 Read 8 New Done Done false
11657 4073 Acked 5072 Flush 0 New Done Done false
11658 4074 Acked 5073 Read 9 New Done Done false
11659 4075 Acked 5074 Read 1 New Done Done false
11660 4076 Acked 5075 Flush 0 New Done Done false
11661 4077 Acked 5076 Read 9 New Done Done false
11662 4078 Acked 5077 Read 5 New Done Done false
11663 4079 Acked 5078 Read 7 New Done Done false
11664 4080 Acked 5079 Read 10 New Done Done false
11665 4081 Acked 5080 Read 5 New Done Done false
11666 4082 Acked 5081 Read 4 New Done Done false
11667 4083 Acked 5082 Read 10 New Done Done false
11668 4084 Acked 5083 Write 8 New Done Done false
11669 4085 Acked 5084 Write 9 New Done Done false
11670 4086 Acked 5085 Read 7 New Done Done false
11671 4087 Acked 5086 Write 8 New Done Done false
11672 4088 Acked 5087 Write 4 New Done Done false
11673 4089 Acked 5088 Read 10 New Done Done false
11674 4090 Acked 5089 Write 8 New Done Done false
11675 4091 Acked 5090 Write 7 New Done Done false
11676 4092 Acked 5091 Read 2 New Done Done false
11677 4093 Acked 5092 Write 4 New Done Done false
11678 4094 Acked 5093 Read 2 New Done Done false
11679 4095 Acked 5094 Write 4 New Done Done false
11680 4096 Acked 5095 Write 5 New Done Done false
11681 4097 Acked 5096 Write 5 New Done Done false
11682 4098 Acked 5097 Read 10 New Done Done false
11683 4099 Acked 5098 Read 5 New Done Done false
11684 4100 Acked 5099 Read 6 New Done Done false
11685 4101 Acked 5100 Write 10 New Done Done false
11686 4102 Acked 5101 Read 8 New Done Done false
11687 4103 Acked 5102 Read 2 New Done Done false
11688 4104 Acked 5103 Read 4 New Done Done false
11689 4105 Acked 5104 Read 1 New Done Done false
11690 4106 Acked 5105 Read 7 New Done Done false
11691 4107 Acked 5106 Write 3 New Done Done false
11692 4108 Acked 5107 Flush 0 New Done Done false
11693 4109 Acked 5108 Read 6 New Done Done false
11694 4110 Acked 5109 Flush 0 New Done Done false
11695 4111 Acked 5110 Read 3 New Done Done false
11696 4112 Acked 5111 Read 4 New Done Done false
11697 4113 Acked 5112 Write 2 New Done Done false
11698 4114 Acked 5113 Write 1 New Done Done false
11699 4115 Acked 5114 Read 8 New Done Done false
11700 4116 Acked 5115 Write 5 New Done Done false
11701 4117 Acked 5116 Write 6 New Done Done false
11702 4118 Acked 5117 Write 3 New Done Done false
11703 4119 Acked 5118 Read 6 New Done Done false
11704 4120 Acked 5119 Read 2 New Done Done false
11705 4121 Acked 5120 Read 1 New Done Done false
11706 4122 Acked 5121 Write 3 New Done Done false
11707 4123 Acked 5122 Read 1 New Done Done false
11708 4124 Acked 5123 Read 3 New Done Done false
11709 4125 Acked 5124 Flush 0 New Done Done false
11710 4126 Acked 5125 Read 9 New Done Done false
11711 4127 Acked 5126 Read 3 New Done Done false
11712 4128 Acked 5127 Write 6 New Done Done false
11713 4129 Acked 5128 Read 1 New Done Done false
11714 4130 Acked 5129 Flush 0 New Done Done false
11715 4131 Acked 5130 Read 3 New Done Done false
11716 4132 Acked 5131 Read 5 New Done Done false
11717 4133 Acked 5132 Read 2 New Done Done false
11718 4134 Acked 5133 Read 1 New Done Done false
11719 4135 Acked 5134 Read 2 New Done Done false
11720 4136 Acked 5135 Read 1 New Done Done false
11721 4137 Acked 5136 Write 5 New Done Done false
11722 4138 Acked 5137 Write 5 New Done Done false
11723 4139 Acked 5138 Write 6 New Done Done false
11724 4140 Acked 5139 Read 7 New Done Done false
11725 4141 Acked 5140 Flush 0 New Done Done false
11726 4142 Acked 5141 Read 2 New Done Done false
11727 4143 Acked 5142 Read 10 New Done Done false
11728 4144 Acked 5143 Write 4 New Done Done false
11729 4145 Acked 5144 Write 1 New Done Done false
11730 4146 Acked 5145 Read 8 New Done Done false
11731 4147 Acked 5146 Read 9 New Done Done false
11732 4148 Acked 5147 Read 1 New Done Done false
11733 4149 Acked 5148 Read 2 New Done Done false
11734 4150 Acked 5149 Write 6 New Done Done false
11735 4151 Acked 5150 Read 5 New Done Done false
11736 4152 Acked 5151 Write 9 New Done Done false
11737 4153 Acked 5152 Flush 0 New Done Done false
11738 4154 Acked 5153 Read 4 New Done Done false
11739 4155 Acked 5154 Write 6 New Done Done false
11740 4156 Acked 5155 Read 7 New Done Done false
11741 4157 Acked 5156 Write 8 New Done Done false
11742 4158 Acked 5157 Read 6 New Done Done false
11743 4159 Acked 5158 Read 6 New Done Done false
11744 4160 Acked 5159 Read 8 New Done Done false
11745 4161 Acked 5160 Read 3 New Done Done false
11746 4162 Acked 5161 Write 6 New Done Done false
11747 4163 Acked 5162 Write 2 New Done Done false
11748 4164 Acked 5163 Write 7 New Done Done false
11749 4165 Acked 5164 Read 8 New Done Done false
11750 4166 Acked 5165 Write 4 New Done Done false
11751 4167 Acked 5166 Flush 0 New Done Done false
11752 4168 Acked 5167 Read 4 New Done Done false
11753 4169 Acked 5168 Write 2 New Done Done false
11754 4170 Acked 5169 Write 5 New Done Done false
11755 4171 Acked 5170 Read 8 New Done Done false
11756 4172 Acked 5171 Read 4 New Done Done false
11757 4173 Acked 5172 Read 4 New Done Done false
11758 4174 Acked 5173 Read 2 New Done Done false
11759 4175 Acked 5174 Write 1 New Done Done false
11760 4176 Acked 5175 Read 6 New Done Done false
11761 4177 Acked 5176 Write 2 New Done Done false
11762 4178 Acked 5177 Write 5 New Done Done false
11763 4179 Acked 5178 Read 6 New Done Done false
11764 4180 Acked 5179 Write 1 New Done Done false
11765 4181 Acked 5180 Write 2 New Done Done false
11766 4182 Acked 5181 Write 5 New Done Done false
11767 4183 Acked 5182 Flush 0 New Done Done false
11768 4184 Acked 5183 Write 10 New Done Done false
11769 4185 Acked 5184 Write 5 New Done Done false
11770 4186 Acked 5185 Write 9 New Done Done false
11771 4187 Acked 5186 Write 6 New Done Done false
11772 4188 Acked 5187 Read 4 New Done Done false
11773 4189 Acked 5188 Read 4 New Done Done false
11774 4190 Acked 5189 Write 5 New Done Done false
11775 4191 Acked 5190 Write 2 New Done Done false
11776 4192 Acked 5191 Write 6 New Done Done false
11777 4193 Acked 5192 Read 3 New Done Done false
11778 4194 Acked 5193 Read 9 New Done Done false
11779 4195 Acked 5194 Write 8 New Done Done false
11780 4196 Acked 5195 Read 7 New Done Done false
11781 4197 Acked 5196 Read 7 New Done Done false
11782 4198 Acked 5197 Write 2 New Done Done false
11783 4199 Acked 5198 Read 8 New Done Done false
11784 4200 Acked 5199 Read 3 New Done Done false
11785 4201 Acked 5200 Write 5 New Done Done false
11786 4202 Acked 5201 Read 9 New Done Done false
11787 4203 Acked 5202 Read 2 New Done Done false
11788 4204 Acked 5203 Write 9 New Done Done false
11789 4205 Acked 5204 Write 4 New Done Done false
11790 4206 Acked 5205 Read 4 New Done Done false
11791 4207 Acked 5206 Read 5 New Done Done false
11792 4208 Acked 5207 Read 9 New Done Done false
11793 4209 Acked 5208 Read 1 New Done Done false
11794 4210 Acked 5209 Write 1 New Done Done false
11795 4211 Acked 5210 Read 8 New Done Done false
11796 4212 Acked 5211 Flush 0 New Done Done false
11797 4213 Acked 5212 Write 5 New Done Done false
11798 4214 Acked 5213 Read 5 New Done Done false
11799 4215 Acked 5214 Read 2 New Done Done false
11800 4216 Acked 5215 Write 3 New Done Done false
11801 4217 Acked 5216 Read 10 New Done Done false
11802 4218 Acked 5217 Write 4 New Done Done false
11803 4219 Acked 5218 Read 4 New Done Done false
11804 4220 Acked 5219 Write 1 New Done Done false
11805 4221 Acked 5220 Read 7 New Done Done false
11806 4222 Acked 5221 Write 4 New Done Done false
11807 4223 Acked 5222 Flush 0 New Done Done false
11808 4224 Acked 5223 Read 9 New Done Done false
11809 4225 Acked 5224 Write 3 New Done Done false
11810 4226 Acked 5225 Write 5 New Done Done false
11811 4227 Acked 5226 Write 9 New Done Done false
11812 4228 Acked 5227 Read 3 New Done Done false
11813 4229 Acked 5228 Write 7 New Done Done false
11814 4230 Acked 5229 Read 6 New Done Done false
11815 4231 Acked 5230 Write 4 New Done Done false
11816 4232 Acked 5231 Read 2 New Done Done false
11817 4233 Acked 5232 Write 9 New Done Done false
11818 4234 Acked 5233 Flush 0 New Done Done false
11819 4235 Acked 5234 Write 2 New Done Done false
11820 4236 Acked 5235 Flush 0 New Done Done false
11821 STATES DS:0 DS:1 DS:2 TOTAL
11822 New 302 0 0 302
11823 Sent 0 0 0 0
11824 Done 0 302 302 604
11825 Skipped 0 0 0 0
11826 Error 0 0 0 0
11827 Last Flush: 4933 5235 5235
11828 Downstairs last five completed: 4933 4932 4931 4930 4929 4928
11829 Upstairs last five completed: 4236 4235 4234 4233 4232 4231
11830 CLIENT: Up:0 ds:302 act:2
11831 [8810] got start action
11832 [8810] I am going to restart
11833 Starting downstairs at port 8810
11834 Make output file at "/tmp/dsc/downstairs-8810.txt"
11835 [8810][0] reports Starting
11836 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1190)
11837 [8810][0] reports Running
11838 ----------------------------------------------------------------
11839 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
11840 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
11841 3935 Acked 4934 Write 6 New Done Done false
11842 3936 Acked 4935 Read 1 New Done Done false
11843 3937 Acked 4936 Read 7 New Done Done false
11844 3938 Acked 4937 Read 2 New Done Done false
11845 3939 Acked 4938 Read 7 New Done Done false
11846 3940 Acked 4939 Write 4 New Done Done false
11847 3941 Acked 4940 Flush 0 New Done Done false
11848 3942 Acked 4941 Read 4 New Done Done false
11849 3943 Acked 4942 Read 6 New Done Done false
11850 3944 Acked 4943 Write 10 New Done Done false
11851 3945 Acked 4944 Write 8 New Done Done false
11852 3946 Acked 4945 Read 9 New Done Done false
11853 3947 Acked 4946 Read 6 New Done Done false
11854 3948 Acked 4947 Read 4 New Done Done false
11855 3949 Acked 4948 Write 5 New Done Done false
11856 3950 Acked 4949 Read 2 New Done Done false
11857 3951 Acked 4950 Write 1 New Done Done false
11858 3952 Acked 4951 Write 1 New Done Done false
11859 3953 Acked 4952 Flush 0 New Done Done false
11860 3954 Acked 4953 Read 10 New Done Done false
11861 3955 Acked 4954 Read 6 New Done Done false
11862 3956 Acked 4955 Write 1 New Done Done false
11863 3957 Acked 4956 Read 4 New Done Done false
11864 3958 Acked 4957 Write 5 New Done Done false
11865 3959 Acked 4958 Read 3 New Done Done false
11866 3960 Acked 4959 Write 10 New Done Done false
11867 3961 Acked 4960 Write 7 New Done Done false
11868 3962 Acked 4961 Write 6 New Done Done false
11869 3963 Acked 4962 Read 8 New Done Done false
11870 3964 Acked 4963 Read 10 New Done Done false
11871 3965 Acked 4964 Read 1 New Done Done false
11872 3966 Acked 4965 Read 10 New Done Done false
11873 3967 Acked 4966 Write 7 New Done Done false
11874 3968 Acked 4967 Write 1 New Done Done false
11875 3969 Acked 4968 Write 6 New Done Done false
11876 3970 Acked 4969 Read 6 New Done Done false
11877 3971 Acked 4970 Write 2 New Done Done false
11878 3972 Acked 4971 Write 7 New Done Done false
11879 3973 Acked 4972 Flush 0 New Done Done false
11880 3974 Acked 4973 Read 6 New Done Done false
11881 3975 Acked 4974 Read 2 New Done Done false
11882 3976 Acked 4975 Read 6 New Done Done false
11883 3977 Acked 4976 Write 3 New Done Done false
11884 3978 Acked 4977 Flush 0 New Done Done false
11885 3979 Acked 4978 Read 1 New Done Done false
11886 3980 Acked 4979 Read 9 New Done Done false
11887 3981 Acked 4980 Read 7 New Done Done false
11888 3982 Acked 4981 Flush 0 New Done Done false
11889 3983 Acked 4982 Read 7 New Done Done false
11890 3984 Acked 4983 Read 4 New Done Done false
11891 3985 Acked 4984 Write 7 New Done Done false
11892 3986 Acked 4985 Read 5 New Done Done false
11893 3987 Acked 4986 Write 7 New Done Done false
11894 3988 Acked 4987 Read 7 New Done Done false
11895 3989 Acked 4988 Flush 0 New Done Done false
11896 3990 Acked 4989 Read 9 New Done Done false
11897 3991 Acked 4990 Write 2 New Done Done false
11898 3992 Acked 4991 Write 4 New Done Done false
11899 3993 Acked 4992 Write 8 New Done Done false
11900 3994 Acked 4993 Read 7 New Done Done false
11901 3995 Acked 4994 Read 8 New Done Done false
11902 3996 Acked 4995 Read 2 New Done Done false
11903 3997 Acked 4996 Write 5 New Done Done false
11904 3998 Acked 4997 Read 8 New Done Done false
11905 3999 Acked 4998 Read 3 New Done Done false
11906 4000 Acked 4999 Write 10 New Done Done false
11907 4001 Acked 5000 Write 1 New Done Done false
11908 4002 Acked 5001 Write 1 New Done Done false
11909 4003 Acked 5002 Read 9 New Done Done false
11910 4004 Acked 5003 Write 3 New Done Done false
11911 4005 Acked 5004 Read 9 New Done Done false
11912 4006 Acked 5005 Flush 0 New Done Done false
11913 4007 Acked 5006 Read 2 New Done Done false
11914 4008 Acked 5007 Read 3 New Done Done false
11915 4009 Acked 5008 Flush 0 New Done Done false
11916 4010 Acked 5009 Write 4 New Done Done false
11917 4011 Acked 5010 Read 6 New Done Done false
11918 4012 Acked 5011 Read 7 New Done Done false
11919 4013 Acked 5012 Write 4 New Done Done false
11920 4014 Acked 5013 Flush 0 New Done Done false
11921 4015 Acked 5014 Write 5 New Done Done false
11922 4016 Acked 5015 Read 7 New Done Done false
11923 4017 Acked 5016 Write 4 New Done Done false
11924 4018 Acked 5017 Read 3 New Done Done false
11925 4019 Acked 5018 Write 1 New Done Done false
11926 4020 Acked 5019 Read 7 New Done Done false
11927 4021 Acked 5020 Write 3 New Done Done false
11928 4022 Acked 5021 Read 10 New Done Done false
11929 4023 Acked 5022 Read 8 New Done Done false
11930 4024 Acked 5023 Read 10 New Done Done false
11931 4025 Acked 5024 Flush 0 New Done Done false
11932 4026 Acked 5025 Read 5 New Done Done false
11933 4027 Acked 5026 Write 7 New Done Done false
11934 4028 Acked 5027 Write 8 New Done Done false
11935 4029 Acked 5028 Write 10 New Done Done false
11936 4030 Acked 5029 Flush 0 New Done Done false
11937 4031 Acked 5030 Write 2 New Done Done false
11938 4032 Acked 5031 Flush 0 New Done Done false
11939 4033 Acked 5032 Flush 0 New Done Done false
11940 4034 Acked 5033 Write 6 New Done Done false
11941 4035 Acked 5034 Write 10 New Done Done false
11942 4036 Acked 5035 Flush 0 New Done Done false
11943 4037 Acked 5036 Write 2 New Done Done false
11944 4038 Acked 5037 Write 8 New Done Done false
11945 4039 Acked 5038 Read 6 New Done Done false
11946 4040 Acked 5039 Read 2 New Done Done false
11947 4041 Acked 5040 Read 6 New Done Done false
11948 4042 Acked 5041 Read 9 New Done Done false
11949 4043 Acked 5042 Flush 0 New Done Done false
11950 4044 Acked 5043 Write 5 New Done Done false
11951 4045 Acked 5044 Read 4 New Done Done false
11952 4046 Acked 5045 Read 5 New Done Done false
11953 4047 Acked 5046 Write 10 New Done Done false
11954 4048 Acked 5047 Write 1 New Done Done false
11955 4049 Acked 5048 Read 8 New Done Done false
11956 4050 Acked 5049 Write 9 New Done Done false
11957 4051 Acked 5050 Read 7 New Done Done false
11958 4052 Acked 5051 Flush 0 New Done Done false
11959 4053 Acked 5052 Read 10 New Done Done false
11960 4054 Acked 5053 Read 6 New Done Done false
11961 4055 Acked 5054 Write 1 New Done Done false
11962 4056 Acked 5055 Read 9 New Done Done false
11963 4057 Acked 5056 Write 1 New Done Done false
11964 4058 Acked 5057 Read 6 New Done Done false
11965 4059 Acked 5058 Write 3 New Done Done false
11966 4060 Acked 5059 Write 7 New Done Done false
11967 4061 Acked 5060 Write 4 New Done Done false
11968 4062 Acked 5061 Write 6 New Done Done false
11969 4063 Acked 5062 Read 6 New Done Done false
11970 4064 Acked 5063 Write 1 New Done Done false
11971 4065 Acked 5064 Write 7 New Done Done false
11972 4066 Acked 5065 Write 2 New Done Done false
11973 4067 Acked 5066 Flush 0 New Done Done false
11974 4068 Acked 5067 Read 4 New Done Done false
11975 4069 Acked 5068 Read 4 New Done Done false
11976 4070 Acked 5069 Write 5 New Done Done false
11977 4071 Acked 5070 Write 4 New Done Done false
11978 4072 Acked 5071 Read 8 New Done Done false
11979 4073 Acked 5072 Flush 0 New Done Done false
11980 4074 Acked 5073 Read 9 New Done Done false
11981 4075 Acked 5074 Read 1 New Done Done false
11982 4076 Acked 5075 Flush 0 New Done Done false
11983 4077 Acked 5076 Read 9 New Done Done false
11984 4078 Acked 5077 Read 5 New Done Done false
11985 4079 Acked 5078 Read 7 New Done Done false
11986 4080 Acked 5079 Read 10 New Done Done false
11987 4081 Acked 5080 Read 5 New Done Done false
11988 4082 Acked 5081 Read 4 New Done Done false
11989 4083 Acked 5082 Read 10 New Done Done false
11990 4084 Acked 5083 Write 8 New Done Done false
11991 4085 Acked 5084 Write 9 New Done Done false
11992 4086 Acked 5085 Read 7 New Done Done false
11993 4087 Acked 5086 Write 8 New Done Done false
11994 4088 Acked 5087 Write 4 New Done Done false
11995 4089 Acked 5088 Read 10 New Done Done false
11996 4090 Acked 5089 Write 8 New Done Done false
11997 4091 Acked 5090 Write 7 New Done Done false
11998 4092 Acked 5091 Read 2 New Done Done false
11999 4093 Acked 5092 Write 4 New Done Done false
12000 4094 Acked 5093 Read 2 New Done Done false
12001 4095 Acked 5094 Write 4 New Done Done false
12002 4096 Acked 5095 Write 5 New Done Done false
12003 4097 Acked 5096 Write 5 New Done Done false
12004 4098 Acked 5097 Read 10 New Done Done false
12005 4099 Acked 5098 Read 5 New Done Done false
12006 4100 Acked 5099 Read 6 New Done Done false
12007 4101 Acked 5100 Write 10 New Done Done false
12008 4102 Acked 5101 Read 8 New Done Done false
12009 4103 Acked 5102 Read 2 New Done Done false
12010 4104 Acked 5103 Read 4 New Done Done false
12011 4105 Acked 5104 Read 1 New Done Done false
12012 4106 Acked 5105 Read 7 New Done Done false
12013 4107 Acked 5106 Write 3 New Done Done false
12014 4108 Acked 5107 Flush 0 New Done Done false
12015 4109 Acked 5108 Read 6 New Done Done false
12016 4110 Acked 5109 Flush 0 New Done Done false
12017 4111 Acked 5110 Read 3 New Done Done false
12018 4112 Acked 5111 Read 4 New Done Done false
12019 4113 Acked 5112 Write 2 New Done Done false
12020 4114 Acked 5113 Write 1 New Done Done false
12021 4115 Acked 5114 Read 8 New Done Done false
12022 4116 Acked 5115 Write 5 New Done Done false
12023 4117 Acked 5116 Write 6 New Done Done false
12024 4118 Acked 5117 Write 3 New Done Done false
12025 4119 Acked 5118 Read 6 New Done Done false
12026 4120 Acked 5119 Read 2 New Done Done false
12027 4121 Acked 5120 Read 1 New Done Done false
12028 4122 Acked 5121 Write 3 New Done Done false
12029 4123 Acked 5122 Read 1 New Done Done false
12030 4124 Acked 5123 Read 3 New Done Done false
12031 4125 Acked 5124 Flush 0 New Done Done false
12032 4126 Acked 5125 Read 9 New Done Done false
12033 4127 Acked 5126 Read 3 New Done Done false
12034 4128 Acked 5127 Write 6 New Done Done false
12035 4129 Acked 5128 Read 1 New Done Done false
12036 4130 Acked 5129 Flush 0 New Done Done false
12037 4131 Acked 5130 Read 3 New Done Done false
12038 4132 Acked 5131 Read 5 New Done Done false
12039 4133 Acked 5132 Read 2 New Done Done false
12040 4134 Acked 5133 Read 1 New Done Done false
12041 4135 Acked 5134 Read 2 New Done Done false
12042 4136 Acked 5135 Read 1 New Done Done false
12043 4137 Acked 5136 Write 5 New Done Done false
12044 4138 Acked 5137 Write 5 New Done Done false
12045 4139 Acked 5138 Write 6 New Done Done false
12046 4140 Acked 5139 Read 7 New Done Done false
12047 4141 Acked 5140 Flush 0 New Done Done false
12048 4142 Acked 5141 Read 2 New Done Done false
12049 4143 Acked 5142 Read 10 New Done Done false
12050 4144 Acked 5143 Write 4 New Done Done false
12051 4145 Acked 5144 Write 1 New Done Done false
12052 4146 Acked 5145 Read 8 New Done Done false
12053 4147 Acked 5146 Read 9 New Done Done false
12054 4148 Acked 5147 Read 1 New Done Done false
12055 4149 Acked 5148 Read 2 New Done Done false
12056 4150 Acked 5149 Write 6 New Done Done false
12057 4151 Acked 5150 Read 5 New Done Done false
12058 4152 Acked 5151 Write 9 New Done Done false
12059 4153 Acked 5152 Flush 0 New Done Done false
12060 4154 Acked 5153 Read 4 New Done Done false
12061 4155 Acked 5154 Write 6 New Done Done false
12062 4156 Acked 5155 Read 7 New Done Done false
12063 4157 Acked 5156 Write 8 New Done Done false
12064 4158 Acked 5157 Read 6 New Done Done false
12065 4159 Acked 5158 Read 6 New Done Done false
12066 4160 Acked 5159 Read 8 New Done Done false
12067 4161 Acked 5160 Read 3 New Done Done false
12068 4162 Acked 5161 Write 6 New Done Done false
12069 4163 Acked 5162 Write 2 New Done Done false
12070 4164 Acked 5163 Write 7 New Done Done false
12071 4165 Acked 5164 Read 8 New Done Done false
12072 4166 Acked 5165 Write 4 New Done Done false
12073 4167 Acked 5166 Flush 0 New Done Done false
12074 4168 Acked 5167 Read 4 New Done Done false
12075 4169 Acked 5168 Write 2 New Done Done false
12076 4170 Acked 5169 Write 5 New Done Done false
12077 4171 Acked 5170 Read 8 New Done Done false
12078 4172 Acked 5171 Read 4 New Done Done false
12079 4173 Acked 5172 Read 4 New Done Done false
12080 4174 Acked 5173 Read 2 New Done Done false
12081 4175 Acked 5174 Write 1 New Done Done false
12082 4176 Acked 5175 Read 6 New Done Done false
12083 4177 Acked 5176 Write 2 New Done Done false
12084 4178 Acked 5177 Write 5 New Done Done false
12085 4179 Acked 5178 Read 6 New Done Done false
12086 4180 Acked 5179 Write 1 New Done Done false
12087 4181 Acked 5180 Write 2 New Done Done false
12088 4182 Acked 5181 Write 5 New Done Done false
12089 4183 Acked 5182 Flush 0 New Done Done false
12090 4184 Acked 5183 Write 10 New Done Done false
12091 4185 Acked 5184 Write 5 New Done Done false
12092 4186 Acked 5185 Write 9 New Done Done false
12093 4187 Acked 5186 Write 6 New Done Done false
12094 4188 Acked 5187 Read 4 New Done Done false
12095 4189 Acked 5188 Read 4 New Done Done false
12096 4190 Acked 5189 Write 5 New Done Done false
12097 4191 Acked 5190 Write 2 New Done Done false
12098 4192 Acked 5191 Write 6 New Done Done false
12099 4193 Acked 5192 Read 3 New Done Done false
12100 4194 Acked 5193 Read 9 New Done Done false
12101 4195 Acked 5194 Write 8 New Done Done false
12102 4196 Acked 5195 Read 7 New Done Done false
12103 4197 Acked 5196 Read 7 New Done Done false
12104 4198 Acked 5197 Write 2 New Done Done false
12105 4199 Acked 5198 Read 8 New Done Done false
12106 4200 Acked 5199 Read 3 New Done Done false
12107 4201 Acked 5200 Write 5 New Done Done false
12108 4202 Acked 5201 Read 9 New Done Done false
12109 4203 Acked 5202 Read 2 New Done Done false
12110 4204 Acked 5203 Write 9 New Done Done false
12111 4205 Acked 5204 Write 4 New Done Done false
12112 4206 Acked 5205 Read 4 New Done Done false
12113 4207 Acked 5206 Read 5 New Done Done false
12114 4208 Acked 5207 Read 9 New Done Done false
12115 4209 Acked 5208 Read 1 New Done Done false
12116 4210 Acked 5209 Write 1 New Done Done false
12117 4211 Acked 5210 Read 8 New Done Done false
12118 4212 Acked 5211 Flush 0 New Done Done false
12119 4213 Acked 5212 Write 5 New Done Done false
12120 4214 Acked 5213 Read 5 New Done Done false
12121 4215 Acked 5214 Read 2 New Done Done false
12122 4216 Acked 5215 Write 3 New Done Done false
12123 4217 Acked 5216 Read 10 New Done Done false
12124 4218 Acked 5217 Write 4 New Done Done false
12125 4219 Acked 5218 Read 4 New Done Done false
12126 4220 Acked 5219 Write 1 New Done Done false
12127 4221 Acked 5220 Read 7 New Done Done false
12128 4222 Acked 5221 Write 4 New Done Done false
12129 4223 Acked 5222 Flush 0 New Done Done false
12130 4224 Acked 5223 Read 9 New Done Done false
12131 4225 Acked 5224 Write 3 New Done Done false
12132 4226 Acked 5225 Write 5 New Done Done false
12133 4227 Acked 5226 Write 9 New Done Done false
12134 4228 Acked 5227 Read 3 New Done Done false
12135 4229 Acked 5228 Write 7 New Done Done false
12136 4230 Acked 5229 Read 6 New Done Done false
12137 4231 Acked 5230 Write 4 New Done Done false
12138 4232 Acked 5231 Read 2 New Done Done false
12139 4233 Acked 5232 Write 9 New Done Done false
12140 4234 Acked 5233 Flush 0 New Done Done false
12141 4235 Acked 5234 Write 2 New Done Done false
12142 4236 Acked 5235 Flush 0 New Done Done false
12143 STATES DS:0 DS:1 DS:2 TOTAL
12144 New 302 0 0 302
12145 Sent 0 0 0 0
12146 Done 0 302 302 604
12147 Skipped 0 0 0 0
12148 Error 0 0 0 0
12149 Last Flush: 4933 5235 5235
12150 Downstairs last five completed: 4933 4932 4931 4930 4929 4928
12151 Upstairs last five completed: 4236 4235 4234 4233 4232 4231
12152 CLIENT: Up:0 ds:302 act:2
121532023-10-13T04:44:22.279ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 0
121542023-10-13T04:44:22.279ZINFOcrucible: [0] Proc runs for 127.0.0.1:8810 in state Offline
121552023-10-13T04:44:22.279ZINFOcrucible: [0] client re-new 302 jobs since flush 4933 = downstairs
121562023-10-13T04:44:22.280ZINFOcrucible: [0] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
121572023-10-13T04:44:22.280ZINFOcrucible: [0] downstairs client at 127.0.0.1:8810 has UUID 12345678-0000-0000-0000-000000008810
121582023-10-13T04:44:22.280ZINFOcrucible: [0] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }
121592023-10-13T04:44:22.280ZINFOcrucible: Returning client:0 UUID:12345678-0000-0000-0000-000000008810 matches
121602023-10-13T04:44:22.280ZINFOcrucible: [0] send last flush ID to this DS: 4933
121612023-10-13T04:44:22.280ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Offline Active Active
121622023-10-13T04:44:22.281ZINFOcrucible: [0] Replied this last flush ID: 4933
121632023-10-13T04:44:22.281ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Offline Active Active ds_transition to Replay
121642023-10-13T04:44:22.281ZINFOcrucible: [0] Transition from Offline to Replay
121652023-10-13T04:44:22.281ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
121662023-10-13T04:44:22.281ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Replay Active Active ds_transition to Active
121672023-10-13T04:44:22.281ZINFOcrucible: [0] Transition from Replay to Active
121682023-10-13T04:44:22.281ZINFOcrucible: [0] Starts cmd_loop
121692023-10-13T04:44:22.281ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:true
121702023-10-13T04:44:22.281ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
121712023-10-13T04:44:22.281ZINFOcrucible: Set check for repair
121722023-10-13T04:44:23.282ZWARNcrucible: [0] flow control end
121732023-10-13T04:44:23.282ZINFOcrucible: Checking if live repair is needed
121742023-10-13T04:44:23.282ZINFOcrucible: No Live Repair required at this time
12175 ----------------------------------------------------------------
12176 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12177 Downstairs last five completed: 5235 5234 5233 5232 5231 5230
12178 Upstairs last five completed: 4236 4235 4234 4233 4232 4231
12179 CLIENT: Up:0 ds:0 act:3
12180 Replay: All jobs finished, all DS active.
12181 Main task has work to do, go find it
12182 got dsc Stop(0)
12183 stop 0
12184 Oct 13 04:44:26.151 INFO request completed, latency_us: 469, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 0efdcdf9-4e89-4676-a7e1-cbf2e3593932, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0[8810] Got stop action so:false kr:false
12185 .1:9998
12186 Oct 13 04:44:26.152 INFO request completed, latency_us: 194, response_code: 200, uri: /state/cid/0, method: GET, req_id: 90603fa8-ccda-43d3-a6f8-d6ff3d0ee2a1, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
121872023-10-13T04:44:26.152ZWARNcrucible: [0] downstairs disconnected
121882023-10-13T04:44:26.153ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
121892023-10-13T04:44:26.153ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8810 closed looper = 0
121902023-10-13T04:44:26.153ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:false
121912023-10-13T04:44:26.153ZWARNcrucible: [0] pm_task rx.recv() is None
121922023-10-13T04:44:26.153ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Offline Active Active
121932023-10-13T04:44:26.153ZINFOcrucible: [0] 127.0.0.1:8810 task reports offline
12194 [8810] Exited with: ExitStatus(unix_wait_status(9))
12195 [8810][0] reports Exit
121962023-10-13T04:44:29.156ZINFOcrucible: [0] connecting to 127.0.0.1:8810 looper = 0
12197 Oct 13 04:44:30.155 INFO request completed, latency_us: 247, response_code: 200, uri: /state/cid/0, method: GET, req_id: ca1b88ad-880a-4389-90d5-c5a320cec24c, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
12198 001/300 Read block 2384 len 20480
12199 002/300 Write block 1286 len 24576 data: 5 5 7 8 8 7
12200 003/300 Write block 2848 len 4096 data: 5
12201 004/300 Write block 561 len 4096 data: 3
12202 005/300 Write block 2048 len 20480 data: 2 3 3 3 3
12203 006/300 Write block 1076 len 36864 data: 5 6 7 6 7 9 9 6 6
12204 007/300 Write block 343 len 32768 data: 2 2 2 2 2 3 3 4
12205 008/300 Write block 508 len 4096 data: 3
12206 009/300 Write block 4411 len 12288 data: 5 5 7
12207 010/300 Read block 4454 len 20480
12208 011/300 Write block 4075 len 28672 data: 4 4 2 3 3 4 5
12209 012/300 Read block 4782 len 36864
12210 013/300 Read block 4026 len 32768
12211 014/300 Read block 2793 len 28672
12212 015/300 Flush
12213 016/300 Read block 2652 len 40960
12214 017/300 Write block 1149 len 32768 data: 4 5 5 5 5 5 4 4
12215 018/300 Write block 1507 len 40960 data: 6 5 5 5 5 5 5 5 6 4
12216 019/300 Flush
12217 020/300 Read block 2689 len 4096
12218 021/300 Flush
12219 022/300 Write block 3703 len 28672 data: 3 3 4 5 4 4 5
12220 023/300 Read block 1951 len 20480
12221 024/300 Read block 627 len 16384
12222 025/300 Read block 4780 len 24576
12223 026/300 Read block 179 len 12288
12224 027/300 Read block 3010 len 4096
12225 028/300 Read block 3283 len 36864
12226 029/300 Write block 1093 len 28672 data: 6 7 5 4 5 6 5
12227 030/300 Read block 135 len 4096
12228 031/300 Read block 3587 len 28672
12229 032/300 Write block 2745 len 16384 data: 2 2 3 5
12230 033/300 Write block 1821 len 4096 data: 3
12231 034/300 Read block 3491 len 8192
12232 035/300 Write block 3535 len 16384 data: 6 5 2 2
12233 036/300 Read block 812 len 40960
12234 037/300 Read block 3047 len 12288
12235 038/300 Read block 393 len 4096
12236 039/300 Write block 2275 len 28672 data: 4 4 3 4 4 4 3
12237 040/300 Read block 4075 len 16384
12238 041/300 Write block 4758 len 24576 data: 4 4 4 3 4 3
12239 042/300 Read block 2847 len 4096
12240 043/300 Read block 4502 len 36864
12241 044/300 Write block 3166 len 24576 data: 4 4 3 3 4 2
12242 045/300 Read block 2207 len 24576
12243 046/300 Read block 2430 len 24576
12244 047/300 Read block 2702 len 20480
12245 048/300 Write block 1537 len 28672 data: 5 5 4 3 3 3 3
12246 049/300 Write block 927 len 16384 data: 4 2 2 2
12247 050/300 Flush
12248 051/300 Write block 1186 len 28672 data: 3 4 4 3 3 3 2
12249 052/300 Read block 1683 len 40960
12250 053/300 Write block 3840 len 8192 data: 5 7
12251 054/300 Write block 2881 len 12288 data: 3 3 3
12252 055/300 Read block 3024 len 24576
12253 056/300 Read block 2215 len 32768
12254 057/300 Write block 2526 len 12288 data: 6 6 6
12255 058/300 Read block 298 len 24576
12256 059/300 Flush
12257 060/300 Read block 4032 len 16384
12258 061/300 Read block 4924 len 40960
12259 062/300 Read block 2830 len 36864
12260 063/300 Flush
12261 064/300 Write block 4527 len 32768 data: 5 5 5 5 5 3 2 2
12262 065/300 Read block 3054 len 20480
12263 066/300 Write block 4556 len 12288 data: 2 3 3
12264 067/300 Write block 4384 len 40960 data: 2 2 4 4 4 4 5 4 4 4
12265 068/300 Write block 2221 len 28672 data: 5 5 4 5 5 3 2
12266 069/300 Flush
12267 070/300 Read block 2708 len 20480
12268 071/300 Read block 3829 len 28672
12269 072/300 Read block 2974 len 8192
12270 073/300 Read block 2461 len 16384
12271 074/300 Read block 3133 len 20480
12272 075/300 Read block 2297 len 8192
12273 076/300 Write block 2753 len 4096 data: 4
12274 077/300 Read block 3160 len 16384
12275 078/300 Write block 2132 len 8192 data: 3 3
12276 079/300 Flush
12277 080/300 Write block 649 len 12288 data: 3 3 2
12278 081/300 Write block 1629 len 20480 data: 4 4 4 4 4
12279 082/300 Read block 4696 len 16384
12280 083/300 Write block 2964 len 36864 data: 3 3 3 4 5 4 4 4 3
12281 084/300 Read block 2330 len 36864
12282 085/300 Write block 1430 len 4096 data: 5
12283 086/300 Read block 754 len 40960
12284 087/300 Write block 1959 len 8192 data: 3 3
12285 088/300 Read block 2486 len 36864
12286 089/300 Read block 3038 len 36864
12287 090/300 Read block 3788 len 24576
12288 091/300 Write block 4923 len 8192 data: 3 3
12289 092/300 Write block 2148 len 28672 data: 3 3 2 2 2 3 3
12290 093/300 Read block 684 len 28672
12291 094/300 Write block 3673 len 4096 data: 3
12292 095/300 Write block 3312 len 32768 data: 3 4 4 5 5 5 4 5
12293 096/300 Read block 733 len 12288
12294 097/300 Write block 3667 len 40960 data: 3 3 3 3 3 3 4 3 3 3
12295 098/300 Read block 1608 len 16384
12296 099/300 Write block 4344 len 4096 data: 6
12297 100/300 Read block 1435 len 4096
12298 101/300 Read block 122 len 8192
12299 102/300 Write block 4252 len 28672 data: 4 5 3 3 4 3 3
12300 103/300 Read block 762 len 16384
12301 104/300 Write block 1954 len 32768 data: 5 3 4 3 3 4 4 3
12302 105/300 Write block 162 len 36864 data: 5 5 4 4 5 5 5 5 4
12303 106/300 Read block 1987 len 40960
12304 107/300 Write block 520 len 20480 data: 7 8 9 9 7
12305 108/300 Read block 1836 len 16384
12306 109/300 Read block 4120 len 28672
12307 110/300 Read block 4539 len 8192
12308 111/300 Write block 3559 len 28672 data: 7 7 8 7 4 3 3
12309 112/300 Read block 1496 len 4096
12310 113/300 Flush
12311 114/300 Read block 1422 len 16384
12312 115/300 Write block 3156 len 32768 data: 3 3 4 4 4 4 4 6
12313 116/300 Read block 4419 len 36864
12314 117/300 Write block 2184 len 16384 data: 4 4 4 3
12315 118/300 Flush
12316 119/300 Read block 1490 len 12288
12317 120/300 Read block 2830 len 36864
12318 121/300 Read block 4913 len 20480
12319 122/300 Write block 897 len 32768 data: 2 5 5 6 6 6 6 6
12320 123/300 Read block 3873 len 20480
12321 124/300 Write block 4621 len 24576 data: 4 7 6 6 6 5
12322 125/300 Write block 2842 len 32768 data: 4 3 3 3 4 4 6 5
12323 126/300 Read block 2887 len 12288
12324 127/300 Read block 2199 len 28672
12325 128/300 Write block 942 len 32768 data: 3 3 3 2 2 2 3 3
12326 129/300 Read block 4232 len 32768
12327 130/300 Write block 3892 len 12288 data: 4 5 4
12328 131/300 Read block 765 len 36864
12329 132/300 Read block 1036 len 8192
12330 133/300 Write block 999 len 36864 data: 2 2 2 2 2 3 4 4 4
12331 134/300 Write block 1176 len 8192 data: 4 4
12332 135/300 Read block 2668 len 28672
12333 136/300 Write block 2856 len 4096 data: 4
12334 137/300 Write block 3851 len 32768 data: 3 5 5 5 4 4 5 6
12335 138/300 Read block 319 len 4096
12336 139/300 Read block 4791 len 4096
12337 140/300 Write block 4583 len 36864 data: 3 4 4 4 3 3 3 3 3
12338 141/300 Read block 1870 len 40960
12339 142/300 Write block 1008 len 12288 data: 2 4 4
12340 143/300 Flush
12341 144/300 Write block 872 len 12288 data: 5 7 7
12342 145/300 Read block 2807 len 16384
12343 146/300 Read block 1602 len 40960
12344 147/300 Write block 780 len 28672 data: 2 2 2 3 3 3 3
12345 148/300 Write block 1303 len 28672 data: 2 3 4 4 3 3 3
12346 149/300 Read block 2462 len 16384
12347 150/300 Write block 2091 len 8192 data: 7 6
12348 151/300 Write block 1580 len 32768 data: 2 2 2 2 3 3 3 4
12349 152/300 Write block 3214 len 12288 data: 3 3 2
12350 153/300 Read block 1468 len 36864
12351 154/300 Read block 2760 len 40960
12352 155/300 Write block 959 len 4096 data: 2
12353 156/300 Write block 4285 len 12288 data: 3 3 4
12354 157/300 Flush
12355 158/300 Read block 1545 len 24576
12356 159/300 Read block 4160 len 40960
12357 160/300 Read block 3743 len 24576
12358 161/300 Read block 902 len 4096
12359 162/300 Read block 3015 len 36864
12360 163/300 Write block 3060 len 8192 data: 2 3
12361 164/300 Read block 4247 len 16384
12362 165/300 Read block 4229 len 24576
12363 166/300 Read block 3290 len 8192
12364 167/300 Read block 2488 len 20480
12365 168/300 Write block 4790 len 20480 data: 2 2 3 3 3
12366 169/300 Read block 616 len 24576
12367 170/300 Read block 4715 len 12288
12368 171/300 Read block 2446 len 20480
12369 172/300 Read block 4010 len 40960
12370 173/300 Read block 4946 len 20480
12371 174/300 Write block 4667 len 36864 data: 2 2 2 3 3 3 2 2 2
12372 175/300 Write block 735 len 36864 data: 3 3 2 3 2 2 4 4 3
12373 176/300 Write block 1644 len 12288 data: 4 4 4
12374 177/300 Read block 3904 len 4096
12375 178/300 Read block 4 len 28672
12376 179/300 Write block 969 len 36864 data: 6 7 4 5 4 4 5 5 4
12377 180/300 Read block 123 len 4096
12378 181/300 Write block 4191 len 24576 data: 3 3 3 3 3 3
12379 182/300 Read block 1595 len 4096
12380 183/300 Write block 3626 len 24576 data: 3 3 3 3 3 3
12381 184/300 Read block 1268 len 20480
12382 185/300 Read block 2799 len 28672
12383 186/300 Read block 2405 len 8192
12384 187/300 Write block 4172 len 24576 data: 3 3 2 2 2 2
12385 188/300 Read block 4550 len 16384
12386 189/300 Write block 185 len 28672 data: 3 3 4 4 4 4 4
12387 190/300 Write block 1054 len 24576 data: 3 3 3 3 2 2
12388 191/300 Read block 991 len 4096
12389 192/300 Write block 3883 len 12288 data: 2 3 3
12390 193/300 Read block 2322 len 40960
12391 194/300 Write block 2622 len 12288 data: 3 3 3
12392 195/300 Read block 1524 len 24576
12393 196/300 Read block 1023 len 28672
12394 197/300 Read block 1706 len 12288
12395 198/300 Read block 798 len 12288
12396 199/300 Flush
12397 200/300 Read block 757 len 4096
12398 201/300 Read block 3308 len 24576
12399 202/300 Read block 1981 len 4096
12400 203/300 Write block 2628 len 16384 data: 4 3 3 3
12401 204/300 Read block 723 len 32768
12402 205/300 Read block 2256 len 28672
12403 206/300 Read block 1510 len 12288
12404 207/300 Read block 3829 len 32768
12405 208/300 Write block 1390 len 32768 data: 3 3 3 3 2 2 2 3
12406 209/300 Read block 4079 len 20480
12407 210/300 Read block 422 len 28672
12408 211/300 Write block 4765 len 20480 data: 3 3 3 4 4
12409 212/300 Write block 4550 len 8192 data: 2 3
12410 213/300 Read block 738 len 16384
12411 214/300 Write block 3217 len 16384 data: 2 2 2 3
12412 215/300 Write block 2606 len 24576 data: 3 4 3 3 4 5
12413 216/300 Read block 1819 len 24576
12414 217/300 Read block 3090 len 28672
12415 218/300 Read block 3798 len 40960
12416 219/300 Read block 3460 len 8192
12417 220/300 Write block 3231 len 32768 data: 2 2 2 3 3 3 4 4
12418 221/300 Write block 3379 len 12288 data: 3 5 5
12419 222/300 Read block 4041 len 32768
12420 223/300 Write block 3956 len 4096 data: 5
12421 224/300 Read block 4090 len 4096
12422 225/300 Write block 1583 len 20480 data: 3 4 4 4 5
12423 226/300 Flush
12424 227/300 Write block 905 len 40960 data: 4 3 3 2 2 3 3 4 6 7
12425 228/300 Write block 1932 len 4096 data: 3
12426 229/300 Write block 4027 len 24576 data: 5 3 3 2 2 3
12427 230/300 Read block 3521 len 20480
12428 231/300 Flush
12429 232/300 Read block 2100 len 16384
12430 233/300 Read block 13 len 16384
12431 234/300 Flush
12432 235/300 Write block 4216 len 8192 data: 6 5
12433 236/300 Read block 839 len 40960
12434 237/300 Read block 2964 len 28672
12435 238/300 Write block 4194 len 16384 data: 4 4 4 4
12436 239/300 Write block 1120 len 12288 data: 3 3 3
12437 240/300 Read block 3329 len 24576
12438 241/300 Read block 1633 len 24576
12439 242/300 Write block 2610 len 12288 data: 5 6 6
12440 243/300 Read block 2419 len 12288
12441 244/300 Write block 714 len 16384 data: 2 3 3 3
12442 245/300 Flush
12443 246/300 Read block 3582 len 20480
12444 247/300 Flush
12445 248/300 Write block 227 len 16384 data: 2 2 2 2
12446 249/300 Read block 3472 len 4096
12447 250/300 Write block 2568 len 24576 data: 6 6 7 8 8 8
12448 251/300 Write block 4318 len 28672 data: 5 4 4 3 3 2 2
12449 252/300 Write block 96 len 20480 data: 4 3 3 3 2
12450 253/300 Read block 2213 len 8192
12451 254/300 Read block 2229 len 8192
12452 255/300 Write block 3349 len 24576 data: 3 4 5 6 5 5
12453 256/300 Read block 434 len 20480
12454 257/300 Read block 911 len 4096
12455 258/300 Write block 3738 len 40960 data: 3 4 3 3 4 4 4 4 4 4
12456 259/300 Read block 3648 len 32768
12457 260/300 Read block 3563 len 4096
12458 261/300 Write block 3116 len 4096 data: 3
12459 262/300 Read block 3990 len 24576
12460 263/300 Write block 293 len 28672 data: 7 7 6 7 8 7 7
12461 264/300 Read block 1070 len 12288
12462 265/300 Write block 3759 len 20480 data: 3 2 3 3 3
12463 266/300 Read block 1112 len 4096
12464 267/300 Write block 596 len 4096 data: 3
12465 268/300 Write block 59 len 40960 data: 4 4 4 2 2 4 4 4 4 3
12466 269/300 Read block 2223 len 8192
12467 270/300 Write block 91 len 28672 data: 3 5 4 3 4 5 4
12468 271/300 Read block 3784 len 4096
12469 272/300 Write block 723 len 28672 data: 5 5 6 5 5 5 5
12470 273/300 Write block 1596 len 8192 data: 3 2
12471 274/300 Write block 1670 len 16384 data: 5 4 3 3
12472 275/300 Write block 3844 len 36864 data: 4 4 5 5 5 6 5 4 6
12473 276/300 Read block 1247 len 4096
12474 277/300 Read block 2373 len 4096
12475 278/300 Write block 3636 len 40960 data: 3 2 3 3 3 3 3 4 4 3
12476 279/300 Flush
12477 280/300 Read block 2904 len 28672
12478 281/300 Read block 4826 len 20480
12479 282/300 Write block 4129 len 20480 data: 5 4 3 4 5
12480 283/300 Write block 4147 len 32768 data: 3 3 3 4 4 5 5 6
12481 284/300 Read block 393 len 8192
12482 285/300 Read block 3221 len 40960
12483 286/300 Write block 784 len 24576 data: 4 4 4 2 3 3
12484 287/300 Flush
12485 288/300 Read block 3467 len 8192
12486 289/300 Flush
12487 290/300 Read block 2590 len 40960
12488 291/300 Read block 862 len 20480
12489 292/300 Read block 4848 len 8192
12490 293/300 Write block 2321 len 20480 data: 5 7 8 8 6
12491 294/300 Read block 3828 len 32768
12492 295/300 Write block 4586 len 20480 data: 5 4 4 4 4
12493 296/300 Read block 4225 len 24576
12494 297/300 Read block 4214 len 32768
12495 298/300 Write block 1000 len 28672 data: 3 3 3 3 4 5 5
12496 299/300 Write block 1253 len 24576 data: 3 4 3 2 2 2
12497 300/300 Flush
12498 Main task has work to do, go find it
12499 got dsc Start(0)
12500 start 0
12501 Oct 13 04:44:31.023 INFO request completed, latency_us: 306, response_code: 204, uri: /start/cid/0, method: POST, req_id: deaa23cc-4f90-4c80-9e71-ad7522289e35, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
12502 Replay: started 0, returned:Ok(())
12503 ----------------------------------------------------------------
12504 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
12505 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
12506 4237 Acked 5236 Read 5 New Done Done false
12507 4238 Acked 5237 Write 6 New Done Done false
12508 4239 Acked 5238 Write 1 New Done Done false
12509 4240 Acked 5239 Write 1 New Done Done false
12510 4241 Acked 5240 Write 5 New Done Done false
12511 4242 Acked 5241 Write 9 New Done Done false
12512 4243 Acked 5242 Write 8 New Done Done false
12513 4244 Acked 5243 Write 1 New Done Done false
12514 4245 Acked 5244 Write 3 New Done Done false
12515 4246 Acked 5245 Read 5 New Done Done false
12516 4247 Acked 5246 Write 7 New Done Done false
12517 4248 Acked 5247 Read 9 New Done Done false
12518 4249 Acked 5248 Read 8 New Done Done false
12519 4250 Acked 5249 Read 7 New Done Done false
12520 4251 Acked 5250 Flush 0 New Done Done false
12521 4252 Acked 5251 Read 10 New Done Done false
12522 4253 Acked 5252 Write 8 New Done Done false
12523 4254 Acked 5253 Write 10 New Done Done false
12524 4255 Acked 5254 Flush 0 New Done Done false
12525 4256 Acked 5255 Read 1 New Done Done false
12526 4257 Acked 5256 Flush 0 New Done Done false
12527 4258 Acked 5257 Write 7 New Done Done false
12528 4259 Acked 5258 Read 5 New Done Done false
12529 4260 Acked 5259 Read 4 New Done Done false
12530 4261 Acked 5260 Read 6 New Done Done false
12531 4262 Acked 5261 Read 3 New Done Done false
12532 4263 Acked 5262 Read 1 New Done Done false
12533 4264 Acked 5263 Read 9 New Done Done false
12534 4265 Acked 5264 Write 7 New Done Done false
12535 4266 Acked 5265 Read 1 New Done Done false
12536 4267 Acked 5266 Read 7 New Done Done false
12537 4268 Acked 5267 Write 4 New Done Done false
12538 4269 Acked 5268 Write 1 New Done Done false
12539 4270 Acked 5269 Read 2 New Done Done false
12540 4271 Acked 5270 Write 4 New Done Done false
12541 4272 Acked 5271 Read 10 New Done Done false
12542 4273 Acked 5272 Read 3 New Done Done false
12543 4274 Acked 5273 Read 1 New Done Done false
12544 4275 Acked 5274 Write 7 New Done Done false
12545 4276 Acked 5275 Read 4 New Done Done false
12546 4277 Acked 5276 Write 6 New Done Done false
12547 4278 Acked 5277 Read 1 New Done Done false
12548 4279 Acked 5278 Read 9 New Done Done false
12549 4280 Acked 5279 Write 6 New Done Done false
12550 4281 Acked 5280 Read 6 New Done Done false
12551 4282 Acked 5281 Read 6 New Done Done false
12552 4283 Acked 5282 Read 5 New Done Done false
12553 4284 Acked 5283 Write 7 New Done Done false
12554 4285 Acked 5284 Write 4 New Done Done false
12555 4286 Acked 5285 Flush 0 New Done Done false
12556 4287 Acked 5286 Write 7 New Done Done false
12557 4288 Acked 5287 Read 10 New Done Done false
12558 4289 Acked 5288 Write 2 New Done Done false
12559 4290 Acked 5289 Write 3 New Done Done false
12560 4291 Acked 5290 Read 6 New Done Done false
12561 4292 Acked 5291 Read 8 New Done Done false
12562 4293 Acked 5292 Write 3 New Done Done false
12563 4294 Acked 5293 Read 6 New Done Done false
12564 4295 Acked 5294 Flush 0 New Done Done false
12565 4296 Acked 5295 Read 4 New Done Done false
12566 4297 Acked 5296 Read 10 New Done Done false
12567 4298 Acked 5297 Read 9 New Done Done false
12568 4299 Acked 5298 Flush 0 New Done Done false
12569 4300 Acked 5299 Write 8 New Done Done false
12570 4301 Acked 5300 Read 5 New Done Done false
12571 4302 Acked 5301 Write 3 New Done Done false
12572 4303 Acked 5302 Write 10 New Done Done false
12573 4304 Acked 5303 Write 7 New Done Done false
12574 4305 Acked 5304 Flush 0 New Done Done false
12575 4306 Acked 5305 Read 5 New Done Done false
12576 4307 Acked 5306 Read 7 New Done Done false
12577 4308 Acked 5307 Read 2 New Done Done false
12578 4309 Acked 5308 Read 4 New Done Done false
12579 4310 Acked 5309 Read 5 New Done Done false
12580 4311 Acked 5310 Read 2 New Done Done false
12581 4312 Acked 5311 Write 1 New Done Done false
12582 4313 Acked 5312 Read 4 New Done Done false
12583 4314 Acked 5313 Write 2 New Done Done false
12584 4315 Acked 5314 Flush 0 New Done Done false
12585 4316 Acked 5315 Write 3 New Done Done false
12586 4317 Acked 5316 Write 5 New Done Done false
12587 4318 Acked 5317 Read 4 New Done Done false
12588 4319 Acked 5318 Write 9 New Done Done false
12589 4320 Acked 5319 Read 9 New Done Done false
12590 4321 Acked 5320 Write 1 New Done Done false
12591 4322 Acked 5321 Read 10 New Done Done false
12592 4323 Acked 5322 Write 2 New Done Done false
12593 4324 Acked 5323 Read 9 New Done Done false
12594 4325 Acked 5324 Read 9 New Done Done false
12595 4326 Acked 5325 Read 6 New Done Done false
12596 4327 Acked 5326 Write 2 New Done Done false
12597 4328 Acked 5327 Write 7 New Done Done false
12598 4329 Acked 5328 Read 7 New Done Done false
12599 4330 Acked 5329 Write 1 New Done Done false
12600 4331 Acked 5330 Write 8 New Done Done false
12601 4332 Acked 5331 Read 3 New Done Done false
12602 4333 Acked 5332 Write 10 New Done Done false
12603 4334 Acked 5333 Read 4 New Done Done false
12604 4335 Acked 5334 Write 1 New Done Done false
12605 4336 Acked 5335 Read 1 New Done Done false
12606 4337 Acked 5336 Read 2 New Done Done false
12607 4338 Acked 5337 Write 7 New Done Done false
12608 4339 Acked 5338 Read 4 New Done Done false
12609 4340 Acked 5339 Write 8 New Done Done false
12610 4341 Acked 5340 Write 9 New Done Done false
12611 4342 Acked 5341 Read 10 New Done Done false
12612 4343 Acked 5342 Write 5 New Done Done false
12613 4344 Acked 5343 Read 4 New Done Done false
12614 4345 Acked 5344 Read 7 New Done Done false
12615 4346 Acked 5345 Read 2 New Done Done false
12616 4347 Acked 5346 Write 7 New Done Done false
12617 4348 Acked 5347 Read 1 New Done Done false
12618 4349 Acked 5348 Flush 0 New Done Done false
12619 4350 Acked 5349 Read 4 New Done Done false
12620 4351 Acked 5350 Write 8 New Done Done false
12621 4352 Acked 5351 Read 9 New Done Done false
12622 4353 Acked 5352 Write 4 New Done Done false
12623 4354 Acked 5353 Flush 0 New Done Done false
12624 4355 Acked 5354 Read 3 New Done Done false
12625 4356 Acked 5355 Read 9 New Done Done false
12626 4357 Acked 5356 Read 5 New Done Done false
12627 4358 Acked 5357 Write 8 New Done Done false
12628 4359 Acked 5358 Read 5 New Done Done false
12629 4360 Acked 5359 Write 6 New Done Done false
12630 4361 Acked 5360 Write 8 New Done Done false
12631 4362 Acked 5361 Read 3 New Done Done false
12632 4363 Acked 5362 Read 7 New Done Done false
12633 4364 Acked 5363 Write 8 New Done Done false
12634 4365 Acked 5364 Read 8 New Done Done false
12635 4366 Acked 5365 Write 3 New Done Done false
12636 4367 Acked 5366 Read 9 New Done Done false
12637 4368 Acked 5367 Read 2 New Done Done false
12638 4369 Acked 5368 Write 9 New Done Done false
12639 4370 Acked 5369 Write 2 New Done Done false
12640 4371 Acked 5370 Read 7 New Done Done false
12641 4372 Acked 5371 Write 1 New Done Done false
12642 4373 Acked 5372 Write 8 New Done Done false
12643 4374 Acked 5373 Read 1 New Done Done false
12644 4375 Acked 5374 Read 1 New Done Done false
12645 4376 Acked 5375 Write 9 New Done Done false
12646 4377 Acked 5376 Read 10 New Done Done false
12647 4378 Acked 5377 Write 3 New Done Done false
12648 4379 Acked 5378 Flush 0 New Done Done false
12649 4380 Acked 5379 Write 3 New Done Done false
12650 4381 Acked 5380 Read 4 New Done Done false
12651 4382 Acked 5381 Read 10 New Done Done false
12652 4383 Acked 5382 Write 7 New Done Done false
12653 4384 Acked 5383 Write 7 New Done Done false
12654 4385 Acked 5384 Read 4 New Done Done false
12655 4386 Acked 5385 Write 2 New Done Done false
12656 4387 Acked 5386 Write 8 New Done Done false
12657 4388 Acked 5387 Write 3 New Done Done false
12658 4389 Acked 5388 Read 9 New Done Done false
12659 4390 Acked 5389 Read 10 New Done Done false
12660 4391 Acked 5390 Write 1 New Done Done false
12661 4392 Acked 5391 Write 3 New Done Done false
12662 4393 Acked 5392 Flush 0 New Done Done false
12663 4394 Acked 5393 Read 6 New Done Done false
12664 4395 Acked 5394 Read 10 New Done Done false
12665 4396 Acked 5395 Read 6 New Done Done false
12666 4397 Acked 5396 Read 1 New Done Done false
12667 4398 Acked 5397 Read 9 New Done Done false
12668 4399 Acked 5398 Write 2 New Done Done false
12669 4400 Acked 5399 Read 4 New Done Done false
12670 4401 Acked 5400 Read 6 New Done Done false
12671 4402 Acked 5401 Read 2 New Done Done false
12672 4403 Acked 5402 Read 5 New Done Done false
12673 4404 Acked 5403 Write 5 New Done Done false
12674 4405 Acked 5404 Read 6 New Done Done false
12675 4406 Acked 5405 Read 3 New Done Done false
12676 4407 Acked 5406 Read 5 New Done Done false
12677 4408 Acked 5407 Read 10 New Done Done false
12678 4409 Acked 5408 Read 5 New Done Done false
12679 4410 Acked 5409 Write 9 New Done Done false
12680 4411 Acked 5410 Write 9 New Done Done false
12681 4412 Acked 5411 Write 3 New Done Done false
12682 4413 Acked 5412 Read 1 New Done Done false
12683 4414 Acked 5413 Read 7 New Done Done false
12684 4415 Acked 5414 Write 9 New Done Done false
12685 4416 Acked 5415 Read 1 New Done Done false
12686 4417 Acked 5416 Write 6 New Done Done false
12687 4418 Acked 5417 Read 1 New Done Done false
12688 4419 Acked 5418 Write 6 New Done Done false
12689 4420 Acked 5419 Read 5 New Done Done false
12690 4421 Acked 5420 Read 7 New Done Done false
12691 4422 Acked 5421 Read 2 New Done Done false
12692 4423 Acked 5422 Write 6 New Done Done false
12693 4424 Acked 5423 Read 4 New Done Done false
12694 4425 Acked 5424 Write 7 New Done Done false
12695 4426 Acked 5425 Write 6 New Done Done false
12696 4427 Acked 5426 Read 1 New Done Done false
12697 4428 Acked 5427 Write 3 New Done Done false
12698 4429 Acked 5428 Read 10 New Done Done false
12699 4430 Acked 5429 Write 3 New Done Done false
12700 4431 Acked 5430 Read 6 New Done Done false
12701 4432 Acked 5431 Read 7 New Done Done false
12702 4433 Acked 5432 Read 3 New Done Done false
12703 4434 Acked 5433 Read 3 New Done Done false
12704 4435 Acked 5434 Flush 0 New Done Done false
12705 4436 Acked 5435 Read 1 New Done Done false
12706 4437 Acked 5436 Read 6 New Done Done false
12707 4438 Acked 5437 Read 1 New Done Done false
12708 4439 Acked 5438 Write 4 New Done Done false
12709 4440 Acked 5439 Read 8 New Done Done false
12710 4441 Acked 5440 Read 7 New Done Done false
12711 4442 Acked 5441 Read 3 New Done Done false
12712 4443 Acked 5442 Read 8 New Done Done false
12713 4444 Acked 5443 Write 8 New Done Done false
12714 4445 Acked 5444 Read 5 New Done Done false
12715 4446 Acked 5445 Read 7 New Done Done false
12716 4447 Acked 5446 Write 5 New Done Done false
12717 4448 Acked 5447 Write 2 New Done Done false
12718 4449 Acked 5448 Read 4 New Done Done false
12719 4450 Acked 5449 Write 4 New Done Done false
12720 4451 Acked 5450 Write 6 New Done Done false
12721 4452 Acked 5451 Read 6 New Done Done false
12722 4453 Acked 5452 Read 7 New Done Done false
12723 4454 Acked 5453 Read 10 New Done Done false
12724 4455 Acked 5454 Read 2 New Done Done false
12725 4456 Acked 5455 Write 8 New Done Done false
12726 4457 Acked 5456 Write 3 New Done Done false
12727 4458 Acked 5457 Read 8 New Done Done false
12728 4459 Acked 5458 Write 1 New Done Done false
12729 4460 Acked 5459 Read 1 New Done Done false
12730 4461 Acked 5460 Write 5 New Done Done false
12731 4462 Acked 5461 Flush 0 New Done Done false
12732 4463 Acked 5462 Write 10 New Done Done false
12733 4464 Acked 5463 Write 1 New Done Done false
12734 4465 Acked 5464 Write 6 New Done Done false
12735 4466 Acked 5465 Read 5 New Done Done false
12736 4467 Acked 5466 Flush 0 New Done Done false
12737 4468 Acked 5467 Read 4 New Done Done false
12738 4469 Acked 5468 Read 4 New Done Done false
12739 4470 Acked 5469 Flush 0 New Done Done false
12740 4471 Acked 5470 Write 2 New Done Done false
12741 4472 Acked 5471 Read 10 New Done Done false
12742 4473 Acked 5472 Read 7 New Done Done false
12743 4474 Acked 5473 Write 4 New Done Done false
12744 4475 Acked 5474 Write 3 New Done Done false
12745 4476 Acked 5475 Read 6 New Done Done false
12746 4477 Acked 5476 Read 6 New Done Done false
12747 4478 Acked 5477 Write 3 New Done Done false
12748 4479 Acked 5478 Read 3 New Done Done false
12749 4480 Acked 5479 Write 4 New Done Done false
12750 4481 Acked 5480 Flush 0 New Done Done false
12751 4482 Acked 5481 Read 5 New Done Done false
12752 4483 Acked 5482 Flush 0 New Done Done false
12753 4484 Acked 5483 Write 4 New Done Done false
12754 4485 Acked 5484 Read 1 New Done Done false
12755 4486 Acked 5485 Write 6 New Done Done false
12756 4487 Acked 5486 Write 7 New Done Done false
12757 4488 Acked 5487 Write 5 New Done Done false
12758 4489 Acked 5488 Read 2 New Done Done false
12759 4490 Acked 5489 Read 2 New Done Done false
12760 4491 Acked 5490 Write 6 New Done Done false
12761 4492 Acked 5491 Read 5 New Done Done false
12762 4493 Acked 5492 Read 1 New Done Done false
12763 4494 Acked 5493 Write 10 New Done Done false
12764 4495 Acked 5494 Read 8 New Done Done false
12765 4496 Acked 5495 Read 1 New Done Done false
12766 4497 Acked 5496 Write 1 New Done Done false
12767 4498 Acked 5497 Read 6 New Done Done false
12768 4499 Acked 5498 Write 7 New Done Done false
12769 4500 Acked 5499 Read 3 New Done Done false
12770 4501 Acked 5500 Write 5 New Done Done false
12771 4502 Acked 5501 Read 1 New Done Done false
12772 4503 Acked 5502 Write 1 New Done Done false
12773 4504 Acked 5503 Write 10 New Done Done false
12774 4505 Acked 5504 Read 2 New Done Done false
12775 4506 Acked 5505 Write 7 New Done Done false
12776 4507 Acked 5506 Read 1 New Done Done false
12777 4508 Acked 5507 Write 7 New Done Done false
12778 4509 Acked 5508 Write 2 New Done Done false
12779 4510 Acked 5509 Write 4 New Done Done false
12780 4511 Acked 5510 Write 9 New Done Done false
12781 4512 Acked 5511 Read 1 New Done Done false
12782 4513 Acked 5512 Read 1 New Done Done false
12783 4514 Acked 5513 Write 10 New Done Done false
12784 4515 Acked 5514 Flush 0 New Done Done false
12785 4516 Acked 5515 Read 7 New Done Done false
12786 4517 Acked 5516 Read 5 New Done Done false
12787 4518 Acked 5517 Write 5 New Done Done false
12788 4519 Acked 5518 Write 8 New Done Done false
12789 4520 Acked 5519 Read 2 New Done Done false
12790 4521 Acked 5520 Read 10 New Done Done false
12791 4522 Acked 5521 Write 6 New Done Done false
12792 4523 Acked 5522 Flush 0 New Done Done false
12793 4524 Acked 5523 Read 2 New Done Done false
12794 4525 Acked 5524 Flush 0 New Done Done false
12795 4526 Acked 5525 Read 10 New Done Done false
12796 4527 Acked 5526 Read 5 New Done Done false
12797 4528 Acked 5527 Read 2 New Done Done false
12798 4529 Acked 5528 Write 5 New Done Done false
12799 4530 Acked 5529 Read 8 New Done Done false
12800 4531 Acked 5530 Write 5 New Done Done false
12801 4532 Acked 5531 Read 6 New Done Done false
12802 4533 Acked 5532 Read 8 New Done Done false
12803 4534 Acked 5533 Write 7 New Done Done false
12804 4535 Acked 5534 Write 6 New Done Done false
12805 4536 Acked 5535 Flush 0 New Done Done false
12806 STATES DS:0 DS:1 DS:2 TOTAL
12807 New 300 0 0 300
12808 Sent 0 0 0 0
12809 Done 0 300 300 600
12810 Skipped 0 0 0 0
12811 Error 0 0 0 0
12812 Last Flush: 5235 5535 5535
12813 Downstairs last five completed: 5235 5234 5233 5232 5231 5230
12814 Upstairs last five completed: 4536 4535 4534 4533 4532 4531
12815 CLIENT: Up:0 ds:300 act:2
12816 [8810] got start action
12817 [8810] I am going to restart
12818 Starting downstairs at port 8810
12819 Make output file at "/tmp/dsc/downstairs-8810.txt"
12820 [8810][0] reports Starting
12821 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1191)
12822 [8810][0] reports Running
128232023-10-13T04:44:33.162ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 0
128242023-10-13T04:44:33.162ZINFOcrucible: [0] Proc runs for 127.0.0.1:8810 in state Offline
128252023-10-13T04:44:33.162ZINFOcrucible: [0] client re-new 300 jobs since flush 5235 = downstairs
128262023-10-13T04:44:33.163ZINFOcrucible: [0] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
128272023-10-13T04:44:33.163ZINFOcrucible: [0] downstairs client at 127.0.0.1:8810 has UUID 12345678-0000-0000-0000-000000008810
128282023-10-13T04:44:33.163ZINFOcrucible: [0] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }
128292023-10-13T04:44:33.163ZINFOcrucible: Returning client:0 UUID:12345678-0000-0000-0000-000000008810 matches
128302023-10-13T04:44:33.163ZINFOcrucible: [0] send last flush ID to this DS: 5235
128312023-10-13T04:44:33.163ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Offline Active Active
128322023-10-13T04:44:33.164ZINFOcrucible: [0] Replied this last flush ID: 5235
128332023-10-13T04:44:33.164ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Offline Active Active ds_transition to Replay
128342023-10-13T04:44:33.164ZINFOcrucible: [0] Transition from Offline to Replay
128352023-10-13T04:44:33.164ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
128362023-10-13T04:44:33.164ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Replay Active Active ds_transition to Active
128372023-10-13T04:44:33.164ZINFOcrucible: [0] Transition from Replay to Active
128382023-10-13T04:44:33.164ZINFOcrucible: [0] Starts cmd_loop
128392023-10-13T04:44:33.164ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:true
128402023-10-13T04:44:33.164ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
128412023-10-13T04:44:33.164ZINFOcrucible: Set check for repair
128422023-10-13T04:44:34.164ZWARNcrucible: [0] flow control end
128432023-10-13T04:44:34.165ZINFOcrucible: Checking if live repair is needed
128442023-10-13T04:44:34.165ZINFOcrucible: No Live Repair required at this time
12845 ----------------------------------------------------------------
12846 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
12847 Downstairs last five completed: 5535 5534 5533 5532 5531 5530
12848 Upstairs last five completed: 4536 4535 4534 4533 4532 4531
12849 CLIENT: Up:0 ds:0 act:3
12850 Replay: All jobs finished, all DS active.
12851 Main task has work to do, go find it
12852 got dsc Stop(2)
12853 stop 2
12854 Oct 13 04:44:35.027 INFO request completed, latency_us: 471, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 466b8d45-5b8a-4dd8-b32d-11e5633ba930, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
12855 [8830] Got stop action so:false kr:false
12856 Oct 13 04:44:35.028 INFO request completed, latency_us: 270, response_code: 200, uri: /state/cid/2, method: GET, req_id: df8583c3-7775-4f68-b384-b81a647b8cf0, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
128572023-10-13T04:44:35.029ZWARNcrucible: [2] downstairs disconnected
128582023-10-13T04:44:35.029ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
128592023-10-13T04:44:35.029ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8830 closed looper = 2
128602023-10-13T04:44:35.029ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
128612023-10-13T04:44:35.029ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
128622023-10-13T04:44:35.029ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
128632023-10-13T04:44:35.029ZWARNcrucible: [2] pm_task rx.recv() is None
12864 [8830] Exited with: ExitStatus(unix_wait_status(9))
12865 [8830][2] reports Exit
128662023-10-13T04:44:38.032ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
12867 Oct 13 04:44:39.030 INFO request completed, latency_us: 250, response_code: 200, uri: /state/cid/2, method: GET, req_id: f2a23543-ca39-4f1d-bd70-e9bc84f6d688, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
12868 001/300 Write block 2553 len 40960 data: 4 3 3 4 4 4 3 3 3 2
12869 002/300 Flush
12870 003/300 Read block 1411 len 36864
12871 004/300 Write block 1942 len 40960 data: 5 4 2 4 4 4 4 4 4 4
12872 005/300 Flush
12873 006/300 Write block 3219 len 20480 data: 3 4 3 2 2
12874 007/300 Write block 353 len 24576 data: 5 5 5 5 4 3
12875 008/300 Read block 3511 len 40960
12876 009/300 Write block 3352 len 20480 data: 7 6 6 5 6
12877 010/300 Read block 1095 len 20480
12878 011/300 Read block 1036 len 20480
12879 012/300 Write block 1637 len 24576 data: 4 4 4 5 3 4
12880 013/300 Read block 3597 len 28672
12881 014/300 Write block 4646 len 8192 data: 5 5
12882 015/300 Flush
12883 016/300 Write block 1586 len 24576 data: 5 6 4 4 3 3
12884 017/300 Write block 4241 len 36864 data: 2 2 3 3 4 5 6 6 6
12885 018/300 Read block 492 len 12288
12886 019/300 Read block 4957 len 28672
12887 020/300 Write block 1363 len 8192 data: 3 3
12888 021/300 Write block 2821 len 32768 data: 2 2 2 2 2 3 3 4
12889 022/300 Read block 1797 len 12288
12890 023/300 Write block 4933 len 16384 data: 2 2 2 2
12891 024/300 Write block 3581 len 36864 data: 5 7 6 6 6 5 5 4 4
12892 025/300 Write block 4889 len 20480 data: 5 5 4 4 4
12893 026/300 Flush
12894 027/300 Read block 3552 len 36864
12895 028/300 Read block 3578 len 8192
12896 029/300 Write block 3272 len 4096 data: 2
12897 030/300 Write block 4491 len 8192 data: 2 2
12898 031/300 Write block 3997 len 12288 data: 2 2 2
12899 032/300 Read block 548 len 20480
12900 033/300 Flush
12901 034/300 Write block 2560 len 28672 data: 4 4 3 2 3 3 5
12902 035/300 Write block 2747 len 32768 data: 4 6 5 4 4 4 5 3
12903 036/300 Write block 2047 len 4096 data: 3
12904 037/300 Read block 2279 len 36864
12905 038/300 Flush
12906 039/300 Write block 2015 len 12288 data: 3 4 4
12907 040/300 Read block 4936 len 4096
12908 041/300 Read block 1143 len 8192
12909 042/300 Read block 2234 len 12288
12910 043/300 Write block 4445 len 12288 data: 4 4 5
12911 044/300 Write block 521 len 40960 data: 9 10 10 8 7 6 5 5 4 3
12912 045/300 Read block 955 len 24576
12913 046/300 Read block 907 len 12288
12914 047/300 Read block 1215 len 32768
12915 048/300 Read block 2499 len 40960
12916 049/300 Read block 4859 len 12288
12917 050/300 Read block 2101 len 40960
12918 051/300 Write block 4307 len 16384 data: 3 3 3 2
12919 052/300 Write block 994 len 16384 data: 6 6 6 4
12920 053/300 Flush
12921 054/300 Read block 4612 len 32768
12922 055/300 Read block 616 len 24576
12923 056/300 Read block 4768 len 24576
12924 057/300 Write block 693 len 36864 data: 6 5 7 7 6 6 5 5 4
12925 058/300 Read block 4006 len 20480
12926 059/300 Read block 1139 len 4096
12927 060/300 Flush
12928 061/300 Read block 2051 len 16384
12929 062/300 Read block 2451 len 4096
12930 063/300 Flush
12931 064/300 Read block 2732 len 28672
12932 065/300 Read block 4159 len 32768
12933 066/300 Flush
12934 067/300 Read block 2980 len 40960
12935 068/300 Flush
12936 069/300 Write block 4784 len 16384 data: 2 3 3 3
12937 070/300 Read block 2685 len 40960
12938 071/300 Read block 1441 len 24576
12939 072/300 Read block 3631 len 40960
12940 073/300 Write block 383 len 28672 data: 4 4 3 3 3 3 3
12941 074/300 Read block 683 len 24576
12942 075/300 Write block 412 len 32768 data: 3 3 3 3 3 3 4 4
12943 076/300 Write block 2022 len 20480 data: 4 4 4 4 3
12944 077/300 Read block 1470 len 12288
12945 078/300 Write block 712 len 40960 data: 4 2 3 4 4 4 4 3 4 4
12946 079/300 Flush
12947 080/300 Read block 3212 len 32768
12948 081/300 Read block 3870 len 32768
12949 082/300 Flush
12950 083/300 Write block 3189 len 36864 data: 7 6 5 4 2 2 3 4 4
12951 084/300 Write block 4404 len 4096 data: 4
12952 085/300 Write block 1976 len 4096 data: 5
12953 086/300 Write block 4550 len 16384 data: 3 4 3 3
12954 087/300 Read block 975 len 28672
12955 088/300 Write block 2033 len 32768 data: 4 4 3 3 4 3 3 4
12956 089/300 Read block 2839 len 20480
12957 090/300 Write block 1974 len 32768 data: 4 4 6 4 4 4 4 4
12958 091/300 Read block 4352 len 12288
12959 092/300 Write block 2672 len 12288 data: 2 2 2
12960 093/300 Read block 70 len 4096
12961 094/300 Read block 3479 len 16384
12962 095/300 Write block 1034 len 24576 data: 3 5 6 5 5 4
12963 096/300 Read block 3788 len 32768
12964 097/300 Read block 1456 len 40960
12965 098/300 Read block 2914 len 24576
12966 099/300 Read block 3253 len 32768
12967 100/300 Write block 3298 len 12288 data: 2 2 2
12968 101/300 Read block 4179 len 28672
12969 102/300 Write block 2102 len 28672 data: 4 4 4 4 5 4 4
12970 103/300 Flush
12971 104/300 Read block 866 len 32768
12972 105/300 Read block 3349 len 40960
12973 106/300 Read block 1250 len 32768
12974 107/300 Write block 3221 len 40960 data: 4 3 3 2 2 2 2 2 2 2
12975 108/300 Read block 1467 len 36864
12976 109/300 Flush
12977 110/300 Write block 126 len 40960 data: 2 2 2 2 2 2 2 2 2 2
12978 111/300 Read block 4722 len 16384
12979 112/300 Write block 4020 len 40960 data: 5 6 7 7 7 6 5 6 4 4
12980 113/300 Read block 1957 len 16384
12981 114/300 Write block 2056 len 16384 data: 3 3 3 2
12982 115/300 Read block 1294 len 40960
12983 116/300 Write block 4871 len 32768 data: 3 3 3 3 5 4 3 2
12984 117/300 Write block 4110 len 16384 data: 4 3 3 2
12985 118/300 Read block 4518 len 4096
12986 119/300 Read block 2861 len 24576
12987 120/300 Write block 3246 len 32768 data: 3 2 3 3 4 3 3 3
12988 121/300 Write block 470 len 40960 data: 2 3 3 4 4 4 4 4 4 3
12989 122/300 Write block 3705 len 40960 data: 5 6 5 5 6 5 6 6 7 7
12990 123/300 Write block 437 len 4096 data: 5
12991 124/300 Write block 557 len 24576 data: 3 3 3 3 4 3
12992 125/300 Read block 3436 len 36864
12993 126/300 Write block 2569 len 8192 data: 7 8
12994 127/300 Read block 3249 len 36864
12995 128/300 Write block 4208 len 28672 data: 4 4 5 6 6 4 5
12996 129/300 Write block 2653 len 28672 data: 3 4 4 5 6 3 3
12997 130/300 Read block 3179 len 16384
12998 131/300 Read block 3773 len 8192
12999 132/300 Read block 2455 len 32768
13000 133/300 Flush
13001 134/300 Flush
13002 135/300 Write block 3325 len 32768 data: 5 4 2 2 2 2 2 2
13003 136/300 Read block 3246 len 24576
13004 137/300 Write block 1696 len 24576 data: 4 5 5 5 5 5
13005 138/300 Read block 562 len 24576
13006 139/300 Flush
13007 140/300 Write block 716 len 8192 data: 5 5
13008 141/300 Write block 213 len 28672 data: 5 6 6 6 8 8 7
13009 142/300 Read block 4268 len 24576
13010 143/300 Read block 4803 len 40960
13011 144/300 Write block 4451 len 4096 data: 4
13012 145/300 Write block 2127 len 12288 data: 5 5 4
13013 146/300 Flush
13014 147/300 Write block 140 len 24576 data: 2 2 2 3 3 3
13015 148/300 Write block 3533 len 24576 data: 8 7 7 6 3 3
13016 149/300 Read block 639 len 20480
13017 150/300 Write block 745 len 24576 data: 3 3 4 3 5 4
13018 151/300 Read block 518 len 40960
13019 152/300 Write block 4015 len 20480 data: 3 3 3 4 5
13020 153/300 Read block 2951 len 36864
13021 154/300 Read block 4465 len 16384
13022 155/300 Write block 2846 len 4096 data: 5
13023 156/300 Read block 734 len 4096
13024 157/300 Read block 1945 len 20480
13025 158/300 Write block 683 len 24576 data: 3 3 2 2 2 2
13026 159/300 Read block 1628 len 36864
13027 160/300 Write block 1790 len 40960 data: 4 4 4 5 4 4 3 3 2 2
13028 161/300 Read block 842 len 36864
13029 162/300 Read block 1963 len 36864
13030 163/300 Read block 406 len 40960
13031 164/300 Write block 4921 len 28672 data: 4 3 4 4 3 3 3
13032 165/300 Write block 559 len 20480 data: 4 4 5 4 3
13033 166/300 Write block 4638 len 4096 data: 3
13034 167/300 Read block 4550 len 28672
13035 168/300 Read block 1271 len 8192
13036 169/300 Read block 136 len 8192
13037 170/300 Write block 3690 len 40960 data: 3 2 2 3 3 3 3 3 4 4
13038 171/300 Read block 325 len 12288
13039 172/300 Read block 4580 len 24576
13040 173/300 Read block 162 len 24576
13041 174/300 Write block 212 len 36864 data: 8 6 7 7 7 9 9 8 3
13042 175/300 Read block 1893 len 4096
13043 176/300 Read block 2358 len 24576
13044 177/300 Read block 1162 len 40960
13045 178/300 Read block 4641 len 16384
13046 179/300 Read block 4096 len 20480
13047 180/300 Write block 2051 len 4096 data: 4
13048 181/300 Flush
13049 182/300 Write block 147 len 36864 data: 3 3 3 3 3 3 3 3 3
13050 183/300 Write block 1771 len 40960 data: 6 6 4 4 6 6 6 4 4 3
13051 184/300 Write block 1167 len 16384 data: 4 3 3 3
13052 185/300 Read block 11 len 32768
13053 186/300 Write block 758 len 36864 data: 5 4 4 3 3 3 2 2 2
13054 187/300 Read block 871 len 4096
13055 188/300 Write block 4822 len 32768 data: 6 5 5 7 6 6 6 6
13056 189/300 Read block 4832 len 4096
13057 190/300 Write block 379 len 4096 data: 3
13058 191/300 Write block 4817 len 32768 data: 4 4 5 4 5 7 6 6
13059 192/300 Flush
13060 193/300 Write block 430 len 28672 data: 2 2 2 4 4 4 5
13061 194/300 Read block 2847 len 32768
13062 195/300 Read block 3929 len 8192
13063 196/300 Read block 2605 len 36864
13064 197/300 Read block 1969 len 32768
13065 198/300 Read block 3408 len 36864
13066 199/300 Read block 4102 len 20480
13067 200/300 Write block 1018 len 4096 data: 4
13068 201/300 Read block 2782 len 12288
13069 202/300 Write block 487 len 8192 data: 5 5
13070 203/300 Write block 269 len 20480 data: 3 3 2 2 2
13071 204/300 Read block 491 len 32768
13072 205/300 Write block 934 len 8192 data: 2 2
13073 206/300 Read block 2429 len 4096
13074 207/300 Read block 131 len 24576
13075 208/300 Read block 3772 len 8192
13076 209/300 Flush
13077 210/300 Write block 863 len 28672 data: 4 4 4 5 5 5 3
13078 211/300 Write block 4240 len 28672 data: 2 3 3 4 4 5 6
13079 212/300 Write block 2062 len 16384 data: 3 3 3 2
13080 213/300 Read block 299 len 16384
13081 214/300 Read block 4365 len 36864
13082 215/300 Read block 2991 len 28672
13083 216/300 Read block 4894 len 32768
13084 217/300 Flush
13085 218/300 Read block 4633 len 32768
13086 219/300 Read block 976 len 24576
13087 220/300 Read block 4762 len 4096
13088 221/300 Write block 3112 len 32768 data: 3 3 2 3 4 4 4 4
13089 222/300 Write block 1285 len 20480 data: 6 6 6 8 9
13090 223/300 Flush
13091 224/300 Write block 973 len 40960 data: 5 5 6 6 5 4 4 3 4 6
13092 225/300 Read block 3200 len 16384
13093 226/300 Read block 4816 len 28672
13094 227/300 Read block 1179 len 20480
13095 228/300 Read block 4177 len 16384
13096 229/300 Write block 4166 len 8192 data: 4 6
13097 230/300 Flush
13098 231/300 Write block 1287 len 32768 data: 7 9 10 9 8 7 6 4
13099 232/300 Read block 3511 len 24576
13100 233/300 Write block 2219 len 32768 data: 5 5 6 6 5 6 6 4
13101 234/300 Read block 2 len 40960
13102 235/300 Read block 2470 len 24576
13103 236/300 Flush
13104 237/300 Flush
13105 238/300 Write block 4226 len 24576 data: 2 2 2 2 2 2
13106 239/300 Read block 3526 len 20480
13107 240/300 Write block 151 len 40960 data: 4 4 4 4 4 3 3 3 4 5
13108 241/300 Read block 4790 len 8192
13109 242/300 Write block 2199 len 20480 data: 4 3 3 3 2
13110 243/300 Write block 2210 len 36864 data: 2 2 3 3 3 3 3 4 4
13111 244/300 Write block 1558 len 40960 data: 4 5 5 2 2 3 3 3 3 2
13112 245/300 Flush
13113 246/300 Read block 4515 len 40960
13114 247/300 Read block 1144 len 32768
13115 248/300 Read block 3681 len 24576
13116 249/300 Read block 1941 len 4096
13117 250/300 Write block 4396 len 4096 data: 3
13118 251/300 Read block 1976 len 20480
13119 252/300 Flush
13120 253/300 Write block 3313 len 36864 data: 5 5 6 6 6 5 6 5 6
13121 254/300 Write block 2835 len 32768 data: 3 4 4 4 4 4 4 5
13122 255/300 Write block 3912 len 24576 data: 3 2 3 3 3 3
13123 256/300 Read block 3207 len 36864
13124 257/300 Read block 2379 len 4096
13125 258/300 Read block 1270 len 16384
13126 259/300 Write block 2879 len 32768 data: 4 3 4 4 4 3 3 3
13127 260/300 Read block 12 len 8192
13128 261/300 Read block 422 len 4096
13129 262/300 Read block 3003 len 24576
13130 263/300 Write block 367 len 4096 data: 2
13131 264/300 Read block 4031 len 20480
13132 265/300 Read block 4582 len 40960
13133 266/300 Write block 3713 len 12288 data: 8 8 7
13134 267/300 Write block 489 len 16384 data: 5 5 5 6
13135 268/300 Write block 2184 len 40960 data: 5 5 5 4 3 2 3 3 3 3
13136 269/300 Read block 4202 len 24576
13137 270/300 Flush
13138 271/300 Write block 1855 len 24576 data: 3 4 5 5 6 6
13139 272/300 Read block 2984 len 28672
13140 273/300 Read block 1967 len 28672
13141 274/300 Read block 2416 len 20480
13142 275/300 Write block 3042 len 12288 data: 8 8 4
13143 276/300 Read block 4424 len 16384
13144 277/300 Write block 778 len 36864 data: 3 2 3 3 3 4 5 5 5
13145 278/300 Read block 4621 len 12288
13146 279/300 Write block 985 len 8192 data: 6 7
13147 280/300 Read block 1198 len 8192
13148 281/300 Flush
13149 282/300 Flush
13150 283/300 Write block 3537 len 24576 data: 4 4 2 3 3 3
13151 284/300 Read block 688 len 24576
13152 285/300 Write block 2883 len 28672 data: 5 4 4 4 3 4 3
13153 286/300 Write block 744 len 8192 data: 3 4
13154 287/300 Flush
13155 288/300 Read block 46 len 28672
13156 289/300 Read block 3525 len 40960
13157 290/300 Read block 1386 len 36864
13158 291/300 Flush
13159 292/300 Write block 2395 len 20480 data: 3 3 4 4 4
13160 293/300 Read block 2950 len 4096
13161 294/300 Read block 4904 len 36864
13162 295/300 Read block 3490 len 20480
13163 296/300 Write block 29 len 24576 data: 3 3 3 3 3 3
13164 297/300 Write block 2941 len 16384 data: 5 3 3 3
13165 298/300 Write block 3075 len 32768 data: 2 2 3 4 5 4 4 3
13166 299/300 Read block 1231 len 32768
13167 300/300 Read block 1345 len 36864
13168 Main task has work to do, go find it
13169 got dsc Start(2)
13170 start 2
13171 Oct 13 04:44:39.991 INFO request completed, latency_us: 285, response_code: 204, uri: /start/cid/2, method: POST, req_id: baad6e0a-d65a-4962-9f04-ad5b3732b9b2, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
13172 Replay: started 2, returned:Ok(())
13173 ----------------------------------------------------------------
13174 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
13175 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
13176 4537 Acked 5536 Write 10 Done Done New false
13177 4538 Acked 5537 Flush 0 Done Done New false
13178 4539 Acked 5538 Read 9 Done Done New false
13179 4540 Acked 5539 Write 10 Done Done New false
13180 4541 Acked 5540 Flush 0 Done Done New false
13181 4542 Acked 5541 Write 5 Done Done New false
13182 4543 Acked 5542 Write 6 Done Done New false
13183 4544 Acked 5543 Read 10 Done Done New false
13184 4545 Acked 5544 Write 5 Done Done New false
13185 4546 Acked 5545 Read 5 Done Done New false
13186 4547 Acked 5546 Read 5 Done Done New false
13187 4548 Acked 5547 Write 6 Done Done New false
13188 4549 Acked 5548 Read 7 Done Done New false
13189 4550 Acked 5549 Write 2 Done Done New false
13190 4551 Acked 5550 Flush 0 Done Done New false
13191 4552 Acked 5551 Write 6 Done Done New false
13192 4553 Acked 5552 Write 9 Done Done New false
13193 4554 Acked 5553 Read 3 Done Done New false
13194 4555 Acked 5554 Read 7 Done Done New false
13195 4556 Acked 5555 Write 2 Done Done New false
13196 4557 Acked 5556 Write 8 Done Done New false
13197 4558 Acked 5557 Read 3 Done Done New false
13198 4559 Acked 5558 Write 4 Done Done New false
13199 4560 Acked 5559 Write 9 Done Done New false
13200 4561 Acked 5560 Write 5 Done Done New false
13201 4562 Acked 5561 Flush 0 Done Done New false
13202 4563 Acked 5562 Read 9 Done Done New false
13203 4564 Acked 5563 Read 2 Done Done New false
13204 4565 Acked 5564 Write 1 Done Done New false
13205 4566 Acked 5565 Write 2 Done Done New false
13206 4567 Acked 5566 Write 3 Done Done New false
13207 4568 Acked 5567 Read 5 Done Done New false
13208 4569 Acked 5568 Flush 0 Done Done New false
13209 4570 Acked 5569 Write 7 Done Done New false
13210 4571 Acked 5570 Write 8 Done Done New false
13211 4572 Acked 5571 Write 1 Done Done New false
13212 4573 Acked 5572 Read 9 Done Done New false
13213 4574 Acked 5573 Flush 0 Done Done New false
13214 4575 Acked 5574 Write 3 Done Done New false
13215 4576 Acked 5575 Read 1 Done Done New false
13216 4577 Acked 5576 Read 2 Done Done New false
13217 4578 Acked 5577 Read 3 Done Done New false
13218 4579 Acked 5578 Write 3 Done Done New false
13219 4580 Acked 5579 Write 10 Done Done New false
13220 4581 Acked 5580 Read 6 Done Done New false
13221 4582 Acked 5581 Read 3 Done Done New false
13222 4583 Acked 5582 Read 8 Done Done New false
13223 4584 Acked 5583 Read 10 Done Done New false
13224 4585 Acked 5584 Read 3 Done Done New false
13225 4586 Acked 5585 Read 10 Done Done New false
13226 4587 Acked 5586 Write 4 Done Done New false
13227 4588 Acked 5587 Write 4 Done Done New false
13228 4589 Acked 5588 Flush 0 Done Done New false
13229 4590 Acked 5589 Read 8 Done Done New false
13230 4591 Acked 5590 Read 6 Done Done New false
13231 4592 Acked 5591 Read 6 Done Done New false
13232 4593 Acked 5592 Write 9 Done Done New false
13233 4594 Acked 5593 Read 5 Done Done New false
13234 4595 Acked 5594 Read 1 Done Done New false
13235 4596 Acked 5595 Flush 0 Done Done New false
13236 4597 Acked 5596 Read 4 Done Done New false
13237 4598 Acked 5597 Read 1 Done Done New false
13238 4599 Acked 5598 Flush 0 Done Done New false
13239 4600 Acked 5599 Read 7 Done Done New false
13240 4601 Acked 5600 Read 8 Done Done New false
13241 4602 Acked 5601 Flush 0 Done Done New false
13242 4603 Acked 5602 Read 10 Done Done New false
13243 4604 Acked 5603 Flush 0 Done Done New false
13244 4605 Acked 5604 Write 4 Done Done New false
13245 4606 Acked 5605 Read 10 Done Done New false
13246 4607 Acked 5606 Read 6 Done Done New false
13247 4608 Acked 5607 Read 10 Done Done New false
13248 4609 Acked 5608 Write 7 Done Done New false
13249 4610 Acked 5609 Read 6 Done Done New false
13250 4611 Acked 5610 Write 8 Done Done New false
13251 4612 Acked 5611 Write 5 Done Done New false
13252 4613 Acked 5612 Read 3 Done Done New false
13253 4614 Acked 5613 Write 10 Done Done New false
13254 4615 Acked 5614 Flush 0 Done Done New false
13255 4616 Acked 5615 Read 8 Done Done New false
13256 4617 Acked 5616 Read 8 Done Done New false
13257 4618 Acked 5617 Flush 0 Done Done New false
13258 4619 Acked 5618 Write 9 Done Done New false
13259 4620 Acked 5619 Write 1 Done Done New false
13260 4621 Acked 5620 Write 1 Done Done New false
13261 4622 Acked 5621 Write 4 Done Done New false
13262 4623 Acked 5622 Read 7 Done Done New false
13263 4624 Acked 5623 Write 8 Done Done New false
13264 4625 Acked 5624 Read 5 Done Done New false
13265 4626 Acked 5625 Write 8 Done Done New false
13266 4627 Acked 5626 Read 3 Done Done New false
13267 4628 Acked 5627 Write 3 Done Done New false
13268 4629 Acked 5628 Read 1 Done Done New false
13269 4630 Acked 5629 Read 4 Done Done New false
13270 4631 Acked 5630 Write 6 Done Done New false
13271 4632 Acked 5631 Read 8 Done Done New false
13272 4633 Acked 5632 Read 10 Done Done New false
13273 4634 Acked 5633 Read 6 Done Done New false
13274 4635 Acked 5634 Read 8 Done Done New false
13275 4636 Acked 5635 Write 3 Done Done New false
13276 4637 Acked 5636 Read 7 Done Done New false
13277 4638 Acked 5637 Write 7 Done Done New false
13278 4639 Acked 5638 Flush 0 Done Done New false
13279 4640 Acked 5639 Read 8 Done Done New false
13280 4641 Acked 5640 Read 10 Done Done New false
13281 4642 Acked 5641 Read 8 Done Done New false
13282 4643 Acked 5642 Write 10 Done Done New false
13283 4644 Acked 5643 Read 9 Done Done New false
13284 4645 Acked 5644 Flush 0 Done Done New false
13285 4646 Acked 5645 Write 10 Done Done New false
13286 4647 Acked 5646 Read 4 Done Done New false
13287 4648 Acked 5647 Write 10 Done Done New false
13288 4649 Acked 5648 Read 4 Done Done New false
13289 4650 Acked 5649 Write 4 Done Done New false
13290 4651 Acked 5650 Read 10 Done Done New false
13291 4652 Acked 5651 Write 8 Done Done New false
13292 4653 Acked 5652 Write 4 Done Done New false
13293 4654 Acked 5653 Read 1 Done Done New false
13294 4655 Acked 5654 Read 6 Done Done New false
13295 4656 Acked 5655 Write 8 Done Done New false
13296 4657 Acked 5656 Write 10 Done Done New false
13297 4658 Acked 5657 Write 10 Done Done New false
13298 4659 Acked 5658 Write 1 Done Done New false
13299 4660 Acked 5659 Write 6 Done Done New false
13300 4661 Acked 5660 Read 9 Done Done New false
13301 4662 Acked 5661 Write 2 Done Done New false
13302 4663 Acked 5662 Read 9 Done Done New false
13303 4664 Acked 5663 Write 7 Done Done New false
13304 4665 Acked 5664 Write 7 Done Done New false
13305 4666 Acked 5665 Read 4 Done Done New false
13306 4667 Acked 5666 Read 2 Done Done New false
13307 4668 Acked 5667 Read 8 Done Done New false
13308 4669 Acked 5668 Flush 0 Done Done New false
13309 4670 Acked 5669 Flush 0 Done Done New false
13310 4671 Acked 5670 Write 8 Done Done New false
13311 4672 Acked 5671 Read 6 Done Done New false
13312 4673 Acked 5672 Write 6 Done Done New false
13313 4674 Acked 5673 Read 6 Done Done New false
13314 4675 Acked 5674 Flush 0 Done Done New false
13315 4676 Acked 5675 Write 2 Done Done New false
13316 4677 Acked 5676 Write 7 Done Done New false
13317 4678 Acked 5677 Read 6 Done Done New false
13318 4679 Acked 5678 Read 10 Done Done New false
13319 4680 Acked 5679 Write 1 Done Done New false
13320 4681 Acked 5680 Write 3 Done Done New false
13321 4682 Acked 5681 Flush 0 Done Done New false
13322 4683 Acked 5682 Write 6 Done Done New false
13323 4684 Acked 5683 Write 6 Done Done New false
13324 4685 Acked 5684 Read 5 Done Done New false
13325 4686 Acked 5685 Write 6 Done Done New false
13326 4687 Acked 5686 Read 10 Done Done New false
13327 4688 Acked 5687 Write 5 Done Done New false
13328 4689 Acked 5688 Read 9 Done Done New false
13329 4690 Acked 5689 Read 4 Done Done New false
13330 4691 Acked 5690 Write 1 Done Done New false
13331 4692 Acked 5691 Read 1 Done Done New false
13332 4693 Acked 5692 Read 5 Done Done New false
13333 4694 Acked 5693 Write 6 Done Done New false
13334 4695 Acked 5694 Read 9 Done Done New false
13335 4696 Acked 5695 Write 10 Done Done New false
13336 4697 Acked 5696 Read 9 Done Done New false
13337 4698 Acked 5697 Read 9 Done Done New false
13338 4699 Acked 5698 Read 10 Done Done New false
13339 4700 Acked 5699 Write 7 Done Done New false
13340 4701 Acked 5700 Write 5 Done Done New false
13341 4702 Acked 5701 Write 1 Done Done New false
13342 4703 Acked 5702 Read 7 Done Done New false
13343 4704 Acked 5703 Read 2 Done Done New false
13344 4705 Acked 5704 Read 2 Done Done New false
13345 4706 Acked 5705 Write 10 Done Done New false
13346 4707 Acked 5706 Read 3 Done Done New false
13347 4708 Acked 5707 Read 6 Done Done New false
13348 4709 Acked 5708 Read 6 Done Done New false
13349 4710 Acked 5709 Write 9 Done Done New false
13350 4711 Acked 5710 Read 1 Done Done New false
13351 4712 Acked 5711 Read 6 Done Done New false
13352 4713 Acked 5712 Read 10 Done Done New false
13353 4714 Acked 5713 Read 4 Done Done New false
13354 4715 Acked 5714 Read 5 Done Done New false
13355 4716 Acked 5715 Write 1 Done Done New false
13356 4717 Acked 5716 Flush 0 Done Done New false
13357 4718 Acked 5717 Write 9 Done Done New false
13358 4719 Acked 5718 Write 10 Done Done New false
13359 4720 Acked 5719 Flush 0 Done Done New false
13360 4721 Acked 5720 Write 4 Done Done New false
13361 4722 Acked 5721 Read 8 Done Done New false
13362 4723 Acked 5722 Write 9 Done Done New false
13363 4724 Acked 5723 Read 1 Done Done New false
13364 4725 Acked 5724 Write 8 Done Done New false
13365 4726 Acked 5725 Read 1 Done Done New false
13366 4727 Acked 5726 Write 1 Done Done New false
13367 4728 Acked 5727 Write 8 Done Done New false
13368 4729 Acked 5728 Flush 0 Done Done New false
13369 4730 Acked 5729 Write 7 Done Done New false
13370 4731 Acked 5730 Read 8 Done Done New false
13371 4732 Acked 5731 Read 2 Done Done New false
13372 4733 Acked 5732 Read 9 Done Done New false
13373 4734 Acked 5733 Read 8 Done Done New false
13374 4735 Acked 5734 Read 9 Done Done New false
13375 4736 Acked 5735 Read 5 Done Done New false
13376 4737 Acked 5736 Write 1 Done Done New false
13377 4738 Acked 5737 Read 3 Done Done New false
13378 4739 Acked 5738 Write 2 Done Done New false
13379 4740 Acked 5739 Write 5 Done Done New false
13380 4741 Acked 5740 Read 8 Done Done New false
13381 4742 Acked 5741 Write 2 Done Done New false
13382 4743 Acked 5742 Read 1 Done Done New false
13383 4744 Acked 5743 Read 6 Done Done New false
13384 4745 Acked 5744 Read 2 Done Done New false
13385 4746 Acked 5745 Flush 0 Done Done New false
13386 4747 Acked 5746 Write 7 Done Done New false
13387 4748 Acked 5747 Write 7 Done Done New false
13388 4749 Acked 5748 Write 4 Done Done New false
13389 4750 Acked 5749 Read 4 Done Done New false
13390 4751 Acked 5750 Read 9 Done Done New false
13391 4752 Acked 5751 Read 7 Done Done New false
13392 4753 Acked 5752 Read 8 Done Done New false
13393 4754 Acked 5753 Flush 0 Done Done New false
13394 4755 Acked 5754 Read 8 Done Done New false
13395 4756 Acked 5755 Read 6 Done Done New false
13396 4757 Acked 5756 Read 1 Done Done New false
13397 4758 Acked 5757 Write 8 Done Done New false
13398 4759 Acked 5758 Write 5 Done Done New false
13399 4760 Acked 5759 Flush 0 Done Done New false
13400 4761 Acked 5760 Write 10 Done Done New false
13401 4762 Acked 5761 Read 4 Done Done New false
13402 4763 Acked 5762 Read 7 Done Done New false
13403 4764 Acked 5763 Read 5 Done Done New false
13404 4765 Acked 5764 Read 4 Done Done New false
13405 4766 Acked 5765 Write 2 Done Done New false
13406 4767 Acked 5766 Flush 0 Done Done New false
13407 4768 Acked 5767 Write 8 Done Done New false
13408 4769 Acked 5768 Read 6 Done Done New false
13409 4770 Acked 5769 Write 8 Done Done New false
13410 4771 Acked 5770 Read 10 Done Done New false
13411 4772 Acked 5771 Read 6 Done Done New false
13412 4773 Acked 5772 Flush 0 Done Done New false
13413 4774 Acked 5773 Flush 0 Done Done New false
13414 4775 Acked 5774 Write 6 Done Done New false
13415 4776 Acked 5775 Read 5 Done Done New false
13416 4777 Acked 5776 Write 10 Done Done New false
13417 4778 Acked 5777 Read 2 Done Done New false
13418 4779 Acked 5778 Write 5 Done Done New false
13419 4780 Acked 5779 Write 9 Done Done New false
13420 4781 Acked 5780 Write 10 Done Done New false
13421 4782 Acked 5781 Flush 0 Done Done New false
13422 4783 Acked 5782 Read 10 Done Done New false
13423 4784 Acked 5783 Read 8 Done Done New false
13424 4785 Acked 5784 Read 6 Done Done New false
13425 4786 Acked 5785 Read 1 Done Done New false
13426 4787 Acked 5786 Write 1 Done Done New false
13427 4788 Acked 5787 Read 5 Done Done New false
13428 4789 Acked 5788 Flush 0 Done Done New false
13429 4790 Acked 5789 Write 9 Done Done New false
13430 4791 Acked 5790 Write 8 Done Done New false
13431 4792 Acked 5791 Write 6 Done Done New false
13432 4793 Acked 5792 Read 9 Done Done New false
13433 4794 Acked 5793 Read 1 Done Done New false
13434 4795 Acked 5794 Read 4 Done Done New false
13435 4796 Acked 5795 Write 8 Done Done New false
13436 4797 Acked 5796 Read 2 Done Done New false
13437 4798 Acked 5797 Read 1 Done Done New false
13438 4799 Acked 5798 Read 6 Done Done New false
13439 4800 Acked 5799 Write 1 Done Done New false
13440 4801 Acked 5800 Read 5 Done Done New false
13441 4802 Acked 5801 Read 10 Done Done New false
13442 4803 Acked 5802 Write 3 Done Done New false
13443 4804 Acked 5803 Write 4 Done Done New false
13444 4805 Acked 5804 Write 10 Done Done New false
13445 4806 Acked 5805 Read 6 Done Done New false
13446 4807 Acked 5806 Flush 0 Done Done New false
13447 4808 Acked 5807 Write 6 Done Done New false
13448 4809 Acked 5808 Read 7 Done Done New false
13449 4810 Acked 5809 Read 7 Done Done New false
13450 4811 Acked 5810 Read 5 Done Done New false
13451 4812 Acked 5811 Write 3 Done Done New false
13452 4813 Acked 5812 Read 4 Done Done New false
13453 4814 Acked 5813 Write 9 Done Done New false
13454 4815 Acked 5814 Read 3 Done Done New false
13455 4816 Acked 5815 Write 2 Done Done New false
13456 4817 Acked 5816 Read 2 Done Done New false
13457 4818 Acked 5817 Flush 0 Done Done New false
13458 4819 Acked 5818 Flush 0 Done Done New false
13459 4820 Acked 5819 Write 6 Done Done New false
13460 4821 Acked 5820 Read 6 Done Done New false
13461 4822 Acked 5821 Write 7 Done Done New false
13462 4823 Acked 5822 Write 2 Done Done New false
13463 4824 Acked 5823 Flush 0 Done Done New false
13464 4825 Acked 5824 Read 7 Done Done New false
13465 4826 Acked 5825 Read 10 Done Done New false
13466 4827 Acked 5826 Read 9 Done Done New false
13467 4828 Acked 5827 Flush 0 Done Done New false
13468 4829 Acked 5828 Write 5 Done Done New false
13469 4830 Acked 5829 Read 1 Done Done New false
13470 4831 Acked 5830 Read 9 Done Done New false
13471 4832 Acked 5831 Read 5 Done Done New false
13472 4833 Acked 5832 Write 6 Done Done New false
13473 4834 Acked 5833 Write 4 Done Done New false
13474 4835 Acked 5834 Write 8 Done Done New false
13475 4836 Acked 5835 Read 8 Done Done New false
13476 4837 Acked 5836 Read 9 Done Done New false
13477 STATES DS:0 DS:1 DS:2 TOTAL
13478 New 0 0 301 301
13479 Sent 0 0 0 0
13480 Done 301 301 0 602
13481 Skipped 0 0 0 0
13482 Error 0 0 0 0
13483 Last Flush: 5827 5827 5535
13484 Downstairs last five completed: 5535 5534 5533 5532 5531 5530
13485 Upstairs last five completed: 4837 4836 4835 4834 4833 4832
13486 CLIENT: Up:0 ds:301 act:2
13487 [8830] got start action
13488 [8830] I am going to restart
13489 Starting downstairs at port 8830
13490 Make output file at "/tmp/dsc/downstairs-8830.txt"
13491 [8830][2] reports Starting
13492 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1192)
13493 [8830][2] reports Running
13494 ----------------------------------------------------------------
13495 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
13496 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
13497 4537 Acked 5536 Write 10 Done Done New false
13498 4538 Acked 5537 Flush 0 Done Done New false
13499 4539 Acked 5538 Read 9 Done Done New false
13500 4540 Acked 5539 Write 10 Done Done New false
13501 4541 Acked 5540 Flush 0 Done Done New false
13502 4542 Acked 5541 Write 5 Done Done New false
13503 4543 Acked 5542 Write 6 Done Done New false
13504 4544 Acked 5543 Read 10 Done Done New false
13505 4545 Acked 5544 Write 5 Done Done New false
13506 4546 Acked 5545 Read 5 Done Done New false
13507 4547 Acked 5546 Read 5 Done Done New false
13508 4548 Acked 5547 Write 6 Done Done New false
13509 4549 Acked 5548 Read 7 Done Done New false
13510 4550 Acked 5549 Write 2 Done Done New false
13511 4551 Acked 5550 Flush 0 Done Done New false
13512 4552 Acked 5551 Write 6 Done Done New false
13513 4553 Acked 5552 Write 9 Done Done New false
13514 4554 Acked 5553 Read 3 Done Done New false
13515 4555 Acked 5554 Read 7 Done Done New false
13516 4556 Acked 5555 Write 2 Done Done New false
13517 4557 Acked 5556 Write 8 Done Done New false
13518 4558 Acked 5557 Read 3 Done Done New false
13519 4559 Acked 5558 Write 4 Done Done New false
13520 4560 Acked 5559 Write 9 Done Done New false
13521 4561 Acked 5560 Write 5 Done Done New false
13522 4562 Acked 5561 Flush 0 Done Done New false
13523 4563 Acked 5562 Read 9 Done Done New false
13524 4564 Acked 5563 Read 2 Done Done New false
13525 4565 Acked 5564 Write 1 Done Done New false
13526 4566 Acked 5565 Write 2 Done Done New false
13527 4567 Acked 5566 Write 3 Done Done New false
13528 4568 Acked 5567 Read 5 Done Done New false
13529 4569 Acked 5568 Flush 0 Done Done New false
13530 4570 Acked 5569 Write 7 Done Done New false
13531 4571 Acked 5570 Write 8 Done Done New false
13532 4572 Acked 5571 Write 1 Done Done New false
13533 4573 Acked 5572 Read 9 Done Done New false
13534 4574 Acked 5573 Flush 0 Done Done New false
13535 4575 Acked 5574 Write 3 Done Done New false
13536 4576 Acked 5575 Read 1 Done Done New false
13537 4577 Acked 5576 Read 2 Done Done New false
13538 4578 Acked 5577 Read 3 Done Done New false
13539 4579 Acked 5578 Write 3 Done Done New false
13540 4580 Acked 5579 Write 10 Done Done New false
13541 4581 Acked 5580 Read 6 Done Done New false
13542 4582 Acked 5581 Read 3 Done Done New false
13543 4583 Acked 5582 Read 8 Done Done New false
13544 4584 Acked 5583 Read 10 Done Done New false
13545 4585 Acked 5584 Read 3 Done Done New false
13546 4586 Acked 5585 Read 10 Done Done New false
13547 4587 Acked 5586 Write 4 Done Done New false
13548 4588 Acked 5587 Write 4 Done Done New false
13549 4589 Acked 5588 Flush 0 Done Done New false
13550 4590 Acked 5589 Read 8 Done Done New false
13551 4591 Acked 5590 Read 6 Done Done New false
13552 4592 Acked 5591 Read 6 Done Done New false
13553 4593 Acked 5592 Write 9 Done Done New false
13554 4594 Acked 5593 Read 5 Done Done New false
13555 4595 Acked 5594 Read 1 Done Done New false
13556 4596 Acked 5595 Flush 0 Done Done New false
13557 4597 Acked 5596 Read 4 Done Done New false
13558 4598 Acked 5597 Read 1 Done Done New false
13559 4599 Acked 5598 Flush 0 Done Done New false
13560 4600 Acked 5599 Read 7 Done Done New false
13561 4601 Acked 5600 Read 8 Done Done New false
13562 4602 Acked 5601 Flush 0 Done Done New false
13563 4603 Acked 5602 Read 10 Done Done New false
13564 4604 Acked 5603 Flush 0 Done Done New false
13565 4605 Acked 5604 Write 4 Done Done New false
13566 4606 Acked 5605 Read 10 Done Done New false
13567 4607 Acked 5606 Read 6 Done Done New false
13568 4608 Acked 5607 Read 10 Done Done New false
13569 4609 Acked 5608 Write 7 Done Done New false
13570 4610 Acked 5609 Read 6 Done Done New false
13571 4611 Acked 5610 Write 8 Done Done New false
13572 4612 Acked 5611 Write 5 Done Done New false
13573 4613 Acked 5612 Read 3 Done Done New false
13574 4614 Acked 5613 Write 10 Done Done New false
13575 4615 Acked 5614 Flush 0 Done Done New false
13576 4616 Acked 5615 Read 8 Done Done New false
13577 4617 Acked 5616 Read 8 Done Done New false
13578 4618 Acked 5617 Flush 0 Done Done New false
13579 4619 Acked 5618 Write 9 Done Done New false
13580 4620 Acked 5619 Write 1 Done Done New false
13581 4621 Acked 5620 Write 1 Done Done New false
13582 4622 Acked 5621 Write 4 Done Done New false
13583 4623 Acked 5622 Read 7 Done Done New false
13584 4624 Acked 5623 Write 8 Done Done New false
13585 4625 Acked 5624 Read 5 Done Done New false
13586 4626 Acked 5625 Write 8 Done Done New false
13587 4627 Acked 5626 Read 3 Done Done New false
13588 4628 Acked 5627 Write 3 Done Done New false
13589 4629 Acked 5628 Read 1 Done Done New false
13590 4630 Acked 5629 Read 4 Done Done New false
13591 4631 Acked 5630 Write 6 Done Done New false
13592 4632 Acked 5631 Read 8 Done Done New false
13593 4633 Acked 5632 Read 10 Done Done New false
13594 4634 Acked 5633 Read 6 Done Done New false
13595 4635 Acked 5634 Read 8 Done Done New false
13596 4636 Acked 5635 Write 3 Done Done New false
13597 4637 Acked 5636 Read 7 Done Done New false
13598 4638 Acked 5637 Write 7 Done Done New false
13599 4639 Acked 5638 Flush 0 Done Done New false
13600 4640 Acked 5639 Read 8 Done Done New false
13601 4641 Acked 5640 Read 10 Done Done New false
13602 4642 Acked 5641 Read 8 Done Done New false
13603 4643 Acked 5642 Write 10 Done Done New false
13604 4644 Acked 5643 Read 9 Done Done New false
13605 4645 Acked 5644 Flush 0 Done Done New false
13606 4646 Acked 5645 Write 10 Done Done New false
13607 4647 Acked 5646 Read 4 Done Done New false
13608 4648 Acked 5647 Write 10 Done Done New false
13609 4649 Acked 5648 Read 4 Done Done New false
13610 4650 Acked 5649 Write 4 Done Done New false
13611 4651 Acked 5650 Read 10 Done Done New false
13612 4652 Acked 5651 Write 8 Done Done New false
13613 4653 Acked 5652 Write 4 Done Done New false
13614 4654 Acked 5653 Read 1 Done Done New false
13615 4655 Acked 5654 Read 6 Done Done New false
13616 4656 Acked 5655 Write 8 Done Done New false
13617 4657 Acked 5656 Write 10 Done Done New false
13618 4658 Acked 5657 Write 10 Done Done New false
13619 4659 Acked 5658 Write 1 Done Done New false
13620 4660 Acked 5659 Write 6 Done Done New false
13621 4661 Acked 5660 Read 9 Done Done New false
13622 4662 Acked 5661 Write 2 Done Done New false
13623 4663 Acked 5662 Read 9 Done Done New false
13624 4664 Acked 5663 Write 7 Done Done New false
13625 4665 Acked 5664 Write 7 Done Done New false
13626 4666 Acked 5665 Read 4 Done Done New false
13627 4667 Acked 5666 Read 2 Done Done New false
13628 4668 Acked 5667 Read 8 Done Done New false
13629 4669 Acked 5668 Flush 0 Done Done New false
13630 4670 Acked 5669 Flush 0 Done Done New false
13631 4671 Acked 5670 Write 8 Done Done New false
13632 4672 Acked 5671 Read 6 Done Done New false
13633 4673 Acked 5672 Write 6 Done Done New false
13634 4674 Acked 5673 Read 6 Done Done New false
13635 4675 Acked 5674 Flush 0 Done Done New false
13636 4676 Acked 5675 Write 2 Done Done New false
13637 4677 Acked 5676 Write 7 Done Done New false
13638 4678 Acked 5677 Read 6 Done Done New false
13639 4679 Acked 5678 Read 10 Done Done New false
13640 4680 Acked 5679 Write 1 Done Done New false
13641 4681 Acked 5680 Write 3 Done Done New false
13642 4682 Acked 5681 Flush 0 Done Done New false
13643 4683 Acked 5682 Write 6 Done Done New false
13644 4684 Acked 5683 Write 6 Done Done New false
13645 4685 Acked 5684 Read 5 Done Done New false
13646 4686 Acked 5685 Write 6 Done Done New false
13647 4687 Acked 5686 Read 10 Done Done New false
13648 4688 Acked 5687 Write 5 Done Done New false
13649 4689 Acked 5688 Read 9 Done Done New false
13650 4690 Acked 5689 Read 4 Done Done New false
13651 4691 Acked 5690 Write 1 Done Done New false
13652 4692 Acked 5691 Read 1 Done Done New false
13653 4693 Acked 5692 Read 5 Done Done New false
13654 4694 Acked 5693 Write 6 Done Done New false
13655 4695 Acked 5694 Read 9 Done Done New false
13656 4696 Acked 5695 Write 10 Done Done New false
13657 4697 Acked 5696 Read 9 Done Done New false
13658 4698 Acked 5697 Read 9 Done Done New false
13659 4699 Acked 5698 Read 10 Done Done New false
13660 4700 Acked 5699 Write 7 Done Done New false
13661 4701 Acked 5700 Write 5 Done Done New false
13662 4702 Acked 5701 Write 1 Done Done New false
13663 4703 Acked 5702 Read 7 Done Done New false
13664 4704 Acked 5703 Read 2 Done Done New false
13665 4705 Acked 5704 Read 2 Done Done New false
13666 4706 Acked 5705 Write 10 Done Done New false
13667 4707 Acked 5706 Read 3 Done Done New false
13668 4708 Acked 5707 Read 6 Done Done New false
13669 4709 Acked 5708 Read 6 Done Done New false
13670 4710 Acked 5709 Write 9 Done Done New false
13671 4711 Acked 5710 Read 1 Done Done New false
13672 4712 Acked 5711 Read 6 Done Done New false
13673 4713 Acked 5712 Read 10 Done Done New false
13674 4714 Acked 5713 Read 4 Done Done New false
13675 4715 Acked 5714 Read 5 Done Done New false
13676 4716 Acked 5715 Write 1 Done Done New false
13677 4717 Acked 5716 Flush 0 Done Done New false
13678 4718 Acked 5717 Write 9 Done Done New false
13679 4719 Acked 5718 Write 10 Done Done New false
13680 4720 Acked 5719 Flush 0 Done Done New false
13681 4721 Acked 5720 Write 4 Done Done New false
13682 4722 Acked 5721 Read 8 Done Done New false
13683 4723 Acked 5722 Write 9 Done Done New false
13684 4724 Acked 5723 Read 1 Done Done New false
13685 4725 Acked 5724 Write 8 Done Done New false
13686 4726 Acked 5725 Read 1 Done Done New false
13687 4727 Acked 5726 Write 1 Done Done New false
13688 4728 Acked 5727 Write 8 Done Done New false
13689 4729 Acked 5728 Flush 0 Done Done New false
13690 4730 Acked 5729 Write 7 Done Done New false
13691 4731 Acked 5730 Read 8 Done Done New false
13692 4732 Acked 5731 Read 2 Done Done New false
13693 4733 Acked 5732 Read 9 Done Done New false
13694 4734 Acked 5733 Read 8 Done Done New false
13695 4735 Acked 5734 Read 9 Done Done New false
13696 4736 Acked 5735 Read 5 Done Done New false
13697 4737 Acked 5736 Write 1 Done Done New false
13698 4738 Acked 5737 Read 3 Done Done New false
13699 4739 Acked 5738 Write 2 Done Done New false
13700 4740 Acked 5739 Write 5 Done Done New false
13701 4741 Acked 5740 Read 8 Done Done New false
13702 4742 Acked 5741 Write 2 Done Done New false
13703 4743 Acked 5742 Read 1 Done Done New false
13704 4744 Acked 5743 Read 6 Done Done New false
13705 4745 Acked 5744 Read 2 Done Done New false
13706 4746 Acked 5745 Flush 0 Done Done New false
13707 4747 Acked 5746 Write 7 Done Done New false
13708 4748 Acked 5747 Write 7 Done Done New false
13709 4749 Acked 5748 Write 4 Done Done New false
13710 4750 Acked 5749 Read 4 Done Done New false
13711 4751 Acked 5750 Read 9 Done Done New false
13712 4752 Acked 5751 Read 7 Done Done New false
13713 4753 Acked 5752 Read 8 Done Done New false
13714 4754 Acked 5753 Flush 0 Done Done New false
13715 4755 Acked 5754 Read 8 Done Done New false
13716 4756 Acked 5755 Read 6 Done Done New false
13717 4757 Acked 5756 Read 1 Done Done New false
13718 4758 Acked 5757 Write 8 Done Done New false
13719 4759 Acked 5758 Write 5 Done Done New false
13720 4760 Acked 5759 Flush 0 Done Done New false
13721 4761 Acked 5760 Write 10 Done Done New false
13722 4762 Acked 5761 Read 4 Done Done New false
13723 4763 Acked 5762 Read 7 Done Done New false
13724 4764 Acked 5763 Read 5 Done Done New false
13725 4765 Acked 5764 Read 4 Done Done New false
13726 4766 Acked 5765 Write 2 Done Done New false
13727 4767 Acked 5766 Flush 0 Done Done New false
13728 4768 Acked 5767 Write 8 Done Done New false
13729 4769 Acked 5768 Read 6 Done Done New false
13730 4770 Acked 5769 Write 8 Done Done New false
13731 4771 Acked 5770 Read 10 Done Done New false
13732 4772 Acked 5771 Read 6 Done Done New false
13733 4773 Acked 5772 Flush 0 Done Done New false
13734 4774 Acked 5773 Flush 0 Done Done New false
13735 4775 Acked 5774 Write 6 Done Done New false
13736 4776 Acked 5775 Read 5 Done Done New false
13737 4777 Acked 5776 Write 10 Done Done New false
13738 4778 Acked 5777 Read 2 Done Done New false
13739 4779 Acked 5778 Write 5 Done Done New false
13740 4780 Acked 5779 Write 9 Done Done New false
13741 4781 Acked 5780 Write 10 Done Done New false
13742 4782 Acked 5781 Flush 0 Done Done New false
13743 4783 Acked 5782 Read 10 Done Done New false
13744 4784 Acked 5783 Read 8 Done Done New false
13745 4785 Acked 5784 Read 6 Done Done New false
13746 4786 Acked 5785 Read 1 Done Done New false
13747 4787 Acked 5786 Write 1 Done Done New false
13748 4788 Acked 5787 Read 5 Done Done New false
13749 4789 Acked 5788 Flush 0 Done Done New false
13750 4790 Acked 5789 Write 9 Done Done New false
13751 4791 Acked 5790 Write 8 Done Done New false
13752 4792 Acked 5791 Write 6 Done Done New false
13753 4793 Acked 5792 Read 9 Done Done New false
13754 4794 Acked 5793 Read 1 Done Done New false
13755 4795 Acked 5794 Read 4 Done Done New false
13756 4796 Acked 5795 Write 8 Done Done New false
13757 4797 Acked 5796 Read 2 Done Done New false
13758 4798 Acked 5797 Read 1 Done Done New false
13759 4799 Acked 5798 Read 6 Done Done New false
13760 4800 Acked 5799 Write 1 Done Done New false
13761 4801 Acked 5800 Read 5 Done Done New false
13762 4802 Acked 5801 Read 10 Done Done New false
13763 4803 Acked 5802 Write 3 Done Done New false
13764 4804 Acked 5803 Write 4 Done Done New false
13765 4805 Acked 5804 Write 10 Done Done New false
13766 4806 Acked 5805 Read 6 Done Done New false
13767 4807 Acked 5806 Flush 0 Done Done New false
13768 4808 Acked 5807 Write 6 Done Done New false
13769 4809 Acked 5808 Read 7 Done Done New false
13770 4810 Acked 5809 Read 7 Done Done New false
13771 4811 Acked 5810 Read 5 Done Done New false
13772 4812 Acked 5811 Write 3 Done Done New false
13773 4813 Acked 5812 Read 4 Done Done New false
13774 4814 Acked 5813 Write 9 Done Done New false
13775 4815 Acked 5814 Read 3 Done Done New false
13776 4816 Acked 5815 Write 2 Done Done New false
13777 4817 Acked 5816 Read 2 Done Done New false
13778 4818 Acked 5817 Flush 0 Done Done New false
13779 4819 Acked 5818 Flush 0 Done Done New false
13780 4820 Acked 5819 Write 6 Done Done New false
13781 4821 Acked 5820 Read 6 Done Done New false
13782 4822 Acked 5821 Write 7 Done Done New false
13783 4823 Acked 5822 Write 2 Done Done New false
13784 4824 Acked 5823 Flush 0 Done Done New false
13785 4825 Acked 5824 Read 7 Done Done New false
13786 4826 Acked 5825 Read 10 Done Done New false
13787 4827 Acked 5826 Read 9 Done Done New false
13788 4828 Acked 5827 Flush 0 Done Done New false
13789 4829 Acked 5828 Write 5 Done Done New false
13790 4830 Acked 5829 Read 1 Done Done New false
13791 4831 Acked 5830 Read 9 Done Done New false
13792 4832 Acked 5831 Read 5 Done Done New false
13793 4833 Acked 5832 Write 6 Done Done New false
13794 4834 Acked 5833 Write 4 Done Done New false
13795 4835 Acked 5834 Write 8 Done Done New false
13796 4836 Acked 5835 Read 8 Done Done New false
13797 4837 Acked 5836 Read 9 Done Done New false
13798 4838 Acked 5837 Flush 0 Done Done New false
13799 STATES DS:0 DS:1 DS:2 TOTAL
13800 New 0 0 302 302
13801 Sent 0 0 0 0
13802 Done 302 302 0 604
13803 Skipped 0 0 0 0
13804 Error 0 0 0 0
13805 Last Flush: 5837 5837 5535
13806 Downstairs last five completed: 5535 5534 5533 5532 5531 5530
13807 Upstairs last five completed: 4838 4837 4836 4835 4834 4833
13808 CLIENT: Up:0 ds:302 act:2
138092023-10-13T04:44:44.039ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 2
138102023-10-13T04:44:44.039ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
138112023-10-13T04:44:44.039ZINFOcrucible: [2] client re-new 302 jobs since flush 5535 = downstairs
138122023-10-13T04:44:44.040ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
138132023-10-13T04:44:44.040ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
138142023-10-13T04:44:44.040ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
138152023-10-13T04:44:44.040ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
138162023-10-13T04:44:44.040ZINFOcrucible: [2] send last flush ID to this DS: 5535
138172023-10-13T04:44:44.040ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
138182023-10-13T04:44:44.041ZINFOcrucible: [2] Replied this last flush ID: 5535
138192023-10-13T04:44:44.041ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Offline ds_transition to Replay
138202023-10-13T04:44:44.041ZINFOcrucible: [2] Transition from Offline to Replay
138212023-10-13T04:44:44.041ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
138222023-10-13T04:44:44.041ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Replay ds_transition to Active
138232023-10-13T04:44:44.041ZINFOcrucible: [2] Transition from Replay to Active
138242023-10-13T04:44:44.041ZINFOcrucible: [2] Starts cmd_loop
138252023-10-13T04:44:44.041ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
138262023-10-13T04:44:44.041ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
138272023-10-13T04:44:44.041ZINFOcrucible: Set check for repair
138282023-10-13T04:44:45.042ZWARNcrucible: [2] flow control end
138292023-10-13T04:44:45.042ZINFOcrucible: Checking if live repair is needed
138302023-10-13T04:44:45.042ZINFOcrucible: No Live Repair required at this time
13831 ----------------------------------------------------------------
13832 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
13833 Downstairs last five completed: 5837 5836 5835 5834 5833 5832
13834 Upstairs last five completed: 4838 4837 4836 4835 4834 4833
13835 CLIENT: Up:0 ds:0 act:3
13836 Replay: All jobs finished, all DS active.
13837 Main task has work to do, go find it
13838 got dsc Stop(1)
13839 stop 1
13840 Oct 13 04:44:47.997 INFO request completed, latency_us: 640, response_code: 204[8820] Got stop action so:false kr:false
13841 , uri: /stop/cid/1, method: POST, req_id: bea1bb84-79fb-4a64-ba7f-f936552fa4e7, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
13842 Oct 13 04:44:47.998 INFO request completed, latency_us: 188, response_code: 200, uri: /state/cid/1, method: GET, req_id: 1aa7f370-caf4-4be4-8f2c-5c360a4e0289, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
138432023-10-13T04:44:47.999ZWARNcrucible: [1] downstairs disconnected
138442023-10-13T04:44:47.999ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
138452023-10-13T04:44:47.999ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8820 closed looper = 1
138462023-10-13T04:44:47.999ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:false
138472023-10-13T04:44:47.999ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
138482023-10-13T04:44:47.999ZINFOcrucible: [1] 127.0.0.1:8820 task reports offline
138492023-10-13T04:44:47.999ZWARNcrucible: [1] pm_task rx.recv() is None
13850 [8820] Exited with: ExitStatus(unix_wait_status(9))
13851 [8820][1] reports Exit
13852 Oct 13 04:44:52.001 INFO request completed, latency_us: 252, response_code: 200, uri: /state/cid/1, method: GET, req_id: 5e0186b8-3955-4587-bd9b-cffd81e0b059, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
13853 001/300 Read block 1695 len 40960
138542023-10-13T04:44:52.003ZINFOcrucible: [1] connecting to 127.0.0.1:8820 looper = 1
13855 002/300 Write block 4313 len 20480 data: 3 2 2 3 5
13856 003/300 Read block 2834 len 36864
13857 004/300 Read block 4356 len 28672
13858 005/300 Write block 2845 len 8192 data: 4 6
13859 006/300 Write block 3117 len 24576 data: 5 5 5 5 5 5
13860 007/300 Read block 1240 len 24576
13861 008/300 Read block 4640 len 24576
13862 009/300 Read block 3761 len 8192
13863 010/300 Read block 4565 len 8192
13864 011/300 Read block 2011 len 12288
13865 012/300 Flush
13866 013/300 Read block 3713 len 40960
13867 014/300 Write block 222 len 4096 data: 2
13868 015/300 Write block 3411 len 8192 data: 4 4
13869 016/300 Read block 1339 len 20480
13870 017/300 Write block 1393 len 28672 data: 4 3 3 3 4 4 5
13871 018/300 Read block 536 len 24576
13872 019/300 Write block 1464 len 24576 data: 2 2 2 2 3 3
13873 020/300 Read block 1524 len 32768
13874 021/300 Read block 536 len 8192
13875 022/300 Read block 2629 len 40960
13876 023/300 Read block 1749 len 8192
13877 024/300 Read block 1890 len 40960
13878 025/300 Read block 365 len 16384
13879 026/300 Read block 2772 len 28672
13880 027/300 Write block 2809 len 24576 data: 3 4 3 3 4 3
13881 028/300 Write block 507 len 32768 data: 4 4 5 5 6 7 6 7
13882 029/300 Write block 2960 len 4096 data: 2
13883 030/300 Write block 1765 len 36864 data: 3 3 3 3 4 6 7 7 5
13884 031/300 Write block 237 len 20480 data: 3 4 4 4 4
13885 032/300 Write block 4065 len 20480 data: 3 3 4 5 6
13886 033/300 Read block 3030 len 36864
13887 034/300 Write block 3405 len 24576 data: 3 4 4 4 4 4
13888 035/300 Read block 1654 len 24576
13889 036/300 Read block 3400 len 20480
13890 037/300 Write block 4123 len 12288 data: 4 4 4
13891 038/300 Read block 810 len 16384
13892 039/300 Write block 614 len 24576 data: 6 5 4 4 3 5
13893 040/300 Read block 4613 len 8192
13894 041/300 Read block 1407 len 4096
13895 042/300 Flush
13896 043/300 Read block 2548 len 28672
13897 044/300 Read block 325 len 12288
13898 045/300 Read block 4983 len 20480
13899 046/300 Read block 2840 len 36864
13900 047/300 Write block 3552 len 20480 data: 4 6 5 5 5
13901 048/300 Read block 157 len 40960
13902 049/300 Read block 3760 len 32768
13903 050/300 Read block 3930 len 40960
13904 051/300 Write block 2507 len 20480 data: 5 4 3 4 5
13905 052/300 Read block 682 len 24576
13906 053/300 Read block 2785 len 36864
13907 054/300 Read block 4647 len 16384
13908 055/300 Read block 1133 len 40960
13909 056/300 Read block 1575 len 4096
13910 057/300 Read block 4714 len 28672
13911 058/300 Read block 65 len 8192
13912 059/300 Write block 611 len 12288 data: 7 8 7
13913 060/300 Write block 676 len 16384 data: 3 3 3 3
13914 061/300 Read block 4186 len 8192
13915 062/300 Read block 3759 len 16384
13916 063/300 Write block 3015 len 36864 data: 2 2 3 3 4 5 5 3 4
13917 064/300 Write block 1464 len 16384 data: 3 3 3 3
13918 065/300 Write block 2343 len 28672 data: 3 3 2 3 3 3 4
13919 066/300 Read block 2544 len 12288
13920 067/300 Read block 4059 len 4096
13921 068/300 Write block 2720 len 20480 data: 4 4 3 3 2
13922 069/300 Read block 2153 len 36864
13923 070/300 Read block 1199 len 24576
13924 071/300 Write block 2749 len 40960 data: 6 5 5 5 6 4 2 2 3 3
13925 072/300 Read block 3727 len 12288
13926 073/300 Read block 863 len 8192
13927 074/300 Read block 3266 len 4096
13928 075/300 Write block 3588 len 8192 data: 5 5
13929 076/300 Write block 4782 len 28672 data: 5 5 3 4 4 4 3
13930 077/300 Write block 284 len 24576 data: 4 3 2 2 4 5
13931 078/300 Read block 2074 len 24576
13932 079/300 Read block 2774 len 40960
13933 080/300 Read block 3004 len 8192
13934 081/300 Write block 2814 len 20480 data: 4 3 3 3 3
13935 082/300 Flush
13936 083/300 Write block 2487 len 32768 data: 4 4 4 5 5 5 4 4
13937 084/300 Read block 2734 len 24576
13938 085/300 Write block 145 len 24576 data: 4 2 4 4 4 4
13939 086/300 Read block 1598 len 16384
13940 087/300 Write block 368 len 4096 data: 2
13941 088/300 Read block 1061 len 8192
13942 089/300 Read block 1421 len 32768
13943 090/300 Write block 1173 len 28672 data: 3 2 3 5 5 4 4
13944 091/300 Write block 3024 len 40960 data: 5 5 5 5 5 4 5 4 4 3
13945 092/300 Read block 1249 len 28672
13946 093/300 Write block 2901 len 12288 data: 2 2 2
13947 094/300 Write block 1488 len 20480 data: 5 5 6 7 5
13948 095/300 Read block 431 len 16384
13949 096/300 Flush
13950 097/300 Write block 1062 len 32768 data: 3 3 3 2 3 3 3 4
13951 098/300 Read block 4024 len 20480
13952 099/300 Read block 3035 len 32768
13953 100/300 Write block 3057 len 24576 data: 3 3 3 3 4 3
13954 101/300 Flush
13955 102/300 Read block 157 len 20480
13956 103/300 Read block 3108 len 20480
13957 104/300 Read block 4465 len 20480
13958 105/300 Write block 4598 len 4096 data: 2
13959 106/300 Read block 2995 len 12288
13960 107/300 Read block 1664 len 32768
13961 108/300 Write block 171 len 8192 data: 4 4
13962 109/300 Write block 33 len 12288 data: 4 4 3
13963 110/300 Read block 1287 len 16384
13964 111/300 Read block 783 len 32768
13965 112/300 Write block 2737 len 40960 data: 5 5 5 4 5 3 2 2 3 3
13966 113/300 Read block 4434 len 16384
13967 114/300 Write block 3519 len 32768 data: 2 2 3 4 4 7 6 6
13968 115/300 Read block 3889 len 24576
13969 116/300 Write block 506 len 4096 data: 3
13970 117/300 Read block 648 len 12288
13971 118/300 Flush
13972 119/300 Flush
13973 120/300 Write block 4609 len 36864 data: 3 3 2 2 2 3 4 3 3
13974 121/300 Flush
13975 122/300 Write block 3114 len 12288 data: 3 4 5
13976 123/300 Read block 1908 len 24576
13977 124/300 Write block 752 len 20480 data: 6 6 7 7 7
13978 125/300 Read block 1711 len 20480
13979 126/300 Read block 4004 len 40960
13980 127/300 Read block 607 len 40960
13981 128/300 Write block 1035 len 36864 data: 6 7 6 6 5 4 4 3 3
13982 129/300 Flush
13983 130/300 Write block 4346 len 12288 data: 8 7 8
13984 131/300 Write block 3942 len 16384 data: 6 6 5 3
13985 132/300 Write block 2122 len 8192 data: 3 3
13986 133/300 Flush
13987 134/300 Read block 2519 len 8192
13988 135/300 Read block 3066 len 16384
13989 136/300 Flush
13990 137/300 Read block 448 len 32768
13991 138/300 Read block 3285 len 4096
13992 139/300 Read block 2905 len 8192
13993 140/300 Read block 1688 len 40960
13994 141/300 Flush
13995 142/300 Read block 69 len 32768
13996 143/300 Write block 929 len 12288 data: 3 3 2
13997 144/300 Read block 572 len 8192
13998 145/300 Write block 989 len 28672 data: 6 5 6 6 7 7 7
13999 146/300 Write block 1126 len 28672 data: 5 6 7 7 5 4 5
14000 147/300 Read block 625 len 28672
14001 148/300 Read block 3819 len 20480
14002 149/300 Read block 259 len 32768
14003 150/300 Read block 3017 len 28672
14004 151/300 Write block 11 len 28672 data: 3 3 4 4 4 4 4
14005 152/300 Write block 2366 len 28672 data: 5 4 4 4 4 5 6
14006 153/300 Write block 2 len 28672 data: 2 3 3 4 5 4 4
14007 154/300 Flush
14008 155/300 Flush
14009 156/300 Read block 604 len 8192
14010 157/300 Read block 4473 len 12288
14011 158/300 Write block 1913 len 4096 data: 3
14012 159/300 Write block 1618 len 36864 data: 4 5 4 4 4 2 2 2 2
14013 160/300 Read block 1301 len 8192
14014 161/300 Write block 2046 len 32768 data: 3 4 3 4 4 5 4 3
14015 162/300 Read block 4097 len 36864
14016 163/300 Flush
14017 164/300 Write block 3880 len 32768 data: 3 3 3 3 4 4 3 3
14018 165/300 Flush
14019 166/300 Flush
14020 167/300 Write block 2412 len 16384 data: 3 3 4 4
14021 168/300 Read block 4569 len 36864
14022 169/300 Write block 2151 len 24576 data: 3 3 4 4 4 4
14023 170/300 Read block 394 len 28672
14024 171/300 Read block 3293 len 8192
14025 172/300 Read block 593 len 24576
14026 173/300 Flush
14027 174/300 Read block 1805 len 28672
14028 175/300 Write block 4639 len 36864 data: 2 3 4 5 5 4 5 6 6
14029 176/300 Write block 261 len 8192 data: 3 2
14030 177/300 Write block 1093 len 24576 data: 7 8 6 5 6 7
14031 178/300 Write block 527 len 16384 data: 6 6 5 4
14032 179/300 Read block 2591 len 8192
14033 180/300 Read block 3347 len 4096
14034 181/300 Read block 4939 len 16384
14035 182/300 Flush
14036 183/300 Read block 4845 len 8192
14037 184/300 Read block 2130 len 20480
14038 185/300 Flush
14039 186/300 Read block 735 len 24576
14040 187/300 Read block 2137 len 16384
14041 188/300 Read block 2785 len 8192
14042 189/300 Write block 2365 len 12288 data: 3 6 5
14043 190/300 Read block 4355 len 36864
14044 191/300 Write block 38 len 36864 data: 3 3 3 2 2 2 4 4 5
14045 192/300 Read block 2317 len 8192
14046 193/300 Read block 882 len 20480
14047 194/300 Read block 2427 len 8192
14048 195/300 Read block 3521 len 40960
14049 196/300 Write block 2110 len 4096 data: 5
14050 197/300 Read block 3987 len 24576
14051 198/300 Write block 2684 len 24576 data: 2 2 3 3 3 3
14052 199/300 Read block 2662 len 8192
14053 200/300 Read block 2114 len 12288
14054 201/300 Read block 1813 len 20480
14055 202/300 Read block 2033 len 8192
14056 203/300 Read block 1793 len 12288
14057 204/300 Flush
14058 205/300 Write block 1349 len 36864 data: 3 3 4 4 4 3 3 3 5
14059 206/300 Write block 3390 len 36864 data: 3 4 4 4 4 4 5 4 4
14060 207/300 Write block 3382 len 32768 data: 5 5 6 4 5 3 3 3
14061 208/300 Read block 519 len 8192
14062 209/300 Write block 3327 len 24576 data: 3 3 3 3 3 3
14063 210/300 Write block 2079 len 16384 data: 2 3 4 4
14064 211/300 Read block 4954 len 4096
14065 212/300 Write block 3837 len 16384 data: 4 5 5 6
14066 213/300 Read block 4021 len 36864
14067 214/300 Read block 4462 len 36864
14068 215/300 Write block 4755 len 4096 data: 4
14069 216/300 Read block 113 len 28672
14070 217/300 Flush
14071 218/300 Read block 3974 len 4096
14072 219/300 Write block 387 len 8192 data: 4 4
14073 220/300 Flush
14074 221/300 Flush
14075 222/300 Write block 2336 len 8192 data: 4 4
14076 223/300 Write block 411 len 24576 data: 4 4 4 4 4 4
14077 224/300 Write block 3698 len 28672 data: 5 5 4 4 4 4 4
14078 225/300 Flush
14079 226/300 Read block 2558 len 24576
14080 227/300 Write block 4725 len 24576 data: 6 6 6 5 4 3
14081 228/300 Write block 615 len 20480 data: 6 5 5 4 6
14082 229/300 Read block 4751 len 16384
14083 230/300 Write block 1045 len 28672 data: 3 3 3 3 3 2 2
14084 231/300 Write block 1032 len 8192 data: 5 5
14085 232/300 Read block 4402 len 16384
14086 233/300 Read block 3960 len 4096
14087 234/300 Write block 143 len 20480 data: 4 4 5 3 5
14088 235/300 Write block 1568 len 36864 data: 2 3 3 3 3 3 3 3 3
14089 236/300 Write block 537 len 32768 data: 4 4 4 4 3 3 2 3
14090 237/300 Read block 1795 len 12288
14091 238/300 Read block 2257 len 32768
14092 239/300 Read block 4597 len 32768
14093 240/300 Read block 3000 len 16384
14094 241/300 Flush
14095 242/300 Read block 4020 len 20480
14096 243/300 Flush
14097 244/300 Flush
14098 245/300 Flush
14099 246/300 Write block 564 len 20480 data: 3 2 2 3 3
14100 247/300 Read block 4090 len 32768
14101 248/300 Write block 745 len 4096 data: 5
14102 249/300 Read block 181 len 4096
14103 250/300 Write block 2492 len 24576 data: 6 5 5 3 2 3
14104 251/300 Read block 107 len 20480
14105 252/300 Read block 2414 len 8192
14106 253/300 Read block 4290 len 36864
14107 254/300 Read block 1284 len 8192
14108 255/300 Read block 2365 len 28672
14109 256/300 Write block 3768 len 20480 data: 5 4 4 4 3
14110 257/300 Write block 1056 len 8192 data: 4 4
14111 258/300 Write block 457 len 4096 data: 3
14112 259/300 Read block 1851 len 32768
14113 260/300 Flush
14114 261/300 Read block 230 len 20480
14115 262/300 Write block 2812 len 36864 data: 4 5 5 4 4 4 4 2 2
14116 263/300 Read block 281 len 36864
14117 264/300 Flush
14118 265/300 Write block 44 len 28672 data: 5 5 6 5 5 5 4
14119 266/300 Write block 3413 len 8192 data: 4 4
14120 267/300 Read block 1983 len 12288
14121 268/300 Read block 4960 len 20480
14122 269/300 Write block 666 len 24576 data: 5 5 5 2 2 2
14123 270/300 Write block 4005 len 16384 data: 5 4 4 5
14124 271/300 Write block 2595 len 28672 data: 5 4 5 4 5 4 3
14125 272/300 Write block 289 len 40960 data: 6 6 6 8 8 8 7 8 9 8
14126 273/300 Write block 3217 len 16384 data: 3 3 4 5
14127 274/300 Read block 3067 len 20480
14128 275/300 Read block 4851 len 8192
14129 276/300 Write block 843 len 40960 data: 4 3 4 4 5 5 4 4 6 6
14130 277/300 Write block 3872 len 12288 data: 3 3 4
14131 278/300 Read block 628 len 8192
14132 279/300 Read block 4379 len 20480
14133 280/300 Read block 2934 len 20480
14134 281/300 Read block 2446 len 32768
14135 282/300 Write block 2555 len 16384 data: 4 5 5 5
14136 283/300 Write block 220 len 40960 data: 4 2 3 2 2 2 2 3 3 3
14137 284/300 Read block 475 len 8192
14138 285/300 Read block 656 len 32768
14139 286/300 Write block 1914 len 16384 data: 3 3 3 3
14140 287/300 Read block 4175 len 8192
14141 288/300 Flush
14142 289/300 Write block 2576 len 4096 data: 4
14143 290/300 Write block 1704 len 20480 data: 2 3 3 4 4
14144 291/300 Read block 87 len 28672
14145 292/300 Flush
14146 293/300 Write block 3760 len 20480 data: 3 4 4 4 3
14147 294/300 Write block 310 len 12288 data: 4 5 4
14148 295/300 Write block 3628 len 28672 data: 4 4 4 4 3 3 3
14149 296/300 Read block 1402 len 8192
14150 297/300 Read block 29 len 4096
14151 298/300 Read block 2264 len 32768
14152 299/300 Flush
14153 300/300 Write block 1472 len 12288 data: 4 4 3
14154 Main task has work to do, go find it
14155 got dsc Start(1)
14156 start 1
14157 Oct 13 04:44:52.879 INFO request completed, latency_us: 338, response_code: 204, uri: /start/cid/1, method: POST, req_id: 1190c635-c560-4b8d-a5e1-69205545af65, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
14158 Replay: started 1, returned:Ok(())
14159 ----------------------------------------------------------------
14160 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
14161 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
14162 4839 Acked 5838 Read 10 Done New Done false
14163 4840 Acked 5839 Write 5 Done New Done false
14164 4841 Acked 5840 Read 9 Done New Done false
14165 4842 Acked 5841 Read 7 Done New Done false
14166 4843 Acked 5842 Write 2 Done New Done false
14167 4844 Acked 5843 Write 6 Done New Done false
14168 4845 Acked 5844 Read 6 Done New Done false
14169 4846 Acked 5845 Read 6 Done New Done false
14170 4847 Acked 5846 Read 2 Done New Done false
14171 4848 Acked 5847 Read 2 Done New Done false
14172 4849 Acked 5848 Read 3 Done New Done false
14173 4850 Acked 5849 Flush 0 Done New Done false
14174 4851 Acked 5850 Read 10 Done New Done false
14175 4852 Acked 5851 Write 1 Done New Done false
14176 4853 Acked 5852 Write 2 Done New Done false
14177 4854 Acked 5853 Read 5 Done New Done false
14178 4855 Acked 5854 Write 7 Done New Done false
14179 4856 Acked 5855 Read 6 Done New Done false
14180 4857 Acked 5856 Write 6 Done New Done false
14181 4858 Acked 5857 Read 8 Done New Done false
14182 4859 Acked 5858 Read 2 Done New Done false
14183 4860 Acked 5859 Read 10 Done New Done false
14184 4861 Acked 5860 Read 2 Done New Done false
14185 4862 Acked 5861 Read 10 Done New Done false
14186 4863 Acked 5862 Read 4 Done New Done false
14187 4864 Acked 5863 Read 7 Done New Done false
14188 4865 Acked 5864 Write 6 Done New Done false
14189 4866 Acked 5865 Write 8 Done New Done false
14190 4867 Acked 5866 Write 1 Done New Done false
14191 4868 Acked 5867 Write 9 Done New Done false
14192 4869 Acked 5868 Write 5 Done New Done false
14193 4870 Acked 5869 Write 5 Done New Done false
14194 4871 Acked 5870 Read 9 Done New Done false
14195 4872 Acked 5871 Write 6 Done New Done false
14196 4873 Acked 5872 Read 6 Done New Done false
14197 4874 Acked 5873 Read 5 Done New Done false
14198 4875 Acked 5874 Write 3 Done New Done false
14199 4876 Acked 5875 Read 4 Done New Done false
14200 4877 Acked 5876 Write 6 Done New Done false
14201 4878 Acked 5877 Read 2 Done New Done false
14202 4879 Acked 5878 Read 1 Done New Done false
14203 4880 Acked 5879 Flush 0 Done New Done false
14204 4881 Acked 5880 Read 7 Done New Done false
14205 4882 Acked 5881 Read 3 Done New Done false
14206 4883 Acked 5882 Read 5 Done New Done false
14207 4884 Acked 5883 Read 9 Done New Done false
14208 4885 Acked 5884 Write 5 Done New Done false
14209 4886 Acked 5885 Read 10 Done New Done false
14210 4887 Acked 5886 Read 8 Done New Done false
14211 4888 Acked 5887 Read 10 Done New Done false
14212 4889 Acked 5888 Flush 0 Done New Done false
14213 4890 Acked 5889 Write 5 Done New Done false
14214 4891 Acked 5890 Read 6 Done New Done false
14215 4892 Acked 5891 Read 9 Done New Done false
14216 4893 Acked 5892 Read 4 Done New Done false
14217 4894 Acked 5893 Read 10 Done New Done false
14218 4895 Acked 5894 Read 1 Done New Done false
14219 4896 Acked 5895 Read 7 Done New Done false
14220 4897 Acked 5896 Read 2 Done New Done false
14221 4898 Acked 5897 Write 3 Done New Done false
14222 4899 Acked 5898 Write 4 Done New Done false
14223 4900 Acked 5899 Read 2 Done New Done false
14224 4901 Acked 5900 Read 4 Done New Done false
14225 4902 Acked 5901 Write 9 Done New Done false
14226 4903 Acked 5902 Write 4 Done New Done false
14227 4904 Acked 5903 Write 7 Done New Done false
14228 4905 Acked 5904 Read 3 Done New Done false
14229 4906 Acked 5905 Read 1 Done New Done false
14230 4907 Acked 5906 Write 5 Done New Done false
14231 4908 Acked 5907 Read 9 Done New Done false
14232 4909 Acked 5908 Read 6 Done New Done false
14233 4910 Acked 5909 Write 10 Done New Done false
14234 4911 Acked 5910 Read 3 Done New Done false
14235 4912 Acked 5911 Read 2 Done New Done false
14236 4913 Acked 5912 Read 1 Done New Done false
14237 4914 Acked 5913 Write 2 Done New Done false
14238 4915 Acked 5914 Write 7 Done New Done false
14239 4916 Acked 5915 Write 6 Done New Done false
14240 4917 Acked 5916 Read 6 Done New Done false
14241 4918 Acked 5917 Read 10 Done New Done false
14242 4919 Acked 5918 Read 2 Done New Done false
14243 4920 Acked 5919 Write 5 Done New Done false
14244 4921 Acked 5920 Flush 0 Done New Done false
14245 4922 Acked 5921 Write 8 Done New Done false
14246 4923 Acked 5922 Read 6 Done New Done false
14247 4924 Acked 5923 Write 6 Done New Done false
14248 4925 Acked 5924 Read 4 Done New Done false
14249 4926 Acked 5925 Write 1 Done New Done false
14250 4927 Acked 5926 Read 2 Done New Done false
14251 4928 Acked 5927 Read 8 Done New Done false
14252 4929 Acked 5928 Write 7 Done New Done false
14253 4930 Acked 5929 Write 10 Done New Done false
14254 4931 Acked 5930 Read 7 Done New Done false
14255 4932 Acked 5931 Write 3 Done New Done false
14256 4933 Acked 5932 Write 5 Done New Done false
14257 4934 Acked 5933 Read 4 Done New Done false
14258 4935 Acked 5934 Flush 0 Done New Done false
14259 4936 Acked 5935 Write 8 Done New Done false
14260 4937 Acked 5936 Read 5 Done New Done false
14261 4938 Acked 5937 Read 8 Done New Done false
14262 4939 Acked 5938 Write 6 Done New Done false
14263 4940 Acked 5939 Flush 0 Done New Done false
14264 4941 Acked 5940 Read 5 Done New Done false
14265 4942 Acked 5941 Read 5 Done New Done false
14266 4943 Acked 5942 Read 5 Done New Done false
14267 4944 Acked 5943 Write 1 Done New Done false
14268 4945 Acked 5944 Read 3 Done New Done false
14269 4946 Acked 5945 Read 8 Done New Done false
14270 4947 Acked 5946 Write 2 Done New Done false
14271 4948 Acked 5947 Write 3 Done New Done false
14272 4949 Acked 5948 Read 4 Done New Done false
14273 4950 Acked 5949 Read 8 Done New Done false
14274 4951 Acked 5950 Write 10 Done New Done false
14275 4952 Acked 5951 Read 4 Done New Done false
14276 4953 Acked 5952 Write 8 Done New Done false
14277 4954 Acked 5953 Read 6 Done New Done false
14278 4955 Acked 5954 Write 1 Done New Done false
14279 4956 Acked 5955 Read 3 Done New Done false
14280 4957 Acked 5956 Flush 0 Done New Done false
14281 4958 Acked 5957 Flush 0 Done New Done false
14282 4959 Acked 5958 Write 9 Done New Done false
14283 4960 Acked 5959 Flush 0 Done New Done false
14284 4961 Acked 5960 Write 3 Done New Done false
14285 4962 Acked 5961 Read 6 Done New Done false
14286 4963 Acked 5962 Write 5 Done New Done false
14287 4964 Acked 5963 Read 5 Done New Done false
14288 4965 Acked 5964 Read 10 Done New Done false
14289 4966 Acked 5965 Read 10 Done New Done false
14290 4967 Acked 5966 Write 9 Done New Done false
14291 4968 Acked 5967 Flush 0 Done New Done false
14292 4969 Acked 5968 Write 3 Done New Done false
14293 4970 Acked 5969 Write 4 Done New Done false
14294 4971 Acked 5970 Write 2 Done New Done false
14295 4972 Acked 5971 Flush 0 Done New Done false
14296 4973 Acked 5972 Read 2 Done New Done false
14297 4974 Acked 5973 Read 4 Done New Done false
14298 4975 Acked 5974 Flush 0 Done New Done false
14299 4976 Acked 5975 Read 8 Done New Done false
14300 4977 Acked 5976 Read 1 Done New Done false
14301 4978 Acked 5977 Read 2 Done New Done false
14302 4979 Acked 5978 Read 10 Done New Done false
14303 4980 Acked 5979 Flush 0 Done New Done false
14304 4981 Acked 5980 Read 8 Done New Done false
14305 4982 Acked 5981 Write 3 Done New Done false
14306 4983 Acked 5982 Read 2 Done New Done false
14307 4984 Acked 5983 Write 7 Done New Done false
14308 4985 Acked 5984 Write 7 Done New Done false
14309 4986 Acked 5985 Read 7 Done New Done false
14310 4987 Acked 5986 Read 5 Done New Done false
14311 4988 Acked 5987 Read 8 Done New Done false
14312 4989 Acked 5988 Read 7 Done New Done false
14313 4990 Acked 5989 Write 7 Done New Done false
14314 4991 Acked 5990 Write 7 Done New Done false
14315 4992 Acked 5991 Write 7 Done New Done false
14316 4993 Acked 5992 Flush 0 Done New Done false
14317 4994 Acked 5993 Flush 0 Done New Done false
14318 4995 Acked 5994 Read 2 Done New Done false
14319 4996 Acked 5995 Read 3 Done New Done false
14320 4997 Acked 5996 Write 1 Done New Done false
14321 4998 Acked 5997 Write 9 Done New Done false
14322 4999 Acked 5998 Read 2 Done New Done false
14323 5000 Acked 5999 Write 8 Done New Done false
14324 5001 Acked 6000 Read 9 Done New Done false
14325 5002 Acked 6001 Flush 0 Done New Done false
14326 5003 Acked 6002 Write 8 Done New Done false
14327 5004 Acked 6003 Flush 0 Done New Done false
14328 5005 Acked 6004 Flush 0 Done New Done false
14329 5006 Acked 6005 Write 4 Done New Done false
14330 5007 Acked 6006 Read 9 Done New Done false
14331 5008 Acked 6007 Write 6 Done New Done false
14332 5009 Acked 6008 Read 7 Done New Done false
14333 5010 Acked 6009 Read 2 Done New Done false
14334 5011 Acked 6010 Read 6 Done New Done false
14335 5012 Acked 6011 Flush 0 Done New Done false
14336 5013 Acked 6012 Read 7 Done New Done false
14337 5014 Acked 6013 Write 9 Done New Done false
14338 5015 Acked 6014 Write 2 Done New Done false
14339 5016 Acked 6015 Write 6 Done New Done false
14340 5017 Acked 6016 Write 4 Done New Done false
14341 5018 Acked 6017 Read 2 Done New Done false
14342 5019 Acked 6018 Read 1 Done New Done false
14343 5020 Acked 6019 Read 4 Done New Done false
14344 5021 Acked 6020 Flush 0 Done New Done false
14345 5022 Acked 6021 Read 2 Done New Done false
14346 5023 Acked 6022 Read 5 Done New Done false
14347 5024 Acked 6023 Flush 0 Done New Done false
14348 5025 Acked 6024 Read 6 Done New Done false
14349 5026 Acked 6025 Read 4 Done New Done false
14350 5027 Acked 6026 Read 2 Done New Done false
14351 5028 Acked 6027 Write 3 Done New Done false
14352 5029 Acked 6028 Read 9 Done New Done false
14353 5030 Acked 6029 Write 9 Done New Done false
14354 5031 Acked 6030 Read 2 Done New Done false
14355 5032 Acked 6031 Read 5 Done New Done false
14356 5033 Acked 6032 Read 2 Done New Done false
14357 5034 Acked 6033 Read 10 Done New Done false
14358 5035 Acked 6034 Write 1 Done New Done false
14359 5036 Acked 6035 Read 6 Done New Done false
14360 5037 Acked 6036 Write 6 Done New Done false
14361 5038 Acked 6037 Read 2 Done New Done false
14362 5039 Acked 6038 Read 3 Done New Done false
14363 5040 Acked 6039 Read 5 Done New Done false
14364 5041 Acked 6040 Read 2 Done New Done false
14365 5042 Acked 6041 Read 3 Done New Done false
14366 5043 Acked 6042 Flush 0 Done New Done false
14367 5044 Acked 6043 Write 9 Done New Done false
14368 5045 Acked 6044 Write 9 Done New Done false
14369 5046 Acked 6045 Write 8 Done New Done false
14370 5047 Acked 6046 Read 2 Done New Done false
14371 5048 Acked 6047 Write 6 Done New Done false
14372 5049 Acked 6048 Write 4 Done New Done false
14373 5050 Acked 6049 Read 1 Done New Done false
14374 5051 Acked 6050 Write 4 Done New Done false
14375 5052 Acked 6051 Read 9 Done New Done false
14376 5053 Acked 6052 Read 9 Done New Done false
14377 5054 Acked 6053 Write 1 Done New Done false
14378 5055 Acked 6054 Read 7 Done New Done false
14379 5056 Acked 6055 Flush 0 Done New Done false
14380 5057 Acked 6056 Read 1 Done New Done false
14381 5058 Acked 6057 Write 2 Done New Done false
14382 5059 Acked 6058 Flush 0 Done New Done false
14383 5060 Acked 6059 Flush 0 Done New Done false
14384 5061 Acked 6060 Write 2 Done New Done false
14385 5062 Acked 6061 Write 6 Done New Done false
14386 5063 Acked 6062 Write 7 Done New Done false
14387 5064 Acked 6063 Flush 0 Done New Done false
14388 5065 Acked 6064 Read 6 Done New Done false
14389 5066 Acked 6065 Write 6 Done New Done false
14390 5067 Acked 6066 Write 5 Done New Done false
14391 5068 Acked 6067 Read 4 Done New Done false
14392 5069 Acked 6068 Write 7 Done New Done false
14393 5070 Acked 6069 Write 2 Done New Done false
14394 5071 Acked 6070 Read 4 Done New Done false
14395 5072 Acked 6071 Read 1 Done New Done false
14396 5073 Acked 6072 Write 5 Done New Done false
14397 5074 Acked 6073 Write 9 Done New Done false
14398 5075 Acked 6074 Write 8 Done New Done false
14399 5076 Acked 6075 Read 3 Done New Done false
14400 5077 Acked 6076 Read 8 Done New Done false
14401 5078 Acked 6077 Read 8 Done New Done false
14402 5079 Acked 6078 Read 4 Done New Done false
14403 5080 Acked 6079 Flush 0 Done New Done false
14404 5081 Acked 6080 Read 5 Done New Done false
14405 5082 Acked 6081 Flush 0 Done New Done false
14406 5083 Acked 6082 Flush 0 Done New Done false
14407 5084 Acked 6083 Flush 0 Done New Done false
14408 5085 Acked 6084 Write 5 Done New Done false
14409 5086 Acked 6085 Read 8 Done New Done false
14410 5087 Acked 6086 Write 1 Done New Done false
14411 5088 Acked 6087 Read 1 Done New Done false
14412 5089 Acked 6088 Write 6 Done New Done false
14413 5090 Acked 6089 Read 5 Done New Done false
14414 5091 Acked 6090 Read 2 Done New Done false
14415 5092 Acked 6091 Read 9 Done New Done false
14416 5093 Acked 6092 Read 2 Done New Done false
14417 5094 Acked 6093 Read 7 Done New Done false
14418 5095 Acked 6094 Write 5 Done New Done false
14419 5096 Acked 6095 Write 2 Done New Done false
14420 5097 Acked 6096 Write 1 Done New Done false
14421 5098 Acked 6097 Read 8 Done New Done false
14422 5099 Acked 6098 Flush 0 Done New Done false
14423 5100 Acked 6099 Read 5 Done New Done false
14424 5101 Acked 6100 Write 9 Done New Done false
14425 5102 Acked 6101 Read 9 Done New Done false
14426 5103 Acked 6102 Flush 0 Done New Done false
14427 5104 Acked 6103 Write 7 Done New Done false
14428 5105 Acked 6104 Write 2 Done New Done false
14429 5106 Acked 6105 Read 3 Done New Done false
14430 5107 Acked 6106 Read 5 Done New Done false
14431 5108 Acked 6107 Write 6 Done New Done false
14432 5109 Acked 6108 Write 4 Done New Done false
14433 5110 Acked 6109 Write 7 Done New Done false
14434 5111 Acked 6110 Write 10 Done New Done false
14435 5112 Acked 6111 Write 4 Done New Done false
14436 5113 Acked 6112 Read 5 Done New Done false
14437 5114 Acked 6113 Read 2 Done New Done false
14438 5115 Acked 6114 Write 10 Done New Done false
14439 5116 Acked 6115 Write 3 Done New Done false
14440 5117 Acked 6116 Read 2 Done New Done false
14441 5118 Acked 6117 Read 5 Done New Done false
14442 5119 Acked 6118 Read 5 Done New Done false
14443 5120 Acked 6119 Read 8 Done New Done false
14444 5121 Acked 6120 Write 4 Done New Done false
14445 5122 Acked 6121 Write 10 Done New Done false
14446 5123 Acked 6122 Read 2 Done New Done false
14447 5124 Acked 6123 Read 8 Done New Done false
14448 5125 Acked 6124 Write 4 Done New Done false
14449 5126 Acked 6125 Read 2 Done New Done false
14450 5127 Acked 6126 Flush 0 Done New Done false
14451 5128 Acked 6127 Write 1 Done New Done false
14452 5129 Acked 6128 Write 5 Done New Done false
14453 5130 Acked 6129 Read 7 Done New Done false
14454 5131 Acked 6130 Flush 0 Done New Done false
14455 5132 Acked 6131 Write 5 Done New Done false
14456 5133 Acked 6132 Write 3 Done New Done false
14457 5134 Acked 6133 Write 7 Done New Done false
14458 5135 Acked 6134 Read 2 Done New Done false
14459 5136 Acked 6135 Read 1 Done New Done false
14460 5137 Acked 6136 Read 8 Done New Done false
14461 5138 Acked 6137 Flush 0 Done New Done false
14462 5139 Acked 6138 Write 3 Sent New Sent false
14463 STATES DS:0 DS:1 DS:2 TOTAL
14464 New 0 301 0 301
14465 Sent 1 0 1 2
14466 Done 300 0 300 600
14467 Skipped 0 0 0 0
14468 Error 0 0 0 0
14469 Last Flush: 6137 5837 6137
14470 Downstairs last five completed: 5837 5836 5835 5834 5833 5832
14471 Upstairs last five completed: 5139 5138 5137 5136 5135 5134
14472 CLIENT: Up:0 ds:301 act:2
14473 [8820] got start action
14474 [8820] I am going to restart
14475 Starting downstairs at port 8820
14476 Make output file at "/tmp/dsc/downstairs-8820.txt"
14477 [8820][1] reports Starting
14478 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1193)
14479 [8820][1] reports Running
14480 ----------------------------------------------------------------
14481 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
14482 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
14483 4839 Acked 5838 Read 10 Done New Done false
14484 4840 Acked 5839 Write 5 Done New Done false
14485 4841 Acked 5840 Read 9 Done New Done false
14486 4842 Acked 5841 Read 7 Done New Done false
14487 4843 Acked 5842 Write 2 Done New Done false
14488 4844 Acked 5843 Write 6 Done New Done false
14489 4845 Acked 5844 Read 6 Done New Done false
14490 4846 Acked 5845 Read 6 Done New Done false
14491 4847 Acked 5846 Read 2 Done New Done false
14492 4848 Acked 5847 Read 2 Done New Done false
14493 4849 Acked 5848 Read 3 Done New Done false
14494 4850 Acked 5849 Flush 0 Done New Done false
14495 4851 Acked 5850 Read 10 Done New Done false
14496 4852 Acked 5851 Write 1 Done New Done false
14497 4853 Acked 5852 Write 2 Done New Done false
14498 4854 Acked 5853 Read 5 Done New Done false
14499 4855 Acked 5854 Write 7 Done New Done false
14500 4856 Acked 5855 Read 6 Done New Done false
14501 4857 Acked 5856 Write 6 Done New Done false
14502 4858 Acked 5857 Read 8 Done New Done false
14503 4859 Acked 5858 Read 2 Done New Done false
14504 4860 Acked 5859 Read 10 Done New Done false
14505 4861 Acked 5860 Read 2 Done New Done false
14506 4862 Acked 5861 Read 10 Done New Done false
14507 4863 Acked 5862 Read 4 Done New Done false
14508 4864 Acked 5863 Read 7 Done New Done false
14509 4865 Acked 5864 Write 6 Done New Done false
14510 4866 Acked 5865 Write 8 Done New Done false
14511 4867 Acked 5866 Write 1 Done New Done false
14512 4868 Acked 5867 Write 9 Done New Done false
14513 4869 Acked 5868 Write 5 Done New Done false
14514 4870 Acked 5869 Write 5 Done New Done false
14515 4871 Acked 5870 Read 9 Done New Done false
14516 4872 Acked 5871 Write 6 Done New Done false
14517 4873 Acked 5872 Read 6 Done New Done false
14518 4874 Acked 5873 Read 5 Done New Done false
14519 4875 Acked 5874 Write 3 Done New Done false
14520 4876 Acked 5875 Read 4 Done New Done false
14521 4877 Acked 5876 Write 6 Done New Done false
14522 4878 Acked 5877 Read 2 Done New Done false
14523 4879 Acked 5878 Read 1 Done New Done false
14524 4880 Acked 5879 Flush 0 Done New Done false
14525 4881 Acked 5880 Read 7 Done New Done false
14526 4882 Acked 5881 Read 3 Done New Done false
14527 4883 Acked 5882 Read 5 Done New Done false
14528 4884 Acked 5883 Read 9 Done New Done false
14529 4885 Acked 5884 Write 5 Done New Done false
14530 4886 Acked 5885 Read 10 Done New Done false
14531 4887 Acked 5886 Read 8 Done New Done false
14532 4888 Acked 5887 Read 10 Done New Done false
14533 4889 Acked 5888 Flush 0 Done New Done false
14534 4890 Acked 5889 Write 5 Done New Done false
14535 4891 Acked 5890 Read 6 Done New Done false
14536 4892 Acked 5891 Read 9 Done New Done false
14537 4893 Acked 5892 Read 4 Done New Done false
14538 4894 Acked 5893 Read 10 Done New Done false
14539 4895 Acked 5894 Read 1 Done New Done false
14540 4896 Acked 5895 Read 7 Done New Done false
14541 4897 Acked 5896 Read 2 Done New Done false
14542 4898 Acked 5897 Write 3 Done New Done false
14543 4899 Acked 5898 Write 4 Done New Done false
14544 4900 Acked 5899 Read 2 Done New Done false
14545 4901 Acked 5900 Read 4 Done New Done false
14546 4902 Acked 5901 Write 9 Done New Done false
14547 4903 Acked 5902 Write 4 Done New Done false
14548 4904 Acked 5903 Write 7 Done New Done false
14549 4905 Acked 5904 Read 3 Done New Done false
14550 4906 Acked 5905 Read 1 Done New Done false
14551 4907 Acked 5906 Write 5 Done New Done false
14552 4908 Acked 5907 Read 9 Done New Done false
14553 4909 Acked 5908 Read 6 Done New Done false
14554 4910 Acked 5909 Write 10 Done New Done false
14555 4911 Acked 5910 Read 3 Done New Done false
14556 4912 Acked 5911 Read 2 Done New Done false
14557 4913 Acked 5912 Read 1 Done New Done false
14558 4914 Acked 5913 Write 2 Done New Done false
14559 4915 Acked 5914 Write 7 Done New Done false
14560 4916 Acked 5915 Write 6 Done New Done false
14561 4917 Acked 5916 Read 6 Done New Done false
14562 4918 Acked 5917 Read 10 Done New Done false
14563 4919 Acked 5918 Read 2 Done New Done false
14564 4920 Acked 5919 Write 5 Done New Done false
14565 4921 Acked 5920 Flush 0 Done New Done false
14566 4922 Acked 5921 Write 8 Done New Done false
14567 4923 Acked 5922 Read 6 Done New Done false
14568 4924 Acked 5923 Write 6 Done New Done false
14569 4925 Acked 5924 Read 4 Done New Done false
14570 4926 Acked 5925 Write 1 Done New Done false
14571 4927 Acked 5926 Read 2 Done New Done false
14572 4928 Acked 5927 Read 8 Done New Done false
14573 4929 Acked 5928 Write 7 Done New Done false
14574 4930 Acked 5929 Write 10 Done New Done false
14575 4931 Acked 5930 Read 7 Done New Done false
14576 4932 Acked 5931 Write 3 Done New Done false
14577 4933 Acked 5932 Write 5 Done New Done false
14578 4934 Acked 5933 Read 4 Done New Done false
14579 4935 Acked 5934 Flush 0 Done New Done false
14580 4936 Acked 5935 Write 8 Done New Done false
14581 4937 Acked 5936 Read 5 Done New Done false
14582 4938 Acked 5937 Read 8 Done New Done false
14583 4939 Acked 5938 Write 6 Done New Done false
14584 4940 Acked 5939 Flush 0 Done New Done false
14585 4941 Acked 5940 Read 5 Done New Done false
14586 4942 Acked 5941 Read 5 Done New Done false
14587 4943 Acked 5942 Read 5 Done New Done false
14588 4944 Acked 5943 Write 1 Done New Done false
14589 4945 Acked 5944 Read 3 Done New Done false
14590 4946 Acked 5945 Read 8 Done New Done false
14591 4947 Acked 5946 Write 2 Done New Done false
14592 4948 Acked 5947 Write 3 Done New Done false
14593 4949 Acked 5948 Read 4 Done New Done false
14594 4950 Acked 5949 Read 8 Done New Done false
14595 4951 Acked 5950 Write 10 Done New Done false
14596 4952 Acked 5951 Read 4 Done New Done false
14597 4953 Acked 5952 Write 8 Done New Done false
14598 4954 Acked 5953 Read 6 Done New Done false
14599 4955 Acked 5954 Write 1 Done New Done false
14600 4956 Acked 5955 Read 3 Done New Done false
14601 4957 Acked 5956 Flush 0 Done New Done false
14602 4958 Acked 5957 Flush 0 Done New Done false
14603 4959 Acked 5958 Write 9 Done New Done false
14604 4960 Acked 5959 Flush 0 Done New Done false
14605 4961 Acked 5960 Write 3 Done New Done false
14606 4962 Acked 5961 Read 6 Done New Done false
14607 4963 Acked 5962 Write 5 Done New Done false
14608 4964 Acked 5963 Read 5 Done New Done false
14609 4965 Acked 5964 Read 10 Done New Done false
14610 4966 Acked 5965 Read 10 Done New Done false
14611 4967 Acked 5966 Write 9 Done New Done false
14612 4968 Acked 5967 Flush 0 Done New Done false
14613 4969 Acked 5968 Write 3 Done New Done false
14614 4970 Acked 5969 Write 4 Done New Done false
14615 4971 Acked 5970 Write 2 Done New Done false
14616 4972 Acked 5971 Flush 0 Done New Done false
14617 4973 Acked 5972 Read 2 Done New Done false
14618 4974 Acked 5973 Read 4 Done New Done false
14619 4975 Acked 5974 Flush 0 Done New Done false
14620 4976 Acked 5975 Read 8 Done New Done false
14621 4977 Acked 5976 Read 1 Done New Done false
14622 4978 Acked 5977 Read 2 Done New Done false
14623 4979 Acked 5978 Read 10 Done New Done false
14624 4980 Acked 5979 Flush 0 Done New Done false
14625 4981 Acked 5980 Read 8 Done New Done false
14626 4982 Acked 5981 Write 3 Done New Done false
14627 4983 Acked 5982 Read 2 Done New Done false
14628 4984 Acked 5983 Write 7 Done New Done false
14629 4985 Acked 5984 Write 7 Done New Done false
14630 4986 Acked 5985 Read 7 Done New Done false
14631 4987 Acked 5986 Read 5 Done New Done false
14632 4988 Acked 5987 Read 8 Done New Done false
14633 4989 Acked 5988 Read 7 Done New Done false
14634 4990 Acked 5989 Write 7 Done New Done false
14635 4991 Acked 5990 Write 7 Done New Done false
14636 4992 Acked 5991 Write 7 Done New Done false
14637 4993 Acked 5992 Flush 0 Done New Done false
14638 4994 Acked 5993 Flush 0 Done New Done false
14639 4995 Acked 5994 Read 2 Done New Done false
14640 4996 Acked 5995 Read 3 Done New Done false
14641 4997 Acked 5996 Write 1 Done New Done false
14642 4998 Acked 5997 Write 9 Done New Done false
14643 4999 Acked 5998 Read 2 Done New Done false
14644 5000 Acked 5999 Write 8 Done New Done false
14645 5001 Acked 6000 Read 9 Done New Done false
14646 5002 Acked 6001 Flush 0 Done New Done false
14647 5003 Acked 6002 Write 8 Done New Done false
14648 5004 Acked 6003 Flush 0 Done New Done false
14649 5005 Acked 6004 Flush 0 Done New Done false
14650 5006 Acked 6005 Write 4 Done New Done false
14651 5007 Acked 6006 Read 9 Done New Done false
14652 5008 Acked 6007 Write 6 Done New Done false
14653 5009 Acked 6008 Read 7 Done New Done false
14654 5010 Acked 6009 Read 2 Done New Done false
14655 5011 Acked 6010 Read 6 Done New Done false
14656 5012 Acked 6011 Flush 0 Done New Done false
14657 5013 Acked 6012 Read 7 Done New Done false
14658 5014 Acked 6013 Write 9 Done New Done false
14659 5015 Acked 6014 Write 2 Done New Done false
14660 5016 Acked 6015 Write 6 Done New Done false
14661 5017 Acked 6016 Write 4 Done New Done false
14662 5018 Acked 6017 Read 2 Done New Done false
14663 5019 Acked 6018 Read 1 Done New Done false
14664 5020 Acked 6019 Read 4 Done New Done false
14665 5021 Acked 6020 Flush 0 Done New Done false
14666 5022 Acked 6021 Read 2 Done New Done false
14667 5023 Acked 6022 Read 5 Done New Done false
14668 5024 Acked 6023 Flush 0 Done New Done false
14669 5025 Acked 6024 Read 6 Done New Done false
14670 5026 Acked 6025 Read 4 Done New Done false
14671 5027 Acked 6026 Read 2 Done New Done false
14672 5028 Acked 6027 Write 3 Done New Done false
14673 5029 Acked 6028 Read 9 Done New Done false
14674 5030 Acked 6029 Write 9 Done New Done false
14675 5031 Acked 6030 Read 2 Done New Done false
14676 5032 Acked 6031 Read 5 Done New Done false
14677 5033 Acked 6032 Read 2 Done New Done false
14678 5034 Acked 6033 Read 10 Done New Done false
14679 5035 Acked 6034 Write 1 Done New Done false
14680 5036 Acked 6035 Read 6 Done New Done false
14681 5037 Acked 6036 Write 6 Done New Done false
14682 5038 Acked 6037 Read 2 Done New Done false
14683 5039 Acked 6038 Read 3 Done New Done false
14684 5040 Acked 6039 Read 5 Done New Done false
14685 5041 Acked 6040 Read 2 Done New Done false
14686 5042 Acked 6041 Read 3 Done New Done false
14687 5043 Acked 6042 Flush 0 Done New Done false
14688 5044 Acked 6043 Write 9 Done New Done false
14689 5045 Acked 6044 Write 9 Done New Done false
14690 5046 Acked 6045 Write 8 Done New Done false
14691 5047 Acked 6046 Read 2 Done New Done false
14692 5048 Acked 6047 Write 6 Done New Done false
14693 5049 Acked 6048 Write 4 Done New Done false
14694 5050 Acked 6049 Read 1 Done New Done false
14695 5051 Acked 6050 Write 4 Done New Done false
14696 5052 Acked 6051 Read 9 Done New Done false
14697 5053 Acked 6052 Read 9 Done New Done false
14698 5054 Acked 6053 Write 1 Done New Done false
14699 5055 Acked 6054 Read 7 Done New Done false
14700 5056 Acked 6055 Flush 0 Done New Done false
14701 5057 Acked 6056 Read 1 Done New Done false
14702 5058 Acked 6057 Write 2 Done New Done false
14703 5059 Acked 6058 Flush 0 Done New Done false
14704 5060 Acked 6059 Flush 0 Done New Done false
14705 5061 Acked 6060 Write 2 Done New Done false
14706 5062 Acked 6061 Write 6 Done New Done false
14707 5063 Acked 6062 Write 7 Done New Done false
14708 5064 Acked 6063 Flush 0 Done New Done false
14709 5065 Acked 6064 Read 6 Done New Done false
14710 5066 Acked 6065 Write 6 Done New Done false
14711 5067 Acked 6066 Write 5 Done New Done false
14712 5068 Acked 6067 Read 4 Done New Done false
14713 5069 Acked 6068 Write 7 Done New Done false
14714 5070 Acked 6069 Write 2 Done New Done false
14715 5071 Acked 6070 Read 4 Done New Done false
14716 5072 Acked 6071 Read 1 Done New Done false
14717 5073 Acked 6072 Write 5 Done New Done false
14718 5074 Acked 6073 Write 9 Done New Done false
14719 5075 Acked 6074 Write 8 Done New Done false
14720 5076 Acked 6075 Read 3 Done New Done false
14721 5077 Acked 6076 Read 8 Done New Done false
14722 5078 Acked 6077 Read 8 Done New Done false
14723 5079 Acked 6078 Read 4 Done New Done false
14724 5080 Acked 6079 Flush 0 Done New Done false
14725 5081 Acked 6080 Read 5 Done New Done false
14726 5082 Acked 6081 Flush 0 Done New Done false
14727 5083 Acked 6082 Flush 0 Done New Done false
14728 5084 Acked 6083 Flush 0 Done New Done false
14729 5085 Acked 6084 Write 5 Done New Done false
14730 5086 Acked 6085 Read 8 Done New Done false
14731 5087 Acked 6086 Write 1 Done New Done false
14732 5088 Acked 6087 Read 1 Done New Done false
14733 5089 Acked 6088 Write 6 Done New Done false
14734 5090 Acked 6089 Read 5 Done New Done false
14735 5091 Acked 6090 Read 2 Done New Done false
14736 5092 Acked 6091 Read 9 Done New Done false
14737 5093 Acked 6092 Read 2 Done New Done false
14738 5094 Acked 6093 Read 7 Done New Done false
14739 5095 Acked 6094 Write 5 Done New Done false
14740 5096 Acked 6095 Write 2 Done New Done false
14741 5097 Acked 6096 Write 1 Done New Done false
14742 5098 Acked 6097 Read 8 Done New Done false
14743 5099 Acked 6098 Flush 0 Done New Done false
14744 5100 Acked 6099 Read 5 Done New Done false
14745 5101 Acked 6100 Write 9 Done New Done false
14746 5102 Acked 6101 Read 9 Done New Done false
14747 5103 Acked 6102 Flush 0 Done New Done false
14748 5104 Acked 6103 Write 7 Done New Done false
14749 5105 Acked 6104 Write 2 Done New Done false
14750 5106 Acked 6105 Read 3 Done New Done false
14751 5107 Acked 6106 Read 5 Done New Done false
14752 5108 Acked 6107 Write 6 Done New Done false
14753 5109 Acked 6108 Write 4 Done New Done false
14754 5110 Acked 6109 Write 7 Done New Done false
14755 5111 Acked 6110 Write 10 Done New Done false
14756 5112 Acked 6111 Write 4 Done New Done false
14757 5113 Acked 6112 Read 5 Done New Done false
14758 5114 Acked 6113 Read 2 Done New Done false
14759 5115 Acked 6114 Write 10 Done New Done false
14760 5116 Acked 6115 Write 3 Done New Done false
14761 5117 Acked 6116 Read 2 Done New Done false
14762 5118 Acked 6117 Read 5 Done New Done false
14763 5119 Acked 6118 Read 5 Done New Done false
14764 5120 Acked 6119 Read 8 Done New Done false
14765 5121 Acked 6120 Write 4 Done New Done false
14766 5122 Acked 6121 Write 10 Done New Done false
14767 5123 Acked 6122 Read 2 Done New Done false
14768 5124 Acked 6123 Read 8 Done New Done false
14769 5125 Acked 6124 Write 4 Done New Done false
14770 5126 Acked 6125 Read 2 Done New Done false
14771 5127 Acked 6126 Flush 0 Done New Done false
14772 5128 Acked 6127 Write 1 Done New Done false
14773 5129 Acked 6128 Write 5 Done New Done false
14774 5130 Acked 6129 Read 7 Done New Done false
14775 5131 Acked 6130 Flush 0 Done New Done false
14776 5132 Acked 6131 Write 5 Done New Done false
14777 5133 Acked 6132 Write 3 Done New Done false
14778 5134 Acked 6133 Write 7 Done New Done false
14779 5135 Acked 6134 Read 2 Done New Done false
14780 5136 Acked 6135 Read 1 Done New Done false
14781 5137 Acked 6136 Read 8 Done New Done false
14782 5138 Acked 6137 Flush 0 Done New Done false
14783 5139 Acked 6138 Write 3 Done New Done false
14784 5140 Acked 6139 Flush 0 Done New Done false
14785 STATES DS:0 DS:1 DS:2 TOTAL
14786 New 0 302 0 302
14787 Sent 0 0 0 0
14788 Done 302 0 302 604
14789 Skipped 0 0 0 0
14790 Error 0 0 0 0
14791 Last Flush: 6139 5837 6139
14792 Downstairs last five completed: 5837 5836 5835 5834 5833 5832
14793 Upstairs last five completed: 5140 5139 5138 5137 5136 5135
14794 CLIENT: Up:0 ds:302 act:2
147952023-10-13T04:44:57.009ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 1
147962023-10-13T04:44:57.009ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state Offline
147972023-10-13T04:44:57.009ZINFOcrucible: [1] client re-new 302 jobs since flush 5837 = downstairs
147982023-10-13T04:44:57.010ZINFOcrucible: [1] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
147992023-10-13T04:44:57.011ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
148002023-10-13T04:44:57.011ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
148012023-10-13T04:44:57.011ZINFOcrucible: Returning client:1 UUID:12345678-0000-0000-0000-000000008820 matches
148022023-10-13T04:44:57.011ZINFOcrucible: [1] send last flush ID to this DS: 5837
148032023-10-13T04:44:57.011ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
148042023-10-13T04:44:57.011ZINFOcrucible: [1] Replied this last flush ID: 5837
148052023-10-13T04:44:57.011ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Offline Active ds_transition to Replay
148062023-10-13T04:44:57.011ZINFOcrucible: [1] Transition from Offline to Replay
148072023-10-13T04:44:57.011ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
148082023-10-13T04:44:57.011ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Replay Active ds_transition to Active
148092023-10-13T04:44:57.011ZINFOcrucible: [1] Transition from Replay to Active
148102023-10-13T04:44:57.011ZINFOcrucible: [1] Starts cmd_loop
148112023-10-13T04:44:57.011ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
148122023-10-13T04:44:57.011ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
148132023-10-13T04:44:57.011ZINFOcrucible: Set check for repair
148142023-10-13T04:44:58.013ZINFOcrucible: Checking if live repair is needed
148152023-10-13T04:44:58.013ZINFOcrucible: No Live Repair required at this time
148162023-10-13T04:44:58.013ZWARNcrucible: [1] flow control end
14817 ----------------------------------------------------------------
14818 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
14819 Downstairs last five completed: 6139 6138 6137 6136 6135 6134
14820 Upstairs last five completed: 5140 5139 5138 5137 5136 5135
14821 CLIENT: Up:0 ds:0 act:3
14822 Replay: All jobs finished, all DS active.
14823 Main task has work to do, go find it
14824 got dsc Stop(2)
14825 stop 2
14826 Oct 13 04:45:00.887 INFO request completed, latency_us: 469, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 4876d254-bb35-4aad-a2fb-c38e2b366c19, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
14827 [8830] Got stop action so:false kr:false
14828 Oct 13 04:45:00.888 INFO request completed, latency_us: 157, response_code: 200, uri: /state/cid/2, method: GET, req_id: 288dda3c-85dd-49ba-b924-5833c1fe8285, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
148292023-10-13T04:45:00.888ZWARNcrucible: [2] downstairs disconnected
148302023-10-13T04:45:00.888ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
148312023-10-13T04:45:00.888ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8830 closed looper = 2
148322023-10-13T04:45:00.888ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
148332023-10-13T04:45:00.888ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
148342023-10-13T04:45:00.888ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
148352023-10-13T04:45:00.888ZWARNcrucible: [2] pm_task rx.recv() is None
14836 [8830] Exited with: ExitStatus(unix_wait_status(9))
14837 [8830][2] reports Exit
14838 Oct 13 04:45:04.891 INFO request completed, latency_us: 255, response_code: 200, uri: /state/cid/2, method: GET, req_id: c226a5ce-0b6d-4a34-a9a2-843e15217c04, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
14839 001/300 Read block 3481 len 12288
148402023-10-13T04:45:04.894ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
14841 002/300 Write block 3061 len 16384 data: 5 4 2 4
14842 003/300 Read block 2822 len 20480
14843 004/300 Read block 4825 len 12288
14844 005/300 Read block 4455 len 36864
14845 006/300 Flush
14846 007/300 Write block 3658 len 20480 data: 4 4 3 2 2
14847 008/300 Read block 4051 len 40960
14848 009/300 Read block 2231 len 20480
14849 010/300 Write block 2892 len 16384 data: 5 6 5 5
14850 011/300 Flush
14851 012/300 Read block 3244 len 16384
14852 013/300 Write block 4753 len 28672 data: 8 7 5 4 4 5 5
14853 014/300 Write block 1558 len 20480 data: 5 6 6 3 3
14854 015/300 Read block 1263 len 20480
14855 016/300 Read block 178 len 28672
14856 017/300 Read block 4732 len 32768
14857 018/300 Flush
14858 019/300 Write block 3460 len 28672 data: 4 3 2 2 4 5 5
14859 020/300 Write block 2283 len 16384 data: 2 3 4 4
14860 021/300 Read block 2179 len 8192
14861 022/300 Read block 1855 len 36864
14862 023/300 Read block 3817 len 20480
14863 024/300 Read block 464 len 32768
14864 025/300 Read block 4988 len 16384
14865 026/300 Write block 4988 len 20480 data: 4 4 4 3 3
14866 027/300 Read block 1612 len 16384
14867 028/300 Read block 1483 len 16384
14868 029/300 Write block 2563 len 36864 data: 3 4 4 6 6 7 8 9 9
14869 030/300 Write block 1045 len 16384 data: 4 4 4 4
14870 031/300 Write block 3589 len 16384 data: 6 4 2 2
14871 032/300 Flush
14872 033/300 Read block 1096 len 40960
14873 034/300 Read block 4490 len 36864
14874 035/300 Flush
14875 036/300 Read block 1275 len 20480
14876 037/300 Write block 4972 len 36864 data: 4 3 4 4 4 4 4 5 5
14877 038/300 Write block 4176 len 16384 data: 3 3 3 3
14878 039/300 Flush
14879 040/300 Write block 2915 len 36864 data: 3 2 2 2 2 3 5 5 5
14880 041/300 Read block 3774 len 24576
14881 042/300 Write block 652 len 8192 data: 2 2
14882 043/300 Read block 2604 len 40960
14883 044/300 Flush
14884 045/300 Write block 2113 len 32768 data: 4 4 4 4 3 2 3 2
14885 046/300 Write block 206 len 12288 data: 4 5 6
14886 047/300 Read block 858 len 4096
14887 048/300 Write block 4754 len 32768 data: 8 6 5 5 6 6 5 4
14888 049/300 Write block 3932 len 12288 data: 3 3 3
14889 050/300 Write block 4092 len 4096 data: 2
14890 051/300 Write block 2845 len 36864 data: 5 7 5 7 6 5 5 6 6
14891 052/300 Read block 413 len 36864
14892 053/300 Read block 4970 len 24576
14893 054/300 Read block 19 len 8192
14894 055/300 Flush
14895 056/300 Write block 4573 len 4096 data: 6
14896 057/300 Read block 724 len 8192
14897 058/300 Read block 865 len 16384
14898 059/300 Read block 862 len 12288
14899 060/300 Read block 4487 len 12288
14900 061/300 Write block 1407 len 32768 data: 3 3 3 3 3 3 3 4
14901 062/300 Write block 2766 len 12288 data: 5 5 6
14902 063/300 Read block 4453 len 8192
14903 064/300 Read block 3620 len 16384
14904 065/300 Read block 3141 len 32768
14905 066/300 Read block 1299 len 4096
14906 067/300 Read block 1418 len 36864
14907 068/300 Write block 4492 len 8192 data: 3 2
14908 069/300 Write block 57 len 24576 data: 5 4 5 5 5 3
14909 070/300 Read block 440 len 16384
14910 071/300 Read block 2706 len 24576
14911 072/300 Read block 955 len 12288
14912 073/300 Write block 1709 len 24576 data: 5 5 5 5 6 4
14913 074/300 Flush
14914 075/300 Read block 734 len 16384
14915 076/300 Read block 97 len 40960
14916 077/300 Read block 3990 len 32768
14917 078/300 Write block 3295 len 16384 data: 2 2 2 3
14918 079/300 Read block 3320 len 16384
14919 080/300 Read block 3873 len 28672
14920 081/300 Read block 2495 len 28672
14921 082/300 Read block 1905 len 32768
14922 083/300 Read block 2318 len 8192
14923 084/300 Write block 1667 len 28672 data: 4 5 5 6 5 4 4
14924 085/300 Read block 1745 len 8192
14925 086/300 Read block 4134 len 40960
14926 087/300 Read block 4026 len 16384
14927 088/300 Write block 4967 len 12288 data: 2 3 3
14928 089/300 Flush
14929 090/300 Read block 3252 len 40960
14930 091/300 Write block 4565 len 4096 data: 2
14931 092/300 Write block 4410 len 4096 data: 5
14932 093/300 Flush
14933 094/300 Read block 2887 len 12288
14934 095/300 Read block 2813 len 4096
14935 096/300 Flush
14936 097/300 Read block 421 len 32768
14937 098/300 Write block 1716 len 4096 data: 5
14938 099/300 Read block 364 len 32768
14939 100/300 Write block 1653 len 4096 data: 3
14940 101/300 Read block 203 len 8192
14941 102/300 Write block 2158 len 12288 data: 3 3 5
14942 103/300 Write block 3721 len 36864 data: 3 4 4 3 4 3 3 5 4
14943 104/300 Write block 4124 len 8192 data: 5 5
14944 105/300 Read block 4843 len 16384
14945 106/300 Write block 2665 len 40960 data: 2 2 2 2 3 5 4 3 3 3
14946 107/300 Write block 151 len 24576 data: 5 5 5 5 5 4
14947 108/300 Write block 2127 len 16384 data: 6 6 5 4
14948 109/300 Write block 4816 len 16384 data: 3 5 5 6
14949 110/300 Write block 2395 len 8192 data: 4 4
14950 111/300 Write block 4454 len 28672 data: 8 6 5 5 5 5 6
14951 112/300 Write block 4690 len 28672 data: 2 2 2 2 3 3 3
14952 113/300 Read block 3373 len 40960
14953 114/300 Write block 835 len 40960 data: 3 3 3 3 3 3 4 4 5 4
14954 115/300 Read block 1199 len 8192
14955 116/300 Read block 2137 len 16384
14956 117/300 Read block 3426 len 36864
14957 118/300 Read block 3643 len 36864
14958 119/300 Flush
14959 120/300 Write block 2325 len 12288 data: 7 6 5
14960 121/300 Read block 4931 len 4096
14961 122/300 Write block 2387 len 32768 data: 3 4 2 2 2 2 2 2
14962 123/300 Read block 257 len 40960
14963 124/300 Flush
14964 125/300 Read block 3350 len 40960
14965 126/300 Read block 4603 len 12288
14966 127/300 Read block 3641 len 20480
14967 128/300 Read block 2267 len 16384
14968 129/300 Write block 2695 len 4096 data: 5
14969 130/300 Write block 4159 len 8192 data: 4 5
14970 131/300 Write block 672 len 4096 data: 3
14971 132/300 Write block 2967 len 40960 data: 5 6 5 5 5 4 2 2 2 2
14972 133/300 Write block 2035 len 12288 data: 4 4 5
14973 134/300 Read block 2249 len 20480
14974 135/300 Write block 54 len 32768 data: 5 5 5 6 5 6 6 6
14975 136/300 Flush
14976 137/300 Read block 2445 len 8192
14977 138/300 Read block 4384 len 20480
14978 139/300 Read block 1348 len 28672
14979 140/300 Write block 4287 len 24576 data: 5 4 5 5 4 3
14980 141/300 Read block 337 len 28672
14981 142/300 Read block 4877 len 4096
14982 143/300 Read block 120 len 20480
14983 144/300 Read block 1572 len 24576
14984 145/300 Read block 4249 len 4096
14985 146/300 Write block 4838 len 20480 data: 4 3 2 2 2
14986 147/300 Read block 2124 len 8192
14987 148/300 Read block 2257 len 4096
14988 149/300 Write block 1321 len 16384 data: 3 2 2 2
14989 150/300 Write block 411 len 28672 data: 5 5 5 5 5 5 4
14990 151/300 Read block 3664 len 12288
14991 152/300 Flush
14992 153/300 Read block 1743 len 20480
14993 154/300 Read block 1795 len 28672
14994 155/300 Write block 2678 len 36864 data: 3 4 2 2 2 3 3 3 4
14995 156/300 Write block 4297 len 4096 data: 8
14996 157/300 Read block 1188 len 28672
14997 158/300 Write block 836 len 8192 data: 4 4
14998 159/300 Write block 4286 len 12288 data: 4 6 5
14999 160/300 Write block 2241 len 16384 data: 5 5 4 4
15000 161/300 Read block 1754 len 8192
15001 162/300 Write block 3323 len 40960 data: 6 6 6 5 4 4 4 4 4 4
15002 163/300 Read block 3702 len 4096
15003 164/300 Read block 2474 len 40960
15004 165/300 Write block 2147 len 4096 data: 4
15005 166/300 Read block 2252 len 12288
15006 167/300 Write block 2776 len 12288 data: 5 4 4
15007 168/300 Write block 842 len 32768 data: 5 6 5 5 5 6 6 5
15008 169/300 Read block 4254 len 8192
15009 170/300 Write block 658 len 20480 data: 3 3 3 4 5
15010 171/300 Write block 4437 len 40960 data: 5 5 5 5 6 7 5 4 5 5
15011 172/300 Read block 4266 len 20480
15012 173/300 Write block 3317 len 4096 data: 7
15013 174/300 Read block 1668 len 16384
15014 175/300 Read block 2508 len 20480
15015 176/300 Read block 1775 len 12288
15016 177/300 Write block 2711 len 8192 data: 4 4
15017 178/300 Read block 4388 len 24576
15018 179/300 Read block 2741 len 24576
15019 180/300 Read block 275 len 32768
15020 181/300 Flush
15021 182/300 Read block 909 len 20480
15022 183/300 Write block 452 len 36864 data: 4 4 3 3 2 4 3 4 4
15023 184/300 Read block 4506 len 4096
15024 185/300 Read block 2856 len 16384
15025 186/300 Read block 4567 len 36864
15026 187/300 Write block 616 len 28672 data: 6 6 5 7 5 6 5
15027 188/300 Write block 1792 len 20480 data: 5 6 5 5 4
15028 189/300 Write block 4901 len 12288 data: 3 3 4
15029 190/300 Read block 536 len 8192
15030 191/300 Write block 1001 len 4096 data: 4
15031 192/300 Read block 1573 len 20480
15032 193/300 Read block 4603 len 12288
15033 194/300 Read block 258 len 8192
15034 195/300 Read block 3802 len 28672
15035 196/300 Read block 242 len 24576
15036 197/300 Write block 4686 len 12288 data: 4 4 4
15037 198/300 Write block 1382 len 20480 data: 3 4 5 4 3
15038 199/300 Write block 1273 len 12288 data: 3 3 3
15039 200/300 Write block 1315 len 8192 data: 3 3
15040 201/300 Flush
15041 202/300 Write block 572 len 36864 data: 6 6 5 5 4 4 4 4 4
15042 203/300 Read block 3561 len 36864
15043 204/300 Write block 3176 len 28672 data: 4 3 2 2 2 3 3
15044 205/300 Read block 371 len 4096
15045 206/300 Write block 4479 len 24576 data: 3 4 4 5 5 4
15046 207/300 Read block 2276 len 24576
15047 208/300 Flush
15048 209/300 Flush
15049 210/300 Read block 3691 len 36864
15050 211/300 Write block 2862 len 32768 data: 3 3 4 3 3 3 3 2
15051 212/300 Read block 73 len 40960
15052 213/300 Write block 1158 len 20480 data: 4 3 3 3 3
15053 214/300 Read block 2487 len 24576
15054 215/300 Read block 3349 len 4096
15055 216/300 Read block 2810 len 24576
15056 217/300 Read block 4826 len 28672
15057 218/300 Read block 2049 len 16384
15058 219/300 Read block 3725 len 40960
15059 220/300 Read block 4139 len 40960
15060 221/300 Read block 2795 len 40960
15061 222/300 Read block 1682 len 20480
15062 223/300 Read block 274 len 4096
15063 224/300 Write block 3582 len 32768 data: 8 7 7 7 6 6 6 7
15064 225/300 Write block 3463 len 24576 data: 3 5 6 6 3 4
15065 226/300 Write block 2099 len 32768 data: 6 6 6 5 5 5 5 6
15066 227/300 Write block 4202 len 40960 data: 5 5 5 5 5 4 5 5 6 7
15067 228/300 Write block 3391 len 40960 data: 5 5 5 5 5 6 5 5 4 2
15068 229/300 Read block 4007 len 40960
15069 230/300 Write block 4687 len 12288 data: 5 5 2
15070 231/300 Write block 3068 len 12288 data: 4 5 6
15071 232/300 Flush
15072 233/300 Write block 1664 len 4096 data: 4
15073 234/300 Flush
15074 235/300 Write block 4506 len 8192 data: 3 3
15075 236/300 Write block 1714 len 4096 data: 5
15076 237/300 Write block 4292 len 36864 data: 4 4 6 7 8 9 6 6 6
15077 238/300 Flush
15078 239/300 Read block 2714 len 4096
15079 240/300 Write block 3548 len 24576 data: 3 3 2 2 5 7
15080 241/300 Read block 3225 len 36864
15081 242/300 Read block 1047 len 32768
15082 243/300 Write block 3362 len 20480 data: 5 5 5 4 4
15083 244/300 Read block 4691 len 16384
15084 245/300 Read block 4152 len 8192
15085 246/300 Write block 4050 len 28672 data: 3 3 4 4 4 3 3
15086 247/300 Write block 2476 len 32768 data: 3 3 2 2 2 2 2 3
15087 248/300 Flush
15088 249/300 Flush
15089 250/300 Read block 4159 len 28672
15090 251/300 Flush
15091 252/300 Read block 1894 len 4096
15092 253/300 Write block 2825 len 40960 data: 3 4 4 5 4 4 3 2 4 3
15093 254/300 Read block 1131 len 12288
15094 255/300 Flush
15095 256/300 Write block 664 len 24576 data: 4 6 6 6 6 3
15096 257/300 Read block 3040 len 16384
15097 258/300 Read block 950 len 4096
15098 259/300 Read block 2381 len 28672
15099 260/300 Read block 1472 len 24576
15100 261/300 Read block 4031 len 40960
15101 262/300 Read block 3837 len 12288
15102 263/300 Read block 531 len 4096
15103 264/300 Read block 4121 len 32768
15104 265/300 Write block 1589 len 8192 data: 5 4
15105 266/300 Read block 3866 len 28672
15106 267/300 Flush
15107 268/300 Read block 1246 len 20480
15108 269/300 Read block 4454 len 32768
15109 270/300 Write block 3882 len 28672 data: 4 4 5 5 4 4 3
15110 271/300 Read block 3600 len 24576
15111 272/300 Read block 2135 len 24576
15112 273/300 Write block 148 len 40960 data: 5 5 5 6 6 6 6 6 5 4
15113 274/300 Write block 2815 len 20480 data: 5 5 5 5 3
15114 275/300 Write block 4455 len 8192 data: 7 6
15115 276/300 Read block 4561 len 40960
15116 277/300 Flush
15117 278/300 Read block 1635 len 20480
15118 279/300 Write block 2971 len 20480 data: 6 5 3 3 3
15119 280/300 Flush
15120 281/300 Read block 938 len 40960
15121 282/300 Read block 4225 len 16384
15122 283/300 Flush
15123 284/300 Write block 2795 len 32768 data: 8 8 8 6 5 4 4 3
15124 285/300 Write block 1715 len 40960 data: 5 6 3 2 2 3 3 4 3 3
15125 286/300 Write block 2627 len 40960 data: 3 5 4 4 4 3 3 3 3 2
15126 287/300 Read block 3875 len 32768
15127 288/300 Write block 826 len 16384 data: 2 2 2 3
15128 289/300 Flush
15129 290/300 Write block 2841 len 40960 data: 5 6 4 4 6 8 6 8 7 6
15130 291/300 Write block 3029 len 40960 data: 5 6 5 5 4 3 3 3 4 4
15131 292/300 Write block 1699 len 16384 data: 6 6 6 2
15132 293/300 Read block 1919 len 40960
15133 294/300 Read block 4710 len 28672
15134 295/300 Flush
15135 296/300 Read block 3048 len 40960
15136 297/300 Write block 1802 len 16384 data: 3 3 3 3
15137 298/300 Write block 1325 len 8192 data: 2 2
15138 299/300 Read block 1318 len 40960
15139 300/300 Write block 3641 len 16384 data: 4 4 5 5
15140 Main task has work to do, go find it
15141 got dsc Start(2)
15142 start 2
15143 Oct 13 04:45:05.793 INFO request completed, latency_us: 353, response_code: 204, uri: /start/cid/2, method: POST, req_id: e44f716b-8438-4ae6-8d8b-43d49b5c44ed, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
15144 Replay: started 2, returned:Ok(())
15145 ----------------------------------------------------------------
15146 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
15147 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
15148 5141 Acked 6140 Read 3 Done Done New false
15149 5142 Acked 6141 Write 4 Done Done New false
15150 5143 Acked 6142 Read 5 Done Done New false
15151 5144 Acked 6143 Read 3 Done Done New false
15152 5145 Acked 6144 Read 9 Done Done New false
15153 5146 Acked 6145 Flush 0 Done Done New false
15154 5147 Acked 6146 Write 5 Done Done New false
15155 5148 Acked 6147 Read 10 Done Done New false
15156 5149 Acked 6148 Read 5 Done Done New false
15157 5150 Acked 6149 Write 4 Done Done New false
15158 5151 Acked 6150 Flush 0 Done Done New false
15159 5152 Acked 6151 Read 4 Done Done New false
15160 5153 Acked 6152 Write 7 Done Done New false
15161 5154 Acked 6153 Write 5 Done Done New false
15162 5155 Acked 6154 Read 5 Done Done New false
15163 5156 Acked 6155 Read 7 Done Done New false
15164 5157 Acked 6156 Read 8 Done Done New false
15165 5158 Acked 6157 Flush 0 Done Done New false
15166 5159 Acked 6158 Write 7 Done Done New false
15167 5160 Acked 6159 Write 4 Done Done New false
15168 5161 Acked 6160 Read 2 Done Done New false
15169 5162 Acked 6161 Read 9 Done Done New false
15170 5163 Acked 6162 Read 5 Done Done New false
15171 5164 Acked 6163 Read 8 Done Done New false
15172 5165 Acked 6164 Read 4 Done Done New false
15173 5166 Acked 6165 Write 5 Done Done New false
15174 5167 Acked 6166 Read 4 Done Done New false
15175 5168 Acked 6167 Read 4 Done Done New false
15176 5169 Acked 6168 Write 9 Done Done New false
15177 5170 Acked 6169 Write 4 Done Done New false
15178 5171 Acked 6170 Write 4 Done Done New false
15179 5172 Acked 6171 Flush 0 Done Done New false
15180 5173 Acked 6172 Read 10 Done Done New false
15181 5174 Acked 6173 Read 9 Done Done New false
15182 5175 Acked 6174 Flush 0 Done Done New false
15183 5176 Acked 6175 Read 5 Done Done New false
15184 5177 Acked 6176 Write 9 Done Done New false
15185 5178 Acked 6177 Write 4 Done Done New false
15186 5179 Acked 6178 Flush 0 Done Done New false
15187 5180 Acked 6179 Write 9 Done Done New false
15188 5181 Acked 6180 Read 6 Done Done New false
15189 5182 Acked 6181 Write 2 Done Done New false
15190 5183 Acked 6182 Read 10 Done Done New false
15191 5184 Acked 6183 Flush 0 Done Done New false
15192 5185 Acked 6184 Write 8 Done Done New false
15193 5186 Acked 6185 Write 3 Done Done New false
15194 5187 Acked 6186 Read 1 Done Done New false
15195 5188 Acked 6187 Write 8 Done Done New false
15196 5189 Acked 6188 Write 3 Done Done New false
15197 5190 Acked 6189 Write 1 Done Done New false
15198 5191 Acked 6190 Write 9 Done Done New false
15199 5192 Acked 6191 Read 9 Done Done New false
15200 5193 Acked 6192 Read 6 Done Done New false
15201 5194 Acked 6193 Read 2 Done Done New false
15202 5195 Acked 6194 Flush 0 Done Done New false
15203 5196 Acked 6195 Write 1 Done Done New false
15204 5197 Acked 6196 Read 2 Done Done New false
15205 5198 Acked 6197 Read 4 Done Done New false
15206 5199 Acked 6198 Read 3 Done Done New false
15207 5200 Acked 6199 Read 3 Done Done New false
15208 5201 Acked 6200 Write 8 Done Done New false
15209 5202 Acked 6201 Write 3 Done Done New false
15210 5203 Acked 6202 Read 2 Done Done New false
15211 5204 Acked 6203 Read 4 Done Done New false
15212 5205 Acked 6204 Read 8 Done Done New false
15213 5206 Acked 6205 Read 1 Done Done New false
15214 5207 Acked 6206 Read 9 Done Done New false
15215 5208 Acked 6207 Write 2 Done Done New false
15216 5209 Acked 6208 Write 6 Done Done New false
15217 5210 Acked 6209 Read 4 Done Done New false
15218 5211 Acked 6210 Read 6 Done Done New false
15219 5212 Acked 6211 Read 3 Done Done New false
15220 5213 Acked 6212 Write 6 Done Done New false
15221 5214 Acked 6213 Flush 0 Done Done New false
15222 5215 Acked 6214 Read 4 Done Done New false
15223 5216 Acked 6215 Read 10 Done Done New false
15224 5217 Acked 6216 Read 8 Done Done New false
15225 5218 Acked 6217 Write 4 Done Done New false
15226 5219 Acked 6218 Read 4 Done Done New false
15227 5220 Acked 6219 Read 7 Done Done New false
15228 5221 Acked 6220 Read 7 Done Done New false
15229 5222 Acked 6221 Read 8 Done Done New false
15230 5223 Acked 6222 Read 2 Done Done New false
15231 5224 Acked 6223 Write 7 Done Done New false
15232 5225 Acked 6224 Read 2 Done Done New false
15233 5226 Acked 6225 Read 10 Done Done New false
15234 5227 Acked 6226 Read 4 Done Done New false
15235 5228 Acked 6227 Write 3 Done Done New false
15236 5229 Acked 6228 Flush 0 Done Done New false
15237 5230 Acked 6229 Read 10 Done Done New false
15238 5231 Acked 6230 Write 1 Done Done New false
15239 5232 Acked 6231 Write 1 Done Done New false
15240 5233 Acked 6232 Flush 0 Done Done New false
15241 5234 Acked 6233 Read 3 Done Done New false
15242 5235 Acked 6234 Read 1 Done Done New false
15243 5236 Acked 6235 Flush 0 Done Done New false
15244 5237 Acked 6236 Read 8 Done Done New false
15245 5238 Acked 6237 Write 1 Done Done New false
15246 5239 Acked 6238 Read 8 Done Done New false
15247 5240 Acked 6239 Write 1 Done Done New false
15248 5241 Acked 6240 Read 2 Done Done New false
15249 5242 Acked 6241 Write 3 Done Done New false
15250 5243 Acked 6242 Write 9 Done Done New false
15251 5244 Acked 6243 Write 2 Done Done New false
15252 5245 Acked 6244 Read 4 Done Done New false
15253 5246 Acked 6245 Write 10 Done Done New false
15254 5247 Acked 6246 Write 6 Done Done New false
15255 5248 Acked 6247 Write 4 Done Done New false
15256 5249 Acked 6248 Write 4 Done Done New false
15257 5250 Acked 6249 Write 2 Done Done New false
15258 5251 Acked 6250 Write 7 Done Done New false
15259 5252 Acked 6251 Write 7 Done Done New false
15260 5253 Acked 6252 Read 10 Done Done New false
15261 5254 Acked 6253 Write 10 Done Done New false
15262 5255 Acked 6254 Read 2 Done Done New false
15263 5256 Acked 6255 Read 4 Done Done New false
15264 5257 Acked 6256 Read 9 Done Done New false
15265 5258 Acked 6257 Read 9 Done Done New false
15266 5259 Acked 6258 Flush 0 Done Done New false
15267 5260 Acked 6259 Write 3 Done Done New false
15268 5261 Acked 6260 Read 1 Done Done New false
15269 5262 Acked 6261 Write 8 Done Done New false
15270 5263 Acked 6262 Read 10 Done Done New false
15271 5264 Acked 6263 Flush 0 Done Done New false
15272 5265 Acked 6264 Read 10 Done Done New false
15273 5266 Acked 6265 Read 3 Done Done New false
15274 5267 Acked 6266 Read 5 Done Done New false
15275 5268 Acked 6267 Read 4 Done Done New false
15276 5269 Acked 6268 Write 1 Done Done New false
15277 5270 Acked 6269 Write 2 Done Done New false
15278 5271 Acked 6270 Write 1 Done Done New false
15279 5272 Acked 6271 Write 10 Done Done New false
15280 5273 Acked 6272 Write 3 Done Done New false
15281 5274 Acked 6273 Read 5 Done Done New false
15282 5275 Acked 6274 Write 8 Done Done New false
15283 5276 Acked 6275 Flush 0 Done Done New false
15284 5277 Acked 6276 Read 2 Done Done New false
15285 5278 Acked 6277 Read 5 Done Done New false
15286 5279 Acked 6278 Read 7 Done Done New false
15287 5280 Acked 6279 Write 6 Done Done New false
15288 5281 Acked 6280 Read 7 Done Done New false
15289 5282 Acked 6281 Read 1 Done Done New false
15290 5283 Acked 6282 Read 5 Done Done New false
15291 5284 Acked 6283 Read 6 Done Done New false
15292 5285 Acked 6284 Read 1 Done Done New false
15293 5286 Acked 6285 Write 5 Done Done New false
15294 5287 Acked 6286 Read 2 Done Done New false
15295 5288 Acked 6287 Read 1 Done Done New false
15296 5289 Acked 6288 Write 4 Done Done New false
15297 5290 Acked 6289 Write 7 Done Done New false
15298 5291 Acked 6290 Read 3 Done Done New false
15299 5292 Acked 6291 Flush 0 Done Done New false
15300 5293 Acked 6292 Read 5 Done Done New false
15301 5294 Acked 6293 Read 7 Done Done New false
15302 5295 Acked 6294 Write 9 Done Done New false
15303 5296 Acked 6295 Write 1 Done Done New false
15304 5297 Acked 6296 Read 7 Done Done New false
15305 5298 Acked 6297 Write 2 Done Done New false
15306 5299 Acked 6298 Write 3 Done Done New false
15307 5300 Acked 6299 Write 4 Done Done New false
15308 5301 Acked 6300 Read 2 Done Done New false
15309 5302 Acked 6301 Write 10 Done Done New false
15310 5303 Acked 6302 Read 1 Done Done New false
15311 5304 Acked 6303 Read 10 Done Done New false
15312 5305 Acked 6304 Write 1 Done Done New false
15313 5306 Acked 6305 Read 3 Done Done New false
15314 5307 Acked 6306 Write 3 Done Done New false
15315 5308 Acked 6307 Write 8 Done Done New false
15316 5309 Acked 6308 Read 2 Done Done New false
15317 5310 Acked 6309 Write 5 Done Done New false
15318 5311 Acked 6310 Write 10 Done Done New false
15319 5312 Acked 6311 Read 5 Done Done New false
15320 5313 Acked 6312 Write 1 Done Done New false
15321 5314 Acked 6313 Read 4 Done Done New false
15322 5315 Acked 6314 Read 5 Done Done New false
15323 5316 Acked 6315 Read 3 Done Done New false
15324 5317 Acked 6316 Write 2 Done Done New false
15325 5318 Acked 6317 Read 6 Done Done New false
15326 5319 Acked 6318 Read 6 Done Done New false
15327 5320 Acked 6319 Read 8 Done Done New false
15328 5321 Acked 6320 Flush 0 Done Done New false
15329 5322 Acked 6321 Read 5 Done Done New false
15330 5323 Acked 6322 Write 9 Done Done New false
15331 5324 Acked 6323 Read 1 Done Done New false
15332 5325 Acked 6324 Read 4 Done Done New false
15333 5326 Acked 6325 Read 9 Done Done New false
15334 5327 Acked 6326 Write 7 Done Done New false
15335 5328 Acked 6327 Write 5 Done Done New false
15336 5329 Acked 6328 Write 3 Done Done New false
15337 5330 Acked 6329 Read 2 Done Done New false
15338 5331 Acked 6330 Write 1 Done Done New false
15339 5332 Acked 6331 Read 5 Done Done New false
15340 5333 Acked 6332 Read 3 Done Done New false
15341 5334 Acked 6333 Read 2 Done Done New false
15342 5335 Acked 6334 Read 7 Done Done New false
15343 5336 Acked 6335 Read 6 Done Done New false
15344 5337 Acked 6336 Write 3 Done Done New false
15345 5338 Acked 6337 Write 5 Done Done New false
15346 5339 Acked 6338 Write 3 Done Done New false
15347 5340 Acked 6339 Write 2 Done Done New false
15348 5341 Acked 6340 Flush 0 Done Done New false
15349 5342 Acked 6341 Write 9 Done Done New false
15350 5343 Acked 6342 Read 9 Done Done New false
15351 5344 Acked 6343 Write 7 Done Done New false
15352 5345 Acked 6344 Read 1 Done Done New false
15353 5346 Acked 6345 Write 6 Done Done New false
15354 5347 Acked 6346 Read 6 Done Done New false
15355 5348 Acked 6347 Flush 0 Done Done New false
15356 5349 Acked 6348 Flush 0 Done Done New false
15357 5350 Acked 6349 Read 9 Done Done New false
15358 5351 Acked 6350 Write 8 Done Done New false
15359 5352 Acked 6351 Read 10 Done Done New false
15360 5353 Acked 6352 Write 5 Done Done New false
15361 5354 Acked 6353 Read 6 Done Done New false
15362 5355 Acked 6354 Read 1 Done Done New false
15363 5356 Acked 6355 Read 6 Done Done New false
15364 5357 Acked 6356 Read 7 Done Done New false
15365 5358 Acked 6357 Read 4 Done Done New false
15366 5359 Acked 6358 Read 10 Done Done New false
15367 5360 Acked 6359 Read 10 Done Done New false
15368 5361 Acked 6360 Read 10 Done Done New false
15369 5362 Acked 6361 Read 5 Done Done New false
15370 5363 Acked 6362 Read 1 Done Done New false
15371 5364 Acked 6363 Write 8 Done Done New false
15372 5365 Acked 6364 Write 6 Done Done New false
15373 5366 Acked 6365 Write 8 Done Done New false
15374 5367 Acked 6366 Write 10 Done Done New false
15375 5368 Acked 6367 Write 10 Done Done New false
15376 5369 Acked 6368 Read 10 Done Done New false
15377 5370 Acked 6369 Write 3 Done Done New false
15378 5371 Acked 6370 Write 3 Done Done New false
15379 5372 Acked 6371 Flush 0 Done Done New false
15380 5373 Acked 6372 Write 1 Done Done New false
15381 5374 Acked 6373 Flush 0 Done Done New false
15382 5375 Acked 6374 Write 2 Done Done New false
15383 5376 Acked 6375 Write 1 Done Done New false
15384 5377 Acked 6376 Write 9 Done Done New false
15385 5378 Acked 6377 Flush 0 Done Done New false
15386 5379 Acked 6378 Read 1 Done Done New false
15387 5380 Acked 6379 Write 6 Done Done New false
15388 5381 Acked 6380 Read 9 Done Done New false
15389 5382 Acked 6381 Read 8 Done Done New false
15390 5383 Acked 6382 Write 5 Done Done New false
15391 5384 Acked 6383 Read 4 Done Done New false
15392 5385 Acked 6384 Read 2 Done Done New false
15393 5386 Acked 6385 Write 7 Done Done New false
15394 5387 Acked 6386 Write 8 Done Done New false
15395 5388 Acked 6387 Flush 0 Done Done New false
15396 5389 Acked 6388 Flush 0 Done Done New false
15397 5390 Acked 6389 Read 7 Done Done New false
15398 5391 Acked 6390 Flush 0 Done Done New false
15399 5392 Acked 6391 Read 1 Done Done New false
15400 5393 Acked 6392 Write 10 Done Done New false
15401 5394 Acked 6393 Read 3 Done Done New false
15402 5395 Acked 6394 Flush 0 Done Done New false
15403 5396 Acked 6395 Write 6 Done Done New false
15404 5397 Acked 6396 Read 4 Done Done New false
15405 5398 Acked 6397 Read 1 Done Done New false
15406 5399 Acked 6398 Read 7 Done Done New false
15407 5400 Acked 6399 Read 6 Done Done New false
15408 5401 Acked 6400 Read 10 Done Done New false
15409 5402 Acked 6401 Read 3 Done Done New false
15410 5403 Acked 6402 Read 1 Done Done New false
15411 5404 Acked 6403 Read 8 Done Done New false
15412 5405 Acked 6404 Write 2 Done Done New false
15413 5406 Acked 6405 Read 7 Done Done New false
15414 5407 Acked 6406 Flush 0 Done Done New false
15415 5408 Acked 6407 Read 5 Done Done New false
15416 5409 Acked 6408 Read 8 Done Done New false
15417 5410 Acked 6409 Write 7 Done Done New false
15418 5411 Acked 6410 Flush 0 Done Done New false
15419 5412 Acked 6411 Read 6 Done Done New false
15420 5413 Acked 6412 Read 6 Done Done New false
15421 5414 Acked 6413 Write 10 Done Done New false
15422 5415 Acked 6414 Write 5 Done Done New false
15423 5416 Acked 6415 Write 2 Done Done New false
15424 5417 Acked 6416 Read 10 Done Done New false
15425 5418 Acked 6417 Flush 0 Done Done New false
15426 5419 Acked 6418 Read 5 Done Done New false
15427 5420 Acked 6419 Write 5 Done Done New false
15428 5421 Acked 6420 Flush 0 Done Done New false
15429 5422 Acked 6421 Read 10 Done Done New false
15430 5423 Acked 6422 Read 4 Done Done New false
15431 5424 Acked 6423 Flush 0 Done Done New false
15432 5425 Acked 6424 Write 8 Done Done New false
15433 5426 Acked 6425 Write 10 Done Done New false
15434 5427 Acked 6426 Write 10 Done Done New false
15435 5428 Acked 6427 Read 8 Done Done New false
15436 5429 Acked 6428 Write 4 Done Done New false
15437 5430 Acked 6429 Flush 0 Done Done New false
15438 5431 Acked 6430 Write 10 Done Done New false
15439 5432 Acked 6431 Write 10 Done Done New false
15440 5433 Acked 6432 Write 4 Done Done New false
15441 5434 Acked 6433 Read 10 Done Done New false
15442 5435 Acked 6434 Read 7 Done Done New false
15443 5436 Acked 6435 Flush 0 Done Done New false
15444 5437 Acked 6436 Read 10 Done Done New false
15445 5438 Acked 6437 Write 4 Done Done New false
15446 5439 Acked 6438 Write 2 Done Done New false
15447 5440 Acked 6439 Read 10 Done Done New false
15448 5441 Acked 6440 Write 4 Sent Sent New false
15449 STATES DS:0 DS:1 DS:2 TOTAL
15450 New 0 0 301 301
15451 Sent 1 1 0 2
15452 Done 300 300 0 600
15453 Skipped 0 0 0 0
15454 Error 0 0 0 0
15455 Last Flush: 6435 6435 6139
15456 Downstairs last five completed: 6139 6138 6137 6136 6135 6134
15457 Upstairs last five completed: 5441 5440 5439 5438 5437 5436
15458 CLIENT: Up:0 ds:301 act:2
15459 [8830] got start action
15460 [8830] I am going to restart
15461 Starting downstairs at port 8830
15462 Make output file at "/tmp/dsc/downstairs-8830.txt"
15463 [8830][2] reports Starting
15464 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1194)
15465 [8830][2] reports Running
154662023-10-13T04:45:07.899ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 2
154672023-10-13T04:45:07.899ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
154682023-10-13T04:45:07.899ZINFOcrucible: [2] client re-new 302 jobs since flush 6139 = downstairs
154692023-10-13T04:45:07.900ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
154702023-10-13T04:45:07.901ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
154712023-10-13T04:45:07.901ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
154722023-10-13T04:45:07.901ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
154732023-10-13T04:45:07.901ZINFOcrucible: [2] send last flush ID to this DS: 6139
154742023-10-13T04:45:07.901ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
154752023-10-13T04:45:07.901ZINFOcrucible: [2] Replied this last flush ID: 6139
154762023-10-13T04:45:07.901ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Offline ds_transition to Replay
154772023-10-13T04:45:07.901ZINFOcrucible: [2] Transition from Offline to Replay
154782023-10-13T04:45:07.901ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
154792023-10-13T04:45:07.901ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Replay ds_transition to Active
154802023-10-13T04:45:07.901ZINFOcrucible: [2] Transition from Replay to Active
154812023-10-13T04:45:07.901ZINFOcrucible: [2] Starts cmd_loop
154822023-10-13T04:45:07.901ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
154832023-10-13T04:45:07.901ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
154842023-10-13T04:45:07.901ZINFOcrucible: Set check for repair
154852023-10-13T04:45:08.902ZINFOcrucible: Checking if live repair is needed
154862023-10-13T04:45:08.902ZINFOcrucible: No Live Repair required at this time
154872023-10-13T04:45:08.902ZWARNcrucible: [2] flow control end
15488 ----------------------------------------------------------------
15489 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
15490 Downstairs last five completed: 6441 6440 6439 6438 6437 6436
15491 Upstairs last five completed: 5442 5441 5440 5439 5438 5437
15492 CLIENT: Up:0 ds:0 act:3
15493 Replay: All jobs finished, all DS active.
15494 Main task has work to do, go find it
15495 got dsc Stop(2)
15496 stop 2
15497 Oct 13 04:45:09.798 INFO request completed, latency_us: 498, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 5804a5f1-c29e-4296-bb43-fc7bd88651ec, remote_addr: 127.0.0.1:52046, [8830] Got stop action so:false kr:false
15498 local_addr: 127.0.0.1:9998
15499 Oct 13 04:45:09.799 INFO request completed, latency_us: 166, response_code: 200, uri: /state/cid/2, method: GET, req_id: 7a359181-214c-4901-af97-fa86993f3176, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
155002023-10-13T04:45:09.799ZWARNcrucible: [2] downstairs disconnected
155012023-10-13T04:45:09.799ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
155022023-10-13T04:45:09.799ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8830 closed looper = 2
155032023-10-13T04:45:09.799ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
155042023-10-13T04:45:09.799ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
155052023-10-13T04:45:09.799ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
155062023-10-13T04:45:09.799ZWARNcrucible: [2] pm_task rx.recv() is None
15507 [8830] Exited with: ExitStatus(unix_wait_status(9))
15508 [8830][2] reports Exit
15509 Oct 13 04:45:13.801 INFO request completed, latency_us: 256, response_code: 200, uri: /state/cid/2, method: GET, req_id: 0f7aadfb-468d-4dce-9064-cc4567cf06c5, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
15510 001/300 Read block 1563 len 24576
15511 002/300 Write block 4658 len 20480 data: 5 5 4 3 4
15512 003/300 Read block 118 len 12288
15513 004/300 Write block 3799 len 8192 data: 3 5
15514 005/300 Read block 244 len 8192
15515 006/300 Write block 64 len 16384 data: 5 5 5 5
15516 007/300 Flush
15517 008/300 Flush
15518 009/300 Flush
15519 010/300 Write block 2956 len 24576 data: 3 3 3 2 3 3
15520 011/300 Flush
15521 012/300 Read block 3828 len 12288
15522 013/300 Write block 2510 len 40960 data: 5 6 5 5 5 6 5 5 5 4
15523 014/300 Write block 3404 len 24576 data: 2 4 5 5 5 5
15524 015/300 Write block 663 len 40960 data: 4 5 7 7 7 7 4 3 3 4
15525 016/300 Write block 4757 len 36864 data: 6 7 7 6 5 5 4 3 4
15526 017/300 Read block 2677 len 12288
15527 018/300 Read block 23 len 8192
15528 019/300 Read block 707 len 36864
15529 020/300 Read block 3281 len 40960
15530 021/300 Read block 2061 len 4096
15531 022/300 Write block 1901 len 32768 data: 5 4 4 3 3 3 4 4
15532 023/300 Write block 3852 len 16384 data: 7 6 6 5
15533 024/300 Read block 3622 len 40960
15534 025/300 Write block 721 len 32768 data: 5 4 6 6 7 6 6 6
15535 026/300 Write block 2115 len 24576 data: 5 5 4 3 4 3
15536 027/300 Read block 2139 len 4096
15537 028/300 Flush
15538 029/300 Write block 2267 len 12288 data: 6 5 4
15539 030/300 Read block 2045 len 12288
15540 031/300 Flush
15541 032/300 Read block 952 len 40960
15542 033/300 Write block 359 len 16384 data: 2 2 3 3
15543 034/300 Flush
15544 035/300 Write block 3782 len 28672 data: 5 5 4 4 5 6 5
15545 036/300 Flush
15546 037/300 Flush
15547 038/300 Write block 1817 len 28672 data: 2 2 2 2 4 3 5
15548 039/300 Write block 3071 len 40960 data: 6 5 5 2 3 3 4 5 6 5
15549 040/300 Write block 1144 len 32768 data: 4 3 3 3 4 5 6 6
15550 041/300 Write block 4146 len 8192 data: 3 4
15551 042/300 Read block 1955 len 8192
15552 043/300 Write block 752 len 28672 data: 7 7 8 8 8 7 6
15553 044/300 Write block 1852 len 40960 data: 2 3 3 4 5 6 6 7 7 6
15554 045/300 Write block 2838 len 16384 data: 5 5 5 6
15555 046/300 Read block 3728 len 16384
15556 047/300 Write block 300 len 16384 data: 5 3 3 4
15557 048/300 Write block 2712 len 8192 data: 5 3
15558 049/300 Read block 1502 len 40960
15559 050/300 Write block 4041 len 12288 data: 2 2 2
15560 051/300 Read block 2481 len 24576
15561 052/300 Flush
15562 053/300 Write block 2189 len 36864 data: 3 4 4 4 4 4 5 5 4
15563 054/300 Read block 3455 len 20480
15564 055/300 Read block 1220 len 20480
15565 056/300 Read block 4258 len 24576
15566 057/300 Flush
15567 058/300 Flush
15568 059/300 Write block 1623 len 28672 data: 3 3 3 3 2 2 5
15569 060/300 Read block 2300 len 12288
15570 061/300 Flush
15571 062/300 Read block 3786 len 16384
15572 063/300 Flush
15573 064/300 Write block 2484 len 28672 data: 3 3 3 5 5 5 6
15574 065/300 Write block 190 len 28672 data: 5 5 4 4 4 4 6
15575 066/300 Read block 4950 len 40960
15576 067/300 Read block 2987 len 28672
15577 068/300 Read block 1507 len 4096
15578 069/300 Write block 264 len 32768 data: 2 2 3 3 3 4 4 3
15579 070/300 Read block 4373 len 40960
15580 071/300 Read block 4441 len 28672
15581 072/300 Read block 2372 len 16384
15582 073/300 Read block 2987 len 16384
15583 074/300 Read block 4115 len 32768
15584 075/300 Flush
15585 076/300 Flush
15586 077/300 Read block 418 len 36864
15587 078/300 Write block 4365 len 36864 data: 2 2 3 4 5 5 5 5 4
15588 079/300 Flush
15589 080/300 Write block 1267 len 12288 data: 2 2 3
15590 081/300 Flush
15591 082/300 Write block 4901 len 8192 data: 4 4
15592 083/300 Read block 2266 len 12288
15593 084/300 Write block 4863 len 16384 data: 6 7 5 6
15594 085/300 Read block 4797 len 20480
15595 086/300 Write block 3356 len 20480 data: 7 7 7 5 6
15596 087/300 Write block 1160 len 40960 data: 4 4 4 3 3 4 4 5 4 4
15597 088/300 Read block 3848 len 36864
15598 089/300 Read block 498 len 4096
15599 090/300 Write block 3697 len 28672 data: 4 6 6 5 5 5 5
15600 091/300 Read block 347 len 8192
15601 092/300 Read block 3816 len 40960
15602 093/300 Write block 1593 len 20480 data: 3 3 3 4 3
15603 094/300 Read block 2186 len 28672
15604 095/300 Read block 906 len 4096
15605 096/300 Read block 1672 len 40960
15606 097/300 Read block 4696 len 32768
15607 098/300 Write block 3357 len 4096 data: 8
15608 099/300 Write block 2802 len 20480 data: 4 3 3 3 3
15609 100/300 Read block 1542 len 20480
15610 101/300 Read block 3256 len 24576
15611 102/300 Write block 2919 len 40960 data: 3 4 6 6 6 6 6 6 6 6
15612 103/300 Read block 1130 len 16384
15613 104/300 Read block 4380 len 8192
15614 105/300 Read block 36 len 12288
15615 106/300 Read block 1571 len 4096
15616 107/300 Read block 3142 len 32768
15617 108/300 Write block 2997 len 4096 data: 2
15618 109/300 Read block 4268 len 28672
15619 110/300 Write block 3167 len 24576 data: 5 4 4 5 3 3
15620 111/300 Read block 2822 len 32768
15621 112/300 Write block 2948 len 36864 data: 3 3 5 4 5 3 3 3 4
15622 113/300 Read block 4138 len 36864
15623 114/300 Read block 4684 len 12288
15624 115/300 Write block 674 len 32768 data: 3 3 4 4 4 4 3 3
15625 116/300 Read block 4132 len 40960
15626 117/300 Write block 596 len 16384 data: 4 3 3 3
15627 118/300 Read block 1652 len 20480
15628 119/300 Write block 276 len 4096 data: 3
15629 120/300 Read block 820 len 32768
15630 121/300 Read block 349 len 36864
15631 122/300 Read block 1842 len 16384
15632 123/300 Read block 997 len 28672
15633 124/300 Read block 795 len 32768
15634 125/300 Flush
15635 126/300 Write block 201 len 36864 data: 3 3 4 4 4 5 6 7 7
15636 127/300 Write block 2329 len 16384 data: 5 5 3 3
15637 128/300 Write block 1983 len 32768 data: 2 2 2 2 2 2 3 3
15638 129/300 Write block 4094 len 16384 data: 2 2 3 3
15639 130/300 Write block 4491 len 24576 data: 3 4 3 3 4 4
15640 131/300 Read block 4954 len 36864
15641 132/300 Write block 757 len 40960 data: 8 7 5 5 4 4 4 3 3 3
15642 133/300 Write block 1362 len 28672 data: 4 4 4 4 4 3 3
15643 134/300 Read block 4290 len 28672
15644 135/300 Read block 714 len 28672
15645 136/300 Write block 1796 len 36864 data: 5 4 3 3 2 2 4 4 4
15646 137/300 Write block 3922 len 40960 data: 3 3 3 3 3 3 3 4 3 3
15647 138/300 Read block 3198 len 12288
15648 139/300 Write block 2210 len 16384 data: 3 3 4 4
15649 140/300 Read block 3674 len 8192
15650 141/300 Read block 602 len 16384
15651 142/300 Read block 2125 len 12288
15652 143/300 Write block 1213 len 40960 data: 3 3 3 3 3 2 3 2 2 2
15653 144/300 Write block 3113 len 16384 data: 4 4 5 6
15654 145/300 Write block 4930 len 20480 data: 2 2 2 3 3
15655 146/300 Read block 3023 len 24576
15656 147/300 Read block 2993 len 4096
15657 148/300 Write block 1462 len 40960 data: 2 2 4 4 4 4 4 4 4 4
15658 149/300 Read block 4476 len 28672
15659 150/300 Write block 3963 len 28672 data: 4 3 2 3 3 3 3
15660 151/300 Write block 2480 len 28672 data: 3 3 3 4 4 4 4
15661 152/300 Write block 4800 len 12288 data: 3 4 4
15662 153/300 Read block 1474 len 32768
15663 154/300 Flush
15664 155/300 Read block 2648 len 20480
15665 156/300 Write block 556 len 28672 data: 3 4 4 5 5 6 5
15666 157/300 Write block 2145 len 36864 data: 4 4 5 4 4 3 4 4 5
15667 158/300 Write block 2134 len 36864 data: 3 3 3 3 3 5 5 5 4
15668 159/300 Read block 4570 len 24576
15669 160/300 Write block 3080 len 12288 data: 6 5 4
15670 161/300 Read block 3572 len 20480
15671 162/300 Write block 1734 len 40960 data: 2 3 3 3 5 4 3 4 4 5
15672 163/300 Write block 2621 len 32768 data: 2 4 4 4 3 4 4 6
15673 164/300 Write block 3467 len 8192 data: 4 5
15674 165/300 Flush
15675 166/300 Flush
15676 167/300 Read block 273 len 32768
15677 168/300 Read block 2584 len 36864
15678 169/300 Flush
15679 170/300 Read block 3761 len 32768
15680 171/300 Flush
15681 172/300 Read block 2149 len 36864
15682 173/300 Read block 880 len 36864
15683 174/300 Read block 1273 len 12288
15684 175/300 Write block 1583 len 20480 data: 4 5 5 6 7
15685 176/300 Read block 4169 len 40960
15686 177/300 Read block 4618 len 12288
15687 178/300 Write block 33 len 16384 data: 5 5 4 2
15688 179/300 Flush
15689 180/300 Read block 3854 len 24576
15690 181/300 Write block 3301 len 24576 data: 2 2 2 4 3 4
15691 182/300 Read block 29 len 20480
15692 183/300 Write block 4660 len 4096 data: 5
15693 184/300 Flush
15694 185/300 Read block 4254 len 40960
15695 186/300 Read block 14 len 32768
15696 187/300 Write block 2228 len 16384 data: 2 2 2 2
15697 188/300 Write block 1778 len 28672 data: 5 5 4 3 3 4 3
15698 189/300 Write block 2246 len 28672 data: 4 4 4 3 3 3 2
15699 190/300 Write block 947 len 16384 data: 3 4 4 3
15700 191/300 Write block 1096 len 28672 data: 6 7 8 6 6 7 7
15701 192/300 Flush
15702 193/300 Write block 900 len 20480 data: 7 7 7 7 7
15703 194/300 Read block 4393 len 12288
15704 195/300 Read block 3181 len 8192
15705 196/300 Read block 843 len 4096
15706 197/300 Write block 3486 len 16384 data: 6 5 6 6
15707 198/300 Write block 2127 len 4096 data: 7
15708 199/300 Read block 897 len 16384
15709 200/300 Read block 193 len 32768
15710 201/300 Write block 2089 len 40960 data: 5 4 8 7 4 4 4 5 7 7
15711 202/300 Read block 230 len 40960
15712 203/300 Flush
15713 204/300 Read block 123 len 36864
15714 205/300 Read block 4777 len 20480
15715 206/300 Write block 4540 len 28672 data: 9 9 10 8 7 5 3
15716 207/300 Read block 2196 len 12288
15717 208/300 Write block 4397 len 36864 data: 3 3 3 2 4 4 4 5 4
15718 209/300 Read block 206 len 20480
15719 210/300 Flush
15720 211/300 Flush
15721 212/300 Read block 3675 len 16384
15722 213/300 Write block 2718 len 28672 data: 3 3 5 5 4 4 3
15723 214/300 Write block 3369 len 20480 data: 4 5 4 5 5
15724 215/300 Write block 1180 len 28672 data: 4 4 4 4 4 5 4
15725 216/300 Read block 2280 len 32768
15726 217/300 Write block 3396 len 4096 data: 7
15727 218/300 Write block 4794 len 12288 data: 4 3 2
15728 219/300 Write block 3050 len 36864 data: 2 2 2 2 2 3 3 4 4
15729 220/300 Read block 1904 len 40960
15730 221/300 Read block 357 len 40960
15731 222/300 Read block 294 len 28672
15732 223/300 Read block 3945 len 24576
15733 224/300 Read block 3612 len 20480
15734 225/300 Write block 4988 len 16384 data: 5 5 5 4
15735 226/300 Write block 3424 len 4096 data: 5
15736 227/300 Read block 2909 len 16384
15737 228/300 Flush
15738 229/300 Read block 449 len 36864
15739 230/300 Write block 2236 len 4096 data: 2
15740 231/300 Write block 4241 len 20480 data: 4 4 5 5 6
15741 232/300 Read block 740 len 8192
15742 233/300 Write block 1544 len 20480 data: 4 4 3 3 4
15743 234/300 Flush
15744 235/300 Write block 3623 len 8192 data: 3 3
15745 236/300 Read block 3324 len 24576
15746 237/300 Write block 2376 len 40960 data: 5 6 5 4 4 5 4 4 5 4
15747 238/300 Write block 831 len 24576 data: 4 3 2 2 4 5
15748 239/300 Read block 2911 len 20480
15749 240/300 Read block 1322 len 28672
15750 241/300 Read block 2935 len 12288
15751 242/300 Read block 4567 len 32768
15752 243/300 Read block 4529 len 28672
15753 244/300 Read block 3027 len 4096
15754 245/300 Read block 4237 len 36864
15755 246/300 Write block 2892 len 16384 data: 6 7 6 6
15756 247/300 Read block 4764 len 32768
15757 248/300 Write block 1643 len 24576 data: 4 5 5 5 4 4
15758 249/300 Read block 1531 len 4096
15759 250/300 Read block 127 len 28672
15760 251/300 Read block 4959 len 16384
15761 252/300 Write block 1641 len 8192 data: 4 5
15762 253/300 Read block 1549 len 40960
15763 254/300 Write block 3723 len 36864 data: 5 4 5 4 4 6 5 3 3
15764 255/300 Read block 3136 len 28672
15765 256/300 Read block 738 len 12288
15766 257/300 Read block 3769 len 24576
15767 258/300 Read block 2639 len 4096
15768 259/300 Read block 2305 len 32768
15769 260/300 Flush
15770 261/300 Write block 1853 len 24576 data: 4 4 5 6 7 7
15771 262/300 Read block 3820 len 20480
15772 263/300 Read block 1439 len 40960
15773 264/300 Write block 2030 len 12288 data: 5 4 4
15774 265/300 Read block 4537 len 16384
15775 266/300 Read block 581 len 8192
15776 267/300 Flush
15777 268/300 Write block 3519 len 20480 data: 3 3 4 5 5
15778 269/300 Write block 591 len 32768 data: 3 3 3 4 4 5 4 4
15779 270/300 Read block 1706 len 20480
15780 271/300 Read block 4761 len 20480
15781 272/300 Flush
15782 273/300 Write block 2288 len 28672 data: 6 5 5 4 5 5 6
15783 274/300 Read block 4976 len 4096
15784 275/300 Flush
15785 276/300 Write block 3044 len 40960 data: 5 4 3 3 4 3 3 3 3 3
15786 277/300 Flush
15787 278/300 Read block 1308 len 8192
15788 279/300 Read block 4093 len 4096
15789 280/300 Read block 3070 len 24576
15790 281/300 Read block 430 len 12288
15791 282/300 Read block 3333 len 40960
15792 283/300 Write block 4134 len 20480 data: 4 6 4 4 4
15793 284/300 Read block 2564 len 12288
15794 285/300 Read block 4012 len 28672
15795 286/300 Write block 3177 len 16384 data: 4 3 3 3
15796 287/300 Write block 245 len 28672 data: 2 2 2 4 4 4 4
15797 288/300 Write block 4593 len 4096 data: 2
15798 289/300 Read block 1327 len 16384
15799 290/300 Write block 1169 len 20480 data: 5 4 3 3 4
15800 291/300 Write block 3323 len 16384 data: 7 7 7 6
15801 292/300 Flush
15802 293/300 Write block 1886 len 12288 data: 4 5 4
15803 294/300 Write block 4778 len 40960 data: 6 6 6 5 6 6 4 5 5 5
15804 295/300 Write block 4094 len 28672 data: 3 3 4 4 4 6 6
15805 296/300 Read block 3564 len 28672
15806 297/300 Read block 930 len 4096
15807 298/300 Read block 15 len 20480
15808 299/300 Flush
15809 300/300 Write block 3823 len 4096 data: 2
15810 Main task has work to do, go find it
15811 got dsc Start(2)
15812 start 2
15813 Oct 13 04:45:14.720 INFO request completed, latency_us: 289, response_code: 204, uri: /start/cid/2, method: POST, req_id: 42e81f56-1454-4a0a-8579-1b5a85c0dc14, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
15814 Replay: started 2, returned:Ok(())
15815 ----------------------------------------------------------------
15816 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
15817 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
15818 5443 Acked 6442 Read 6 Done Done New false
15819 5444 Acked 6443 Write 5 Done Done New false
15820 5445 Acked 6444 Read 3 Done Done New false
15821 5446 Acked 6445 Write 2 Done Done New false
15822 5447 Acked 6446 Read 2 Done Done New false
15823 5448 Acked 6447 Write 4 Done Done New false
15824 5449 Acked 6448 Flush 0 Done Done New false
15825 5450 Acked 6449 Flush 0 Done Done New false
15826 5451 Acked 6450 Flush 0 Done Done New false
15827 5452 Acked 6451 Write 6 Done Done New false
15828 5453 Acked 6452 Flush 0 Done Done New false
15829 5454 Acked 6453 Read 3 Done Done New false
15830 5455 Acked 6454 Write 10 Done Done New false
15831 5456 Acked 6455 Write 6 Done Done New false
15832 5457 Acked 6456 Write 10 Done Done New false
15833 5458 Acked 6457 Write 9 Done Done New false
15834 5459 Acked 6458 Read 3 Done Done New false
15835 5460 Acked 6459 Read 2 Done Done New false
15836 5461 Acked 6460 Read 9 Done Done New false
15837 5462 Acked 6461 Read 10 Done Done New false
15838 5463 Acked 6462 Read 1 Done Done New false
15839 5464 Acked 6463 Write 8 Done Done New false
15840 5465 Acked 6464 Write 4 Done Done New false
15841 5466 Acked 6465 Read 10 Done Done New false
15842 5467 Acked 6466 Write 8 Done Done New false
15843 5468 Acked 6467 Write 6 Done Done New false
15844 5469 Acked 6468 Read 1 Done Done New false
15845 5470 Acked 6469 Flush 0 Done Done New false
15846 5471 Acked 6470 Write 3 Done Done New false
15847 5472 Acked 6471 Read 3 Done Done New false
15848 5473 Acked 6472 Flush 0 Done Done New false
15849 5474 Acked 6473 Read 10 Done Done New false
15850 5475 Acked 6474 Write 4 Done Done New false
15851 5476 Acked 6475 Flush 0 Done Done New false
15852 5477 Acked 6476 Write 7 Done Done New false
15853 5478 Acked 6477 Flush 0 Done Done New false
15854 5479 Acked 6478 Flush 0 Done Done New false
15855 5480 Acked 6479 Write 7 Done Done New false
15856 5481 Acked 6480 Write 10 Done Done New false
15857 5482 Acked 6481 Write 8 Done Done New false
15858 5483 Acked 6482 Write 2 Done Done New false
15859 5484 Acked 6483 Read 2 Done Done New false
15860 5485 Acked 6484 Write 7 Done Done New false
15861 5486 Acked 6485 Write 10 Done Done New false
15862 5487 Acked 6486 Write 4 Done Done New false
15863 5488 Acked 6487 Read 4 Done Done New false
15864 5489 Acked 6488 Write 4 Done Done New false
15865 5490 Acked 6489 Write 2 Done Done New false
15866 5491 Acked 6490 Read 10 Done Done New false
15867 5492 Acked 6491 Write 3 Done Done New false
15868 5493 Acked 6492 Read 6 Done Done New false
15869 5494 Acked 6493 Flush 0 Done Done New false
15870 5495 Acked 6494 Write 9 Done Done New false
15871 5496 Acked 6495 Read 5 Done Done New false
15872 5497 Acked 6496 Read 5 Done Done New false
15873 5498 Acked 6497 Read 6 Done Done New false
15874 5499 Acked 6498 Flush 0 Done Done New false
15875 5500 Acked 6499 Flush 0 Done Done New false
15876 5501 Acked 6500 Write 7 Done Done New false
15877 5502 Acked 6501 Read 3 Done Done New false
15878 5503 Acked 6502 Flush 0 Done Done New false
15879 5504 Acked 6503 Read 4 Done Done New false
15880 5505 Acked 6504 Flush 0 Done Done New false
15881 5506 Acked 6505 Write 7 Done Done New false
15882 5507 Acked 6506 Write 7 Done Done New false
15883 5508 Acked 6507 Read 10 Done Done New false
15884 5509 Acked 6508 Read 7 Done Done New false
15885 5510 Acked 6509 Read 1 Done Done New false
15886 5511 Acked 6510 Write 8 Done Done New false
15887 5512 Acked 6511 Read 10 Done Done New false
15888 5513 Acked 6512 Read 7 Done Done New false
15889 5514 Acked 6513 Read 4 Done Done New false
15890 5515 Acked 6514 Read 4 Done Done New false
15891 5516 Acked 6515 Read 8 Done Done New false
15892 5517 Acked 6516 Flush 0 Done Done New false
15893 5518 Acked 6517 Flush 0 Done Done New false
15894 5519 Acked 6518 Read 9 Done Done New false
15895 5520 Acked 6519 Write 9 Done Done New false
15896 5521 Acked 6520 Flush 0 Done Done New false
15897 5522 Acked 6521 Write 3 Done Done New false
15898 5523 Acked 6522 Flush 0 Done Done New false
15899 5524 Acked 6523 Write 2 Done Done New false
15900 5525 Acked 6524 Read 3 Done Done New false
15901 5526 Acked 6525 Write 4 Done Done New false
15902 5527 Acked 6526 Read 5 Done Done New false
15903 5528 Acked 6527 Write 5 Done Done New false
15904 5529 Acked 6528 Write 10 Done Done New false
15905 5530 Acked 6529 Read 9 Done Done New false
15906 5531 Acked 6530 Read 1 Done Done New false
15907 5532 Acked 6531 Write 7 Done Done New false
15908 5533 Acked 6532 Read 2 Done Done New false
15909 5534 Acked 6533 Read 10 Done Done New false
15910 5535 Acked 6534 Write 5 Done Done New false
15911 5536 Acked 6535 Read 7 Done Done New false
15912 5537 Acked 6536 Read 1 Done Done New false
15913 5538 Acked 6537 Read 10 Done Done New false
15914 5539 Acked 6538 Read 8 Done Done New false
15915 5540 Acked 6539 Write 1 Done Done New false
15916 5541 Acked 6540 Write 5 Done Done New false
15917 5542 Acked 6541 Read 5 Done Done New false
15918 5543 Acked 6542 Read 6 Done Done New false
15919 5544 Acked 6543 Write 10 Done Done New false
15920 5545 Acked 6544 Read 4 Done Done New false
15921 5546 Acked 6545 Read 2 Done Done New false
15922 5547 Acked 6546 Read 3 Done Done New false
15923 5548 Acked 6547 Read 1 Done Done New false
15924 5549 Acked 6548 Read 8 Done Done New false
15925 5550 Acked 6549 Write 1 Done Done New false
15926 5551 Acked 6550 Read 7 Done Done New false
15927 5552 Acked 6551 Write 6 Done Done New false
15928 5553 Acked 6552 Read 8 Done Done New false
15929 5554 Acked 6553 Write 9 Done Done New false
15930 5555 Acked 6554 Read 9 Done Done New false
15931 5556 Acked 6555 Read 3 Done Done New false
15932 5557 Acked 6556 Write 8 Done Done New false
15933 5558 Acked 6557 Read 10 Done Done New false
15934 5559 Acked 6558 Write 4 Done Done New false
15935 5560 Acked 6559 Read 5 Done Done New false
15936 5561 Acked 6560 Write 1 Done Done New false
15937 5562 Acked 6561 Read 8 Done Done New false
15938 5563 Acked 6562 Read 9 Done Done New false
15939 5564 Acked 6563 Read 4 Done Done New false
15940 5565 Acked 6564 Read 7 Done Done New false
15941 5566 Acked 6565 Read 8 Done Done New false
15942 5567 Acked 6566 Flush 0 Done Done New false
15943 5568 Acked 6567 Write 9 Done Done New false
15944 5569 Acked 6568 Write 4 Done Done New false
15945 5570 Acked 6569 Write 8 Done Done New false
15946 5571 Acked 6570 Write 4 Done Done New false
15947 5572 Acked 6571 Write 6 Done Done New false
15948 5573 Acked 6572 Read 9 Done Done New false
15949 5574 Acked 6573 Flush 0 Done Done New false
15950 5575 Acked 6574 Write 10 Done Done New false
15951 5576 Acked 6575 Write 7 Done Done New false
15952 5577 Acked 6576 Read 7 Done Done New false
15953 5578 Acked 6577 Read 7 Done Done New false
15954 5579 Acked 6578 Write 9 Done Done New false
15955 5580 Acked 6579 Write 10 Done Done New false
15956 5581 Acked 6580 Read 3 Done Done New false
15957 5582 Acked 6581 Write 4 Done Done New false
15958 5583 Acked 6582 Read 2 Done Done New false
15959 5584 Acked 6583 Read 4 Done Done New false
15960 5585 Acked 6584 Read 3 Done Done New false
15961 5586 Acked 6585 Write 10 Done Done New false
15962 5587 Acked 6586 Write 4 Done Done New false
15963 5588 Acked 6587 Write 5 Done Done New false
15964 5589 Acked 6588 Read 6 Done Done New false
15965 5590 Acked 6589 Read 1 Done Done New false
15966 5591 Acked 6590 Write 10 Done Done New false
15967 5592 Acked 6591 Read 7 Done Done New false
15968 5593 Acked 6592 Write 7 Done Done New false
15969 5594 Acked 6593 Write 7 Done Done New false
15970 5595 Acked 6594 Write 3 Done Done New false
15971 5596 Acked 6595 Read 8 Done Done New false
15972 5597 Acked 6596 Flush 0 Done Done New false
15973 5598 Acked 6597 Read 5 Done Done New false
15974 5599 Acked 6598 Write 7 Done Done New false
15975 5600 Acked 6599 Write 9 Done Done New false
15976 5601 Acked 6600 Write 9 Done Done New false
15977 5602 Acked 6601 Read 6 Done Done New false
15978 5603 Acked 6602 Write 3 Done Done New false
15979 5604 Acked 6603 Read 5 Done Done New false
15980 5605 Acked 6604 Write 10 Done Done New false
15981 5606 Acked 6605 Write 8 Done Done New false
15982 5607 Acked 6606 Write 2 Done Done New false
15983 5608 Acked 6607 Flush 0 Done Done New false
15984 5609 Acked 6608 Flush 0 Done Done New false
15985 5610 Acked 6609 Read 8 Done Done New false
15986 5611 Acked 6610 Read 9 Done Done New false
15987 5612 Acked 6611 Flush 0 Done Done New false
15988 5613 Acked 6612 Read 8 Done Done New false
15989 5614 Acked 6613 Flush 0 Done Done New false
15990 5615 Acked 6614 Read 9 Done Done New false
15991 5616 Acked 6615 Read 9 Done Done New false
15992 5617 Acked 6616 Read 3 Done Done New false
15993 5618 Acked 6617 Write 5 Done Done New false
15994 5619 Acked 6618 Read 10 Done Done New false
15995 5620 Acked 6619 Read 3 Done Done New false
15996 5621 Acked 6620 Write 4 Done Done New false
15997 5622 Acked 6621 Flush 0 Done Done New false
15998 5623 Acked 6622 Read 6 Done Done New false
15999 5624 Acked 6623 Write 6 Done Done New false
16000 5625 Acked 6624 Read 5 Done Done New false
16001 5626 Acked 6625 Write 1 Done Done New false
16002 5627 Acked 6626 Flush 0 Done Done New false
16003 5628 Acked 6627 Read 10 Done Done New false
16004 5629 Acked 6628 Read 8 Done Done New false
16005 5630 Acked 6629 Write 4 Done Done New false
16006 5631 Acked 6630 Write 7 Done Done New false
16007 5632 Acked 6631 Write 7 Done Done New false
16008 5633 Acked 6632 Write 4 Done Done New false
16009 5634 Acked 6633 Write 7 Done Done New false
16010 5635 Acked 6634 Flush 0 Done Done New false
16011 5636 Acked 6635 Write 5 Done Done New false
16012 5637 Acked 6636 Read 3 Done Done New false
16013 5638 Acked 6637 Read 2 Done Done New false
16014 5639 Acked 6638 Read 1 Done Done New false
16015 5640 Acked 6639 Write 4 Done Done New false
16016 5641 Acked 6640 Write 1 Done Done New false
16017 5642 Acked 6641 Read 4 Done Done New false
16018 5643 Acked 6642 Read 8 Done Done New false
16019 5644 Acked 6643 Write 10 Done Done New false
16020 5645 Acked 6644 Read 10 Done Done New false
16021 5646 Acked 6645 Flush 0 Done Done New false
16022 5647 Acked 6646 Read 9 Done Done New false
16023 5648 Acked 6647 Read 5 Done Done New false
16024 5649 Acked 6648 Write 7 Done Done New false
16025 5650 Acked 6649 Read 3 Done Done New false
16026 5651 Acked 6650 Write 9 Done Done New false
16027 5652 Acked 6651 Read 5 Done Done New false
16028 5653 Acked 6652 Flush 0 Done Done New false
16029 5654 Acked 6653 Flush 0 Done Done New false
16030 5655 Acked 6654 Read 4 Done Done New false
16031 5656 Acked 6655 Write 7 Done Done New false
16032 5657 Acked 6656 Write 5 Done Done New false
16033 5658 Acked 6657 Write 7 Done Done New false
16034 5659 Acked 6658 Read 8 Done Done New false
16035 5660 Acked 6659 Write 1 Done Done New false
16036 5661 Acked 6660 Write 3 Done Done New false
16037 5662 Acked 6661 Write 9 Done Done New false
16038 5663 Acked 6662 Read 10 Done Done New false
16039 5664 Acked 6663 Read 10 Done Done New false
16040 5665 Acked 6664 Read 7 Done Done New false
16041 5666 Acked 6665 Read 6 Done Done New false
16042 5667 Acked 6666 Read 5 Done Done New false
16043 5668 Acked 6667 Write 4 Done Done New false
16044 5669 Acked 6668 Write 1 Done Done New false
16045 5670 Acked 6669 Read 4 Done Done New false
16046 5671 Acked 6670 Flush 0 Done Done New false
16047 5672 Acked 6671 Read 9 Done Done New false
16048 5673 Acked 6672 Write 1 Done Done New false
16049 5674 Acked 6673 Write 5 Done Done New false
16050 5675 Acked 6674 Read 2 Done Done New false
16051 5676 Acked 6675 Write 5 Done Done New false
16052 5677 Acked 6676 Flush 0 Done Done New false
16053 5678 Acked 6677 Write 2 Done Done New false
16054 5679 Acked 6678 Read 6 Done Done New false
16055 5680 Acked 6679 Write 10 Done Done New false
16056 5681 Acked 6680 Write 6 Done Done New false
16057 5682 Acked 6681 Read 5 Done Done New false
16058 5683 Acked 6682 Read 7 Done Done New false
16059 5684 Acked 6683 Read 3 Done Done New false
16060 5685 Acked 6684 Read 8 Done Done New false
16061 5686 Acked 6685 Read 7 Done Done New false
16062 5687 Acked 6686 Read 1 Done Done New false
16063 5688 Acked 6687 Read 9 Done Done New false
16064 5689 Acked 6688 Write 4 Done Done New false
16065 5690 Acked 6689 Read 8 Done Done New false
16066 5691 Acked 6690 Write 6 Done Done New false
16067 5692 Acked 6691 Read 1 Done Done New false
16068 5693 Acked 6692 Read 7 Done Done New false
16069 5694 Acked 6693 Read 4 Done Done New false
16070 5695 Acked 6694 Write 2 Done Done New false
16071 5696 Acked 6695 Read 10 Done Done New false
16072 5697 Acked 6696 Write 9 Done Done New false
16073 5698 Acked 6697 Read 7 Done Done New false
16074 5699 Acked 6698 Read 3 Done Done New false
16075 5700 Acked 6699 Read 6 Done Done New false
16076 5701 Acked 6700 Read 1 Done Done New false
16077 5702 Acked 6701 Read 8 Done Done New false
16078 5703 Acked 6702 Flush 0 Done Done New false
16079 5704 Acked 6703 Write 6 Done Done New false
16080 5705 Acked 6704 Read 5 Done Done New false
16081 5706 Acked 6705 Read 10 Done Done New false
16082 5707 Acked 6706 Write 3 Done Done New false
16083 5708 Acked 6707 Read 4 Done Done New false
16084 5709 Acked 6708 Read 2 Done Done New false
16085 5710 Acked 6709 Flush 0 Done Done New false
16086 5711 Acked 6710 Write 5 Done Done New false
16087 5712 Acked 6711 Write 8 Done Done New false
16088 5713 Acked 6712 Read 5 Done Done New false
16089 5714 Acked 6713 Read 5 Done Done New false
16090 5715 Acked 6714 Flush 0 Done Done New false
16091 5716 Acked 6715 Write 7 Done Done New false
16092 5717 Acked 6716 Read 1 Done Done New false
16093 5718 Acked 6717 Flush 0 Done Done New false
16094 5719 Acked 6718 Write 10 Done Done New false
16095 5720 Acked 6719 Flush 0 Done Done New false
16096 5721 Acked 6720 Read 2 Done Done New false
16097 5722 Acked 6721 Read 1 Done Done New false
16098 5723 Acked 6722 Read 6 Done Done New false
16099 5724 Acked 6723 Read 3 Done Done New false
16100 5725 Acked 6724 Read 10 Done Done New false
16101 5726 Acked 6725 Write 5 Done Done New false
16102 5727 Acked 6726 Read 3 Done Done New false
16103 5728 Acked 6727 Read 7 Done Done New false
16104 5729 Acked 6728 Write 4 Done Done New false
16105 5730 Acked 6729 Write 7 Done Done New false
16106 5731 Acked 6730 Write 1 Done Done New false
16107 5732 Acked 6731 Read 4 Done Done New false
16108 5733 Acked 6732 Write 5 Done Done New false
16109 5734 Acked 6733 Write 4 Done Done New false
16110 5735 Acked 6734 Flush 0 Done Done New false
16111 5736 Acked 6735 Write 3 Done Done New false
16112 5737 Acked 6736 Write 10 Done Done New false
16113 5738 Acked 6737 Write 7 Done Done New false
16114 5739 Acked 6738 Read 7 Done Done New false
16115 5740 Acked 6739 Flush 0 Done Done New false
16116 5741 Acked 6740 Read 1 Done Done New false
16117 5742 Acked 6741 Read 5 Done Done New false
16118 5743 Acked 6742 Flush 0 Done Done New false
16119 5744 Acked 6743 Write 1 Sent Sent New false
16120 STATES DS:0 DS:1 DS:2 TOTAL
16121 New 0 0 302 302
16122 Sent 1 1 0 2
16123 Done 301 301 0 602
16124 Skipped 0 0 0 0
16125 Error 0 0 0 0
16126 Last Flush: 6742 6742 6441
16127 Downstairs last five completed: 6441 6440 6439 6438 6437 6436
16128 Upstairs last five completed: 5744 5743 5742 5741 5740 5739
16129 CLIENT: Up:0 ds:302 act:2
161302023-10-13T04:45:16.808ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
16131 [8830] got start action
16132 [8830] I am going to restart
16133 Starting downstairs at port 8830
16134 Make output file at "/tmp/dsc/downstairs-8830.txt"
16135 [8830][2] reports Starting
16136 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1195)
16137 [8830][2] reports Running
16138 ----------------------------------------------------------------
16139 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
16140 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
16141 5443 Acked 6442 Read 6 Done Done New false
16142 5444 Acked 6443 Write 5 Done Done New false
16143 5445 Acked 6444 Read 3 Done Done New false
16144 5446 Acked 6445 Write 2 Done Done New false
16145 5447 Acked 6446 Read 2 Done Done New false
16146 5448 Acked 6447 Write 4 Done Done New false
16147 5449 Acked 6448 Flush 0 Done Done New false
16148 5450 Acked 6449 Flush 0 Done Done New false
16149 5451 Acked 6450 Flush 0 Done Done New false
16150 5452 Acked 6451 Write 6 Done Done New false
16151 5453 Acked 6452 Flush 0 Done Done New false
16152 5454 Acked 6453 Read 3 Done Done New false
16153 5455 Acked 6454 Write 10 Done Done New false
16154 5456 Acked 6455 Write 6 Done Done New false
16155 5457 Acked 6456 Write 10 Done Done New false
16156 5458 Acked 6457 Write 9 Done Done New false
16157 5459 Acked 6458 Read 3 Done Done New false
16158 5460 Acked 6459 Read 2 Done Done New false
16159 5461 Acked 6460 Read 9 Done Done New false
16160 5462 Acked 6461 Read 10 Done Done New false
16161 5463 Acked 6462 Read 1 Done Done New false
16162 5464 Acked 6463 Write 8 Done Done New false
16163 5465 Acked 6464 Write 4 Done Done New false
16164 5466 Acked 6465 Read 10 Done Done New false
16165 5467 Acked 6466 Write 8 Done Done New false
16166 5468 Acked 6467 Write 6 Done Done New false
16167 5469 Acked 6468 Read 1 Done Done New false
16168 5470 Acked 6469 Flush 0 Done Done New false
16169 5471 Acked 6470 Write 3 Done Done New false
16170 5472 Acked 6471 Read 3 Done Done New false
16171 5473 Acked 6472 Flush 0 Done Done New false
16172 5474 Acked 6473 Read 10 Done Done New false
16173 5475 Acked 6474 Write 4 Done Done New false
16174 5476 Acked 6475 Flush 0 Done Done New false
16175 5477 Acked 6476 Write 7 Done Done New false
16176 5478 Acked 6477 Flush 0 Done Done New false
16177 5479 Acked 6478 Flush 0 Done Done New false
16178 5480 Acked 6479 Write 7 Done Done New false
16179 5481 Acked 6480 Write 10 Done Done New false
16180 5482 Acked 6481 Write 8 Done Done New false
16181 5483 Acked 6482 Write 2 Done Done New false
16182 5484 Acked 6483 Read 2 Done Done New false
16183 5485 Acked 6484 Write 7 Done Done New false
16184 5486 Acked 6485 Write 10 Done Done New false
16185 5487 Acked 6486 Write 4 Done Done New false
16186 5488 Acked 6487 Read 4 Done Done New false
16187 5489 Acked 6488 Write 4 Done Done New false
16188 5490 Acked 6489 Write 2 Done Done New false
16189 5491 Acked 6490 Read 10 Done Done New false
16190 5492 Acked 6491 Write 3 Done Done New false
16191 5493 Acked 6492 Read 6 Done Done New false
16192 5494 Acked 6493 Flush 0 Done Done New false
16193 5495 Acked 6494 Write 9 Done Done New false
16194 5496 Acked 6495 Read 5 Done Done New false
16195 5497 Acked 6496 Read 5 Done Done New false
16196 5498 Acked 6497 Read 6 Done Done New false
16197 5499 Acked 6498 Flush 0 Done Done New false
16198 5500 Acked 6499 Flush 0 Done Done New false
16199 5501 Acked 6500 Write 7 Done Done New false
16200 5502 Acked 6501 Read 3 Done Done New false
16201 5503 Acked 6502 Flush 0 Done Done New false
16202 5504 Acked 6503 Read 4 Done Done New false
16203 5505 Acked 6504 Flush 0 Done Done New false
16204 5506 Acked 6505 Write 7 Done Done New false
16205 5507 Acked 6506 Write 7 Done Done New false
16206 5508 Acked 6507 Read 10 Done Done New false
16207 5509 Acked 6508 Read 7 Done Done New false
16208 5510 Acked 6509 Read 1 Done Done New false
16209 5511 Acked 6510 Write 8 Done Done New false
16210 5512 Acked 6511 Read 10 Done Done New false
16211 5513 Acked 6512 Read 7 Done Done New false
16212 5514 Acked 6513 Read 4 Done Done New false
16213 5515 Acked 6514 Read 4 Done Done New false
16214 5516 Acked 6515 Read 8 Done Done New false
16215 5517 Acked 6516 Flush 0 Done Done New false
16216 5518 Acked 6517 Flush 0 Done Done New false
16217 5519 Acked 6518 Read 9 Done Done New false
16218 5520 Acked 6519 Write 9 Done Done New false
16219 5521 Acked 6520 Flush 0 Done Done New false
16220 5522 Acked 6521 Write 3 Done Done New false
16221 5523 Acked 6522 Flush 0 Done Done New false
16222 5524 Acked 6523 Write 2 Done Done New false
16223 5525 Acked 6524 Read 3 Done Done New false
16224 5526 Acked 6525 Write 4 Done Done New false
16225 5527 Acked 6526 Read 5 Done Done New false
16226 5528 Acked 6527 Write 5 Done Done New false
16227 5529 Acked 6528 Write 10 Done Done New false
16228 5530 Acked 6529 Read 9 Done Done New false
16229 5531 Acked 6530 Read 1 Done Done New false
16230 5532 Acked 6531 Write 7 Done Done New false
16231 5533 Acked 6532 Read 2 Done Done New false
16232 5534 Acked 6533 Read 10 Done Done New false
16233 5535 Acked 6534 Write 5 Done Done New false
16234 5536 Acked 6535 Read 7 Done Done New false
16235 5537 Acked 6536 Read 1 Done Done New false
16236 5538 Acked 6537 Read 10 Done Done New false
16237 5539 Acked 6538 Read 8 Done Done New false
16238 5540 Acked 6539 Write 1 Done Done New false
16239 5541 Acked 6540 Write 5 Done Done New false
16240 5542 Acked 6541 Read 5 Done Done New false
16241 5543 Acked 6542 Read 6 Done Done New false
16242 5544 Acked 6543 Write 10 Done Done New false
16243 5545 Acked 6544 Read 4 Done Done New false
16244 5546 Acked 6545 Read 2 Done Done New false
16245 5547 Acked 6546 Read 3 Done Done New false
16246 5548 Acked 6547 Read 1 Done Done New false
16247 5549 Acked 6548 Read 8 Done Done New false
16248 5550 Acked 6549 Write 1 Done Done New false
16249 5551 Acked 6550 Read 7 Done Done New false
16250 5552 Acked 6551 Write 6 Done Done New false
16251 5553 Acked 6552 Read 8 Done Done New false
16252 5554 Acked 6553 Write 9 Done Done New false
16253 5555 Acked 6554 Read 9 Done Done New false
16254 5556 Acked 6555 Read 3 Done Done New false
16255 5557 Acked 6556 Write 8 Done Done New false
16256 5558 Acked 6557 Read 10 Done Done New false
16257 5559 Acked 6558 Write 4 Done Done New false
16258 5560 Acked 6559 Read 5 Done Done New false
16259 5561 Acked 6560 Write 1 Done Done New false
16260 5562 Acked 6561 Read 8 Done Done New false
16261 5563 Acked 6562 Read 9 Done Done New false
16262 5564 Acked 6563 Read 4 Done Done New false
16263 5565 Acked 6564 Read 7 Done Done New false
16264 5566 Acked 6565 Read 8 Done Done New false
16265 5567 Acked 6566 Flush 0 Done Done New false
16266 5568 Acked 6567 Write 9 Done Done New false
16267 5569 Acked 6568 Write 4 Done Done New false
16268 5570 Acked 6569 Write 8 Done Done New false
16269 5571 Acked 6570 Write 4 Done Done New false
16270 5572 Acked 6571 Write 6 Done Done New false
16271 5573 Acked 6572 Read 9 Done Done New false
16272 5574 Acked 6573 Flush 0 Done Done New false
16273 5575 Acked 6574 Write 10 Done Done New false
16274 5576 Acked 6575 Write 7 Done Done New false
16275 5577 Acked 6576 Read 7 Done Done New false
16276 5578 Acked 6577 Read 7 Done Done New false
16277 5579 Acked 6578 Write 9 Done Done New false
16278 5580 Acked 6579 Write 10 Done Done New false
16279 5581 Acked 6580 Read 3 Done Done New false
16280 5582 Acked 6581 Write 4 Done Done New false
16281 5583 Acked 6582 Read 2 Done Done New false
16282 5584 Acked 6583 Read 4 Done Done New false
16283 5585 Acked 6584 Read 3 Done Done New false
16284 5586 Acked 6585 Write 10 Done Done New false
16285 5587 Acked 6586 Write 4 Done Done New false
16286 5588 Acked 6587 Write 5 Done Done New false
16287 5589 Acked 6588 Read 6 Done Done New false
16288 5590 Acked 6589 Read 1 Done Done New false
16289 5591 Acked 6590 Write 10 Done Done New false
16290 5592 Acked 6591 Read 7 Done Done New false
16291 5593 Acked 6592 Write 7 Done Done New false
16292 5594 Acked 6593 Write 7 Done Done New false
16293 5595 Acked 6594 Write 3 Done Done New false
16294 5596 Acked 6595 Read 8 Done Done New false
16295 5597 Acked 6596 Flush 0 Done Done New false
16296 5598 Acked 6597 Read 5 Done Done New false
16297 5599 Acked 6598 Write 7 Done Done New false
16298 5600 Acked 6599 Write 9 Done Done New false
16299 5601 Acked 6600 Write 9 Done Done New false
16300 5602 Acked 6601 Read 6 Done Done New false
16301 5603 Acked 6602 Write 3 Done Done New false
16302 5604 Acked 6603 Read 5 Done Done New false
16303 5605 Acked 6604 Write 10 Done Done New false
16304 5606 Acked 6605 Write 8 Done Done New false
16305 5607 Acked 6606 Write 2 Done Done New false
16306 5608 Acked 6607 Flush 0 Done Done New false
16307 5609 Acked 6608 Flush 0 Done Done New false
16308 5610 Acked 6609 Read 8 Done Done New false
16309 5611 Acked 6610 Read 9 Done Done New false
16310 5612 Acked 6611 Flush 0 Done Done New false
16311 5613 Acked 6612 Read 8 Done Done New false
16312 5614 Acked 6613 Flush 0 Done Done New false
16313 5615 Acked 6614 Read 9 Done Done New false
16314 5616 Acked 6615 Read 9 Done Done New false
16315 5617 Acked 6616 Read 3 Done Done New false
16316 5618 Acked 6617 Write 5 Done Done New false
16317 5619 Acked 6618 Read 10 Done Done New false
16318 5620 Acked 6619 Read 3 Done Done New false
16319 5621 Acked 6620 Write 4 Done Done New false
16320 5622 Acked 6621 Flush 0 Done Done New false
16321 5623 Acked 6622 Read 6 Done Done New false
16322 5624 Acked 6623 Write 6 Done Done New false
16323 5625 Acked 6624 Read 5 Done Done New false
16324 5626 Acked 6625 Write 1 Done Done New false
16325 5627 Acked 6626 Flush 0 Done Done New false
16326 5628 Acked 6627 Read 10 Done Done New false
16327 5629 Acked 6628 Read 8 Done Done New false
16328 5630 Acked 6629 Write 4 Done Done New false
16329 5631 Acked 6630 Write 7 Done Done New false
16330 5632 Acked 6631 Write 7 Done Done New false
16331 5633 Acked 6632 Write 4 Done Done New false
16332 5634 Acked 6633 Write 7 Done Done New false
16333 5635 Acked 6634 Flush 0 Done Done New false
16334 5636 Acked 6635 Write 5 Done Done New false
16335 5637 Acked 6636 Read 3 Done Done New false
16336 5638 Acked 6637 Read 2 Done Done New false
16337 5639 Acked 6638 Read 1 Done Done New false
16338 5640 Acked 6639 Write 4 Done Done New false
16339 5641 Acked 6640 Write 1 Done Done New false
16340 5642 Acked 6641 Read 4 Done Done New false
16341 5643 Acked 6642 Read 8 Done Done New false
16342 5644 Acked 6643 Write 10 Done Done New false
16343 5645 Acked 6644 Read 10 Done Done New false
16344 5646 Acked 6645 Flush 0 Done Done New false
16345 5647 Acked 6646 Read 9 Done Done New false
16346 5648 Acked 6647 Read 5 Done Done New false
16347 5649 Acked 6648 Write 7 Done Done New false
16348 5650 Acked 6649 Read 3 Done Done New false
16349 5651 Acked 6650 Write 9 Done Done New false
16350 5652 Acked 6651 Read 5 Done Done New false
16351 5653 Acked 6652 Flush 0 Done Done New false
16352 5654 Acked 6653 Flush 0 Done Done New false
16353 5655 Acked 6654 Read 4 Done Done New false
16354 5656 Acked 6655 Write 7 Done Done New false
16355 5657 Acked 6656 Write 5 Done Done New false
16356 5658 Acked 6657 Write 7 Done Done New false
16357 5659 Acked 6658 Read 8 Done Done New false
16358 5660 Acked 6659 Write 1 Done Done New false
16359 5661 Acked 6660 Write 3 Done Done New false
16360 5662 Acked 6661 Write 9 Done Done New false
16361 5663 Acked 6662 Read 10 Done Done New false
16362 5664 Acked 6663 Read 10 Done Done New false
16363 5665 Acked 6664 Read 7 Done Done New false
16364 5666 Acked 6665 Read 6 Done Done New false
16365 5667 Acked 6666 Read 5 Done Done New false
16366 5668 Acked 6667 Write 4 Done Done New false
16367 5669 Acked 6668 Write 1 Done Done New false
16368 5670 Acked 6669 Read 4 Done Done New false
16369 5671 Acked 6670 Flush 0 Done Done New false
16370 5672 Acked 6671 Read 9 Done Done New false
16371 5673 Acked 6672 Write 1 Done Done New false
16372 5674 Acked 6673 Write 5 Done Done New false
16373 5675 Acked 6674 Read 2 Done Done New false
16374 5676 Acked 6675 Write 5 Done Done New false
16375 5677 Acked 6676 Flush 0 Done Done New false
16376 5678 Acked 6677 Write 2 Done Done New false
16377 5679 Acked 6678 Read 6 Done Done New false
16378 5680 Acked 6679 Write 10 Done Done New false
16379 5681 Acked 6680 Write 6 Done Done New false
16380 5682 Acked 6681 Read 5 Done Done New false
16381 5683 Acked 6682 Read 7 Done Done New false
16382 5684 Acked 6683 Read 3 Done Done New false
16383 5685 Acked 6684 Read 8 Done Done New false
16384 5686 Acked 6685 Read 7 Done Done New false
16385 5687 Acked 6686 Read 1 Done Done New false
16386 5688 Acked 6687 Read 9 Done Done New false
16387 5689 Acked 6688 Write 4 Done Done New false
16388 5690 Acked 6689 Read 8 Done Done New false
16389 5691 Acked 6690 Write 6 Done Done New false
16390 5692 Acked 6691 Read 1 Done Done New false
16391 5693 Acked 6692 Read 7 Done Done New false
16392 5694 Acked 6693 Read 4 Done Done New false
16393 5695 Acked 6694 Write 2 Done Done New false
16394 5696 Acked 6695 Read 10 Done Done New false
16395 5697 Acked 6696 Write 9 Done Done New false
16396 5698 Acked 6697 Read 7 Done Done New false
16397 5699 Acked 6698 Read 3 Done Done New false
16398 5700 Acked 6699 Read 6 Done Done New false
16399 5701 Acked 6700 Read 1 Done Done New false
16400 5702 Acked 6701 Read 8 Done Done New false
16401 5703 Acked 6702 Flush 0 Done Done New false
16402 5704 Acked 6703 Write 6 Done Done New false
16403 5705 Acked 6704 Read 5 Done Done New false
16404 5706 Acked 6705 Read 10 Done Done New false
16405 5707 Acked 6706 Write 3 Done Done New false
16406 5708 Acked 6707 Read 4 Done Done New false
16407 5709 Acked 6708 Read 2 Done Done New false
16408 5710 Acked 6709 Flush 0 Done Done New false
16409 5711 Acked 6710 Write 5 Done Done New false
16410 5712 Acked 6711 Write 8 Done Done New false
16411 5713 Acked 6712 Read 5 Done Done New false
16412 5714 Acked 6713 Read 5 Done Done New false
16413 5715 Acked 6714 Flush 0 Done Done New false
16414 5716 Acked 6715 Write 7 Done Done New false
16415 5717 Acked 6716 Read 1 Done Done New false
16416 5718 Acked 6717 Flush 0 Done Done New false
16417 5719 Acked 6718 Write 10 Done Done New false
16418 5720 Acked 6719 Flush 0 Done Done New false
16419 5721 Acked 6720 Read 2 Done Done New false
16420 5722 Acked 6721 Read 1 Done Done New false
16421 5723 Acked 6722 Read 6 Done Done New false
16422 5724 Acked 6723 Read 3 Done Done New false
16423 5725 Acked 6724 Read 10 Done Done New false
16424 5726 Acked 6725 Write 5 Done Done New false
16425 5727 Acked 6726 Read 3 Done Done New false
16426 5728 Acked 6727 Read 7 Done Done New false
16427 5729 Acked 6728 Write 4 Done Done New false
16428 5730 Acked 6729 Write 7 Done Done New false
16429 5731 Acked 6730 Write 1 Done Done New false
16430 5732 Acked 6731 Read 4 Done Done New false
16431 5733 Acked 6732 Write 5 Done Done New false
16432 5734 Acked 6733 Write 4 Done Done New false
16433 5735 Acked 6734 Flush 0 Done Done New false
16434 5736 Acked 6735 Write 3 Done Done New false
16435 5737 Acked 6736 Write 10 Done Done New false
16436 5738 Acked 6737 Write 7 Done Done New false
16437 5739 Acked 6738 Read 7 Done Done New false
16438 5740 Acked 6739 Flush 0 Done Done New false
16439 5741 Acked 6740 Read 1 Done Done New false
16440 5742 Acked 6741 Read 5 Done Done New false
16441 5743 Acked 6742 Flush 0 Done Done New false
16442 5744 Acked 6743 Write 1 Done Done New false
16443 5745 Acked 6744 Flush 0 Done Done New false
16444 STATES DS:0 DS:1 DS:2 TOTAL
16445 New 0 0 303 303
16446 Sent 0 0 0 0
16447 Done 303 303 0 606
16448 Skipped 0 0 0 0
16449 Error 0 0 0 0
16450 Last Flush: 6744 6744 6441
16451 Downstairs last five completed: 6441 6440 6439 6438 6437 6436
16452 Upstairs last five completed: 5745 5744 5743 5742 5741 5740
16453 CLIENT: Up:0 ds:303 act:2
164542023-10-13T04:45:18.811ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 2
164552023-10-13T04:45:18.811ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
164562023-10-13T04:45:18.811ZINFOcrucible: [2] client re-new 303 jobs since flush 6441 = downstairs
164572023-10-13T04:45:18.812ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
164582023-10-13T04:45:18.813ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
164592023-10-13T04:45:18.813ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
164602023-10-13T04:45:18.813ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
164612023-10-13T04:45:18.813ZINFOcrucible: [2] send last flush ID to this DS: 6441
164622023-10-13T04:45:18.813ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
164632023-10-13T04:45:18.813ZINFOcrucible: [2] Replied this last flush ID: 6441
164642023-10-13T04:45:18.813ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Offline ds_transition to Replay
164652023-10-13T04:45:18.813ZINFOcrucible: [2] Transition from Offline to Replay
164662023-10-13T04:45:18.813ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
164672023-10-13T04:45:18.813ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Replay ds_transition to Active
164682023-10-13T04:45:18.813ZINFOcrucible: [2] Transition from Replay to Active
164692023-10-13T04:45:18.813ZINFOcrucible: [2] Starts cmd_loop
164702023-10-13T04:45:18.813ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
164712023-10-13T04:45:18.813ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
164722023-10-13T04:45:18.813ZINFOcrucible: Set check for repair
164732023-10-13T04:45:19.814ZINFOcrucible: Checking if live repair is needed
164742023-10-13T04:45:19.814ZINFOcrucible: No Live Repair required at this time
164752023-10-13T04:45:19.814ZWARNcrucible: [2] flow control end
16476 ----------------------------------------------------------------
16477 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
16478 Downstairs last five completed: 6744 6743 6742 6741 6740 6739
16479 Upstairs last five completed: 5745 5744 5743 5742 5741 5740
16480 CLIENT: Up:0 ds:0 act:3
16481 Replay: All jobs finished, all DS active.
16482 Main task has work to do, go find it
16483 got dsc Stop(2)
16484 stop 2
16485 Oct 13 04:45:22.727 INFO request completed, latency_us: 522, response_code: 204, [8830] Got stop action so:false kr:false
16486 uri: /stop/cid/2, method: POST, req_id: eb8a4fb0-c81d-4f78-aa7a-3f52a3c84e6b, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
16487 Oct 13 04:45:22.728 INFO request completed, latency_us: 175, response_code: 200, uri: /state/cid/2, method: GET, req_id: a5792f2f-01d4-4131-ae35-4401def6d3ca, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
164882023-10-13T04:45:22.729ZWARNcrucible: [2] downstairs disconnected
164892023-10-13T04:45:22.729ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
164902023-10-13T04:45:22.729ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8830 closed looper = 2
164912023-10-13T04:45:22.729ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
164922023-10-13T04:45:22.729ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
164932023-10-13T04:45:22.729ZWARNcrucible: [2] pm_task rx.recv() is None
164942023-10-13T04:45:22.729ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
16495 [8830] Exited with: ExitStatus(unix_wait_status(9))
16496 [8830][2] reports Exit
16497 Oct 13 04:45:26.731 INFO request completed, latency_us: 289, response_code: 200, uri: /state/cid/2, method: GET, req_id: e9b0e48d-13bd-4fe9-8321-7ba9733a703d, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
16498 001/300 Read block 29 len 32768
16499 002/300 Write block 1394 len 28672 data: 4 4 4 5 5 6 5
16500 003/300 Read block 1433 len 32768
16501 004/300 Read block 2823 len 36864
16502 005/300 Read block 4908 len 12288
16503 006/300 Flush
16504 007/300 Write block 1762 len 8192 data: 2 2
16505 008/300 Write block 3669 len 40960 data: 4 4 4 4 5 4 4 4 3 3
16506 009/300 Read block 834 len 24576
16507 010/300 Write block 2274 len 8192 data: 4 5
16508 011/300 Write block 2062 len 32768 data: 4 4 4 3 2 3 3 3
16509 012/300 Write block 2992 len 4096 data: 5
16510 013/300 Flush
16511 014/300 Read block 107 len 4096
16512 015/300 Read block 3698 len 24576
16513 016/300 Write block 3161 len 12288 data: 5 5 7
16514 017/300 Read block 2782 len 12288
16515 018/300 Read block 1368 len 36864
16516 019/300 Write block 3549 len 20480 data: 4 3 3 6 8
16517 020/300 Write block 2065 len 8192 data: 4 3
16518 021/300 Read block 1091 len 16384
16519 022/300 Write block 2827 len 4096 data: 5
16520 023/300 Flush
16521 024/300 Write block 1661 len 16384 data: 6 4 5 5
16522 025/300 Write block 4894 len 24576 data: 4 4 3 2 2 2
16523 026/300 Read block 4306 len 4096
16524 027/300 Read block 983 len 32768
16525 028/300 Read block 3665 len 32768
16526 029/300 Write block 2345 len 28672 data: 3 4 4 4 5 4 2
16527 030/300 Read block 812 len 36864
16528 031/300 Read block 4511 len 32768
16529 032/300 Write block 2761 len 12288 data: 3 4 5
16530 033/300 Write block 2622 len 16384 data: 5 5 5 4
16531 034/300 Read block 670 len 32768
16532 035/300 Write block 1277 len 20480 data: 4 4 3 3 3
16533 036/300 Write block 217 len 4096 data: 10
16534 037/300 Read block 3921 len 24576
16535 038/300 Read block 1470 len 4096
16536 039/300 Read block 1095 len 16384
16537 040/300 Read block 4531 len 20480
16538 041/300 Read block 3214 len 36864
16539 042/300 Read block 3772 len 20480
16540 043/300 Write block 3324 len 36864 data: 8 8 7 5 5 5 5 5 5
16541 044/300 Read block 137 len 8192
16542 045/300 Read block 1880 len 28672
16543 046/300 Write block 2031 len 12288 data: 5 5 5
16544 047/300 Write block 3925 len 36864 data: 4 4 4 4 5 4 4 4 4
16545 048/300 Flush
16546 049/300 Read block 483 len 8192
16547 050/300 Read block 4812 len 8192
16548 051/300 Read block 2205 len 16384
16549 052/300 Write block 1722 len 32768 data: 5 4 4 3 3 3 3 3
16550 053/300 Flush
16551 054/300 Write block 1692 len 36864 data: 5 5 6 6 5 6 6 7 7
16552 055/300 Read block 4184 len 16384
16553 056/300 Write block 2862 len 24576 data: 4 4 5 4 4 4
16554 057/300 Read block 2226 len 32768
16555 058/300 Read block 3728 len 32768
16556 059/300 Write block 128 len 32768 data: 3 3 3 3 3 3 3 3
16557 060/300 Write block 535 len 32768 data: 6 4 5 5 5 5 4 4
16558 061/300 Write block 2596 len 4096 data: 5
16559 062/300 Write block 4729 len 24576 data: 5 4 2 3 4 4
16560 063/300 Write block 3277 len 12288 data: 2 2 2
16561 064/300 Write block 3936 len 8192 data: 3 4
16562 065/300 Read block 272 len 20480
16563 066/300 Write block 3555 len 28672 data: 6 6 4 7 8 8 9
16564 067/300 Read block 3106 len 20480
16565 068/300 Flush
16566 069/300 Read block 727 len 20480
16567 070/300 Write block 1584 len 36864 data: 6 6 7 8 5 6 5 4 3
16568 071/300 Read block 2519 len 12288
16569 072/300 Write block 4441 len 8192 data: 7 8
16570 073/300 Write block 4942 len 4096 data: 6
16571 074/300 Flush
16572 075/300 Read block 4040 len 40960
16573 076/300 Write block 1702 len 4096 data: 3
16574 077/300 Write block 2824 len 24576 data: 3 4 5 6 6 5
16575 078/300 Write block 2246 len 20480 data: 5 5 5 4 4
16576 079/300 Write block 4280 len 40960 data: 5 5 5 5 4 4 5 7 6 6
16577 080/300 Flush
16578 081/300 Read block 2721 len 20480
16579 082/300 Write block 3779 len 24576 data: 5 6 6 6 6 5
16580 083/300 Write block 4377 len 4096 data: 3
16581 084/300 Write block 2387 len 40960 data: 4 5 3 3 3 3 3 3 5 5
16582 085/300 Write block 4766 len 28672 data: 4 4 5 5 3 2 2
16583 086/300 Write block 4301 len 32768 data: 6 5 4 3 3 3 4 4
16584 087/300 Write block 3936 len 40960 data: 4 5 4 5 5 5 7 7 6 4
16585 088/300 Write block 200 len 28672 data: 4 4 4 5 5 5 6
16586 089/300 Read block 4764 len 36864
16587 090/300 Read block 3855 len 20480
16588 091/300 Write block 3447 len 32768 data: 3 4 4 4 4 4 5 6
16589 092/300 Write block 1520 len 36864 data: 3 3 3 3 3 3 2 2 3
16590 093/300 Write block 972 len 8192 data: 6 6
16591 094/300 Write block 2112 len 36864 data: 4 5 5 6 6 5 4 5 4
16592 095/300 Read block 195 len 8192
16593 096/300 Flush
16594 097/300 Read block 1809 len 16384
16595 098/300 Read block 1151 len 28672
16596 099/300 Flush
16597 100/300 Flush
16598 101/300 Write block 2437 len 4096 data: 2
16599 102/300 Write block 1442 len 20480 data: 4 4 4 6 5
16600 103/300 Flush
16601 104/300 Read block 1317 len 32768
16602 105/300 Read block 3730 len 4096
16603 106/300 Flush
16604 107/300 Flush
16605 108/300 Flush
16606 109/300 Read block 3056 len 28672
16607 110/300 Read block 4815 len 16384
16608 111/300 Read block 4984 len 24576
16609 112/300 Write block 2794 len 36864 data: 8 9 9 9 7 6 5 5 5
16610 113/300 Write block 4289 len 8192 data: 7 6
16611 114/300 Read block 414 len 20480
16612 115/300 Flush
16613 116/300 Read block 2593 len 20480
16614 117/300 Write block 1310 len 28672 data: 3 3 3 4 3 4 4
16615 118/300 Write block 3388 len 4096 data: 4
16616 119/300 Write block 4916 len 28672 data: 6 5 6 6 6 5 4
16617 120/300 Read block 530 len 4096
16618 121/300 Read block 3886 len 16384
16619 122/300 Read block 4816 len 36864
16620 123/300 Write block 661 len 8192 data: 5 6
16621 124/300 Read block 656 len 8192
16622 125/300 Write block 4992 len 24576 data: 4 3 3 3 3 3
16623 126/300 Read block 722 len 40960
16624 127/300 Read block 1851 len 40960
16625 128/300 Write block 3401 len 16384 data: 2 2 2 3
16626 129/300 Write block 3805 len 40960 data: 7 6 6 5 5 3 3 3 3 3
16627 130/300 Read block 1682 len 24576
16628 131/300 Read block 4553 len 32768
16629 132/300 Read block 1497 len 4096
16630 133/300 Read block 1427 len 20480
16631 134/300 Read block 1348 len 4096
16632 135/300 Write block 3081 len 12288 data: 6 5 3
16633 136/300 Write block 4421 len 28672 data: 3 5 5 7 7 7 6
16634 137/300 Read block 669 len 24576
16635 138/300 Read block 4139 len 24576
16636 139/300 Write block 3930 len 12288 data: 5 5 5
16637 140/300 Write block 1102 len 12288 data: 8 5 5
16638 141/300 Flush
16639 142/300 Flush
16640 143/300 Read block 3973 len 40960
16641 144/300 Read block 2791 len 4096
16642 145/300 Flush
16643 146/300 Read block 1490 len 36864
16644 147/300 Write block 4038 len 32768 data: 2 3 3 3 3 3 2 2
16645 148/300 Write block 2129 len 4096 data: 6
16646 149/300 Flush
16647 150/300 Flush
16648 151/300 Read block 903 len 16384
16649 152/300 Read block 999 len 36864
16650 153/300 Write block 1536 len 28672 data: 5 6 6 5 4 4 4
16651 154/300 Read block 238 len 24576
16652 155/300 Write block 477 len 20480 data: 5 5 4 3 2
16653 156/300 Flush
16654 157/300 Write block 1466 len 8192 data: 5 5
16655 158/300 Write block 3384 len 32768 data: 7 5 6 4 5 4 4 6
16656 159/300 Write block 3296 len 32768 data: 3 3 4 3 3 3 3 3
16657 160/300 Read block 4146 len 4096
16658 161/300 Read block 4595 len 4096
16659 162/300 Read block 616 len 12288
16660 163/300 Read block 2220 len 28672
16661 164/300 Write block 3762 len 16384 data: 5 5 4 3
16662 165/300 Read block 139 len 4096
16663 166/300 Write block 4264 len 20480 data: 4 4 4 3 3
16664 167/300 Flush
16665 168/300 Read block 3783 len 36864
16666 169/300 Read block 194 len 36864
16667 170/300 Write block 2752 len 28672 data: 6 7 5 3 3 4 4
16668 171/300 Write block 4546 len 28672 data: 4 3 3 3 4 5 4
16669 172/300 Write block 3143 len 4096 data: 4
16670 173/300 Read block 3886 len 28672
16671 174/300 Read block 652 len 20480
16672 175/300 Read block 4406 len 4096
16673 176/300 Write block 842 len 16384 data: 6 7 6 6
16674 177/300 Write block 817 len 24576 data: 3 3 3 3 3 3
16675 178/300 Write block 1118 len 32768 data: 3 3 4 4 4 3 3 4
16676 179/300 Write block 3980 len 32768 data: 5 5 5 5 4 3 3 3
16677 180/300 Write block 660 len 16384 data: 4 6 7 5
16678 181/300 Read block 847 len 36864
16679 182/300 Read block 779 len 4096
16680 183/300 Read block 8 len 32768
16681 184/300 Write block 3392 len 4096 data: 6
16682 185/300 Read block 613 len 16384
16683 186/300 Write block 432 len 28672 data: 3 5 5 5 6 6 4
16684 187/300 Read block 4243 len 32768
16685 188/300 Read block 1410 len 8192
16686 189/300 Write block 832 len 40960 data: 4 3 3 5 6 5 4 4 4 5
16687 190/300 Write block 1049 len 28672 data: 4 3 3 2 2 4 4
16688 191/300 Read block 1295 len 40960
16689 192/300 Write block 18 len 36864 data: 5 5 6 6 4 4 4 5 5
16690 193/300 Write block 3812 len 32768 data: 4 4 4 5 5 4 4 4
16691 194/300 Read block 2760 len 20480
16692 195/300 Write block 3277 len 20480 data: 3 3 3 2 2
16693 196/300 Read block 1623 len 4096
16694 197/300 Write block 2859 len 40960 data: 3 3 2 5 5 6 5 5 5 4
16695 198/300 Write block 2120 len 28672 data: 5 3 4 4 5 6 6
16696 199/300 Read block 3465 len 4096
16697 200/300 Read block 4978 len 12288
16698 201/300 Flush
16699 202/300 Write block 1496 len 8192 data: 6 7
16700 203/300 Write block 4300 len 12288 data: 7 7 6
16701 204/300 Write block 1417 len 12288 data: 4 4 4
16702 205/300 Write block 3049 len 28672 data: 4 4 4 4 4 3 4
16703 206/300 Read block 2761 len 4096
16704 207/300 Read block 3211 len 20480
16705 208/300 Flush
16706 209/300 Write block 2729 len 16384 data: 3 3 3 4
16707 210/300 Read block 4837 len 24576
16708 211/300 Read block 4232 len 12288
16709 212/300 Write block 809 len 36864 data: 2 2 2 3 3 3 3 3 4
16710 213/300 Write block 3735 len 36864 data: 3 3 3 4 5 4 4 5 5
16711 214/300 Flush
16712 215/300 Read block 2341 len 32768
16713 216/300 Read block 1148 len 16384
16714 217/300 Read block 1206 len 8192
16715 218/300 Write block 4371 len 20480 data: 6 6 5 4 3
16716 219/300 Read block 178 len 36864
16717 220/300 Read block 3916 len 36864
16718 221/300 Read block 4617 len 20480
16719 222/300 Read block 621 len 24576
16720 223/300 Write block 2240 len 12288 data: 5 6 6
16721 224/300 Read block 4709 len 12288
16722 225/300 Read block 3109 len 24576
16723 226/300 Write block 2270 len 40960 data: 3 3 4 4 5 6 5 4 5 5
16724 227/300 Write block 3989 len 16384 data: 3 2 2 3
16725 228/300 Write block 3127 len 36864 data: 4 3 3 3 3 3 3 3 3
16726 229/300 Write block 4712 len 36864 data: 5 6 5 4 3 3 4 3 3
16727 230/300 Read block 792 len 40960
16728 231/300 Write block 1500 len 28672 data: 4 3 3 3 3 4 6
16729 232/300 Read block 1336 len 16384
16730 233/300 Write block 3411 len 20480 data: 5 5 5 5 3
16731 234/300 Write block 2555 len 12288 data: 5 6 6
16732 235/300 Read block 3272 len 28672
16733 236/300 Flush
16734 237/300 Read block 2514 len 28672
16735 238/300 Write block 3672 len 8192 data: 5 6
16736 239/300 Read block 3224 len 32768
16737 240/300 Write block 3690 len 36864 data: 4 3 3 4 4 4 4 5 7
16738 241/300 Write block 4381 len 24576 data: 2 2 2 3 3 5
16739 242/300 Read block 561 len 16384
16740 243/300 Read block 492 len 24576
16741 244/300 Write block 3888 len 20480 data: 4 4 4 4 5
16742 245/300 Write block 2723 len 40960 data: 5 4 2 2 2 3 4 4 4 5
16743 246/300 Flush
16744 247/300 Read block 689 len 16384
16745 248/300 Write block 4155 len 24576 data: 6 5 5 5 5 6
16746 249/300 Read block 1781 len 16384
16747 250/300 Write block 924 len 28672 data: 4 4 4 5 3 4 4
16748 251/300 Read block 3707 len 4096
16749 252/300 Read block 105 len 32768
16750 253/300 Read block 2745 len 36864
16751 254/300 Read block 993 len 4096
16752 255/300 Read block 600 len 4096
16753 256/300 Write block 549 len 40960 data: 3 3 3 4 3 3 3 4 5 5
16754 257/300 Read block 2561 len 40960
16755 258/300 Read block 375 len 36864
16756 259/300 Write block 148 len 20480 data: 6 6 6 7 7
16757 260/300 Flush
16758 261/300 Write block 1506 len 24576 data: 7 7 6 6 6 6
16759 262/300 Flush
16760 263/300 Write block 3817 len 36864 data: 5 5 5 4 4 2 3 3 5
16761 264/300 Read block 592 len 16384
16762 265/300 Read block 4610 len 36864
16763 266/300 Write block 1708 len 40960 data: 5 6 6 6 6 7 6 6 7 4
16764 267/300 Read block 598 len 36864
16765 268/300 Flush
16766 269/300 Write block 2551 len 16384 data: 3 3 5 4
16767 270/300 Write block 4690 len 20480 data: 3 3 3 3 4
16768 271/300 Read block 335 len 36864
16769 272/300 Write block 1664 len 4096 data: 6
16770 273/300 Read block 3460 len 24576
16771 274/300 Read block 2820 len 32768
16772 275/300 Write block 4391 len 24576 data: 5 5 5 3 4 4
16773 276/300 Flush
16774 277/300 Write block 4617 len 20480 data: 4 3 3 3 5
16775 278/300 Write block 2095 len 16384 data: 5 6 8 8
16776 279/300 Read block 1172 len 40960
16777 280/300 Read block 1877 len 32768
16778 281/300 Read block 2420 len 40960
16779 282/300 Write block 2654 len 24576 data: 5 5 6 7 4 4
16780 283/300 Write block 4092 len 28672 data: 3 2 4 4 5 5 5
16781 284/300 Read block 2671 len 4096
16782 285/300 Read block 212 len 12288
16783 286/300 Write block 3364 len 20480 data: 6 5 5 4 4
16784 287/300 Read block 4292 len 24576
16785 288/300 Read block 4985 len 12288
16786 289/300 Read block 3117 len 28672
16787 290/300 Write block 933 len 28672 data: 2 3 3 2 2 2 2
16788 291/300 Read block 1212 len 32768
16789 292/300 Write block 2216 len 36864 data: 4 5 5 6 6 7 7 6 7
16790 293/300 Read block 4010 len 40960
16791 294/300 Write block 2930 len 20480 data: 5 4 3 2 2
16792 295/300 Read block 4388 len 8192
16793 296/300 Read block 531 len 24576
16794 297/300 Read block 4142 len 16384
16795 298/300 Read block 4719 len 16384
16796 299/300 Read block 2144 len 4096
16797 300/300 Flush
16798 Main task has work to do, go find it
16799 got dsc Start(2)
16800 start 2
16801 Oct 13 04:45:27.614 INFO request completed, latency_us: 284, response_code: 204, uri: /start/cid/2, method: POST, req_id: 2a211c0c-db6c-4b29-87cc-5bb331ef4424, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
16802 Replay: started 2, returned:Ok(())
16803 ----------------------------------------------------------------
16804 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
16805 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
16806 5746 Acked 6745 Read 8 Done Done New false
16807 5747 Acked 6746 Flush 0 Done Done New false
16808 5748 Acked 6747 Write 7 Done Done New false
16809 5749 Acked 6748 Read 8 Done Done New false
16810 5750 Acked 6749 Read 9 Done Done New false
16811 5751 Acked 6750 Read 3 Done Done New false
16812 5752 Acked 6751 Flush 0 Done Done New false
16813 5753 Acked 6752 Write 2 Done Done New false
16814 5754 Acked 6753 Write 10 Done Done New false
16815 5755 Acked 6754 Read 6 Done Done New false
16816 5756 Acked 6755 Write 2 Done Done New false
16817 5757 Acked 6756 Write 8 Done Done New false
16818 5758 Acked 6757 Write 1 Done Done New false
16819 5759 Acked 6758 Flush 0 Done Done New false
16820 5760 Acked 6759 Read 1 Done Done New false
16821 5761 Acked 6760 Read 6 Done Done New false
16822 5762 Acked 6761 Write 3 Done Done New false
16823 5763 Acked 6762 Read 3 Done Done New false
16824 5764 Acked 6763 Read 9 Done Done New false
16825 5765 Acked 6764 Write 5 Done Done New false
16826 5766 Acked 6765 Write 2 Done Done New false
16827 5767 Acked 6766 Read 4 Done Done New false
16828 5768 Acked 6767 Write 1 Done Done New false
16829 5769 Acked 6768 Flush 0 Done Done New false
16830 5770 Acked 6769 Write 4 Done Done New false
16831 5771 Acked 6770 Write 6 Done Done New false
16832 5772 Acked 6771 Read 1 Done Done New false
16833 5773 Acked 6772 Read 8 Done Done New false
16834 5774 Acked 6773 Read 8 Done Done New false
16835 5775 Acked 6774 Write 7 Done Done New false
16836 5776 Acked 6775 Read 9 Done Done New false
16837 5777 Acked 6776 Read 8 Done Done New false
16838 5778 Acked 6777 Write 3 Done Done New false
16839 5779 Acked 6778 Write 4 Done Done New false
16840 5780 Acked 6779 Read 8 Done Done New false
16841 5781 Acked 6780 Write 5 Done Done New false
16842 5782 Acked 6781 Write 1 Done Done New false
16843 5783 Acked 6782 Read 6 Done Done New false
16844 5784 Acked 6783 Read 1 Done Done New false
16845 5785 Acked 6784 Read 4 Done Done New false
16846 5786 Acked 6785 Read 5 Done Done New false
16847 5787 Acked 6786 Read 9 Done Done New false
16848 5788 Acked 6787 Read 5 Done Done New false
16849 5789 Acked 6788 Write 9 Done Done New false
16850 5790 Acked 6789 Read 2 Done Done New false
16851 5791 Acked 6790 Read 7 Done Done New false
16852 5792 Acked 6791 Write 3 Done Done New false
16853 5793 Acked 6792 Write 9 Done Done New false
16854 5794 Acked 6793 Flush 0 Done Done New false
16855 5795 Acked 6794 Read 2 Done Done New false
16856 5796 Acked 6795 Read 2 Done Done New false
16857 5797 Acked 6796 Read 4 Done Done New false
16858 5798 Acked 6797 Write 8 Done Done New false
16859 5799 Acked 6798 Flush 0 Done Done New false
16860 5800 Acked 6799 Write 9 Done Done New false
16861 5801 Acked 6800 Read 4 Done Done New false
16862 5802 Acked 6801 Write 6 Done Done New false
16863 5803 Acked 6802 Read 8 Done Done New false
16864 5804 Acked 6803 Read 8 Done Done New false
16865 5805 Acked 6804 Write 8 Done Done New false
16866 5806 Acked 6805 Write 8 Done Done New false
16867 5807 Acked 6806 Write 1 Done Done New false
16868 5808 Acked 6807 Write 6 Done Done New false
16869 5809 Acked 6808 Write 3 Done Done New false
16870 5810 Acked 6809 Write 2 Done Done New false
16871 5811 Acked 6810 Read 5 Done Done New false
16872 5812 Acked 6811 Write 7 Done Done New false
16873 5813 Acked 6812 Read 5 Done Done New false
16874 5814 Acked 6813 Flush 0 Done Done New false
16875 5815 Acked 6814 Read 5 Done Done New false
16876 5816 Acked 6815 Write 9 Done Done New false
16877 5817 Acked 6816 Read 3 Done Done New false
16878 5818 Acked 6817 Write 2 Done Done New false
16879 5819 Acked 6818 Write 1 Done Done New false
16880 5820 Acked 6819 Flush 0 Done Done New false
16881 5821 Acked 6820 Read 10 Done Done New false
16882 5822 Acked 6821 Write 1 Done Done New false
16883 5823 Acked 6822 Write 6 Done Done New false
16884 5824 Acked 6823 Write 5 Done Done New false
16885 5825 Acked 6824 Write 10 Done Done New false
16886 5826 Acked 6825 Flush 0 Done Done New false
16887 5827 Acked 6826 Read 5 Done Done New false
16888 5828 Acked 6827 Write 6 Done Done New false
16889 5829 Acked 6828 Write 1 Done Done New false
16890 5830 Acked 6829 Write 10 Done Done New false
16891 5831 Acked 6830 Write 7 Done Done New false
16892 5832 Acked 6831 Write 8 Done Done New false
16893 5833 Acked 6832 Write 10 Done Done New false
16894 5834 Acked 6833 Write 7 Done Done New false
16895 5835 Acked 6834 Read 9 Done Done New false
16896 5836 Acked 6835 Read 5 Done Done New false
16897 5837 Acked 6836 Write 8 Done Done New false
16898 5838 Acked 6837 Write 9 Done Done New false
16899 5839 Acked 6838 Write 2 Done Done New false
16900 5840 Acked 6839 Write 9 Done Done New false
16901 5841 Acked 6840 Read 2 Done Done New false
16902 5842 Acked 6841 Flush 0 Done Done New false
16903 5843 Acked 6842 Read 4 Done Done New false
16904 5844 Acked 6843 Read 7 Done Done New false
16905 5845 Acked 6844 Flush 0 Done Done New false
16906 5846 Acked 6845 Flush 0 Done Done New false
16907 5847 Acked 6846 Write 1 Done Done New false
16908 5848 Acked 6847 Write 5 Done Done New false
16909 5849 Acked 6848 Flush 0 Done Done New false
16910 5850 Acked 6849 Read 8 Done Done New false
16911 5851 Acked 6850 Read 1 Done Done New false
16912 5852 Acked 6851 Flush 0 Done Done New false
16913 5853 Acked 6852 Flush 0 Done Done New false
16914 5854 Acked 6853 Flush 0 Done Done New false
16915 5855 Acked 6854 Read 7 Done Done New false
16916 5856 Acked 6855 Read 4 Done Done New false
16917 5857 Acked 6856 Read 6 Done Done New false
16918 5858 Acked 6857 Write 9 Done Done New false
16919 5859 Acked 6858 Write 2 Done Done New false
16920 5860 Acked 6859 Read 5 Done Done New false
16921 5861 Acked 6860 Flush 0 Done Done New false
16922 5862 Acked 6861 Read 5 Done Done New false
16923 5863 Acked 6862 Write 7 Done Done New false
16924 5864 Acked 6863 Write 1 Done Done New false
16925 5865 Acked 6864 Write 7 Done Done New false
16926 5866 Acked 6865 Read 1 Done Done New false
16927 5867 Acked 6866 Read 4 Done Done New false
16928 5868 Acked 6867 Read 9 Done Done New false
16929 5869 Acked 6868 Write 2 Done Done New false
16930 5870 Acked 6869 Read 2 Done Done New false
16931 5871 Acked 6870 Write 6 Done Done New false
16932 5872 Acked 6871 Read 10 Done Done New false
16933 5873 Acked 6872 Read 10 Done Done New false
16934 5874 Acked 6873 Write 4 Done Done New false
16935 5875 Acked 6874 Write 10 Done Done New false
16936 5876 Acked 6875 Read 6 Done Done New false
16937 5877 Acked 6876 Read 8 Done Done New false
16938 5878 Acked 6877 Read 1 Done Done New false
16939 5879 Acked 6878 Read 5 Done Done New false
16940 5880 Acked 6879 Read 1 Done Done New false
16941 5881 Acked 6880 Write 3 Done Done New false
16942 5882 Acked 6881 Write 7 Done Done New false
16943 5883 Acked 6882 Read 6 Done Done New false
16944 5884 Acked 6883 Read 6 Done Done New false
16945 5885 Acked 6884 Write 3 Done Done New false
16946 5886 Acked 6885 Write 3 Done Done New false
16947 5887 Acked 6886 Flush 0 Done Done New false
16948 5888 Acked 6887 Flush 0 Done Done New false
16949 5889 Acked 6888 Read 10 Done Done New false
16950 5890 Acked 6889 Read 1 Done Done New false
16951 5891 Acked 6890 Flush 0 Done Done New false
16952 5892 Acked 6891 Read 9 Done Done New false
16953 5893 Acked 6892 Write 8 Done Done New false
16954 5894 Acked 6893 Write 1 Done Done New false
16955 5895 Acked 6894 Flush 0 Done Done New false
16956 5896 Acked 6895 Flush 0 Done Done New false
16957 5897 Acked 6896 Read 4 Done Done New false
16958 5898 Acked 6897 Read 9 Done Done New false
16959 5899 Acked 6898 Write 7 Done Done New false
16960 5900 Acked 6899 Read 6 Done Done New false
16961 5901 Acked 6900 Write 5 Done Done New false
16962 5902 Acked 6901 Flush 0 Done Done New false
16963 5903 Acked 6902 Write 2 Done Done New false
16964 5904 Acked 6903 Write 8 Done Done New false
16965 5905 Acked 6904 Write 8 Done Done New false
16966 5906 Acked 6905 Read 1 Done Done New false
16967 5907 Acked 6906 Read 1 Done Done New false
16968 5908 Acked 6907 Read 3 Done Done New false
16969 5909 Acked 6908 Read 7 Done Done New false
16970 5910 Acked 6909 Write 4 Done Done New false
16971 5911 Acked 6910 Read 1 Done Done New false
16972 5912 Acked 6911 Write 5 Done Done New false
16973 5913 Acked 6912 Flush 0 Done Done New false
16974 5914 Acked 6913 Read 9 Done Done New false
16975 5915 Acked 6914 Read 9 Done Done New false
16976 5916 Acked 6915 Write 7 Done Done New false
16977 5917 Acked 6916 Write 7 Done Done New false
16978 5918 Acked 6917 Write 1 Done Done New false
16979 5919 Acked 6918 Read 7 Done Done New false
16980 5920 Acked 6919 Read 5 Done Done New false
16981 5921 Acked 6920 Read 1 Done Done New false
16982 5922 Acked 6921 Write 4 Done Done New false
16983 5923 Acked 6922 Write 6 Done Done New false
16984 5924 Acked 6923 Write 8 Done Done New false
16985 5925 Acked 6924 Write 8 Done Done New false
16986 5926 Acked 6925 Write 4 Done Done New false
16987 5927 Acked 6926 Read 9 Done Done New false
16988 5928 Acked 6927 Flush 0 Done Done New false
16989 5929 Acked 6928 Read 1 Done Done New false
16990 5930 Acked 6929 Read 8 Done Done New false
16991 5931 Acked 6930 Write 1 Done Done New false
16992 5932 Acked 6931 Read 4 Done Done New false
16993 5933 Acked 6932 Write 7 Done Done New false
16994 5934 Acked 6933 Read 8 Done Done New false
16995 5935 Acked 6934 Read 2 Done Done New false
16996 5936 Acked 6935 Write 10 Done Done New false
16997 5937 Acked 6936 Write 7 Done Done New false
16998 5938 Acked 6937 Read 10 Done Done New false
16999 5939 Acked 6938 Write 9 Done Done New false
17000 5940 Acked 6939 Write 8 Done Done New false
17001 5941 Acked 6940 Read 5 Done Done New false
17002 5942 Acked 6941 Write 5 Done Done New false
17003 5943 Acked 6942 Read 1 Done Done New false
17004 5944 Acked 6943 Write 10 Done Done New false
17005 5945 Acked 6944 Write 7 Done Done New false
17006 5946 Acked 6945 Read 1 Done Done New false
17007 5947 Acked 6946 Read 3 Done Done New false
17008 5948 Acked 6947 Flush 0 Done Done New false
17009 5949 Acked 6948 Write 2 Done Done New false
17010 5950 Acked 6949 Write 3 Done Done New false
17011 5951 Acked 6950 Write 3 Done Done New false
17012 5952 Acked 6951 Write 7 Done Done New false
17013 5953 Acked 6952 Read 1 Done Done New false
17014 5954 Acked 6953 Read 5 Done Done New false
17015 5955 Acked 6954 Flush 0 Done Done New false
17016 5956 Acked 6955 Write 4 Done Done New false
17017 5957 Acked 6956 Read 6 Done Done New false
17018 5958 Acked 6957 Read 3 Done Done New false
17019 5959 Acked 6958 Write 9 Done Done New false
17020 5960 Acked 6959 Write 9 Done Done New false
17021 5961 Acked 6960 Flush 0 Done Done New false
17022 5962 Acked 6961 Read 8 Done Done New false
17023 5963 Acked 6962 Read 4 Done Done New false
17024 5964 Acked 6963 Read 2 Done Done New false
17025 5965 Acked 6964 Write 5 Done Done New false
17026 5966 Acked 6965 Read 9 Done Done New false
17027 5967 Acked 6966 Read 9 Done Done New false
17028 5968 Acked 6967 Read 5 Done Done New false
17029 5969 Acked 6968 Read 6 Done Done New false
17030 5970 Acked 6969 Write 3 Done Done New false
17031 5971 Acked 6970 Read 3 Done Done New false
17032 5972 Acked 6971 Read 6 Done Done New false
17033 5973 Acked 6972 Write 10 Done Done New false
17034 5974 Acked 6973 Write 4 Done Done New false
17035 5975 Acked 6974 Write 9 Done Done New false
17036 5976 Acked 6975 Write 9 Done Done New false
17037 5977 Acked 6976 Read 10 Done Done New false
17038 5978 Acked 6977 Write 7 Done Done New false
17039 5979 Acked 6978 Read 4 Done Done New false
17040 5980 Acked 6979 Write 5 Done Done New false
17041 5981 Acked 6980 Write 3 Done Done New false
17042 5982 Acked 6981 Read 7 Done Done New false
17043 5983 Acked 6982 Flush 0 Done Done New false
17044 5984 Acked 6983 Read 7 Done Done New false
17045 5985 Acked 6984 Write 2 Done Done New false
17046 5986 Acked 6985 Read 8 Done Done New false
17047 5987 Acked 6986 Write 9 Done Done New false
17048 5988 Acked 6987 Write 6 Done Done New false
17049 5989 Acked 6988 Read 4 Done Done New false
17050 5990 Acked 6989 Read 6 Done Done New false
17051 5991 Acked 6990 Write 5 Done Done New false
17052 5992 Acked 6991 Write 10 Done Done New false
17053 5993 Acked 6992 Flush 0 Done Done New false
17054 5994 Acked 6993 Read 4 Done Done New false
17055 5995 Acked 6994 Write 6 Done Done New false
17056 5996 Acked 6995 Read 4 Done Done New false
17057 5997 Acked 6996 Write 7 Done Done New false
17058 5998 Acked 6997 Read 1 Done Done New false
17059 5999 Acked 6998 Read 8 Done Done New false
17060 6000 Acked 6999 Read 9 Done Done New false
17061 6001 Acked 7000 Read 1 Done Done New false
17062 6002 Acked 7001 Read 1 Done Done New false
17063 6003 Acked 7002 Write 10 Done Done New false
17064 6004 Acked 7003 Read 10 Done Done New false
17065 6005 Acked 7004 Read 9 Done Done New false
17066 6006 Acked 7005 Write 5 Done Done New false
17067 6007 Acked 7006 Flush 0 Done Done New false
17068 6008 Acked 7007 Write 6 Done Done New false
17069 6009 Acked 7008 Flush 0 Done Done New false
17070 6010 Acked 7009 Write 9 Done Done New false
17071 6011 Acked 7010 Read 4 Done Done New false
17072 6012 Acked 7011 Read 9 Done Done New false
17073 6013 Acked 7012 Write 10 Done Done New false
17074 6014 Acked 7013 Read 9 Done Done New false
17075 6015 Acked 7014 Flush 0 Done Done New false
17076 6016 Acked 7015 Write 4 Done Done New false
17077 6017 Acked 7016 Write 5 Done Done New false
17078 6018 Acked 7017 Read 9 Done Done New false
17079 6019 Acked 7018 Write 1 Done Done New false
17080 6020 Acked 7019 Read 6 Done Done New false
17081 6021 Acked 7020 Read 8 Done Done New false
17082 6022 Acked 7021 Write 6 Done Done New false
17083 6023 Acked 7022 Flush 0 Done Done New false
17084 6024 Acked 7023 Write 5 Done Done New false
17085 6025 Acked 7024 Write 4 Done Done New false
17086 6026 Acked 7025 Read 10 Done Done New false
17087 6027 Acked 7026 Read 8 Done Done New false
17088 6028 Acked 7027 Read 10 Done Done New false
17089 6029 Acked 7028 Write 6 Done Done New false
17090 6030 Acked 7029 Write 7 Done Done New false
17091 6031 Acked 7030 Read 1 Done Done New false
17092 6032 Acked 7031 Read 3 Done Done New false
17093 6033 Acked 7032 Write 5 Done Done New false
17094 6034 Acked 7033 Read 6 Done Done New false
17095 6035 Acked 7034 Read 3 Done Done New false
17096 6036 Acked 7035 Read 7 Done Done New false
17097 6037 Acked 7036 Write 7 Done Done New false
17098 6038 Acked 7037 Read 8 Done Done New false
17099 6039 Acked 7038 Write 9 Done Done New false
17100 6040 Acked 7039 Read 10 Done Done New false
17101 6041 Acked 7040 Write 5 Done Done New false
17102 6042 Acked 7041 Read 2 Done Done New false
17103 6043 Acked 7042 Read 6 Done Done New false
17104 6044 Acked 7043 Read 4 Done Done New false
17105 6045 Acked 7044 Read 4 Done Done New false
17106 6046 Acked 7045 Read 1 Done Done New false
17107 6047 Acked 7046 Flush 0 Done Done New false
17108 STATES DS:0 DS:1 DS:2 TOTAL
17109 New 0 0 302 302
17110 Sent 0 0 0 0
17111 Done 302 302 0 604
17112 Skipped 0 0 0 0
17113 Error 0 0 0 0
17114 Last Flush: 7046 7046 6744
17115 Downstairs last five completed: 6744 6743 6742 6741 6740 6739
17116 Upstairs last five completed: 6047 6046 6045 6044 6043 6042
17117 CLIENT: Up:0 ds:302 act:2
17118 [8830] got start action
17119 [8830] I am going to restart
17120 Starting downstairs at port 8830
17121 Make output file at "/tmp/dsc/downstairs-8830.txt"
17122 [8830][2] reports Starting
17123 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1196)
17124 [8830][2] reports Running
171252023-10-13T04:45:29.737ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 2
171262023-10-13T04:45:29.737ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
171272023-10-13T04:45:29.737ZINFOcrucible: [2] client re-new 302 jobs since flush 6744 = downstairs
171282023-10-13T04:45:29.738ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
171292023-10-13T04:45:29.739ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
171302023-10-13T04:45:29.739ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
171312023-10-13T04:45:29.739ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
171322023-10-13T04:45:29.739ZINFOcrucible: [2] send last flush ID to this DS: 6744
171332023-10-13T04:45:29.739ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
171342023-10-13T04:45:29.739ZINFOcrucible: [2] Replied this last flush ID: 6744
171352023-10-13T04:45:29.739ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Offline ds_transition to Replay
171362023-10-13T04:45:29.739ZINFOcrucible: [2] Transition from Offline to Replay
171372023-10-13T04:45:29.739ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
171382023-10-13T04:45:29.739ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Replay ds_transition to Active
171392023-10-13T04:45:29.739ZINFOcrucible: [2] Transition from Replay to Active
171402023-10-13T04:45:29.739ZINFOcrucible: [2] Starts cmd_loop
171412023-10-13T04:45:29.739ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
171422023-10-13T04:45:29.739ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
171432023-10-13T04:45:29.739ZINFOcrucible: Set check for repair
171442023-10-13T04:45:30.741ZINFOcrucible: Checking if live repair is needed
171452023-10-13T04:45:30.741ZINFOcrucible: No Live Repair required at this time
171462023-10-13T04:45:30.741ZWARNcrucible: [2] flow control end
17147 ----------------------------------------------------------------
17148 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
17149 Downstairs last five completed: 7046 7045 7044 7043 7042 7041
17150 Upstairs last five completed: 6047 6046 6045 6044 6043 6042
17151 CLIENT: Up:0 ds:0 act:3
17152 Replay: All jobs finished, all DS active.
17153 Main task has work to do, go find it
17154 got dsc Stop(2)
17155 stop 2
17156 [8830] Got stop action so:false kr:false
17157 Oct 13 04:45:31.618 INFO request completed, latency_us: 459, response_code: 204, uri: /stop/cid/2, method: POST, req_id: c4a11bc2-b27b-4f04-91f1-d58389b6cbfb, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
17158 Oct 13 04:45:31.619 INFO request completed, latency_us: 189, response_code: 200, uri: /state/cid/2, method: GET, req_id: 9e087e7b-4ff2-43e4-9a7f-05dc31e348c1, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
171592023-10-13T04:45:31.620ZWARNcrucible: [2] downstairs disconnected
171602023-10-13T04:45:31.620ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
171612023-10-13T04:45:31.620ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8830 closed looper = 2
171622023-10-13T04:45:31.620ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
171632023-10-13T04:45:31.620ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
171642023-10-13T04:45:31.620ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
171652023-10-13T04:45:31.620ZWARNcrucible: [2] pm_task rx.recv() is None
17166 [8830] Exited with: ExitStatus(unix_wait_status(9))
17167 [8830][2] reports Exit
171682023-10-13T04:45:32.622ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
17169 Oct 13 04:45:35.622 INFO request completed, latency_us: 256, response_code: 200, uri: /state/cid/2, method: GET, req_id: b8353799-b584-4125-a323-110a4869226e, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
17170 001/300 Read block 2112 len 8192
17171 002/300 Flush
17172 003/300 Write block 4352 len 40960 data: 5 6 4 4 4 4 4 4 4 3
17173 004/300 Write block 1780 len 32768 data: 5 4 4 5 4 3 4 4
17174 005/300 Read block 3530 len 24576
17175 006/300 Read block 4625 len 12288
17176 007/300 Flush
17177 008/300 Write block 2657 len 28672 data: 8 5 5 3 3 3 2
17178 009/300 Write block 3171 len 36864 data: 4 4 3 4 4 5 5 4 4
17179 010/300 Read block 796 len 8192
17180 011/300 Read block 4416 len 40960
17181 012/300 Read block 1559 len 20480
17182 013/300 Write block 1241 len 24576 data: 3 3 3 4 4 3
17183 014/300 Write block 1426 len 24576 data: 4 4 5 6 6 4
17184 015/300 Read block 2712 len 36864
17185 016/300 Read block 2467 len 24576
17186 017/300 Write block 3041 len 36864 data: 7 9 9 6 5 4 4 5 5
17187 018/300 Write block 3516 len 12288 data: 2 2 2
17188 019/300 Write block 221 len 28672 data: 3 4 3 3 3 3 4
17189 020/300 Write block 4938 len 32768 data: 2 4 7 7 7 6 5 5
17190 021/300 Flush
17191 022/300 Read block 159 len 28672
17192 023/300 Read block 1705 len 12288
17193 024/300 Write block 996 len 4096 data: 7
17194 025/300 Write block 2116 len 24576 data: 7 6 5 6 6 4
17195 026/300 Read block 2454 len 20480
17196 027/300 Read block 4056 len 24576
17197 028/300 Write block 3512 len 28672 data: 4 4 2 2 3 3 3
17198 029/300 Write block 1758 len 24576 data: 3 2 2 2 3 3
17199 030/300 Read block 2788 len 36864
17200 031/300 Write block 3168 len 20480 data: 5 5 6 5 5
17201 032/300 Read block 839 len 4096
17202 033/300 Read block 946 len 28672
17203 034/300 Write block 630 len 28672 data: 2 2 2 3 2 2 3
17204 035/300 Read block 776 len 20480
17205 036/300 Read block 4612 len 28672
17206 037/300 Write block 2255 len 8192 data: 2 2
17207 038/300 Read block 2337 len 32768
17208 039/300 Write block 3221 len 36864 data: 5 4 4 3 3 3 3 3 3
17209 040/300 Write block 458 len 8192 data: 4 5
17210 041/300 Write block 3387 len 40960 data: 5 6 5 5 7 7 6 6 6 8
17211 042/300 Read block 1829 len 20480
17212 043/300 Flush
17213 044/300 Read block 3961 len 40960
17214 045/300 Flush
17215 046/300 Read block 1653 len 8192
17216 047/300 Flush
17217 048/300 Write block 1862 len 40960 data: 4 6 5 5 4 4 3 3 5 5
17218 049/300 Read block 497 len 24576
17219 050/300 Write block 357 len 4096 data: 5
17220 051/300 Read block 2784 len 32768
17221 052/300 Write block 1306 len 28672 data: 5 4 4 4 4 4 4
17222 053/300 Write block 1799 len 8192 data: 4 3
17223 054/300 Write block 3247 len 28672 data: 3 4 4 5 4 4 4
17224 055/300 Write block 1671 len 40960 data: 6 5 5 2 2 2 2 3 4 4
17225 056/300 Read block 1410 len 32768
17226 057/300 Read block 2307 len 28672
17227 058/300 Write block 2202 len 32768 data: 4 3 2 2 2 2 2 3
17228 059/300 Write block 4623 len 24576 data: 7 7 7 6 4 3
17229 060/300 Write block 1497 len 32768 data: 8 7 6 5 4 4 4 4
17230 061/300 Write block 904 len 40960 data: 8 5 4 4 3 3 4 4 5 7
17231 062/300 Write block 4125 len 20480 data: 6 5 6 5 6
17232 063/300 Read block 2921 len 16384
17233 064/300 Flush
17234 065/300 Write block 2164 len 4096 data: 7
17235 066/300 Write block 2507 len 36864 data: 6 5 4 6 7 6 6 6 7
17236 067/300 Read block 3871 len 4096
17237 068/300 Write block 2231 len 40960 data: 3 2 2 2 4 3 3 3 5 6
17238 069/300 Flush
17239 070/300 Write block 35 len 20480 data: 5 3 3 4 4
17240 071/300 Read block 3826 len 28672
17241 072/300 Read block 2569 len 36864
17242 073/300 Read block 3089 len 4096
17243 074/300 Write block 2160 len 16384 data: 6 7 8 7
17244 075/300 Write block 756 len 4096 data: 9
17245 076/300 Read block 4512 len 36864
17246 077/300 Write block 799 len 8192 data: 3 4
17247 078/300 Write block 4308 len 24576 data: 5 4 3 2 3 4
17248 079/300 Read block 3259 len 24576
17249 080/300 Read block 1250 len 36864
17250 081/300 Write block 4388 len 12288 data: 5 5 6
17251 082/300 Write block 1223 len 16384 data: 3 4 4 4
17252 083/300 Read block 2736 len 4096
17253 084/300 Read block 4339 len 4096
17254 085/300 Write block 4664 len 16384 data: 4 4 3 3
17255 086/300 Read block 1037 len 12288
17256 087/300 Write block 4573 len 36864 data: 7 6 6 6 6 6 5 3 3
17257 088/300 Write block 1328 len 20480 data: 2 2 2 2 2
17258 089/300 Read block 4 len 24576
17259 090/300 Read block 2934 len 12288
17260 091/300 Read block 3192 len 8192
17261 092/300 Read block 2775 len 12288
17262 093/300 Flush
17263 094/300 Write block 437 len 28672 data: 7 5 4 6 6 6 6
17264 095/300 Flush
17265 096/300 Read block 366 len 20480
17266 097/300 Flush
17267 098/300 Write block 4050 len 28672 data: 4 4 5 5 5 4 4
17268 099/300 Read block 4226 len 24576
17269 100/300 Write block 1869 len 32768 data: 4 6 6 5 4 5 5 6
17270 101/300 Write block 1753 len 8192 data: 6 5
17271 102/300 Write block 2597 len 20480 data: 6 5 6 5 4
17272 103/300 Read block 813 len 4096
17273 104/300 Write block 3761 len 40960 data: 5 6 6 5 4 4 6 6 5 5
17274 105/300 Flush
17275 106/300 Read block 4028 len 40960
17276 107/300 Read block 2943 len 4096
17277 108/300 Read block 1528 len 32768
17278 109/300 Read block 4665 len 36864
17279 110/300 Read block 835 len 40960
17280 111/300 Read block 2945 len 20480
17281 112/300 Read block 4156 len 32768
17282 113/300 Write block 2248 len 40960 data: 6 5 5 4 3 2 2 3 3 2
17283 114/300 Flush
17284 115/300 Read block 4281 len 20480
17285 116/300 Write block 418 len 24576 data: 5 5 4 4 4 4
17286 117/300 Read block 2077 len 12288
17287 118/300 Read block 3543 len 32768
17288 119/300 Read block 4586 len 24576
17289 120/300 Read block 2349 len 40960
17290 121/300 Read block 1 len 12288
17291 122/300 Read block 3960 len 40960
17292 123/300 Write block 2727 len 16384 data: 3 4 5 5
17293 124/300 Read block 1518 len 28672
17294 125/300 Read block 3170 len 4096
17295 126/300 Read block 3271 len 24576
17296 127/300 Write block 3756 len 12288 data: 3 4 4
17297 128/300 Read block 1433 len 32768
17298 129/300 Write block 1419 len 8192 data: 5 2
17299 130/300 Read block 2530 len 4096
17300 131/300 Write block 1109 len 28672 data: 5 4 4 2 3 2 3
17301 132/300 Write block 2493 len 16384 data: 6 6 4 3
17302 133/300 Read block 1465 len 16384
17303 134/300 Write block 1785 len 12288 data: 4 5 5
17304 135/300 Flush
17305 136/300 Write block 2466 len 24576 data: 3 3 3 5 5 6
17306 137/300 Read block 1278 len 20480
17307 138/300 Read block 2651 len 16384
17308 139/300 Read block 1567 len 28672
17309 140/300 Read block 488 len 16384
17310 141/300 Read block 3868 len 36864
17311 142/300 Write block 2965 len 24576 data: 4 4 6 7 6 6
17312 143/300 Read block 4903 len 4096
17313 144/300 Read block 2382 len 8192
17314 145/300 Read block 3156 len 4096
17315 146/300 Read block 1135 len 20480
17316 147/300 Read block 2510 len 16384
17317 148/300 Write block 956 len 40960 data: 3 2 2 3 2 2 2 2 2 2
17318 149/300 Write block 1135 len 20480 data: 3 2 2 4 3
17319 150/300 Read block 1660 len 28672
17320 151/300 Write block 4052 len 36864 data: 6 6 6 5 5 4 4 4 3
17321 152/300 Write block 1268 len 12288 data: 3 4 3
17322 153/300 Read block 1198 len 28672
17323 154/300 Write block 1161 len 40960 data: 5 5 4 4 5 5 6 5 6 5
17324 155/300 Write block 3507 len 40960 data: 4 4 4 4 4 5 5 3 3 4
17325 156/300 Write block 739 len 4096 data: 3
17326 157/300 Read block 381 len 36864
17327 158/300 Read block 1505 len 40960
17328 159/300 Write block 4682 len 40960 data: 7 7 6 5 5 6 6 3 4 4
17329 160/300 Read block 3565 len 16384
17330 161/300 Read block 172 len 32768
17331 162/300 Write block 827 len 24576 data: 3 3 4 4 5 5
17332 163/300 Write block 764 len 36864 data: 4 4 4 2 2 2 3 3 3
17333 164/300 Write block 2231 len 4096 data: 4
17334 165/300 Flush
17335 166/300 Read block 989 len 40960
17336 167/300 Read block 2278 len 32768
17337 168/300 Write block 1166 len 4096 data: 6
17338 169/300 Read block 3292 len 24576
17339 170/300 Read block 4625 len 36864
17340 171/300 Read block 2508 len 20480
17341 172/300 Write block 4005 len 24576 data: 6 5 5 6 3 3
17342 173/300 Read block 4519 len 36864
17343 174/300 Write block 3211 len 4096 data: 5
17344 175/300 Read block 2062 len 20480
17345 176/300 Read block 1934 len 24576
17346 177/300 Read block 3485 len 4096
17347 178/300 Flush
17348 179/300 Flush
17349 180/300 Read block 3189 len 32768
17350 181/300 Write block 2613 len 12288 data: 7 7 7
17351 182/300 Read block 2996 len 12288
17352 183/300 Read block 3465 len 40960
17353 184/300 Write block 822 len 20480 data: 4 3 2 2 3
17354 185/300 Read block 3448 len 28672
17355 186/300 Write block 1638 len 28672 data: 5 5 6 5 6 5 6
17356 187/300 Write block 605 len 36864 data: 4 4 3 3 5 7 8 9 8
17357 188/300 Flush
17358 189/300 Read block 2578 len 8192
17359 190/300 Read block 1616 len 12288
17360 191/300 Read block 767 len 20480
17361 192/300 Write block 915 len 20480 data: 6 5 5 6 6
17362 193/300 Write block 417 len 36864 data: 5 6 6 5 5 5 5 5 5
17363 194/300 Read block 2070 len 16384
17364 195/300 Read block 4169 len 36864
17365 196/300 Read block 1136 len 36864
17366 197/300 Write block 591 len 32768 data: 4 4 4 5 5 6 5 5
17367 198/300 Write block 1807 len 36864 data: 3 3 4 6 7 6 5 3 3
17368 199/300 Read block 4638 len 4096
17369 200/300 Read block 418 len 12288
17370 201/300 Write block 2718 len 12288 data: 4 4 6
17371 202/300 Write block 3764 len 20480 data: 6 5 5 7 7
17372 203/300 Read block 3981 len 8192
17373 204/300 Write block 3430 len 12288 data: 8 7 6
17374 205/300 Write block 3167 len 24576 data: 6 6 6 7 6 6
17375 206/300 Read block 4881 len 16384
17376 207/300 Write block 4017 len 20480 data: 4 5 6 6 7
17377 208/300 Read block 673 len 36864
17378 209/300 Write block 968 len 40960 data: 5 7 8 5 7 7 6 7 7 6
17379 210/300 Read block 1300 len 8192
17380 211/300 Write block 2558 len 32768 data: 6 4 5 5 4 4 5 5
17381 212/300 Read block 1547 len 8192
17382 213/300 Write block 586 len 8192 data: 3 5
17383 214/300 Write block 366 len 40960 data: 3 3 3 2 3 3 3 3 3 5
17384 215/300 Read block 693 len 8192
17385 216/300 Read block 67 len 4096
17386 217/300 Write block 2177 len 16384 data: 4 3 3 3
17387 218/300 Flush
17388 219/300 Read block 2881 len 4096
17389 220/300 Write block 4403 len 4096 data: 5
17390 221/300 Read block 680 len 12288
17391 222/300 Read block 4092 len 16384
17392 223/300 Read block 2057 len 36864
17393 224/300 Write block 3609 len 16384 data: 5 4 6 5
17394 225/300 Write block 1699 len 20480 data: 8 8 7 4 2
17395 226/300 Read block 819 len 24576
17396 227/300 Read block 23 len 12288
17397 228/300 Write block 3545 len 24576 data: 3 3 3 4 5 4
17398 229/300 Flush
17399 230/300 Write block 3658 len 36864 data: 5 5 4 3 3 2 3 3 3
17400 231/300 Write block 714 len 16384 data: 4 5 6 6
17401 232/300 Write block 4207 len 8192 data: 5 6
17402 233/300 Read block 158 len 36864
17403 234/300 Read block 2304 len 8192
17404 235/300 Read block 2781 len 4096
17405 236/300 Write block 3337 len 20480 data: 3 3 3 3 3
17406 237/300 Read block 914 len 28672
17407 238/300 Read block 3992 len 4096
17408 239/300 Read block 3805 len 20480
17409 240/300 Read block 1288 len 24576
17410 241/300 Read block 4528 len 8192
17411 242/300 Write block 45 len 24576 data: 6 7 6 6 6 5
17412 243/300 Write block 4592 len 40960 data: 3 3 2 2 2 2 3 4 4 5
17413 244/300 Flush
17414 245/300 Write block 3032 len 36864 data: 6 5 4 4 4 5 5 5 6
17415 246/300 Write block 2728 len 28672 data: 5 6 6 5 6 4 4
17416 247/300 Write block 3576 len 12288 data: 5 5 4
17417 248/300 Read block 14 len 4096
17418 249/300 Write block 1675 len 16384 data: 3 3 3 4
17419 250/300 Write block 2538 len 8192 data: 4 4
17420 251/300 Write block 929 len 12288 data: 5 5 3
17421 252/300 Read block 4637 len 28672
17422 253/300 Read block 4368 len 12288
17423 254/300 Flush
17424 255/300 Write block 4727 len 24576 data: 7 6 6 5 3 4
17425 256/300 Read block 2485 len 40960
17426 257/300 Read block 3972 len 28672
17427 258/300 Flush
17428 259/300 Write block 4269 len 4096 data: 2
17429 260/300 Write block 496 len 8192 data: 4 4
17430 261/300 Read block 2132 len 40960
17431 262/300 Read block 2635 len 32768
17432 263/300 Write block 4660 len 40960 data: 6 4 5 5 5 5 4 4 3 3
17433 264/300 Write block 1918 len 32768 data: 4 4 4 6 7 6 6 6
17434 265/300 Read block 113 len 12288
17435 266/300 Read block 4961 len 16384
17436 267/300 Write block 2676 len 8192 data: 2 3
17437 268/300 Write block 1087 len 28672 data: 6 7 7 6 5 5 8
17438 269/300 Write block 4742 len 20480 data: 2 3 3 4 4
17439 270/300 Write block 3551 len 24576 data: 4 7 9 6 7 7
17440 271/300 Write block 48 len 24576 data: 7 7 6 3 4 4
17441 272/300 Read block 3711 len 16384
17442 273/300 Write block 2851 len 36864 data: 6 7 7 5 4 5 4 3 4
17443 274/300 Read block 625 len 28672
17444 275/300 Read block 1950 len 20480
17445 276/300 Read block 1168 len 12288
17446 277/300 Write block 4311 len 28672 data: 3 4 5 3 3 4 6
17447 278/300 Write block 4676 len 12288 data: 2 2 2
17448 279/300 Read block 4224 len 40960
17449 280/300 Write block 2805 len 28672 data: 4 4 2 2 4 5 4
17450 281/300 Read block 2590 len 12288
17451 282/300 Read block 3014 len 36864
17452 283/300 Read block 3419 len 28672
17453 284/300 Read block 2464 len 12288
17454 285/300 Read block 2767 len 8192
17455 286/300 Read block 1581 len 40960
17456 287/300 Read block 1734 len 32768
17457 288/300 Read block 2032 len 32768
17458 289/300 Read block 4859 len 36864
17459 290/300 Read block 3834 len 40960
17460 291/300 Flush
17461 292/300 Write block 251 len 36864 data: 5 3 5 6 9 7 7 6 5
17462 293/300 Write block 305 len 28672 data: 4 4 6 5 5 5 6
17463 294/300 Flush
17464 295/300 Read block 3816 len 4096
17465 296/300 Write block 1695 len 20480 data: 7 6 7 7 9
17466 297/300 Read block 3605 len 36864
17467 298/300 Read block 4878 len 12288
17468 299/300 Read block 1655 len 36864
17469 300/300 Write block 1009 len 36864 data: 5 5 4 5 6 7 6 6 5
17470 Main task has work to do, go find it
17471 got dsc Start(2)
17472 start 2
17473 Oct 13 04:45:36.510 INFO request completed, latency_us: 304, response_code: 204, uri: /start/cid/2, method: POST, req_id: a606585e-0897-4ec1-b37f-247423f0a073, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
17474 Replay: started 2, returned:Ok(())
17475 ----------------------------------------------------------------
17476 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
17477 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
17478 6048 Acked 7047 Read 2 Done Done New false
17479 6049 Acked 7048 Flush 0 Done Done New false
17480 6050 Acked 7049 Write 10 Done Done New false
17481 6051 Acked 7050 Write 8 Done Done New false
17482 6052 Acked 7051 Read 6 Done Done New false
17483 6053 Acked 7052 Read 3 Done Done New false
17484 6054 Acked 7053 Flush 0 Done Done New false
17485 6055 Acked 7054 Write 7 Done Done New false
17486 6056 Acked 7055 Write 9 Done Done New false
17487 6057 Acked 7056 Read 2 Done Done New false
17488 6058 Acked 7057 Read 10 Done Done New false
17489 6059 Acked 7058 Read 5 Done Done New false
17490 6060 Acked 7059 Write 6 Done Done New false
17491 6061 Acked 7060 Write 6 Done Done New false
17492 6062 Acked 7061 Read 9 Done Done New false
17493 6063 Acked 7062 Read 6 Done Done New false
17494 6064 Acked 7063 Write 9 Done Done New false
17495 6065 Acked 7064 Write 3 Done Done New false
17496 6066 Acked 7065 Write 7 Done Done New false
17497 6067 Acked 7066 Write 8 Done Done New false
17498 6068 Acked 7067 Flush 0 Done Done New false
17499 6069 Acked 7068 Read 7 Done Done New false
17500 6070 Acked 7069 Read 3 Done Done New false
17501 6071 Acked 7070 Write 1 Done Done New false
17502 6072 Acked 7071 Write 6 Done Done New false
17503 6073 Acked 7072 Read 5 Done Done New false
17504 6074 Acked 7073 Read 6 Done Done New false
17505 6075 Acked 7074 Write 7 Done Done New false
17506 6076 Acked 7075 Write 6 Done Done New false
17507 6077 Acked 7076 Read 9 Done Done New false
17508 6078 Acked 7077 Write 5 Done Done New false
17509 6079 Acked 7078 Read 1 Done Done New false
17510 6080 Acked 7079 Read 7 Done Done New false
17511 6081 Acked 7080 Write 7 Done Done New false
17512 6082 Acked 7081 Read 5 Done Done New false
17513 6083 Acked 7082 Read 7 Done Done New false
17514 6084 Acked 7083 Write 2 Done Done New false
17515 6085 Acked 7084 Read 8 Done Done New false
17516 6086 Acked 7085 Write 9 Done Done New false
17517 6087 Acked 7086 Write 2 Done Done New false
17518 6088 Acked 7087 Write 10 Done Done New false
17519 6089 Acked 7088 Read 5 Done Done New false
17520 6090 Acked 7089 Flush 0 Done Done New false
17521 6091 Acked 7090 Read 10 Done Done New false
17522 6092 Acked 7091 Flush 0 Done Done New false
17523 6093 Acked 7092 Read 2 Done Done New false
17524 6094 Acked 7093 Flush 0 Done Done New false
17525 6095 Acked 7094 Flush 0 Done Done New false
17526 6096 Acked 7095 Write 10 Done Done New false
17527 6097 Acked 7096 Read 6 Done Done New false
17528 6098 Acked 7097 Write 1 Done Done New false
17529 6099 Acked 7098 Read 8 Done Done New false
17530 6100 Acked 7099 Write 7 Done Done New false
17531 6101 Acked 7100 Write 2 Done Done New false
17532 6102 Acked 7101 Write 7 Done Done New false
17533 6103 Acked 7102 Write 10 Done Done New false
17534 6104 Acked 7103 Read 8 Done Done New false
17535 6105 Acked 7104 Read 7 Done Done New false
17536 6106 Acked 7105 Write 8 Done Done New false
17537 6107 Acked 7106 Write 6 Done Done New false
17538 6108 Acked 7107 Write 8 Done Done New false
17539 6109 Acked 7108 Write 10 Done Done New false
17540 6110 Acked 7109 Write 5 Done Done New false
17541 6111 Acked 7110 Read 4 Done Done New false
17542 6112 Acked 7111 Flush 0 Done Done New false
17543 6113 Acked 7112 Write 1 Done Done New false
17544 6114 Acked 7113 Write 9 Done Done New false
17545 6115 Acked 7114 Read 1 Done Done New false
17546 6116 Acked 7115 Write 10 Done Done New false
17547 6117 Acked 7116 Flush 0 Done Done New false
17548 6118 Acked 7117 Write 5 Done Done New false
17549 6119 Acked 7118 Read 7 Done Done New false
17550 6120 Acked 7119 Read 9 Done Done New false
17551 6121 Acked 7120 Read 1 Done Done New false
17552 6122 Acked 7121 Write 4 Done Done New false
17553 6123 Acked 7122 Write 1 Done Done New false
17554 6124 Acked 7123 Read 9 Done Done New false
17555 6125 Acked 7124 Write 2 Done Done New false
17556 6126 Acked 7125 Write 6 Done Done New false
17557 6127 Acked 7126 Read 6 Done Done New false
17558 6128 Acked 7127 Read 9 Done Done New false
17559 6129 Acked 7128 Write 3 Done Done New false
17560 6130 Acked 7129 Write 4 Done Done New false
17561 6131 Acked 7130 Read 1 Done Done New false
17562 6132 Acked 7131 Read 1 Done Done New false
17563 6133 Acked 7132 Write 4 Done Done New false
17564 6134 Acked 7133 Read 3 Done Done New false
17565 6135 Acked 7134 Write 9 Done Done New false
17566 6136 Acked 7135 Write 5 Done Done New false
17567 6137 Acked 7136 Read 6 Done Done New false
17568 6138 Acked 7137 Read 3 Done Done New false
17569 6139 Acked 7138 Read 2 Done Done New false
17570 6140 Acked 7139 Read 3 Done Done New false
17571 6141 Acked 7140 Flush 0 Done Done New false
17572 6142 Acked 7141 Write 7 Done Done New false
17573 6143 Acked 7142 Flush 0 Done Done New false
17574 6144 Acked 7143 Read 5 Done Done New false
17575 6145 Acked 7144 Flush 0 Done Done New false
17576 6146 Acked 7145 Write 7 Done Done New false
17577 6147 Acked 7146 Read 6 Done Done New false
17578 6148 Acked 7147 Write 8 Done Done New false
17579 6149 Acked 7148 Write 2 Done Done New false
17580 6150 Acked 7149 Write 5 Done Done New false
17581 6151 Acked 7150 Read 1 Done Done New false
17582 6152 Acked 7151 Write 10 Done Done New false
17583 6153 Acked 7152 Flush 0 Done Done New false
17584 6154 Acked 7153 Read 10 Done Done New false
17585 6155 Acked 7154 Read 1 Done Done New false
17586 6156 Acked 7155 Read 8 Done Done New false
17587 6157 Acked 7156 Read 9 Done Done New false
17588 6158 Acked 7157 Read 10 Done Done New false
17589 6159 Acked 7158 Read 5 Done Done New false
17590 6160 Acked 7159 Read 8 Done Done New false
17591 6161 Acked 7160 Write 10 Done Done New false
17592 6162 Acked 7161 Flush 0 Done Done New false
17593 6163 Acked 7162 Read 5 Done Done New false
17594 6164 Acked 7163 Write 6 Done Done New false
17595 6165 Acked 7164 Read 3 Done Done New false
17596 6166 Acked 7165 Read 8 Done Done New false
17597 6167 Acked 7166 Read 6 Done Done New false
17598 6168 Acked 7167 Read 10 Done Done New false
17599 6169 Acked 7168 Read 3 Done Done New false
17600 6170 Acked 7169 Read 10 Done Done New false
17601 6171 Acked 7170 Write 4 Done Done New false
17602 6172 Acked 7171 Read 7 Done Done New false
17603 6173 Acked 7172 Read 1 Done Done New false
17604 6174 Acked 7173 Read 6 Done Done New false
17605 6175 Acked 7174 Write 3 Done Done New false
17606 6176 Acked 7175 Read 8 Done Done New false
17607 6177 Acked 7176 Write 2 Done Done New false
17608 6178 Acked 7177 Read 1 Done Done New false
17609 6179 Acked 7178 Write 7 Done Done New false
17610 6180 Acked 7179 Write 4 Done Done New false
17611 6181 Acked 7180 Read 4 Done Done New false
17612 6182 Acked 7181 Write 3 Done Done New false
17613 6183 Acked 7182 Flush 0 Done Done New false
17614 6184 Acked 7183 Write 6 Done Done New false
17615 6185 Acked 7184 Read 5 Done Done New false
17616 6186 Acked 7185 Read 4 Done Done New false
17617 6187 Acked 7186 Read 7 Done Done New false
17618 6188 Acked 7187 Read 4 Done Done New false
17619 6189 Acked 7188 Read 9 Done Done New false
17620 6190 Acked 7189 Write 6 Done Done New false
17621 6191 Acked 7190 Read 1 Done Done New false
17622 6192 Acked 7191 Read 2 Done Done New false
17623 6193 Acked 7192 Read 1 Done Done New false
17624 6194 Acked 7193 Read 5 Done Done New false
17625 6195 Acked 7194 Read 4 Done Done New false
17626 6196 Acked 7195 Write 10 Done Done New false
17627 6197 Acked 7196 Write 5 Done Done New false
17628 6198 Acked 7197 Read 7 Done Done New false
17629 6199 Acked 7198 Write 9 Done Done New false
17630 6200 Acked 7199 Write 3 Done Done New false
17631 6201 Acked 7200 Read 7 Done Done New false
17632 6202 Acked 7201 Write 10 Done Done New false
17633 6203 Acked 7202 Write 10 Done Done New false
17634 6204 Acked 7203 Write 1 Done Done New false
17635 6205 Acked 7204 Read 9 Done Done New false
17636 6206 Acked 7205 Read 10 Done Done New false
17637 6207 Acked 7206 Write 10 Done Done New false
17638 6208 Acked 7207 Read 4 Done Done New false
17639 6209 Acked 7208 Read 8 Done Done New false
17640 6210 Acked 7209 Write 6 Done Done New false
17641 6211 Acked 7210 Write 9 Done Done New false
17642 6212 Acked 7211 Write 1 Done Done New false
17643 6213 Acked 7212 Flush 0 Done Done New false
17644 6214 Acked 7213 Read 10 Done Done New false
17645 6215 Acked 7214 Read 8 Done Done New false
17646 6216 Acked 7215 Write 1 Done Done New false
17647 6217 Acked 7216 Read 6 Done Done New false
17648 6218 Acked 7217 Read 9 Done Done New false
17649 6219 Acked 7218 Read 5 Done Done New false
17650 6220 Acked 7219 Write 6 Done Done New false
17651 6221 Acked 7220 Read 9 Done Done New false
17652 6222 Acked 7221 Write 1 Done Done New false
17653 6223 Acked 7222 Read 5 Done Done New false
17654 6224 Acked 7223 Read 6 Done Done New false
17655 6225 Acked 7224 Read 1 Done Done New false
17656 6226 Acked 7225 Flush 0 Done Done New false
17657 6227 Acked 7226 Flush 0 Done Done New false
17658 6228 Acked 7227 Read 8 Done Done New false
17659 6229 Acked 7228 Write 3 Done Done New false
17660 6230 Acked 7229 Read 3 Done Done New false
17661 6231 Acked 7230 Read 10 Done Done New false
17662 6232 Acked 7231 Write 5 Done Done New false
17663 6233 Acked 7232 Read 7 Done Done New false
17664 6234 Acked 7233 Write 7 Done Done New false
17665 6235 Acked 7234 Write 9 Done Done New false
17666 6236 Acked 7235 Flush 0 Done Done New false
17667 6237 Acked 7236 Read 2 Done Done New false
17668 6238 Acked 7237 Read 3 Done Done New false
17669 6239 Acked 7238 Read 5 Done Done New false
17670 6240 Acked 7239 Write 5 Done Done New false
17671 6241 Acked 7240 Write 9 Done Done New false
17672 6242 Acked 7241 Read 4 Done Done New false
17673 6243 Acked 7242 Read 9 Done Done New false
17674 6244 Acked 7243 Read 9 Done Done New false
17675 6245 Acked 7244 Write 8 Done Done New false
17676 6246 Acked 7245 Write 9 Done Done New false
17677 6247 Acked 7246 Read 1 Done Done New false
17678 6248 Acked 7247 Read 3 Done Done New false
17679 6249 Acked 7248 Write 3 Done Done New false
17680 6250 Acked 7249 Write 5 Done Done New false
17681 6251 Acked 7250 Read 2 Done Done New false
17682 6252 Acked 7251 Write 3 Done Done New false
17683 6253 Acked 7252 Write 6 Done Done New false
17684 6254 Acked 7253 Read 4 Done Done New false
17685 6255 Acked 7254 Write 5 Done Done New false
17686 6256 Acked 7255 Read 9 Done Done New false
17687 6257 Acked 7256 Write 10 Done Done New false
17688 6258 Acked 7257 Read 2 Done Done New false
17689 6259 Acked 7258 Write 8 Done Done New false
17690 6260 Acked 7259 Read 2 Done Done New false
17691 6261 Acked 7260 Write 2 Done Done New false
17692 6262 Acked 7261 Write 10 Done Done New false
17693 6263 Acked 7262 Read 2 Done Done New false
17694 6264 Acked 7263 Read 1 Done Done New false
17695 6265 Acked 7264 Write 4 Done Done New false
17696 6266 Acked 7265 Flush 0 Done Done New false
17697 6267 Acked 7266 Read 1 Done Done New false
17698 6268 Acked 7267 Write 1 Done Done New false
17699 6269 Acked 7268 Read 3 Done Done New false
17700 6270 Acked 7269 Read 4 Done Done New false
17701 6271 Acked 7270 Read 9 Done Done New false
17702 6272 Acked 7271 Write 4 Done Done New false
17703 6273 Acked 7272 Write 5 Done Done New false
17704 6274 Acked 7273 Read 6 Done Done New false
17705 6275 Acked 7274 Read 3 Done Done New false
17706 6276 Acked 7275 Write 6 Done Done New false
17707 6277 Acked 7276 Flush 0 Done Done New false
17708 6278 Acked 7277 Write 9 Done Done New false
17709 6279 Acked 7278 Write 4 Done Done New false
17710 6280 Acked 7279 Write 2 Done Done New false
17711 6281 Acked 7280 Read 9 Done Done New false
17712 6282 Acked 7281 Read 2 Done Done New false
17713 6283 Acked 7282 Read 1 Done Done New false
17714 6284 Acked 7283 Write 5 Done Done New false
17715 6285 Acked 7284 Read 7 Done Done New false
17716 6286 Acked 7285 Read 1 Done Done New false
17717 6287 Acked 7286 Read 5 Done Done New false
17718 6288 Acked 7287 Read 6 Done Done New false
17719 6289 Acked 7288 Read 2 Done Done New false
17720 6290 Acked 7289 Write 6 Done Done New false
17721 6291 Acked 7290 Write 10 Done Done New false
17722 6292 Acked 7291 Flush 0 Done Done New false
17723 6293 Acked 7292 Write 9 Done Done New false
17724 6294 Acked 7293 Write 7 Done Done New false
17725 6295 Acked 7294 Write 3 Done Done New false
17726 6296 Acked 7295 Read 1 Done Done New false
17727 6297 Acked 7296 Write 4 Done Done New false
17728 6298 Acked 7297 Write 2 Done Done New false
17729 6299 Acked 7298 Write 3 Done Done New false
17730 6300 Acked 7299 Read 7 Done Done New false
17731 6301 Acked 7300 Read 3 Done Done New false
17732 6302 Acked 7301 Flush 0 Done Done New false
17733 6303 Acked 7302 Write 6 Done Done New false
17734 6304 Acked 7303 Read 10 Done Done New false
17735 6305 Acked 7304 Read 7 Done Done New false
17736 6306 Acked 7305 Flush 0 Done Done New false
17737 6307 Acked 7306 Write 1 Done Done New false
17738 6308 Acked 7307 Write 2 Done Done New false
17739 6309 Acked 7308 Read 10 Done Done New false
17740 6310 Acked 7309 Read 8 Done Done New false
17741 6311 Acked 7310 Write 10 Done Done New false
17742 6312 Acked 7311 Write 8 Done Done New false
17743 6313 Acked 7312 Read 3 Done Done New false
17744 6314 Acked 7313 Read 4 Done Done New false
17745 6315 Acked 7314 Write 2 Done Done New false
17746 6316 Acked 7315 Write 7 Done Done New false
17747 6317 Acked 7316 Write 5 Done Done New false
17748 6318 Acked 7317 Write 6 Done Done New false
17749 6319 Acked 7318 Write 6 Done Done New false
17750 6320 Acked 7319 Read 4 Done Done New false
17751 6321 Acked 7320 Write 9 Done Done New false
17752 6322 Acked 7321 Read 7 Done Done New false
17753 6323 Acked 7322 Read 5 Done Done New false
17754 6324 Acked 7323 Read 3 Done Done New false
17755 6325 Acked 7324 Write 7 Done Done New false
17756 6326 Acked 7325 Write 3 Done Done New false
17757 6327 Acked 7326 Read 10 Done Done New false
17758 6328 Acked 7327 Write 7 Done Done New false
17759 6329 Acked 7328 Read 3 Done Done New false
17760 6330 Acked 7329 Read 9 Done Done New false
17761 6331 Acked 7330 Read 7 Done Done New false
17762 6332 Acked 7331 Read 3 Done Done New false
17763 6333 Acked 7332 Read 2 Done Done New false
17764 6334 Acked 7333 Read 10 Done Done New false
17765 6335 Acked 7334 Read 8 Done Done New false
17766 6336 Acked 7335 Read 8 Done Done New false
17767 6337 Acked 7336 Read 9 Done Done New false
17768 6338 Acked 7337 Read 10 Done Done New false
17769 6339 Acked 7338 Flush 0 Done Done New false
17770 6340 Acked 7339 Write 9 Done Done New false
17771 6341 Acked 7340 Write 7 Done Done New false
17772 6342 Acked 7341 Flush 0 Done Done New false
17773 6343 Acked 7342 Read 1 Done Done New false
17774 6344 Acked 7343 Write 5 Done Done New false
17775 6345 Acked 7344 Read 9 Done Done New false
17776 6346 Acked 7345 Read 3 Done Done New false
17777 6347 Acked 7346 Read 9 Done Done New false
17778 6348 Acked 7347 Write 9 Sent Sent New false
17779 STATES DS:0 DS:1 DS:2 TOTAL
17780 New 0 0 301 301
17781 Sent 1 1 0 2
17782 Done 300 300 0 600
17783 Skipped 0 0 0 0
17784 Error 0 0 0 0
17785 Last Flush: 7341 7341 7046
17786 Downstairs last five completed: 7046 7045 7044 7043 7042 7041
17787 Upstairs last five completed: 6348 6347 6346 6345 6344 6343
17788 CLIENT: Up:0 ds:301 act:2
17789 ----------------------------------------------------------------
17790 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
17791 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
17792 6048 Acked 7047 Read 2 Done Done New false
17793 6049 Acked 7048 Flush 0 Done Done New false
17794 6050 Acked 7049 Write 10 Done Done New false
17795 6051 Acked 7050 Write 8 Done Done New false
17796 6052 Acked 7051 Read 6 Done Done New false
17797 6053 Acked 7052 Read 3 Done Done New false
17798 6054 Acked 7053 Flush 0 Done Done New false
17799 6055 Acked 7054 Write 7 Done Done New false
17800 6056 Acked 7055 Write 9 Done Done New false
17801 6057 Acked 7056 Read 2 Done Done New false
17802 6058 Acked 7057 Read 10 Done Done New false
17803 6059 Acked 7058 Read 5 Done Done New false
17804 6060 Acked 7059 Write 6 Done Done New false
17805 6061 Acked 7060 Write 6 Done Done New false
17806 6062 Acked 7061 Read 9 Done Done New false
17807 6063 Acked 7062 Read 6 Done Done New false
17808 6064 Acked 7063 Write 9 Done Done New false
17809 6065 Acked 7064 Write 3 Done Done New false
17810 6066 Acked 7065 Write 7 Done Done New false
17811 6067 Acked 7066 Write 8 Done Done New false
17812 6068 Acked 7067 Flush 0 Done Done New false
17813 6069 Acked 7068 Read 7 Done Done New false
17814 6070 Acked 7069 Read 3 Done Done New false
17815 6071 Acked 7070 Write 1 Done Done New false
17816 6072 Acked 7071 Write 6 Done Done New false
17817 6073 Acked 7072 Read 5 Done Done New false
17818 6074 Acked 7073 Read 6 Done Done New false
17819 6075 Acked 7074 Write 7 Done Done New false
17820 6076 Acked 7075 Write 6 Done Done New false
17821 6077 Acked 7076 Read 9 Done Done New false
17822 6078 Acked 7077 Write 5 Done Done New false
17823 6079 Acked 7078 Read 1 Done Done New false
17824 6080 Acked 7079 Read 7 Done Done New false
17825 6081 Acked 7080 Write 7 Done Done New false
17826 6082 Acked 7081 Read 5 Done Done New false
17827 6083 Acked 7082 Read 7 Done Done New false
17828 6084 Acked 7083 Write 2 Done Done New false
17829 6085 Acked 7084 Read 8 Done Done New false
17830 6086 Acked 7085 Write 9 Done Done New false
17831 6087 Acked 7086 Write 2 Done Done New false
17832 6088 Acked 7087 Write 10 Done Done New false
17833 6089 Acked 7088 Read 5 Done Done New false
17834 6090 Acked 7089 Flush 0 Done Done New false
17835 6091 Acked 7090 Read 10 Done Done New false
17836 6092 Acked 7091 Flush 0 Done Done New false
17837 6093 Acked 7092 Read 2 Done Done New false
17838 6094 Acked 7093 Flush 0 Done Done New false
17839 6095 Acked 7094 Flush 0 Done Done New false
17840 6096 Acked 7095 Write 10 Done Done New false
17841 6097 Acked 7096 Read 6 Done Done New false
17842 6098 Acked 7097 Write 1 Done Done New false
17843 6099 Acked 7098 Read 8 Done Done New false
17844 6100 Acked 7099 Write 7 Done Done New false
17845 6101 Acked 7100 Write 2 Done Done New false
17846 6102 Acked 7101 Write 7 Done Done New false
17847 6103 Acked 7102 Write 10 Done Done New false
17848 6104 Acked 7103 Read 8 Done Done New false
17849 6105 Acked 7104 Read 7 Done Done New false
17850 6106 Acked 7105 Write 8 Done Done New false
17851 6107 Acked 7106 Write 6 Done Done New false
17852 6108 Acked 7107 Write 8 Done Done New false
17853 6109 Acked 7108 Write 10 Done Done New false
17854 6110 Acked 7109 Write 5 Done Done New false
17855 6111 Acked 7110 Read 4 Done Done New false
17856 6112 Acked 7111 Flush 0 Done Done New false
17857 6113 Acked 7112 Write 1 Done Done New false
17858 6114 Acked 7113 Write 9 Done Done New false
17859 6115 Acked 7114 Read 1 Done Done New false
17860 6116 Acked 7115 Write 10 Done Done New false
17861 6117 Acked 7116 Flush 0 Done Done New false
17862 6118 Acked 7117 Write 5 Done Done New false
17863 6119 Acked 7118 Read 7 Done Done New false
17864 6120 Acked 7119 Read 9 Done Done New false
17865 6121 Acked 7120 Read 1 Done Done New false
17866 6122 Acked 7121 Write 4 Done Done New false
17867 6123 Acked 7122 Write 1 Done Done New false
17868 6124 Acked 7123 Read 9 Done Done New false
17869 6125 Acked 7124 Write 2 Done Done New false
17870 6126 Acked 7125 Write 6 Done Done New false
17871 6127 Acked 7126 Read 6 Done Done New false
17872 6128 Acked 7127 Read 9 Done Done New false
17873 6129 Acked 7128 Write 3 Done Done New false
17874 6130 Acked 7129 Write 4 Done Done New false
17875 6131 Acked 7130 Read 1 Done Done New false
17876 6132 Acked 7131 Read 1 Done Done New false
17877 6133 Acked 7132 Write 4 Done Done New false
17878 6134 Acked 7133 Read 3 Done Done New false
17879 6135 Acked 7134 Write 9 Done Done New false
17880 6136 Acked 7135 Write 5 Done Done New false
17881 6137 Acked 7136 Read 6 Done Done New false
17882 6138 Acked 7137 Read 3 Done Done New false
17883 6139 Acked 7138 Read 2 Done Done New false
17884 6140 Acked 7139 Read 3 Done Done New false
17885 6141 Acked 7140 Flush 0 Done Done New false
17886 6142 Acked 7141 Write 7 Done Done New false
17887 6143 Acked 7142 Flush 0 Done Done New false
17888 6144 Acked 7143 Read 5 Done Done New false
17889 6145 Acked 7144 Flush 0 Done Done New false
17890 6146 Acked 7145 Write 7 Done Done New false
17891 6147 Acked 7146 Read 6 Done Done New false
17892 6148 Acked 7147 Write 8 Done Done New false
17893 6149 Acked 7148 Write 2 Done Done New false
17894 6150 Acked 7149 Write 5 Done Done New false
17895 6151 Acked 7150 Read 1 Done Done New false
17896 6152 Acked 7151 Write 10 Done Done New false
17897 6153 Acked 7152 Flush 0 Done Done New false
17898 6154 Acked 7153 Read 10 Done Done New false
17899 6155 Acked 7154 Read 1 Done Done New false
17900 6156 Acked 7155 Read 8 Done Done New false
17901 6157 Acked 7156 Read 9 Done Done New false
17902 6158 Acked 7157 Read 10 Done Done New false
17903 6159 Acked 7158 Read 5 Done Done New false
17904 6160 Acked 7159 Read 8 Done Done New false
17905 6161 Acked 7160 Write 10 Done Done New false
17906 6162 Acked 7161 Flush 0 Done Done New false
17907 6163 Acked 7162 Read 5 Done Done New false
17908 6164 Acked 7163 Write 6 Done Done New false
17909 6165 Acked 7164 Read 3 Done Done New false
17910 6166 Acked 7165 Read 8 Done Done New false
17911 6167 Acked 7166 Read 6 Done Done New false
17912 6168 Acked 7167 Read 10 Done Done New false
17913 6169 Acked 7168 Read 3 Done Done New false
17914 6170 Acked 7169 Read 10 Done Done New false
17915 6171 Acked 7170 Write 4 Done Done New false
17916 6172 Acked 7171 Read 7 Done Done New false
17917 6173 Acked 7172 Read 1 Done Done New false
17918 6174 Acked 7173 Read 6 Done Done New false
17919 6175 Acked 7174 Write 3 Done Done New false
17920 6176 Acked 7175 Read 8 Done Done New false
17921 6177 Acked 7176 Write 2 Done Done New false
17922 6178 Acked 7177 Read 1 Done Done New false
17923 6179 Acked 7178 Write 7 Done Done New false
17924 6180 Acked 7179 Write 4 Done Done New false
17925 6181 Acked 7180 Read 4 Done Done New false
17926 6182 Acked 7181 Write 3 Done Done New false
17927 6183 Acked 7182 Flush 0 Done Done New false
17928 6184 Acked 7183 Write 6 Done Done New false
17929 6185 Acked 7184 Read 5 Done Done New false
17930 6186 Acked 7185 Read 4 Done Done New false
17931 6187 Acked 7186 Read 7 Done Done New false
17932 6188 Acked 7187 Read 4 Done Done New false
17933 6189 Acked 7188 Read 9 Done Done New false
17934 6190 Acked 7189 Write 6 Done Done New false
17935 6191 Acked 7190 Read 1 Done Done New false
17936 6192 Acked 7191 Read 2 Done Done New false
17937 6193 Acked 7192 Read 1 Done Done New false
17938 6194 Acked 7193 Read 5 Done Done New false
17939 6195 Acked 7194 Read 4 Done Done New false
17940 6196 Acked 7195 Write 10 Done Done New false
17941 6197 Acked 7196 Write 5 Done Done New false
17942 6198 Acked 7197 Read 7 Done Done New false
17943 6199 Acked 7198 Write 9 Done Done New false
17944 6200 Acked 7199 Write 3 Done Done New false
17945 6201 Acked 7200 Read 7 Done Done New false
17946 6202 Acked 7201 Write 10 Done Done New false
17947 6203 Acked 7202 Write 10 Done Done New false
17948 6204 Acked 7203 Write 1 Done Done New false
17949 6205 Acked 7204 Read 9 Done Done New false
17950 6206 Acked 7205 Read 10 Done Done New false
17951 6207 Acked 7206 Write 10 Done Done New false
17952 6208 Acked 7207 Read 4 Done Done New false
17953 6209 Acked 7208 Read 8 Done Done New false
17954 6210 Acked 7209 Write 6 Done Done New false
17955 6211 Acked 7210 Write 9 Done Done New false
17956 6212 Acked 7211 Write 1 Done Done New false
17957 6213 Acked 7212 Flush 0 Done Done New false
17958 6214 Acked 7213 Read 10 Done Done New false
17959 6215 Acked 7214 Read 8 Done Done New false
17960 6216 Acked 7215 Write 1 Done Done New false
17961 6217 Acked 7216 Read 6 Done Done New false
17962 6218 Acked 7217 Read 9 Done Done New false
17963 6219 Acked 7218 Read 5 Done Done New false
17964 6220 Acked 7219 Write 6 Done Done New false
17965 6221 Acked 7220 Read 9 Done Done New false
17966 6222 Acked 7221 Write 1 Done Done New false
17967 6223 Acked 7222 Read 5 Done Done New false
17968 6224 Acked 7223 Read 6 Done Done New false
17969 6225 Acked 7224 Read 1 Done Done New false
17970 6226 Acked 7225 Flush 0 Done Done New false
17971 6227 Acked 7226 Flush 0 Done Done New false
17972 6228 Acked 7227 Read 8 Done Done New false
17973 6229 Acked 7228 Write 3 Done Done New false
17974 6230 Acked 7229 Read 3 Done Done New false
17975 6231 Acked 7230 Read 10 Done Done New false
17976 6232 Acked 7231 Write 5 Done Done New false
17977 6233 Acked 7232 Read 7 Done Done New false
17978 6234 Acked 7233 Write 7 Done Done New false
17979 6235 Acked 7234 Write 9 Done Done New false
17980 6236 Acked 7235 Flush 0 Done Done New false
17981 6237 Acked 7236 Read 2 Done Done New false
17982 6238 Acked 7237 Read 3 Done Done New false
17983 6239 Acked 7238 Read 5 Done Done New false
17984 6240 Acked 7239 Write 5 Done Done New false
17985 6241 Acked 7240 Write 9 Done Done New false
17986 6242 Acked 7241 Read 4 Done Done New false
17987 6243 Acked 7242 Read 9 Done Done New false
17988 6244 Acked 7243 Read 9 Done Done New false
17989 6245 Acked 7244 Write 8 Done Done New false
17990 6246 Acked 7245 Write 9 Done Done New false
17991 6247 Acked 7246 Read 1 Done Done New false
17992 6248 Acked 7247 Read 3 Done Done New false
17993 6249 Acked 7248 Write 3 Done Done New false
17994 6250 Acked 7249 Write 5 Done Done New false
17995 6251 Acked 7250 Read 2 Done Done New false
17996 6252 Acked 7251 Write 3 Done Done New false
17997 6253 Acked 7252 Write 6 Done Done New false
17998 6254 Acked 7253 Read 4 Done Done New false
17999 6255 Acked 7254 Write 5 Done Done New false
18000 6256 Acked 7255 Read 9 Done Done New false
18001 6257 Acked 7256 Write 10 Done Done New false
18002 6258 Acked 7257 Read 2 Done Done New false
18003 6259 Acked 7258 Write 8 Done Done New false
18004 6260 Acked 7259 Read 2 Done Done New false
18005 6261 Acked 7260 Write 2 Done Done New false
18006 6262 Acked 7261 Write 10 Done Done New false
18007 6263 Acked 7262 Read 2 Done Done New false
18008 6264 Acked 7263 Read 1 Done Done New false
18009 6265 Acked 7264 Write 4 Done Done New false
18010 6266 Acked 7265 Flush 0 Done Done New false
18011 6267 Acked 7266 Read 1 Done Done New false
18012 6268 Acked 7267 Write 1 Done Done New false
18013 6269 Acked 7268 Read 3 Done Done New false
18014 6270 Acked 7269 Read 4 Done Done New false
18015 6271 Acked 7270 Read 9 Done Done New false
18016 6272 Acked 7271 Write 4 Done Done New false
18017 6273 Acked 7272 Write 5 Done Done New false
18018 6274 Acked 7273 Read 6 Done Done New false
18019 6275 Acked 7274 Read 3 Done Done New false
18020 6276 Acked 7275 Write 6 Done Done New false
18021 6277 Acked 7276 Flush 0 Done Done New false
18022 6278 Acked 7277 Write 9 Done Done New false
18023 6279 Acked 7278 Write 4 Done Done New false
18024 6280 Acked 7279 Write 2 Done Done New false
18025 6281 Acked 7280 Read 9 Done Done New false
18026 6282 Acked 7281 Read 2 Done Done New false
18027 6283 Acked 7282 Read 1 Done Done New false
18028 6284 Acked 7283 Write 5 Done Done New false
18029 6285 Acked 7284 Read 7 Done Done New false
18030 6286 Acked 7285 Read 1 Done Done New false
18031 6287 Acked 7286 Read 5 Done Done New false
18032 6288 Acked 7287 Read 6 Done Done New false
18033 6289 Acked 7288 Read 2 Done Done New false
18034 6290 Acked 7289 Write 6 Done Done New false
18035 6291 Acked 7290 Write 10 Done Done New false
18036 6292 Acked 7291 Flush 0 Done Done New false
18037 6293 Acked 7292 Write 9 Done Done New false
18038 6294 Acked 7293 Write 7 Done Done New false
18039 6295 Acked 7294 Write 3 Done Done New false
18040 6296 Acked 7295 Read 1 Done Done New false
18041 6297 Acked 7296 Write 4 Done Done New false
18042 6298 Acked 7297 Write 2 Done Done New false
18043 6299 Acked 7298 Write 3 Done Done New false
18044 6300 Acked 7299 Read 7 Done Done New false
18045 6301 Acked 7300 Read 3 Done Done New false
18046 6302 Acked 7301 Flush 0 Done Done New false
18047 6303 Acked 7302 Write 6 Done Done New false
18048 6304 Acked 7303 Read 10 Done Done New false
18049 6305 Acked 7304 Read 7 Done Done New false
18050 6306 Acked 7305 Flush 0 Done Done New false
18051 6307 Acked 7306 Write 1 Done Done New false
18052 6308 Acked 7307 Write 2 Done Done New false
18053 6309 Acked 7308 Read 10 Done Done New false
18054 6310 Acked 7309 Read 8 Done Done New false
18055 6311 Acked 7310 Write 10 Done Done New false
18056 6312 Acked 7311 Write 8 Done Done New false
18057 6313 Acked 7312 Read 3 Done Done New false
18058 6314 Acked 7313 Read 4 Done Done New false
18059 6315 Acked 7314 Write 2 Done Done New false
18060 6316 Acked 7315 Write 7 Done Done New false
18061 6317 Acked 7316 Write 5 Done Done New false
18062 6318 Acked 7317 Write 6 Done Done New false
18063 6319 Acked 7318 Write 6 Done Done New false
18064 6320 Acked 7319 Read 4 Done Done New false
18065 6321 Acked 7320 Write 9 Done Done New false
18066 6322 Acked 7321 Read 7 Done Done New false
18067 6323 Acked 7322 Read 5 Done Done New false
18068 6324 Acked 7323 Read 3 Done Done New false
18069 6325 Acked 7324 Write 7 Done Done New false
18070 6326 Acked 7325 Write 3 Done Done New false
18071 6327 Acked 7326 Read 10 Done Done New false
18072 6328 Acked 7327 Write 7 Done Done New false
18073 6329 Acked 7328 Read 3 Done Done New false
18074 6330 Acked 7329 Read 9 Done Done New false
18075 6331 Acked 7330 Read 7 Done Done New false
18076 6332 Acked 7331 Read 3 Done Done New false
18077 6333 Acked 7332 Read 2 Done Done New false
18078 6334 Acked 7333 Read 10 Done Done New false
18079 6335 Acked 7334 Read 8 Done Done New false
18080 6336 Acked 7335 Read 8 Done Done New false
18081 6337 Acked 7336 Read 9 Done Done New false
18082 6338 Acked 7337 Read 10 Done Done New false
18083 6339 Acked 7338 Flush 0 Done Done New false
18084 6340 Acked 7339 Write 9 Done Done New false
18085 6341 Acked 7340 Write 7 Done Done New false
18086 6342 Acked 7341 Flush 0 Done Done New false
18087 6343 Acked 7342 Read 1 Done Done New false
18088 6344 Acked 7343 Write 5 Done Done New false
18089 6345 Acked 7344 Read 9 Done Done New false
18090 6346 Acked 7345 Read 3 Done Done New false
18091 6347 Acked 7346 Read 9 Done Done New false
18092 6348 Acked 7347 Write 9 Done Done New false
18093 6349 Acked 7348 Flush 0 Done Done New false
18094 STATES DS:0 DS:1 DS:2 TOTAL
18095 New 0 0 302 302
18096 Sent 0 0 0 0
18097 Done 302 302 0 604
18098 Skipped 0 0 0 0
18099 Error 0 0 0 0
18100 Last Flush: 7348 7348 7046
18101 Downstairs last five completed: 7046 7045 7044 7043 7042 7041
18102 Upstairs last five completed: 6349 6348 6347 6346 6345 6344
18103 CLIENT: Up:0 ds:302 act:2
18104 [8830] got start action
18105 [8830] I am going to restart
18106 Starting downstairs at port 8830
18107 Make output file at "/tmp/dsc/downstairs-8830.txt"
18108 [8830][2] reports Starting
18109 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1197)
18110 [8830][2] reports Running
181112023-10-13T04:45:42.635ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
181122023-10-13T04:45:42.635ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 2
181132023-10-13T04:45:42.635ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
181142023-10-13T04:45:42.635ZINFOcrucible: [2] client re-new 302 jobs since flush 7046 = downstairs
181152023-10-13T04:45:42.636ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
181162023-10-13T04:45:42.636ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
181172023-10-13T04:45:42.637ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
181182023-10-13T04:45:42.637ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
181192023-10-13T04:45:42.637ZINFOcrucible: [2] send last flush ID to this DS: 7046
181202023-10-13T04:45:42.637ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
181212023-10-13T04:45:42.637ZINFOcrucible: [2] Replied this last flush ID: 7046
181222023-10-13T04:45:42.637ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Offline ds_transition to Replay
181232023-10-13T04:45:42.637ZINFOcrucible: [2] Transition from Offline to Replay
181242023-10-13T04:45:42.637ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
181252023-10-13T04:45:42.637ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Replay ds_transition to Active
181262023-10-13T04:45:42.637ZINFOcrucible: [2] Transition from Replay to Active
181272023-10-13T04:45:42.637ZINFOcrucible: [2] Starts cmd_loop
181282023-10-13T04:45:42.637ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
181292023-10-13T04:45:42.637ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
181302023-10-13T04:45:42.637ZINFOcrucible: Set check for repair
181312023-10-13T04:45:43.637ZWARNcrucible: [2] flow control end
181322023-10-13T04:45:43.638ZINFOcrucible: Checking if live repair is needed
181332023-10-13T04:45:43.638ZINFOcrucible: No Live Repair required at this time
18134 ----------------------------------------------------------------
18135 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
18136 Downstairs last five completed: 7348 7347 7346 7345 7344 7343
18137 Upstairs last five completed: 6349 6348 6347 6346 6345 6344
18138 CLIENT: Up:0 ds:0 act:3
18139 Replay: All jobs finished, all DS active.
18140 Main task has work to do, go find it
18141 got dsc Stop(0)
18142 stop 0
18143 Oct 13 04:45:44.517 INFO request completed, latency_us: 493, response_code: 204, uri: /stop/cid/0, method: POST, req_id: 7f2471af-8a56-4e90-9839-7fe7eee1915c, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
18144 [8810] Got stop action so:false kr:false
18145 Oct 13 04:45:44.518 INFO request completed, latency_us: 265, response_code: 200, uri: /state/cid/0, method: GET, req_id: b7197fe0-12f4-4732-9b80-9222c309d5cf, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
181462023-10-13T04:45:44.519ZWARNcrucible: [0] downstairs disconnected
181472023-10-13T04:45:44.519ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
181482023-10-13T04:45:44.519ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8810 closed looper = 0
181492023-10-13T04:45:44.519ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:false
181502023-10-13T04:45:44.519ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Offline Active Active
181512023-10-13T04:45:44.519ZINFOcrucible: [0] 127.0.0.1:8810 task reports offline
181522023-10-13T04:45:44.519ZWARNcrucible: [0] pm_task rx.recv() is None
18153 [8810] Exited with: ExitStatus(unix_wait_status(9))
18154 [8810][0] reports Exit
18155 Oct 13 04:45:48.520 INFO request completed, latency_us: 257, response_code: 200, uri: /state/cid/0, method: GET, req_id: 41cedc17-3933-4f9d-b698-dbb721d767b6, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
18156 001/300 Flush
18157 002/300 Write block 3477 len 32768 data: 4 4 4 4 5 4 5 5
18158 003/300 Read block 2430 len 36864
18159 004/300 Read block 4482 len 4096
18160 005/300 Read block 3656 len 40960
18161 006/300 Write block 4625 len 40960 data: 8 7 5 4 3 4 4 4 4 5
18162 007/300 Write block 311 len 4096 data: 7
18163 008/300 Read block 4785 len 28672
18164 009/300 Read block 4159 len 20480
18165 010/300 Write block 3050 len 12288 data: 5 5 5
18166 011/300 Read block 4462 len 36864
18167 012/300 Read block 1509 len 16384
18168 013/300 Flush
18169 014/300 Flush
18170 015/300 Write block 3127 len 40960 data: 5 4 4 4 4 4 4 4 4 5
18171 016/300 Write block 1201 len 28672 data: 3 3 3 4 4 4 4
18172 017/300 Read block 4421 len 20480
18173 018/300 Read block 1103 len 12288
18174 019/300 Flush
18175 020/300 Read block 753 len 36864
18176 021/300 Read block 3555 len 20480
18177 022/300 Write block 2529 len 8192 data: 5 5
18178 023/300 Read block 4357 len 28672
18179 024/300 Read block 4334 len 32768
18180 025/300 Write block 1288 len 16384 data: 10 11 10 9
18181 026/300 Write block 1624 len 16384 data: 4 4 4 3
18182 027/300 Write block 4592 len 16384 data: 4 4 3 3
18183 028/300 Write block 4934 len 32768 data: 4 3 3 2 3 5 8 8
18184 029/300 Read block 4262 len 24576
18185 030/300 Write block 3402 len 28672 data: 3 3 4 5 6 6 6
18186 031/300 Read block 452 len 8192
18187 032/300 Read block 2823 len 36864
18188 033/300 Write block 575 len 24576 data: 6 5 5 5 5 5
18189 034/300 Read block 4683 len 20480
18190 035/300 Read block 1533 len 16384
18191 036/300 Write block 3520 len 32768 data: 4 5 6 6 8 7 7 9
18192 037/300 Read block 1512 len 36864
18193 038/300 Write block 1629 len 36864 data: 6 5 5 5 5 4 4 4 5
18194 039/300 Read block 2032 len 4096
18195 040/300 Read block 3935 len 28672
18196 041/300 Write block 1845 len 8192 data: 3 4
18197 042/300 Read block 2354 len 12288
18198 043/300 Read block 2750 len 24576
18199 044/300 Write block 4550 len 32768 data: 5 6 5 4 3 3 3 4
18200 045/300 Write block 2388 len 20480 data: 6 4 4 4 4
18201 046/300 Read block 2184 len 32768
18202 047/300 Read block 4824 len 20480
18203 048/300 Write block 1128 len 32768 data: 8 8 6 5 6 5 4 4
18204 049/300 Read block 2116 len 24576
18205 050/300 Flush
18206 051/300 Read block 3049 len 4096
18207 052/300 Read block 3527 len 24576
18208 053/300 Write block 2259 len 20480 data: 3 4 4 4 4
18209 054/300 Write block 2948 len 24576 data: 4 4 6 5 6 4
18210 055/300 Write block 33 len 16384 data: 6 6 6 4
18211 056/300 Write block 1312 len 28672 data: 5 5 4 5 5 3 3
18212 057/300 Write block 2017 len 4096 data: 5
18213 058/300 Read block 911 len 32768
18214 059/300 Write block 1019 len 32768 data: 3 3 4 4 5 5 6 5
18215 060/300 Write block 2318 len 4096 data: 3
18216 061/300 Flush
18217 062/300 Write block 1881 len 4096 data: 2
18218 063/300 Read block 2904 len 12288
18219 064/300 Read block 3910 len 20480
18220 065/300 Write block 4512 len 40960 data: 2 2 2 2 2 2 3 4 5 5
18221 066/300 Read block 575 len 4096
18222 067/300 Write block 2036 len 4096 data: 5
18223 068/300 Read block 4918 len 16384
18224 069/300 Read block 2810 len 16384
18225 070/300 Read block 93 len 24576
18226 071/300 Write block 1003 len 24576 data: 4 5 6 6 5 3
18227 072/300 Write block 4932 len 40960 data: 3 4 5 4 4 3 4 6 9 9
18228 073/300 Read block 4142 len 32768
18229 074/300 Read block 16 len 28672
18230 075/300 Write block 900 len 12288 data: 8 8 8
18231 076/300 Flush
18232 077/300 Read block 3506 len 32768
18233 078/300 Read block 3222 len 12288
18234 079/300 Write block 3533 len 28672 data: 9 8 8 7 5 5 3
18235 080/300 Read block 1130 len 32768
18236 081/300 Write block 230 len 4096 data: 3
18237 082/300 Read block 1538 len 24576
18238 083/300 Write block 1729 len 16384 data: 4 3 2 2
18239 084/300 Flush
18240 085/300 Write block 2157 len 24576 data: 4 4 4 7 8 9
18241 086/300 Flush
18242 087/300 Read block 3658 len 36864
18243 088/300 Write block 1300 len 12288 data: 3 3 3
18244 089/300 Flush
18245 090/300 Write block 2922 len 40960 data: 7 7 7 7 7 7 7 6 6 5
18246 091/300 Read block 1507 len 40960
18247 092/300 Write block 2851 len 36864 data: 7 8 8 6 5 6 5 4 5
18248 093/300 Read block 1258 len 8192
18249 094/300 Write block 4161 len 12288 data: 5 6 7
18250 095/300 Read block 4535 len 36864
18251 096/300 Read block 1333 len 36864
18252 097/300 Read block 3652 len 32768
18253 098/300 Flush
18254 099/300 Write block 1656 len 12288 data: 3 4 5
18255 100/300 Read block 2539 len 20480
18256 101/300 Write block 3673 len 36864 data: 7 5 5 5 4 4 3 2 2
18257 102/300 Read block 725 len 16384
18258 103/300 Write block 93 len 12288 data: 5 4 5
18259 104/300 Read block 2062 len 20480
18260 105/300 Write block 3355 len 20480 data: 6 8 9 8 6
18261 106/300 Write block 4467 len 40960 data: 3 4 4 4 2 2 2 2 2 2
18262 107/300 Read block 4230 len 4096
18263 108/300 Read block 2620 len 40960
18264 109/300 Write block 2455 len 16384 data: 3 3 3 3
18265 110/300 Write block 2916 len 8192 data: 3 3
18266 111/300 Write block 1262 len 28672 data: 4 4 4 3 3 3 4
18267 112/300 Read block 2231 len 4096
18268 113/300 Read block 3494 len 40960
18269 114/300 Write block 1184 len 32768 data: 5 6 5 5 5 4 4 4
18270 115/300 Write block 1017 len 12288 data: 6 5 4
18271 116/300 Read block 2878 len 40960
18272 117/300 Write block 1532 len 16384 data: 6 6 5 6
18273 118/300 Write block 360 len 16384 data: 3 4 4 3
18274 119/300 Read block 2416 len 32768
18275 120/300 Write block 597 len 16384 data: 6 6 4 3
18276 121/300 Read block 3230 len 28672
18277 122/300 Read block 1486 len 32768
18278 123/300 Write block 2618 len 12288 data: 4 4 4
18279 124/300 Flush
18280 125/300 Read block 3186 len 4096
18281 126/300 Read block 3187 len 40960
18282 127/300 Flush
18283 128/300 Read block 4197 len 12288
18284 129/300 Write block 1200 len 24576 data: 3 4 4 4 5 5
18285 130/300 Read block 1166 len 36864
18286 131/300 Read block 3852 len 32768
18287 132/300 Write block 4613 len 28672 data: 3 4 5 4 5 4 4
18288 133/300 Read block 2358 len 28672
18289 134/300 Read block 4025 len 4096
18290 135/300 Write block 707 len 8192 data: 5 6
18291 136/300 Write block 805 len 8192 data: 4 2
18292 137/300 Read block 4369 len 40960
18293 138/300 Write block 3387 len 16384 data: 6 7 6 6
18294 139/300 Write block 4324 len 40960 data: 3 2 2 2 2 2 2 2 2 2
18295 140/300 Read block 422 len 32768
18296 141/300 Write block 724 len 4096 data: 7
18297 142/300 Read block 501 len 12288
18298 143/300 Write block 3476 len 28672 data: 3 5 5 5 5 6 5
18299 144/300 Write block 532 len 16384 data: 3 3 5 7
18300 145/300 Flush
18301 146/300 Write block 4659 len 24576 data: 6 7 5 6 6 6
18302 147/300 Write block 4437 len 8192 data: 6 6
18303 148/300 Write block 1804 len 16384 data: 5 4 3 4
18304 149/300 Write block 1855 len 36864 data: 6 7 8 8 8 8 7 5 7
18305 150/300 Read block 3143 len 20480
18306 151/300 Read block 687 len 36864
18307 152/300 Read block 491 len 4096
18308 153/300 Write block 2346 len 4096 data: 5
18309 154/300 Write block 2085 len 32768 data: 4 5 6 6 6 5 9 8
18310 155/300 Read block 1781 len 8192
18311 156/300 Read block 2326 len 20480
18312 157/300 Write block 4794 len 36864 data: 5 4 3 3 3 3 4 5 5
18313 158/300 Write block 2754 len 36864 data: 6 4 4 5 5 3 3 4 5
18314 159/300 Flush
18315 160/300 Read block 4162 len 32768
18316 161/300 Write block 593 len 12288 data: 5 6 6
18317 162/300 Write block 3220 len 12288 data: 6 6 5
18318 163/300 Read block 4224 len 32768
18319 164/300 Read block 3638 len 36864
18320 165/300 Write block 872 len 36864 data: 6 8 8 6 6 5 5 5 4
18321 166/300 Read block 3144 len 28672
18322 167/300 Write block 2107 len 24576 data: 5 5 4 6 4 5
18323 168/300 Read block 3110 len 36864
18324 169/300 Read block 216 len 40960
18325 170/300 Read block 4332 len 12288
18326 171/300 Flush
18327 172/300 Write block 3516 len 4096 data: 5
18328 173/300 Read block 2014 len 40960
18329 174/300 Flush
18330 175/300 Read block 2436 len 12288
18331 176/300 Read block 2497 len 36864
18332 177/300 Read block 386 len 8192
18333 178/300 Read block 1832 len 32768
18334 179/300 Read block 1619 len 12288
18335 180/300 Read block 3118 len 32768
18336 181/300 Flush
18337 182/300 Read block 776 len 12288
18338 183/300 Write block 843 len 40960 data: 8 7 7 6 7 7 6 5 7 7
18339 184/300 Read block 3969 len 32768
18340 185/300 Flush
18341 186/300 Read block 2257 len 4096
18342 187/300 Write block 3318 len 24576 data: 6 7 6 7 7 8
18343 188/300 Read block 2792 len 12288
18344 189/300 Write block 4348 len 12288 data: 9 9 8
18345 190/300 Flush
18346 191/300 Read block 324 len 36864
18347 192/300 Write block 2118 len 12288 data: 6 7 7
18348 193/300 Read block 665 len 12288
18349 194/300 Read block 4037 len 32768
18350 195/300 Read block 879 len 4096
18351 196/300 Write block 4719 len 4096 data: 4
18352 197/300 Flush
18353 198/300 Write block 3155 len 24576 data: 4 4 4 5 5 5
18354 199/300 Read block 544 len 12288
18355 200/300 Read block 1356 len 16384
18356 201/300 Write block 2601 len 4096 data: 5
18357 202/300 Write block 2120 len 36864 data: 8 5 5 5 6 7 7 8 7
18358 203/300 Flush
18359 204/300 Read block 3502 len 28672
18360 205/300 Write block 1557 len 20480 data: 5 6 7 7 4
18361 206/300 Write block 1082 len 32768 data: 10 7 7 8 9 7 8 8
18362 207/300 Flush
18363 208/300 Write block 3776 len 28672 data: 3 4 5 6 7 7 7
18364 209/300 Read block 1321 len 16384
18365 210/300 Read block 1332 len 24576
18366 211/300 Flush
18367 212/300 Read block 2329 len 4096
18368 213/300 Write block 910 len 20480 data: 5 5 6 8 8
18369 214/300 Read block 2262 len 16384
18370 215/300 Read block 4059 len 32768
18371 216/300 Write block 532 len 4096 data: 4
18372 217/300 Flush
18373 218/300 Read block 2452 len 12288
18374 219/300 Flush
18375 220/300 Flush
18376 221/300 Write block 4455 len 4096 data: 8
18377 222/300 Write block 3924 len 16384 data: 4 5 5 5
18378 223/300 Flush
18379 224/300 Read block 3894 len 8192
18380 225/300 Read block 1684 len 16384
18381 226/300 Read block 140 len 8192
18382 227/300 Write block 1266 len 24576 data: 4 4 5 5 4 3
18383 228/300 Read block 2204 len 20480
18384 229/300 Read block 4314 len 8192
18385 230/300 Read block 1851 len 20480
18386 231/300 Write block 499 len 36864 data: 5 4 3 3 3 3 3 4 5
18387 232/300 Read block 4193 len 20480
18388 233/300 Read block 1966 len 16384
18389 234/300 Write block 186 len 4096 data: 4
18390 235/300 Read block 3374 len 4096
18391 236/300 Read block 4819 len 12288
18392 237/300 Write block 4743 len 32768 data: 4 4 5 5 5 5 5 5
18393 238/300 Read block 800 len 28672
18394 239/300 Read block 3757 len 12288
18395 240/300 Read block 3812 len 24576
18396 241/300 Read block 2720 len 8192
18397 242/300 Write block 4387 len 36864 data: 5 6 6 7 6 6 6 4 5
18398 243/300 Read block 2796 len 24576
18399 244/300 Write block 2583 len 28672 data: 4 4 4 4 3 3 3
18400 245/300 Write block 3151 len 8192 data: 5 5
18401 246/300 Read block 3869 len 16384
18402 247/300 Read block 3089 len 28672
18403 248/300 Read block 171 len 16384
18404 249/300 Write block 639 len 28672 data: 5 4 5 6 5 5 5
18405 250/300 Write block 4024 len 36864 data: 8 7 6 7 5 5 3 3 4
18406 251/300 Read block 4766 len 40960
18407 252/300 Flush
18408 253/300 Read block 3052 len 20480
18409 254/300 Read block 1499 len 20480
18410 255/300 Flush
18411 256/300 Write block 554 len 16384 data: 4 4 5 6
18412 257/300 Read block 1119 len 36864
18413 258/300 Write block 2440 len 36864 data: 2 2 2 3 3 3 4 3 3
18414 259/300 Read block 3113 len 32768
18415 260/300 Write block 343 len 28672 data: 3 3 3 3 3 4 4
18416 261/300 Read block 3533 len 24576
18417 262/300 Flush
18418 263/300 Write block 2864 len 12288 data: 7 6 6
18419 264/300 Flush
18420 265/300 Read block 129 len 12288
18421 266/300 Read block 1799 len 20480
18422 267/300 Flush
18423 268/300 Read block 2709 len 4096
18424 269/300 Read block 2873 len 28672
18425 270/300 Write block 866 len 20480 data: 6 6 6 4 3
18426 271/300 Read block 2267 len 24576
18427 272/300 Write block 4859 len 8192 data: 3 6
18428 273/300 Read block 1031 len 24576
18429 274/300 Read block 3484 len 16384
18430 275/300 Write block 2246 len 8192 data: 6 6
18431 276/300 Read block 303 len 40960
18432 277/300 Read block 992 len 12288
18433 278/300 Write block 1578 len 28672 data: 3 2 3 3 3 5 7
18434 279/300 Read block 547 len 12288
18435 280/300 Write block 260 len 20480 data: 4 4 3 2 3
18436 281/300 Write block 4154 len 32768 data: 7 7 6 6 6 6 7 6
18437 282/300 Read block 714 len 24576
18438 283/300 Write block 2306 len 28672 data: 2 2 3 3 3 3 2
18439 284/300 Flush
18440 285/300 Flush
18441 286/300 Read block 1387 len 40960
18442 287/300 Read block 1991 len 8192
18443 288/300 Read block 46 len 28672
18444 289/300 Write block 2821 len 8192 data: 3 3
18445 290/300 Read block 4905 len 32768
18446 291/300 Read block 2050 len 24576
18447 292/300 Read block 4625 len 4096
18448 293/300 Write block 2873 len 32768 data: 3 3 4 4 4 4 5 4
18449 294/300 Read block 1474 len 32768
18450 295/300 Read block 4186 len 24576
18451 296/300 Write block 974 len 40960 data: 7 8 8 7 5 5 4 5 7 6
18452 297/300 Read block 1237 len 28672
18453 298/300 Read block 1868 len 28672
18454 299/300 Write block 270 len 40960 data: 5 4 3 3 2 3 4 3 3 3
18455 300/300 Read block 1574 len 8192
18456 Main task has work to do, go find it
18457 got dsc Start(0)
18458 start 0
18459 Oct 13 04:45:49.408 INFO request completed, latency_us: 293, response_code: 204, uri: /start/cid/0, method: POST, req_id: 47856267-8e94-485b-9a71-9bcdfb11f96b, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
18460 Replay: started 0, returned:Ok(())
18461 ----------------------------------------------------------------
18462 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
18463 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
18464 6350 Acked 7349 Flush 0 New Done Done false
18465 6351 Acked 7350 Write 8 New Done Done false
18466 6352 Acked 7351 Read 9 New Done Done false
18467 6353 Acked 7352 Read 1 New Done Done false
18468 6354 Acked 7353 Read 10 New Done Done false
18469 6355 Acked 7354 Write 10 New Done Done false
18470 6356 Acked 7355 Write 1 New Done Done false
18471 6357 Acked 7356 Read 7 New Done Done false
18472 6358 Acked 7357 Read 5 New Done Done false
18473 6359 Acked 7358 Write 3 New Done Done false
18474 6360 Acked 7359 Read 9 New Done Done false
18475 6361 Acked 7360 Read 4 New Done Done false
18476 6362 Acked 7361 Flush 0 New Done Done false
18477 6363 Acked 7362 Flush 0 New Done Done false
18478 6364 Acked 7363 Write 10 New Done Done false
18479 6365 Acked 7364 Write 7 New Done Done false
18480 6366 Acked 7365 Read 5 New Done Done false
18481 6367 Acked 7366 Read 3 New Done Done false
18482 6368 Acked 7367 Flush 0 New Done Done false
18483 6369 Acked 7368 Read 9 New Done Done false
18484 6370 Acked 7369 Read 5 New Done Done false
18485 6371 Acked 7370 Write 2 New Done Done false
18486 6372 Acked 7371 Read 7 New Done Done false
18487 6373 Acked 7372 Read 8 New Done Done false
18488 6374 Acked 7373 Write 4 New Done Done false
18489 6375 Acked 7374 Write 4 New Done Done false
18490 6376 Acked 7375 Write 4 New Done Done false
18491 6377 Acked 7376 Write 8 New Done Done false
18492 6378 Acked 7377 Read 6 New Done Done false
18493 6379 Acked 7378 Write 7 New Done Done false
18494 6380 Acked 7379 Read 2 New Done Done false
18495 6381 Acked 7380 Read 9 New Done Done false
18496 6382 Acked 7381 Write 6 New Done Done false
18497 6383 Acked 7382 Read 5 New Done Done false
18498 6384 Acked 7383 Read 4 New Done Done false
18499 6385 Acked 7384 Write 8 New Done Done false
18500 6386 Acked 7385 Read 9 New Done Done false
18501 6387 Acked 7386 Write 9 New Done Done false
18502 6388 Acked 7387 Read 1 New Done Done false
18503 6389 Acked 7388 Read 7 New Done Done false
18504 6390 Acked 7389 Write 2 New Done Done false
18505 6391 Acked 7390 Read 3 New Done Done false
18506 6392 Acked 7391 Read 6 New Done Done false
18507 6393 Acked 7392 Write 8 New Done Done false
18508 6394 Acked 7393 Write 5 New Done Done false
18509 6395 Acked 7394 Read 8 New Done Done false
18510 6396 Acked 7395 Read 5 New Done Done false
18511 6397 Acked 7396 Write 8 New Done Done false
18512 6398 Acked 7397 Read 6 New Done Done false
18513 6399 Acked 7398 Flush 0 New Done Done false
18514 6400 Acked 7399 Read 1 New Done Done false
18515 6401 Acked 7400 Read 6 New Done Done false
18516 6402 Acked 7401 Write 5 New Done Done false
18517 6403 Acked 7402 Write 6 New Done Done false
18518 6404 Acked 7403 Write 4 New Done Done false
18519 6405 Acked 7404 Write 7 New Done Done false
18520 6406 Acked 7405 Write 1 New Done Done false
18521 6407 Acked 7406 Read 8 New Done Done false
18522 6408 Acked 7407 Write 8 New Done Done false
18523 6409 Acked 7408 Write 1 New Done Done false
18524 6410 Acked 7409 Flush 0 New Done Done false
18525 6411 Acked 7410 Write 1 New Done Done false
18526 6412 Acked 7411 Read 3 New Done Done false
18527 6413 Acked 7412 Read 5 New Done Done false
18528 6414 Acked 7413 Write 10 New Done Done false
18529 6415 Acked 7414 Read 1 New Done Done false
18530 6416 Acked 7415 Write 1 New Done Done false
18531 6417 Acked 7416 Read 4 New Done Done false
18532 6418 Acked 7417 Read 4 New Done Done false
18533 6419 Acked 7418 Read 6 New Done Done false
18534 6420 Acked 7419 Write 6 New Done Done false
18535 6421 Acked 7420 Write 10 New Done Done false
18536 6422 Acked 7421 Read 8 New Done Done false
18537 6423 Acked 7422 Read 7 New Done Done false
18538 6424 Acked 7423 Write 3 New Done Done false
18539 6425 Acked 7424 Flush 0 New Done Done false
18540 6426 Acked 7425 Read 8 New Done Done false
18541 6427 Acked 7426 Read 3 New Done Done false
18542 6428 Acked 7427 Write 7 New Done Done false
18543 6429 Acked 7428 Read 8 New Done Done false
18544 6430 Acked 7429 Write 1 New Done Done false
18545 6431 Acked 7430 Read 6 New Done Done false
18546 6432 Acked 7431 Write 4 New Done Done false
18547 6433 Acked 7432 Flush 0 New Done Done false
18548 6434 Acked 7433 Write 6 New Done Done false
18549 6435 Acked 7434 Flush 0 New Done Done false
18550 6436 Acked 7435 Read 9 New Done Done false
18551 6437 Acked 7436 Write 3 New Done Done false
18552 6438 Acked 7437 Flush 0 New Done Done false
18553 6439 Acked 7438 Write 10 New Done Done false
18554 6440 Acked 7439 Read 10 New Done Done false
18555 6441 Acked 7440 Write 9 New Done Done false
18556 6442 Acked 7441 Read 2 New Done Done false
18557 6443 Acked 7442 Write 3 New Done Done false
18558 6444 Acked 7443 Read 9 New Done Done false
18559 6445 Acked 7444 Read 9 New Done Done false
18560 6446 Acked 7445 Read 8 New Done Done false
18561 6447 Acked 7446 Flush 0 New Done Done false
18562 6448 Acked 7447 Write 3 New Done Done false
18563 6449 Acked 7448 Read 5 New Done Done false
18564 6450 Acked 7449 Write 9 New Done Done false
18565 6451 Acked 7450 Read 4 New Done Done false
18566 6452 Acked 7451 Write 3 New Done Done false
18567 6453 Acked 7452 Read 5 New Done Done false
18568 6454 Acked 7453 Write 5 New Done Done false
18569 6455 Acked 7454 Write 10 New Done Done false
18570 6456 Acked 7455 Read 1 New Done Done false
18571 6457 Acked 7456 Read 10 New Done Done false
18572 6458 Acked 7457 Write 4 New Done Done false
18573 6459 Acked 7458 Write 2 New Done Done false
18574 6460 Acked 7459 Write 7 New Done Done false
18575 6461 Acked 7460 Read 1 New Done Done false
18576 6462 Acked 7461 Read 10 New Done Done false
18577 6463 Acked 7462 Write 8 New Done Done false
18578 6464 Acked 7463 Write 3 New Done Done false
18579 6465 Acked 7464 Read 10 New Done Done false
18580 6466 Acked 7465 Write 4 New Done Done false
18581 6467 Acked 7466 Write 4 New Done Done false
18582 6468 Acked 7467 Read 8 New Done Done false
18583 6469 Acked 7468 Write 4 New Done Done false
18584 6470 Acked 7469 Read 7 New Done Done false
18585 6471 Acked 7470 Read 8 New Done Done false
18586 6472 Acked 7471 Write 3 New Done Done false
18587 6473 Acked 7472 Flush 0 New Done Done false
18588 6474 Acked 7473 Read 1 New Done Done false
18589 6475 Acked 7474 Read 10 New Done Done false
18590 6476 Acked 7475 Flush 0 New Done Done false
18591 6477 Acked 7476 Read 3 New Done Done false
18592 6478 Acked 7477 Write 6 New Done Done false
18593 6479 Acked 7478 Read 9 New Done Done false
18594 6480 Acked 7479 Read 8 New Done Done false
18595 6481 Acked 7480 Write 7 New Done Done false
18596 6482 Acked 7481 Read 7 New Done Done false
18597 6483 Acked 7482 Read 1 New Done Done false
18598 6484 Acked 7483 Write 2 New Done Done false
18599 6485 Acked 7484 Write 2 New Done Done false
18600 6486 Acked 7485 Read 10 New Done Done false
18601 6487 Acked 7486 Write 4 New Done Done false
18602 6488 Acked 7487 Write 10 New Done Done false
18603 6489 Acked 7488 Read 8 New Done Done false
18604 6490 Acked 7489 Write 1 New Done Done false
18605 6491 Acked 7490 Read 3 New Done Done false
18606 6492 Acked 7491 Write 7 New Done Done false
18607 6493 Acked 7492 Write 4 New Done Done false
18608 6494 Acked 7493 Flush 0 New Done Done false
18609 6495 Acked 7494 Write 6 New Done Done false
18610 6496 Acked 7495 Write 2 New Done Done false
18611 6497 Acked 7496 Write 4 New Done Done false
18612 6498 Acked 7497 Write 9 New Done Done false
18613 6499 Acked 7498 Read 5 New Done Done false
18614 6500 Acked 7499 Read 9 New Done Done false
18615 6501 Acked 7500 Read 1 New Done Done false
18616 6502 Acked 7501 Write 1 New Done Done false
18617 6503 Acked 7502 Write 8 New Done Done false
18618 6504 Acked 7503 Read 2 New Done Done false
18619 6505 Acked 7504 Read 5 New Done Done false
18620 6506 Acked 7505 Write 9 New Done Done false
18621 6507 Acked 7506 Write 9 New Done Done false
18622 6508 Acked 7507 Flush 0 New Done Done false
18623 6509 Acked 7508 Read 8 New Done Done false
18624 6510 Acked 7509 Write 3 New Done Done false
18625 6511 Acked 7510 Write 3 New Done Done false
18626 6512 Acked 7511 Read 8 New Done Done false
18627 6513 Acked 7512 Read 9 New Done Done false
18628 6514 Acked 7513 Write 9 New Done Done false
18629 6515 Acked 7514 Read 7 New Done Done false
18630 6516 Acked 7515 Write 6 New Done Done false
18631 6517 Acked 7516 Read 9 New Done Done false
18632 6518 Acked 7517 Read 10 New Done Done false
18633 6519 Acked 7518 Read 3 New Done Done false
18634 6520 Acked 7519 Flush 0 New Done Done false
18635 6521 Acked 7520 Write 1 New Done Done false
18636 6522 Acked 7521 Read 10 New Done Done false
18637 6523 Acked 7522 Flush 0 New Done Done false
18638 6524 Acked 7523 Read 3 New Done Done false
18639 6525 Acked 7524 Read 9 New Done Done false
18640 6526 Acked 7525 Read 2 New Done Done false
18641 6527 Acked 7526 Read 8 New Done Done false
18642 6528 Acked 7527 Read 3 New Done Done false
18643 6529 Acked 7528 Read 8 New Done Done false
18644 6530 Acked 7529 Flush 0 New Done Done false
18645 6531 Acked 7530 Read 3 New Done Done false
18646 6532 Acked 7531 Write 10 New Done Done false
18647 6533 Acked 7532 Read 8 New Done Done false
18648 6534 Acked 7533 Flush 0 New Done Done false
18649 6535 Acked 7534 Read 1 New Done Done false
18650 6536 Acked 7535 Write 6 New Done Done false
18651 6537 Acked 7536 Read 3 New Done Done false
18652 6538 Acked 7537 Write 3 New Done Done false
18653 6539 Acked 7538 Flush 0 New Done Done false
18654 6540 Acked 7539 Read 9 New Done Done false
18655 6541 Acked 7540 Write 3 New Done Done false
18656 6542 Acked 7541 Read 3 New Done Done false
18657 6543 Acked 7542 Read 8 New Done Done false
18658 6544 Acked 7543 Read 1 New Done Done false
18659 6545 Acked 7544 Write 1 New Done Done false
18660 6546 Acked 7545 Flush 0 New Done Done false
18661 6547 Acked 7546 Write 6 New Done Done false
18662 6548 Acked 7547 Read 3 New Done Done false
18663 6549 Acked 7548 Read 4 New Done Done false
18664 6550 Acked 7549 Write 1 New Done Done false
18665 6551 Acked 7550 Write 9 New Done Done false
18666 6552 Acked 7551 Flush 0 New Done Done false
18667 6553 Acked 7552 Read 7 New Done Done false
18668 6554 Acked 7553 Write 5 New Done Done false
18669 6555 Acked 7554 Write 8 New Done Done false
18670 6556 Acked 7555 Flush 0 New Done Done false
18671 6557 Acked 7556 Write 7 New Done Done false
18672 6558 Acked 7557 Read 4 New Done Done false
18673 6559 Acked 7558 Read 6 New Done Done false
18674 6560 Acked 7559 Flush 0 New Done Done false
18675 6561 Acked 7560 Read 1 New Done Done false
18676 6562 Acked 7561 Write 5 New Done Done false
18677 6563 Acked 7562 Read 4 New Done Done false
18678 6564 Acked 7563 Read 8 New Done Done false
18679 6565 Acked 7564 Write 1 New Done Done false
18680 6566 Acked 7565 Flush 0 New Done Done false
18681 6567 Acked 7566 Read 3 New Done Done false
18682 6568 Acked 7567 Flush 0 New Done Done false
18683 6569 Acked 7568 Flush 0 New Done Done false
18684 6570 Acked 7569 Write 1 New Done Done false
18685 6571 Acked 7570 Write 4 New Done Done false
18686 6572 Acked 7571 Flush 0 New Done Done false
18687 6573 Acked 7572 Read 2 New Done Done false
18688 6574 Acked 7573 Read 4 New Done Done false
18689 6575 Acked 7574 Read 2 New Done Done false
18690 6576 Acked 7575 Write 6 New Done Done false
18691 6577 Acked 7576 Read 5 New Done Done false
18692 6578 Acked 7577 Read 2 New Done Done false
18693 6579 Acked 7578 Read 5 New Done Done false
18694 6580 Acked 7579 Write 9 New Done Done false
18695 6581 Acked 7580 Read 5 New Done Done false
18696 6582 Acked 7581 Read 4 New Done Done false
18697 6583 Acked 7582 Write 1 New Done Done false
18698 6584 Acked 7583 Read 1 New Done Done false
18699 6585 Acked 7584 Read 3 New Done Done false
18700 6586 Acked 7585 Write 8 New Done Done false
18701 6587 Acked 7586 Read 7 New Done Done false
18702 6588 Acked 7587 Read 3 New Done Done false
18703 6589 Acked 7588 Read 6 New Done Done false
18704 6590 Acked 7589 Read 2 New Done Done false
18705 6591 Acked 7590 Write 9 New Done Done false
18706 6592 Acked 7591 Read 6 New Done Done false
18707 6593 Acked 7592 Write 7 New Done Done false
18708 6594 Acked 7593 Write 2 New Done Done false
18709 6595 Acked 7594 Read 4 New Done Done false
18710 6596 Acked 7595 Read 7 New Done Done false
18711 6597 Acked 7596 Read 4 New Done Done false
18712 6598 Acked 7597 Write 7 New Done Done false
18713 6599 Acked 7598 Write 9 New Done Done false
18714 6600 Acked 7599 Read 10 New Done Done false
18715 6601 Acked 7600 Flush 0 New Done Done false
18716 6602 Acked 7601 Read 5 New Done Done false
18717 6603 Acked 7602 Read 5 New Done Done false
18718 6604 Acked 7603 Flush 0 New Done Done false
18719 6605 Acked 7604 Write 4 New Done Done false
18720 6606 Acked 7605 Read 9 New Done Done false
18721 6607 Acked 7606 Write 9 New Done Done false
18722 6608 Acked 7607 Read 8 New Done Done false
18723 6609 Acked 7608 Flush 0 New Done Done false
18724 6610 Acked 7609 Write 7 New Done Done false
18725 6611 Acked 7610 Read 6 New Done Done false
18726 6612 Acked 7611 Flush 0 New Done Done false
18727 6613 Acked 7612 Write 3 New Done Done false
18728 6614 Acked 7613 Flush 0 New Done Done false
18729 6615 Acked 7614 Read 3 New Done Done false
18730 6616 Acked 7615 Read 5 New Done Done false
18731 6617 Acked 7616 Flush 0 New Done Done false
18732 6618 Acked 7617 Read 1 New Done Done false
18733 6619 Acked 7618 Read 7 New Done Done false
18734 6620 Acked 7619 Write 5 New Done Done false
18735 6621 Acked 7620 Read 6 New Done Done false
18736 6622 Acked 7621 Write 2 New Done Done false
18737 6623 Acked 7622 Read 6 New Done Done false
18738 6624 Acked 7623 Read 4 New Done Done false
18739 6625 Acked 7624 Write 2 New Done Done false
18740 6626 Acked 7625 Read 10 New Done Done false
18741 6627 Acked 7626 Read 3 New Done Done false
18742 6628 Acked 7627 Write 7 New Done Done false
18743 6629 Acked 7628 Read 3 New Done Done false
18744 6630 Acked 7629 Write 5 New Done Done false
18745 6631 Acked 7630 Write 8 New Done Done false
18746 6632 Acked 7631 Read 6 New Done Done false
18747 6633 Acked 7632 Write 7 New Done Done false
18748 6634 Acked 7633 Flush 0 New Done Done false
18749 6635 Acked 7634 Flush 0 New Done Done false
18750 6636 Acked 7635 Read 10 New Done Done false
18751 6637 Acked 7636 Read 2 New Done Done false
18752 6638 Acked 7637 Read 7 New Done Done false
18753 6639 Acked 7638 Write 2 New Done Done false
18754 6640 Acked 7639 Read 8 New Done Done false
18755 6641 Acked 7640 Read 6 New Done Done false
18756 6642 Acked 7641 Read 1 New Done Done false
18757 6643 Acked 7642 Write 8 New Done Done false
18758 6644 Acked 7643 Read 8 New Done Done false
18759 6645 Acked 7644 Read 6 New Done Done false
18760 6646 Acked 7645 Write 10 New Done Done false
18761 6647 Acked 7646 Read 7 New Done Done false
18762 6648 Acked 7647 Read 7 New Done Done false
18763 6649 Acked 7648 Write 10 New Done Done false
18764 6650 Acked 7649 Read 2 New Done Done false
18765 STATES DS:0 DS:1 DS:2 TOTAL
18766 New 301 0 0 301
18767 Sent 0 0 0 0
18768 Done 0 301 301 602
18769 Skipped 0 0 0 0
18770 Error 0 0 0 0
18771 Last Flush: 7348 7634 7634
18772 Downstairs last five completed: 7348 7347 7346 7345 7344 7343
18773 Upstairs last five completed: 6650 6649 6648 6647 6646 6645
18774 CLIENT: Up:0 ds:301 act:2
187752023-10-13T04:45:50.526ZINFOcrucible: [0] connecting to 127.0.0.1:8810 looper = 0
18776 ----------------------------------------------------------------
18777 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
18778 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
18779 6350 Acked 7349 Flush 0 New Done Done false
18780 6351 Acked 7350 Write 8 New Done Done false
18781 6352 Acked 7351 Read 9 New Done Done false
18782 6353 Acked 7352 Read 1 New Done Done false
18783 6354 Acked 7353 Read 10 New Done Done false
18784 6355 Acked 7354 Write 10 New Done Done false
18785 6356 Acked 7355 Write 1 New Done Done false
18786 6357 Acked 7356 Read 7 New Done Done false
18787 6358 Acked 7357 Read 5 New Done Done false
18788 6359 Acked 7358 Write 3 New Done Done false
18789 6360 Acked 7359 Read 9 New Done Done false
18790 6361 Acked 7360 Read 4 New Done Done false
18791 6362 Acked 7361 Flush 0 New Done Done false
18792 6363 Acked 7362 Flush 0 New Done Done false
18793 6364 Acked 7363 Write 10 New Done Done false
18794 6365 Acked 7364 Write 7 New Done Done false
18795 6366 Acked 7365 Read 5 New Done Done false
18796 6367 Acked 7366 Read 3 New Done Done false
18797 6368 Acked 7367 Flush 0 New Done Done false
18798 6369 Acked 7368 Read 9 New Done Done false
18799 6370 Acked 7369 Read 5 New Done Done false
18800 6371 Acked 7370 Write 2 New Done Done false
18801 6372 Acked 7371 Read 7 New Done Done false
18802 6373 Acked 7372 Read 8 New Done Done false
18803 6374 Acked 7373 Write 4 New Done Done false
18804 6375 Acked 7374 Write 4 New Done Done false
18805 6376 Acked 7375 Write 4 New Done Done false
18806 6377 Acked 7376 Write 8 New Done Done false
18807 6378 Acked 7377 Read 6 New Done Done false
18808 6379 Acked 7378 Write 7 New Done Done false
18809 6380 Acked 7379 Read 2 New Done Done false
18810 6381 Acked 7380 Read 9 New Done Done false
18811 6382 Acked 7381 Write 6 New Done Done false
18812 6383 Acked 7382 Read 5 New Done Done false
18813 6384 Acked 7383 Read 4 New Done Done false
18814 6385 Acked 7384 Write 8 New Done Done false
18815 6386 Acked 7385 Read 9 New Done Done false
18816 6387 Acked 7386 Write 9 New Done Done false
18817 6388 Acked 7387 Read 1 New Done Done false
18818 6389 Acked 7388 Read 7 New Done Done false
18819 6390 Acked 7389 Write 2 New Done Done false
18820 6391 Acked 7390 Read 3 New Done Done false
18821 6392 Acked 7391 Read 6 New Done Done false
18822 6393 Acked 7392 Write 8 New Done Done false
18823 6394 Acked 7393 Write 5 New Done Done false
18824 6395 Acked 7394 Read 8 New Done Done false
18825 6396 Acked 7395 Read 5 New Done Done false
18826 6397 Acked 7396 Write 8 New Done Done false
18827 6398 Acked 7397 Read 6 New Done Done false
18828 6399 Acked 7398 Flush 0 New Done Done false
18829 6400 Acked 7399 Read 1 New Done Done false
18830 6401 Acked 7400 Read 6 New Done Done false
18831 6402 Acked 7401 Write 5 New Done Done false
18832 6403 Acked 7402 Write 6 New Done Done false
18833 6404 Acked 7403 Write 4 New Done Done false
18834 6405 Acked 7404 Write 7 New Done Done false
18835 6406 Acked 7405 Write 1 New Done Done false
18836 6407 Acked 7406 Read 8 New Done Done false
18837 6408 Acked 7407 Write 8 New Done Done false
18838 6409 Acked 7408 Write 1 New Done Done false
18839 6410 Acked 7409 Flush 0 New Done Done false
18840 6411 Acked 7410 Write 1 New Done Done false
18841 6412 Acked 7411 Read 3 New Done Done false
18842 6413 Acked 7412 Read 5 New Done Done false
18843 6414 Acked 7413 Write 10 New Done Done false
18844 6415 Acked 7414 Read 1 New Done Done false
18845 6416 Acked 7415 Write 1 New Done Done false
18846 6417 Acked 7416 Read 4 New Done Done false
18847 6418 Acked 7417 Read 4 New Done Done false
18848 6419 Acked 7418 Read 6 New Done Done false
18849 6420 Acked 7419 Write 6 New Done Done false
18850 6421 Acked 7420 Write 10 New Done Done false
18851 6422 Acked 7421 Read 8 New Done Done false
18852 6423 Acked 7422 Read 7 New Done Done false
18853 6424 Acked 7423 Write 3 New Done Done false
18854 6425 Acked 7424 Flush 0 New Done Done false
18855 6426 Acked 7425 Read 8 New Done Done false
18856 6427 Acked 7426 Read 3 New Done Done false
18857 6428 Acked 7427 Write 7 New Done Done false
18858 6429 Acked 7428 Read 8 New Done Done false
18859 6430 Acked 7429 Write 1 New Done Done false
18860 6431 Acked 7430 Read 6 New Done Done false
18861 6432 Acked 7431 Write 4 New Done Done false
18862 6433 Acked 7432 Flush 0 New Done Done false
18863 6434 Acked 7433 Write 6 New Done Done false
18864 6435 Acked 7434 Flush 0 New Done Done false
18865 6436 Acked 7435 Read 9 New Done Done false
18866 6437 Acked 7436 Write 3 New Done Done false
18867 6438 Acked 7437 Flush 0 New Done Done false
18868 6439 Acked 7438 Write 10 New Done Done false
18869 6440 Acked 7439 Read 10 New Done Done false
18870 6441 Acked 7440 Write 9 New Done Done false
18871 6442 Acked 7441 Read 2 New Done Done false
18872 6443 Acked 7442 Write 3 New Done Done false
18873 6444 Acked 7443 Read 9 New Done Done false
18874 6445 Acked 7444 Read 9 New Done Done false
18875 6446 Acked 7445 Read 8 New Done Done false
18876 6447 Acked 7446 Flush 0 New Done Done false
18877 6448 Acked 7447 Write 3 New Done Done false
18878 6449 Acked 7448 Read 5 New Done Done false
18879 6450 Acked 7449 Write 9 New Done Done false
18880 6451 Acked 7450 Read 4 New Done Done false
18881 6452 Acked 7451 Write 3 New Done Done false
18882 6453 Acked 7452 Read 5 New Done Done false
18883 6454 Acked 7453 Write 5 New Done Done false
18884 6455 Acked 7454 Write 10 New Done Done false
18885 6456 Acked 7455 Read 1 New Done Done false
18886 6457 Acked 7456 Read 10 New Done Done false
18887 6458 Acked 7457 Write 4 New Done Done false
18888 6459 Acked 7458 Write 2 New Done Done false
18889 6460 Acked 7459 Write 7 New Done Done false
18890 6461 Acked 7460 Read 1 New Done Done false
18891 6462 Acked 7461 Read 10 New Done Done false
18892 6463 Acked 7462 Write 8 New Done Done false
18893 6464 Acked 7463 Write 3 New Done Done false
18894 6465 Acked 7464 Read 10 New Done Done false
18895 6466 Acked 7465 Write 4 New Done Done false
18896 6467 Acked 7466 Write 4 New Done Done false
18897 6468 Acked 7467 Read 8 New Done Done false
18898 6469 Acked 7468 Write 4 New Done Done false
18899 6470 Acked 7469 Read 7 New Done Done false
18900 6471 Acked 7470 Read 8 New Done Done false
18901 6472 Acked 7471 Write 3 New Done Done false
18902 6473 Acked 7472 Flush 0 New Done Done false
18903 6474 Acked 7473 Read 1 New Done Done false
18904 6475 Acked 7474 Read 10 New Done Done false
18905 6476 Acked 7475 Flush 0 New Done Done false
18906 6477 Acked 7476 Read 3 New Done Done false
18907 6478 Acked 7477 Write 6 New Done Done false
18908 6479 Acked 7478 Read 9 New Done Done false
18909 6480 Acked 7479 Read 8 New Done Done false
18910 6481 Acked 7480 Write 7 New Done Done false
18911 6482 Acked 7481 Read 7 New Done Done false
18912 6483 Acked 7482 Read 1 New Done Done false
18913 6484 Acked 7483 Write 2 New Done Done false
18914 6485 Acked 7484 Write 2 New Done Done false
18915 6486 Acked 7485 Read 10 New Done Done false
18916 6487 Acked 7486 Write 4 New Done Done false
18917 6488 Acked 7487 Write 10 New Done Done false
18918 6489 Acked 7488 Read 8 New Done Done false
18919 6490 Acked 7489 Write 1 New Done Done false
18920 6491 Acked 7490 Read 3 New Done Done false
18921 6492 Acked 7491 Write 7 New Done Done false
18922 6493 Acked 7492 Write 4 New Done Done false
18923 6494 Acked 7493 Flush 0 New Done Done false
18924 6495 Acked 7494 Write 6 New Done Done false
18925 6496 Acked 7495 Write 2 New Done Done false
18926 6497 Acked 7496 Write 4 New Done Done false
18927 6498 Acked 7497 Write 9 New Done Done false
18928 6499 Acked 7498 Read 5 New Done Done false
18929 6500 Acked 7499 Read 9 New Done Done false
18930 6501 Acked 7500 Read 1 New Done Done false
18931 6502 Acked 7501 Write 1 New Done Done false
18932 6503 Acked 7502 Write 8 New Done Done false
18933 6504 Acked 7503 Read 2 New Done Done false
18934 6505 Acked 7504 Read 5 New Done Done false
18935 6506 Acked 7505 Write 9 New Done Done false
18936 6507 Acked 7506 Write 9 New Done Done false
18937 6508 Acked 7507 Flush 0 New Done Done false
18938 6509 Acked 7508 Read 8 New Done Done false
18939 6510 Acked 7509 Write 3 New Done Done false
18940 6511 Acked 7510 Write 3 New Done Done false
18941 6512 Acked 7511 Read 8 New Done Done false
18942 6513 Acked 7512 Read 9 New Done Done false
18943 6514 Acked 7513 Write 9 New Done Done false
18944 6515 Acked 7514 Read 7 New Done Done false
18945 6516 Acked 7515 Write 6 New Done Done false
18946 6517 Acked 7516 Read 9 New Done Done false
18947 6518 Acked 7517 Read 10 New Done Done false
18948 6519 Acked 7518 Read 3 New Done Done false
18949 6520 Acked 7519 Flush 0 New Done Done false
18950 6521 Acked 7520 Write 1 New Done Done false
18951 6522 Acked 7521 Read 10 New Done Done false
18952 6523 Acked 7522 Flush 0 New Done Done false
18953 6524 Acked 7523 Read 3 New Done Done false
18954 6525 Acked 7524 Read 9 New Done Done false
18955 6526 Acked 7525 Read 2 New Done Done false
18956 6527 Acked 7526 Read 8 New Done Done false
18957 6528 Acked 7527 Read 3 New Done Done false
18958 6529 Acked 7528 Read 8 New Done Done false
18959 6530 Acked 7529 Flush 0 New Done Done false
18960 6531 Acked 7530 Read 3 New Done Done false
18961 6532 Acked 7531 Write 10 New Done Done false
18962 6533 Acked 7532 Read 8 New Done Done false
18963 6534 Acked 7533 Flush 0 New Done Done false
18964 6535 Acked 7534 Read 1 New Done Done false
18965 6536 Acked 7535 Write 6 New Done Done false
18966 6537 Acked 7536 Read 3 New Done Done false
18967 6538 Acked 7537 Write 3 New Done Done false
18968 6539 Acked 7538 Flush 0 New Done Done false
18969 6540 Acked 7539 Read 9 New Done Done false
18970 6541 Acked 7540 Write 3 New Done Done false
18971 6542 Acked 7541 Read 3 New Done Done false
18972 6543 Acked 7542 Read 8 New Done Done false
18973 6544 Acked 7543 Read 1 New Done Done false
18974 6545 Acked 7544 Write 1 New Done Done false
18975 6546 Acked 7545 Flush 0 New Done Done false
18976 6547 Acked 7546 Write 6 New Done Done false
18977 6548 Acked 7547 Read 3 New Done Done false
18978 6549 Acked 7548 Read 4 New Done Done false
18979 6550 Acked 7549 Write 1 New Done Done false
18980 6551 Acked 7550 Write 9 New Done Done false
18981 6552 Acked 7551 Flush 0 New Done Done false
18982 6553 Acked 7552 Read 7 New Done Done false
18983 6554 Acked 7553 Write 5 New Done Done false
18984 6555 Acked 7554 Write 8 New Done Done false
18985 6556 Acked 7555 Flush 0 New Done Done false
18986 6557 Acked 7556 Write 7 New Done Done false
18987 6558 Acked 7557 Read 4 New Done Done false
18988 6559 Acked 7558 Read 6 New Done Done false
18989 6560 Acked 7559 Flush 0 New Done Done false
18990 6561 Acked 7560 Read 1 New Done Done false
18991 6562 Acked 7561 Write 5 New Done Done false
18992 6563 Acked 7562 Read 4 New Done Done false
18993 6564 Acked 7563 Read 8 New Done Done false
18994 6565 Acked 7564 Write 1 New Done Done false
18995 6566 Acked 7565 Flush 0 New Done Done false
18996 6567 Acked 7566 Read 3 New Done Done false
18997 6568 Acked 7567 Flush 0 New Done Done false
18998 6569 Acked 7568 Flush 0 New Done Done false
18999 6570 Acked 7569 Write 1 New Done Done false
19000 6571 Acked 7570 Write 4 New Done Done false
19001 6572 Acked 7571 Flush 0 New Done Done false
19002 6573 Acked 7572 Read 2 New Done Done false
19003 6574 Acked 7573 Read 4 New Done Done false
19004 6575 Acked 7574 Read 2 New Done Done false
19005 6576 Acked 7575 Write 6 New Done Done false
19006 6577 Acked 7576 Read 5 New Done Done false
19007 6578 Acked 7577 Read 2 New Done Done false
19008 6579 Acked 7578 Read 5 New Done Done false
19009 6580 Acked 7579 Write 9 New Done Done false
19010 6581 Acked 7580 Read 5 New Done Done false
19011 6582 Acked 7581 Read 4 New Done Done false
19012 6583 Acked 7582 Write 1 New Done Done false
19013 6584 Acked 7583 Read 1 New Done Done false
19014 6585 Acked 7584 Read 3 New Done Done false
19015 6586 Acked 7585 Write 8 New Done Done false
19016 6587 Acked 7586 Read 7 New Done Done false
19017 6588 Acked 7587 Read 3 New Done Done false
19018 6589 Acked 7588 Read 6 New Done Done false
19019 6590 Acked 7589 Read 2 New Done Done false
19020 6591 Acked 7590 Write 9 New Done Done false
19021 6592 Acked 7591 Read 6 New Done Done false
19022 6593 Acked 7592 Write 7 New Done Done false
19023 6594 Acked 7593 Write 2 New Done Done false
19024 6595 Acked 7594 Read 4 New Done Done false
19025 6596 Acked 7595 Read 7 New Done Done false
19026 6597 Acked 7596 Read 4 New Done Done false
19027 6598 Acked 7597 Write 7 New Done Done false
19028 6599 Acked 7598 Write 9 New Done Done false
19029 6600 Acked 7599 Read 10 New Done Done false
19030 6601 Acked 7600 Flush 0 New Done Done false
19031 6602 Acked 7601 Read 5 New Done Done false
19032 6603 Acked 7602 Read 5 New Done Done false
19033 6604 Acked 7603 Flush 0 New Done Done false
19034 6605 Acked 7604 Write 4 New Done Done false
19035 6606 Acked 7605 Read 9 New Done Done false
19036 6607 Acked 7606 Write 9 New Done Done false
19037 6608 Acked 7607 Read 8 New Done Done false
19038 6609 Acked 7608 Flush 0 New Done Done false
19039 6610 Acked 7609 Write 7 New Done Done false
19040 6611 Acked 7610 Read 6 New Done Done false
19041 6612 Acked 7611 Flush 0 New Done Done false
19042 6613 Acked 7612 Write 3 New Done Done false
19043 6614 Acked 7613 Flush 0 New Done Done false
19044 6615 Acked 7614 Read 3 New Done Done false
19045 6616 Acked 7615 Read 5 New Done Done false
19046 6617 Acked 7616 Flush 0 New Done Done false
19047 6618 Acked 7617 Read 1 New Done Done false
19048 6619 Acked 7618 Read 7 New Done Done false
19049 6620 Acked 7619 Write 5 New Done Done false
19050 6621 Acked 7620 Read 6 New Done Done false
19051 6622 Acked 7621 Write 2 New Done Done false
19052 6623 Acked 7622 Read 6 New Done Done false
19053 6624 Acked 7623 Read 4 New Done Done false
19054 6625 Acked 7624 Write 2 New Done Done false
19055 6626 Acked 7625 Read 10 New Done Done false
19056 6627 Acked 7626 Read 3 New Done Done false
19057 6628 Acked 7627 Write 7 New Done Done false
19058 6629 Acked 7628 Read 3 New Done Done false
19059 6630 Acked 7629 Write 5 New Done Done false
19060 6631 Acked 7630 Write 8 New Done Done false
19061 6632 Acked 7631 Read 6 New Done Done false
19062 6633 Acked 7632 Write 7 New Done Done false
19063 6634 Acked 7633 Flush 0 New Done Done false
19064 6635 Acked 7634 Flush 0 New Done Done false
19065 6636 Acked 7635 Read 10 New Done Done false
19066 6637 Acked 7636 Read 2 New Done Done false
19067 6638 Acked 7637 Read 7 New Done Done false
19068 6639 Acked 7638 Write 2 New Done Done false
19069 6640 Acked 7639 Read 8 New Done Done false
19070 6641 Acked 7640 Read 6 New Done Done false
19071 6642 Acked 7641 Read 1 New Done Done false
19072 6643 Acked 7642 Write 8 New Done Done false
19073 6644 Acked 7643 Read 8 New Done Done false
19074 6645 Acked 7644 Read 6 New Done Done false
19075 6646 Acked 7645 Write 10 New Done Done false
19076 6647 Acked 7646 Read 7 New Done Done false
19077 6648 Acked 7647 Read 7 New Done Done false
19078 6649 Acked 7648 Write 10 New Done Done false
19079 6650 Acked 7649 Read 2 New Done Done false
19080 6651 Acked 7650 Flush 0 New Done Done false
19081 STATES DS:0 DS:1 DS:2 TOTAL
19082 New 302 0 0 302
19083 Sent 0 0 0 0
19084 Done 0 302 302 604
19085 Skipped 0 0 0 0
19086 Error 0 0 0 0
19087 Last Flush: 7348 7650 7650
19088 Downstairs last five completed: 7348 7347 7346 7345 7344 7343
19089 Upstairs last five completed: 6651 6650 6649 6648 6647 6646
19090 CLIENT: Up:0 ds:302 act:2
19091 [8810] got start action
19092 [8810] I am going to restart
19093 Starting downstairs at port 8810
19094 Make output file at "/tmp/dsc/downstairs-8810.txt"
19095 [8810][0] reports Starting
19096 Downstairs /var/tmp/dsc/region/8810 port 8810 PID:Some(1198)
19097 [8810][0] reports Running
190982023-10-13T04:45:55.532ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 0
190992023-10-13T04:45:55.532ZINFOcrucible: [0] Proc runs for 127.0.0.1:8810 in state Offline
191002023-10-13T04:45:55.532ZINFOcrucible: [0] client re-new 302 jobs since flush 7348 = downstairs
191012023-10-13T04:45:55.533ZINFOcrucible: [0] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
191022023-10-13T04:45:55.534ZINFOcrucible: [0] downstairs client at 127.0.0.1:8810 has UUID 12345678-0000-0000-0000-000000008810
191032023-10-13T04:45:55.534ZINFOcrucible: [0] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }
191042023-10-13T04:45:55.534ZINFOcrucible: Returning client:0 UUID:12345678-0000-0000-0000-000000008810 matches
191052023-10-13T04:45:55.534ZINFOcrucible: [0] send last flush ID to this DS: 7348
191062023-10-13T04:45:55.534ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Offline Active Active
191072023-10-13T04:45:55.534ZINFOcrucible: [0] Replied this last flush ID: 7348
191082023-10-13T04:45:55.534ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Offline Active Active ds_transition to Replay
191092023-10-13T04:45:55.534ZINFOcrucible: [0] Transition from Offline to Replay
191102023-10-13T04:45:55.534ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
191112023-10-13T04:45:55.534ZINFOcrucible: [0] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Replay Active Active ds_transition to Active
191122023-10-13T04:45:55.534ZINFOcrucible: [0] Transition from Replay to Active
191132023-10-13T04:45:55.534ZINFOcrucible: [0] Starts cmd_loop
191142023-10-13T04:45:55.534ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:true
191152023-10-13T04:45:55.534ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
191162023-10-13T04:45:55.534ZINFOcrucible: Set check for repair
191172023-10-13T04:45:56.536ZINFOcrucible: Checking if live repair is needed
191182023-10-13T04:45:56.536ZINFOcrucible: No Live Repair required at this time
191192023-10-13T04:45:56.536ZWARNcrucible: [0] flow control end
19120 ----------------------------------------------------------------
19121 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
19122 Downstairs last five completed: 7650 7649 7648 7647 7646 7645
19123 Upstairs last five completed: 6651 6650 6649 6648 6647 6646
19124 CLIENT: Up:0 ds:0 act:3
19125 Replay: All jobs finished, all DS active.
19126 Main task has work to do, go find it
19127 got dsc Stop(2)
19128 stop 2
19129 Oct 13 04:45:57.416 INFO request completed, latency_us: 515, response_code: 204, uri: /stop/cid/2, method: POST[8830] Got stop action so:false kr:false
19130 , req_id: 3da5af57-d38b-4f32-b89d-1551ef8f7e11, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
19131 Oct 13 04:45:57.417 INFO request completed, latency_us: 209, response_code: 200, uri: /state/cid/2, method: GET, req_id: 2694b1ed-6648-4ef5-9cd4-85ad0370c3cf, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
191322023-10-13T04:45:57.417ZWARNcrucible: [2] downstairs disconnected
191332023-10-13T04:45:57.417ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
191342023-10-13T04:45:57.417ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8830 closed looper = 2
191352023-10-13T04:45:57.417ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
191362023-10-13T04:45:57.417ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
191372023-10-13T04:45:57.417ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
191382023-10-13T04:45:57.417ZWARNcrucible: [2] pm_task rx.recv() is None
19139 [8830] Exited with: ExitStatus(unix_wait_status(9))
19140 [8830][2] reports Exit
19141 Oct 13 04:46:01.420 INFO request completed, latency_us: 254, response_code: 200, uri: /state/cid/2, method: GET, req_id: 523a0c8a-c5f2-419f-9101-9be22bfae9b0, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
19142 001/300 Read block 248 len 8192
19143 002/300 Write block 3491 len 40960 data: 3 5 6 5 4 5 5 4 3 3
19144 003/300 Read block 4675 len 12288
19145 004/300 Read block 1136 len 36864
19146 005/300 Read block 3638 len 20480
19147 006/300 Flush
19148 007/300 Read block 4293 len 32768
19149 008/300 Write block 4391 len 24576 data: 7 7 7 5 6 5
19150 009/300 Read block 2366 len 40960
19151 010/300 Write block 2434 len 4096 data: 2
19152 011/300 Write block 4357 len 20480 data: 5 5 5 5 4
19153 012/300 Read block 1275 len 32768
19154 013/300 Read block 3963 len 24576
19155 014/300 Read block 4752 len 16384
19156 015/300 Write block 3931 len 20480 data: 6 6 5 4 4
19157 016/300 Read block 2975 len 24576
19158 017/300 Write block 518 len 16384 data: 6 6 8 10
19159 018/300 Read block 1030 len 40960
19160 019/300 Write block 546 len 36864 data: 3 3 3 4 4 4 5 4 5
19161 020/300 Flush
19162 021/300 Flush
19163 022/300 Read block 2607 len 4096
19164 023/300 Read block 1418 len 16384
19165 024/300 Write block 6 len 36864 data: 6 5 5 3 3 4 4 5 5
19166 025/300 Write block 1281 len 12288 data: 4 4 4
19167 026/300 Read block 3392 len 32768
19168 027/300 Read block 547 len 16384
19169 028/300 Read block 2594 len 36864
19170 029/300 Write block 388 len 24576 data: 5 4 3 3 3 4
19171 030/300 Write block 148 len 28672 data: 7 7 7 8 8 7 7
19172 031/300 Write block 1706 len 4096 data: 4
19173 032/300 Read block 3856 len 32768
19174 033/300 Write block 355 len 12288 data: 6 6 6
19175 034/300 Read block 2129 len 32768
19176 035/300 Write block 2403 len 12288 data: 4 3 3
19177 036/300 Flush
19178 037/300 Read block 4752 len 4096
19179 038/300 Read block 2743 len 20480
19180 039/300 Write block 1537 len 40960 data: 7 7 6 5 5 5 4 5 5 4
19181 040/300 Read block 706 len 28672
19182 041/300 Read block 2137 len 40960
19183 042/300 Write block 2607 len 36864 data: 5 4 4 6 7 7 8 8 8
19184 043/300 Read block 4012 len 24576
19185 044/300 Write block 2838 len 8192 data: 6 6
19186 045/300 Read block 2097 len 20480
19187 046/300 Read block 214 len 16384
19188 047/300 Write block 4177 len 8192 data: 4 4
19189 048/300 Write block 2297 len 20480 data: 4 3 3 4 4
19190 049/300 Write block 3345 len 40960 data: 5 4 3 3 4 5 6 8 7 7
19191 050/300 Read block 4172 len 36864
19192 051/300 Read block 2311 len 28672
19193 052/300 Read block 2913 len 24576
19194 053/300 Write block 4884 len 28672 data: 3 3 6 7 6 6 6
19195 054/300 Read block 4219 len 28672
19196 055/300 Read block 3279 len 32768
19197 056/300 Read block 1033 len 28672
19198 057/300 Write block 2962 len 36864 data: 3 3 4 5 5 7 8 7 7
19199 058/300 Write block 955 len 28672 data: 3 4 3 3 4 3 3
19200 059/300 Read block 4689 len 36864
19201 060/300 Write block 3508 len 12288 data: 5 5 5
19202 061/300 Read block 2306 len 4096
19203 062/300 Flush
19204 063/300 Flush
19205 064/300 Write block 3123 len 4096 data: 5
19206 065/300 Read block 3443 len 36864
19207 066/300 Read block 3850 len 24576
19208 067/300 Read block 3126 len 36864
19209 068/300 Read block 222 len 28672
19210 069/300 Flush
19211 070/300 Flush
19212 071/300 Write block 3959 len 28672 data: 5 5 5 5 5 4 3
19213 072/300 Read block 4169 len 20480
19214 073/300 Read block 1721 len 36864
19215 074/300 Read block 72 len 20480
19216 075/300 Read block 4381 len 24576
19217 076/300 Read block 2559 len 12288
19218 077/300 Read block 1352 len 32768
19219 078/300 Read block 229 len 36864
19220 079/300 Read block 4694 len 36864
19221 080/300 Read block 317 len 8192
19222 081/300 Read block 1679 len 36864
19223 082/300 Read block 2128 len 20480
19224 083/300 Read block 3774 len 8192
19225 084/300 Write block 2776 len 8192 data: 6 5
19226 085/300 Write block 4596 len 12288 data: 3 3 4
19227 086/300 Read block 1399 len 36864
19228 087/300 Write block 1530 len 20480 data: 5 7 7 7 6
19229 088/300 Write block 70 len 8192 data: 3 4
19230 089/300 Read block 2252 len 4096
19231 090/300 Flush
19232 091/300 Read block 2914 len 36864
19233 092/300 Write block 3738 len 20480 data: 5 6 5 5 6
19234 093/300 Write block 464 len 28672 data: 4 4 3 3 2 2 3
19235 094/300 Write block 3268 len 28672 data: 2 2 2 2 3 2 3
19236 095/300 Read block 2757 len 16384
19237 096/300 Read block 3406 len 4096
19238 097/300 Write block 2979 len 28672 data: 2 3 3 3 4 4 4
19239 098/300 Read block 990 len 28672
19240 099/300 Flush
19241 100/300 Write block 3858 len 12288 data: 7 6 8
19242 101/300 Write block 1140 len 40960 data: 3 4 4 4 5 4 4 4 5 6
19243 102/300 Read block 4759 len 4096
19244 103/300 Flush
19245 104/300 Write block 3296 len 20480 data: 4 4 5 4 4
19246 105/300 Read block 2077 len 4096
19247 106/300 Read block 2571 len 40960
19248 107/300 Write block 2391 len 28672 data: 5 5 4 4 6 6 5
19249 108/300 Write block 4136 len 28672 data: 5 5 5 4 3 3 4
19250 109/300 Write block 1935 len 4096 data: 3
19251 110/300 Write block 1067 len 36864 data: 4 4 5 4 4 5 3 4 5
19252 111/300 Write block 3245 len 20480 data: 3 4 4 5 5
19253 112/300 Write block 96 len 4096 data: 6
19254 113/300 Write block 4556 len 32768 data: 4 5 4 3 3 3 2 2
19255 114/300 Read block 4863 len 12288
19256 115/300 Flush
19257 116/300 Flush
19258 117/300 Write block 452 len 24576 data: 5 5 4 4 3 5
19259 118/300 Flush
19260 119/300 Write block 632 len 28672 data: 3 4 3 3 4 3 6
19261 120/300 Read block 4917 len 24576
19262 121/300 Write block 1290 len 16384 data: 11 10 8 7
19263 122/300 Read block 2134 len 28672
19264 123/300 Write block 3244 len 20480 data: 3 4 5 5 6
19265 124/300 Read block 2554 len 4096
19266 125/300 Write block 1837 len 12288 data: 5 5 5
19267 126/300 Read block 4407 len 36864
19268 127/300 Read block 1505 len 8192
19269 128/300 Write block 2741 len 36864 data: 6 4 3 3 4 4 5 7 7
19270 129/300 Write block 3304 len 4096 data: 5
19271 130/300 Read block 1971 len 40960
19272 131/300 Write block 1050 len 24576 data: 4 4 3 3 5 5
19273 132/300 Flush
19274 133/300 Write block 4894 len 20480 data: 5 5 4 3 3
19275 134/300 Flush
19276 135/300 Read block 3876 len 8192
19277 136/300 Write block 154 len 4096 data: 8
19278 137/300 Read block 2975 len 28672
19279 138/300 Write block 2107 len 20480 data: 6 6 5 7 5
19280 139/300 Read block 3631 len 32768
19281 140/300 Write block 3955 len 20480 data: 4 6 5 5 6
19282 141/300 Write block 736 len 40960 data: 4 3 4 4 3 5 5 4 4 6
19283 142/300 Read block 1255 len 40960
19284 143/300 Read block 1172 len 24576
19285 144/300 Read block 1634 len 16384
19286 145/300 Read block 417 len 20480
19287 146/300 Read block 1351 len 20480
19288 147/300 Write block 818 len 4096 data: 4
19289 148/300 Write block 4436 len 36864 data: 5 7 7 6 6 8 9 6 5
19290 149/300 Read block 4554 len 16384
19291 150/300 Flush
19292 151/300 Write block 2617 len 4096 data: 4
19293 152/300 Read block 1741 len 20480
19294 153/300 Write block 2440 len 12288 data: 3 3 3
19295 154/300 Write block 4162 len 28672 data: 7 8 6 5 5 7 6
19296 155/300 Read block 4075 len 4096
19297 156/300 Write block 439 len 32768 data: 5 7 7 7 7 5 4 5
19298 157/300 Read block 4386 len 20480
19299 158/300 Read block 1508 len 36864
19300 159/300 Read block 4497 len 12288
19301 160/300 Read block 1148 len 20480
19302 161/300 Read block 1693 len 28672
19303 162/300 Read block 1987 len 12288
19304 163/300 Write block 149 len 32768 data: 8 8 9 9 8 9 7 6
19305 164/300 Read block 2585 len 20480
19306 165/300 Write block 4233 len 20480 data: 2 2 2 2 2
19307 166/300 Write block 2870 len 40960 data: 2 2 2 4 4 5 5 5 5 6
19308 167/300 Write block 2266 len 16384 data: 5 7 6 5
19309 168/300 Write block 2608 len 40960 data: 5 5 7 8 8 9 9 9 7 5
19310 169/300 Write block 4502 len 4096 data: 4
19311 170/300 Write block 3965 len 40960 data: 4 4 4 4 4 3 3 2 2 2
19312 171/300 Read block 2025 len 20480
19313 172/300 Write block 1875 len 8192 data: 6 7
19314 173/300 Write block 2009 len 24576 data: 5 6 5 5 4 3
19315 174/300 Flush
19316 175/300 Write block 3050 len 12288 data: 6 6 6
19317 176/300 Read block 3657 len 28672
19318 177/300 Flush
19319 178/300 Write block 1104 len 36864 data: 6 5 5 5 5 6 5 5 3
19320 179/300 Write block 4339 len 28672 data: 3 3 3 3 4 7 7
19321 180/300 Write block 3059 len 32768 data: 4 4 6 5 3 5 4 5
19322 181/300 Write block 3597 len 28672 data: 3 3 4 4 4 6 6
19323 182/300 Write block 4489 len 8192 data: 4 3
19324 183/300 Write block 3856 len 16384 data: 5 6 8 7
19325 184/300 Read block 3785 len 12288
19326 185/300 Read block 1263 len 12288
19327 186/300 Read block 2105 len 28672
19328 187/300 Read block 3480 len 8192
19329 188/300 Read block 1621 len 8192
19330 189/300 Read block 2572 len 8192
19331 190/300 Flush
19332 191/300 Read block 3567 len 20480
19333 192/300 Read block 4846 len 28672
19334 193/300 Write block 358 len 12288 data: 4 3 4
19335 194/300 Read block 3945 len 16384
19336 195/300 Write block 1137 len 24576 data: 3 5 4 4 5 5
19337 196/300 Write block 2445 len 32768 data: 4 5 4 4 3 3 3 3
19338 197/300 Read block 2925 len 32768
19339 198/300 Write block 475 len 32768 data: 5 5 6 6 5 4 3 2
19340 199/300 Read block 3427 len 12288
19341 200/300 Write block 1098 len 28672 data: 9 7 7 8 9 6 7
19342 201/300 Write block 4530 len 16384 data: 6 6 4 3
19343 202/300 Write block 2061 len 32768 data: 4 5 5 5 5 4 4 4
19344 203/300 Flush
19345 204/300 Write block 4831 len 28672 data: 5 4 5 4 4 3 3
19346 205/300 Read block 4848 len 8192
19347 206/300 Write block 1381 len 16384 data: 3 4 5 6
19348 207/300 Read block 3677 len 16384
19349 208/300 Write block 4942 len 32768 data: 8 7 6 6 5 5 5 4
19350 209/300 Flush
19351 210/300 Flush
19352 211/300 Write block 4978 len 8192 data: 5 6
19353 212/300 Read block 1834 len 12288
19354 213/300 Flush
19355 214/300 Write block 3250 len 12288 data: 6 5 5
19356 215/300 Write block 275 len 20480 data: 4 5 4 4 4
19357 216/300 Read block 4312 len 8192
19358 217/300 Read block 3693 len 24576
19359 218/300 Read block 2659 len 4096
19360 219/300 Write block 4845 len 36864 data: 2 2 2 2 2 2 3 3 2
19361 220/300 Read block 3830 len 4096
19362 221/300 Read block 2143 len 24576
19363 222/300 Read block 4411 len 40960
19364 223/300 Read block 284 len 24576
19365 224/300 Read block 4728 len 12288
19366 225/300 Read block 34 len 24576
19367 226/300 Flush
19368 227/300 Flush
19369 228/300 Read block 4256 len 28672
19370 229/300 Read block 1762 len 24576
19371 230/300 Read block 1149 len 32768
19372 231/300 Write block 4715 len 20480 data: 5 4 4 5 5
19373 232/300 Read block 1143 len 36864
19374 233/300 Write block 2041 len 12288 data: 4 4 4
19375 234/300 Read block 4972 len 36864
19376 235/300 Write block 4385 len 28672 data: 4 6 6 7 7 8 8
19377 236/300 Write block 4973 len 4096 data: 4
19378 237/300 Write block 450 len 32768 data: 3 3 6 6 5 5 4 6
19379 238/300 Read block 1330 len 40960
19380 239/300 Read block 2302 len 16384
19381 240/300 Write block 190 len 32768 data: 6 6 5 5 5 5 7 6
19382 241/300 Read block 866 len 24576
19383 242/300 Flush
19384 243/300 Flush
19385 244/300 Write block 1270 len 32768 data: 5 4 3 4 4 4 3 5
19386 245/300 Read block 439 len 16384
19387 246/300 Write block 4170 len 40960 data: 5 3 4 4 3 3 4 5 5 4
19388 247/300 Read block 1867 len 16384
19389 248/300 Read block 4704 len 24576
19390 249/300 Read block 1029 len 16384
19391 250/300 Write block 372 len 24576 data: 4 4 4 6 5 4
19392 251/300 Read block 4900 len 4096
19393 252/300 Read block 1521 len 12288
19394 253/300 Read block 2113 len 16384
19395 254/300 Write block 4198 len 24576 data: 4 3 5 5 6 6
19396 255/300 Flush
19397 256/300 Read block 4372 len 16384
19398 257/300 Read block 355 len 4096
19399 258/300 Write block 501 len 16384 data: 4 4 4 4
19400 259/300 Read block 776 len 12288
19401 260/300 Read block 957 len 4096
19402 261/300 Read block 3690 len 12288
19403 262/300 Read block 4433 len 28672
19404 263/300 Read block 1461 len 28672
19405 264/300 Write block 4606 len 32768 data: 6 5 4 4 4 3 3 4
19406 265/300 Write block 4782 len 36864 data: 7 7 5 6 6 6 4 2 3
19407 266/300 Write block 3881 len 4096 data: 4
19408 267/300 Write block 1093 len 4096 data: 9
19409 268/300 Read block 1506 len 4096
19410 269/300 Write block 2561 len 36864 data: 6 5 5 6 6 7 7 8 9
19411 270/300 Read block 2205 len 32768
19412 271/300 Write block 1812 len 28672 data: 7 6 4 4 2 3 3
19413 272/300 Read block 2322 len 32768
19414 273/300 Write block 4013 len 28672 data: 2 2 4 4 5 6 7
19415 274/300 Read block 1800 len 40960
19416 275/300 Write block 3241 len 12288 data: 3 4 4
19417 276/300 Write block 673 len 24576 data: 3 4 4 5 5 5
19418 277/300 Read block 4471 len 12288
19419 278/300 Write block 2660 len 16384 data: 4 4 4 3
19420 279/300 Write block 1026 len 24576 data: 6 6 5 5 5 5
19421 280/300 Read block 2127 len 4096
19422 281/300 Write block 2584 len 16384 data: 5 5 5 4
19423 282/300 Write block 614 len 40960 data: 7 7 7 7 6 8 6 7 6 4
19424 283/300 Write block 1854 len 20480 data: 5 7 8 9 9
19425 284/300 Read block 2719 len 20480
19426 285/300 Write block 3436 len 24576 data: 3 3 2 2 3 3
19427 286/300 Write block 3889 len 16384 data: 5 5 5 6
19428 287/300 Write block 3000 len 4096 data: 2
19429 288/300 Read block 3848 len 8192
19430 289/300 Write block 1239 len 16384 data: 4 4 4 4
19431 290/300 Read block 2595 len 36864
19432 291/300 Write block 543 len 40960 data: 3 4 3 4 4 4 5 5 5 6
19433 292/300 Write block 2204 len 32768 data: 3 3 3 3 3 4 4 4
19434 293/300 Write block 1272 len 24576 data: 4 5 5 5 4 6
19435 294/300 Flush
19436 295/300 Read block 2507 len 28672
19437 296/300 Read block 1224 len 36864
19438 297/300 Write block 136 len 4096 data: 2
19439 298/300 Write block 2509 len 24576 data: 5 7 8 7 7 7
19440 299/300 Read block 2507 len 4096
19441 300/300 Read block 3004 len 40960
19442 Main task has work to do, go find it
19443 got dsc Start(2)
19444 start 2
19445 Oct 13 04:46:02.338 INFO request completed, latency_us: 290, response_code: 204, uri: /start/cid/2, method: POST, req_id: fc0cb3dc-42fb-42f0-8240-53c5045435b5, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
19446 Replay: started 2, returned:Ok(())
19447 ----------------------------------------------------------------
19448 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
19449 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
19450 6652 Acked 7651 Read 2 Done Done New false
19451 6653 Acked 7652 Write 10 Done Done New false
19452 6654 Acked 7653 Read 3 Done Done New false
19453 6655 Acked 7654 Read 9 Done Done New false
19454 6656 Acked 7655 Read 5 Done Done New false
19455 6657 Acked 7656 Flush 0 Done Done New false
19456 6658 Acked 7657 Read 8 Done Done New false
19457 6659 Acked 7658 Write 6 Done Done New false
19458 6660 Acked 7659 Read 10 Done Done New false
19459 6661 Acked 7660 Write 1 Done Done New false
19460 6662 Acked 7661 Write 5 Done Done New false
19461 6663 Acked 7662 Read 8 Done Done New false
19462 6664 Acked 7663 Read 6 Done Done New false
19463 6665 Acked 7664 Read 4 Done Done New false
19464 6666 Acked 7665 Write 5 Done Done New false
19465 6667 Acked 7666 Read 6 Done Done New false
19466 6668 Acked 7667 Write 4 Done Done New false
19467 6669 Acked 7668 Read 10 Done Done New false
19468 6670 Acked 7669 Write 9 Done Done New false
19469 6671 Acked 7670 Flush 0 Done Done New false
19470 6672 Acked 7671 Flush 0 Done Done New false
19471 6673 Acked 7672 Read 1 Done Done New false
19472 6674 Acked 7673 Read 4 Done Done New false
19473 6675 Acked 7674 Write 9 Done Done New false
19474 6676 Acked 7675 Write 3 Done Done New false
19475 6677 Acked 7676 Read 8 Done Done New false
19476 6678 Acked 7677 Read 4 Done Done New false
19477 6679 Acked 7678 Read 9 Done Done New false
19478 6680 Acked 7679 Write 6 Done Done New false
19479 6681 Acked 7680 Write 7 Done Done New false
19480 6682 Acked 7681 Write 1 Done Done New false
19481 6683 Acked 7682 Read 8 Done Done New false
19482 6684 Acked 7683 Write 3 Done Done New false
19483 6685 Acked 7684 Read 8 Done Done New false
19484 6686 Acked 7685 Write 3 Done Done New false
19485 6687 Acked 7686 Flush 0 Done Done New false
19486 6688 Acked 7687 Read 1 Done Done New false
19487 6689 Acked 7688 Read 5 Done Done New false
19488 6690 Acked 7689 Write 10 Done Done New false
19489 6691 Acked 7690 Read 7 Done Done New false
19490 6692 Acked 7691 Read 10 Done Done New false
19491 6693 Acked 7692 Write 9 Done Done New false
19492 6694 Acked 7693 Read 6 Done Done New false
19493 6695 Acked 7694 Write 2 Done Done New false
19494 6696 Acked 7695 Read 5 Done Done New false
19495 6697 Acked 7696 Read 4 Done Done New false
19496 6698 Acked 7697 Write 2 Done Done New false
19497 6699 Acked 7698 Write 5 Done Done New false
19498 6700 Acked 7699 Write 10 Done Done New false
19499 6701 Acked 7700 Read 9 Done Done New false
19500 6702 Acked 7701 Read 7 Done Done New false
19501 6703 Acked 7702 Read 6 Done Done New false
19502 6704 Acked 7703 Write 7 Done Done New false
19503 6705 Acked 7704 Read 7 Done Done New false
19504 6706 Acked 7705 Read 8 Done Done New false
19505 6707 Acked 7706 Read 7 Done Done New false
19506 6708 Acked 7707 Write 9 Done Done New false
19507 6709 Acked 7708 Write 7 Done Done New false
19508 6710 Acked 7709 Read 9 Done Done New false
19509 6711 Acked 7710 Write 3 Done Done New false
19510 6712 Acked 7711 Read 1 Done Done New false
19511 6713 Acked 7712 Flush 0 Done Done New false
19512 6714 Acked 7713 Flush 0 Done Done New false
19513 6715 Acked 7714 Write 1 Done Done New false
19514 6716 Acked 7715 Read 9 Done Done New false
19515 6717 Acked 7716 Read 6 Done Done New false
19516 6718 Acked 7717 Read 9 Done Done New false
19517 6719 Acked 7718 Read 7 Done Done New false
19518 6720 Acked 7719 Flush 0 Done Done New false
19519 6721 Acked 7720 Flush 0 Done Done New false
19520 6722 Acked 7721 Write 7 Done Done New false
19521 6723 Acked 7722 Read 5 Done Done New false
19522 6724 Acked 7723 Read 9 Done Done New false
19523 6725 Acked 7724 Read 5 Done Done New false
19524 6726 Acked 7725 Read 6 Done Done New false
19525 6727 Acked 7726 Read 3 Done Done New false
19526 6728 Acked 7727 Read 8 Done Done New false
19527 6729 Acked 7728 Read 9 Done Done New false
19528 6730 Acked 7729 Read 9 Done Done New false
19529 6731 Acked 7730 Read 2 Done Done New false
19530 6732 Acked 7731 Read 9 Done Done New false
19531 6733 Acked 7732 Read 5 Done Done New false
19532 6734 Acked 7733 Read 2 Done Done New false
19533 6735 Acked 7734 Write 2 Done Done New false
19534 6736 Acked 7735 Write 3 Done Done New false
19535 6737 Acked 7736 Read 9 Done Done New false
19536 6738 Acked 7737 Write 5 Done Done New false
19537 6739 Acked 7738 Write 2 Done Done New false
19538 6740 Acked 7739 Read 1 Done Done New false
19539 6741 Acked 7740 Flush 0 Done Done New false
19540 6742 Acked 7741 Read 9 Done Done New false
19541 6743 Acked 7742 Write 5 Done Done New false
19542 6744 Acked 7743 Write 7 Done Done New false
19543 6745 Acked 7744 Write 7 Done Done New false
19544 6746 Acked 7745 Read 4 Done Done New false
19545 6747 Acked 7746 Read 1 Done Done New false
19546 6748 Acked 7747 Write 7 Done Done New false
19547 6749 Acked 7748 Read 7 Done Done New false
19548 6750 Acked 7749 Flush 0 Done Done New false
19549 6751 Acked 7750 Write 3 Done Done New false
19550 6752 Acked 7751 Write 10 Done Done New false
19551 6753 Acked 7752 Read 1 Done Done New false
19552 6754 Acked 7753 Flush 0 Done Done New false
19553 6755 Acked 7754 Write 5 Done Done New false
19554 6756 Acked 7755 Read 1 Done Done New false
19555 6757 Acked 7756 Read 10 Done Done New false
19556 6758 Acked 7757 Write 7 Done Done New false
19557 6759 Acked 7758 Write 7 Done Done New false
19558 6760 Acked 7759 Write 1 Done Done New false
19559 6761 Acked 7760 Write 9 Done Done New false
19560 6762 Acked 7761 Write 5 Done Done New false
19561 6763 Acked 7762 Write 1 Done Done New false
19562 6764 Acked 7763 Write 8 Done Done New false
19563 6765 Acked 7764 Read 3 Done Done New false
19564 6766 Acked 7765 Flush 0 Done Done New false
19565 6767 Acked 7766 Flush 0 Done Done New false
19566 6768 Acked 7767 Write 6 Done Done New false
19567 6769 Acked 7768 Flush 0 Done Done New false
19568 6770 Acked 7769 Write 7 Done Done New false
19569 6771 Acked 7770 Read 6 Done Done New false
19570 6772 Acked 7771 Write 4 Done Done New false
19571 6773 Acked 7772 Read 7 Done Done New false
19572 6774 Acked 7773 Write 5 Done Done New false
19573 6775 Acked 7774 Read 1 Done Done New false
19574 6776 Acked 7775 Write 3 Done Done New false
19575 6777 Acked 7776 Read 9 Done Done New false
19576 6778 Acked 7777 Read 2 Done Done New false
19577 6779 Acked 7778 Write 9 Done Done New false
19578 6780 Acked 7779 Write 1 Done Done New false
19579 6781 Acked 7780 Read 10 Done Done New false
19580 6782 Acked 7781 Write 6 Done Done New false
19581 6783 Acked 7782 Flush 0 Done Done New false
19582 6784 Acked 7783 Write 5 Done Done New false
19583 6785 Acked 7784 Flush 0 Done Done New false
19584 6786 Acked 7785 Read 2 Done Done New false
19585 6787 Acked 7786 Write 1 Done Done New false
19586 6788 Acked 7787 Read 7 Done Done New false
19587 6789 Acked 7788 Write 5 Done Done New false
19588 6790 Acked 7789 Read 8 Done Done New false
19589 6791 Acked 7790 Write 5 Done Done New false
19590 6792 Acked 7791 Write 10 Done Done New false
19591 6793 Acked 7792 Read 10 Done Done New false
19592 6794 Acked 7793 Read 6 Done Done New false
19593 6795 Acked 7794 Read 4 Done Done New false
19594 6796 Acked 7795 Read 5 Done Done New false
19595 6797 Acked 7796 Read 5 Done Done New false
19596 6798 Acked 7797 Write 1 Done Done New false
19597 6799 Acked 7798 Write 9 Done Done New false
19598 6800 Acked 7799 Read 4 Done Done New false
19599 6801 Acked 7800 Flush 0 Done Done New false
19600 6802 Acked 7801 Write 1 Done Done New false
19601 6803 Acked 7802 Read 5 Done Done New false
19602 6804 Acked 7803 Write 3 Done Done New false
19603 6805 Acked 7804 Write 7 Done Done New false
19604 6806 Acked 7805 Read 1 Done Done New false
19605 6807 Acked 7806 Write 8 Done Done New false
19606 6808 Acked 7807 Read 5 Done Done New false
19607 6809 Acked 7808 Read 9 Done Done New false
19608 6810 Acked 7809 Read 3 Done Done New false
19609 6811 Acked 7810 Read 5 Done Done New false
19610 6812 Acked 7811 Read 7 Done Done New false
19611 6813 Acked 7812 Read 3 Done Done New false
19612 6814 Acked 7813 Write 8 Done Done New false
19613 6815 Acked 7814 Read 5 Done Done New false
19614 6816 Acked 7815 Write 5 Done Done New false
19615 6817 Acked 7816 Write 10 Done Done New false
19616 6818 Acked 7817 Write 4 Done Done New false
19617 6819 Acked 7818 Write 10 Done Done New false
19618 6820 Acked 7819 Write 1 Done Done New false
19619 6821 Acked 7820 Write 10 Done Done New false
19620 6822 Acked 7821 Read 5 Done Done New false
19621 6823 Acked 7822 Write 2 Done Done New false
19622 6824 Acked 7823 Write 6 Done Done New false
19623 6825 Acked 7824 Flush 0 Done Done New false
19624 6826 Acked 7825 Write 3 Done Done New false
19625 6827 Acked 7826 Read 7 Done Done New false
19626 6828 Acked 7827 Flush 0 Done Done New false
19627 6829 Acked 7828 Write 9 Done Done New false
19628 6830 Acked 7829 Write 7 Done Done New false
19629 6831 Acked 7830 Write 8 Done Done New false
19630 6832 Acked 7831 Write 7 Done Done New false
19631 6833 Acked 7832 Write 2 Done Done New false
19632 6834 Acked 7833 Write 4 Done Done New false
19633 6835 Acked 7834 Read 3 Done Done New false
19634 6836 Acked 7835 Read 3 Done Done New false
19635 6837 Acked 7836 Read 7 Done Done New false
19636 6838 Acked 7837 Read 2 Done Done New false
19637 6839 Acked 7838 Read 2 Done Done New false
19638 6840 Acked 7839 Read 2 Done Done New false
19639 6841 Acked 7840 Flush 0 Done Done New false
19640 6842 Acked 7841 Read 5 Done Done New false
19641 6843 Acked 7842 Read 7 Done Done New false
19642 6844 Acked 7843 Write 3 Done Done New false
19643 6845 Acked 7844 Read 4 Done Done New false
19644 6846 Acked 7845 Write 6 Done Done New false
19645 6847 Acked 7846 Write 8 Done Done New false
19646 6848 Acked 7847 Read 8 Done Done New false
19647 6849 Acked 7848 Write 8 Done Done New false
19648 6850 Acked 7849 Read 3 Done Done New false
19649 6851 Acked 7850 Write 7 Done Done New false
19650 6852 Acked 7851 Write 4 Done Done New false
19651 6853 Acked 7852 Write 8 Done Done New false
19652 6854 Acked 7853 Flush 0 Done Done New false
19653 6855 Acked 7854 Write 7 Done Done New false
19654 6856 Acked 7855 Read 2 Done Done New false
19655 6857 Acked 7856 Write 4 Done Done New false
19656 6858 Acked 7857 Read 4 Done Done New false
19657 6859 Acked 7858 Write 8 Done Done New false
19658 6860 Acked 7859 Flush 0 Done Done New false
19659 6861 Acked 7860 Flush 0 Done Done New false
19660 6862 Acked 7861 Write 2 Done Done New false
19661 6863 Acked 7862 Read 3 Done Done New false
19662 6864 Acked 7863 Flush 0 Done Done New false
19663 6865 Acked 7864 Write 3 Done Done New false
19664 6866 Acked 7865 Write 5 Done Done New false
19665 6867 Acked 7866 Read 2 Done Done New false
19666 6868 Acked 7867 Read 6 Done Done New false
19667 6869 Acked 7868 Read 1 Done Done New false
19668 6870 Acked 7869 Write 9 Done Done New false
19669 6871 Acked 7870 Read 1 Done Done New false
19670 6872 Acked 7871 Read 6 Done Done New false
19671 6873 Acked 7872 Read 10 Done Done New false
19672 6874 Acked 7873 Read 6 Done Done New false
19673 6875 Acked 7874 Read 3 Done Done New false
19674 6876 Acked 7875 Read 6 Done Done New false
19675 6877 Acked 7876 Flush 0 Done Done New false
19676 6878 Acked 7877 Flush 0 Done Done New false
19677 6879 Acked 7878 Read 7 Done Done New false
19678 6880 Acked 7879 Read 6 Done Done New false
19679 6881 Acked 7880 Read 8 Done Done New false
19680 6882 Acked 7881 Write 5 Done Done New false
19681 6883 Acked 7882 Read 9 Done Done New false
19682 6884 Acked 7883 Write 3 Done Done New false
19683 6885 Acked 7884 Read 9 Done Done New false
19684 6886 Acked 7885 Write 7 Done Done New false
19685 6887 Acked 7886 Write 1 Done Done New false
19686 6888 Acked 7887 Write 8 Done Done New false
19687 6889 Acked 7888 Read 10 Done Done New false
19688 6890 Acked 7889 Read 4 Done Done New false
19689 6891 Acked 7890 Write 8 Done Done New false
19690 6892 Acked 7891 Read 6 Done Done New false
19691 6893 Acked 7892 Flush 0 Done Done New false
19692 6894 Acked 7893 Flush 0 Done Done New false
19693 6895 Acked 7894 Write 8 Done Done New false
19694 6896 Acked 7895 Read 4 Done Done New false
19695 6897 Acked 7896 Write 10 Done Done New false
19696 6898 Acked 7897 Read 4 Done Done New false
19697 6899 Acked 7898 Read 6 Done Done New false
19698 6900 Acked 7899 Read 4 Done Done New false
19699 6901 Acked 7900 Write 6 Done Done New false
19700 6902 Acked 7901 Read 1 Done Done New false
19701 6903 Acked 7902 Read 3 Done Done New false
19702 6904 Acked 7903 Read 4 Done Done New false
19703 6905 Acked 7904 Write 6 Done Done New false
19704 6906 Acked 7905 Flush 0 Done Done New false
19705 6907 Acked 7906 Read 4 Done Done New false
19706 6908 Acked 7907 Read 1 Done Done New false
19707 6909 Acked 7908 Write 4 Done Done New false
19708 6910 Acked 7909 Read 3 Done Done New false
19709 6911 Acked 7910 Read 1 Done Done New false
19710 6912 Acked 7911 Read 3 Done Done New false
19711 6913 Acked 7912 Read 7 Done Done New false
19712 6914 Acked 7913 Read 7 Done Done New false
19713 6915 Acked 7914 Write 8 Done Done New false
19714 6916 Acked 7915 Write 9 Done Done New false
19715 6917 Acked 7916 Write 1 Done Done New false
19716 6918 Acked 7917 Write 1 Done Done New false
19717 6919 Acked 7918 Read 1 Done Done New false
19718 6920 Acked 7919 Write 9 Done Done New false
19719 6921 Acked 7920 Read 8 Done Done New false
19720 6922 Acked 7921 Write 7 Done Done New false
19721 6923 Acked 7922 Read 8 Done Done New false
19722 6924 Acked 7923 Write 7 Done Done New false
19723 6925 Acked 7924 Read 10 Done Done New false
19724 6926 Acked 7925 Write 3 Done Done New false
19725 6927 Acked 7926 Write 6 Done Done New false
19726 6928 Acked 7927 Read 3 Done Done New false
19727 6929 Acked 7928 Write 4 Done Done New false
19728 6930 Acked 7929 Write 6 Done Done New false
19729 6931 Acked 7930 Read 1 Done Done New false
19730 6932 Acked 7931 Write 4 Done Done New false
19731 6933 Acked 7932 Write 10 Done Done New false
19732 6934 Acked 7933 Write 5 Done Done New false
19733 6935 Acked 7934 Read 5 Done Done New false
19734 6936 Acked 7935 Write 6 Done Done New false
19735 6937 Acked 7936 Write 4 Done Done New false
19736 6938 Acked 7937 Write 1 Done Done New false
19737 6939 Acked 7938 Read 2 Done Done New false
19738 6940 Acked 7939 Write 4 Done Done New false
19739 6941 Acked 7940 Read 9 Done Done New false
19740 6942 Acked 7941 Write 10 Done Done New false
19741 6943 Acked 7942 Write 8 Done Done New false
19742 6944 Acked 7943 Write 6 Done Done New false
19743 6945 Acked 7944 Flush 0 Done Done New false
19744 6946 Acked 7945 Read 7 Done Done New false
19745 6947 Acked 7946 Read 9 Done Done New false
19746 6948 Acked 7947 Write 1 Done Done New false
19747 6949 Acked 7948 Write 6 Done Done New false
19748 6950 Acked 7949 Read 1 Done Done New false
19749 6951 Acked 7950 Read 10 Done Done New false
19750 STATES DS:0 DS:1 DS:2 TOTAL
19751 New 0 0 300 300
19752 Sent 0 0 0 0
19753 Done 300 300 0 600
19754 Skipped 0 0 0 0
19755 Error 0 0 0 0
19756 Last Flush: 7944 7944 7650
19757 Downstairs last five completed: 7650 7649 7648 7647 7646 7645
19758 Upstairs last five completed: 6951 6950 6949 6948 6947 6946
19759 CLIENT: Up:0 ds:300 act:2
19760 ----------------------------------------------------------------
19761 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
19762 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
19763 6652 Acked 7651 Read 2 Done Done New false
19764 6653 Acked 7652 Write 10 Done Done New false
19765 6654 Acked 7653 Read 3 Done Done New false
19766 6655 Acked 7654 Read 9 Done Done New false
19767 6656 Acked 7655 Read 5 Done Done New false
19768 6657 Acked 7656 Flush 0 Done Done New false
19769 6658 Acked 7657 Read 8 Done Done New false
19770 6659 Acked 7658 Write 6 Done Done New false
19771 6660 Acked 7659 Read 10 Done Done New false
19772 6661 Acked 7660 Write 1 Done Done New false
19773 6662 Acked 7661 Write 5 Done Done New false
19774 6663 Acked 7662 Read 8 Done Done New false
19775 6664 Acked 7663 Read 6 Done Done New false
19776 6665 Acked 7664 Read 4 Done Done New false
19777 6666 Acked 7665 Write 5 Done Done New false
19778 6667 Acked 7666 Read 6 Done Done New false
19779 6668 Acked 7667 Write 4 Done Done New false
19780 6669 Acked 7668 Read 10 Done Done New false
19781 6670 Acked 7669 Write 9 Done Done New false
19782 6671 Acked 7670 Flush 0 Done Done New false
19783 6672 Acked 7671 Flush 0 Done Done New false
19784 6673 Acked 7672 Read 1 Done Done New false
19785 6674 Acked 7673 Read 4 Done Done New false
19786 6675 Acked 7674 Write 9 Done Done New false
19787 6676 Acked 7675 Write 3 Done Done New false
19788 6677 Acked 7676 Read 8 Done Done New false
19789 6678 Acked 7677 Read 4 Done Done New false
19790 6679 Acked 7678 Read 9 Done Done New false
19791 6680 Acked 7679 Write 6 Done Done New false
19792 6681 Acked 7680 Write 7 Done Done New false
19793 6682 Acked 7681 Write 1 Done Done New false
19794 6683 Acked 7682 Read 8 Done Done New false
19795 6684 Acked 7683 Write 3 Done Done New false
19796 6685 Acked 7684 Read 8 Done Done New false
19797 6686 Acked 7685 Write 3 Done Done New false
19798 6687 Acked 7686 Flush 0 Done Done New false
19799 6688 Acked 7687 Read 1 Done Done New false
19800 6689 Acked 7688 Read 5 Done Done New false
19801 6690 Acked 7689 Write 10 Done Done New false
19802 6691 Acked 7690 Read 7 Done Done New false
19803 6692 Acked 7691 Read 10 Done Done New false
19804 6693 Acked 7692 Write 9 Done Done New false
19805 6694 Acked 7693 Read 6 Done Done New false
19806 6695 Acked 7694 Write 2 Done Done New false
19807 6696 Acked 7695 Read 5 Done Done New false
19808 6697 Acked 7696 Read 4 Done Done New false
19809 6698 Acked 7697 Write 2 Done Done New false
19810 6699 Acked 7698 Write 5 Done Done New false
19811 6700 Acked 7699 Write 10 Done Done New false
19812 6701 Acked 7700 Read 9 Done Done New false
19813 6702 Acked 7701 Read 7 Done Done New false
19814 6703 Acked 7702 Read 6 Done Done New false
19815 6704 Acked 7703 Write 7 Done Done New false
19816 6705 Acked 7704 Read 7 Done Done New false
19817 6706 Acked 7705 Read 8 Done Done New false
19818 6707 Acked 7706 Read 7 Done Done New false
19819 6708 Acked 7707 Write 9 Done Done New false
19820 6709 Acked 7708 Write 7 Done Done New false
19821 6710 Acked 7709 Read 9 Done Done New false
19822 6711 Acked 7710 Write 3 Done Done New false
19823 6712 Acked 7711 Read 1 Done Done New false
19824 6713 Acked 7712 Flush 0 Done Done New false
19825 6714 Acked 7713 Flush 0 Done Done New false
19826 6715 Acked 7714 Write 1 Done Done New false
19827 6716 Acked 7715 Read 9 Done Done New false
19828 6717 Acked 7716 Read 6 Done Done New false
19829 6718 Acked 7717 Read 9 Done Done New false
19830 6719 Acked 7718 Read 7 Done Done New false
19831 6720 Acked 7719 Flush 0 Done Done New false
19832 6721 Acked 7720 Flush 0 Done Done New false
19833 6722 Acked 7721 Write 7 Done Done New false
19834 6723 Acked 7722 Read 5 Done Done New false
19835 6724 Acked 7723 Read 9 Done Done New false
19836 6725 Acked 7724 Read 5 Done Done New false
19837 6726 Acked 7725 Read 6 Done Done New false
19838 6727 Acked 7726 Read 3 Done Done New false
19839 6728 Acked 7727 Read 8 Done Done New false
19840 6729 Acked 7728 Read 9 Done Done New false
19841 6730 Acked 7729 Read 9 Done Done New false
19842 6731 Acked 7730 Read 2 Done Done New false
19843 6732 Acked 7731 Read 9 Done Done New false
19844 6733 Acked 7732 Read 5 Done Done New false
19845 6734 Acked 7733 Read 2 Done Done New false
19846 6735 Acked 7734 Write 2 Done Done New false
19847 6736 Acked 7735 Write 3 Done Done New false
19848 6737 Acked 7736 Read 9 Done Done New false
19849 6738 Acked 7737 Write 5 Done Done New false
19850 6739 Acked 7738 Write 2 Done Done New false
19851 6740 Acked 7739 Read 1 Done Done New false
19852 6741 Acked 7740 Flush 0 Done Done New false
19853 6742 Acked 7741 Read 9 Done Done New false
19854 6743 Acked 7742 Write 5 Done Done New false
19855 6744 Acked 7743 Write 7 Done Done New false
19856 6745 Acked 7744 Write 7 Done Done New false
19857 6746 Acked 7745 Read 4 Done Done New false
19858 6747 Acked 7746 Read 1 Done Done New false
19859 6748 Acked 7747 Write 7 Done Done New false
19860 6749 Acked 7748 Read 7 Done Done New false
19861 6750 Acked 7749 Flush 0 Done Done New false
19862 6751 Acked 7750 Write 3 Done Done New false
19863 6752 Acked 7751 Write 10 Done Done New false
19864 6753 Acked 7752 Read 1 Done Done New false
19865 6754 Acked 7753 Flush 0 Done Done New false
19866 6755 Acked 7754 Write 5 Done Done New false
19867 6756 Acked 7755 Read 1 Done Done New false
19868 6757 Acked 7756 Read 10 Done Done New false
19869 6758 Acked 7757 Write 7 Done Done New false
19870 6759 Acked 7758 Write 7 Done Done New false
19871 6760 Acked 7759 Write 1 Done Done New false
19872 6761 Acked 7760 Write 9 Done Done New false
19873 6762 Acked 7761 Write 5 Done Done New false
19874 6763 Acked 7762 Write 1 Done Done New false
19875 6764 Acked 7763 Write 8 Done Done New false
19876 6765 Acked 7764 Read 3 Done Done New false
19877 6766 Acked 7765 Flush 0 Done Done New false
19878 6767 Acked 7766 Flush 0 Done Done New false
19879 6768 Acked 7767 Write 6 Done Done New false
19880 6769 Acked 7768 Flush 0 Done Done New false
19881 6770 Acked 7769 Write 7 Done Done New false
19882 6771 Acked 7770 Read 6 Done Done New false
19883 6772 Acked 7771 Write 4 Done Done New false
19884 6773 Acked 7772 Read 7 Done Done New false
19885 6774 Acked 7773 Write 5 Done Done New false
19886 6775 Acked 7774 Read 1 Done Done New false
19887 6776 Acked 7775 Write 3 Done Done New false
19888 6777 Acked 7776 Read 9 Done Done New false
19889 6778 Acked 7777 Read 2 Done Done New false
19890 6779 Acked 7778 Write 9 Done Done New false
19891 6780 Acked 7779 Write 1 Done Done New false
19892 6781 Acked 7780 Read 10 Done Done New false
19893 6782 Acked 7781 Write 6 Done Done New false
19894 6783 Acked 7782 Flush 0 Done Done New false
19895 6784 Acked 7783 Write 5 Done Done New false
19896 6785 Acked 7784 Flush 0 Done Done New false
19897 6786 Acked 7785 Read 2 Done Done New false
19898 6787 Acked 7786 Write 1 Done Done New false
19899 6788 Acked 7787 Read 7 Done Done New false
19900 6789 Acked 7788 Write 5 Done Done New false
19901 6790 Acked 7789 Read 8 Done Done New false
19902 6791 Acked 7790 Write 5 Done Done New false
19903 6792 Acked 7791 Write 10 Done Done New false
19904 6793 Acked 7792 Read 10 Done Done New false
19905 6794 Acked 7793 Read 6 Done Done New false
19906 6795 Acked 7794 Read 4 Done Done New false
19907 6796 Acked 7795 Read 5 Done Done New false
19908 6797 Acked 7796 Read 5 Done Done New false
19909 6798 Acked 7797 Write 1 Done Done New false
19910 6799 Acked 7798 Write 9 Done Done New false
19911 6800 Acked 7799 Read 4 Done Done New false
19912 6801 Acked 7800 Flush 0 Done Done New false
19913 6802 Acked 7801 Write 1 Done Done New false
19914 6803 Acked 7802 Read 5 Done Done New false
19915 6804 Acked 7803 Write 3 Done Done New false
19916 6805 Acked 7804 Write 7 Done Done New false
19917 6806 Acked 7805 Read 1 Done Done New false
19918 6807 Acked 7806 Write 8 Done Done New false
19919 6808 Acked 7807 Read 5 Done Done New false
19920 6809 Acked 7808 Read 9 Done Done New false
19921 6810 Acked 7809 Read 3 Done Done New false
19922 6811 Acked 7810 Read 5 Done Done New false
19923 6812 Acked 7811 Read 7 Done Done New false
19924 6813 Acked 7812 Read 3 Done Done New false
19925 6814 Acked 7813 Write 8 Done Done New false
19926 6815 Acked 7814 Read 5 Done Done New false
19927 6816 Acked 7815 Write 5 Done Done New false
19928 6817 Acked 7816 Write 10 Done Done New false
19929 6818 Acked 7817 Write 4 Done Done New false
19930 6819 Acked 7818 Write 10 Done Done New false
19931 6820 Acked 7819 Write 1 Done Done New false
19932 6821 Acked 7820 Write 10 Done Done New false
19933 6822 Acked 7821 Read 5 Done Done New false
19934 6823 Acked 7822 Write 2 Done Done New false
19935 6824 Acked 7823 Write 6 Done Done New false
19936 6825 Acked 7824 Flush 0 Done Done New false
19937 6826 Acked 7825 Write 3 Done Done New false
19938 6827 Acked 7826 Read 7 Done Done New false
19939 6828 Acked 7827 Flush 0 Done Done New false
19940 6829 Acked 7828 Write 9 Done Done New false
19941 6830 Acked 7829 Write 7 Done Done New false
19942 6831 Acked 7830 Write 8 Done Done New false
19943 6832 Acked 7831 Write 7 Done Done New false
19944 6833 Acked 7832 Write 2 Done Done New false
19945 6834 Acked 7833 Write 4 Done Done New false
19946 6835 Acked 7834 Read 3 Done Done New false
19947 6836 Acked 7835 Read 3 Done Done New false
19948 6837 Acked 7836 Read 7 Done Done New false
19949 6838 Acked 7837 Read 2 Done Done New false
19950 6839 Acked 7838 Read 2 Done Done New false
19951 6840 Acked 7839 Read 2 Done Done New false
19952 6841 Acked 7840 Flush 0 Done Done New false
19953 6842 Acked 7841 Read 5 Done Done New false
19954 6843 Acked 7842 Read 7 Done Done New false
19955 6844 Acked 7843 Write 3 Done Done New false
19956 6845 Acked 7844 Read 4 Done Done New false
19957 6846 Acked 7845 Write 6 Done Done New false
19958 6847 Acked 7846 Write 8 Done Done New false
19959 6848 Acked 7847 Read 8 Done Done New false
19960 6849 Acked 7848 Write 8 Done Done New false
19961 6850 Acked 7849 Read 3 Done Done New false
19962 6851 Acked 7850 Write 7 Done Done New false
19963 6852 Acked 7851 Write 4 Done Done New false
19964 6853 Acked 7852 Write 8 Done Done New false
19965 6854 Acked 7853 Flush 0 Done Done New false
19966 6855 Acked 7854 Write 7 Done Done New false
19967 6856 Acked 7855 Read 2 Done Done New false
19968 6857 Acked 7856 Write 4 Done Done New false
19969 6858 Acked 7857 Read 4 Done Done New false
19970 6859 Acked 7858 Write 8 Done Done New false
19971 6860 Acked 7859 Flush 0 Done Done New false
19972 6861 Acked 7860 Flush 0 Done Done New false
19973 6862 Acked 7861 Write 2 Done Done New false
19974 6863 Acked 7862 Read 3 Done Done New false
19975 6864 Acked 7863 Flush 0 Done Done New false
19976 6865 Acked 7864 Write 3 Done Done New false
19977 6866 Acked 7865 Write 5 Done Done New false
19978 6867 Acked 7866 Read 2 Done Done New false
19979 6868 Acked 7867 Read 6 Done Done New false
19980 6869 Acked 7868 Read 1 Done Done New false
19981 6870 Acked 7869 Write 9 Done Done New false
19982 6871 Acked 7870 Read 1 Done Done New false
19983 6872 Acked 7871 Read 6 Done Done New false
19984 6873 Acked 7872 Read 10 Done Done New false
19985 6874 Acked 7873 Read 6 Done Done New false
19986 6875 Acked 7874 Read 3 Done Done New false
19987 6876 Acked 7875 Read 6 Done Done New false
19988 6877 Acked 7876 Flush 0 Done Done New false
19989 6878 Acked 7877 Flush 0 Done Done New false
19990 6879 Acked 7878 Read 7 Done Done New false
19991 6880 Acked 7879 Read 6 Done Done New false
19992 6881 Acked 7880 Read 8 Done Done New false
19993 6882 Acked 7881 Write 5 Done Done New false
19994 6883 Acked 7882 Read 9 Done Done New false
19995 6884 Acked 7883 Write 3 Done Done New false
19996 6885 Acked 7884 Read 9 Done Done New false
19997 6886 Acked 7885 Write 7 Done Done New false
19998 6887 Acked 7886 Write 1 Done Done New false
19999 6888 Acked 7887 Write 8 Done Done New false
20000 6889 Acked 7888 Read 10 Done Done New false
20001 6890 Acked 7889 Read 4 Done Done New false
20002 6891 Acked 7890 Write 8 Done Done New false
20003 6892 Acked 7891 Read 6 Done Done New false
20004 6893 Acked 7892 Flush 0 Done Done New false
20005 6894 Acked 7893 Flush 0 Done Done New false
20006 6895 Acked 7894 Write 8 Done Done New false
20007 6896 Acked 7895 Read 4 Done Done New false
20008 6897 Acked 7896 Write 10 Done Done New false
20009 6898 Acked 7897 Read 4 Done Done New false
20010 6899 Acked 7898 Read 6 Done Done New false
20011 6900 Acked 7899 Read 4 Done Done New false
20012 6901 Acked 7900 Write 6 Done Done New false
20013 6902 Acked 7901 Read 1 Done Done New false
20014 6903 Acked 7902 Read 3 Done Done New false
20015 6904 Acked 7903 Read 4 Done Done New false
20016 6905 Acked 7904 Write 6 Done Done New false
20017 6906 Acked 7905 Flush 0 Done Done New false
20018 6907 Acked 7906 Read 4 Done Done New false
20019 6908 Acked 7907 Read 1 Done Done New false
20020 6909 Acked 7908 Write 4 Done Done New false
20021 6910 Acked 7909 Read 3 Done Done New false
20022 6911 Acked 7910 Read 1 Done Done New false
20023 6912 Acked 7911 Read 3 Done Done New false
20024 6913 Acked 7912 Read 7 Done Done New false
20025 6914 Acked 7913 Read 7 Done Done New false
20026 6915 Acked 7914 Write 8 Done Done New false
20027 6916 Acked 7915 Write 9 Done Done New false
20028 6917 Acked 7916 Write 1 Done Done New false
20029 6918 Acked 7917 Write 1 Done Done New false
20030 6919 Acked 7918 Read 1 Done Done New false
20031 6920 Acked 7919 Write 9 Done Done New false
20032 6921 Acked 7920 Read 8 Done Done New false
20033 6922 Acked 7921 Write 7 Done Done New false
20034 6923 Acked 7922 Read 8 Done Done New false
20035 6924 Acked 7923 Write 7 Done Done New false
20036 6925 Acked 7924 Read 10 Done Done New false
20037 6926 Acked 7925 Write 3 Done Done New false
20038 6927 Acked 7926 Write 6 Done Done New false
20039 6928 Acked 7927 Read 3 Done Done New false
20040 6929 Acked 7928 Write 4 Done Done New false
20041 6930 Acked 7929 Write 6 Done Done New false
20042 6931 Acked 7930 Read 1 Done Done New false
20043 6932 Acked 7931 Write 4 Done Done New false
20044 6933 Acked 7932 Write 10 Done Done New false
20045 6934 Acked 7933 Write 5 Done Done New false
20046 6935 Acked 7934 Read 5 Done Done New false
20047 6936 Acked 7935 Write 6 Done Done New false
20048 6937 Acked 7936 Write 4 Done Done New false
20049 6938 Acked 7937 Write 1 Done Done New false
20050 6939 Acked 7938 Read 2 Done Done New false
20051 6940 Acked 7939 Write 4 Done Done New false
20052 6941 Acked 7940 Read 9 Done Done New false
20053 6942 Acked 7941 Write 10 Done Done New false
20054 6943 Acked 7942 Write 8 Done Done New false
20055 6944 Acked 7943 Write 6 Done Done New false
20056 6945 Acked 7944 Flush 0 Done Done New false
20057 6946 Acked 7945 Read 7 Done Done New false
20058 6947 Acked 7946 Read 9 Done Done New false
20059 6948 Acked 7947 Write 1 Done Done New false
20060 6949 Acked 7948 Write 6 Done Done New false
20061 6950 Acked 7949 Read 1 Done Done New false
20062 6951 Acked 7950 Read 10 Done Done New false
20063 6952 Acked 7951 Flush 0 Done Done New false
20064 STATES DS:0 DS:1 DS:2 TOTAL
20065 New 0 0 301 301
20066 Sent 0 0 0 0
20067 Done 301 301 0 602
20068 Skipped 0 0 0 0
20069 Error 0 0 0 0
20070 Last Flush: 7951 7951 7650
20071 Downstairs last five completed: 7650 7649 7648 7647 7646 7645
20072 Upstairs last five completed: 6952 6951 6950 6949 6948 6947
20073 CLIENT: Up:0 ds:301 act:2
200742023-10-13T04:46:07.429ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
20075 [8830] got start action
20076 [8830] I am going to restart
20077 Starting downstairs at port 8830
20078 Make output file at "/tmp/dsc/downstairs-8830.txt"
20079 [8830][2] reports Starting
20080 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1199)
20081 [8830][2] reports Running
200822023-10-13T04:46:08.431ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 2
200832023-10-13T04:46:08.431ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
200842023-10-13T04:46:08.431ZINFOcrucible: [2] client re-new 301 jobs since flush 7650 = downstairs
200852023-10-13T04:46:08.432ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
200862023-10-13T04:46:08.433ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
200872023-10-13T04:46:08.433ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
200882023-10-13T04:46:08.433ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
200892023-10-13T04:46:08.433ZINFOcrucible: [2] send last flush ID to this DS: 7650
200902023-10-13T04:46:08.433ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
200912023-10-13T04:46:08.433ZINFOcrucible: [2] Replied this last flush ID: 7650
200922023-10-13T04:46:08.433ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Offline ds_transition to Replay
200932023-10-13T04:46:08.433ZINFOcrucible: [2] Transition from Offline to Replay
200942023-10-13T04:46:08.433ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
200952023-10-13T04:46:08.433ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Replay ds_transition to Active
200962023-10-13T04:46:08.433ZINFOcrucible: [2] Transition from Replay to Active
200972023-10-13T04:46:08.433ZINFOcrucible: [2] Starts cmd_loop
200982023-10-13T04:46:08.433ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
200992023-10-13T04:46:08.433ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
201002023-10-13T04:46:08.433ZINFOcrucible: Set check for repair
201012023-10-13T04:46:09.434ZINFOcrucible: Checking if live repair is needed
201022023-10-13T04:46:09.434ZINFOcrucible: No Live Repair required at this time
201032023-10-13T04:46:09.434ZWARNcrucible: [2] flow control end
20104 ----------------------------------------------------------------
20105 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
20106 Downstairs last five completed: 7951 7950 7949 7948 7947 7946
20107 Upstairs last five completed: 6952 6951 6950 6949 6948 6947
20108 CLIENT: Up:0 ds:0 act:3
20109 Replay: All jobs finished, all DS active.
20110 Main task has work to do, go find it
20111 got dsc Stop(1)
20112 stop 1
20113 Oct 13 04:46:10.345 INFO request completed, latency_us: 467, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 2b34306a-3278-471c-87ff-d2747b254249, remote_addr: 127.0.0.1:52046[8820] Got stop action so:false kr:false
20114 , local_addr: 127.0.0.1:9998
20115 Oct 13 04:46:10.346 INFO request completed, latency_us: 252, response_code: 200, uri: /state/cid/1, method: GET, req_id: f3c40b17-17f9-4e42-9d6a-e6308385cae9, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
201162023-10-13T04:46:10.347ZWARNcrucible: [1] downstairs disconnected
201172023-10-13T04:46:10.347ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
201182023-10-13T04:46:10.347ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8820 closed looper = 1
201192023-10-13T04:46:10.347ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:false
201202023-10-13T04:46:10.347ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
201212023-10-13T04:46:10.347ZINFOcrucible: [1] 127.0.0.1:8820 task reports offline
201222023-10-13T04:46:10.347ZWARNcrucible: [1] pm_task rx.recv() is None
20123 [8820] Exited with: ExitStatus(unix_wait_status(9))
20124 [8820][1] reports Exit
20125 Oct 13 04:46:14.348 INFO request completed, latency_us: 249, response_code: 200, uri: /state/cid/1, method: GET, req_id: 64fed2f2-ca2a-4c88-99b8-6dac9aec69d9, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
20126 001/300 Read block 4867 len 32768
20127 002/300 Read block 2205 len 24576
20128 003/300 Write block 2560 len 32768 data: 6 7 6 6 7 7 8 8
20129 004/300 Write block 4687 len 20480 data: 7 7 4 5 5
20130 005/300 Read block 2794 len 32768
20131 006/300 Read block 877 len 40960
20132 007/300 Read block 2002 len 12288
20133 008/300 Write block 3413 len 40960 data: 6 6 4 2 2 2 3 3 4 4
20134 009/300 Read block 4740 len 36864
20135 010/300 Read block 3456 len 28672
20136 011/300 Read block 1965 len 36864
20137 012/300 Read block 4987 len 20480
20138 013/300 Read block 774 len 8192
20139 014/300 Write block 3114 len 8192 data: 5 6
20140 015/300 Read block 1770 len 8192
20141 016/300 Read block 2040 len 16384
20142 017/300 Write block 4427 len 28672 data: 7 7 7 4 3 2 3
20143 018/300 Write block 1912 len 36864 data: 4 4 4 4 4 4 5 5 5
20144 019/300 Read block 4707 len 16384
20145 020/300 Write block 3606 len 24576 data: 5 5 5 6 5 7
20146 021/300 Write block 572 len 12288 data: 7 7 6
20147 022/300 Flush
20148 023/300 Write block 3527 len 20480 data: 10 9 9 10 8
20149 024/300 Read block 1758 len 4096
20150 025/300 Write block 2409 len 4096 data: 4
20151 026/300 Read block 3438 len 12288
20152 027/300 Read block 4458 len 24576
20153 028/300 Flush
20154 029/300 Flush
20155 030/300 Flush
20156 031/300 Read block 3098 len 40960
20157 032/300 Read block 4800 len 8192
20158 033/300 Write block 3025 len 20480 data: 6 6 6 6 6
20159 034/300 Write block 1407 len 36864 data: 4 4 4 4 4 4 4 5 5
20160 035/300 Read block 717 len 40960
20161 036/300 Write block 1830 len 4096 data: 6
20162 037/300 Read block 625 len 12288
20163 038/300 Write block 1745 len 16384 data: 6 5 6 5
20164 039/300 Read block 2963 len 12288
20165 040/300 Write block 4723 len 20480 data: 6 6 7 7 8
20166 041/300 Read block 4406 len 8192
20167 042/300 Read block 2203 len 28672
20168 043/300 Write block 2356 len 24576 data: 4 3 4 3 3 3
20169 044/300 Write block 3724 len 24576 data: 5 6 5 5 7 6
20170 045/300 Write block 935 len 40960 data: 4 3 3 3 3 2 3 4 4 4
20171 046/300 Read block 1919 len 28672
20172 047/300 Read block 100 len 12288
20173 048/300 Write block 840 len 28672 data: 5 6 7 9 8 8 7
20174 049/300 Write block 4392 len 40960 data: 8 8 6 7 6 4 4 4 3 5
20175 050/300 Write block 119 len 20480 data: 4 4 5 5 4
20176 051/300 Read block 2658 len 40960
20177 052/300 Read block 3228 len 36864
20178 053/300 Read block 1691 len 32768
20179 054/300 Read block 2851 len 32768
20180 055/300 Write block 1124 len 12288 data: 4 5 6
20181 056/300 Read block 1066 len 8192
20182 057/300 Read block 2345 len 20480
20183 058/300 Read block 1063 len 24576
20184 059/300 Read block 3721 len 24576
20185 060/300 Write block 1417 len 16384 data: 5 5 6 3
20186 061/300 Write block 4698 len 20480 data: 5 7 7 7 6
20187 062/300 Read block 3094 len 4096
20188 063/300 Read block 585 len 28672
20189 064/300 Read block 2180 len 12288
20190 065/300 Read block 1890 len 24576
20191 066/300 Read block 3789 len 40960
20192 067/300 Write block 1911 len 8192 data: 4 5
20193 068/300 Write block 2544 len 8192 data: 4 4
20194 069/300 Read block 4244 len 24576
20195 070/300 Read block 4243 len 24576
20196 071/300 Read block 3876 len 16384
20197 072/300 Write block 4327 len 20480 data: 3 3 3 3 3
20198 073/300 Flush
20199 074/300 Read block 2686 len 20480
20200 075/300 Write block 95 len 8192 data: 6 7
20201 076/300 Write block 1778 len 28672 data: 6 6 6 5 5 6 5
20202 077/300 Read block 1328 len 16384
20203 078/300 Read block 1537 len 16384
20204 079/300 Write block 4295 len 36864 data: 8 9 10 7 7 8 8 7 5
20205 080/300 Read block 3715 len 28672
20206 081/300 Write block 2745 len 16384 data: 5 5 6 8
20207 082/300 Write block 9 len 28672 data: 4 4 5 5 6 6 5
20208 083/300 Flush
20209 084/300 Read block 3019 len 32768
20210 085/300 Flush
20211 086/300 Write block 2907 len 4096 data: 4
20212 087/300 Read block 2521 len 28672
20213 088/300 Write block 2705 len 24576 data: 8 7 6 6 6 5
20214 089/300 Write block 4808 len 24576 data: 5 5 4 4 4 4
20215 090/300 Flush
20216 091/300 Read block 2167 len 28672
20217 092/300 Write block 2403 len 12288 data: 5 4 4
20218 093/300 Write block 3276 len 36864 data: 3 4 4 4 3 3 3 3 5
20219 094/300 Read block 3351 len 12288
20220 095/300 Write block 137 len 36864 data: 2 2 2 3 3 3 5 5 6
20221 096/300 Read block 4243 len 8192
20222 097/300 Write block 4549 len 40960 data: 4 6 7 6 5 4 4 5 6 5
20223 098/300 Read block 1129 len 12288
20224 099/300 Read block 3271 len 40960
20225 100/300 Flush
20226 101/300 Flush
20227 102/300 Write block 3224 len 8192 data: 4 4
20228 103/300 Flush
20229 104/300 Read block 4743 len 4096
20230 105/300 Read block 1897 len 12288
20231 106/300 Read block 1364 len 20480
20232 107/300 Read block 1595 len 8192
20233 108/300 Read block 4035 len 20480
20234 109/300 Write block 838 len 8192 data: 5 5
20235 110/300 Write block 1795 len 20480 data: 6 6 5 4 5
20236 111/300 Read block 1672 len 12288
20237 112/300 Read block 732 len 28672
20238 113/300 Read block 2490 len 24576
20239 114/300 Read block 380 len 40960
20240 115/300 Write block 1018 len 8192 data: 6 5
20241 116/300 Read block 3285 len 36864
20242 117/300 Read block 3997 len 36864
20243 118/300 Flush
20244 119/300 Flush
20245 120/300 Flush
20246 121/300 Read block 3242 len 24576
20247 122/300 Write block 3627 len 32768 data: 4 5 5 5 5 4 4 4
20248 123/300 Write block 2717 len 24576 data: 3 5 5 7 6 5
20249 124/300 Read block 2823 len 36864
20250 125/300 Read block 750 len 36864
20251 126/300 Read block 4692 len 28672
20252 127/300 Write block 1147 len 36864 data: 5 6 7 7 7 6 6 6 5
20253 128/300 Read block 2357 len 12288
20254 129/300 Flush
20255 130/300 Read block 2351 len 40960
20256 131/300 Flush
20257 132/300 Read block 3108 len 28672
20258 133/300 Write block 4598 len 16384 data: 5 5 5 6
20259 134/300 Read block 4745 len 24576
20260 135/300 Write block 4514 len 40960 data: 3 3 3 3 4 5 6 6 4 4
20261 136/300 Write block 3892 len 20480 data: 7 6 5 4 2
20262 137/300 Flush
20263 138/300 Write block 582 len 40960 data: 3 3 3 3 4 6 3 3 3 5
20264 139/300 Write block 4181 len 4096 data: 6
20265 140/300 Write block 13 len 28672 data: 7 7 6 5 5 6 6
20266 141/300 Read block 942 len 36864
20267 142/300 Read block 4378 len 32768
20268 143/300 Read block 4025 len 40960
20269 144/300 Read block 3706 len 4096
20270 145/300 Read block 3870 len 28672
20271 146/300 Write block 4006 len 20480 data: 6 6 7 4 4
20272 147/300 Write block 4177 len 28672 data: 6 6 5 4 7 5 5
20273 148/300 Write block 1771 len 32768 data: 8 8 6 5 7 7 7 7
20274 149/300 Flush
20275 150/300 Read block 127 len 12288
20276 151/300 Write block 715 len 40960 data: 6 7 7 5 4 5 6 5 7 8
20277 152/300 Write block 1629 len 40960 data: 7 6 6 6 6 5 5 5 6 6
20278 153/300 Flush
20279 154/300 Read block 1979 len 36864
20280 155/300 Write block 4346 len 8192 data: 9 8
20281 156/300 Flush
20282 157/300 Write block 1293 len 36864 data: 8 5 5 4 4 4 3 4 4
20283 158/300 Read block 4058 len 8192
20284 159/300 Write block 4842 len 8192 data: 3 2
20285 160/300 Write block 429 len 12288 data: 3 3 3
20286 161/300 Flush
20287 162/300 Write block 3246 len 20480 data: 6 6 7 6 7
20288 163/300 Read block 3169 len 20480
20289 164/300 Write block 4811 len 28672 data: 5 5 5 4 3 4 6
20290 165/300 Flush
20291 166/300 Flush
20292 167/300 Flush
20293 168/300 Read block 184 len 12288
20294 169/300 Read block 2237 len 36864
20295 170/300 Read block 1521 len 20480
20296 171/300 Write block 2035 len 16384 data: 5 6 6 4
20297 172/300 Read block 10 len 36864
20298 173/300 Read block 4369 len 40960
20299 174/300 Write block 1556 len 32768 data: 5 6 7 8 8 5 4 4
20300 175/300 Write block 2616 len 40960 data: 8 6 5 5 5 3 6 6 6 5
20301 176/300 Flush
20302 177/300 Write block 3929 len 16384 data: 6 6 7 7
20303 178/300 Write block 3533 len 32768 data: 10 9 9 8 6 6 4 4
20304 179/300 Write block 476 len 36864 data: 6 7 7 6 5 4 3 2 2
20305 180/300 Read block 2431 len 4096
20306 181/300 Flush
20307 182/300 Flush
20308 183/300 Write block 2824 len 4096 data: 4
20309 184/300 Read block 1660 len 8192
20310 185/300 Write block 4779 len 20480 data: 7 7 6 8 8
20311 186/300 Write block 3090 len 36864 data: 3 3 3 3 3 3 3 2 2
20312 187/300 Write block 2504 len 16384 data: 4 3 4 7
20313 188/300 Write block 4860 len 40960 data: 7 6 7 7 8 6 7 4 3 3
20314 189/300 Read block 400 len 12288
20315 190/300 Read block 3608 len 20480
20316 191/300 Read block 2411 len 24576
20317 192/300 Flush
20318 193/300 Read block 38 len 12288
20319 194/300 Read block 2444 len 40960
20320 195/300 Read block 3876 len 12288
20321 196/300 Read block 746 len 4096
20322 197/300 Write block 1360 len 20480 data: 3 3 5 5 5
20323 198/300 Write block 2154 len 12288 data: 5 5 5
20324 199/300 Flush
20325 200/300 Read block 1370 len 28672
20326 201/300 Write block 4393 len 24576 data: 9 7 8 7 5 5
20327 202/300 Read block 1118 len 4096
20328 203/300 Write block 3321 len 24576 data: 8 8 9 9 9 8
20329 204/300 Flush
20330 205/300 Flush
20331 206/300 Write block 962 len 28672 data: 3 3 3 3 3 4 6
20332 207/300 Write block 1163 len 4096 data: 5
20333 208/300 Write block 1280 len 24576 data: 4 5 5 5 5 7
20334 209/300 Write block 1366 len 32768 data: 5 4 4 3 3 3 2 2
20335 210/300 Write block 9 len 28672 data: 5 5 6 6 8 8 7
20336 211/300 Write block 258 len 24576 data: 7 6 5 5 4 3
20337 212/300 Read block 4369 len 20480
20338 213/300 Write block 4448 len 36864 data: 3 4 3 5 6 6 9 9 7
20339 214/300 Write block 2974 len 20480 data: 4 4 3 2 2
20340 215/300 Read block 993 len 24576
20341 216/300 Read block 3047 len 28672
20342 217/300 Write block 1243 len 20480 data: 4 5 5 4 3
20343 218/300 Read block 3063 len 20480
20344 219/300 Read block 817 len 8192
20345 220/300 Write block 2056 len 16384 data: 4 4 4 3
20346 221/300 Read block 2186 len 12288
20347 222/300 Read block 2181 len 12288
20348 223/300 Flush
20349 224/300 Write block 3749 len 28672 data: 3 3 4 4 4 4 4
20350 225/300 Read block 2487 len 16384
20351 226/300 Write block 2431 len 12288 data: 3 3 2
20352 227/300 Write block 2415 len 8192 data: 5 4
20353 228/300 Read block 1454 len 12288
20354 229/300 Write block 338 len 28672 data: 2 2 2 3 2 4 4
20355 230/300 Flush
20356 231/300 Write block 1667 len 16384 data: 5 6 6 7
20357 232/300 Read block 1408 len 24576
20358 233/300 Write block 952 len 24576 data: 2 3 3 4 5 4
20359 234/300 Read block 1741 len 4096
20360 235/300 Write block 3664 len 12288 data: 4 4 4
20361 236/300 Write block 684 len 28672 data: 4 3 3 3 3 2 2
20362 237/300 Write block 2167 len 28672 data: 4 4 6 5 5 4 4
20363 238/300 Write block 3752 len 32768 data: 5 5 5 5 4 5 5 4
20364 239/300 Read block 817 len 12288
20365 240/300 Read block 1391 len 36864
20366 241/300 Write block 3152 len 8192 data: 6 5
20367 242/300 Read block 1055 len 40960
20368 243/300 Write block 2474 len 28672 data: 4 3 4 4 3 3 4
20369 244/300 Read block 1297 len 28672
20370 245/300 Read block 1301 len 12288
20371 246/300 Read block 940 len 28672
20372 247/300 Write block 485 len 32768 data: 2 3 6 6 6 6 6 7
20373 248/300 Read block 806 len 12288
20374 249/300 Write block 114 len 16384 data: 4 5 3 3
20375 250/300 Read block 3222 len 20480
20376 251/300 Write block 4658 len 28672 data: 6 7 8 6 7 7 7
20377 252/300 Read block 4670 len 8192
20378 253/300 Read block 1249 len 4096
20379 254/300 Write block 4966 len 20480 data: 2 3 4 4 3
20380 255/300 Flush
20381 256/300 Write block 2173 len 8192 data: 5 3
20382 257/300 Read block 4446 len 36864
20383 258/300 Read block 3022 len 28672
20384 259/300 Write block 4649 len 36864 data: 2 2 2 4 5 3 3 4 4
20385 260/300 Write block 4663 len 32768 data: 8 8 6 5 5 4 4 4
20386 261/300 Write block 2655 len 40960 data: 6 7 9 6 6 5 5 5 4 2
20387 262/300 Read block 941 len 40960
20388 263/300 Write block 2555 len 20480 data: 6 7 7 7 5
20389 264/300 Write block 2752 len 40960 data: 7 8 7 5 5 6 6 4 4 5
20390 265/300 Read block 1444 len 36864
20391 266/300 Write block 4950 len 8192 data: 3 4
20392 267/300 Flush
20393 268/300 Read block 2670 len 4096
20394 269/300 Write block 1630 len 40960 data: 7 7 7 7 6 6 6 7 7 6
20395 270/300 Write block 4383 len 8192 data: 3 4
20396 271/300 Write block 2453 len 40960 data: 2 2 4 4 4 4 4 3 3 3
20397 272/300 Write block 1192 len 16384 data: 3 2 2 2
20398 273/300 Write block 516 len 20480 data: 5 7 7 7 9
20399 274/300 Write block 1471 len 28672 data: 5 5 5 4 3 3 3
20400 275/300 Flush
20401 276/300 Read block 1904 len 4096
20402 277/300 Write block 776 len 8192 data: 4 4
20403 278/300 Read block 1131 len 12288
20404 279/300 Write block 806 len 40960 data: 3 2 2 3 3 3 4 4 4 4
20405 280/300 Read block 3635 len 36864
20406 281/300 Read block 2194 len 12288
20407 282/300 Write block 525 len 32768 data: 8 7 7 7 6 5 3 5
20408 283/300 Read block 1799 len 36864
20409 284/300 Read block 1290 len 16384
20410 285/300 Write block 2828 len 16384 data: 7 6 5 4
20411 286/300 Read block 666 len 36864
20412 287/300 Flush
20413 288/300 Read block 150 len 4096
20414 289/300 Read block 2490 len 32768
20415 290/300 Write block 4652 len 12288 data: 5 6 4
20416 291/300 Read block 944 len 32768
20417 292/300 Write block 2566 len 28672 data: 9 9 9 10 10 10 9
20418 293/300 Read block 3730 len 40960
20419 294/300 Write block 1773 len 28672 data: 7 6 8 8 8 8 7
20420 295/300 Write block 3911 len 36864 data: 4 4 3 4 4 4 4 2 2
20421 296/300 Read block 4815 len 28672
20422 297/300 Read block 307 len 36864
20423 298/300 Read block 411 len 28672
20424 299/300 Read block 938 len 28672
20425 300/300 Write block 1985 len 4096 data: 3
20426 Main task has work to do, go find it
20427 got dsc Start(1)
20428 start 1
20429 Oct 13 04:46:15.263 INFO request completed, latency_us: 289, response_code: 204, uri: /start/cid/1, method: POST, req_id: cf584124-9b6a-4acb-937a-fc31a4c2f466, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
20430 Replay: started 1, returned:Ok(())
20431 ----------------------------------------------------------------
20432 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:300
20433 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
20434 6953 Acked 7952 Read 8 Done New Done false
20435 6954 Acked 7953 Read 6 Done New Done false
20436 6955 Acked 7954 Write 8 Done New Done false
20437 6956 Acked 7955 Write 5 Done New Done false
20438 6957 Acked 7956 Read 8 Done New Done false
20439 6958 Acked 7957 Read 10 Done New Done false
20440 6959 Acked 7958 Read 3 Done New Done false
20441 6960 Acked 7959 Write 10 Done New Done false
20442 6961 Acked 7960 Read 9 Done New Done false
20443 6962 Acked 7961 Read 7 Done New Done false
20444 6963 Acked 7962 Read 9 Done New Done false
20445 6964 Acked 7963 Read 5 Done New Done false
20446 6965 Acked 7964 Read 2 Done New Done false
20447 6966 Acked 7965 Write 2 Done New Done false
20448 6967 Acked 7966 Read 2 Done New Done false
20449 6968 Acked 7967 Read 4 Done New Done false
20450 6969 Acked 7968 Write 7 Done New Done false
20451 6970 Acked 7969 Write 9 Done New Done false
20452 6971 Acked 7970 Read 4 Done New Done false
20453 6972 Acked 7971 Write 6 Done New Done false
20454 6973 Acked 7972 Write 3 Done New Done false
20455 6974 Acked 7973 Flush 0 Done New Done false
20456 6975 Acked 7974 Write 5 Done New Done false
20457 6976 Acked 7975 Read 1 Done New Done false
20458 6977 Acked 7976 Write 1 Done New Done false
20459 6978 Acked 7977 Read 3 Done New Done false
20460 6979 Acked 7978 Read 6 Done New Done false
20461 6980 Acked 7979 Flush 0 Done New Done false
20462 6981 Acked 7980 Flush 0 Done New Done false
20463 6982 Acked 7981 Flush 0 Done New Done false
20464 6983 Acked 7982 Read 10 Done New Done false
20465 6984 Acked 7983 Read 2 Done New Done false
20466 6985 Acked 7984 Write 5 Done New Done false
20467 6986 Acked 7985 Write 9 Done New Done false
20468 6987 Acked 7986 Read 10 Done New Done false
20469 6988 Acked 7987 Write 1 Done New Done false
20470 6989 Acked 7988 Read 3 Done New Done false
20471 6990 Acked 7989 Write 4 Done New Done false
20472 6991 Acked 7990 Read 3 Done New Done false
20473 6992 Acked 7991 Write 5 Done New Done false
20474 6993 Acked 7992 Read 2 Done New Done false
20475 6994 Acked 7993 Read 7 Done New Done false
20476 6995 Acked 7994 Write 6 Done New Done false
20477 6996 Acked 7995 Write 6 Done New Done false
20478 6997 Acked 7996 Write 10 Done New Done false
20479 6998 Acked 7997 Read 7 Done New Done false
20480 6999 Acked 7998 Read 3 Done New Done false
20481 7000 Acked 7999 Write 7 Done New Done false
20482 7001 Acked 8000 Write 10 Done New Done false
20483 7002 Acked 8001 Write 5 Done New Done false
20484 7003 Acked 8002 Read 10 Done New Done false
20485 7004 Acked 8003 Read 9 Done New Done false
20486 7005 Acked 8004 Read 8 Done New Done false
20487 7006 Acked 8005 Read 8 Done New Done false
20488 7007 Acked 8006 Write 3 Done New Done false
20489 7008 Acked 8007 Read 2 Done New Done false
20490 7009 Acked 8008 Read 5 Done New Done false
20491 7010 Acked 8009 Read 6 Done New Done false
20492 7011 Acked 8010 Read 6 Done New Done false
20493 7012 Acked 8011 Write 4 Done New Done false
20494 7013 Acked 8012 Write 5 Done New Done false
20495 7014 Acked 8013 Read 1 Done New Done false
20496 7015 Acked 8014 Read 7 Done New Done false
20497 7016 Acked 8015 Read 3 Done New Done false
20498 7017 Acked 8016 Read 6 Done New Done false
20499 7018 Acked 8017 Read 10 Done New Done false
20500 7019 Acked 8018 Write 2 Done New Done false
20501 7020 Acked 8019 Write 2 Done New Done false
20502 7021 Acked 8020 Read 6 Done New Done false
20503 7022 Acked 8021 Read 6 Done New Done false
20504 7023 Acked 8022 Read 4 Done New Done false
20505 7024 Acked 8023 Write 5 Done New Done false
20506 7025 Acked 8024 Flush 0 Done New Done false
20507 7026 Acked 8025 Read 5 Done New Done false
20508 7027 Acked 8026 Write 2 Done New Done false
20509 7028 Acked 8027 Write 7 Done New Done false
20510 7029 Acked 8028 Read 4 Done New Done false
20511 7030 Acked 8029 Read 4 Done New Done false
20512 7031 Acked 8030 Write 9 Done New Done false
20513 7032 Acked 8031 Read 7 Done New Done false
20514 7033 Acked 8032 Write 4 Done New Done false
20515 7034 Acked 8033 Write 7 Done New Done false
20516 7035 Acked 8034 Flush 0 Done New Done false
20517 7036 Acked 8035 Read 8 Done New Done false
20518 7037 Acked 8036 Flush 0 Done New Done false
20519 7038 Acked 8037 Write 1 Done New Done false
20520 7039 Acked 8038 Read 7 Done New Done false
20521 7040 Acked 8039 Write 6 Done New Done false
20522 7041 Acked 8040 Write 6 Done New Done false
20523 7042 Acked 8041 Flush 0 Done New Done false
20524 7043 Acked 8042 Read 7 Done New Done false
20525 7044 Acked 8043 Write 3 Done New Done false
20526 7045 Acked 8044 Write 9 Done New Done false
20527 7046 Acked 8045 Read 3 Done New Done false
20528 7047 Acked 8046 Write 9 Done New Done false
20529 7048 Acked 8047 Read 2 Done New Done false
20530 7049 Acked 8048 Write 10 Done New Done false
20531 7050 Acked 8049 Read 3 Done New Done false
20532 7051 Acked 8050 Read 10 Done New Done false
20533 7052 Acked 8051 Flush 0 Done New Done false
20534 7053 Acked 8052 Flush 0 Done New Done false
20535 7054 Acked 8053 Write 2 Done New Done false
20536 7055 Acked 8054 Flush 0 Done New Done false
20537 7056 Acked 8055 Read 1 Done New Done false
20538 7057 Acked 8056 Read 3 Done New Done false
20539 7058 Acked 8057 Read 5 Done New Done false
20540 7059 Acked 8058 Read 2 Done New Done false
20541 7060 Acked 8059 Read 5 Done New Done false
20542 7061 Acked 8060 Write 2 Done New Done false
20543 7062 Acked 8061 Write 5 Done New Done false
20544 7063 Acked 8062 Read 3 Done New Done false
20545 7064 Acked 8063 Read 7 Done New Done false
20546 7065 Acked 8064 Read 6 Done New Done false
20547 7066 Acked 8065 Read 10 Done New Done false
20548 7067 Acked 8066 Write 2 Done New Done false
20549 7068 Acked 8067 Read 9 Done New Done false
20550 7069 Acked 8068 Read 9 Done New Done false
20551 7070 Acked 8069 Flush 0 Done New Done false
20552 7071 Acked 8070 Flush 0 Done New Done false
20553 7072 Acked 8071 Flush 0 Done New Done false
20554 7073 Acked 8072 Read 6 Done New Done false
20555 7074 Acked 8073 Write 8 Done New Done false
20556 7075 Acked 8074 Write 6 Done New Done false
20557 7076 Acked 8075 Read 9 Done New Done false
20558 7077 Acked 8076 Read 9 Done New Done false
20559 7078 Acked 8077 Read 7 Done New Done false
20560 7079 Acked 8078 Write 9 Done New Done false
20561 7080 Acked 8079 Read 3 Done New Done false
20562 7081 Acked 8080 Flush 0 Done New Done false
20563 7082 Acked 8081 Read 10 Done New Done false
20564 7083 Acked 8082 Flush 0 Done New Done false
20565 7084 Acked 8083 Read 7 Done New Done false
20566 7085 Acked 8084 Write 4 Done New Done false
20567 7086 Acked 8085 Read 6 Done New Done false
20568 7087 Acked 8086 Write 10 Done New Done false
20569 7088 Acked 8087 Write 5 Done New Done false
20570 7089 Acked 8088 Flush 0 Done New Done false
20571 7090 Acked 8089 Write 10 Done New Done false
20572 7091 Acked 8090 Write 1 Done New Done false
20573 7092 Acked 8091 Write 7 Done New Done false
20574 7093 Acked 8092 Read 9 Done New Done false
20575 7094 Acked 8093 Read 8 Done New Done false
20576 7095 Acked 8094 Read 10 Done New Done false
20577 7096 Acked 8095 Read 1 Done New Done false
20578 7097 Acked 8096 Read 7 Done New Done false
20579 7098 Acked 8097 Write 5 Done New Done false
20580 7099 Acked 8098 Write 7 Done New Done false
20581 7100 Acked 8099 Write 8 Done New Done false
20582 7101 Acked 8100 Flush 0 Done New Done false
20583 7102 Acked 8101 Read 3 Done New Done false
20584 7103 Acked 8102 Write 10 Done New Done false
20585 7104 Acked 8103 Write 10 Done New Done false
20586 7105 Acked 8104 Flush 0 Done New Done false
20587 7106 Acked 8105 Read 9 Done New Done false
20588 7107 Acked 8106 Write 2 Done New Done false
20589 7108 Acked 8107 Flush 0 Done New Done false
20590 7109 Acked 8108 Write 9 Done New Done false
20591 7110 Acked 8109 Read 2 Done New Done false
20592 7111 Acked 8110 Write 2 Done New Done false
20593 7112 Acked 8111 Write 3 Done New Done false
20594 7113 Acked 8112 Flush 0 Done New Done false
20595 7114 Acked 8113 Write 5 Done New Done false
20596 7115 Acked 8114 Read 5 Done New Done false
20597 7116 Acked 8115 Write 7 Done New Done false
20598 7117 Acked 8116 Flush 0 Done New Done false
20599 7118 Acked 8117 Flush 0 Done New Done false
20600 7119 Acked 8118 Flush 0 Done New Done false
20601 7120 Acked 8119 Read 3 Done New Done false
20602 7121 Acked 8120 Read 9 Done New Done false
20603 7122 Acked 8121 Read 5 Done New Done false
20604 7123 Acked 8122 Write 4 Done New Done false
20605 7124 Acked 8123 Read 9 Done New Done false
20606 7125 Acked 8124 Read 10 Done New Done false
20607 7126 Acked 8125 Write 8 Done New Done false
20608 7127 Acked 8126 Write 10 Done New Done false
20609 7128 Acked 8127 Flush 0 Done New Done false
20610 7129 Acked 8128 Write 4 Done New Done false
20611 7130 Acked 8129 Write 8 Done New Done false
20612 7131 Acked 8130 Write 9 Done New Done false
20613 7132 Acked 8131 Read 1 Done New Done false
20614 7133 Acked 8132 Flush 0 Done New Done false
20615 7134 Acked 8133 Flush 0 Done New Done false
20616 7135 Acked 8134 Write 1 Done New Done false
20617 7136 Acked 8135 Read 2 Done New Done false
20618 7137 Acked 8136 Write 5 Done New Done false
20619 7138 Acked 8137 Write 9 Done New Done false
20620 7139 Acked 8138 Write 4 Done New Done false
20621 7140 Acked 8139 Write 10 Done New Done false
20622 7141 Acked 8140 Read 3 Done New Done false
20623 7142 Acked 8141 Read 5 Done New Done false
20624 7143 Acked 8142 Read 6 Done New Done false
20625 7144 Acked 8143 Flush 0 Done New Done false
20626 7145 Acked 8144 Read 3 Done New Done false
20627 7146 Acked 8145 Read 10 Done New Done false
20628 7147 Acked 8146 Read 3 Done New Done false
20629 7148 Acked 8147 Read 1 Done New Done false
20630 7149 Acked 8148 Write 5 Done New Done false
20631 7150 Acked 8149 Write 3 Done New Done false
20632 7151 Acked 8150 Flush 0 Done New Done false
20633 7152 Acked 8151 Read 7 Done New Done false
20634 7153 Acked 8152 Write 6 Done New Done false
20635 7154 Acked 8153 Read 1 Done New Done false
20636 7155 Acked 8154 Write 6 Done New Done false
20637 7156 Acked 8155 Flush 0 Done New Done false
20638 7157 Acked 8156 Flush 0 Done New Done false
20639 7158 Acked 8157 Write 7 Done New Done false
20640 7159 Acked 8158 Write 1 Done New Done false
20641 7160 Acked 8159 Write 6 Done New Done false
20642 7161 Acked 8160 Write 8 Done New Done false
20643 7162 Acked 8161 Write 7 Done New Done false
20644 7163 Acked 8162 Write 6 Done New Done false
20645 7164 Acked 8163 Read 5 Done New Done false
20646 7165 Acked 8164 Write 9 Done New Done false
20647 7166 Acked 8165 Write 5 Done New Done false
20648 7167 Acked 8166 Read 6 Done New Done false
20649 7168 Acked 8167 Read 7 Done New Done false
20650 7169 Acked 8168 Write 5 Done New Done false
20651 7170 Acked 8169 Read 5 Done New Done false
20652 7171 Acked 8170 Read 2 Done New Done false
20653 7172 Acked 8171 Write 4 Done New Done false
20654 7173 Acked 8172 Read 3 Done New Done false
20655 7174 Acked 8173 Read 3 Done New Done false
20656 7175 Acked 8174 Flush 0 Done New Done false
20657 7176 Acked 8175 Write 7 Done New Done false
20658 7177 Acked 8176 Read 4 Done New Done false
20659 7178 Acked 8177 Write 3 Done New Done false
20660 7179 Acked 8178 Write 2 Done New Done false
20661 7180 Acked 8179 Read 3 Done New Done false
20662 7181 Acked 8180 Write 7 Done New Done false
20663 7182 Acked 8181 Flush 0 Done New Done false
20664 7183 Acked 8182 Write 4 Done New Done false
20665 7184 Acked 8183 Read 6 Done New Done false
20666 7185 Acked 8184 Write 6 Done New Done false
20667 7186 Acked 8185 Read 1 Done New Done false
20668 7187 Acked 8186 Write 3 Done New Done false
20669 7188 Acked 8187 Write 7 Done New Done false
20670 7189 Acked 8188 Write 7 Done New Done false
20671 7190 Acked 8189 Write 8 Done New Done false
20672 7191 Acked 8190 Read 3 Done New Done false
20673 7192 Acked 8191 Read 9 Done New Done false
20674 7193 Acked 8192 Write 2 Done New Done false
20675 7194 Acked 8193 Read 10 Done New Done false
20676 7195 Acked 8194 Write 7 Done New Done false
20677 7196 Acked 8195 Read 7 Done New Done false
20678 7197 Acked 8196 Read 3 Done New Done false
20679 7198 Acked 8197 Read 7 Done New Done false
20680 7199 Acked 8198 Write 8 Done New Done false
20681 7200 Acked 8199 Read 3 Done New Done false
20682 7201 Acked 8200 Write 4 Done New Done false
20683 7202 Acked 8201 Read 5 Done New Done false
20684 7203 Acked 8202 Write 7 Done New Done false
20685 7204 Acked 8203 Read 2 Done New Done false
20686 7205 Acked 8204 Read 1 Done New Done false
20687 7206 Acked 8205 Write 5 Done New Done false
20688 7207 Acked 8206 Flush 0 Done New Done false
20689 7208 Acked 8207 Write 2 Done New Done false
20690 7209 Acked 8208 Read 9 Done New Done false
20691 7210 Acked 8209 Read 7 Done New Done false
20692 7211 Acked 8210 Write 9 Done New Done false
20693 7212 Acked 8211 Write 8 Done New Done false
20694 7213 Acked 8212 Write 10 Done New Done false
20695 7214 Acked 8213 Read 10 Done New Done false
20696 7215 Acked 8214 Write 5 Done New Done false
20697 7216 Acked 8215 Write 10 Done New Done false
20698 7217 Acked 8216 Read 9 Done New Done false
20699 7218 Acked 8217 Write 2 Done New Done false
20700 7219 Acked 8218 Flush 0 Done New Done false
20701 7220 Acked 8219 Read 1 Done New Done false
20702 7221 Acked 8220 Write 10 Done New Done false
20703 7222 Acked 8221 Write 2 Done New Done false
20704 7223 Acked 8222 Write 10 Done New Done false
20705 7224 Acked 8223 Write 4 Done New Done false
20706 7225 Acked 8224 Write 5 Done New Done false
20707 7226 Acked 8225 Write 7 Done New Done false
20708 7227 Acked 8226 Flush 0 Done New Done false
20709 7228 Acked 8227 Read 1 Done New Done false
20710 7229 Acked 8228 Write 2 Done New Done false
20711 7230 Acked 8229 Read 3 Done New Done false
20712 7231 Acked 8230 Write 10 Done New Done false
20713 7232 Acked 8231 Read 9 Done New Done false
20714 7233 Acked 8232 Read 3 Done New Done false
20715 7234 Acked 8233 Write 8 Done New Done false
20716 7235 Acked 8234 Read 9 Done New Done false
20717 7236 Acked 8235 Read 4 Done New Done false
20718 7237 Acked 8236 Write 4 Done New Done false
20719 7238 Acked 8237 Read 9 Done New Done false
20720 7239 Acked 8238 Flush 0 Done New Done false
20721 7240 Acked 8239 Read 1 Done New Done false
20722 7241 Acked 8240 Read 8 Done New Done false
20723 7242 Acked 8241 Write 3 Done New Done false
20724 7243 Acked 8242 Read 8 Done New Done false
20725 7244 Acked 8243 Write 7 Done New Done false
20726 7245 Acked 8244 Read 10 Done New Done false
20727 7246 Acked 8245 Write 7 Done New Done false
20728 7247 Acked 8246 Write 9 Done New Done false
20729 7248 Acked 8247 Read 7 Done New Done false
20730 7249 Acked 8248 Read 9 Done New Done false
20731 7250 Acked 8249 Read 7 Done New Done false
20732 7251 Acked 8250 Read 7 Done New Done false
20733 7252 Acked 8251 Write 1 Sent New Sent false
20734 STATES DS:0 DS:1 DS:2 TOTAL
20735 New 0 300 0 300
20736 Sent 1 0 1 2
20737 Done 299 0 299 598
20738 Skipped 0 0 0 0
20739 Error 0 0 0 0
20740 Last Flush: 8238 7951 8238
20741 Downstairs last five completed: 7951 7950 7949 7948 7947 7946
20742 Upstairs last five completed: 7252 7251 7250 7249 7248 7247
20743 CLIENT: Up:0 ds:300 act:2
207442023-10-13T04:46:15.353ZINFOcrucible: [1] connecting to 127.0.0.1:8820 looper = 1
20745 ----------------------------------------------------------------
20746 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
20747 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
20748 6953 Acked 7952 Read 8 Done New Done false
20749 6954 Acked 7953 Read 6 Done New Done false
20750 6955 Acked 7954 Write 8 Done New Done false
20751 6956 Acked 7955 Write 5 Done New Done false
20752 6957 Acked 7956 Read 8 Done New Done false
20753 6958 Acked 7957 Read 10 Done New Done false
20754 6959 Acked 7958 Read 3 Done New Done false
20755 6960 Acked 7959 Write 10 Done New Done false
20756 6961 Acked 7960 Read 9 Done New Done false
20757 6962 Acked 7961 Read 7 Done New Done false
20758 6963 Acked 7962 Read 9 Done New Done false
20759 6964 Acked 7963 Read 5 Done New Done false
20760 6965 Acked 7964 Read 2 Done New Done false
20761 6966 Acked 7965 Write 2 Done New Done false
20762 6967 Acked 7966 Read 2 Done New Done false
20763 6968 Acked 7967 Read 4 Done New Done false
20764 6969 Acked 7968 Write 7 Done New Done false
20765 6970 Acked 7969 Write 9 Done New Done false
20766 6971 Acked 7970 Read 4 Done New Done false
20767 6972 Acked 7971 Write 6 Done New Done false
20768 6973 Acked 7972 Write 3 Done New Done false
20769 6974 Acked 7973 Flush 0 Done New Done false
20770 6975 Acked 7974 Write 5 Done New Done false
20771 6976 Acked 7975 Read 1 Done New Done false
20772 6977 Acked 7976 Write 1 Done New Done false
20773 6978 Acked 7977 Read 3 Done New Done false
20774 6979 Acked 7978 Read 6 Done New Done false
20775 6980 Acked 7979 Flush 0 Done New Done false
20776 6981 Acked 7980 Flush 0 Done New Done false
20777 6982 Acked 7981 Flush 0 Done New Done false
20778 6983 Acked 7982 Read 10 Done New Done false
20779 6984 Acked 7983 Read 2 Done New Done false
20780 6985 Acked 7984 Write 5 Done New Done false
20781 6986 Acked 7985 Write 9 Done New Done false
20782 6987 Acked 7986 Read 10 Done New Done false
20783 6988 Acked 7987 Write 1 Done New Done false
20784 6989 Acked 7988 Read 3 Done New Done false
20785 6990 Acked 7989 Write 4 Done New Done false
20786 6991 Acked 7990 Read 3 Done New Done false
20787 6992 Acked 7991 Write 5 Done New Done false
20788 6993 Acked 7992 Read 2 Done New Done false
20789 6994 Acked 7993 Read 7 Done New Done false
20790 6995 Acked 7994 Write 6 Done New Done false
20791 6996 Acked 7995 Write 6 Done New Done false
20792 6997 Acked 7996 Write 10 Done New Done false
20793 6998 Acked 7997 Read 7 Done New Done false
20794 6999 Acked 7998 Read 3 Done New Done false
20795 7000 Acked 7999 Write 7 Done New Done false
20796 7001 Acked 8000 Write 10 Done New Done false
20797 7002 Acked 8001 Write 5 Done New Done false
20798 7003 Acked 8002 Read 10 Done New Done false
20799 7004 Acked 8003 Read 9 Done New Done false
20800 7005 Acked 8004 Read 8 Done New Done false
20801 7006 Acked 8005 Read 8 Done New Done false
20802 7007 Acked 8006 Write 3 Done New Done false
20803 7008 Acked 8007 Read 2 Done New Done false
20804 7009 Acked 8008 Read 5 Done New Done false
20805 7010 Acked 8009 Read 6 Done New Done false
20806 7011 Acked 8010 Read 6 Done New Done false
20807 7012 Acked 8011 Write 4 Done New Done false
20808 7013 Acked 8012 Write 5 Done New Done false
20809 7014 Acked 8013 Read 1 Done New Done false
20810 7015 Acked 8014 Read 7 Done New Done false
20811 7016 Acked 8015 Read 3 Done New Done false
20812 7017 Acked 8016 Read 6 Done New Done false
20813 7018 Acked 8017 Read 10 Done New Done false
20814 7019 Acked 8018 Write 2 Done New Done false
20815 7020 Acked 8019 Write 2 Done New Done false
20816 7021 Acked 8020 Read 6 Done New Done false
20817 7022 Acked 8021 Read 6 Done New Done false
20818 7023 Acked 8022 Read 4 Done New Done false
20819 7024 Acked 8023 Write 5 Done New Done false
20820 7025 Acked 8024 Flush 0 Done New Done false
20821 7026 Acked 8025 Read 5 Done New Done false
20822 7027 Acked 8026 Write 2 Done New Done false
20823 7028 Acked 8027 Write 7 Done New Done false
20824 7029 Acked 8028 Read 4 Done New Done false
20825 7030 Acked 8029 Read 4 Done New Done false
20826 7031 Acked 8030 Write 9 Done New Done false
20827 7032 Acked 8031 Read 7 Done New Done false
20828 7033 Acked 8032 Write 4 Done New Done false
20829 7034 Acked 8033 Write 7 Done New Done false
20830 7035 Acked 8034 Flush 0 Done New Done false
20831 7036 Acked 8035 Read 8 Done New Done false
20832 7037 Acked 8036 Flush 0 Done New Done false
20833 7038 Acked 8037 Write 1 Done New Done false
20834 7039 Acked 8038 Read 7 Done New Done false
20835 7040 Acked 8039 Write 6 Done New Done false
20836 7041 Acked 8040 Write 6 Done New Done false
20837 7042 Acked 8041 Flush 0 Done New Done false
20838 7043 Acked 8042 Read 7 Done New Done false
20839 7044 Acked 8043 Write 3 Done New Done false
20840 7045 Acked 8044 Write 9 Done New Done false
20841 7046 Acked 8045 Read 3 Done New Done false
20842 7047 Acked 8046 Write 9 Done New Done false
20843 7048 Acked 8047 Read 2 Done New Done false
20844 7049 Acked 8048 Write 10 Done New Done false
20845 7050 Acked 8049 Read 3 Done New Done false
20846 7051 Acked 8050 Read 10 Done New Done false
20847 7052 Acked 8051 Flush 0 Done New Done false
20848 7053 Acked 8052 Flush 0 Done New Done false
20849 7054 Acked 8053 Write 2 Done New Done false
20850 7055 Acked 8054 Flush 0 Done New Done false
20851 7056 Acked 8055 Read 1 Done New Done false
20852 7057 Acked 8056 Read 3 Done New Done false
20853 7058 Acked 8057 Read 5 Done New Done false
20854 7059 Acked 8058 Read 2 Done New Done false
20855 7060 Acked 8059 Read 5 Done New Done false
20856 7061 Acked 8060 Write 2 Done New Done false
20857 7062 Acked 8061 Write 5 Done New Done false
20858 7063 Acked 8062 Read 3 Done New Done false
20859 7064 Acked 8063 Read 7 Done New Done false
20860 7065 Acked 8064 Read 6 Done New Done false
20861 7066 Acked 8065 Read 10 Done New Done false
20862 7067 Acked 8066 Write 2 Done New Done false
20863 7068 Acked 8067 Read 9 Done New Done false
20864 7069 Acked 8068 Read 9 Done New Done false
20865 7070 Acked 8069 Flush 0 Done New Done false
20866 7071 Acked 8070 Flush 0 Done New Done false
20867 7072 Acked 8071 Flush 0 Done New Done false
20868 7073 Acked 8072 Read 6 Done New Done false
20869 7074 Acked 8073 Write 8 Done New Done false
20870 7075 Acked 8074 Write 6 Done New Done false
20871 7076 Acked 8075 Read 9 Done New Done false
20872 7077 Acked 8076 Read 9 Done New Done false
20873 7078 Acked 8077 Read 7 Done New Done false
20874 7079 Acked 8078 Write 9 Done New Done false
20875 7080 Acked 8079 Read 3 Done New Done false
20876 7081 Acked 8080 Flush 0 Done New Done false
20877 7082 Acked 8081 Read 10 Done New Done false
20878 7083 Acked 8082 Flush 0 Done New Done false
20879 7084 Acked 8083 Read 7 Done New Done false
20880 7085 Acked 8084 Write 4 Done New Done false
20881 7086 Acked 8085 Read 6 Done New Done false
20882 7087 Acked 8086 Write 10 Done New Done false
20883 7088 Acked 8087 Write 5 Done New Done false
20884 7089 Acked 8088 Flush 0 Done New Done false
20885 7090 Acked 8089 Write 10 Done New Done false
20886 7091 Acked 8090 Write 1 Done New Done false
20887 7092 Acked 8091 Write 7 Done New Done false
20888 7093 Acked 8092 Read 9 Done New Done false
20889 7094 Acked 8093 Read 8 Done New Done false
20890 7095 Acked 8094 Read 10 Done New Done false
20891 7096 Acked 8095 Read 1 Done New Done false
20892 7097 Acked 8096 Read 7 Done New Done false
20893 7098 Acked 8097 Write 5 Done New Done false
20894 7099 Acked 8098 Write 7 Done New Done false
20895 7100 Acked 8099 Write 8 Done New Done false
20896 7101 Acked 8100 Flush 0 Done New Done false
20897 7102 Acked 8101 Read 3 Done New Done false
20898 7103 Acked 8102 Write 10 Done New Done false
20899 7104 Acked 8103 Write 10 Done New Done false
20900 7105 Acked 8104 Flush 0 Done New Done false
20901 7106 Acked 8105 Read 9 Done New Done false
20902 7107 Acked 8106 Write 2 Done New Done false
20903 7108 Acked 8107 Flush 0 Done New Done false
20904 7109 Acked 8108 Write 9 Done New Done false
20905 7110 Acked 8109 Read 2 Done New Done false
20906 7111 Acked 8110 Write 2 Done New Done false
20907 7112 Acked 8111 Write 3 Done New Done false
20908 7113 Acked 8112 Flush 0 Done New Done false
20909 7114 Acked 8113 Write 5 Done New Done false
20910 7115 Acked 8114 Read 5 Done New Done false
20911 7116 Acked 8115 Write 7 Done New Done false
20912 7117 Acked 8116 Flush 0 Done New Done false
20913 7118 Acked 8117 Flush 0 Done New Done false
20914 7119 Acked 8118 Flush 0 Done New Done false
20915 7120 Acked 8119 Read 3 Done New Done false
20916 7121 Acked 8120 Read 9 Done New Done false
20917 7122 Acked 8121 Read 5 Done New Done false
20918 7123 Acked 8122 Write 4 Done New Done false
20919 7124 Acked 8123 Read 9 Done New Done false
20920 7125 Acked 8124 Read 10 Done New Done false
20921 7126 Acked 8125 Write 8 Done New Done false
20922 7127 Acked 8126 Write 10 Done New Done false
20923 7128 Acked 8127 Flush 0 Done New Done false
20924 7129 Acked 8128 Write 4 Done New Done false
20925 7130 Acked 8129 Write 8 Done New Done false
20926 7131 Acked 8130 Write 9 Done New Done false
20927 7132 Acked 8131 Read 1 Done New Done false
20928 7133 Acked 8132 Flush 0 Done New Done false
20929 7134 Acked 8133 Flush 0 Done New Done false
20930 7135 Acked 8134 Write 1 Done New Done false
20931 7136 Acked 8135 Read 2 Done New Done false
20932 7137 Acked 8136 Write 5 Done New Done false
20933 7138 Acked 8137 Write 9 Done New Done false
20934 7139 Acked 8138 Write 4 Done New Done false
20935 7140 Acked 8139 Write 10 Done New Done false
20936 7141 Acked 8140 Read 3 Done New Done false
20937 7142 Acked 8141 Read 5 Done New Done false
20938 7143 Acked 8142 Read 6 Done New Done false
20939 7144 Acked 8143 Flush 0 Done New Done false
20940 7145 Acked 8144 Read 3 Done New Done false
20941 7146 Acked 8145 Read 10 Done New Done false
20942 7147 Acked 8146 Read 3 Done New Done false
20943 7148 Acked 8147 Read 1 Done New Done false
20944 7149 Acked 8148 Write 5 Done New Done false
20945 7150 Acked 8149 Write 3 Done New Done false
20946 7151 Acked 8150 Flush 0 Done New Done false
20947 7152 Acked 8151 Read 7 Done New Done false
20948 7153 Acked 8152 Write 6 Done New Done false
20949 7154 Acked 8153 Read 1 Done New Done false
20950 7155 Acked 8154 Write 6 Done New Done false
20951 7156 Acked 8155 Flush 0 Done New Done false
20952 7157 Acked 8156 Flush 0 Done New Done false
20953 7158 Acked 8157 Write 7 Done New Done false
20954 7159 Acked 8158 Write 1 Done New Done false
20955 7160 Acked 8159 Write 6 Done New Done false
20956 7161 Acked 8160 Write 8 Done New Done false
20957 7162 Acked 8161 Write 7 Done New Done false
20958 7163 Acked 8162 Write 6 Done New Done false
20959 7164 Acked 8163 Read 5 Done New Done false
20960 7165 Acked 8164 Write 9 Done New Done false
20961 7166 Acked 8165 Write 5 Done New Done false
20962 7167 Acked 8166 Read 6 Done New Done false
20963 7168 Acked 8167 Read 7 Done New Done false
20964 7169 Acked 8168 Write 5 Done New Done false
20965 7170 Acked 8169 Read 5 Done New Done false
20966 7171 Acked 8170 Read 2 Done New Done false
20967 7172 Acked 8171 Write 4 Done New Done false
20968 7173 Acked 8172 Read 3 Done New Done false
20969 7174 Acked 8173 Read 3 Done New Done false
20970 7175 Acked 8174 Flush 0 Done New Done false
20971 7176 Acked 8175 Write 7 Done New Done false
20972 7177 Acked 8176 Read 4 Done New Done false
20973 7178 Acked 8177 Write 3 Done New Done false
20974 7179 Acked 8178 Write 2 Done New Done false
20975 7180 Acked 8179 Read 3 Done New Done false
20976 7181 Acked 8180 Write 7 Done New Done false
20977 7182 Acked 8181 Flush 0 Done New Done false
20978 7183 Acked 8182 Write 4 Done New Done false
20979 7184 Acked 8183 Read 6 Done New Done false
20980 7185 Acked 8184 Write 6 Done New Done false
20981 7186 Acked 8185 Read 1 Done New Done false
20982 7187 Acked 8186 Write 3 Done New Done false
20983 7188 Acked 8187 Write 7 Done New Done false
20984 7189 Acked 8188 Write 7 Done New Done false
20985 7190 Acked 8189 Write 8 Done New Done false
20986 7191 Acked 8190 Read 3 Done New Done false
20987 7192 Acked 8191 Read 9 Done New Done false
20988 7193 Acked 8192 Write 2 Done New Done false
20989 7194 Acked 8193 Read 10 Done New Done false
20990 7195 Acked 8194 Write 7 Done New Done false
20991 7196 Acked 8195 Read 7 Done New Done false
20992 7197 Acked 8196 Read 3 Done New Done false
20993 7198 Acked 8197 Read 7 Done New Done false
20994 7199 Acked 8198 Write 8 Done New Done false
20995 7200 Acked 8199 Read 3 Done New Done false
20996 7201 Acked 8200 Write 4 Done New Done false
20997 7202 Acked 8201 Read 5 Done New Done false
20998 7203 Acked 8202 Write 7 Done New Done false
20999 7204 Acked 8203 Read 2 Done New Done false
21000 7205 Acked 8204 Read 1 Done New Done false
21001 7206 Acked 8205 Write 5 Done New Done false
21002 7207 Acked 8206 Flush 0 Done New Done false
21003 7208 Acked 8207 Write 2 Done New Done false
21004 7209 Acked 8208 Read 9 Done New Done false
21005 7210 Acked 8209 Read 7 Done New Done false
21006 7211 Acked 8210 Write 9 Done New Done false
21007 7212 Acked 8211 Write 8 Done New Done false
21008 7213 Acked 8212 Write 10 Done New Done false
21009 7214 Acked 8213 Read 10 Done New Done false
21010 7215 Acked 8214 Write 5 Done New Done false
21011 7216 Acked 8215 Write 10 Done New Done false
21012 7217 Acked 8216 Read 9 Done New Done false
21013 7218 Acked 8217 Write 2 Done New Done false
21014 7219 Acked 8218 Flush 0 Done New Done false
21015 7220 Acked 8219 Read 1 Done New Done false
21016 7221 Acked 8220 Write 10 Done New Done false
21017 7222 Acked 8221 Write 2 Done New Done false
21018 7223 Acked 8222 Write 10 Done New Done false
21019 7224 Acked 8223 Write 4 Done New Done false
21020 7225 Acked 8224 Write 5 Done New Done false
21021 7226 Acked 8225 Write 7 Done New Done false
21022 7227 Acked 8226 Flush 0 Done New Done false
21023 7228 Acked 8227 Read 1 Done New Done false
21024 7229 Acked 8228 Write 2 Done New Done false
21025 7230 Acked 8229 Read 3 Done New Done false
21026 7231 Acked 8230 Write 10 Done New Done false
21027 7232 Acked 8231 Read 9 Done New Done false
21028 7233 Acked 8232 Read 3 Done New Done false
21029 7234 Acked 8233 Write 8 Done New Done false
21030 7235 Acked 8234 Read 9 Done New Done false
21031 7236 Acked 8235 Read 4 Done New Done false
21032 7237 Acked 8236 Write 4 Done New Done false
21033 7238 Acked 8237 Read 9 Done New Done false
21034 7239 Acked 8238 Flush 0 Done New Done false
21035 7240 Acked 8239 Read 1 Done New Done false
21036 7241 Acked 8240 Read 8 Done New Done false
21037 7242 Acked 8241 Write 3 Done New Done false
21038 7243 Acked 8242 Read 8 Done New Done false
21039 7244 Acked 8243 Write 7 Done New Done false
21040 7245 Acked 8244 Read 10 Done New Done false
21041 7246 Acked 8245 Write 7 Done New Done false
21042 7247 Acked 8246 Write 9 Done New Done false
21043 7248 Acked 8247 Read 7 Done New Done false
21044 7249 Acked 8248 Read 9 Done New Done false
21045 7250 Acked 8249 Read 7 Done New Done false
21046 7251 Acked 8250 Read 7 Done New Done false
21047 7252 Acked 8251 Write 1 Done New Done false
21048 7253 Acked 8252 Flush 0 Done New Done false
21049 STATES DS:0 DS:1 DS:2 TOTAL
21050 New 0 301 0 301
21051 Sent 0 0 0 0
21052 Done 301 0 301 602
21053 Skipped 0 0 0 0
21054 Error 0 0 0 0
21055 Last Flush: 8252 7951 8252
21056 Downstairs last five completed: 7951 7950 7949 7948 7947 7946
21057 Upstairs last five completed: 7253 7252 7251 7250 7249 7248
21058 CLIENT: Up:0 ds:301 act:2
21059 [8820] got start action
21060 [8820] I am going to restart
21061 Starting downstairs at port 8820
21062 Make output file at "/tmp/dsc/downstairs-8820.txt"
21063 [8820][1] reports Starting
21064 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1200)
21065 [8820][1] reports Running
210662023-10-13T04:46:21.360ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 1
210672023-10-13T04:46:21.360ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state Offline
210682023-10-13T04:46:21.360ZINFOcrucible: [1] client re-new 301 jobs since flush 7951 = downstairs
210692023-10-13T04:46:21.361ZINFOcrucible: [1] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
210702023-10-13T04:46:21.361ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
210712023-10-13T04:46:21.361ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
210722023-10-13T04:46:21.361ZINFOcrucible: Returning client:1 UUID:12345678-0000-0000-0000-000000008820 matches
210732023-10-13T04:46:21.361ZINFOcrucible: [1] send last flush ID to this DS: 7951
210742023-10-13T04:46:21.361ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
210752023-10-13T04:46:21.361ZINFOcrucible: [1] Replied this last flush ID: 7951
210762023-10-13T04:46:21.361ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Offline Active ds_transition to Replay
210772023-10-13T04:46:21.362ZINFOcrucible: [1] Transition from Offline to Replay
210782023-10-13T04:46:21.362ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
210792023-10-13T04:46:21.362ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Replay Active ds_transition to Active
210802023-10-13T04:46:21.362ZINFOcrucible: [1] Transition from Replay to Active
210812023-10-13T04:46:21.362ZINFOcrucible: [1] Starts cmd_loop
210822023-10-13T04:46:21.362ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
210832023-10-13T04:46:21.362ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
210842023-10-13T04:46:21.362ZINFOcrucible: Set check for repair
210852023-10-13T04:46:22.363ZINFOcrucible: Checking if live repair is needed
210862023-10-13T04:46:22.363ZINFOcrucible: No Live Repair required at this time
210872023-10-13T04:46:22.363ZWARNcrucible: [1] flow control end
21088 ----------------------------------------------------------------
21089 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
21090 Downstairs last five completed: 8252 8251 8250 8249 8248 8247
21091 Upstairs last five completed: 7253 7252 7251 7250 7249 7248
21092 CLIENT: Up:0 ds:0 act:3
21093 Replay: All jobs finished, all DS active.
21094 Main task has work to do, go find it
21095 got dsc Stop(2)
21096 stop 2
21097 Oct 13 04:46:23.270 INFO request completed, latency_us: 468, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 99fe20f7-ab05-4d87-b510-2eee84e735c9, remote_addr: [8830] Got stop action so:false kr:false
21098 127.0.0.1:52046, local_addr: 127.0.0.1:9998
21099 Oct 13 04:46:23.271 INFO request completed, latency_us: 168, response_code: 200, uri: /state/cid/2, method: GET, req_id: c79840e6-c222-443a-912d-935dc5a389f1, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
211002023-10-13T04:46:23.271ZWARNcrucible: [2] downstairs disconnected
211012023-10-13T04:46:23.271ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
211022023-10-13T04:46:23.271ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8830 closed looper = 2
211032023-10-13T04:46:23.271ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
211042023-10-13T04:46:23.271ZWARNcrucible: [2] pm_task rx.recv() is None
211052023-10-13T04:46:23.271ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
211062023-10-13T04:46:23.271ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
21107 [8830] Exited with: ExitStatus(unix_wait_status(9))
21108 [8830][2] reports Exit
21109 Oct 13 04:46:27.273 INFO request completed, latency_us: 248, response_code: 200, uri: /state/cid/2, method: GET, req_id: 805e5aa9-42cf-4e90-a770-1e6fe7e77b3b, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
21110 001/300 Flush
21111 002/300 Write block 4439 len 24576 data: 7 7 9 10 7 6
21112 003/300 Write block 3482 len 24576 data: 6 6 6 5 7 6
21113 004/300 Write block 1039 len 32768 data: 6 5 5 4 4 3 5 5
21114 005/300 Write block 2441 len 32768 data: 4 4 4 4 5 6 5 5
21115 006/300 Write block 469 len 36864 data: 3 4 4 4 5 5 6 7 8
21116 007/300 Flush
21117 008/300 Read block 2179 len 12288
21118 009/300 Read block 1484 len 40960
21119 010/300 Write block 2358 len 16384 data: 5 4 4 4
21120 011/300 Flush
21121 012/300 Write block 1621 len 24576 data: 5 5 4 5 5 5
21122 013/300 Write block 4716 len 40960 data: 5 5 6 6 4 4 6 7 7 8
21123 014/300 Write block 3912 len 32768 data: 5 4 5 5 5 5 3 3
21124 015/300 Write block 3165 len 28672 data: 4 5 7 7 7 8 7
21125 016/300 Read block 2652 len 36864
21126 017/300 Flush
21127 018/300 Read block 2676 len 28672
21128 019/300 Write block 3845 len 40960 data: 5 6 6 6 7 6 5 8 7 7
21129 020/300 Flush
21130 021/300 Read block 3450 len 40960
21131 022/300 Read block 584 len 24576
21132 023/300 Write block 974 len 8192 data: 8 9
21133 024/300 Read block 3959 len 8192
21134 025/300 Write block 3759 len 20480 data: 5 4 6 7 7
21135 026/300 Write block 515 len 36864 data: 7 6 8 8 8 10 11 11 11
21136 027/300 Read block 2198 len 28672
21137 028/300 Flush
21138 029/300 Read block 1963 len 24576
21139 030/300 Read block 1659 len 12288
21140 031/300 Write block 4160 len 32768 data: 8 7 8 9 7 6 6 8
21141 032/300 Read block 4219 len 36864
21142 033/300 Read block 3339 len 24576
21143 034/300 Read block 4380 len 32768
21144 035/300 Write block 238 len 40960 data: 5 5 5 5 5 4 3 3 3 3
21145 036/300 Read block 138 len 40960
21146 037/300 Write block 1613 len 12288 data: 5 4 4
21147 038/300 Write block 2952 len 20480 data: 7 5 4 4 5
21148 039/300 Read block 880 len 12288
21149 040/300 Write block 4078 len 40960 data: 4 4 5 6 6 5 6 6 7 6
21150 041/300 Write block 3393 len 16384 data: 7 7 7 9
21151 042/300 Write block 3672 len 16384 data: 6 8 6 6
21152 043/300 Write block 2501 len 28672 data: 4 4 4 5 4 5 8
21153 044/300 Write block 3009 len 24576 data: 4 3 2 2 2 3
21154 045/300 Flush
21155 046/300 Write block 1395 len 24576 data: 5 5 6 6 7 6
21156 047/300 Read block 4409 len 12288
21157 048/300 Flush
21158 049/300 Read block 1576 len 24576
21159 050/300 Write block 4940 len 28672 data: 10 10 9 8 7 7 6
21160 051/300 Read block 3074 len 20480
21161 052/300 Write block 1131 len 28672 data: 6 7 6 5 5 3 4
21162 053/300 Flush
21163 054/300 Read block 458 len 12288
21164 055/300 Read block 4984 len 32768
21165 056/300 Read block 1877 len 24576
21166 057/300 Read block 2922 len 36864
21167 058/300 Write block 4830 len 12288 data: 5 6 5
21168 059/300 Read block 1303 len 24576
21169 060/300 Read block 2282 len 28672
21170 061/300 Read block 4203 len 12288
21171 062/300 Flush
21172 063/300 Write block 1038 len 4096 data: 7
21173 064/300 Read block 1291 len 40960
21174 065/300 Read block 4341 len 28672
21175 066/300 Write block 4019 len 28672 data: 8 7 8 8 8 9 8
21176 067/300 Flush
21177 068/300 Read block 2407 len 16384
21178 069/300 Read block 1547 len 36864
21179 070/300 Read block 78 len 32768
21180 071/300 Read block 1014 len 24576
21181 072/300 Write block 1877 len 12288 data: 5 4 2
21182 073/300 Read block 325 len 16384
21183 074/300 Write block 2460 len 12288 data: 4 4 4
21184 075/300 Read block 2694 len 36864
21185 076/300 Read block 2950 len 32768
21186 077/300 Read block 2860 len 8192
21187 078/300 Write block 3551 len 16384 data: 5 8 10 7
21188 079/300 Write block 395 len 16384 data: 5 6 6 7
21189 080/300 Read block 1373 len 36864
21190 081/300 Read block 714 len 24576
21191 082/300 Write block 3370 len 16384 data: 6 5 6 6
21192 083/300 Read block 4870 len 16384
21193 084/300 Read block 974 len 28672
21194 085/300 Read block 1748 len 20480
21195 086/300 Flush
21196 087/300 Write block 1942 len 4096 data: 6
21197 088/300 Read block 265 len 40960
21198 089/300 Write block 2223 len 8192 data: 7 8
21199 090/300 Flush
21200 091/300 Write block 3110 len 40960 data: 4 3 4 5 6 7 7 6 6 6
21201 092/300 Write block 3274 len 32768 data: 4 3 4 5 5 5 4 4
21202 093/300 Write block 4289 len 40960 data: 8 7 5 5 5 7 9 10 11 8
21203 094/300 Write block 1942 len 4096 data: 7
21204 095/300 Read block 3765 len 16384
21205 096/300 Flush
21206 097/300 Write block 3342 len 40960 data: 4 3 4 6 5 4 4 5 6 7
21207 098/300 Read block 274 len 40960
21208 099/300 Flush
21209 100/300 Write block 3641 len 8192 data: 5 5
21210 101/300 Read block 4653 len 24576
21211 102/300 Read block 4403 len 12288
21212 103/300 Write block 974 len 12288 data: 9 10 9
21213 104/300 Write block 1201 len 28672 data: 5 5 5 6 6 5 5
21214 105/300 Write block 2448 len 40960 data: 6 4 4 4 4 3 3 5 5 5
21215 106/300 Write block 4471 len 28672 data: 3 3 3 3 3 3 4
21216 107/300 Read block 3893 len 40960
21217 108/300 Read block 2164 len 40960
21218 109/300 Read block 539 len 16384
21219 110/300 Read block 4621 len 8192
21220 111/300 Read block 2212 len 40960
21221 112/300 Write block 2191 len 36864 data: 5 5 5 5 6 6 5 4 5
21222 113/300 Flush
21223 114/300 Read block 2332 len 32768
21224 115/300 Read block 1992 len 36864
21225 116/300 Write block 347 len 16384 data: 4 5 5 5
21226 117/300 Read block 2166 len 28672
21227 118/300 Write block 482 len 28672 data: 4 3 3 3 4 7 7
21228 119/300 Write block 3011 len 36864 data: 3 3 3 4 3 3 4 4 5
21229 120/300 Read block 4673 len 20480
21230 121/300 Read block 1163 len 12288
21231 122/300 Write block 2683 len 40960 data: 4 4 4 5 4 4 4 4 5 5
21232 123/300 Write block 638 len 24576 data: 7 6 5 6 7 6
21233 124/300 Write block 1017 len 28672 data: 7 7 6 4 5 5 6
21234 125/300 Write block 2014 len 28672 data: 4 4 5 6 4 3 3
21235 126/300 Write block 4469 len 20480 data: 5 5 4 4 4
21236 127/300 Read block 857 len 28672
21237 128/300 Flush
21238 129/300 Write block 2479 len 28672 data: 4 5 4 4 5 5 5
21239 130/300 Read block 2009 len 36864
21240 131/300 Read block 37 len 28672
21241 132/300 Write block 3339 len 40960 data: 4 4 4 5 4 5 7 6 5 5
21242 133/300 Read block 3345 len 32768
21243 134/300 Write block 3432 len 28672 data: 7 6 4 4 4 4 3
21244 135/300 Write block 2813 len 12288 data: 6 6 6
21245 136/300 Write block 2225 len 8192 data: 7 5
21246 137/300 Read block 4055 len 36864
21247 138/300 Write block 4504 len 32768 data: 4 4 4 4 3 2 2 2
21248 139/300 Read block 1312 len 36864
21249 140/300 Read block 3197 len 4096
21250 141/300 Write block 4209 len 32768 data: 6 7 8 7 5 6 6 7
21251 142/300 Read block 920 len 24576
21252 143/300 Read block 1788 len 28672
21253 144/300 Write block 3061 len 4096 data: 7
21254 145/300 Read block 4516 len 36864
21255 146/300 Write block 3 len 8192 data: 4 4
21256 147/300 Write block 3470 len 4096 data: 4
21257 148/300 Write block 1695 len 20480 data: 8 7 8 8 10
21258 149/300 Write block 4950 len 32768 data: 4 5 3 4 3 3 3 3
21259 150/300 Write block 1732 len 4096 data: 3
21260 151/300 Read block 228 len 8192
21261 152/300 Read block 519 len 32768
21262 153/300 Read block 1179 len 12288
21263 154/300 Read block 3147 len 12288
21264 155/300 Write block 2026 len 40960 data: 4 4 4 4 6 6 6 6 5 6
21265 156/300 Write block 2997 len 28672 data: 3 2 2 3 3 4 4
21266 157/300 Read block 1368 len 4096
21267 158/300 Write block 2304 len 36864 data: 3 3 3 3 4 4 4 4 3
21268 159/300 Write block 4066 len 16384 data: 4 5 6 7
21269 160/300 Flush
21270 161/300 Read block 774 len 28672
21271 162/300 Read block 63 len 12288
21272 163/300 Read block 222 len 28672
21273 164/300 Read block 1061 len 20480
21274 165/300 Flush
21275 166/300 Flush
21276 167/300 Read block 1822 len 16384
21277 168/300 Read block 3575 len 28672
21278 169/300 Read block 1058 len 28672
21279 170/300 Read block 4437 len 8192
21280 171/300 Flush
21281 172/300 Read block 3894 len 24576
21282 173/300 Write block 3123 len 36864 data: 6 5 5 4 6 5 5 5 5
21283 174/300 Read block 2492 len 4096
21284 175/300 Read block 2598 len 20480
21285 176/300 Write block 2483 len 4096 data: 6
21286 177/300 Write block 4850 len 40960 data: 3 4 4 3 2 2 2 2 3 4
21287 178/300 Flush
21288 179/300 Read block 1759 len 40960
21289 180/300 Write block 3586 len 12288 data: 7 7 7
21290 181/300 Write block 1265 len 20480 data: 4 5 5 6 6
21291 182/300 Read block 1497 len 8192
21292 183/300 Read block 2522 len 24576
21293 184/300 Read block 2384 len 32768
21294 185/300 Read block 3074 len 32768
21295 186/300 Read block 3886 len 20480
21296 187/300 Read block 4294 len 8192
21297 188/300 Read block 2538 len 4096
21298 189/300 Read block 3780 len 8192
21299 190/300 Read block 4249 len 8192
21300 191/300 Write block 705 len 8192 data: 4 5
21301 192/300 Write block 277 len 8192 data: 5 5
21302 193/300 Write block 1303 len 40960 data: 3 4 5 6 5 5 5 5 5 6
21303 194/300 Read block 4241 len 4096
21304 195/300 Write block 866 len 12288 data: 7 7 7
21305 196/300 Read block 2599 len 16384
21306 197/300 Read block 1827 len 40960
21307 198/300 Read block 3880 len 40960
21308 199/300 Flush
21309 200/300 Write block 3794 len 36864 data: 4 3 3 3 3 4 6 5 8
21310 201/300 Write block 1702 len 40960 data: 5 3 3 4 5 5 6 7 7 7
21311 202/300 Flush
21312 203/300 Write block 2520 len 8192 data: 3 3
21313 204/300 Read block 4183 len 32768
21314 205/300 Read block 2396 len 24576
21315 206/300 Write block 4367 len 4096 data: 4
21316 207/300 Write block 2866 len 4096 data: 7
21317 208/300 Read block 1553 len 40960
21318 209/300 Read block 3164 len 32768
21319 210/300 Write block 4488 len 32768 data: 5 5 4 4 5 4 4 5
21320 211/300 Write block 3400 len 4096 data: 3
21321 212/300 Read block 3493 len 4096
21322 213/300 Write block 4312 len 4096 data: 5
21323 214/300 Write block 4463 len 24576 data: 5 4 3 3 4 5
21324 215/300 Read block 2659 len 28672
21325 216/300 Read block 979 len 28672
21326 217/300 Read block 902 len 40960
21327 218/300 Read block 3104 len 28672
21328 219/300 Read block 2292 len 36864
21329 220/300 Read block 1128 len 16384
21330 221/300 Read block 3030 len 40960
21331 222/300 Write block 718 len 12288 data: 6 5 6
21332 223/300 Read block 76 len 36864
21333 224/300 Write block 2135 len 4096 data: 4
21334 225/300 Read block 4259 len 32768
21335 226/300 Write block 4714 len 24576 data: 6 6 6 6 7 7
21336 227/300 Write block 3172 len 36864 data: 7 4 5 5 6 6 5 5 4
21337 228/300 Write block 66 len 16384 data: 6 6 4 3
21338 229/300 Write block 193 len 28672 data: 6 6 6 8 7 7 4
21339 230/300 Write block 2965 len 20480 data: 6 6 8 9 8
21340 231/300 Read block 2692 len 32768
21341 232/300 Write block 3298 len 8192 data: 6 5
21342 233/300 Write block 732 len 28672 data: 4 3 2 4 5 4 5
21343 234/300 Write block 2468 len 24576 data: 4 6 6 7 6 4
21344 235/300 Write block 4857 len 16384 data: 3 4 5 8
21345 236/300 Read block 3652 len 36864
21346 237/300 Write block 2591 len 4096 data: 2
21347 238/300 Read block 4982 len 28672
21348 239/300 Read block 3185 len 40960
21349 240/300 Write block 2756 len 32768 data: 6 7 7 5 5 6 6 6
21350 241/300 Read block 3701 len 32768
21351 242/300 Read block 3558 len 12288
21352 243/300 Read block 2372 len 4096
21353 244/300 Flush
21354 245/300 Read block 1021 len 36864
21355 246/300 Read block 4469 len 4096
21356 247/300 Read block 4279 len 36864
21357 248/300 Read block 4154 len 12288
21358 249/300 Write block 3284 len 4096 data: 6
21359 250/300 Read block 3495 len 32768
21360 251/300 Read block 1528 len 32768
21361 252/300 Read block 2656 len 4096
21362 253/300 Read block 1420 len 16384
21363 254/300 Read block 3725 len 40960
21364 255/300 Read block 3197 len 36864
21365 256/300 Flush
21366 257/300 Read block 2872 len 12288
21367 258/300 Read block 1741 len 24576
21368 259/300 Read block 4411 len 8192
21369 260/300 Read block 858 len 8192
21370 261/300 Write block 2501 len 12288 data: 5 5 5
21371 262/300 Read block 4349 len 12288
21372 263/300 Read block 2572 len 40960
21373 264/300 Read block 3930 len 4096
21374 265/300 Read block 2033 len 8192
21375 266/300 Read block 1045 len 12288
21376 267/300 Write block 4533 len 36864 data: 4 3 3 4 4 5 8 10 10
21377 268/300 Read block 829 len 32768
21378 269/300 Write block 1198 len 32768 data: 3 3 4 6 6 6 7 7
21379 270/300 Read block 3368 len 16384
21380 271/300 Flush
21381 272/300 Write block 2965 len 36864 data: 7 7 9 10 9 8 7 6 4
21382 273/300 Flush
21383 274/300 Write block 2490 len 40960 data: 7 6 7 7 7 5 4 4 4 4
21384 275/300 Flush
21385 276/300 Write block 3169 len 40960 data: 8 9 8 8 5 6 6 7 7 6
21386 277/300 Write block 4307 len 8192 data: 5 6
21387 278/300 Write block 2723 len 28672 data: 6 5 3 3 4 6 7
21388 279/300 Read block 4450 len 8192
21389 280/300 Write block 1292 len 40960 data: 9 9 6 6 5 5 5 4 5 5
21390 281/300 Write block 4546 len 4096 data: 5
21391 282/300 Flush
21392 283/300 Read block 4096 len 40960
21393 284/300 Write block 2749 len 32768 data: 8 6 6 8 9 8 6 7
21394 285/300 Write block 4318 len 20480 data: 6 5 5 4 4
21395 286/300 Write block 1799 len 16384 data: 6 4 3 5
21396 287/300 Write block 358 len 32768 data: 5 4 5 5 5 4 3 3
21397 288/300 Read block 4353 len 40960
21398 289/300 Write block 4329 len 8192 data: 4 4
21399 290/300 Flush
21400 291/300 Write block 2768 len 36864 data: 7 7 5 4 4 3 3 5 7
21401 292/300 Flush
21402 293/300 Read block 3007 len 32768
21403 294/300 Read block 4371 len 4096
21404 295/300 Write block 1943 len 24576 data: 5 3 5 5 5 5
21405 296/300 Read block 4905 len 8192
21406 297/300 Read block 3762 len 8192
21407 298/300 Read block 4584 len 4096
21408 299/300 Read block 1869 len 4096
21409 300/300 Write block 2582 len 36864 data: 4 5 6 6 6 5 4 4 2
21410 Main task has work to do, go find it
21411 got dsc Start(2)
21412 start 2
21413 Oct 13 04:46:28.190 INFO request completed, latency_us: 365, response_code: 204, uri: /start/cid/2, method: POST, req_id: 4f08fcec-717d-4558-9791-3d8305486a57, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
21414 Replay: started 2, returned:Ok(())
21415 ----------------------------------------------------------------
21416 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
21417 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
21418 7254 Acked 8253 Flush 0 Done Done New false
21419 7255 Acked 8254 Write 6 Done Done New false
21420 7256 Acked 8255 Write 6 Done Done New false
21421 7257 Acked 8256 Write 8 Done Done New false
21422 7258 Acked 8257 Write 8 Done Done New false
21423 7259 Acked 8258 Write 9 Done Done New false
21424 7260 Acked 8259 Flush 0 Done Done New false
21425 7261 Acked 8260 Read 3 Done Done New false
21426 7262 Acked 8261 Read 10 Done Done New false
21427 7263 Acked 8262 Write 4 Done Done New false
21428 7264 Acked 8263 Flush 0 Done Done New false
21429 7265 Acked 8264 Write 6 Done Done New false
21430 7266 Acked 8265 Write 10 Done Done New false
21431 7267 Acked 8266 Write 8 Done Done New false
21432 7268 Acked 8267 Write 7 Done Done New false
21433 7269 Acked 8268 Read 9 Done Done New false
21434 7270 Acked 8269 Flush 0 Done Done New false
21435 7271 Acked 8270 Read 7 Done Done New false
21436 7272 Acked 8271 Write 10 Done Done New false
21437 7273 Acked 8272 Flush 0 Done Done New false
21438 7274 Acked 8273 Read 10 Done Done New false
21439 7275 Acked 8274 Read 6 Done Done New false
21440 7276 Acked 8275 Write 2 Done Done New false
21441 7277 Acked 8276 Read 2 Done Done New false
21442 7278 Acked 8277 Write 5 Done Done New false
21443 7279 Acked 8278 Write 9 Done Done New false
21444 7280 Acked 8279 Read 7 Done Done New false
21445 7281 Acked 8280 Flush 0 Done Done New false
21446 7282 Acked 8281 Read 6 Done Done New false
21447 7283 Acked 8282 Read 3 Done Done New false
21448 7284 Acked 8283 Write 8 Done Done New false
21449 7285 Acked 8284 Read 9 Done Done New false
21450 7286 Acked 8285 Read 6 Done Done New false
21451 7287 Acked 8286 Read 8 Done Done New false
21452 7288 Acked 8287 Flush 0 Done Done New false
21453 7289 Acked 8288 Write 10 Done Done New false
21454 7290 Acked 8289 Read 10 Done Done New false
21455 7291 Acked 8290 Write 3 Done Done New false
21456 7292 Acked 8291 Write 5 Done Done New false
21457 7293 Acked 8292 Read 3 Done Done New false
21458 7294 Acked 8293 Write 10 Done Done New false
21459 7295 Acked 8294 Write 4 Done Done New false
21460 7296 Acked 8295 Write 4 Done Done New false
21461 7297 Acked 8296 Write 7 Done Done New false
21462 7298 Acked 8297 Write 6 Done Done New false
21463 7299 Acked 8298 Flush 0 Done Done New false
21464 7300 Acked 8299 Write 6 Done Done New false
21465 7301 Acked 8300 Read 3 Done Done New false
21466 7302 Acked 8301 Flush 0 Done Done New false
21467 7303 Acked 8302 Read 6 Done Done New false
21468 7304 Acked 8303 Write 7 Done Done New false
21469 7305 Acked 8304 Read 5 Done Done New false
21470 7306 Acked 8305 Write 7 Done Done New false
21471 7307 Acked 8306 Flush 0 Done Done New false
21472 7308 Acked 8307 Read 3 Done Done New false
21473 7309 Acked 8308 Read 8 Done Done New false
21474 7310 Acked 8309 Read 6 Done Done New false
21475 7311 Acked 8310 Read 9 Done Done New false
21476 7312 Acked 8311 Write 3 Done Done New false
21477 7313 Acked 8312 Read 6 Done Done New false
21478 7314 Acked 8313 Read 7 Done Done New false
21479 7315 Acked 8314 Read 3 Done Done New false
21480 7316 Acked 8315 Flush 0 Done Done New false
21481 7317 Acked 8316 Write 1 Done Done New false
21482 7318 Acked 8317 Read 10 Done Done New false
21483 7319 Acked 8318 Read 7 Done Done New false
21484 7320 Acked 8319 Write 7 Done Done New false
21485 7321 Acked 8320 Flush 0 Done Done New false
21486 7322 Acked 8321 Read 4 Done Done New false
21487 7323 Acked 8322 Read 9 Done Done New false
21488 7324 Acked 8323 Read 8 Done Done New false
21489 7325 Acked 8324 Read 6 Done Done New false
21490 7326 Acked 8325 Write 3 Done Done New false
21491 7327 Acked 8326 Read 4 Done Done New false
21492 7328 Acked 8327 Write 3 Done Done New false
21493 7329 Acked 8328 Read 9 Done Done New false
21494 7330 Acked 8329 Read 8 Done Done New false
21495 7331 Acked 8330 Read 2 Done Done New false
21496 7332 Acked 8331 Write 4 Done Done New false
21497 7333 Acked 8332 Write 4 Done Done New false
21498 7334 Acked 8333 Read 9 Done Done New false
21499 7335 Acked 8334 Read 6 Done Done New false
21500 7336 Acked 8335 Write 4 Done Done New false
21501 7337 Acked 8336 Read 4 Done Done New false
21502 7338 Acked 8337 Read 7 Done Done New false
21503 7339 Acked 8338 Read 5 Done Done New false
21504 7340 Acked 8339 Flush 0 Done Done New false
21505 7341 Acked 8340 Write 1 Done Done New false
21506 7342 Acked 8341 Read 10 Done Done New false
21507 7343 Acked 8342 Write 2 Done Done New false
21508 7344 Acked 8343 Flush 0 Done Done New false
21509 7345 Acked 8344 Write 10 Done Done New false
21510 7346 Acked 8345 Write 8 Done Done New false
21511 7347 Acked 8346 Write 10 Done Done New false
21512 7348 Acked 8347 Write 1 Done Done New false
21513 7349 Acked 8348 Read 4 Done Done New false
21514 7350 Acked 8349 Flush 0 Done Done New false
21515 7351 Acked 8350 Write 10 Done Done New false
21516 7352 Acked 8351 Read 10 Done Done New false
21517 7353 Acked 8352 Flush 0 Done Done New false
21518 7354 Acked 8353 Write 2 Done Done New false
21519 7355 Acked 8354 Read 6 Done Done New false
21520 7356 Acked 8355 Read 3 Done Done New false
21521 7357 Acked 8356 Write 3 Done Done New false
21522 7358 Acked 8357 Write 7 Done Done New false
21523 7359 Acked 8358 Write 10 Done Done New false
21524 7360 Acked 8359 Write 7 Done Done New false
21525 7361 Acked 8360 Read 10 Done Done New false
21526 7362 Acked 8361 Read 10 Done Done New false
21527 7363 Acked 8362 Read 4 Done Done New false
21528 7364 Acked 8363 Read 2 Done Done New false
21529 7365 Acked 8364 Read 10 Done Done New false
21530 7366 Acked 8365 Write 9 Done Done New false
21531 7367 Acked 8366 Flush 0 Done Done New false
21532 7368 Acked 8367 Read 8 Done Done New false
21533 7369 Acked 8368 Read 9 Done Done New false
21534 7370 Acked 8369 Write 4 Done Done New false
21535 7371 Acked 8370 Read 7 Done Done New false
21536 7372 Acked 8371 Write 7 Done Done New false
21537 7373 Acked 8372 Write 9 Done Done New false
21538 7374 Acked 8373 Read 5 Done Done New false
21539 7375 Acked 8374 Read 3 Done Done New false
21540 7376 Acked 8375 Write 10 Done Done New false
21541 7377 Acked 8376 Write 6 Done Done New false
21542 7378 Acked 8377 Write 7 Done Done New false
21543 7379 Acked 8378 Write 7 Done Done New false
21544 7380 Acked 8379 Write 5 Done Done New false
21545 7381 Acked 8380 Read 7 Done Done New false
21546 7382 Acked 8381 Flush 0 Done Done New false
21547 7383 Acked 8382 Write 7 Done Done New false
21548 7384 Acked 8383 Read 9 Done Done New false
21549 7385 Acked 8384 Read 7 Done Done New false
21550 7386 Acked 8385 Write 10 Done Done New false
21551 7387 Acked 8386 Read 8 Done Done New false
21552 7388 Acked 8387 Write 7 Done Done New false
21553 7389 Acked 8388 Write 3 Done Done New false
21554 7390 Acked 8389 Write 2 Done Done New false
21555 7391 Acked 8390 Read 9 Done Done New false
21556 7392 Acked 8391 Write 8 Done Done New false
21557 7393 Acked 8392 Read 9 Done Done New false
21558 7394 Acked 8393 Read 1 Done Done New false
21559 7395 Acked 8394 Write 8 Done Done New false
21560 7396 Acked 8395 Read 6 Done Done New false
21561 7397 Acked 8396 Read 7 Done Done New false
21562 7398 Acked 8397 Write 1 Done Done New false
21563 7399 Acked 8398 Read 9 Done Done New false
21564 7400 Acked 8399 Write 2 Done Done New false
21565 7401 Acked 8400 Write 1 Done Done New false
21566 7402 Acked 8401 Write 5 Done Done New false
21567 7403 Acked 8402 Write 8 Done Done New false
21568 7404 Acked 8403 Write 1 Done Done New false
21569 7405 Acked 8404 Read 2 Done Done New false
21570 7406 Acked 8405 Read 8 Done Done New false
21571 7407 Acked 8406 Read 3 Done Done New false
21572 7408 Acked 8407 Read 3 Done Done New false
21573 7409 Acked 8408 Write 10 Done Done New false
21574 7410 Acked 8409 Write 7 Done Done New false
21575 7411 Acked 8410 Read 1 Done Done New false
21576 7412 Acked 8411 Write 9 Done Done New false
21577 7413 Acked 8412 Write 4 Done Done New false
21578 7414 Acked 8413 Flush 0 Done Done New false
21579 7415 Acked 8414 Read 7 Done Done New false
21580 7416 Acked 8415 Read 3 Done Done New false
21581 7417 Acked 8416 Read 7 Done Done New false
21582 7418 Acked 8417 Read 5 Done Done New false
21583 7419 Acked 8418 Flush 0 Done Done New false
21584 7420 Acked 8419 Flush 0 Done Done New false
21585 7421 Acked 8420 Read 4 Done Done New false
21586 7422 Acked 8421 Read 7 Done Done New false
21587 7423 Acked 8422 Read 7 Done Done New false
21588 7424 Acked 8423 Read 2 Done Done New false
21589 7425 Acked 8424 Flush 0 Done Done New false
21590 7426 Acked 8425 Read 6 Done Done New false
21591 7427 Acked 8426 Write 9 Done Done New false
21592 7428 Acked 8427 Read 1 Done Done New false
21593 7429 Acked 8428 Read 5 Done Done New false
21594 7430 Acked 8429 Write 1 Done Done New false
21595 7431 Acked 8430 Write 10 Done Done New false
21596 7432 Acked 8431 Flush 0 Done Done New false
21597 7433 Acked 8432 Read 10 Done Done New false
21598 7434 Acked 8433 Write 3 Done Done New false
21599 7435 Acked 8434 Write 5 Done Done New false
21600 7436 Acked 8435 Read 2 Done Done New false
21601 7437 Acked 8436 Read 6 Done Done New false
21602 7438 Acked 8437 Read 8 Done Done New false
21603 7439 Acked 8438 Read 8 Done Done New false
21604 7440 Acked 8439 Read 5 Done Done New false
21605 7441 Acked 8440 Read 2 Done Done New false
21606 7442 Acked 8441 Read 1 Done Done New false
21607 7443 Acked 8442 Read 2 Done Done New false
21608 7444 Acked 8443 Read 2 Done Done New false
21609 7445 Acked 8444 Write 2 Done Done New false
21610 7446 Acked 8445 Write 2 Done Done New false
21611 7447 Acked 8446 Write 10 Done Done New false
21612 7448 Acked 8447 Read 1 Done Done New false
21613 7449 Acked 8448 Write 3 Done Done New false
21614 7450 Acked 8449 Read 4 Done Done New false
21615 7451 Acked 8450 Read 10 Done Done New false
21616 7452 Acked 8451 Read 10 Done Done New false
21617 7453 Acked 8452 Flush 0 Done Done New false
21618 7454 Acked 8453 Write 9 Done Done New false
21619 7455 Acked 8454 Flush 0 Done Done New false
21620 7456 Acked 8455 Write 10 Done Done New false
21621 7457 Acked 8456 Flush 0 Done Done New false
21622 7458 Acked 8457 Write 2 Done Done New false
21623 7459 Acked 8458 Read 8 Done Done New false
21624 7460 Acked 8459 Read 6 Done Done New false
21625 7461 Acked 8460 Write 1 Done Done New false
21626 7462 Acked 8461 Write 1 Done Done New false
21627 7463 Acked 8462 Read 10 Done Done New false
21628 7464 Acked 8463 Read 8 Done Done New false
21629 7465 Acked 8464 Write 8 Done Done New false
21630 7466 Acked 8465 Write 1 Done Done New false
21631 7467 Acked 8466 Read 1 Done Done New false
21632 7468 Acked 8467 Write 1 Done Done New false
21633 7469 Acked 8468 Write 6 Done Done New false
21634 7470 Acked 8469 Read 7 Done Done New false
21635 7471 Acked 8470 Read 7 Done Done New false
21636 7472 Acked 8471 Read 10 Done Done New false
21637 7473 Acked 8472 Read 7 Done Done New false
21638 7474 Acked 8473 Read 9 Done Done New false
21639 7475 Acked 8474 Read 4 Done Done New false
21640 7476 Acked 8475 Read 10 Done Done New false
21641 7477 Acked 8476 Write 3 Done Done New false
21642 7478 Acked 8477 Read 9 Done Done New false
21643 7479 Acked 8478 Write 1 Done Done New false
21644 7480 Acked 8479 Read 8 Done Done New false
21645 7481 Acked 8480 Write 6 Done Done New false
21646 7482 Acked 8481 Write 9 Done Done New false
21647 7483 Acked 8482 Write 4 Done Done New false
21648 7484 Acked 8483 Write 7 Done Done New false
21649 7485 Acked 8484 Write 5 Done Done New false
21650 7486 Acked 8485 Read 8 Done Done New false
21651 7487 Acked 8486 Write 2 Done Done New false
21652 7488 Acked 8487 Write 7 Done Done New false
21653 7489 Acked 8488 Write 6 Done Done New false
21654 7490 Acked 8489 Write 4 Done Done New false
21655 7491 Acked 8490 Read 9 Done Done New false
21656 7492 Acked 8491 Write 1 Done Done New false
21657 7493 Acked 8492 Read 7 Done Done New false
21658 7494 Acked 8493 Read 10 Done Done New false
21659 7495 Acked 8494 Write 8 Done Done New false
21660 7496 Acked 8495 Read 8 Done Done New false
21661 7497 Acked 8496 Read 3 Done Done New false
21662 7498 Acked 8497 Read 1 Done Done New false
21663 7499 Acked 8498 Flush 0 Done Done New false
21664 7500 Acked 8499 Read 9 Done Done New false
21665 7501 Acked 8500 Read 1 Done Done New false
21666 7502 Acked 8501 Read 9 Done Done New false
21667 7503 Acked 8502 Read 3 Done Done New false
21668 7504 Acked 8503 Write 1 Done Done New false
21669 7505 Acked 8504 Read 8 Done Done New false
21670 7506 Acked 8505 Read 8 Done Done New false
21671 7507 Acked 8506 Read 1 Done Done New false
21672 7508 Acked 8507 Read 4 Done Done New false
21673 7509 Acked 8508 Read 10 Done Done New false
21674 7510 Acked 8509 Read 9 Done Done New false
21675 7511 Acked 8510 Flush 0 Done Done New false
21676 7512 Acked 8511 Read 3 Done Done New false
21677 7513 Acked 8512 Read 6 Done Done New false
21678 7514 Acked 8513 Read 2 Done Done New false
21679 7515 Acked 8514 Read 2 Done Done New false
21680 7516 Acked 8515 Write 3 Done Done New false
21681 7517 Acked 8516 Read 3 Done Done New false
21682 7518 Acked 8517 Read 10 Done Done New false
21683 7519 Acked 8518 Read 1 Done Done New false
21684 7520 Acked 8519 Read 2 Done Done New false
21685 7521 Acked 8520 Read 3 Done Done New false
21686 7522 Acked 8521 Write 9 Done Done New false
21687 7523 Acked 8522 Read 8 Done Done New false
21688 7524 Acked 8523 Write 8 Done Done New false
21689 7525 Acked 8524 Read 4 Done Done New false
21690 7526 Acked 8525 Flush 0 Done Done New false
21691 7527 Acked 8526 Write 9 Done Done New false
21692 7528 Acked 8527 Flush 0 Done Done New false
21693 7529 Acked 8528 Write 10 Done Done New false
21694 7530 Acked 8529 Flush 0 Done Done New false
21695 7531 Acked 8530 Write 10 Done Done New false
21696 7532 Acked 8531 Write 2 Done Done New false
21697 7533 Acked 8532 Write 7 Done Done New false
21698 7534 Acked 8533 Read 2 Done Done New false
21699 7535 Acked 8534 Write 10 Done Done New false
21700 7536 Acked 8535 Write 1 Done Done New false
21701 7537 Acked 8536 Flush 0 Done Done New false
21702 7538 Acked 8537 Read 10 Done Done New false
21703 7539 Acked 8538 Write 8 Done Done New false
21704 7540 Acked 8539 Write 5 Done Done New false
21705 7541 Acked 8540 Write 4 Done Done New false
21706 7542 Acked 8541 Write 8 Done Done New false
21707 7543 Acked 8542 Read 10 Done Done New false
21708 7544 Acked 8543 Write 2 Done Done New false
21709 7545 Acked 8544 Flush 0 Done Done New false
21710 7546 Acked 8545 Write 9 Done Done New false
21711 7547 Acked 8546 Flush 0 Done Done New false
21712 7548 Acked 8547 Read 8 Done Done New false
21713 7549 Acked 8548 Read 1 Done Done New false
21714 7550 Acked 8549 Write 6 Done Done New false
21715 7551 Acked 8550 Read 2 Done Done New false
21716 7552 Acked 8551 Read 2 Done Done New false
21717 7553 Acked 8552 Read 1 Done Done New false
21718 7554 Acked 8553 Read 1 Done Done New false
21719 7555 Acked 8554 Write 9 Sent Sent New false
21720 STATES DS:0 DS:1 DS:2 TOTAL
21721 New 0 0 302 302
21722 Sent 1 1 0 2
21723 Done 301 301 0 602
21724 Skipped 0 0 0 0
21725 Error 0 0 0 0
21726 Last Flush: 8546 8546 8252
21727 Downstairs last five completed: 8252 8251 8250 8249 8248 8247
21728 Upstairs last five completed: 7555 7554 7553 7552 7551 7550
21729 CLIENT: Up:0 ds:302 act:2
21730 [8830] got start action
21731 [8830] I am going to restart
21732 Starting downstairs at port 8830
21733 Make output file at "/tmp/dsc/downstairs-8830.txt"
21734 [8830][2] reports Starting
21735 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1201)
21736 [8830][2] reports Running
21737 ----------------------------------------------------------------
21738 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
21739 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
21740 7254 Acked 8253 Flush 0 Done Done New false
21741 7255 Acked 8254 Write 6 Done Done New false
21742 7256 Acked 8255 Write 6 Done Done New false
21743 7257 Acked 8256 Write 8 Done Done New false
21744 7258 Acked 8257 Write 8 Done Done New false
21745 7259 Acked 8258 Write 9 Done Done New false
21746 7260 Acked 8259 Flush 0 Done Done New false
21747 7261 Acked 8260 Read 3 Done Done New false
21748 7262 Acked 8261 Read 10 Done Done New false
21749 7263 Acked 8262 Write 4 Done Done New false
21750 7264 Acked 8263 Flush 0 Done Done New false
21751 7265 Acked 8264 Write 6 Done Done New false
21752 7266 Acked 8265 Write 10 Done Done New false
21753 7267 Acked 8266 Write 8 Done Done New false
21754 7268 Acked 8267 Write 7 Done Done New false
21755 7269 Acked 8268 Read 9 Done Done New false
21756 7270 Acked 8269 Flush 0 Done Done New false
21757 7271 Acked 8270 Read 7 Done Done New false
21758 7272 Acked 8271 Write 10 Done Done New false
21759 7273 Acked 8272 Flush 0 Done Done New false
21760 7274 Acked 8273 Read 10 Done Done New false
21761 7275 Acked 8274 Read 6 Done Done New false
21762 7276 Acked 8275 Write 2 Done Done New false
21763 7277 Acked 8276 Read 2 Done Done New false
21764 7278 Acked 8277 Write 5 Done Done New false
21765 7279 Acked 8278 Write 9 Done Done New false
21766 7280 Acked 8279 Read 7 Done Done New false
21767 7281 Acked 8280 Flush 0 Done Done New false
21768 7282 Acked 8281 Read 6 Done Done New false
21769 7283 Acked 8282 Read 3 Done Done New false
21770 7284 Acked 8283 Write 8 Done Done New false
21771 7285 Acked 8284 Read 9 Done Done New false
21772 7286 Acked 8285 Read 6 Done Done New false
21773 7287 Acked 8286 Read 8 Done Done New false
21774 7288 Acked 8287 Flush 0 Done Done New false
21775 7289 Acked 8288 Write 10 Done Done New false
21776 7290 Acked 8289 Read 10 Done Done New false
21777 7291 Acked 8290 Write 3 Done Done New false
21778 7292 Acked 8291 Write 5 Done Done New false
21779 7293 Acked 8292 Read 3 Done Done New false
21780 7294 Acked 8293 Write 10 Done Done New false
21781 7295 Acked 8294 Write 4 Done Done New false
21782 7296 Acked 8295 Write 4 Done Done New false
21783 7297 Acked 8296 Write 7 Done Done New false
21784 7298 Acked 8297 Write 6 Done Done New false
21785 7299 Acked 8298 Flush 0 Done Done New false
21786 7300 Acked 8299 Write 6 Done Done New false
21787 7301 Acked 8300 Read 3 Done Done New false
21788 7302 Acked 8301 Flush 0 Done Done New false
21789 7303 Acked 8302 Read 6 Done Done New false
21790 7304 Acked 8303 Write 7 Done Done New false
21791 7305 Acked 8304 Read 5 Done Done New false
21792 7306 Acked 8305 Write 7 Done Done New false
21793 7307 Acked 8306 Flush 0 Done Done New false
21794 7308 Acked 8307 Read 3 Done Done New false
21795 7309 Acked 8308 Read 8 Done Done New false
21796 7310 Acked 8309 Read 6 Done Done New false
21797 7311 Acked 8310 Read 9 Done Done New false
21798 7312 Acked 8311 Write 3 Done Done New false
21799 7313 Acked 8312 Read 6 Done Done New false
21800 7314 Acked 8313 Read 7 Done Done New false
21801 7315 Acked 8314 Read 3 Done Done New false
21802 7316 Acked 8315 Flush 0 Done Done New false
21803 7317 Acked 8316 Write 1 Done Done New false
21804 7318 Acked 8317 Read 10 Done Done New false
21805 7319 Acked 8318 Read 7 Done Done New false
21806 7320 Acked 8319 Write 7 Done Done New false
21807 7321 Acked 8320 Flush 0 Done Done New false
21808 7322 Acked 8321 Read 4 Done Done New false
21809 7323 Acked 8322 Read 9 Done Done New false
21810 7324 Acked 8323 Read 8 Done Done New false
21811 7325 Acked 8324 Read 6 Done Done New false
21812 7326 Acked 8325 Write 3 Done Done New false
21813 7327 Acked 8326 Read 4 Done Done New false
21814 7328 Acked 8327 Write 3 Done Done New false
21815 7329 Acked 8328 Read 9 Done Done New false
21816 7330 Acked 8329 Read 8 Done Done New false
21817 7331 Acked 8330 Read 2 Done Done New false
21818 7332 Acked 8331 Write 4 Done Done New false
21819 7333 Acked 8332 Write 4 Done Done New false
21820 7334 Acked 8333 Read 9 Done Done New false
21821 7335 Acked 8334 Read 6 Done Done New false
21822 7336 Acked 8335 Write 4 Done Done New false
21823 7337 Acked 8336 Read 4 Done Done New false
21824 7338 Acked 8337 Read 7 Done Done New false
21825 7339 Acked 8338 Read 5 Done Done New false
21826 7340 Acked 8339 Flush 0 Done Done New false
21827 7341 Acked 8340 Write 1 Done Done New false
21828 7342 Acked 8341 Read 10 Done Done New false
21829 7343 Acked 8342 Write 2 Done Done New false
21830 7344 Acked 8343 Flush 0 Done Done New false
21831 7345 Acked 8344 Write 10 Done Done New false
21832 7346 Acked 8345 Write 8 Done Done New false
21833 7347 Acked 8346 Write 10 Done Done New false
21834 7348 Acked 8347 Write 1 Done Done New false
21835 7349 Acked 8348 Read 4 Done Done New false
21836 7350 Acked 8349 Flush 0 Done Done New false
21837 7351 Acked 8350 Write 10 Done Done New false
21838 7352 Acked 8351 Read 10 Done Done New false
21839 7353 Acked 8352 Flush 0 Done Done New false
21840 7354 Acked 8353 Write 2 Done Done New false
21841 7355 Acked 8354 Read 6 Done Done New false
21842 7356 Acked 8355 Read 3 Done Done New false
21843 7357 Acked 8356 Write 3 Done Done New false
21844 7358 Acked 8357 Write 7 Done Done New false
21845 7359 Acked 8358 Write 10 Done Done New false
21846 7360 Acked 8359 Write 7 Done Done New false
21847 7361 Acked 8360 Read 10 Done Done New false
21848 7362 Acked 8361 Read 10 Done Done New false
21849 7363 Acked 8362 Read 4 Done Done New false
21850 7364 Acked 8363 Read 2 Done Done New false
21851 7365 Acked 8364 Read 10 Done Done New false
21852 7366 Acked 8365 Write 9 Done Done New false
21853 7367 Acked 8366 Flush 0 Done Done New false
21854 7368 Acked 8367 Read 8 Done Done New false
21855 7369 Acked 8368 Read 9 Done Done New false
21856 7370 Acked 8369 Write 4 Done Done New false
21857 7371 Acked 8370 Read 7 Done Done New false
21858 7372 Acked 8371 Write 7 Done Done New false
21859 7373 Acked 8372 Write 9 Done Done New false
21860 7374 Acked 8373 Read 5 Done Done New false
21861 7375 Acked 8374 Read 3 Done Done New false
21862 7376 Acked 8375 Write 10 Done Done New false
21863 7377 Acked 8376 Write 6 Done Done New false
21864 7378 Acked 8377 Write 7 Done Done New false
21865 7379 Acked 8378 Write 7 Done Done New false
21866 7380 Acked 8379 Write 5 Done Done New false
21867 7381 Acked 8380 Read 7 Done Done New false
21868 7382 Acked 8381 Flush 0 Done Done New false
21869 7383 Acked 8382 Write 7 Done Done New false
21870 7384 Acked 8383 Read 9 Done Done New false
21871 7385 Acked 8384 Read 7 Done Done New false
21872 7386 Acked 8385 Write 10 Done Done New false
21873 7387 Acked 8386 Read 8 Done Done New false
21874 7388 Acked 8387 Write 7 Done Done New false
21875 7389 Acked 8388 Write 3 Done Done New false
21876 7390 Acked 8389 Write 2 Done Done New false
21877 7391 Acked 8390 Read 9 Done Done New false
21878 7392 Acked 8391 Write 8 Done Done New false
21879 7393 Acked 8392 Read 9 Done Done New false
21880 7394 Acked 8393 Read 1 Done Done New false
21881 7395 Acked 8394 Write 8 Done Done New false
21882 7396 Acked 8395 Read 6 Done Done New false
21883 7397 Acked 8396 Read 7 Done Done New false
21884 7398 Acked 8397 Write 1 Done Done New false
21885 7399 Acked 8398 Read 9 Done Done New false
21886 7400 Acked 8399 Write 2 Done Done New false
21887 7401 Acked 8400 Write 1 Done Done New false
21888 7402 Acked 8401 Write 5 Done Done New false
21889 7403 Acked 8402 Write 8 Done Done New false
21890 7404 Acked 8403 Write 1 Done Done New false
21891 7405 Acked 8404 Read 2 Done Done New false
21892 7406 Acked 8405 Read 8 Done Done New false
21893 7407 Acked 8406 Read 3 Done Done New false
21894 7408 Acked 8407 Read 3 Done Done New false
21895 7409 Acked 8408 Write 10 Done Done New false
21896 7410 Acked 8409 Write 7 Done Done New false
21897 7411 Acked 8410 Read 1 Done Done New false
21898 7412 Acked 8411 Write 9 Done Done New false
21899 7413 Acked 8412 Write 4 Done Done New false
21900 7414 Acked 8413 Flush 0 Done Done New false
21901 7415 Acked 8414 Read 7 Done Done New false
21902 7416 Acked 8415 Read 3 Done Done New false
21903 7417 Acked 8416 Read 7 Done Done New false
21904 7418 Acked 8417 Read 5 Done Done New false
21905 7419 Acked 8418 Flush 0 Done Done New false
21906 7420 Acked 8419 Flush 0 Done Done New false
21907 7421 Acked 8420 Read 4 Done Done New false
21908 7422 Acked 8421 Read 7 Done Done New false
21909 7423 Acked 8422 Read 7 Done Done New false
21910 7424 Acked 8423 Read 2 Done Done New false
21911 7425 Acked 8424 Flush 0 Done Done New false
21912 7426 Acked 8425 Read 6 Done Done New false
21913 7427 Acked 8426 Write 9 Done Done New false
21914 7428 Acked 8427 Read 1 Done Done New false
21915 7429 Acked 8428 Read 5 Done Done New false
21916 7430 Acked 8429 Write 1 Done Done New false
21917 7431 Acked 8430 Write 10 Done Done New false
21918 7432 Acked 8431 Flush 0 Done Done New false
21919 7433 Acked 8432 Read 10 Done Done New false
21920 7434 Acked 8433 Write 3 Done Done New false
21921 7435 Acked 8434 Write 5 Done Done New false
21922 7436 Acked 8435 Read 2 Done Done New false
21923 7437 Acked 8436 Read 6 Done Done New false
21924 7438 Acked 8437 Read 8 Done Done New false
21925 7439 Acked 8438 Read 8 Done Done New false
21926 7440 Acked 8439 Read 5 Done Done New false
21927 7441 Acked 8440 Read 2 Done Done New false
21928 7442 Acked 8441 Read 1 Done Done New false
21929 7443 Acked 8442 Read 2 Done Done New false
21930 7444 Acked 8443 Read 2 Done Done New false
21931 7445 Acked 8444 Write 2 Done Done New false
21932 7446 Acked 8445 Write 2 Done Done New false
21933 7447 Acked 8446 Write 10 Done Done New false
21934 7448 Acked 8447 Read 1 Done Done New false
21935 7449 Acked 8448 Write 3 Done Done New false
21936 7450 Acked 8449 Read 4 Done Done New false
21937 7451 Acked 8450 Read 10 Done Done New false
21938 7452 Acked 8451 Read 10 Done Done New false
21939 7453 Acked 8452 Flush 0 Done Done New false
21940 7454 Acked 8453 Write 9 Done Done New false
21941 7455 Acked 8454 Flush 0 Done Done New false
21942 7456 Acked 8455 Write 10 Done Done New false
21943 7457 Acked 8456 Flush 0 Done Done New false
21944 7458 Acked 8457 Write 2 Done Done New false
21945 7459 Acked 8458 Read 8 Done Done New false
21946 7460 Acked 8459 Read 6 Done Done New false
21947 7461 Acked 8460 Write 1 Done Done New false
21948 7462 Acked 8461 Write 1 Done Done New false
21949 7463 Acked 8462 Read 10 Done Done New false
21950 7464 Acked 8463 Read 8 Done Done New false
21951 7465 Acked 8464 Write 8 Done Done New false
21952 7466 Acked 8465 Write 1 Done Done New false
21953 7467 Acked 8466 Read 1 Done Done New false
21954 7468 Acked 8467 Write 1 Done Done New false
21955 7469 Acked 8468 Write 6 Done Done New false
21956 7470 Acked 8469 Read 7 Done Done New false
21957 7471 Acked 8470 Read 7 Done Done New false
21958 7472 Acked 8471 Read 10 Done Done New false
21959 7473 Acked 8472 Read 7 Done Done New false
21960 7474 Acked 8473 Read 9 Done Done New false
21961 7475 Acked 8474 Read 4 Done Done New false
21962 7476 Acked 8475 Read 10 Done Done New false
21963 7477 Acked 8476 Write 3 Done Done New false
21964 7478 Acked 8477 Read 9 Done Done New false
21965 7479 Acked 8478 Write 1 Done Done New false
21966 7480 Acked 8479 Read 8 Done Done New false
21967 7481 Acked 8480 Write 6 Done Done New false
21968 7482 Acked 8481 Write 9 Done Done New false
21969 7483 Acked 8482 Write 4 Done Done New false
21970 7484 Acked 8483 Write 7 Done Done New false
21971 7485 Acked 8484 Write 5 Done Done New false
21972 7486 Acked 8485 Read 8 Done Done New false
21973 7487 Acked 8486 Write 2 Done Done New false
21974 7488 Acked 8487 Write 7 Done Done New false
21975 7489 Acked 8488 Write 6 Done Done New false
21976 7490 Acked 8489 Write 4 Done Done New false
21977 7491 Acked 8490 Read 9 Done Done New false
21978 7492 Acked 8491 Write 1 Done Done New false
21979 7493 Acked 8492 Read 7 Done Done New false
21980 7494 Acked 8493 Read 10 Done Done New false
21981 7495 Acked 8494 Write 8 Done Done New false
21982 7496 Acked 8495 Read 8 Done Done New false
21983 7497 Acked 8496 Read 3 Done Done New false
21984 7498 Acked 8497 Read 1 Done Done New false
21985 7499 Acked 8498 Flush 0 Done Done New false
21986 7500 Acked 8499 Read 9 Done Done New false
21987 7501 Acked 8500 Read 1 Done Done New false
21988 7502 Acked 8501 Read 9 Done Done New false
21989 7503 Acked 8502 Read 3 Done Done New false
21990 7504 Acked 8503 Write 1 Done Done New false
21991 7505 Acked 8504 Read 8 Done Done New false
21992 7506 Acked 8505 Read 8 Done Done New false
21993 7507 Acked 8506 Read 1 Done Done New false
21994 7508 Acked 8507 Read 4 Done Done New false
21995 7509 Acked 8508 Read 10 Done Done New false
21996 7510 Acked 8509 Read 9 Done Done New false
21997 7511 Acked 8510 Flush 0 Done Done New false
21998 7512 Acked 8511 Read 3 Done Done New false
21999 7513 Acked 8512 Read 6 Done Done New false
22000 7514 Acked 8513 Read 2 Done Done New false
22001 7515 Acked 8514 Read 2 Done Done New false
22002 7516 Acked 8515 Write 3 Done Done New false
22003 7517 Acked 8516 Read 3 Done Done New false
22004 7518 Acked 8517 Read 10 Done Done New false
22005 7519 Acked 8518 Read 1 Done Done New false
22006 7520 Acked 8519 Read 2 Done Done New false
22007 7521 Acked 8520 Read 3 Done Done New false
22008 7522 Acked 8521 Write 9 Done Done New false
22009 7523 Acked 8522 Read 8 Done Done New false
22010 7524 Acked 8523 Write 8 Done Done New false
22011 7525 Acked 8524 Read 4 Done Done New false
22012 7526 Acked 8525 Flush 0 Done Done New false
22013 7527 Acked 8526 Write 9 Done Done New false
22014 7528 Acked 8527 Flush 0 Done Done New false
22015 7529 Acked 8528 Write 10 Done Done New false
22016 7530 Acked 8529 Flush 0 Done Done New false
22017 7531 Acked 8530 Write 10 Done Done New false
22018 7532 Acked 8531 Write 2 Done Done New false
22019 7533 Acked 8532 Write 7 Done Done New false
22020 7534 Acked 8533 Read 2 Done Done New false
22021 7535 Acked 8534 Write 10 Done Done New false
22022 7536 Acked 8535 Write 1 Done Done New false
22023 7537 Acked 8536 Flush 0 Done Done New false
22024 7538 Acked 8537 Read 10 Done Done New false
22025 7539 Acked 8538 Write 8 Done Done New false
22026 7540 Acked 8539 Write 5 Done Done New false
22027 7541 Acked 8540 Write 4 Done Done New false
22028 7542 Acked 8541 Write 8 Done Done New false
22029 7543 Acked 8542 Read 10 Done Done New false
22030 7544 Acked 8543 Write 2 Done Done New false
22031 7545 Acked 8544 Flush 0 Done Done New false
22032 7546 Acked 8545 Write 9 Done Done New false
22033 7547 Acked 8546 Flush 0 Done Done New false
22034 7548 Acked 8547 Read 8 Done Done New false
22035 7549 Acked 8548 Read 1 Done Done New false
22036 7550 Acked 8549 Write 6 Done Done New false
22037 7551 Acked 8550 Read 2 Done Done New false
22038 7552 Acked 8551 Read 2 Done Done New false
22039 7553 Acked 8552 Read 1 Done Done New false
22040 7554 Acked 8553 Read 1 Done Done New false
22041 7555 Acked 8554 Write 9 Done Done New false
22042 7556 Acked 8555 Flush 0 Done Done New false
22043 STATES DS:0 DS:1 DS:2 TOTAL
22044 New 0 0 303 303
22045 Sent 0 0 0 0
22046 Done 303 303 0 606
22047 Skipped 0 0 0 0
22048 Error 0 0 0 0
22049 Last Flush: 8555 8555 8252
22050 Downstairs last five completed: 8252 8251 8250 8249 8248 8247
22051 Upstairs last five completed: 7556 7555 7554 7553 7552 7551
22052 CLIENT: Up:0 ds:303 act:2
220532023-10-13T04:46:32.283ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
220542023-10-13T04:46:32.284ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 2
220552023-10-13T04:46:32.284ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
220562023-10-13T04:46:32.284ZINFOcrucible: [2] client re-new 303 jobs since flush 8252 = downstairs
220572023-10-13T04:46:32.285ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
220582023-10-13T04:46:32.285ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
220592023-10-13T04:46:32.285ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
220602023-10-13T04:46:32.285ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
220612023-10-13T04:46:32.285ZINFOcrucible: [2] send last flush ID to this DS: 8252
220622023-10-13T04:46:32.285ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
220632023-10-13T04:46:32.285ZINFOcrucible: [2] Replied this last flush ID: 8252
220642023-10-13T04:46:32.286ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Offline ds_transition to Replay
220652023-10-13T04:46:32.286ZINFOcrucible: [2] Transition from Offline to Replay
220662023-10-13T04:46:32.286ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
220672023-10-13T04:46:32.286ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Replay ds_transition to Active
220682023-10-13T04:46:32.286ZINFOcrucible: [2] Transition from Replay to Active
220692023-10-13T04:46:32.286ZINFOcrucible: [2] Starts cmd_loop
220702023-10-13T04:46:32.286ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
220712023-10-13T04:46:32.286ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
220722023-10-13T04:46:32.286ZINFOcrucible: Set check for repair
220732023-10-13T04:46:33.286ZINFOcrucible: Checking if live repair is needed
220742023-10-13T04:46:33.286ZINFOcrucible: No Live Repair required at this time
220752023-10-13T04:46:33.286ZWARNcrucible: [2] flow control end
22076 ----------------------------------------------------------------
22077 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22078 Downstairs last five completed: 8555 8554 8553 8552 8551 8550
22079 Upstairs last five completed: 7556 7555 7554 7553 7552 7551
22080 CLIENT: Up:0 ds:0 act:3
22081 Replay: All jobs finished, all DS active.
22082 Main task has work to do, go find it
22083 got dsc Stop(1)
22084 stop 1
22085 Oct 13 04:46:36.200 INFO request completed, latency_us: 532, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 8d58743a-d3da-4f71-942c-6b4cb142e9f5, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
22086 [8820] Got stop action so:false kr:false
22087 Oct 13 04:46:36.201 INFO request completed, latency_us: 188, response_code: 200, uri: /state/cid/1, method: GET, req_id: 2cb45ed3-516c-4ffe-94ab-db93e1050f76, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
220882023-10-13T04:46:36.201ZWARNcrucible: [1] downstairs disconnected
220892023-10-13T04:46:36.201ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
220902023-10-13T04:46:36.201ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8820 closed looper = 1
220912023-10-13T04:46:36.201ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:false
220922023-10-13T04:46:36.201ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
220932023-10-13T04:46:36.201ZINFOcrucible: [1] 127.0.0.1:8820 task reports offline
220942023-10-13T04:46:36.201ZWARNcrucible: [1] pm_task rx.recv() is None
22095 [8820] Exited with: ExitStatus(unix_wait_status(9))
22096 [8820][1] reports Exit
22097 Oct 13 04:46:40.203 INFO request completed, latency_us: 257, response_code: 200, uri: /state/cid/1, method: GET, req_id: 11f40634-f30a-4620-b6c9-e71cc479c903, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
22098 001/300 Read block 323 len 36864
220992023-10-13T04:46:40.206ZINFOcrucible: [1] connecting to 127.0.0.1:8820 looper = 1
22100 002/300 Write block 3284 len 12288 data: 7 5 5
22101 003/300 Write block 2206 len 28672 data: 4 4 4 5 5 5 5
22102 004/300 Read block 2768 len 36864
22103 005/300 Write block 2408 len 20480 data: 4 5 4 5 4
22104 006/300 Read block 3820 len 36864
22105 007/300 Read block 1502 len 16384
22106 008/300 Write block 862 len 20480 data: 5 5 5 5 8
22107 009/300 Write block 302 len 12288 data: 4 5 4
22108 010/300 Read block 3294 len 16384
22109 011/300 Read block 1186 len 16384
22110 012/300 Read block 18 len 36864
22111 013/300 Write block 2468 len 12288 data: 5 7 7
22112 014/300 Write block 1011 len 20480 data: 5 6 7 8 7
22113 015/300 Write block 2326 len 36864 data: 7 6 6 6 6 4 4 3 4
22114 016/300 Flush
22115 017/300 Write block 1210 len 8192 data: 3 3
22116 018/300 Read block 2955 len 28672
22117 019/300 Flush
22118 020/300 Read block 811 len 24576
22119 021/300 Read block 4003 len 16384
22120 022/300 Read block 733 len 24576
22121 023/300 Read block 1763 len 24576
22122 024/300 Flush
22123 025/300 Read block 2209 len 32768
22124 026/300 Write block 2293 len 40960 data: 6 7 6 6 5 4 4 5 5 3
22125 027/300 Write block 2439 len 16384 data: 2 4 5 5
22126 028/300 Read block 1367 len 40960
22127 029/300 Write block 1716 len 8192 data: 8 5
22128 030/300 Read block 2203 len 32768
22129 031/300 Read block 1825 len 36864
22130 032/300 Read block 3602 len 8192
22131 033/300 Flush
22132 034/300 Read block 3091 len 20480
22133 035/300 Flush
22134 036/300 Write block 3042 len 24576 data: 10 10 7 6 5 5
22135 037/300 Read block 4091 len 4096
22136 038/300 Write block 1399 len 32768 data: 8 7 6 6 6 6 5 4
22137 039/300 Write block 530 len 16384 data: 6 4 6 4
22138 040/300 Read block 2723 len 32768
22139 041/300 Read block 1727 len 24576
22140 042/300 Write block 4249 len 40960 data: 7 4 4 5 6 4 4 5 4 4
22141 043/300 Read block 357 len 32768
22142 044/300 Write block 2473 len 40960 data: 5 5 4 5 5 4 5 6 5 5
22143 045/300 Write block 850 len 36864 data: 6 8 8 6 5 4 5 5 5
22144 046/300 Read block 3819 len 28672
22145 047/300 Read block 902 len 8192
22146 048/300 Write block 3861 len 32768 data: 8 6 6 5 5 6 5 4
22147 049/300 Read block 812 len 12288
22148 050/300 Write block 3388 len 4096 data: 8
22149 051/300 Write block 4220 len 28672 data: 3 2 2 2 3 3 3
22150 052/300 Read block 2426 len 16384
22151 053/300 Read block 3031 len 16384
22152 054/300 Write block 1611 len 8192 data: 5 5
22153 055/300 Write block 152 len 16384 data: 10 9 10 8
22154 056/300 Write block 884 len 8192 data: 4 5
22155 057/300 Read block 3451 len 12288
22156 058/300 Read block 652 len 24576
22157 059/300 Read block 386 len 4096
22158 060/300 Write block 374 len 36864 data: 5 7 6 5 4 4 4 3 3
22159 061/300 Read block 913 len 32768
22160 062/300 Write block 499 len 20480 data: 6 5 5 5 5
22161 063/300 Write block 2751 len 12288 data: 7 9 10
22162 064/300 Flush
22163 065/300 Write block 3627 len 16384 data: 5 6 6 6
22164 066/300 Write block 2861 len 12288 data: 3 6 6
22165 067/300 Read block 3939 len 20480
22166 068/300 Read block 420 len 20480
22167 069/300 Flush
22168 070/300 Read block 596 len 24576
22169 071/300 Write block 4824 len 4096 data: 7
22170 072/300 Write block 4599 len 20480 data: 6 6 7 6 6
22171 073/300 Write block 4915 len 40960 data: 7 7 6 7 7 7 6 5 5 5
22172 074/300 Read block 4033 len 20480
22173 075/300 Flush
22174 076/300 Read block 1326 len 16384
22175 077/300 Read block 1152 len 20480
22176 078/300 Read block 3727 len 12288
22177 079/300 Write block 3487 len 20480 data: 7 7 7 5 4
22178 080/300 Read block 995 len 12288
22179 081/300 Write block 1488 len 36864 data: 6 6 7 8 6 6 6 5 7
22180 082/300 Flush
22181 083/300 Read block 3790 len 32768
22182 084/300 Flush
22183 085/300 Read block 1525 len 32768
22184 086/300 Write block 4087 len 28672 data: 7 6 4 4 3 4 3
22185 087/300 Read block 1381 len 8192
22186 088/300 Write block 1268 len 8192 data: 7 7
22187 089/300 Write block 3037 len 20480 data: 6 6 6 7 8
22188 090/300 Write block 1987 len 32768 data: 3 3 4 4 3 4 4 5
22189 091/300 Flush
22190 092/300 Flush
22191 093/300 Read block 4315 len 40960
22192 094/300 Read block 3351 len 12288
22193 095/300 Flush
22194 096/300 Read block 106 len 32768
22195 097/300 Read block 3420 len 28672
22196 098/300 Read block 3886 len 20480
22197 099/300 Read block 1349 len 4096
22198 100/300 Read block 3673 len 28672
22199 101/300 Read block 2598 len 4096
22200 102/300 Read block 4561 len 4096
22201 103/300 Write block 3545 len 40960 data: 4 4 4 5 6 5 6 9 11 8
22202 104/300 Read block 1561 len 40960
22203 105/300 Write block 4845 len 28672 data: 3 3 3 3 3 4 5
22204 106/300 Read block 4670 len 8192
22205 107/300 Write block 4435 len 20480 data: 5 6 8 8 8
22206 108/300 Read block 2468 len 8192
22207 109/300 Write block 1861 len 32768 data: 8 6 8 6 6 5 5 4
22208 110/300 Read block 4716 len 20480
22209 111/300 Write block 4369 len 40960 data: 6 6 7 7 6 5 4 3 4 2
22210 112/300 Write block 3327 len 20480 data: 6 6 6 6 6
22211 113/300 Read block 3837 len 16384
22212 114/300 Write block 1550 len 24576 data: 5 5 5 5 6 6
22213 115/300 Flush
22214 116/300 Read block 2738 len 36864
22215 117/300 Read block 4069 len 24576
22216 118/300 Read block 2758 len 12288
22217 119/300 Write block 3836 len 20480 data: 4 5 6 6 7
22218 120/300 Read block 4260 len 12288
22219 121/300 Read block 1407 len 20480
22220 122/300 Read block 2097 len 4096
22221 123/300 Write block 1196 len 32768 data: 2 2 4 4 5 7 7 7
22222 124/300 Write block 4651 len 32768 data: 3 6 7 5 4 5 5 7
22223 125/300 Write block 5 len 24576 data: 5 7 6 6 6 6
22224 126/300 Write block 2624 len 4096 data: 7
22225 127/300 Write block 3807 len 20480 data: 7 6 6 4 4
22226 128/300 Read block 4206 len 32768
22227 129/300 Write block 1181 len 16384 data: 5 5 5 6
22228 130/300 Write block 1305 len 12288 data: 6 7 6
22229 131/300 Write block 2256 len 12288 data: 4 3 2
22230 132/300 Write block 3477 len 32768 data: 6 6 6 6 7 7 7 7
22231 133/300 Flush
22232 134/300 Read block 3217 len 20480
22233 135/300 Read block 192 len 20480
22234 136/300 Read block 3489 len 24576
22235 137/300 Read block 2009 len 24576
22236 138/300 Read block 506 len 8192
22237 139/300 Read block 2386 len 12288
22238 140/300 Flush
22239 141/300 Flush
22240 142/300 Read block 2249 len 4096
22241 143/300 Read block 418 len 8192
22242 144/300 Read block 37 len 32768
22243 145/300 Read block 2729 len 36864
22244 146/300 Read block 1910 len 32768
22245 147/300 Read block 1269 len 4096
22246 148/300 Write block 1796 len 16384 data: 7 6 5 7
22247 149/300 Read block 3074 len 40960
22248 150/300 Read block 193 len 12288
22249 151/300 Write block 2236 len 24576 data: 4 4 4 6 7 7
22250 152/300 Read block 1833 len 40960
22251 153/300 Read block 2635 len 28672
22252 154/300 Read block 2987 len 12288
22253 155/300 Write block 1052 len 40960 data: 4 4 6 6 5 5 3 3 2 3
22254 156/300 Write block 3927 len 4096 data: 6
22255 157/300 Write block 272 len 8192 data: 4 4
22256 158/300 Read block 4458 len 8192
22257 159/300 Read block 2747 len 28672
22258 160/300 Read block 4289 len 4096
22259 161/300 Read block 1339 len 16384
22260 162/300 Read block 3011 len 40960
22261 163/300 Flush
22262 164/300 Read block 3927 len 24576
22263 165/300 Write block 4329 len 20480 data: 5 5 4 3 3
22264 166/300 Write block 2255 len 20480 data: 4 5 4 3 4
22265 167/300 Write block 2987 len 16384 data: 6 6 6 5
22266 168/300 Read block 2532 len 28672
22267 169/300 Read block 3850 len 16384
22268 170/300 Flush
22269 171/300 Read block 3339 len 12288
22270 172/300 Write block 3107 len 20480 data: 3 4 4 5 4
22271 173/300 Read block 1166 len 24576
22272 174/300 Read block 1816 len 4096
22273 175/300 Read block 707 len 24576
22274 176/300 Write block 4464 len 28672 data: 5 4 4 5 6 6 6
22275 177/300 Read block 4016 len 24576
22276 178/300 Read block 1967 len 36864
22277 179/300 Read block 4483 len 40960
22278 180/300 Read block 3184 len 20480
22279 181/300 Write block 1424 len 20480 data: 3 4 5 5 6
22280 182/300 Write block 4387 len 36864 data: 7 8 8 9 9 9 10 8 9
22281 183/300 Read block 3201 len 16384
22282 184/300 Write block 4308 len 12288 data: 7 5 4
22283 185/300 Write block 4495 len 20480 data: 6 5 6 7 8
22284 186/300 Flush
22285 187/300 Read block 498 len 8192
22286 188/300 Read block 4961 len 8192
22287 189/300 Write block 1177 len 40960 data: 6 5 5 5 6 6 6 7 7 6
22288 190/300 Write block 1694 len 16384 data: 7 9 8 9
22289 191/300 Read block 534 len 12288
22290 192/300 Flush
22291 193/300 Write block 3883 len 12288 data: 5 6 6
22292 194/300 Read block 1466 len 24576
22293 195/300 Read block 3082 len 20480
22294 196/300 Read block 2631 len 36864
22295 197/300 Read block 1344 len 28672
22296 198/300 Read block 3840 len 28672
22297 199/300 Flush
22298 200/300 Write block 1580 len 12288 data: 4 4 4
22299 201/300 Write block 837 len 4096 data: 6
22300 202/300 Read block 770 len 16384
22301 203/300 Write block 3083 len 40960 data: 4 4 4 4 4 5 4 4 4 4
22302 204/300 Read block 4944 len 8192
22303 205/300 Read block 3883 len 8192
22304 206/300 Read block 3035 len 12288
22305 207/300 Read block 2278 len 12288
22306 208/300 Write block 1032 len 32768 data: 6 6 4 7 8 7 8 7
22307 209/300 Read block 1262 len 36864
22308 210/300 Flush
22309 211/300 Read block 4335 len 32768
22310 212/300 Write block 1280 len 12288 data: 5 6 6
22311 213/300 Flush
22312 214/300 Read block 3204 len 16384
22313 215/300 Read block 701 len 40960
22314 216/300 Read block 488 len 28672
22315 217/300 Read block 860 len 28672
22316 218/300 Read block 3150 len 12288
22317 219/300 Write block 67 len 16384 data: 7 5 4 4
22318 220/300 Read block 4828 len 36864
22319 221/300 Write block 849 len 40960 data: 7 7 9 9 7 6 5 6 6 6
22320 222/300 Write block 2682 len 4096 data: 3
22321 223/300 Write block 1925 len 8192 data: 7 6
22322 224/300 Write block 2740 len 32768 data: 5 7 5 4 4 6 6 7
22323 225/300 Read block 953 len 28672
22324 226/300 Read block 2143 len 40960
22325 227/300 Read block 3120 len 28672
22326 228/300 Flush
22327 229/300 Write block 366 len 28672 data: 4 4 4 3 4 4 5
22328 230/300 Read block 1994 len 40960
22329 231/300 Write block 4701 len 20480 data: 8 7 4 2 3
22330 232/300 Write block 2195 len 16384 data: 7 7 6 5
22331 233/300 Read block 3216 len 28672
22332 234/300 Read block 438 len 28672
22333 235/300 Read block 2953 len 12288
22334 236/300 Write block 101 len 24576 data: 3 4 4 4 4 5
22335 237/300 Write block 4054 len 20480 data: 7 6 6 5 5
22336 238/300 Write block 3766 len 24576 data: 6 8 8 6 6 5
22337 239/300 Flush
22338 240/300 Write block 4242 len 20480 data: 5 6 6 7 7
22339 241/300 Read block 1683 len 24576
22340 242/300 Read block 981 len 16384
22341 243/300 Read block 3151 len 24576
22342 244/300 Read block 1923 len 40960
22343 245/300 Write block 495 len 36864 data: 5 5 5 5 7 6 6 6 6
22344 246/300 Read block 3377 len 12288
22345 247/300 Read block 1133 len 40960
22346 248/300 Read block 921 len 4096
22347 249/300 Read block 136 len 36864
22348 250/300 Write block 116 len 20480 data: 4 4 4 5 5
22349 251/300 Read block 1274 len 24576
22350 252/300 Write block 680 len 24576 data: 4 4 3 4 5 4
22351 253/300 Write block 1878 len 16384 data: 5 3 3 3
22352 254/300 Read block 1746 len 24576
22353 255/300 Read block 147 len 16384
22354 256/300 Flush
22355 257/300 Read block 4384 len 20480
22356 258/300 Write block 2010 len 12288 data: 7 6 6
22357 259/300 Write block 4275 len 32768 data: 5 4 4 4 5 6 6 6
22358 260/300 Write block 4329 len 16384 data: 6 6 5 4
22359 261/300 Read block 4629 len 24576
22360 262/300 Read block 621 len 12288
22361 263/300 Write block 1241 len 36864 data: 5 5 5 6 6 5 4 3 3
22362 264/300 Read block 4116 len 8192
22363 265/300 Write block 492 len 28672 data: 8 6 5 6 6 6 6
22364 266/300 Read block 1366 len 40960
22365 267/300 Read block 3811 len 12288
22366 268/300 Read block 1454 len 4096
22367 269/300 Read block 2522 len 32768
22368 270/300 Read block 2941 len 32768
22369 271/300 Read block 4250 len 4096
22370 272/300 Read block 3885 len 8192
22371 273/300 Write block 1036 len 12288 data: 9 8 9
22372 274/300 Write block 4947 len 36864 data: 6 6 5 5 6 4 5 4 4
22373 275/300 Read block 1262 len 8192
22374 276/300 Read block 320 len 24576
22375 277/300 Flush
22376 278/300 Read block 3315 len 4096
22377 279/300 Read block 4357 len 16384
22378 280/300 Flush
22379 281/300 Flush
22380 282/300 Flush
22381 283/300 Write block 4727 len 28672 data: 9 7 7 6 4 5 5
22382 284/300 Read block 1846 len 4096
22383 285/300 Write block 1110 len 20480 data: 6 6 4 4 3
22384 286/300 Write block 3285 len 40960 data: 6 6 5 4 3 3 3 2 2 2
22385 287/300 Flush
22386 288/300 Write block 1895 len 20480 data: 3 3 3 4 4
22387 289/300 Read block 1065 len 16384
22388 290/300 Write block 144 len 12288 data: 6 7 4
22389 291/300 Flush
22390 292/300 Read block 2525 len 20480
22391 293/300 Flush
22392 294/300 Read block 1811 len 24576
22393 295/300 Read block 2333 len 12288
22394 296/300 Read block 1877 len 16384
22395 297/300 Read block 4179 len 4096
22396 298/300 Read block 2233 len 24576
22397 299/300 Write block 3277 len 16384 data: 6 6 6 5
22398 300/300 Read block 4570 len 32768
22399 Main task has work to do, go find it
22400 got dsc Start(1)
22401 start 1
22402 Oct 13 04:46:41.074 INFO request completed, latency_us: 294, response_code: 204, uri: /start/cid/1, method: POST, req_id: ca12eec8-4cc0-41ff-8243-a9112f80c268, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
22403 Replay: started 1, returned:Ok(())
22404 ----------------------------------------------------------------
22405 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
22406 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
22407 7557 Acked 8556 Read 9 Done New Done false
22408 7558 Acked 8557 Write 3 Done New Done false
22409 7559 Acked 8558 Write 7 Done New Done false
22410 7560 Acked 8559 Read 9 Done New Done false
22411 7561 Acked 8560 Write 5 Done New Done false
22412 7562 Acked 8561 Read 9 Done New Done false
22413 7563 Acked 8562 Read 4 Done New Done false
22414 7564 Acked 8563 Write 5 Done New Done false
22415 7565 Acked 8564 Write 3 Done New Done false
22416 7566 Acked 8565 Read 4 Done New Done false
22417 7567 Acked 8566 Read 4 Done New Done false
22418 7568 Acked 8567 Read 9 Done New Done false
22419 7569 Acked 8568 Write 3 Done New Done false
22420 7570 Acked 8569 Write 5 Done New Done false
22421 7571 Acked 8570 Write 9 Done New Done false
22422 7572 Acked 8571 Flush 0 Done New Done false
22423 7573 Acked 8572 Write 2 Done New Done false
22424 7574 Acked 8573 Read 7 Done New Done false
22425 7575 Acked 8574 Flush 0 Done New Done false
22426 7576 Acked 8575 Read 6 Done New Done false
22427 7577 Acked 8576 Read 4 Done New Done false
22428 7578 Acked 8577 Read 6 Done New Done false
22429 7579 Acked 8578 Read 6 Done New Done false
22430 7580 Acked 8579 Flush 0 Done New Done false
22431 7581 Acked 8580 Read 8 Done New Done false
22432 7582 Acked 8581 Write 10 Done New Done false
22433 7583 Acked 8582 Write 4 Done New Done false
22434 7584 Acked 8583 Read 10 Done New Done false
22435 7585 Acked 8584 Write 2 Done New Done false
22436 7586 Acked 8585 Read 8 Done New Done false
22437 7587 Acked 8586 Read 9 Done New Done false
22438 7588 Acked 8587 Read 2 Done New Done false
22439 7589 Acked 8588 Flush 0 Done New Done false
22440 7590 Acked 8589 Read 5 Done New Done false
22441 7591 Acked 8590 Flush 0 Done New Done false
22442 7592 Acked 8591 Write 6 Done New Done false
22443 7593 Acked 8592 Read 1 Done New Done false
22444 7594 Acked 8593 Write 8 Done New Done false
22445 7595 Acked 8594 Write 4 Done New Done false
22446 7596 Acked 8595 Read 8 Done New Done false
22447 7597 Acked 8596 Read 6 Done New Done false
22448 7598 Acked 8597 Write 10 Done New Done false
22449 7599 Acked 8598 Read 8 Done New Done false
22450 7600 Acked 8599 Write 10 Done New Done false
22451 7601 Acked 8600 Write 9 Done New Done false
22452 7602 Acked 8601 Read 7 Done New Done false
22453 7603 Acked 8602 Read 2 Done New Done false
22454 7604 Acked 8603 Write 8 Done New Done false
22455 7605 Acked 8604 Read 3 Done New Done false
22456 7606 Acked 8605 Write 1 Done New Done false
22457 7607 Acked 8606 Write 7 Done New Done false
22458 7608 Acked 8607 Read 4 Done New Done false
22459 7609 Acked 8608 Read 4 Done New Done false
22460 7610 Acked 8609 Write 2 Done New Done false
22461 7611 Acked 8610 Write 4 Done New Done false
22462 7612 Acked 8611 Write 2 Done New Done false
22463 7613 Acked 8612 Read 3 Done New Done false
22464 7614 Acked 8613 Read 6 Done New Done false
22465 7615 Acked 8614 Read 1 Done New Done false
22466 7616 Acked 8615 Write 9 Done New Done false
22467 7617 Acked 8616 Read 8 Done New Done false
22468 7618 Acked 8617 Write 5 Done New Done false
22469 7619 Acked 8618 Write 3 Done New Done false
22470 7620 Acked 8619 Flush 0 Done New Done false
22471 7621 Acked 8620 Write 4 Done New Done false
22472 7622 Acked 8621 Write 3 Done New Done false
22473 7623 Acked 8622 Read 5 Done New Done false
22474 7624 Acked 8623 Read 5 Done New Done false
22475 7625 Acked 8624 Flush 0 Done New Done false
22476 7626 Acked 8625 Flush 0 Done New Done false
22477 7627 Acked 8626 Read 6 Done New Done false
22478 7628 Acked 8627 Write 1 Done New Done false
22479 7629 Acked 8628 Write 5 Done New Done false
22480 7630 Acked 8629 Write 10 Done New Done false
22481 7631 Acked 8630 Read 5 Done New Done false
22482 7632 Acked 8631 Flush 0 Done New Done false
22483 7633 Acked 8632 Read 4 Done New Done false
22484 7634 Acked 8633 Read 5 Done New Done false
22485 7635 Acked 8634 Read 3 Done New Done false
22486 7636 Acked 8635 Write 5 Done New Done false
22487 7637 Acked 8636 Read 3 Done New Done false
22488 7638 Acked 8637 Write 9 Done New Done false
22489 7639 Acked 8638 Flush 0 Done New Done false
22490 7640 Acked 8639 Read 8 Done New Done false
22491 7641 Acked 8640 Flush 0 Done New Done false
22492 7642 Acked 8641 Read 8 Done New Done false
22493 7643 Acked 8642 Write 7 Done New Done false
22494 7644 Acked 8643 Read 2 Done New Done false
22495 7645 Acked 8644 Write 2 Done New Done false
22496 7646 Acked 8645 Write 5 Done New Done false
22497 7647 Acked 8646 Write 8 Done New Done false
22498 7648 Acked 8647 Flush 0 Done New Done false
22499 7649 Acked 8648 Flush 0 Done New Done false
22500 7650 Acked 8649 Read 10 Done New Done false
22501 7651 Acked 8650 Read 3 Done New Done false
22502 7652 Acked 8651 Flush 0 Done New Done false
22503 7653 Acked 8652 Read 8 Done New Done false
22504 7654 Acked 8653 Read 7 Done New Done false
22505 7655 Acked 8654 Read 5 Done New Done false
22506 7656 Acked 8655 Read 1 Done New Done false
22507 7657 Acked 8656 Read 7 Done New Done false
22508 7658 Acked 8657 Read 1 Done New Done false
22509 7659 Acked 8658 Read 1 Done New Done false
22510 7660 Acked 8659 Write 10 Done New Done false
22511 7661 Acked 8660 Read 10 Done New Done false
22512 7662 Acked 8661 Write 7 Done New Done false
22513 7663 Acked 8662 Read 2 Done New Done false
22514 7664 Acked 8663 Write 5 Done New Done false
22515 7665 Acked 8664 Read 2 Done New Done false
22516 7666 Acked 8665 Write 8 Done New Done false
22517 7667 Acked 8666 Read 5 Done New Done false
22518 7668 Acked 8667 Write 10 Done New Done false
22519 7669 Acked 8668 Write 5 Done New Done false
22520 7670 Acked 8669 Read 4 Done New Done false
22521 7671 Acked 8670 Write 6 Done New Done false
22522 7672 Acked 8671 Flush 0 Done New Done false
22523 7673 Acked 8672 Read 9 Done New Done false
22524 7674 Acked 8673 Read 6 Done New Done false
22525 7675 Acked 8674 Read 3 Done New Done false
22526 7676 Acked 8675 Write 5 Done New Done false
22527 7677 Acked 8676 Read 3 Done New Done false
22528 7678 Acked 8677 Read 5 Done New Done false
22529 7679 Acked 8678 Read 1 Done New Done false
22530 7680 Acked 8679 Write 8 Done New Done false
22531 7681 Acked 8680 Write 8 Done New Done false
22532 7682 Acked 8681 Write 6 Done New Done false
22533 7683 Acked 8682 Write 1 Done New Done false
22534 7684 Acked 8683 Write 5 Done New Done false
22535 7685 Acked 8684 Read 8 Done New Done false
22536 7686 Acked 8685 Write 4 Done New Done false
22537 7687 Acked 8686 Write 3 Done New Done false
22538 7688 Acked 8687 Write 3 Done New Done false
22539 7689 Acked 8688 Write 8 Done New Done false
22540 7690 Acked 8689 Flush 0 Done New Done false
22541 7691 Acked 8690 Read 5 Done New Done false
22542 7692 Acked 8691 Read 5 Done New Done false
22543 7693 Acked 8692 Read 6 Done New Done false
22544 7694 Acked 8693 Read 6 Done New Done false
22545 7695 Acked 8694 Read 2 Done New Done false
22546 7696 Acked 8695 Read 3 Done New Done false
22547 7697 Acked 8696 Flush 0 Done New Done false
22548 7698 Acked 8697 Flush 0 Done New Done false
22549 7699 Acked 8698 Read 1 Done New Done false
22550 7700 Acked 8699 Read 2 Done New Done false
22551 7701 Acked 8700 Read 8 Done New Done false
22552 7702 Acked 8701 Read 9 Done New Done false
22553 7703 Acked 8702 Read 8 Done New Done false
22554 7704 Acked 8703 Read 1 Done New Done false
22555 7705 Acked 8704 Write 4 Done New Done false
22556 7706 Acked 8705 Read 10 Done New Done false
22557 7707 Acked 8706 Read 3 Done New Done false
22558 7708 Acked 8707 Write 6 Done New Done false
22559 7709 Acked 8708 Read 10 Done New Done false
22560 7710 Acked 8709 Read 7 Done New Done false
22561 7711 Acked 8710 Read 3 Done New Done false
22562 7712 Acked 8711 Write 10 Done New Done false
22563 7713 Acked 8712 Write 1 Done New Done false
22564 7714 Acked 8713 Write 2 Done New Done false
22565 7715 Acked 8714 Read 2 Done New Done false
22566 7716 Acked 8715 Read 7 Done New Done false
22567 7717 Acked 8716 Read 1 Done New Done false
22568 7718 Acked 8717 Read 4 Done New Done false
22569 7719 Acked 8718 Read 10 Done New Done false
22570 7720 Acked 8719 Flush 0 Done New Done false
22571 7721 Acked 8720 Read 6 Done New Done false
22572 7722 Acked 8721 Write 5 Done New Done false
22573 7723 Acked 8722 Write 5 Done New Done false
22574 7724 Acked 8723 Write 4 Done New Done false
22575 7725 Acked 8724 Read 7 Done New Done false
22576 7726 Acked 8725 Read 4 Done New Done false
22577 7727 Acked 8726 Flush 0 Done New Done false
22578 7728 Acked 8727 Read 3 Done New Done false
22579 7729 Acked 8728 Write 5 Done New Done false
22580 7730 Acked 8729 Read 6 Done New Done false
22581 7731 Acked 8730 Read 1 Done New Done false
22582 7732 Acked 8731 Read 6 Done New Done false
22583 7733 Acked 8732 Write 7 Done New Done false
22584 7734 Acked 8733 Read 6 Done New Done false
22585 7735 Acked 8734 Read 9 Done New Done false
22586 7736 Acked 8735 Read 10 Done New Done false
22587 7737 Acked 8736 Read 5 Done New Done false
22588 7738 Acked 8737 Write 5 Done New Done false
22589 7739 Acked 8738 Write 9 Done New Done false
22590 7740 Acked 8739 Read 4 Done New Done false
22591 7741 Acked 8740 Write 3 Done New Done false
22592 7742 Acked 8741 Write 5 Done New Done false
22593 7743 Acked 8742 Flush 0 Done New Done false
22594 7744 Acked 8743 Read 2 Done New Done false
22595 7745 Acked 8744 Read 2 Done New Done false
22596 7746 Acked 8745 Write 10 Done New Done false
22597 7747 Acked 8746 Write 4 Done New Done false
22598 7748 Acked 8747 Read 3 Done New Done false
22599 7749 Acked 8748 Flush 0 Done New Done false
22600 7750 Acked 8749 Write 3 Done New Done false
22601 7751 Acked 8750 Read 6 Done New Done false
22602 7752 Acked 8751 Read 5 Done New Done false
22603 7753 Acked 8752 Read 9 Done New Done false
22604 7754 Acked 8753 Read 7 Done New Done false
22605 7755 Acked 8754 Read 7 Done New Done false
22606 7756 Acked 8755 Flush 0 Done New Done false
22607 7757 Acked 8756 Write 3 Done New Done false
22608 7758 Acked 8757 Write 1 Done New Done false
22609 7759 Acked 8758 Read 4 Done New Done false
22610 7760 Acked 8759 Write 10 Done New Done false
22611 7761 Acked 8760 Read 2 Done New Done false
22612 7762 Acked 8761 Read 2 Done New Done false
22613 7763 Acked 8762 Read 3 Done New Done false
22614 7764 Acked 8763 Read 3 Done New Done false
22615 7765 Acked 8764 Write 8 Done New Done false
22616 7766 Acked 8765 Read 9 Done New Done false
22617 7767 Acked 8766 Flush 0 Done New Done false
22618 7768 Acked 8767 Read 8 Done New Done false
22619 7769 Acked 8768 Write 3 Done New Done false
22620 7770 Acked 8769 Flush 0 Done New Done false
22621 7771 Acked 8770 Read 4 Done New Done false
22622 7772 Acked 8771 Read 10 Done New Done false
22623 7773 Acked 8772 Read 7 Done New Done false
22624 7774 Acked 8773 Read 7 Done New Done false
22625 7775 Acked 8774 Read 3 Done New Done false
22626 7776 Acked 8775 Write 4 Done New Done false
22627 7777 Acked 8776 Read 9 Done New Done false
22628 7778 Acked 8777 Write 10 Done New Done false
22629 7779 Acked 8778 Write 1 Done New Done false
22630 7780 Acked 8779 Write 2 Done New Done false
22631 7781 Acked 8780 Write 8 Done New Done false
22632 7782 Acked 8781 Read 7 Done New Done false
22633 7783 Acked 8782 Read 10 Done New Done false
22634 7784 Acked 8783 Read 7 Done New Done false
22635 7785 Acked 8784 Flush 0 Done New Done false
22636 7786 Acked 8785 Write 7 Done New Done false
22637 7787 Acked 8786 Read 10 Done New Done false
22638 7788 Acked 8787 Write 5 Done New Done false
22639 7789 Acked 8788 Write 4 Done New Done false
22640 7790 Acked 8789 Read 7 Done New Done false
22641 7791 Acked 8790 Read 7 Done New Done false
22642 7792 Acked 8791 Read 3 Done New Done false
22643 7793 Acked 8792 Write 6 Done New Done false
22644 7794 Acked 8793 Write 5 Done New Done false
22645 7795 Acked 8794 Write 6 Done New Done false
22646 7796 Acked 8795 Flush 0 Done New Done false
22647 7797 Acked 8796 Write 5 Done New Done false
22648 7798 Acked 8797 Read 6 Done New Done false
22649 7799 Acked 8798 Flush 0 Done New Done false
22650 7800 Acked 8799 Read 4 Done New Done false
22651 7801 Acked 8800 Read 6 Done New Done false
22652 7802 Acked 8801 Read 10 Done New Done false
22653 7803 Acked 8802 Write 9 Done New Done false
22654 7804 Acked 8803 Read 3 Done New Done false
22655 7805 Acked 8804 Read 10 Done New Done false
22656 7806 Acked 8805 Read 1 Done New Done false
22657 7807 Acked 8806 Read 9 Done New Done false
22658 7808 Acked 8807 Write 5 Done New Done false
22659 7809 Acked 8808 Read 6 Done New Done false
22660 7810 Acked 8809 Write 6 Done New Done false
22661 7811 Acked 8810 Write 4 Done New Done false
22662 7812 Acked 8811 Read 6 Done New Done false
22663 7813 Acked 8812 Read 4 Done New Done false
22664 7814 Acked 8813 Flush 0 Done New Done false
22665 7815 Acked 8814 Read 5 Done New Done false
22666 7816 Acked 8815 Write 3 Done New Done false
22667 7817 Acked 8816 Write 8 Done New Done false
22668 7818 Acked 8817 Write 4 Done New Done false
22669 7819 Acked 8818 Read 6 Done New Done false
22670 7820 Acked 8819 Read 3 Done New Done false
22671 7821 Acked 8820 Write 9 Done New Done false
22672 7822 Acked 8821 Read 2 Done New Done false
22673 7823 Acked 8822 Write 7 Done New Done false
22674 7824 Acked 8823 Read 10 Done New Done false
22675 7825 Acked 8824 Read 3 Done New Done false
22676 7826 Acked 8825 Read 1 Done New Done false
22677 7827 Acked 8826 Read 8 Done New Done false
22678 7828 Acked 8827 Read 8 Done New Done false
22679 7829 Acked 8828 Read 1 Done New Done false
22680 7830 Acked 8829 Read 2 Done New Done false
22681 7831 Acked 8830 Write 3 Done New Done false
22682 7832 Acked 8831 Write 9 Done New Done false
22683 7833 Acked 8832 Read 2 Done New Done false
22684 7834 Acked 8833 Read 6 Done New Done false
22685 7835 Acked 8834 Flush 0 Done New Done false
22686 7836 Acked 8835 Read 1 Done New Done false
22687 7837 Acked 8836 Read 4 Done New Done false
22688 7838 Acked 8837 Flush 0 Done New Done false
22689 7839 Acked 8838 Flush 0 Done New Done false
22690 7840 Acked 8839 Flush 0 Done New Done false
22691 7841 Acked 8840 Write 7 Done New Done false
22692 7842 Acked 8841 Read 1 Done New Done false
22693 7843 Acked 8842 Write 5 Done New Done false
22694 7844 Acked 8843 Write 10 Done New Done false
22695 7845 Acked 8844 Flush 0 Done New Done false
22696 7846 Acked 8845 Write 5 Done New Done false
22697 7847 Acked 8846 Read 4 Done New Done false
22698 7848 Acked 8847 Write 3 Done New Done false
22699 7849 Acked 8848 Flush 0 Done New Done false
22700 7850 Acked 8849 Read 5 Done New Done false
22701 7851 Acked 8850 Flush 0 Done New Done false
22702 7852 Acked 8851 Read 6 Done New Done false
22703 7853 Acked 8852 Read 3 Done New Done false
22704 7854 Acked 8853 Read 4 Done New Done false
22705 7855 Acked 8854 Read 1 Done New Done false
22706 7856 Acked 8855 Read 6 Done New Done false
22707 7857 Acked 8856 Write 4 Done New Done false
22708 7858 Acked 8857 Read 8 Done New Done false
22709 STATES DS:0 DS:1 DS:2 TOTAL
22710 New 0 302 0 302
22711 Sent 0 0 0 0
22712 Done 302 0 302 604
22713 Skipped 0 0 0 0
22714 Error 0 0 0 0
22715 Last Flush: 8850 8555 8850
22716 Downstairs last five completed: 8555 8554 8553 8552 8551 8550
22717 Upstairs last five completed: 7858 7857 7856 7855 7854 7853
22718 CLIENT: Up:0 ds:302 act:2
22719 [8820] got start action
22720 [8820] I am going to restart
22721 Starting downstairs at port 8820
22722 Make output file at "/tmp/dsc/downstairs-8820.txt"
22723 [8820][1] reports Starting
22724 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1202)
22725 [8820][1] reports Running
227262023-10-13T04:46:43.209ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 1
227272023-10-13T04:46:43.209ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state Offline
227282023-10-13T04:46:43.209ZINFOcrucible: [1] client re-new 303 jobs since flush 8555 = downstairs
227292023-10-13T04:46:43.210ZINFOcrucible: [1] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
227302023-10-13T04:46:43.211ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
227312023-10-13T04:46:43.211ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
227322023-10-13T04:46:43.211ZINFOcrucible: Returning client:1 UUID:12345678-0000-0000-0000-000000008820 matches
227332023-10-13T04:46:43.211ZINFOcrucible: [1] send last flush ID to this DS: 8555
227342023-10-13T04:46:43.211ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
227352023-10-13T04:46:43.211ZINFOcrucible: [1] Replied this last flush ID: 8555
227362023-10-13T04:46:43.211ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Offline Active ds_transition to Replay
227372023-10-13T04:46:43.211ZINFOcrucible: [1] Transition from Offline to Replay
227382023-10-13T04:46:43.211ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
227392023-10-13T04:46:43.211ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Replay Active ds_transition to Active
227402023-10-13T04:46:43.211ZINFOcrucible: [1] Transition from Replay to Active
227412023-10-13T04:46:43.211ZINFOcrucible: [1] Starts cmd_loop
227422023-10-13T04:46:43.211ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
227432023-10-13T04:46:43.211ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
227442023-10-13T04:46:43.211ZINFOcrucible: Set check for repair
227452023-10-13T04:46:44.212ZINFOcrucible: Checking if live repair is needed
227462023-10-13T04:46:44.212ZINFOcrucible: No Live Repair required at this time
227472023-10-13T04:46:44.212ZWARNcrucible: [1] flow control end
22748 ----------------------------------------------------------------
22749 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
22750 Downstairs last five completed: 8858 8857 8856 8855 8854 8853
22751 Upstairs last five completed: 7859 7858 7857 7856 7855 7854
22752 CLIENT: Up:0 ds:0 act:3
22753 Replay: All jobs finished, all DS active.
22754 Main task has work to do, go find it
22755 got dsc Stop(1)
22756 stop 1
22757 Oct 13 04:46:45.078 INFO request completed, latency_us: 493, response_code: 204, uri: /stop/cid/1, method: POST, req_id: 6ede4217-d170-40b7-b457-e9989faada45, remote_addr: 127.0[8820] Got stop action so:false kr:false
22758 .0.1:52046, local_addr: 127.0.0.1:9998
22759 Oct 13 04:46:45.080 INFO request completed, latency_us: 247, response_code: 200, uri: /state/cid/1, method: GET, req_id: 16374dda-9a3e-4812-93f4-999539b80895, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
227602023-10-13T04:46:45.080ZWARNcrucible: [1] downstairs disconnected
227612023-10-13T04:46:45.080ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
227622023-10-13T04:46:45.080ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8820 closed looper = 1
227632023-10-13T04:46:45.080ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:false
227642023-10-13T04:46:45.080ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
227652023-10-13T04:46:45.081ZINFOcrucible: [1] 127.0.0.1:8820 task reports offline
227662023-10-13T04:46:45.081ZWARNcrucible: [1] pm_task rx.recv() is None
22767 [8820] Exited with: ExitStatus(unix_wait_status(9))
22768 [8820][1] reports Exit
22769 Oct 13 04:46:49.082 INFO request completed, latency_us: 254, response_code: 200, uri: /state/cid/1, method: GET, req_id: b4ee540f-3b83-4bf8-a2b6-8fdd253da0e7, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
22770 001/300 Write block 3025 len 16384 data: 7 7 7 7
22771 002/300 Read block 4228 len 4096
22772 003/300 Read block 2295 len 32768
22773 004/300 Read block 1392 len 12288
22774 005/300 Read block 967 len 16384
22775 006/300 Read block 1079 len 32768
22776 007/300 Read block 2222 len 36864
22777 008/300 Read block 1579 len 40960
22778 009/300 Read block 4289 len 36864
22779 010/300 Read block 3685 len 8192
22780 011/300 Write block 1355 len 40960 data: 4 4 6 3 3 4 4 6 6 6
22781 012/300 Read block 3719 len 40960
22782 013/300 Read block 3032 len 24576
22783 014/300 Read block 799 len 20480
22784 015/300 Write block 1786 len 8192 data: 6 6
22785 016/300 Read block 3275 len 32768
22786 017/300 Write block 1297 len 32768 data: 6 6 5 6 6 4 4 5
22787 018/300 Write block 947 len 8192 data: 4 5
22788 019/300 Read block 3853 len 40960
22789 020/300 Write block 1111 len 40960 data: 7 5 5 4 4 3 3 4 4 5
22790 021/300 Write block 4585 len 40960 data: 5 6 5 5 5 5 4 5 5 4
22791 022/300 Write block 1056 len 40960 data: 6 6 4 4 3 4 4 4 4 3
22792 023/300 Write block 2460 len 20480 data: 5 5 5 3 2
22793 024/300 Read block 1658 len 36864
22794 025/300 Flush
22795 026/300 Write block 32 len 28672 data: 4 7 7 7 5 4 5
22796 027/300 Write block 802 len 8192 data: 5 5
22797 028/300 Read block 4810 len 24576
22798 029/300 Read block 1349 len 40960
22799 030/300 Write block 3159 len 40960 data: 6 6 6 6 8 4 5 6 8 8
22800 031/300 Write block 2486 len 28672 data: 5 6 6 6 8 7 8
22801 032/300 Read block 4458 len 12288
22802 033/300 Read block 2253 len 20480
22803 034/300 Write block 2172 len 36864 data: 5 6 4 3 4 5 4 4 4
22804 035/300 Read block 3100 len 4096
22805 036/300 Read block 1014 len 28672
22806 037/300 Write block 1029 len 36864 data: 6 6 6 7 7 5 8 10 9
22807 038/300 Read block 1849 len 36864
22808 039/300 Write block 3512 len 40960 data: 6 6 4 4 6 4 4 4 5 6
22809 040/300 Read block 1390 len 8192
22810 041/300 Write block 3221 len 32768 data: 7 6 5 5 5 4 4 4
22811 042/300 Read block 3022 len 16384
22812 043/300 Read block 2978 len 16384
22813 044/300 Read block 2180 len 20480
22814 045/300 Write block 4131 len 12288 data: 4 5 6
22815 046/300 Write block 4849 len 36864 data: 4 5 6 5 4 3 3 3 4
22816 047/300 Read block 4022 len 20480
22817 048/300 Write block 804 len 8192 data: 4 5
22818 049/300 Read block 1790 len 8192
22819 050/300 Read block 1407 len 24576
22820 051/300 Read block 2347 len 24576
22821 052/300 Read block 3208 len 12288
22822 053/300 Read block 4273 len 16384
22823 054/300 Read block 1544 len 16384
22824 055/300 Read block 1170 len 8192
22825 056/300 Read block 2692 len 32768
22826 057/300 Write block 3303 len 12288 data: 4 6 4
22827 058/300 Write block 3848 len 24576 data: 7 8 7 6 9 8
22828 059/300 Write block 553 len 32768 data: 5 6 5 6 7 6 6 6
22829 060/300 Write block 4262 len 16384 data: 4 5 5 5
22830 061/300 Write block 1715 len 32768 data: 7 9 6 3 3 4 4 6
22831 062/300 Read block 3482 len 8192
22832 063/300 Write block 4933 len 20480 data: 5 6 5 5 4
22833 064/300 Read block 3278 len 12288
22834 065/300 Read block 2576 len 16384
22835 066/300 Write block 3179 len 36864 data: 6 5 4 4 4 6 6 6 8
22836 067/300 Read block 4911 len 40960
22837 068/300 Read block 4950 len 20480
22838 069/300 Write block 3849 len 32768 data: 9 8 7 10 9 8 6 6
22839 070/300 Flush
22840 071/300 Read block 1128 len 16384
22841 072/300 Read block 3349 len 4096
22842 073/300 Write block 3171 len 4096 data: 9
22843 074/300 Write block 4375 len 28672 data: 5 4 5 3 3 2 3
22844 075/300 Read block 2022 len 40960
22845 076/300 Write block 4773 len 40960 data: 3 3 3 4 6 7 8 8 7 9
22846 077/300 Read block 2198 len 36864
22847 078/300 Flush
22848 079/300 Write block 2248 len 16384 data: 7 6 6 5
22849 080/300 Write block 650 len 12288 data: 4 3 3
22850 081/300 Write block 1499 len 36864 data: 7 6 5 5 5 5 5 8 8
22851 082/300 Read block 3334 len 8192
22852 083/300 Read block 2070 len 24576
22853 084/300 Read block 4035 len 4096
22854 085/300 Read block 510 len 32768
22855 086/300 Read block 3170 len 8192
22856 087/300 Read block 3641 len 8192
22857 088/300 Read block 2739 len 16384
22858 089/300 Write block 3295 len 40960 data: 3 5 5 7 6 5 4 4 5 7
22859 090/300 Read block 1775 len 24576
22860 091/300 Read block 1775 len 32768
22861 092/300 Read block 3905 len 4096
22862 093/300 Write block 1714 len 4096 data: 7
22863 094/300 Read block 3324 len 36864
22864 095/300 Write block 2257 len 8192 data: 5 4
22865 096/300 Write block 2641 len 40960 data: 2 2 3 3 3 3 3 3 3 3
22866 097/300 Read block 2933 len 36864
22867 098/300 Read block 629 len 16384
22868 099/300 Read block 2448 len 40960
22869 100/300 Flush
22870 101/300 Read block 3836 len 16384
22871 102/300 Read block 3117 len 24576
22872 103/300 Read block 929 len 36864
22873 104/300 Read block 1428 len 20480
22874 105/300 Read block 1398 len 32768
22875 106/300 Read block 1525 len 12288
22876 107/300 Read block 3843 len 32768
22877 108/300 Read block 1139 len 16384
22878 109/300 Write block 2555 len 20480 data: 7 8 8 8 6
22879 110/300 Read block 499 len 28672
22880 111/300 Read block 835 len 12288
22881 112/300 Write block 2934 len 8192 data: 3 2
22882 113/300 Write block 4213 len 32768 data: 6 7 7 8 6 5 4 4
22883 114/300 Read block 3362 len 4096
22884 115/300 Write block 3543 len 24576 data: 3 3 5 5 5 6
22885 116/300 Write block 491 len 4096 data: 7
22886 117/300 Read block 731 len 40960
22887 118/300 Read block 3403 len 12288
22888 119/300 Read block 4729 len 20480
22889 120/300 Write block 2412 len 12288 data: 5 4 5
22890 121/300 Flush
22891 122/300 Write block 3052 len 28672 data: 7 5 4 5 4 5 5
22892 123/300 Read block 2026 len 20480
22893 124/300 Flush
22894 125/300 Write block 4561 len 8192 data: 4 3
22895 126/300 Read block 2804 len 12288
22896 127/300 Read block 3699 len 36864
22897 128/300 Flush
22898 129/300 Read block 512 len 36864
22899 130/300 Read block 4268 len 4096
22900 131/300 Read block 1965 len 32768
22901 132/300 Write block 671 len 24576 data: 4 5 4 5 5 6
22902 133/300 Write block 3285 len 32768 data: 7 7 6 5 4 4 4 3
22903 134/300 Write block 772 len 40960 data: 4 3 3 5 5 5 4 3 4 4
22904 135/300 Read block 4342 len 32768
22905 136/300 Read block 723 len 4096
22906 137/300 Write block 288 len 40960 data: 5 7 7 7 9 9 9 8 9 10
22907 138/300 Flush
22908 139/300 Write block 4661 len 20480 data: 7 8 9 9 7
22909 140/300 Write block 0 len 32768 data: 2 2 3 5 5 6 8 7
22910 141/300 Read block 4520 len 20480
22911 142/300 Write block 3966 len 16384 data: 5 5 5 5
22912 143/300 Read block 3342 len 36864
22913 144/300 Read block 2021 len 16384
22914 145/300 Write block 4100 len 36864 data: 7 7 7 6 5 4 4 5 4
22915 146/300 Flush
22916 147/300 Read block 2338 len 16384
22917 148/300 Read block 4213 len 32768
22918 149/300 Write block 713 len 8192 data: 3 5
22919 150/300 Read block 323 len 8192
22920 151/300 Read block 3123 len 40960
22921 152/300 Read block 3226 len 28672
22922 153/300 Flush
22923 154/300 Read block 3745 len 4096
22924 155/300 Read block 4453 len 8192
22925 156/300 Read block 3742 len 12288
22926 157/300 Read block 4271 len 20480
22927 158/300 Write block 4030 len 28672 data: 4 4 5 3 3 4 4
22928 159/300 Flush
22929 160/300 Read block 4157 len 20480
22930 161/300 Read block 4216 len 24576
22931 162/300 Write block 4685 len 28672 data: 6 6 8 8 5 6 6
22932 163/300 Write block 3461 len 36864 data: 4 3 4 6 7 7 5 6 4
22933 164/300 Read block 2660 len 28672
22934 165/300 Write block 3863 len 40960 data: 7 6 6 7 6 5 3 2 2 4
22935 166/300 Read block 4760 len 40960
22936 167/300 Write block 963 len 36864 data: 4 4 4 4 5 7 8 9 6
22937 168/300 Write block 4759 len 28672 data: 8 7 6 6 5 4 5
22938 169/300 Write block 3278 len 32768 data: 7 7 6 5 4 4 8 8
22939 170/300 Write block 4409 len 32768 data: 4 6 6 6 8 7 6 6
22940 171/300 Write block 1727 len 8192 data: 4 4
22941 172/300 Write block 1819 len 20480 data: 3 3 5 4 6
22942 173/300 Write block 566 len 24576 data: 3 4 4 4 4 5
22943 174/300 Read block 3537 len 32768
22944 175/300 Write block 975 len 40960 data: 11 10 8 6 6 5 6 8 7 6
22945 176/300 Read block 4191 len 4096
22946 177/300 Read block 2907 len 24576
22947 178/300 Write block 1368 len 12288 data: 5 4 4
22948 179/300 Write block 576 len 4096 data: 6
22949 180/300 Read block 4704 len 36864
22950 181/300 Write block 1658 len 40960 data: 6 5 6 7 5 6 7 3 3 6
22951 182/300 Write block 3037 len 16384 data: 7 7 7 8
22952 183/300 Read block 796 len 36864
22953 184/300 Read block 3452 len 40960
22954 185/300 Read block 415 len 28672
22955 186/300 Read block 1431 len 12288
22956 187/300 Write block 684 len 8192 data: 6 5
22957 188/300 Read block 4618 len 16384
22958 189/300 Write block 3089 len 36864 data: 5 5 5 5 4 4 4 4 3
22959 190/300 Write block 2532 len 4096 data: 5
22960 191/300 Write block 926 len 36864 data: 5 6 4 6 6 4 2 3 4
22961 192/300 Write block 891 len 40960 data: 2 2 2 2 2 2 3 6 6 9
22962 193/300 Read block 2131 len 12288
22963 194/300 Write block 1486 len 36864 data: 2 4 7 7 8 9 7 7 7
22964 195/300 Write block 4955 len 24576 data: 5 4 4 4 3 2
22965 196/300 Flush
22966 197/300 Write block 1394 len 36864 data: 5 6 6 7 7 9 8 7 7
22967 198/300 Write block 1529 len 16384 data: 3 6 8 8
22968 199/300 Write block 1744 len 28672 data: 5 7 6 7 6 6 5
22969 200/300 Read block 1448 len 20480
22970 201/300 Read block 3599 len 16384
22971 202/300 Read block 2246 len 24576
22972 203/300 Write block 1425 len 8192 data: 5 6
22973 204/300 Write block 3365 len 28672 data: 6 6 5 5 5 7 6
22974 205/300 Read block 947 len 16384
22975 206/300 Write block 3436 len 28672 data: 5 5 4 3 4 4 3
22976 207/300 Flush
22977 208/300 Read block 4793 len 8192
22978 209/300 Write block 4122 len 8192 data: 4 5
22979 210/300 Flush
22980 211/300 Read block 648 len 28672
22981 212/300 Read block 686 len 32768
22982 213/300 Write block 4017 len 28672 data: 6 7 9 8 9 9 9
22983 214/300 Read block 1546 len 16384
22984 215/300 Write block 187 len 20480 data: 5 5 5 7 7
22985 216/300 Write block 3179 len 28672 data: 7 6 5 5 5 7 7
22986 217/300 Read block 1952 len 32768
22987 218/300 Read block 1493 len 24576
22988 219/300 Read block 2289 len 4096
22989 220/300 Read block 2482 len 28672
22990 221/300 Read block 3496 len 32768
22991 222/300 Write block 2777 len 16384 data: 6 5 2 2
22992 223/300 Write block 637 len 28672 data: 4 8 7 6 7 8 7
22993 224/300 Write block 763 len 4096 data: 5
22994 225/300 Read block 239 len 8192
22995 226/300 Read block 914 len 12288
22996 227/300 Write block 2950 len 28672 data: 7 6 8 6 5 5 6
22997 228/300 Read block 4279 len 24576
22998 229/300 Read block 1434 len 16384
22999 230/300 Read block 3757 len 8192
23000 231/300 Write block 846 len 36864 data: 8 8 8 8 8 10 10 8 7
23001 232/300 Read block 4810 len 12288
23002 233/300 Read block 4864 len 40960
23003 234/300 Write block 2131 len 8192 data: 3 4
23004 235/300 Write block 489 len 32768 data: 7 7 8 9 7 6 7 7
23005 236/300 Flush
23006 237/300 Write block 3060 len 36864 data: 5 8 6 4 6 5 6 4 5
23007 238/300 Write block 2592 len 8192 data: 3 5
23008 239/300 Read block 481 len 32768
23009 240/300 Write block 4268 len 12288 data: 4 3 2
23010 241/300 Read block 4572 len 8192
23011 242/300 Read block 596 len 20480
23012 243/300 Write block 2094 len 28672 data: 5 6 7 9 9 7 7
23013 244/300 Read block 168 len 4096
23014 245/300 Write block 2063 len 16384 data: 6 6 6 5
23015 246/300 Write block 1071 len 16384 data: 5 6 4 5
23016 247/300 Write block 3715 len 12288 data: 8 4 4
23017 248/300 Read block 4637 len 12288
23018 249/300 Read block 998 len 28672
23019 250/300 Read block 3517 len 24576
23020 251/300 Write block 4161 len 24576 data: 8 9 10 8 7 7
23021 252/300 Read block 4359 len 40960
23022 253/300 Read block 12 len 4096
23023 254/300 Flush
23024 255/300 Read block 727 len 4096
23025 256/300 Write block 2699 len 8192 data: 7 8
23026 257/300 Read block 441 len 20480
23027 258/300 Write block 1307 len 20480 data: 7 6 6 6 6
23028 259/300 Read block 994 len 32768
23029 260/300 Read block 2154 len 28672
23030 261/300 Write block 2649 len 20480 data: 4 4 2 3 4
23031 262/300 Read block 1721 len 32768
23032 263/300 Write block 4993 len 8192 data: 4 4
23033 264/300 Read block 4693 len 16384
23034 265/300 Write block 612 len 32768 data: 10 9 8 8 8 8 7 9
23035 266/300 Read block 3792 len 28672
23036 267/300 Write block 4443 len 32768 data: 8 7 6 6 6 4 5 4
23037 268/300 Write block 3125 len 12288 data: 6 5 7
23038 269/300 Write block 1301 len 16384 data: 7 5 5 6
23039 270/300 Read block 2583 len 32768
23040 271/300 Write block 3782 len 16384 data: 8 7 6 5
23041 272/300 Read block 634 len 4096
23042 273/300 Read block 368 len 20480
23043 274/300 Write block 4066 len 40960 data: 5 6 7 8 6 7 7 6 6 5
23044 275/300 Read block 2307 len 36864
23045 276/300 Read block 971 len 24576
23046 277/300 Read block 2917 len 36864
23047 278/300 Write block 2864 len 12288 data: 8 7 8
23048 279/300 Read block 2204 len 36864
23049 280/300 Read block 652 len 4096
23050 281/300 Write block 3118 len 40960 data: 7 7 6 6 6 7 6 7 6 8
23051 282/300 Read block 2969 len 40960
23052 283/300 Write block 2499 len 8192 data: 5 4
23053 284/300 Read block 1090 len 16384
23054 285/300 Write block 4068 len 24576 data: 8 9 7 8 8 7
23055 286/300 Read block 3518 len 12288
23056 287/300 Read block 4109 len 32768
23057 288/300 Read block 577 len 8192
23058 289/300 Flush
23059 290/300 Write block 4542 len 20480 data: 11 9 8 6 6
23060 291/300 Read block 2867 len 16384
23061 292/300 Read block 2008 len 32768
23062 293/300 Write block 2284 len 12288 data: 4 5 5
23063 294/300 Write block 1922 len 20480 data: 8 7 7 8 7
23064 295/300 Write block 4418 len 40960 data: 6 4 4 4 6 6 8 8 8 8
23065 296/300 Flush
23066 297/300 Read block 2968 len 8192
23067 298/300 Read block 2749 len 32768
23068 299/300 Write block 4731 len 24576 data: 5 6 6 5 5 4
23069 300/300 Write block 1447 len 20480 data: 3 3 3 4 5
23070 Main task has work to do, go find it
23071 got dsc Start(1)
23072 start 1
23073 Oct 13 04:46:50.013 INFO request completed, latency_us: 336, response_code: 204, uri: /start/cid/1, method: POST, req_id: f6e3b93a-6fa8-4c9d-b678-919f59f049ad, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
23074 Replay: started 1, returned:Ok(())
23075 ----------------------------------------------------------------
23076 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
23077 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
23078 7860 Acked 8859 Write 4 Done New Done false
23079 7861 Acked 8860 Read 1 Done New Done false
23080 7862 Acked 8861 Read 8 Done New Done false
23081 7863 Acked 8862 Read 3 Done New Done false
23082 7864 Acked 8863 Read 4 Done New Done false
23083 7865 Acked 8864 Read 8 Done New Done false
23084 7866 Acked 8865 Read 9 Done New Done false
23085 7867 Acked 8866 Read 10 Done New Done false
23086 7868 Acked 8867 Read 9 Done New Done false
23087 7869 Acked 8868 Read 2 Done New Done false
23088 7870 Acked 8869 Write 10 Done New Done false
23089 7871 Acked 8870 Read 10 Done New Done false
23090 7872 Acked 8871 Read 6 Done New Done false
23091 7873 Acked 8872 Read 5 Done New Done false
23092 7874 Acked 8873 Write 2 Done New Done false
23093 7875 Acked 8874 Read 8 Done New Done false
23094 7876 Acked 8875 Write 8 Done New Done false
23095 7877 Acked 8876 Write 2 Done New Done false
23096 7878 Acked 8877 Read 10 Done New Done false
23097 7879 Acked 8878 Write 10 Done New Done false
23098 7880 Acked 8879 Write 10 Done New Done false
23099 7881 Acked 8880 Write 10 Done New Done false
23100 7882 Acked 8881 Write 5 Done New Done false
23101 7883 Acked 8882 Read 9 Done New Done false
23102 7884 Acked 8883 Flush 0 Done New Done false
23103 7885 Acked 8884 Write 7 Done New Done false
23104 7886 Acked 8885 Write 2 Done New Done false
23105 7887 Acked 8886 Read 6 Done New Done false
23106 7888 Acked 8887 Read 10 Done New Done false
23107 7889 Acked 8888 Write 10 Done New Done false
23108 7890 Acked 8889 Write 7 Done New Done false
23109 7891 Acked 8890 Read 3 Done New Done false
23110 7892 Acked 8891 Read 5 Done New Done false
23111 7893 Acked 8892 Write 9 Done New Done false
23112 7894 Acked 8893 Read 1 Done New Done false
23113 7895 Acked 8894 Read 7 Done New Done false
23114 7896 Acked 8895 Write 9 Done New Done false
23115 7897 Acked 8896 Read 9 Done New Done false
23116 7898 Acked 8897 Write 10 Done New Done false
23117 7899 Acked 8898 Read 2 Done New Done false
23118 7900 Acked 8899 Write 8 Done New Done false
23119 7901 Acked 8900 Read 4 Done New Done false
23120 7902 Acked 8901 Read 4 Done New Done false
23121 7903 Acked 8902 Read 5 Done New Done false
23122 7904 Acked 8903 Write 3 Done New Done false
23123 7905 Acked 8904 Write 9 Done New Done false
23124 7906 Acked 8905 Read 5 Done New Done false
23125 7907 Acked 8906 Write 2 Done New Done false
23126 7908 Acked 8907 Read 2 Done New Done false
23127 7909 Acked 8908 Read 6 Done New Done false
23128 7910 Acked 8909 Read 6 Done New Done false
23129 7911 Acked 8910 Read 3 Done New Done false
23130 7912 Acked 8911 Read 4 Done New Done false
23131 7913 Acked 8912 Read 4 Done New Done false
23132 7914 Acked 8913 Read 2 Done New Done false
23133 7915 Acked 8914 Read 8 Done New Done false
23134 7916 Acked 8915 Write 3 Done New Done false
23135 7917 Acked 8916 Write 6 Done New Done false
23136 7918 Acked 8917 Write 8 Done New Done false
23137 7919 Acked 8918 Write 4 Done New Done false
23138 7920 Acked 8919 Write 8 Done New Done false
23139 7921 Acked 8920 Read 2 Done New Done false
23140 7922 Acked 8921 Write 5 Done New Done false
23141 7923 Acked 8922 Read 3 Done New Done false
23142 7924 Acked 8923 Read 4 Done New Done false
23143 7925 Acked 8924 Write 9 Done New Done false
23144 7926 Acked 8925 Read 10 Done New Done false
23145 7927 Acked 8926 Read 5 Done New Done false
23146 7928 Acked 8927 Write 8 Done New Done false
23147 7929 Acked 8928 Flush 0 Done New Done false
23148 7930 Acked 8929 Read 4 Done New Done false
23149 7931 Acked 8930 Read 1 Done New Done false
23150 7932 Acked 8931 Write 1 Done New Done false
23151 7933 Acked 8932 Write 7 Done New Done false
23152 7934 Acked 8933 Read 10 Done New Done false
23153 7935 Acked 8934 Write 10 Done New Done false
23154 7936 Acked 8935 Read 9 Done New Done false
23155 7937 Acked 8936 Flush 0 Done New Done false
23156 7938 Acked 8937 Write 4 Done New Done false
23157 7939 Acked 8938 Write 3 Done New Done false
23158 7940 Acked 8939 Write 9 Done New Done false
23159 7941 Acked 8940 Read 2 Done New Done false
23160 7942 Acked 8941 Read 6 Done New Done false
23161 7943 Acked 8942 Read 1 Done New Done false
23162 7944 Acked 8943 Read 8 Done New Done false
23163 7945 Acked 8944 Read 2 Done New Done false
23164 7946 Acked 8945 Read 2 Done New Done false
23165 7947 Acked 8946 Read 4 Done New Done false
23166 7948 Acked 8947 Write 10 Done New Done false
23167 7949 Acked 8948 Read 6 Done New Done false
23168 7950 Acked 8949 Read 8 Done New Done false
23169 7951 Acked 8950 Read 1 Done New Done false
23170 7952 Acked 8951 Write 1 Done New Done false
23171 7953 Acked 8952 Read 9 Done New Done false
23172 7954 Acked 8953 Write 2 Done New Done false
23173 7955 Acked 8954 Write 10 Done New Done false
23174 7956 Acked 8955 Read 9 Done New Done false
23175 7957 Acked 8956 Read 4 Done New Done false
23176 7958 Acked 8957 Read 10 Done New Done false
23177 7959 Acked 8958 Flush 0 Done New Done false
23178 7960 Acked 8959 Read 4 Done New Done false
23179 7961 Acked 8960 Read 6 Done New Done false
23180 7962 Acked 8961 Read 9 Done New Done false
23181 7963 Acked 8962 Read 5 Done New Done false
23182 7964 Acked 8963 Read 8 Done New Done false
23183 7965 Acked 8964 Read 3 Done New Done false
23184 7966 Acked 8965 Read 8 Done New Done false
23185 7967 Acked 8966 Flush 0 Done New Done false
23186 7968 Acked 8967 Read 4 Done New Done false
23187 7969 Acked 8968 Write 5 Done New Done false
23188 7970 Acked 8969 Read 7 Done New Done false
23189 7971 Acked 8970 Read 3 Done New Done false
23190 7972 Acked 8971 Write 2 Done New Done false
23191 7973 Acked 8972 Write 8 Done New Done false
23192 7974 Acked 8973 Read 1 Done New Done false
23193 7975 Acked 8974 Write 6 Done New Done false
23194 7976 Acked 8975 Write 1 Done New Done false
23195 7977 Acked 8976 Read 10 Done New Done false
23196 7978 Acked 8977 Read 3 Done New Done false
23197 7979 Acked 8978 Read 5 Done New Done false
23198 7980 Acked 8979 Write 3 Done New Done false
23199 7981 Acked 8980 Flush 0 Done New Done false
23200 7982 Acked 8981 Write 7 Done New Done false
23201 7983 Acked 8982 Read 5 Done New Done false
23202 7984 Acked 8983 Flush 0 Done New Done false
23203 7985 Acked 8984 Write 2 Done New Done false
23204 7986 Acked 8985 Read 3 Done New Done false
23205 7987 Acked 8986 Read 9 Done New Done false
23206 7988 Acked 8987 Flush 0 Done New Done false
23207 7989 Acked 8988 Read 9 Done New Done false
23208 7990 Acked 8989 Read 1 Done New Done false
23209 7991 Acked 8990 Read 8 Done New Done false
23210 7992 Acked 8991 Write 6 Done New Done false
23211 7993 Acked 8992 Write 8 Done New Done false
23212 7994 Acked 8993 Write 10 Done New Done false
23213 7995 Acked 8994 Read 8 Done New Done false
23214 7996 Acked 8995 Read 1 Done New Done false
23215 7997 Acked 8996 Write 10 Done New Done false
23216 7998 Acked 8997 Flush 0 Done New Done false
23217 7999 Acked 8998 Write 5 Done New Done false
23218 8000 Acked 8999 Write 8 Done New Done false
23219 8001 Acked 9000 Read 5 Done New Done false
23220 8002 Acked 9001 Write 4 Done New Done false
23221 8003 Acked 9002 Read 9 Done New Done false
23222 8004 Acked 9003 Read 4 Done New Done false
23223 8005 Acked 9004 Write 9 Done New Done false
23224 8006 Acked 9005 Flush 0 Done New Done false
23225 8007 Acked 9006 Read 4 Done New Done false
23226 8008 Acked 9007 Read 8 Done New Done false
23227 8009 Acked 9008 Write 2 Done New Done false
23228 8010 Acked 9009 Read 2 Done New Done false
23229 8011 Acked 9010 Read 10 Done New Done false
23230 8012 Acked 9011 Read 7 Done New Done false
23231 8013 Acked 9012 Flush 0 Done New Done false
23232 8014 Acked 9013 Read 1 Done New Done false
23233 8015 Acked 9014 Read 2 Done New Done false
23234 8016 Acked 9015 Read 3 Done New Done false
23235 8017 Acked 9016 Read 5 Done New Done false
23236 8018 Acked 9017 Write 7 Done New Done false
23237 8019 Acked 9018 Flush 0 Done New Done false
23238 8020 Acked 9019 Read 5 Done New Done false
23239 8021 Acked 9020 Read 6 Done New Done false
23240 8022 Acked 9021 Write 7 Done New Done false
23241 8023 Acked 9022 Write 9 Done New Done false
23242 8024 Acked 9023 Read 7 Done New Done false
23243 8025 Acked 9024 Write 10 Done New Done false
23244 8026 Acked 9025 Read 10 Done New Done false
23245 8027 Acked 9026 Write 9 Done New Done false
23246 8028 Acked 9027 Write 7 Done New Done false
23247 8029 Acked 9028 Write 8 Done New Done false
23248 8030 Acked 9029 Write 8 Done New Done false
23249 8031 Acked 9030 Write 2 Done New Done false
23250 8032 Acked 9031 Write 5 Done New Done false
23251 8033 Acked 9032 Write 6 Done New Done false
23252 8034 Acked 9033 Read 8 Done New Done false
23253 8035 Acked 9034 Write 10 Done New Done false
23254 8036 Acked 9035 Read 1 Done New Done false
23255 8037 Acked 9036 Read 6 Done New Done false
23256 8038 Acked 9037 Write 3 Done New Done false
23257 8039 Acked 9038 Write 1 Done New Done false
23258 8040 Acked 9039 Read 9 Done New Done false
23259 8041 Acked 9040 Write 10 Done New Done false
23260 8042 Acked 9041 Write 4 Done New Done false
23261 8043 Acked 9042 Read 9 Done New Done false
23262 8044 Acked 9043 Read 10 Done New Done false
23263 8045 Acked 9044 Read 7 Done New Done false
23264 8046 Acked 9045 Read 3 Done New Done false
23265 8047 Acked 9046 Write 2 Done New Done false
23266 8048 Acked 9047 Read 4 Done New Done false
23267 8049 Acked 9048 Write 9 Done New Done false
23268 8050 Acked 9049 Write 1 Done New Done false
23269 8051 Acked 9050 Write 9 Done New Done false
23270 8052 Acked 9051 Write 10 Done New Done false
23271 8053 Acked 9052 Read 3 Done New Done false
23272 8054 Acked 9053 Write 9 Done New Done false
23273 8055 Acked 9054 Write 6 Done New Done false
23274 8056 Acked 9055 Flush 0 Done New Done false
23275 8057 Acked 9056 Write 9 Done New Done false
23276 8058 Acked 9057 Write 4 Done New Done false
23277 8059 Acked 9058 Write 7 Done New Done false
23278 8060 Acked 9059 Read 5 Done New Done false
23279 8061 Acked 9060 Read 4 Done New Done false
23280 8062 Acked 9061 Read 6 Done New Done false
23281 8063 Acked 9062 Write 2 Done New Done false
23282 8064 Acked 9063 Write 7 Done New Done false
23283 8065 Acked 9064 Read 4 Done New Done false
23284 8066 Acked 9065 Write 7 Done New Done false
23285 8067 Acked 9066 Flush 0 Done New Done false
23286 8068 Acked 9067 Read 2 Done New Done false
23287 8069 Acked 9068 Write 2 Done New Done false
23288 8070 Acked 9069 Flush 0 Done New Done false
23289 8071 Acked 9070 Read 7 Done New Done false
23290 8072 Acked 9071 Read 8 Done New Done false
23291 8073 Acked 9072 Write 7 Done New Done false
23292 8074 Acked 9073 Read 4 Done New Done false
23293 8075 Acked 9074 Write 5 Done New Done false
23294 8076 Acked 9075 Write 7 Done New Done false
23295 8077 Acked 9076 Read 8 Done New Done false
23296 8078 Acked 9077 Read 6 Done New Done false
23297 8079 Acked 9078 Read 1 Done New Done false
23298 8080 Acked 9079 Read 7 Done New Done false
23299 8081 Acked 9080 Read 8 Done New Done false
23300 8082 Acked 9081 Write 4 Done New Done false
23301 8083 Acked 9082 Write 7 Done New Done false
23302 8084 Acked 9083 Write 1 Done New Done false
23303 8085 Acked 9084 Read 2 Done New Done false
23304 8086 Acked 9085 Read 3 Done New Done false
23305 8087 Acked 9086 Write 7 Done New Done false
23306 8088 Acked 9087 Read 6 Done New Done false
23307 8089 Acked 9088 Read 4 Done New Done false
23308 8090 Acked 9089 Read 2 Done New Done false
23309 8091 Acked 9090 Write 9 Done New Done false
23310 8092 Acked 9091 Read 3 Done New Done false
23311 8093 Acked 9092 Read 10 Done New Done false
23312 8094 Acked 9093 Write 2 Done New Done false
23313 8095 Acked 9094 Write 8 Done New Done false
23314 8096 Acked 9095 Flush 0 Done New Done false
23315 8097 Acked 9096 Write 9 Done New Done false
23316 8098 Acked 9097 Write 2 Done New Done false
23317 8099 Acked 9098 Read 8 Done New Done false
23318 8100 Acked 9099 Write 3 Done New Done false
23319 8101 Acked 9100 Read 2 Done New Done false
23320 8102 Acked 9101 Read 5 Done New Done false
23321 8103 Acked 9102 Write 7 Done New Done false
23322 8104 Acked 9103 Read 1 Done New Done false
23323 8105 Acked 9104 Write 4 Done New Done false
23324 8106 Acked 9105 Write 4 Done New Done false
23325 8107 Acked 9106 Write 3 Done New Done false
23326 8108 Acked 9107 Read 3 Done New Done false
23327 8109 Acked 9108 Read 7 Done New Done false
23328 8110 Acked 9109 Read 6 Done New Done false
23329 8111 Acked 9110 Write 6 Done New Done false
23330 8112 Acked 9111 Read 10 Done New Done false
23331 8113 Acked 9112 Read 1 Done New Done false
23332 8114 Acked 9113 Flush 0 Done New Done false
23333 8115 Acked 9114 Read 1 Done New Done false
23334 8116 Acked 9115 Write 2 Done New Done false
23335 8117 Acked 9116 Read 5 Done New Done false
23336 8118 Acked 9117 Write 5 Done New Done false
23337 8119 Acked 9118 Read 8 Done New Done false
23338 8120 Acked 9119 Read 7 Done New Done false
23339 8121 Acked 9120 Write 5 Done New Done false
23340 8122 Acked 9121 Read 8 Done New Done false
23341 8123 Acked 9122 Write 2 Done New Done false
23342 8124 Acked 9123 Read 4 Done New Done false
23343 8125 Acked 9124 Write 8 Done New Done false
23344 8126 Acked 9125 Read 7 Done New Done false
23345 8127 Acked 9126 Write 8 Done New Done false
23346 8128 Acked 9127 Write 3 Done New Done false
23347 8129 Acked 9128 Write 4 Done New Done false
23348 8130 Acked 9129 Read 8 Done New Done false
23349 8131 Acked 9130 Write 4 Done New Done false
23350 8132 Acked 9131 Read 1 Done New Done false
23351 8133 Acked 9132 Read 5 Done New Done false
23352 8134 Acked 9133 Write 10 Done New Done false
23353 8135 Acked 9134 Read 9 Done New Done false
23354 8136 Acked 9135 Flush 0 Done New Done false
23355 8137 Acked 9136 Read 6 Done New Done false
23356 8138 Acked 9137 Read 9 Done New Done false
23357 8139 Acked 9138 Write 3 Done New Done false
23358 8140 Acked 9139 Read 9 Done New Done false
23359 8141 Acked 9140 Read 1 Done New Done false
23360 8142 Acked 9141 Write 10 Done New Done false
23361 8143 Acked 9142 Read 10 Done New Done false
23362 8144 Acked 9143 Write 2 Done New Done false
23363 8145 Acked 9144 Read 4 Done New Done false
23364 8146 Acked 9145 Write 6 Done New Done false
23365 8147 Acked 9146 Read 3 Done New Done false
23366 8148 Acked 9147 Read 8 Done New Done false
23367 8149 Acked 9148 Read 2 Done New Done false
23368 8150 Acked 9149 Flush 0 Done New Done false
23369 8151 Acked 9150 Write 5 Done New Done false
23370 8152 Acked 9151 Read 4 Done New Done false
23371 8153 Acked 9152 Read 8 Done New Done false
23372 8154 Acked 9153 Write 3 Done New Done false
23373 8155 Acked 9154 Write 5 Done New Done false
23374 8156 Acked 9155 Write 10 Done New Done false
23375 8157 Acked 9156 Flush 0 Done New Done false
23376 8158 Acked 9157 Read 2 Done New Done false
23377 8159 Acked 9158 Read 8 Done New Done false
23378 8160 Acked 9159 Write 6 Sent New Sent false
23379 8161 Acked 9160 Write 5 Sent New Sent false
23380 STATES DS:0 DS:1 DS:2 TOTAL
23381 New 0 302 0 302
23382 Sent 2 0 2 4
23383 Done 300 0 300 600
23384 Skipped 0 0 0 0
23385 Error 0 0 0 0
23386 Last Flush: 9156 8858 9156
23387 Downstairs last five completed: 8858 8857 8856 8855 8854 8853
23388 Upstairs last five completed: 8161 8160 8159 8158 8157 8156
23389 CLIENT: Up:0 ds:302 act:2
23390 [8820] got start action
23391 [8820] I am going to restart
23392 Starting downstairs at port 8820
23393 Make output file at "/tmp/dsc/downstairs-8820.txt"
23394 [8820][1] reports Starting
23395 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1203)
23396 [8820][1] reports Running
233972023-10-13T04:46:52.089ZINFOcrucible: [1] connecting to 127.0.0.1:8820 looper = 1
233982023-10-13T04:46:52.089ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 1
233992023-10-13T04:46:52.089ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state Offline
234002023-10-13T04:46:52.089ZINFOcrucible: [1] client re-new 303 jobs since flush 8858 = downstairs
234012023-10-13T04:46:52.090ZINFOcrucible: [1] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
234022023-10-13T04:46:52.091ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
234032023-10-13T04:46:52.091ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
234042023-10-13T04:46:52.091ZINFOcrucible: Returning client:1 UUID:12345678-0000-0000-0000-000000008820 matches
234052023-10-13T04:46:52.091ZINFOcrucible: [1] send last flush ID to this DS: 8858
234062023-10-13T04:46:52.091ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
234072023-10-13T04:46:52.091ZINFOcrucible: [1] Replied this last flush ID: 8858
234082023-10-13T04:46:52.091ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Offline Active ds_transition to Replay
234092023-10-13T04:46:52.091ZINFOcrucible: [1] Transition from Offline to Replay
234102023-10-13T04:46:52.091ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
234112023-10-13T04:46:52.091ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Replay Active ds_transition to Active
234122023-10-13T04:46:52.091ZINFOcrucible: [1] Transition from Replay to Active
234132023-10-13T04:46:52.091ZINFOcrucible: [1] Starts cmd_loop
234142023-10-13T04:46:52.091ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
234152023-10-13T04:46:52.091ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
234162023-10-13T04:46:52.091ZINFOcrucible: Set check for repair
234172023-10-13T04:46:53.093ZINFOcrucible: Checking if live repair is needed
234182023-10-13T04:46:53.093ZINFOcrucible: No Live Repair required at this time
234192023-10-13T04:46:53.093ZWARNcrucible: [1] flow control end
23420 ----------------------------------------------------------------
23421 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
23422 Downstairs last five completed: 9161 9160 9159 9158 9157 9156
23423 Upstairs last five completed: 8162 8161 8160 8159 8158 8157
23424 CLIENT: Up:0 ds:0 act:3
23425 Replay: All jobs finished, all DS active.
23426 Main task has work to do, go find it
23427 got dsc Stop(1)
23428 stop 1
23429 Oct 13 04:46:54.019 INFO [8820] Got stop action so:false kr:false
23430 request completed, latency_us: 470, response_code: 204, uri: /stop/cid/1, method: POST, req_id: c50bf03a-fbf3-4229-b2a1-e7842129e3dc, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
23431 Oct 13 04:46:54.020 INFO request completed, latency_us: 178, response_code: 200, uri: /state/cid/1, method: GET, req_id: 297ee51a-bbe6-45a6-b8ef-3fe6b372efcd, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
234322023-10-13T04:46:54.020ZWARNcrucible: [1] downstairs disconnected
234332023-10-13T04:46:54.020ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
234342023-10-13T04:46:54.020ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8820 closed looper = 1
234352023-10-13T04:46:54.020ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:false
234362023-10-13T04:46:54.020ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
234372023-10-13T04:46:54.020ZINFOcrucible: [1] 127.0.0.1:8820 task reports offline
234382023-10-13T04:46:54.020ZWARNcrucible: [1] pm_task rx.recv() is None
23439 [8820] Exited with: ExitStatus(unix_wait_status(9))
23440 [8820][1] reports Exit
23441 Oct 13 04:46:58.022 INFO request completed, latency_us: 249, response_code: 200, uri: /state/cid/1, method: GET, req_id: 0be4dbfd-9f35-4a1f-b6f1-c95d3da2e9a5, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
23442 001/300 Write block 3030 len 4096 data: 7
23443 002/300 Read block 4225 len 8192
23444 003/300 Read block 3324 len 36864
23445 004/300 Read block 958 len 36864
23446 005/300 Read block 2366 len 32768
23447 006/300 Read block 4376 len 16384
23448 007/300 Write block 4429 len 36864 data: 8 5 4 3 4 3 6 7 9
23449 008/300 Write block 2361 len 32768 data: 5 3 3 3 4 7 6 5
23450 009/300 Read block 4502 len 4096
23451 010/300 Write block 4894 len 36864 data: 6 6 5 4 4 3 2 5 5
23452 011/300 Flush
23453 012/300 Write block 4678 len 20480 data: 3 5 6 6 8
23454 013/300 Flush
23455 014/300 Write block 1573 len 32768 data: 4 4 4 4 3 4 3 5
23456 015/300 Flush
23457 016/300 Flush
23458 017/300 Read block 1491 len 8192
23459 018/300 Read block 2591 len 20480
23460 019/300 Write block 1157 len 20480 data: 3 5 4 5 6
23461 020/300 Flush
23462 021/300 Write block 4926 len 36864 data: 4 4 3 3 3 3 4 6 7
23463 022/300 Read block 1744 len 4096
23464 023/300 Write block 83 len 36864 data: 2 2 2 2 2 3 3 3 4
23465 024/300 Read block 3431 len 28672
23466 025/300 Flush
23467 026/300 Write block 893 len 4096 data: 3
23468 027/300 Write block 1705 len 16384 data: 5 6 6 7
23469 028/300 Write block 1680 len 12288 data: 5 5 4
23470 029/300 Flush
23471 030/300 Read block 3024 len 20480
23472 031/300 Read block 3298 len 12288
23473 032/300 Write block 3095 len 12288 data: 5 5 4
23474 033/300 Read block 1646 len 20480
23475 034/300 Flush
23476 035/300 Flush
23477 036/300 Read block 1115 len 40960
23478 037/300 Write block 1301 len 8192 data: 8 6
23479 038/300 Read block 623 len 4096
23480 039/300 Write block 1796 len 8192 data: 8 7
23481 040/300 Read block 378 len 24576
23482 041/300 Write block 4013 len 16384 data: 3 3 5 5
23483 042/300 Read block 299 len 28672
23484 043/300 Read block 1668 len 20480
23485 044/300 Read block 3391 len 4096
23486 045/300 Read block 2451 len 8192
23487 046/300 Write block 3944 len 16384 data: 7 5 4 4
23488 047/300 Read block 4356 len 24576
23489 048/300 Read block 3943 len 36864
23490 049/300 Write block 3844 len 4096 data: 5
23491 050/300 Write block 1350 len 32768 data: 4 5 5 5 4 5 5 7
23492 051/300 Write block 4519 len 12288 data: 6 7 7
23493 052/300 Write block 2048 len 28672 data: 4 5 5 6 5 4 4
23494 053/300 Write block 2299 len 4096 data: 5
23495 054/300 Write block 4895 len 28672 data: 7 6 5 5 4 3 6
23496 055/300 Read block 1215 len 36864
23497 056/300 Write block 1168 len 12288 data: 6 7 6
23498 057/300 Write block 4428 len 40960 data: 8 9 6 5 4 5 4 7 8 10
23499 058/300 Flush
23500 059/300 Read block 4220 len 16384
23501 060/300 Read block 3774 len 32768
23502 061/300 Flush
23503 062/300 Write block 1662 len 32768 data: 6 7 8 4 4 7 7 7
23504 063/300 Read block 1416 len 4096
23505 064/300 Read block 4859 len 24576
23506 065/300 Read block 1144 len 40960
23507 066/300 Read block 4946 len 36864
23508 067/300 Write block 4570 len 16384 data: 3 4 5 8
23509 068/300 Read block 921 len 4096
23510 069/300 Read block 49 len 36864
23511 070/300 Write block 940 len 12288 data: 3 4 5
23512 071/300 Read block 1155 len 4096
23513 072/300 Read block 1653 len 24576
23514 073/300 Read block 72 len 32768
23515 074/300 Read block 4205 len 12288
23516 075/300 Flush
23517 076/300 Write block 233 len 12288 data: 2 4 5
23518 077/300 Read block 788 len 20480
23519 078/300 Write block 3807 len 28672 data: 8 7 7 5 5 5 5
23520 079/300 Write block 3754 len 4096 data: 6
23521 080/300 Read block 3895 len 4096
23522 081/300 Write block 499 len 32768 data: 8 7 7 7 7 5 4 5
23523 082/300 Flush
23524 083/300 Read block 2627 len 32768
23525 084/300 Read block 4798 len 20480
23526 085/300 Read block 3822 len 36864
23527 086/300 Read block 1245 len 40960
23528 087/300 Write block 1650 len 28672 data: 3 2 2 4 3 3 4
23529 088/300 Read block 3762 len 12288
23530 089/300 Read block 4081 len 4096
23531 090/300 Write block 1343 len 36864 data: 5 4 4 3 3 3 4 5 6
23532 091/300 Write block 1830 len 28672 data: 7 5 5 4 5 5 5
23533 092/300 Write block 1691 len 24576 data: 4 6 6 8 10 9
23534 093/300 Read block 628 len 8192
23535 094/300 Read block 1598 len 40960
23536 095/300 Read block 2324 len 8192
23537 096/300 Write block 836 len 36864 data: 7 7 6 6 6 7 8 10 9
23538 097/300 Write block 4957 len 32768 data: 5 5 4 3 2 2 3 3
23539 098/300 Read block 4615 len 28672
23540 099/300 Write block 937 len 36864 data: 4 4 4 4 5 6 5 5 3
23541 100/300 Read block 1454 len 12288
23542 101/300 Read block 3020 len 20480
23543 102/300 Read block 521 len 12288
23544 103/300 Write block 2546 len 8192 data: 4 3
23545 104/300 Read block 2683 len 8192
23546 105/300 Read block 2710 len 40960
23547 106/300 Write block 2073 len 20480 data: 3 3 2 2 2
23548 107/300 Read block 944 len 24576
23549 108/300 Read block 4407 len 16384
23550 109/300 Read block 1376 len 12288
23551 110/300 Read block 2443 len 4096
23552 111/300 Write block 423 len 32768 data: 6 6 6 4 4 3 4 4
23553 112/300 Write block 1330 len 24576 data: 3 3 3 2 2 2
23554 113/300 Read block 1490 len 40960
23555 114/300 Read block 4822 len 4096
23556 115/300 Read block 1433 len 16384
23557 116/300 Write block 33 len 36864 data: 8 8 8 6 5 6 5 4 3
23558 117/300 Write block 360 len 40960 data: 6 6 6 5 4 4 5 5 5 4
23559 118/300 Write block 109 len 40960 data: 6 5 6 6 5 5 6 5 5 5
23560 119/300 Read block 2510 len 24576
23561 120/300 Write block 1064 len 20480 data: 5 4 4 5 5
23562 121/300 Read block 2466 len 24576
23563 122/300 Read block 2510 len 40960
23564 123/300 Read block 4051 len 12288
23565 124/300 Flush
23566 125/300 Read block 4885 len 20480
23567 126/300 Read block 1364 len 32768
23568 127/300 Write block 457 len 8192 data: 7 5
23569 128/300 Flush
23570 129/300 Flush
23571 130/300 Write block 4402 len 24576 data: 5 6 6 5 4 3
23572 131/300 Write block 2340 len 32768 data: 4 3 2 4 4 4 6 5
23573 132/300 Flush
23574 133/300 Write block 1079 len 36864 data: 7 8 10 11 8 8 9 10 8
23575 134/300 Read block 331 len 20480
23576 135/300 Read block 2751 len 16384
23577 136/300 Write block 3161 len 28672 data: 7 7 9 5 6 7 9
23578 137/300 Read block 3613 len 36864
23579 138/300 Write block 1767 len 8192 data: 4 4
23580 139/300 Write block 3204 len 28672 data: 6 6 7 7 6 6 6
23581 140/300 Write block 4832 len 24576 data: 6 6 5 5 4 4
23582 141/300 Read block 4940 len 12288
23583 142/300 Write block 3033 len 8192 data: 6 5
23584 143/300 Read block 458 len 24576
23585 144/300 Read block 3586 len 24576
23586 145/300 Write block 1608 len 36864 data: 4 3 4 6 6 6 5 5 4
23587 146/300 Flush
23588 147/300 Read block 2562 len 24576
23589 148/300 Read block 2369 len 40960
23590 149/300 Write block 2115 len 12288 data: 7 8 7
23591 150/300 Write block 583 len 8192 data: 4 4
23592 151/300 Flush
23593 152/300 Flush
23594 153/300 Write block 1590 len 12288 data: 6 5 4
23595 154/300 Write block 375 len 4096 data: 8
23596 155/300 Read block 1863 len 8192
23597 156/300 Write block 4368 len 16384 data: 5 7 7 8
23598 157/300 Read block 3896 len 24576
23599 158/300 Write block 5 len 4096 data: 7
23600 159/300 Write block 3241 len 4096 data: 4
23601 160/300 Read block 1141 len 40960
23602 161/300 Write block 3221 len 16384 data: 8 7 6 6
23603 162/300 Read block 4582 len 16384
23604 163/300 Read block 244 len 8192
23605 164/300 Read block 570 len 36864
23606 165/300 Read block 2077 len 12288
23607 166/300 Flush
23608 167/300 Read block 1423 len 12288
23609 168/300 Read block 1585 len 40960
23610 169/300 Write block 1432 len 24576 data: 4 5 5 5 5 4
23611 170/300 Write block 2479 len 36864 data: 6 7 6 6 7 6 6 6 7
23612 171/300 Read block 1497 len 40960
23613 172/300 Read block 1290 len 16384
23614 173/300 Write block 4098 len 12288 data: 6 7 8
23615 174/300 Write block 4498 len 16384 data: 8 9 6 4
23616 175/300 Read block 2381 len 24576
23617 176/300 Write block 4965 len 20480 data: 3 3 4 5 5
23618 177/300 Write block 329 len 4096 data: 4
23619 178/300 Write block 4796 len 40960 data: 4 4 4 4 5 6 6 4 4 3
23620 179/300 Flush
23621 180/300 Write block 351 len 20480 data: 4 4 6 6 7
23622 181/300 Write block 2079 len 12288 data: 3 4 5
23623 182/300 Read block 1097 len 16384
23624 183/300 Write block 4211 len 40960 data: 9 8 7 8 8 9 7 6 5 5
23625 184/300 Read block 2022 len 16384
23626 185/300 Read block 3664 len 36864
23627 186/300 Flush
23628 187/300 Read block 2760 len 4096
23629 188/300 Read block 1920 len 8192
23630 189/300 Flush
23631 190/300 Read block 713 len 20480
23632 191/300 Write block 2534 len 32768 data: 5 5 3 3 5 5 5 4
23633 192/300 Read block 419 len 32768
23634 193/300 Read block 3274 len 8192
23635 194/300 Write block 2285 len 28672 data: 6 6 4 7 6 6 5
23636 195/300 Read block 3098 len 24576
23637 196/300 Read block 4457 len 40960
23638 197/300 Read block 1634 len 12288
23639 198/300 Flush
23640 199/300 Read block 3316 len 36864
23641 200/300 Write block 2987 len 8192 data: 7 7
23642 201/300 Read block 1182 len 24576
23643 202/300 Read block 4274 len 24576
23644 203/300 Write block 2239 len 12288 data: 7 8 8
23645 204/300 Write block 1396 len 8192 data: 7 8
23646 205/300 Read block 3953 len 32768
23647 206/300 Read block 4805 len 40960
23648 207/300 Read block 30 len 4096
23649 208/300 Write block 211 len 4096 data: 6
23650 209/300 Write block 1371 len 32768 data: 4 3 3 2 2 3 3 3
23651 210/300 Write block 2666 len 12288 data: 3 3 3
23652 211/300 Write block 2531 len 16384 data: 5 6 5 6
23653 212/300 Read block 222 len 8192
23654 213/300 Write block 840 len 24576 data: 7 8 9 11 10 9
23655 214/300 Write block 1255 len 20480 data: 4 3 3 3 2
23656 215/300 Read block 4794 len 32768
23657 216/300 Read block 2613 len 20480
23658 217/300 Read block 1216 len 4096
23659 218/300 Read block 2311 len 24576
23660 219/300 Read block 1062 len 28672
23661 220/300 Read block 1011 len 8192
23662 221/300 Read block 11 len 32768
23663 222/300 Write block 3116 len 24576 data: 8 7 8 8 7 7
23664 223/300 Read block 712 len 28672
23665 224/300 Read block 4779 len 4096
23666 225/300 Read block 3820 len 20480
23667 226/300 Write block 60 len 8192 data: 7 7
23668 227/300 Flush
23669 228/300 Read block 1809 len 12288
23670 229/300 Flush
23671 230/300 Read block 3597 len 40960
23672 231/300 Write block 1359 len 24576 data: 4 5 5 7 7 7
23673 232/300 Write block 1767 len 12288 data: 5 5 5
23674 233/300 Write block 8 len 24576 data: 7 7 7 7 7 9
23675 234/300 Read block 1904 len 40960
23676 235/300 Read block 1754 len 40960
23677 236/300 Read block 4231 len 12288
23678 237/300 Read block 4094 len 16384
23679 238/300 Read block 4048 len 36864
23680 239/300 Write block 1116 len 36864 data: 4 4 5 5 6 5 5 4 5
23681 240/300 Read block 2466 len 12288
23682 241/300 Read block 2840 len 8192
23683 242/300 Flush
23684 243/300 Read block 2201 len 36864
23685 244/300 Read block 969 len 36864
23686 245/300 Write block 151 len 12288 data: 10 11 10
23687 246/300 Flush
23688 247/300 Read block 4417 len 32768
23689 248/300 Read block 1800 len 24576
23690 249/300 Flush
23691 250/300 Read block 1049 len 4096
23692 251/300 Flush
23693 252/300 Write block 3975 len 12288 data: 4 4 4
23694 253/300 Write block 2019 len 16384 data: 4 4 4 5
23695 254/300 Write block 4660 len 12288 data: 9 8 9
23696 255/300 Write block 2529 len 12288 data: 6 6 6
23697 256/300 Read block 214 len 36864
23698 257/300 Write block 1016 len 28672 data: 7 8 8 7 5 6 6
23699 258/300 Flush
23700 259/300 Read block 1919 len 16384
23701 260/300 Write block 3919 len 32768 data: 4 2 2 4 4 5 6 6
23702 261/300 Write block 1387 len 12288 data: 3 3 2
23703 262/300 Read block 4455 len 20480
23704 263/300 Write block 2491 len 12288 data: 8 9 8
23705 264/300 Read block 1739 len 40960
23706 265/300 Read block 714 len 20480
23707 266/300 Write block 3280 len 32768 data: 7 6 5 5 9 9 8 7
23708 267/300 Write block 2252 len 8192 data: 4 3
23709 268/300 Write block 506 len 28672 data: 6 6 5 6 6 7 8
23710 269/300 Read block 4662 len 24576
23711 270/300 Write block 490 len 40960 data: 8 9 10 8 7 8 8 7 7 9
23712 271/300 Write block 607 len 8192 data: 4 4
23713 272/300 Read block 1103 len 32768
23714 273/300 Read block 4072 len 28672
23715 274/300 Read block 4676 len 20480
23716 275/300 Read block 2063 len 40960
23717 276/300 Read block 3121 len 16384
23718 277/300 Write block 1321 len 20480 data: 4 3 3 3 3
23719 278/300 Read block 2271 len 24576
23720 279/300 Read block 1890 len 24576
23721 280/300 Read block 642 len 20480
23722 281/300 Flush
23723 282/300 Write block 4773 len 40960 data: 4 4 4 5 7 8 9 9 8 10
23724 283/300 Flush
23725 284/300 Write block 718 len 4096 data: 7
23726 285/300 Read block 4810 len 20480
23727 286/300 Write block 1584 len 12288 data: 8 7 8
23728 287/300 Write block 1288 len 36864 data: 11 12 12 11 10 10 7 7 6
23729 288/300 Flush
23730 289/300 Write block 4617 len 40960 data: 6 5 5 4 6 8 8 8 9 8
23731 290/300 Write block 734 len 12288 data: 3 5 6
23732 291/300 Read block 1030 len 4096
23733 292/300 Write block 904 len 24576 data: 9 6 5 5 4 4
23734 293/300 Read block 3664 len 16384
23735 294/300 Read block 4737 len 12288
23736 295/300 Read block 4577 len 12288
23737 296/300 Write block 1997 len 36864 data: 5 5 5 6 7 4 4 5 6
23738 297/300 Write block 4659 len 20480 data: 8 10 9 10 10
23739 298/300 Read block 435 len 20480
23740 299/300 Read block 856 len 36864
23741 300/300 Flush
23742 Main task has work to do, go find it
23743 got dsc Start(1)
23744 start 1
23745 Oct 13 04:46:58.897 INFO request completed, latency_us: 285, response_code: 204, uri: /start/cid/1, method: POST, req_id: c27ca49e-450b-43b6-a84c-9a424203a547, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
23746 Replay: started 1, returned:Ok(())
23747 ----------------------------------------------------------------
23748 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:301
23749 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
23750 8163 Acked 9162 Write 1 Done New Done false
23751 8164 Acked 9163 Read 2 Done New Done false
23752 8165 Acked 9164 Read 9 Done New Done false
23753 8166 Acked 9165 Read 9 Done New Done false
23754 8167 Acked 9166 Read 8 Done New Done false
23755 8168 Acked 9167 Read 4 Done New Done false
23756 8169 Acked 9168 Write 9 Done New Done false
23757 8170 Acked 9169 Write 8 Done New Done false
23758 8171 Acked 9170 Read 1 Done New Done false
23759 8172 Acked 9171 Write 9 Done New Done false
23760 8173 Acked 9172 Flush 0 Done New Done false
23761 8174 Acked 9173 Write 5 Done New Done false
23762 8175 Acked 9174 Flush 0 Done New Done false
23763 8176 Acked 9175 Write 8 Done New Done false
23764 8177 Acked 9176 Flush 0 Done New Done false
23765 8178 Acked 9177 Flush 0 Done New Done false
23766 8179 Acked 9178 Read 2 Done New Done false
23767 8180 Acked 9179 Read 5 Done New Done false
23768 8181 Acked 9180 Write 5 Done New Done false
23769 8182 Acked 9181 Flush 0 Done New Done false
23770 8183 Acked 9182 Write 9 Done New Done false
23771 8184 Acked 9183 Read 1 Done New Done false
23772 8185 Acked 9184 Write 9 Done New Done false
23773 8186 Acked 9185 Read 7 Done New Done false
23774 8187 Acked 9186 Flush 0 Done New Done false
23775 8188 Acked 9187 Write 1 Done New Done false
23776 8189 Acked 9188 Write 4 Done New Done false
23777 8190 Acked 9189 Write 3 Done New Done false
23778 8191 Acked 9190 Flush 0 Done New Done false
23779 8192 Acked 9191 Read 5 Done New Done false
23780 8193 Acked 9192 Read 3 Done New Done false
23781 8194 Acked 9193 Write 3 Done New Done false
23782 8195 Acked 9194 Read 5 Done New Done false
23783 8196 Acked 9195 Flush 0 Done New Done false
23784 8197 Acked 9196 Flush 0 Done New Done false
23785 8198 Acked 9197 Read 10 Done New Done false
23786 8199 Acked 9198 Write 2 Done New Done false
23787 8200 Acked 9199 Read 1 Done New Done false
23788 8201 Acked 9200 Write 2 Done New Done false
23789 8202 Acked 9201 Read 6 Done New Done false
23790 8203 Acked 9202 Write 4 Done New Done false
23791 8204 Acked 9203 Read 7 Done New Done false
23792 8205 Acked 9204 Read 5 Done New Done false
23793 8206 Acked 9205 Read 1 Done New Done false
23794 8207 Acked 9206 Read 2 Done New Done false
23795 8208 Acked 9207 Write 4 Done New Done false
23796 8209 Acked 9208 Read 6 Done New Done false
23797 8210 Acked 9209 Read 9 Done New Done false
23798 8211 Acked 9210 Write 1 Done New Done false
23799 8212 Acked 9211 Write 8 Done New Done false
23800 8213 Acked 9212 Write 3 Done New Done false
23801 8214 Acked 9213 Write 7 Done New Done false
23802 8215 Acked 9214 Write 1 Done New Done false
23803 8216 Acked 9215 Write 7 Done New Done false
23804 8217 Acked 9216 Read 9 Done New Done false
23805 8218 Acked 9217 Write 3 Done New Done false
23806 8219 Acked 9218 Write 10 Done New Done false
23807 8220 Acked 9219 Flush 0 Done New Done false
23808 8221 Acked 9220 Read 4 Done New Done false
23809 8222 Acked 9221 Read 8 Done New Done false
23810 8223 Acked 9222 Flush 0 Done New Done false
23811 8224 Acked 9223 Write 8 Done New Done false
23812 8225 Acked 9224 Read 1 Done New Done false
23813 8226 Acked 9225 Read 6 Done New Done false
23814 8227 Acked 9226 Read 10 Done New Done false
23815 8228 Acked 9227 Read 9 Done New Done false
23816 8229 Acked 9228 Write 4 Done New Done false
23817 8230 Acked 9229 Read 1 Done New Done false
23818 8231 Acked 9230 Read 9 Done New Done false
23819 8232 Acked 9231 Write 3 Done New Done false
23820 8233 Acked 9232 Read 1 Done New Done false
23821 8234 Acked 9233 Read 6 Done New Done false
23822 8235 Acked 9234 Read 8 Done New Done false
23823 8236 Acked 9235 Read 3 Done New Done false
23824 8237 Acked 9236 Flush 0 Done New Done false
23825 8238 Acked 9237 Write 3 Done New Done false
23826 8239 Acked 9238 Read 5 Done New Done false
23827 8240 Acked 9239 Write 7 Done New Done false
23828 8241 Acked 9240 Write 1 Done New Done false
23829 8242 Acked 9241 Read 1 Done New Done false
23830 8243 Acked 9242 Write 8 Done New Done false
23831 8244 Acked 9243 Flush 0 Done New Done false
23832 8245 Acked 9244 Read 8 Done New Done false
23833 8246 Acked 9245 Read 5 Done New Done false
23834 8247 Acked 9246 Read 9 Done New Done false
23835 8248 Acked 9247 Read 10 Done New Done false
23836 8249 Acked 9248 Write 7 Done New Done false
23837 8250 Acked 9249 Read 3 Done New Done false
23838 8251 Acked 9250 Read 1 Done New Done false
23839 8252 Acked 9251 Write 9 Done New Done false
23840 8253 Acked 9252 Write 7 Done New Done false
23841 8254 Acked 9253 Write 6 Done New Done false
23842 8255 Acked 9254 Read 2 Done New Done false
23843 8256 Acked 9255 Read 10 Done New Done false
23844 8257 Acked 9256 Read 2 Done New Done false
23845 8258 Acked 9257 Write 9 Done New Done false
23846 8259 Acked 9258 Write 8 Done New Done false
23847 8260 Acked 9259 Read 7 Done New Done false
23848 8261 Acked 9260 Write 9 Done New Done false
23849 8262 Acked 9261 Read 3 Done New Done false
23850 8263 Acked 9262 Read 5 Done New Done false
23851 8264 Acked 9263 Read 3 Done New Done false
23852 8265 Acked 9264 Write 2 Done New Done false
23853 8266 Acked 9265 Read 2 Done New Done false
23854 8267 Acked 9266 Read 10 Done New Done false
23855 8268 Acked 9267 Write 5 Done New Done false
23856 8269 Acked 9268 Read 6 Done New Done false
23857 8270 Acked 9269 Read 4 Done New Done false
23858 8271 Acked 9270 Read 3 Done New Done false
23859 8272 Acked 9271 Read 1 Done New Done false
23860 8273 Acked 9272 Write 8 Done New Done false
23861 8274 Acked 9273 Write 6 Done New Done false
23862 8275 Acked 9274 Read 10 Done New Done false
23863 8276 Acked 9275 Read 1 Done New Done false
23864 8277 Acked 9276 Read 4 Done New Done false
23865 8278 Acked 9277 Write 9 Done New Done false
23866 8279 Acked 9278 Write 10 Done New Done false
23867 8280 Acked 9279 Write 10 Done New Done false
23868 8281 Acked 9280 Read 6 Done New Done false
23869 8282 Acked 9281 Write 5 Done New Done false
23870 8283 Acked 9282 Read 6 Done New Done false
23871 8284 Acked 9283 Read 10 Done New Done false
23872 8285 Acked 9284 Read 3 Done New Done false
23873 8286 Acked 9285 Flush 0 Done New Done false
23874 8287 Acked 9286 Read 5 Done New Done false
23875 8288 Acked 9287 Read 8 Done New Done false
23876 8289 Acked 9288 Write 2 Done New Done false
23877 8290 Acked 9289 Flush 0 Done New Done false
23878 8291 Acked 9290 Flush 0 Done New Done false
23879 8292 Acked 9291 Write 6 Done New Done false
23880 8293 Acked 9292 Write 8 Done New Done false
23881 8294 Acked 9293 Flush 0 Done New Done false
23882 8295 Acked 9294 Write 9 Done New Done false
23883 8296 Acked 9295 Read 5 Done New Done false
23884 8297 Acked 9296 Read 4 Done New Done false
23885 8298 Acked 9297 Write 7 Done New Done false
23886 8299 Acked 9298 Read 9 Done New Done false
23887 8300 Acked 9299 Write 2 Done New Done false
23888 8301 Acked 9300 Write 7 Done New Done false
23889 8302 Acked 9301 Write 6 Done New Done false
23890 8303 Acked 9302 Read 3 Done New Done false
23891 8304 Acked 9303 Write 2 Done New Done false
23892 8305 Acked 9304 Read 6 Done New Done false
23893 8306 Acked 9305 Read 6 Done New Done false
23894 8307 Acked 9306 Write 9 Done New Done false
23895 8308 Acked 9307 Flush 0 Done New Done false
23896 8309 Acked 9308 Read 6 Done New Done false
23897 8310 Acked 9309 Read 10 Done New Done false
23898 8311 Acked 9310 Flush 0 Done New Done false
23899 8312 Acked 9311 Write 3 Done New Done false
23900 8313 Acked 9312 Write 2 Done New Done false
23901 8314 Acked 9313 Flush 0 Done New Done false
23902 8315 Acked 9314 Flush 0 Done New Done false
23903 8316 Acked 9315 Write 3 Done New Done false
23904 8317 Acked 9316 Write 1 Done New Done false
23905 8318 Acked 9317 Read 2 Done New Done false
23906 8319 Acked 9318 Write 4 Done New Done false
23907 8320 Acked 9319 Read 6 Done New Done false
23908 8321 Acked 9320 Write 1 Done New Done false
23909 8322 Acked 9321 Write 1 Done New Done false
23910 8323 Acked 9322 Read 10 Done New Done false
23911 8324 Acked 9323 Write 4 Done New Done false
23912 8325 Acked 9324 Read 4 Done New Done false
23913 8326 Acked 9325 Read 2 Done New Done false
23914 8327 Acked 9326 Read 9 Done New Done false
23915 8328 Acked 9327 Read 3 Done New Done false
23916 8329 Acked 9328 Flush 0 Done New Done false
23917 8330 Acked 9329 Read 3 Done New Done false
23918 8331 Acked 9330 Read 10 Done New Done false
23919 8332 Acked 9331 Write 6 Done New Done false
23920 8333 Acked 9332 Write 9 Done New Done false
23921 8334 Acked 9333 Read 10 Done New Done false
23922 8335 Acked 9334 Read 4 Done New Done false
23923 8336 Acked 9335 Write 3 Done New Done false
23924 8337 Acked 9336 Write 4 Done New Done false
23925 8338 Acked 9337 Read 6 Done New Done false
23926 8339 Acked 9338 Write 5 Done New Done false
23927 8340 Acked 9339 Write 1 Done New Done false
23928 8341 Acked 9340 Write 10 Done New Done false
23929 8342 Acked 9341 Flush 0 Done New Done false
23930 8343 Acked 9342 Write 5 Done New Done false
23931 8344 Acked 9343 Write 3 Done New Done false
23932 8345 Acked 9344 Read 4 Done New Done false
23933 8346 Acked 9345 Write 10 Done New Done false
23934 8347 Acked 9346 Read 4 Done New Done false
23935 8348 Acked 9347 Read 9 Done New Done false
23936 8349 Acked 9348 Flush 0 Done New Done false
23937 8350 Acked 9349 Read 1 Done New Done false
23938 8351 Acked 9350 Read 2 Done New Done false
23939 8352 Acked 9351 Flush 0 Done New Done false
23940 8353 Acked 9352 Read 5 Done New Done false
23941 8354 Acked 9353 Write 8 Done New Done false
23942 8355 Acked 9354 Read 8 Done New Done false
23943 8356 Acked 9355 Read 2 Done New Done false
23944 8357 Acked 9356 Write 7 Done New Done false
23945 8358 Acked 9357 Read 6 Done New Done false
23946 8359 Acked 9358 Read 10 Done New Done false
23947 8360 Acked 9359 Read 3 Done New Done false
23948 8361 Acked 9360 Flush 0 Done New Done false
23949 8362 Acked 9361 Read 9 Done New Done false
23950 8363 Acked 9362 Write 2 Done New Done false
23951 8364 Acked 9363 Read 6 Done New Done false
23952 8365 Acked 9364 Read 6 Done New Done false
23953 8366 Acked 9365 Write 3 Done New Done false
23954 8367 Acked 9366 Write 2 Done New Done false
23955 8368 Acked 9367 Read 8 Done New Done false
23956 8369 Acked 9368 Read 10 Done New Done false
23957 8370 Acked 9369 Read 1 Done New Done false
23958 8371 Acked 9370 Write 1 Done New Done false
23959 8372 Acked 9371 Write 8 Done New Done false
23960 8373 Acked 9372 Write 3 Done New Done false
23961 8374 Acked 9373 Write 4 Done New Done false
23962 8375 Acked 9374 Read 2 Done New Done false
23963 8376 Acked 9375 Write 6 Done New Done false
23964 8377 Acked 9376 Write 5 Done New Done false
23965 8378 Acked 9377 Read 8 Done New Done false
23966 8379 Acked 9378 Read 5 Done New Done false
23967 8380 Acked 9379 Read 1 Done New Done false
23968 8381 Acked 9380 Read 6 Done New Done false
23969 8382 Acked 9381 Read 7 Done New Done false
23970 8383 Acked 9382 Read 2 Done New Done false
23971 8384 Acked 9383 Read 8 Done New Done false
23972 8385 Acked 9384 Write 6 Done New Done false
23973 8386 Acked 9385 Read 7 Done New Done false
23974 8387 Acked 9386 Read 1 Done New Done false
23975 8388 Acked 9387 Read 5 Done New Done false
23976 8389 Acked 9388 Write 2 Done New Done false
23977 8390 Acked 9389 Flush 0 Done New Done false
23978 8391 Acked 9390 Read 3 Done New Done false
23979 8392 Acked 9391 Flush 0 Done New Done false
23980 8393 Acked 9392 Read 10 Done New Done false
23981 8394 Acked 9393 Write 6 Done New Done false
23982 8395 Acked 9394 Write 3 Done New Done false
23983 8396 Acked 9395 Write 6 Done New Done false
23984 8397 Acked 9396 Read 10 Done New Done false
23985 8398 Acked 9397 Read 10 Done New Done false
23986 8399 Acked 9398 Read 3 Done New Done false
23987 8400 Acked 9399 Read 4 Done New Done false
23988 8401 Acked 9400 Read 9 Done New Done false
23989 8402 Acked 9401 Write 9 Done New Done false
23990 8403 Acked 9402 Read 3 Done New Done false
23991 8404 Acked 9403 Read 2 Done New Done false
23992 8405 Acked 9404 Flush 0 Done New Done false
23993 8406 Acked 9405 Read 9 Done New Done false
23994 8407 Acked 9406 Read 9 Done New Done false
23995 8408 Acked 9407 Write 3 Done New Done false
23996 8409 Acked 9408 Flush 0 Done New Done false
23997 8410 Acked 9409 Read 8 Done New Done false
23998 8411 Acked 9410 Read 6 Done New Done false
23999 8412 Acked 9411 Flush 0 Done New Done false
24000 8413 Acked 9412 Read 1 Done New Done false
24001 8414 Acked 9413 Flush 0 Done New Done false
24002 8415 Acked 9414 Write 3 Done New Done false
24003 8416 Acked 9415 Write 4 Done New Done false
24004 8417 Acked 9416 Write 3 Done New Done false
24005 8418 Acked 9417 Write 3 Done New Done false
24006 8419 Acked 9418 Read 9 Done New Done false
24007 8420 Acked 9419 Write 7 Done New Done false
24008 8421 Acked 9420 Flush 0 Done New Done false
24009 8422 Acked 9421 Read 4 Done New Done false
24010 8423 Acked 9422 Write 8 Done New Done false
24011 8424 Acked 9423 Write 3 Done New Done false
24012 8425 Acked 9424 Read 5 Done New Done false
24013 8426 Acked 9425 Write 3 Done New Done false
24014 8427 Acked 9426 Read 10 Done New Done false
24015 8428 Acked 9427 Read 5 Done New Done false
24016 8429 Acked 9428 Write 8 Done New Done false
24017 8430 Acked 9429 Write 2 Done New Done false
24018 8431 Acked 9430 Write 7 Done New Done false
24019 8432 Acked 9431 Read 6 Done New Done false
24020 8433 Acked 9432 Write 10 Done New Done false
24021 8434 Acked 9433 Write 2 Done New Done false
24022 8435 Acked 9434 Read 8 Done New Done false
24023 8436 Acked 9435 Read 7 Done New Done false
24024 8437 Acked 9436 Read 5 Done New Done false
24025 8438 Acked 9437 Read 10 Done New Done false
24026 8439 Acked 9438 Read 4 Done New Done false
24027 8440 Acked 9439 Write 5 Done New Done false
24028 8441 Acked 9440 Read 6 Done New Done false
24029 8442 Acked 9441 Read 6 Done New Done false
24030 8443 Acked 9442 Read 5 Done New Done false
24031 8444 Acked 9443 Flush 0 Done New Done false
24032 8445 Acked 9444 Write 10 Done New Done false
24033 8446 Acked 9445 Flush 0 Done New Done false
24034 8447 Acked 9446 Write 1 Done New Done false
24035 8448 Acked 9447 Read 5 Done New Done false
24036 8449 Acked 9448 Write 3 Done New Done false
24037 8450 Acked 9449 Write 9 Done New Done false
24038 8451 Acked 9450 Flush 0 Done New Done false
24039 8452 Acked 9451 Write 10 Done New Done false
24040 8453 Acked 9452 Write 3 Done New Done false
24041 8454 Acked 9453 Read 1 Done New Done false
24042 8455 Acked 9454 Write 6 Done New Done false
24043 8456 Acked 9455 Read 4 Done New Done false
24044 8457 Acked 9456 Read 3 Done New Done false
24045 8458 Acked 9457 Read 3 Done New Done false
24046 8459 Acked 9458 Write 9 Done New Done false
24047 8460 Acked 9459 Write 5 Done New Done false
24048 8461 Acked 9460 Read 5 Done New Done false
24049 8462 Acked 9461 Read 9 Done New Done false
24050 8463 Acked 9462 Flush 0 Done New Done false
24051 STATES DS:0 DS:1 DS:2 TOTAL
24052 New 0 301 0 301
24053 Sent 0 0 0 0
24054 Done 301 0 301 602
24055 Skipped 0 0 0 0
24056 Error 0 0 0 0
24057 Last Flush: 9462 9161 9462
24058 Downstairs last five completed: 9161 9160 9159 9158 9157 9156
24059 Upstairs last five completed: 8463 8462 8461 8460 8459 8458
24060 CLIENT: Up:0 ds:301 act:2
24061 [8820] got start action
24062 [8820] I am going to restart
24063 Starting downstairs at port 8820
24064 Make output file at "/tmp/dsc/downstairs-8820.txt"
24065 [8820][1] reports Starting
24066 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1204)
24067 [8820][1] reports Running
240682023-10-13T04:47:01.029ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 1
240692023-10-13T04:47:01.029ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state Offline
240702023-10-13T04:47:01.029ZINFOcrucible: [1] client re-new 301 jobs since flush 9161 = downstairs
240712023-10-13T04:47:01.030ZINFOcrucible: [1] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
240722023-10-13T04:47:01.031ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
240732023-10-13T04:47:01.031ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
240742023-10-13T04:47:01.031ZINFOcrucible: Returning client:1 UUID:12345678-0000-0000-0000-000000008820 matches
240752023-10-13T04:47:01.031ZINFOcrucible: [1] send last flush ID to this DS: 9161
240762023-10-13T04:47:01.031ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
240772023-10-13T04:47:01.031ZINFOcrucible: [1] Replied this last flush ID: 9161
240782023-10-13T04:47:01.031ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Offline Active ds_transition to Replay
240792023-10-13T04:47:01.031ZINFOcrucible: [1] Transition from Offline to Replay
240802023-10-13T04:47:01.031ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
240812023-10-13T04:47:01.031ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Replay Active ds_transition to Active
240822023-10-13T04:47:01.031ZINFOcrucible: [1] Transition from Replay to Active
240832023-10-13T04:47:01.031ZINFOcrucible: [1] Starts cmd_loop
240842023-10-13T04:47:01.031ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
240852023-10-13T04:47:01.031ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
240862023-10-13T04:47:01.031ZINFOcrucible: Set check for repair
240872023-10-13T04:47:02.033ZINFOcrucible: Checking if live repair is needed
240882023-10-13T04:47:02.033ZINFOcrucible: No Live Repair required at this time
240892023-10-13T04:47:02.033ZWARNcrucible: [1] flow control end
24090 ----------------------------------------------------------------
24091 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
24092 Downstairs last five completed: 9462 9461 9460 9459 9458 9457
24093 Upstairs last five completed: 8463 8462 8461 8460 8459 8458
24094 CLIENT: Up:0 ds:0 act:3
24095 Replay: All jobs finished, all DS active.
24096 Main task has work to do, go find it
24097 got dsc Stop(2)
24098 stop 2
24099 Oct 13 04:47:02.903 INFO request completed, latency_us: 505, response_code: 204, uri: /stop/cid/2, method: POST, req_id: 2c8bdcb7-f6b3-438a-a800-9ebd4143bf83, remote_addr: [8830] Got stop action so:false kr:false
24100 127.0.0.1:52046, local_addr: 127.0.0.1:9998
24101 Oct 13 04:47:02.904 INFO request completed, latency_us: 168, response_code: 200, uri: /state/cid/2, method: GET, req_id: 139f18f8-ffaa-4347-9040-9cdc4218db43, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
241022023-10-13T04:47:02.904ZWARNcrucible: [2] downstairs disconnected
241032023-10-13T04:47:02.904ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
241042023-10-13T04:47:02.904ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8830 closed looper = 2
241052023-10-13T04:47:02.904ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:false
241062023-10-13T04:47:02.904ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
241072023-10-13T04:47:02.905ZINFOcrucible: [2] 127.0.0.1:8830 task reports offline
241082023-10-13T04:47:02.905ZWARNcrucible: [2] pm_task rx.recv() is None
24109 [8830] Exited with: ExitStatus(unix_wait_status(9))
24110 [8830][2] reports Exit
24111 Oct 13 04:47:06.906 INFO request completed, latency_us: 247, response_code: 200, uri: /state/cid/2, method: GET, req_id: 757ef83b-c81e-407c-aae2-3184922f94d5, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
24112 001/300 Read block 2045 len 32768
24113 002/300 Read block 3862 len 8192
24114 003/300 Write block 4770 len 40960 data: 4 3 3 5 5 5 6 8 9 10
24115 004/300 Read block 3533 len 40960
24116 005/300 Read block 4647 len 28672
24117 006/300 Write block 3942 len 16384 data: 8 8 8 6
24118 007/300 Read block 2446 len 4096
24119 008/300 Write block 1016 len 4096 data: 8
24120 009/300 Write block 3340 len 24576 data: 5 5 6 5 6 8
24121 010/300 Read block 4216 len 20480
24122 011/300 Read block 4985 len 16384
24123 012/300 Write block 4032 len 28672 data: 6 4 4 5 5 4 3
24124 013/300 Read block 173 len 20480
24125 014/300 Write block 2860 len 32768 data: 4 4 7 7 9 8 9 6
24126 015/300 Read block 4834 len 8192
24127 016/300 Write block 3941 len 36864 data: 6 9 9 9 7 5 5 4 4
24128 017/300 Read block 4653 len 16384
24129 018/300 Write block 834 len 24576 data: 4 6 8 8 7 7
24130 019/300 Read block 909 len 12288
24131 020/300 Flush
24132 021/300 Write block 3149 len 28672 data: 5 4 6 7 6 5 5
24133 022/300 Read block 1876 len 24576
24134 023/300 Write block 904 len 12288 data: 10 7 6
24135 024/300 Read block 3914 len 16384
24136 025/300 Write block 2244 len 4096 data: 5
24137 026/300 Flush
24138 027/300 Write block 4651 len 20480 data: 4 7 8 6 5
24139 028/300 Write block 3271 len 4096 data: 3
24140 029/300 Write block 3739 len 36864 data: 7 6 6 7 6 5 5 5 5
24141 030/300 Read block 3509 len 32768
24142 031/300 Write block 756 len 32768 data: 10 9 8 6 6 5 5 6
24143 032/300 Read block 1833 len 36864
24144 033/300 Read block 1423 len 24576
24145 034/300 Flush
24146 035/300 Read block 3473 len 36864
24147 036/300 Write block 3008 len 20480 data: 4 5 4 4 4
24148 037/300 Read block 3697 len 40960
24149 038/300 Read block 4981 len 36864
24150 039/300 Read block 114 len 8192
24151 040/300 Read block 303 len 40960
24152 041/300 Read block 4390 len 28672
24153 042/300 Read block 565 len 20480
24154 043/300 Write block 4973 len 40960 data: 5 5 5 5 5 6 7 6 4 5
24155 044/300 Read block 2001 len 20480
24156 045/300 Write block 1967 len 24576 data: 3 4 4 4 4 3
24157 046/300 Read block 4276 len 24576
24158 047/300 Write block 400 len 32768 data: 7 7 7 7 6 6 5 4
24159 048/300 Read block 638 len 8192
24160 049/300 Write block 3108 len 20480 data: 5 5 6 5 5
24161 050/300 Write block 3587 len 32768 data: 8 8 8 5 3 3 2 3
24162 051/300 Read block 3481 len 20480
24163 052/300 Read block 563 len 40960
24164 053/300 Write block 373 len 32768 data: 5 6 9 7 6 5 5 5
24165 054/300 Flush
24166 055/300 Write block 3635 len 8192 data: 3 4
24167 056/300 Flush
24168 057/300 Read block 3330 len 16384
24169 058/300 Read block 1439 len 32768
24170 059/300 Read block 46 len 8192
24171 060/300 Read block 2435 len 28672
24172 061/300 Read block 2974 len 12288
24173 062/300 Read block 1174 len 12288
24174 063/300 Write block 2691 len 8192 data: 6 6
24175 064/300 Write block 3428 len 36864 data: 7 8 9 8 8 7 5 5 6
24176 065/300 Write block 2546 len 20480 data: 5 4 3 3 3
24177 066/300 Write block 1435 len 32768 data: 6 6 5 4 4 4 4 5
24178 067/300 Read block 2133 len 36864
24179 068/300 Write block 1144 len 8192 data: 6 5
24180 069/300 Write block 280 len 32768 data: 3 3 4 4 5 4 3 3
24181 070/300 Write block 506 len 40960 data: 7 7 6 7 7 8 9 7 8 8
24182 071/300 Read block 1514 len 24576
24183 072/300 Write block 261 len 40960 data: 6 5 4 4 3 4 4 4 5 6
24184 073/300 Write block 2824 len 16384 data: 5 5 6 7
24185 074/300 Read block 1915 len 8192
24186 075/300 Read block 4566 len 16384
24187 076/300 Read block 1641 len 28672
24188 077/300 Write block 1213 len 20480 data: 4 4 4 4 4
24189 078/300 Write block 2160 len 32768 data: 8 9 10 8 8 5 4 5
24190 079/300 Read block 1952 len 16384
24191 080/300 Write block 1177 len 4096 data: 7
24192 081/300 Read block 1626 len 16384
24193 082/300 Write block 1503 len 20480 data: 6 6 6 9 9
24194 083/300 Write block 1951 len 36864 data: 5 4 4 6 4 5 4 4 5
24195 084/300 Read block 1284 len 8192
24196 085/300 Read block 259 len 20480
24197 086/300 Write block 4258 len 4096 data: 5
24198 087/300 Read block 2427 len 16384
24199 088/300 Write block 1790 len 4096 data: 5
24200 089/300 Read block 4851 len 4096
24201 090/300 Write block 4592 len 20480 data: 6 6 5 4 4
24202 091/300 Read block 2819 len 36864
24203 092/300 Flush
24204 093/300 Flush
24205 094/300 Read block 970 len 12288
24206 095/300 Write block 4463 len 36864 data: 6 6 5 5 6 7 7 7 5
24207 096/300 Read block 1815 len 16384
24208 097/300 Read block 1445 len 36864
24209 098/300 Flush
24210 099/300 Read block 106 len 40960
24211 100/300 Read block 2926 len 40960
24212 101/300 Write block 1130 len 36864 data: 7 7 8 7 6 6 4 5 6
24213 102/300 Read block 536 len 40960
24214 103/300 Read block 1130 len 20480
24215 104/300 Write block 1205 len 4096 data: 8
24216 105/300 Read block 2164 len 40960
24217 106/300 Write block 4095 len 32768 data: 5 6 6 7 8 9 8 8
24218 107/300 Read block 737 len 40960
24219 108/300 Write block 4537 len 24576 data: 5 6 9 11 11 12
24220 109/300 Write block 121 len 16384 data: 6 6 5 3
24221 110/300 Write block 4411 len 24576 data: 7 7 9 8 7 7
24222 111/300 Flush
24223 112/300 Flush
24224 113/300 Write block 1047 len 24576 data: 5 5 5 5 5 5
24225 114/300 Read block 2809 len 12288
24226 115/300 Read block 909 len 4096
24227 116/300 Write block 1061 len 12288 data: 5 5 5
24228 117/300 Write block 986 len 8192 data: 8 7
24229 118/300 Write block 972 len 40960 data: 8 8 10 12 11 9 7 7 6 7
24230 119/300 Write block 1772 len 12288 data: 9 8 7
24231 120/300 Read block 3308 len 32768
24232 121/300 Write block 3567 len 8192 data: 3 2
24233 122/300 Write block 909 len 24576 data: 5 6 6 7 9 9
24234 123/300 Write block 2747 len 32768 data: 8 9 9 7 8 10 11 9
24235 124/300 Read block 4921 len 16384
24236 125/300 Write block 3865 len 8192 data: 7 8
24237 126/300 Read block 4715 len 20480
24238 127/300 Flush
24239 128/300 Flush
24240 129/300 Read block 4956 len 24576
24241 130/300 Read block 684 len 28672
24242 131/300 Read block 1248 len 8192
24243 132/300 Write block 4129 len 12288 data: 7 5 5
24244 133/300 Read block 4819 len 24576
24245 134/300 Write block 1630 len 36864 data: 8 8 8 8 7 7 7 8 8
24246 135/300 Write block 546 len 28672 data: 5 5 5 6 6 6 7
24247 136/300 Read block 651 len 36864
24248 137/300 Write block 2568 len 8192 data: 10 11
24249 138/300 Write block 4762 len 24576 data: 7 6 5 6 5 5
24250 139/300 Write block 1413 len 28672 data: 5 6 6 5 6 6 7
24251 140/300 Write block 3572 len 16384 data: 4 4 5 6
24252 141/300 Read block 46 len 28672
24253 142/300 Read block 4113 len 32768
24254 143/300 Read block 4567 len 24576
24255 144/300 Write block 3898 len 28672 data: 3 3 3 3 3 3 3
24256 145/300 Read block 3701 len 12288
24257 146/300 Write block 3116 len 24576 data: 9 8 9 9 8 8
24258 147/300 Write block 781 len 32768 data: 5 4 5 6 6 6 3 4
24259 148/300 Write block 4048 len 32768 data: 3 2 5 5 7 7 8 7
24260 149/300 Flush
24261 150/300 Write block 4668 len 12288 data: 5 5 5
24262 151/300 Read block 731 len 28672
24263 152/300 Read block 1437 len 28672
24264 153/300 Read block 206 len 20480
24265 154/300 Read block 4920 len 36864
24266 155/300 Read block 3148 len 24576
24267 156/300 Write block 745 len 16384 data: 7 4 5 4
24268 157/300 Read block 3152 len 24576
24269 158/300 Write block 2657 len 24576 data: 10 7 7 6 6 6
24270 159/300 Write block 4516 len 4096 data: 4
24271 160/300 Read block 2827 len 24576
24272 161/300 Write block 1155 len 28672 data: 6 5 4 6 5 6 7
24273 162/300 Read block 2502 len 28672
24274 163/300 Read block 2981 len 16384
24275 164/300 Flush
24276 165/300 Read block 222 len 20480
24277 166/300 Read block 2921 len 16384
24278 167/300 Read block 386 len 16384
24279 168/300 Read block 2916 len 28672
24280 169/300 Read block 346 len 36864
24281 170/300 Write block 2553 len 40960 data: 6 5 8 9 9 9 7 7 8 7
24282 171/300 Write block 3906 len 36864 data: 4 4 4 5 4 5 6 5 6
24283 172/300 Read block 1220 len 32768
24284 173/300 Read block 1227 len 20480
24285 174/300 Read block 274 len 4096
24286 175/300 Read block 3903 len 36864
24287 176/300 Write block 2346 len 24576 data: 7 6 5 6 5 3
24288 177/300 Read block 2354 len 24576
24289 178/300 Read block 3977 len 20480
24290 179/300 Flush
24291 180/300 Write block 38 len 20480 data: 7 6 5 4 3
24292 181/300 Read block 3498 len 28672
24293 182/300 Read block 521 len 20480
24294 183/300 Flush
24295 184/300 Read block 4738 len 20480
24296 185/300 Flush
24297 186/300 Write block 3550 len 12288 data: 6 7 10
24298 187/300 Write block 710 len 24576 data: 5 5 5 4 6 7
24299 188/300 Read block 4981 len 36864
24300 189/300 Read block 3593 len 20480
24301 190/300 Read block 3186 len 20480
24302 191/300 Read block 3275 len 12288
24303 192/300 Read block 1309 len 8192
24304 193/300 Read block 1893 len 16384
24305 194/300 Write block 2762 len 32768 data: 7 7 5 4 6 6 8 8
24306 195/300 Read block 1423 len 8192
24307 196/300 Write block 3156 len 36864 data: 5 5 6 7 7 8 8 10 6
24308 197/300 Read block 2193 len 24576
24309 198/300 Flush
24310 199/300 Write block 2930 len 12288 data: 7 6 4
24311 200/300 Flush
24312 201/300 Write block 3192 len 16384 data: 5 3 3 4
24313 202/300 Write block 328 len 20480 data: 4 5 3 3 4
24314 203/300 Write block 2747 len 12288 data: 9 10 10
24315 204/300 Flush
24316 205/300 Write block 2962 len 36864 data: 4 4 5 8 8 10 11 10 9
24317 206/300 Read block 3630 len 36864
24318 207/300 Read block 1183 len 40960
24319 208/300 Write block 4939 len 36864 data: 7 11 11 10 9 8 8 7 7
24320 209/300 Write block 511 len 4096 data: 9
24321 210/300 Read block 1111 len 24576
24322 211/300 Write block 4871 len 20480 data: 4 4 4 4 6
24323 212/300 Read block 4862 len 28672
24324 213/300 Read block 826 len 24576
24325 214/300 Read block 1377 len 16384
24326 215/300 Read block 4459 len 16384
24327 216/300 Read block 1357 len 4096
24328 217/300 Write block 2312 len 12288 data: 4 2 2
24329 218/300 Read block 2880 len 8192
24330 219/300 Read block 2104 len 4096
24331 220/300 Read block 1807 len 32768
24332 221/300 Write block 4882 len 36864 data: 2 3 4 4 7 8 7 7 7
24333 222/300 Write block 2337 len 24576 data: 5 4 4 5 4 3
24334 223/300 Flush
24335 224/300 Read block 2160 len 12288
24336 225/300 Write block 2155 len 20480 data: 6 6 5 5 5
24337 226/300 Write block 1364 len 36864 data: 8 5 6 5 6 5 5 5 4
24338 227/300 Write block 1365 len 8192 data: 6 7
24339 228/300 Write block 2778 len 24576 data: 6 3 3 3 2 3
24340 229/300 Read block 2775 len 12288
24341 230/300 Write block 1915 len 24576 data: 5 5 5 6 6 6
24342 231/300 Write block 3320 len 12288 data: 7 9 9
24343 232/300 Read block 1947 len 28672
24344 233/300 Read block 1606 len 40960
24345 234/300 Write block 4604 len 20480 data: 6 6 7 6 5
24346 235/300 Write block 1478 len 20480 data: 2 2 2 2 2
24347 236/300 Read block 428 len 32768
24348 237/300 Flush
24349 238/300 Flush
24350 239/300 Read block 3793 len 16384
24351 240/300 Flush
24352 241/300 Flush
24353 242/300 Write block 2312 len 8192 data: 5 3
24354 243/300 Flush
24355 244/300 Read block 3784 len 16384
24356 245/300 Read block 3608 len 24576
24357 246/300 Write block 899 len 16384 data: 7 10 9 9
24358 247/300 Read block 699 len 40960
24359 248/300 Flush
24360 249/300 Write block 312 len 28672 data: 5 5 5 4 3 2 2
24361 250/300 Read block 1457 len 40960
24362 251/300 Write block 2437 len 24576 data: 3 2 3 5 6 6
24363 252/300 Read block 3665 len 8192
24364 253/300 Read block 2057 len 20480
24365 254/300 Read block 2172 len 12288
24366 255/300 Flush
24367 256/300 Read block 2665 len 8192
24368 257/300 Flush
24369 258/300 Write block 2109 len 32768 data: 6 8 6 6 6 6 8 9
24370 259/300 Write block 2532 len 4096 data: 7
24371 260/300 Read block 235 len 36864
24372 261/300 Write block 802 len 28672 data: 6 6 5 6 4 3 3
24373 262/300 Read block 2859 len 16384
24374 263/300 Write block 2961 len 8192 data: 4 5
24375 264/300 Read block 1385 len 28672
24376 265/300 Write block 4655 len 24576 data: 6 6 6 8 9 11
24377 266/300 Read block 4264 len 40960
24378 267/300 Read block 2995 len 16384
24379 268/300 Read block 3168 len 8192
24380 269/300 Write block 3420 len 8192 data: 4 5
24381 270/300 Flush
24382 271/300 Write block 2138 len 36864 data: 4 6 6 6 5 4 4 5 5
24383 272/300 Read block 4841 len 40960
24384 273/300 Write block 1444 len 40960 data: 5 7 6 4 4 4 5 6 6 4
24385 274/300 Read block 681 len 24576
24386 275/300 Read block 3926 len 16384
24387 276/300 Write block 4 len 8192 data: 6 8
24388 277/300 Write block 1764 len 28672 data: 3 4 4 6 6 6 7
24389 278/300 Read block 4482 len 36864
24390 279/300 Read block 2062 len 8192
24391 280/300 Read block 3632 len 24576
24392 281/300 Read block 3094 len 36864
24393 282/300 Read block 3682 len 36864
24394 283/300 Read block 1499 len 28672
24395 284/300 Read block 3827 len 12288
24396 285/300 Write block 2069 len 32768 data: 4 4 4 4 4 4 3 3
24397 286/300 Read block 4173 len 36864
24398 287/300 Read block 3226 len 20480
24399 288/300 Flush
24400 289/300 Read block 3940 len 32768
24401 290/300 Write block 4115 len 40960 data: 4 4 3 3 3 4 6 5 6 6
24402 291/300 Read block 11 len 20480
24403 292/300 Read block 2548 len 32768
24404 293/300 Flush
24405 294/300 Read block 3540 len 4096
24406 295/300 Flush
24407 296/300 Read block 2963 len 28672
24408 297/300 Write block 1148 len 12288 data: 7 8 8
24409 298/300 Write block 2877 len 36864 data: 6 6 7 5 5 5 6 5 5
24410 299/300 Write block 4986 len 16384 data: 3 3 6 6
24411 300/300 Read block 3086 len 12288
24412 Main task has work to do, go find it
24413 got dsc Start(2)
24414 start 2
24415 Oct 13 04:47:07.810 INFO request completed, latency_us: 293, response_code: 204, uri: /start/cid/2, method: POST, req_id: 686ab8bb-0cc5-465f-b68e-0d9373d19773, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
24416 Replay: started 2, returned:Ok(())
24417 ----------------------------------------------------------------
24418 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
24419 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
24420 8464 Acked 9463 Read 8 Done Done New false
24421 8465 Acked 9464 Read 2 Done Done New false
24422 8466 Acked 9465 Write 10 Done Done New false
24423 8467 Acked 9466 Read 10 Done Done New false
24424 8468 Acked 9467 Read 7 Done Done New false
24425 8469 Acked 9468 Write 4 Done Done New false
24426 8470 Acked 9469 Read 1 Done Done New false
24427 8471 Acked 9470 Write 1 Done Done New false
24428 8472 Acked 9471 Write 6 Done Done New false
24429 8473 Acked 9472 Read 5 Done Done New false
24430 8474 Acked 9473 Read 4 Done Done New false
24431 8475 Acked 9474 Write 7 Done Done New false
24432 8476 Acked 9475 Read 5 Done Done New false
24433 8477 Acked 9476 Write 8 Done Done New false
24434 8478 Acked 9477 Read 2 Done Done New false
24435 8479 Acked 9478 Write 9 Done Done New false
24436 8480 Acked 9479 Read 4 Done Done New false
24437 8481 Acked 9480 Write 6 Done Done New false
24438 8482 Acked 9481 Read 3 Done Done New false
24439 8483 Acked 9482 Flush 0 Done Done New false
24440 8484 Acked 9483 Write 7 Done Done New false
24441 8485 Acked 9484 Read 6 Done Done New false
24442 8486 Acked 9485 Write 3 Done Done New false
24443 8487 Acked 9486 Read 4 Done Done New false
24444 8488 Acked 9487 Flush 0 Done Done New false
24445 8489 Acked 9488 Write 1 Done Done New false
24446 8490 Acked 9489 Flush 0 Done Done New false
24447 8491 Acked 9490 Write 5 Done Done New false
24448 8492 Acked 9491 Write 1 Done Done New false
24449 8493 Acked 9492 Write 9 Done Done New false
24450 8494 Acked 9493 Read 8 Done Done New false
24451 8495 Acked 9494 Write 8 Done Done New false
24452 8496 Acked 9495 Read 9 Done Done New false
24453 8497 Acked 9496 Read 6 Done Done New false
24454 8498 Acked 9497 Flush 0 Done Done New false
24455 8499 Acked 9498 Read 9 Done Done New false
24456 8500 Acked 9499 Write 5 Done Done New false
24457 8501 Acked 9500 Read 10 Done Done New false
24458 8502 Acked 9501 Read 9 Done Done New false
24459 8503 Acked 9502 Read 2 Done Done New false
24460 8504 Acked 9503 Read 10 Done Done New false
24461 8505 Acked 9504 Read 7 Done Done New false
24462 8506 Acked 9505 Read 5 Done Done New false
24463 8507 Acked 9506 Write 10 Done Done New false
24464 8508 Acked 9507 Read 5 Done Done New false
24465 8509 Acked 9508 Write 6 Done Done New false
24466 8510 Acked 9509 Read 6 Done Done New false
24467 8511 Acked 9510 Write 8 Done Done New false
24468 8512 Acked 9511 Read 2 Done Done New false
24469 8513 Acked 9512 Write 5 Done Done New false
24470 8514 Acked 9513 Write 8 Done Done New false
24471 8515 Acked 9514 Read 5 Done Done New false
24472 8516 Acked 9515 Read 10 Done Done New false
24473 8517 Acked 9516 Write 8 Done Done New false
24474 8518 Acked 9517 Flush 0 Done Done New false
24475 8519 Acked 9518 Write 2 Done Done New false
24476 8520 Acked 9519 Flush 0 Done Done New false
24477 8521 Acked 9520 Read 4 Done Done New false
24478 8522 Acked 9521 Read 8 Done Done New false
24479 8523 Acked 9522 Read 2 Done Done New false
24480 8524 Acked 9523 Read 7 Done Done New false
24481 8525 Acked 9524 Read 3 Done Done New false
24482 8526 Acked 9525 Read 3 Done Done New false
24483 8527 Acked 9526 Write 2 Done Done New false
24484 8528 Acked 9527 Write 9 Done Done New false
24485 8529 Acked 9528 Write 5 Done Done New false
24486 8530 Acked 9529 Write 8 Done Done New false
24487 8531 Acked 9530 Read 9 Done Done New false
24488 8532 Acked 9531 Write 2 Done Done New false
24489 8533 Acked 9532 Write 8 Done Done New false
24490 8534 Acked 9533 Write 10 Done Done New false
24491 8535 Acked 9534 Read 6 Done Done New false
24492 8536 Acked 9535 Write 10 Done Done New false
24493 8537 Acked 9536 Write 4 Done Done New false
24494 8538 Acked 9537 Read 2 Done Done New false
24495 8539 Acked 9538 Read 4 Done Done New false
24496 8540 Acked 9539 Read 7 Done Done New false
24497 8541 Acked 9540 Write 5 Done Done New false
24498 8542 Acked 9541 Write 8 Done Done New false
24499 8543 Acked 9542 Read 4 Done Done New false
24500 8544 Acked 9543 Write 1 Done Done New false
24501 8545 Acked 9544 Read 4 Done Done New false
24502 8546 Acked 9545 Write 5 Done Done New false
24503 8547 Acked 9546 Write 9 Done Done New false
24504 8548 Acked 9547 Read 2 Done Done New false
24505 8549 Acked 9548 Read 5 Done Done New false
24506 8550 Acked 9549 Write 1 Done Done New false
24507 8551 Acked 9550 Read 4 Done Done New false
24508 8552 Acked 9551 Write 1 Done Done New false
24509 8553 Acked 9552 Read 1 Done Done New false
24510 8554 Acked 9553 Write 5 Done Done New false
24511 8555 Acked 9554 Read 9 Done Done New false
24512 8556 Acked 9555 Flush 0 Done Done New false
24513 8557 Acked 9556 Flush 0 Done Done New false
24514 8558 Acked 9557 Read 3 Done Done New false
24515 8559 Acked 9558 Write 9 Done Done New false
24516 8560 Acked 9559 Read 4 Done Done New false
24517 8561 Acked 9560 Read 9 Done Done New false
24518 8562 Acked 9561 Flush 0 Done Done New false
24519 8563 Acked 9562 Read 10 Done Done New false
24520 8564 Acked 9563 Read 10 Done Done New false
24521 8565 Acked 9564 Write 9 Done Done New false
24522 8566 Acked 9565 Read 10 Done Done New false
24523 8567 Acked 9566 Read 5 Done Done New false
24524 8568 Acked 9567 Write 1 Done Done New false
24525 8569 Acked 9568 Read 10 Done Done New false
24526 8570 Acked 9569 Write 8 Done Done New false
24527 8571 Acked 9570 Read 10 Done Done New false
24528 8572 Acked 9571 Write 6 Done Done New false
24529 8573 Acked 9572 Write 4 Done Done New false
24530 8574 Acked 9573 Write 6 Done Done New false
24531 8575 Acked 9574 Flush 0 Done Done New false
24532 8576 Acked 9575 Flush 0 Done Done New false
24533 8577 Acked 9576 Write 6 Done Done New false
24534 8578 Acked 9577 Read 3 Done Done New false
24535 8579 Acked 9578 Read 1 Done Done New false
24536 8580 Acked 9579 Write 3 Done Done New false
24537 8581 Acked 9580 Write 2 Done Done New false
24538 8582 Acked 9581 Write 10 Done Done New false
24539 8583 Acked 9582 Write 3 Done Done New false
24540 8584 Acked 9583 Read 8 Done Done New false
24541 8585 Acked 9584 Write 2 Done Done New false
24542 8586 Acked 9585 Write 6 Done Done New false
24543 8587 Acked 9586 Write 8 Done Done New false
24544 8588 Acked 9587 Read 4 Done Done New false
24545 8589 Acked 9588 Write 2 Done Done New false
24546 8590 Acked 9589 Read 5 Done Done New false
24547 8591 Acked 9590 Flush 0 Done Done New false
24548 8592 Acked 9591 Flush 0 Done Done New false
24549 8593 Acked 9592 Read 6 Done Done New false
24550 8594 Acked 9593 Read 7 Done Done New false
24551 8595 Acked 9594 Read 2 Done Done New false
24552 8596 Acked 9595 Write 3 Done Done New false
24553 8597 Acked 9596 Read 6 Done Done New false
24554 8598 Acked 9597 Write 9 Done Done New false
24555 8599 Acked 9598 Write 7 Done Done New false
24556 8600 Acked 9599 Read 9 Done Done New false
24557 8601 Acked 9600 Write 2 Done Done New false
24558 8602 Acked 9601 Write 6 Done Done New false
24559 8603 Acked 9602 Write 7 Done Done New false
24560 8604 Acked 9603 Write 4 Done Done New false
24561 8605 Acked 9604 Read 7 Done Done New false
24562 8606 Acked 9605 Read 8 Done Done New false
24563 8607 Acked 9606 Read 6 Done Done New false
24564 8608 Acked 9607 Write 7 Done Done New false
24565 8609 Acked 9608 Read 3 Done Done New false
24566 8610 Acked 9609 Write 6 Done Done New false
24567 8611 Acked 9610 Write 8 Done Done New false
24568 8612 Acked 9611 Write 8 Done Done New false
24569 8613 Acked 9612 Flush 0 Done Done New false
24570 8614 Acked 9613 Write 3 Done Done New false
24571 8615 Acked 9614 Read 7 Done Done New false
24572 8616 Acked 9615 Read 7 Done Done New false
24573 8617 Acked 9616 Read 5 Done Done New false
24574 8618 Acked 9617 Read 9 Done Done New false
24575 8619 Acked 9618 Read 6 Done Done New false
24576 8620 Acked 9619 Write 4 Done Done New false
24577 8621 Acked 9620 Read 6 Done Done New false
24578 8622 Acked 9621 Write 6 Done Done New false
24579 8623 Acked 9622 Write 1 Done Done New false
24580 8624 Acked 9623 Read 6 Done Done New false
24581 8625 Acked 9624 Write 7 Done Done New false
24582 8626 Acked 9625 Read 7 Done Done New false
24583 8627 Acked 9626 Read 4 Done Done New false
24584 8628 Acked 9627 Flush 0 Done Done New false
24585 8629 Acked 9628 Read 5 Done Done New false
24586 8630 Acked 9629 Read 4 Done Done New false
24587 8631 Acked 9630 Read 4 Done Done New false
24588 8632 Acked 9631 Read 7 Done Done New false
24589 8633 Acked 9632 Read 9 Done Done New false
24590 8634 Acked 9633 Write 10 Done Done New false
24591 8635 Acked 9634 Write 9 Done Done New false
24592 8636 Acked 9635 Read 8 Done Done New false
24593 8637 Acked 9636 Read 5 Done Done New false
24594 8638 Acked 9637 Read 1 Done Done New false
24595 8639 Acked 9638 Read 9 Done Done New false
24596 8640 Acked 9639 Write 6 Done Done New false
24597 8641 Acked 9640 Read 6 Done Done New false
24598 8642 Acked 9641 Read 5 Done Done New false
24599 8643 Acked 9642 Flush 0 Done Done New false
24600 8644 Acked 9643 Write 5 Done Done New false
24601 8645 Acked 9644 Read 7 Done Done New false
24602 8646 Acked 9645 Read 5 Done Done New false
24603 8647 Acked 9646 Flush 0 Done Done New false
24604 8648 Acked 9647 Read 5 Done Done New false
24605 8649 Acked 9648 Flush 0 Done Done New false
24606 8650 Acked 9649 Write 3 Done Done New false
24607 8651 Acked 9650 Write 6 Done Done New false
24608 8652 Acked 9651 Read 9 Done Done New false
24609 8653 Acked 9652 Flush 0 Done Done New false
24610 8654 Acked 9653 Read 5 Done Done New false
24611 8655 Acked 9654 Read 5 Done Done New false
24612 8656 Acked 9655 Read 3 Done Done New false
24613 8657 Acked 9656 Read 2 Done Done New false
24614 8658 Acked 9657 Read 4 Done Done New false
24615 8659 Acked 9658 Write 8 Done Done New false
24616 8660 Acked 9659 Read 2 Done Done New false
24617 8661 Acked 9660 Write 9 Done Done New false
24618 8662 Acked 9661 Read 6 Done Done New false
24619 8663 Acked 9662 Flush 0 Done Done New false
24620 8664 Acked 9663 Write 3 Done Done New false
24621 8665 Acked 9664 Flush 0 Done Done New false
24622 8666 Acked 9665 Write 4 Done Done New false
24623 8667 Acked 9666 Write 5 Done Done New false
24624 8668 Acked 9667 Write 3 Done Done New false
24625 8669 Acked 9668 Flush 0 Done Done New false
24626 8670 Acked 9669 Write 9 Done Done New false
24627 8671 Acked 9670 Read 9 Done Done New false
24628 8672 Acked 9671 Read 10 Done Done New false
24629 8673 Acked 9672 Write 9 Done Done New false
24630 8674 Acked 9673 Write 1 Done Done New false
24631 8675 Acked 9674 Read 6 Done Done New false
24632 8676 Acked 9675 Write 5 Done Done New false
24633 8677 Acked 9676 Read 7 Done Done New false
24634 8678 Acked 9677 Read 6 Done Done New false
24635 8679 Acked 9678 Read 4 Done Done New false
24636 8680 Acked 9679 Read 4 Done Done New false
24637 8681 Acked 9680 Read 1 Done Done New false
24638 8682 Acked 9681 Write 3 Done Done New false
24639 8683 Acked 9682 Read 2 Done Done New false
24640 8684 Acked 9683 Read 1 Done Done New false
24641 8685 Acked 9684 Read 8 Done Done New false
24642 8686 Acked 9685 Write 9 Done Done New false
24643 8687 Acked 9686 Write 6 Done Done New false
24644 8688 Acked 9687 Flush 0 Done Done New false
24645 8689 Acked 9688 Read 3 Done Done New false
24646 8690 Acked 9689 Write 5 Done Done New false
24647 8691 Acked 9690 Write 9 Done Done New false
24648 8692 Acked 9691 Write 2 Done Done New false
24649 8693 Acked 9692 Write 6 Done Done New false
24650 8694 Acked 9693 Read 3 Done Done New false
24651 8695 Acked 9694 Write 6 Done Done New false
24652 8696 Acked 9695 Write 3 Done Done New false
24653 8697 Acked 9696 Read 7 Done Done New false
24654 8698 Acked 9697 Read 10 Done Done New false
24655 8699 Acked 9698 Write 5 Done Done New false
24656 8700 Acked 9699 Write 5 Done Done New false
24657 8701 Acked 9700 Read 8 Done Done New false
24658 8702 Acked 9701 Flush 0 Done Done New false
24659 8703 Acked 9702 Flush 0 Done Done New false
24660 8704 Acked 9703 Read 4 Done Done New false
24661 8705 Acked 9704 Flush 0 Done Done New false
24662 8706 Acked 9705 Flush 0 Done Done New false
24663 8707 Acked 9706 Write 2 Done Done New false
24664 8708 Acked 9707 Flush 0 Done Done New false
24665 8709 Acked 9708 Read 4 Done Done New false
24666 8710 Acked 9709 Read 6 Done Done New false
24667 8711 Acked 9710 Write 4 Done Done New false
24668 8712 Acked 9711 Read 10 Done Done New false
24669 8713 Acked 9712 Flush 0 Done Done New false
24670 8714 Acked 9713 Write 7 Done Done New false
24671 8715 Acked 9714 Read 10 Done Done New false
24672 8716 Acked 9715 Write 6 Done Done New false
24673 8717 Acked 9716 Read 2 Done Done New false
24674 8718 Acked 9717 Read 5 Done Done New false
24675 8719 Acked 9718 Read 3 Done Done New false
24676 8720 Acked 9719 Flush 0 Done Done New false
24677 8721 Acked 9720 Read 2 Done Done New false
24678 8722 Acked 9721 Flush 0 Done Done New false
24679 8723 Acked 9722 Write 8 Done Done New false
24680 8724 Acked 9723 Write 1 Done Done New false
24681 8725 Acked 9724 Read 9 Done Done New false
24682 8726 Acked 9725 Write 7 Done Done New false
24683 8727 Acked 9726 Read 4 Done Done New false
24684 8728 Acked 9727 Write 2 Done Done New false
24685 8729 Acked 9728 Read 7 Done Done New false
24686 8730 Acked 9729 Write 6 Done Done New false
24687 8731 Acked 9730 Read 10 Done Done New false
24688 8732 Acked 9731 Read 4 Done Done New false
24689 8733 Acked 9732 Read 2 Done Done New false
24690 8734 Acked 9733 Write 2 Done Done New false
24691 8735 Acked 9734 Flush 0 Done Done New false
24692 8736 Acked 9735 Write 9 Done Done New false
24693 8737 Acked 9736 Read 10 Done Done New false
24694 8738 Acked 9737 Write 10 Done Done New false
24695 8739 Acked 9738 Read 6 Done Done New false
24696 8740 Acked 9739 Read 4 Done Done New false
24697 8741 Acked 9740 Write 2 Done Done New false
24698 8742 Acked 9741 Write 7 Done Done New false
24699 8743 Acked 9742 Read 9 Done Done New false
24700 8744 Acked 9743 Read 2 Done Done New false
24701 8745 Acked 9744 Read 6 Done Done New false
24702 8746 Acked 9745 Read 9 Done Done New false
24703 8747 Acked 9746 Read 9 Done Done New false
24704 8748 Acked 9747 Read 7 Done Done New false
24705 8749 Acked 9748 Read 3 Done Done New false
24706 8750 Acked 9749 Write 8 Done Done New false
24707 8751 Acked 9750 Read 9 Done Done New false
24708 8752 Acked 9751 Read 5 Done Done New false
24709 8753 Acked 9752 Flush 0 Done Done New false
24710 8754 Acked 9753 Read 8 Done Done New false
24711 8755 Acked 9754 Write 10 Done Done New false
24712 8756 Acked 9755 Read 5 Done Done New false
24713 8757 Acked 9756 Read 8 Done Done New false
24714 8758 Acked 9757 Flush 0 Done Done New false
24715 8759 Acked 9758 Read 1 Done Done New false
24716 8760 Acked 9759 Flush 0 Done Done New false
24717 8761 Acked 9760 Read 7 Done Done New false
24718 8762 Acked 9761 Write 3 Done Done New false
24719 8763 Acked 9762 Write 9 Done Done New false
24720 8764 Acked 9763 Write 4 Done Done New false
24721 8765 Acked 9764 Read 3 Done Done New false
24722 STATES DS:0 DS:1 DS:2 TOTAL
24723 New 0 0 302 302
24724 Sent 0 0 0 0
24725 Done 302 302 0 604
24726 Skipped 0 0 0 0
24727 Error 0 0 0 0
24728 Last Flush: 9759 9759 9462
24729 Downstairs last five completed: 9462 9461 9460 9459 9458 9457
24730 Upstairs last five completed: 8765 8764 8763 8762 8761 8760
24731 CLIENT: Up:0 ds:302 act:2
24732 ----------------------------------------------------------------
24733 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:303
24734 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
24735 8464 Acked 9463 Read 8 Done Done New false
24736 8465 Acked 9464 Read 2 Done Done New false
24737 8466 Acked 9465 Write 10 Done Done New false
24738 8467 Acked 9466 Read 10 Done Done New false
24739 8468 Acked 9467 Read 7 Done Done New false
24740 8469 Acked 9468 Write 4 Done Done New false
24741 8470 Acked 9469 Read 1 Done Done New false
24742 8471 Acked 9470 Write 1 Done Done New false
24743 8472 Acked 9471 Write 6 Done Done New false
24744 8473 Acked 9472 Read 5 Done Done New false
24745 8474 Acked 9473 Read 4 Done Done New false
24746 8475 Acked 9474 Write 7 Done Done New false
24747 8476 Acked 9475 Read 5 Done Done New false
24748 8477 Acked 9476 Write 8 Done Done New false
24749 8478 Acked 9477 Read 2 Done Done New false
24750 8479 Acked 9478 Write 9 Done Done New false
24751 8480 Acked 9479 Read 4 Done Done New false
24752 8481 Acked 9480 Write 6 Done Done New false
24753 8482 Acked 9481 Read 3 Done Done New false
24754 8483 Acked 9482 Flush 0 Done Done New false
24755 8484 Acked 9483 Write 7 Done Done New false
24756 8485 Acked 9484 Read 6 Done Done New false
24757 8486 Acked 9485 Write 3 Done Done New false
24758 8487 Acked 9486 Read 4 Done Done New false
24759 8488 Acked 9487 Flush 0 Done Done New false
24760 8489 Acked 9488 Write 1 Done Done New false
24761 8490 Acked 9489 Flush 0 Done Done New false
24762 8491 Acked 9490 Write 5 Done Done New false
24763 8492 Acked 9491 Write 1 Done Done New false
24764 8493 Acked 9492 Write 9 Done Done New false
24765 8494 Acked 9493 Read 8 Done Done New false
24766 8495 Acked 9494 Write 8 Done Done New false
24767 8496 Acked 9495 Read 9 Done Done New false
24768 8497 Acked 9496 Read 6 Done Done New false
24769 8498 Acked 9497 Flush 0 Done Done New false
24770 8499 Acked 9498 Read 9 Done Done New false
24771 8500 Acked 9499 Write 5 Done Done New false
24772 8501 Acked 9500 Read 10 Done Done New false
24773 8502 Acked 9501 Read 9 Done Done New false
24774 8503 Acked 9502 Read 2 Done Done New false
24775 8504 Acked 9503 Read 10 Done Done New false
24776 8505 Acked 9504 Read 7 Done Done New false
24777 8506 Acked 9505 Read 5 Done Done New false
24778 8507 Acked 9506 Write 10 Done Done New false
24779 8508 Acked 9507 Read 5 Done Done New false
24780 8509 Acked 9508 Write 6 Done Done New false
24781 8510 Acked 9509 Read 6 Done Done New false
24782 8511 Acked 9510 Write 8 Done Done New false
24783 8512 Acked 9511 Read 2 Done Done New false
24784 8513 Acked 9512 Write 5 Done Done New false
24785 8514 Acked 9513 Write 8 Done Done New false
24786 8515 Acked 9514 Read 5 Done Done New false
24787 8516 Acked 9515 Read 10 Done Done New false
24788 8517 Acked 9516 Write 8 Done Done New false
24789 8518 Acked 9517 Flush 0 Done Done New false
24790 8519 Acked 9518 Write 2 Done Done New false
24791 8520 Acked 9519 Flush 0 Done Done New false
24792 8521 Acked 9520 Read 4 Done Done New false
24793 8522 Acked 9521 Read 8 Done Done New false
24794 8523 Acked 9522 Read 2 Done Done New false
24795 8524 Acked 9523 Read 7 Done Done New false
24796 8525 Acked 9524 Read 3 Done Done New false
24797 8526 Acked 9525 Read 3 Done Done New false
24798 8527 Acked 9526 Write 2 Done Done New false
24799 8528 Acked 9527 Write 9 Done Done New false
24800 8529 Acked 9528 Write 5 Done Done New false
24801 8530 Acked 9529 Write 8 Done Done New false
24802 8531 Acked 9530 Read 9 Done Done New false
24803 8532 Acked 9531 Write 2 Done Done New false
24804 8533 Acked 9532 Write 8 Done Done New false
24805 8534 Acked 9533 Write 10 Done Done New false
24806 8535 Acked 9534 Read 6 Done Done New false
24807 8536 Acked 9535 Write 10 Done Done New false
24808 8537 Acked 9536 Write 4 Done Done New false
24809 8538 Acked 9537 Read 2 Done Done New false
24810 8539 Acked 9538 Read 4 Done Done New false
24811 8540 Acked 9539 Read 7 Done Done New false
24812 8541 Acked 9540 Write 5 Done Done New false
24813 8542 Acked 9541 Write 8 Done Done New false
24814 8543 Acked 9542 Read 4 Done Done New false
24815 8544 Acked 9543 Write 1 Done Done New false
24816 8545 Acked 9544 Read 4 Done Done New false
24817 8546 Acked 9545 Write 5 Done Done New false
24818 8547 Acked 9546 Write 9 Done Done New false
24819 8548 Acked 9547 Read 2 Done Done New false
24820 8549 Acked 9548 Read 5 Done Done New false
24821 8550 Acked 9549 Write 1 Done Done New false
24822 8551 Acked 9550 Read 4 Done Done New false
24823 8552 Acked 9551 Write 1 Done Done New false
24824 8553 Acked 9552 Read 1 Done Done New false
24825 8554 Acked 9553 Write 5 Done Done New false
24826 8555 Acked 9554 Read 9 Done Done New false
24827 8556 Acked 9555 Flush 0 Done Done New false
24828 8557 Acked 9556 Flush 0 Done Done New false
24829 8558 Acked 9557 Read 3 Done Done New false
24830 8559 Acked 9558 Write 9 Done Done New false
24831 8560 Acked 9559 Read 4 Done Done New false
24832 8561 Acked 9560 Read 9 Done Done New false
24833 8562 Acked 9561 Flush 0 Done Done New false
24834 8563 Acked 9562 Read 10 Done Done New false
24835 8564 Acked 9563 Read 10 Done Done New false
24836 8565 Acked 9564 Write 9 Done Done New false
24837 8566 Acked 9565 Read 10 Done Done New false
24838 8567 Acked 9566 Read 5 Done Done New false
24839 8568 Acked 9567 Write 1 Done Done New false
24840 8569 Acked 9568 Read 10 Done Done New false
24841 8570 Acked 9569 Write 8 Done Done New false
24842 8571 Acked 9570 Read 10 Done Done New false
24843 8572 Acked 9571 Write 6 Done Done New false
24844 8573 Acked 9572 Write 4 Done Done New false
24845 8574 Acked 9573 Write 6 Done Done New false
24846 8575 Acked 9574 Flush 0 Done Done New false
24847 8576 Acked 9575 Flush 0 Done Done New false
24848 8577 Acked 9576 Write 6 Done Done New false
24849 8578 Acked 9577 Read 3 Done Done New false
24850 8579 Acked 9578 Read 1 Done Done New false
24851 8580 Acked 9579 Write 3 Done Done New false
24852 8581 Acked 9580 Write 2 Done Done New false
24853 8582 Acked 9581 Write 10 Done Done New false
24854 8583 Acked 9582 Write 3 Done Done New false
24855 8584 Acked 9583 Read 8 Done Done New false
24856 8585 Acked 9584 Write 2 Done Done New false
24857 8586 Acked 9585 Write 6 Done Done New false
24858 8587 Acked 9586 Write 8 Done Done New false
24859 8588 Acked 9587 Read 4 Done Done New false
24860 8589 Acked 9588 Write 2 Done Done New false
24861 8590 Acked 9589 Read 5 Done Done New false
24862 8591 Acked 9590 Flush 0 Done Done New false
24863 8592 Acked 9591 Flush 0 Done Done New false
24864 8593 Acked 9592 Read 6 Done Done New false
24865 8594 Acked 9593 Read 7 Done Done New false
24866 8595 Acked 9594 Read 2 Done Done New false
24867 8596 Acked 9595 Write 3 Done Done New false
24868 8597 Acked 9596 Read 6 Done Done New false
24869 8598 Acked 9597 Write 9 Done Done New false
24870 8599 Acked 9598 Write 7 Done Done New false
24871 8600 Acked 9599 Read 9 Done Done New false
24872 8601 Acked 9600 Write 2 Done Done New false
24873 8602 Acked 9601 Write 6 Done Done New false
24874 8603 Acked 9602 Write 7 Done Done New false
24875 8604 Acked 9603 Write 4 Done Done New false
24876 8605 Acked 9604 Read 7 Done Done New false
24877 8606 Acked 9605 Read 8 Done Done New false
24878 8607 Acked 9606 Read 6 Done Done New false
24879 8608 Acked 9607 Write 7 Done Done New false
24880 8609 Acked 9608 Read 3 Done Done New false
24881 8610 Acked 9609 Write 6 Done Done New false
24882 8611 Acked 9610 Write 8 Done Done New false
24883 8612 Acked 9611 Write 8 Done Done New false
24884 8613 Acked 9612 Flush 0 Done Done New false
24885 8614 Acked 9613 Write 3 Done Done New false
24886 8615 Acked 9614 Read 7 Done Done New false
24887 8616 Acked 9615 Read 7 Done Done New false
24888 8617 Acked 9616 Read 5 Done Done New false
24889 8618 Acked 9617 Read 9 Done Done New false
24890 8619 Acked 9618 Read 6 Done Done New false
24891 8620 Acked 9619 Write 4 Done Done New false
24892 8621 Acked 9620 Read 6 Done Done New false
24893 8622 Acked 9621 Write 6 Done Done New false
24894 8623 Acked 9622 Write 1 Done Done New false
24895 8624 Acked 9623 Read 6 Done Done New false
24896 8625 Acked 9624 Write 7 Done Done New false
24897 8626 Acked 9625 Read 7 Done Done New false
24898 8627 Acked 9626 Read 4 Done Done New false
24899 8628 Acked 9627 Flush 0 Done Done New false
24900 8629 Acked 9628 Read 5 Done Done New false
24901 8630 Acked 9629 Read 4 Done Done New false
24902 8631 Acked 9630 Read 4 Done Done New false
24903 8632 Acked 9631 Read 7 Done Done New false
24904 8633 Acked 9632 Read 9 Done Done New false
24905 8634 Acked 9633 Write 10 Done Done New false
24906 8635 Acked 9634 Write 9 Done Done New false
24907 8636 Acked 9635 Read 8 Done Done New false
24908 8637 Acked 9636 Read 5 Done Done New false
24909 8638 Acked 9637 Read 1 Done Done New false
24910 8639 Acked 9638 Read 9 Done Done New false
24911 8640 Acked 9639 Write 6 Done Done New false
24912 8641 Acked 9640 Read 6 Done Done New false
24913 8642 Acked 9641 Read 5 Done Done New false
24914 8643 Acked 9642 Flush 0 Done Done New false
24915 8644 Acked 9643 Write 5 Done Done New false
24916 8645 Acked 9644 Read 7 Done Done New false
24917 8646 Acked 9645 Read 5 Done Done New false
24918 8647 Acked 9646 Flush 0 Done Done New false
24919 8648 Acked 9647 Read 5 Done Done New false
24920 8649 Acked 9648 Flush 0 Done Done New false
24921 8650 Acked 9649 Write 3 Done Done New false
24922 8651 Acked 9650 Write 6 Done Done New false
24923 8652 Acked 9651 Read 9 Done Done New false
24924 8653 Acked 9652 Flush 0 Done Done New false
24925 8654 Acked 9653 Read 5 Done Done New false
24926 8655 Acked 9654 Read 5 Done Done New false
24927 8656 Acked 9655 Read 3 Done Done New false
24928 8657 Acked 9656 Read 2 Done Done New false
24929 8658 Acked 9657 Read 4 Done Done New false
24930 8659 Acked 9658 Write 8 Done Done New false
24931 8660 Acked 9659 Read 2 Done Done New false
24932 8661 Acked 9660 Write 9 Done Done New false
24933 8662 Acked 9661 Read 6 Done Done New false
24934 8663 Acked 9662 Flush 0 Done Done New false
24935 8664 Acked 9663 Write 3 Done Done New false
24936 8665 Acked 9664 Flush 0 Done Done New false
24937 8666 Acked 9665 Write 4 Done Done New false
24938 8667 Acked 9666 Write 5 Done Done New false
24939 8668 Acked 9667 Write 3 Done Done New false
24940 8669 Acked 9668 Flush 0 Done Done New false
24941 8670 Acked 9669 Write 9 Done Done New false
24942 8671 Acked 9670 Read 9 Done Done New false
24943 8672 Acked 9671 Read 10 Done Done New false
24944 8673 Acked 9672 Write 9 Done Done New false
24945 8674 Acked 9673 Write 1 Done Done New false
24946 8675 Acked 9674 Read 6 Done Done New false
24947 8676 Acked 9675 Write 5 Done Done New false
24948 8677 Acked 9676 Read 7 Done Done New false
24949 8678 Acked 9677 Read 6 Done Done New false
24950 8679 Acked 9678 Read 4 Done Done New false
24951 8680 Acked 9679 Read 4 Done Done New false
24952 8681 Acked 9680 Read 1 Done Done New false
24953 8682 Acked 9681 Write 3 Done Done New false
24954 8683 Acked 9682 Read 2 Done Done New false
24955 8684 Acked 9683 Read 1 Done Done New false
24956 8685 Acked 9684 Read 8 Done Done New false
24957 8686 Acked 9685 Write 9 Done Done New false
24958 8687 Acked 9686 Write 6 Done Done New false
24959 8688 Acked 9687 Flush 0 Done Done New false
24960 8689 Acked 9688 Read 3 Done Done New false
24961 8690 Acked 9689 Write 5 Done Done New false
24962 8691 Acked 9690 Write 9 Done Done New false
24963 8692 Acked 9691 Write 2 Done Done New false
24964 8693 Acked 9692 Write 6 Done Done New false
24965 8694 Acked 9693 Read 3 Done Done New false
24966 8695 Acked 9694 Write 6 Done Done New false
24967 8696 Acked 9695 Write 3 Done Done New false
24968 8697 Acked 9696 Read 7 Done Done New false
24969 8698 Acked 9697 Read 10 Done Done New false
24970 8699 Acked 9698 Write 5 Done Done New false
24971 8700 Acked 9699 Write 5 Done Done New false
24972 8701 Acked 9700 Read 8 Done Done New false
24973 8702 Acked 9701 Flush 0 Done Done New false
24974 8703 Acked 9702 Flush 0 Done Done New false
24975 8704 Acked 9703 Read 4 Done Done New false
24976 8705 Acked 9704 Flush 0 Done Done New false
24977 8706 Acked 9705 Flush 0 Done Done New false
24978 8707 Acked 9706 Write 2 Done Done New false
24979 8708 Acked 9707 Flush 0 Done Done New false
24980 8709 Acked 9708 Read 4 Done Done New false
24981 8710 Acked 9709 Read 6 Done Done New false
24982 8711 Acked 9710 Write 4 Done Done New false
24983 8712 Acked 9711 Read 10 Done Done New false
24984 8713 Acked 9712 Flush 0 Done Done New false
24985 8714 Acked 9713 Write 7 Done Done New false
24986 8715 Acked 9714 Read 10 Done Done New false
24987 8716 Acked 9715 Write 6 Done Done New false
24988 8717 Acked 9716 Read 2 Done Done New false
24989 8718 Acked 9717 Read 5 Done Done New false
24990 8719 Acked 9718 Read 3 Done Done New false
24991 8720 Acked 9719 Flush 0 Done Done New false
24992 8721 Acked 9720 Read 2 Done Done New false
24993 8722 Acked 9721 Flush 0 Done Done New false
24994 8723 Acked 9722 Write 8 Done Done New false
24995 8724 Acked 9723 Write 1 Done Done New false
24996 8725 Acked 9724 Read 9 Done Done New false
24997 8726 Acked 9725 Write 7 Done Done New false
24998 8727 Acked 9726 Read 4 Done Done New false
24999 8728 Acked 9727 Write 2 Done Done New false
25000 8729 Acked 9728 Read 7 Done Done New false
25001 8730 Acked 9729 Write 6 Done Done New false
25002 8731 Acked 9730 Read 10 Done Done New false
25003 8732 Acked 9731 Read 4 Done Done New false
25004 8733 Acked 9732 Read 2 Done Done New false
25005 8734 Acked 9733 Write 2 Done Done New false
25006 8735 Acked 9734 Flush 0 Done Done New false
25007 8736 Acked 9735 Write 9 Done Done New false
25008 8737 Acked 9736 Read 10 Done Done New false
25009 8738 Acked 9737 Write 10 Done Done New false
25010 8739 Acked 9738 Read 6 Done Done New false
25011 8740 Acked 9739 Read 4 Done Done New false
25012 8741 Acked 9740 Write 2 Done Done New false
25013 8742 Acked 9741 Write 7 Done Done New false
25014 8743 Acked 9742 Read 9 Done Done New false
25015 8744 Acked 9743 Read 2 Done Done New false
25016 8745 Acked 9744 Read 6 Done Done New false
25017 8746 Acked 9745 Read 9 Done Done New false
25018 8747 Acked 9746 Read 9 Done Done New false
25019 8748 Acked 9747 Read 7 Done Done New false
25020 8749 Acked 9748 Read 3 Done Done New false
25021 8750 Acked 9749 Write 8 Done Done New false
25022 8751 Acked 9750 Read 9 Done Done New false
25023 8752 Acked 9751 Read 5 Done Done New false
25024 8753 Acked 9752 Flush 0 Done Done New false
25025 8754 Acked 9753 Read 8 Done Done New false
25026 8755 Acked 9754 Write 10 Done Done New false
25027 8756 Acked 9755 Read 5 Done Done New false
25028 8757 Acked 9756 Read 8 Done Done New false
25029 8758 Acked 9757 Flush 0 Done Done New false
25030 8759 Acked 9758 Read 1 Done Done New false
25031 8760 Acked 9759 Flush 0 Done Done New false
25032 8761 Acked 9760 Read 7 Done Done New false
25033 8762 Acked 9761 Write 3 Done Done New false
25034 8763 Acked 9762 Write 9 Done Done New false
25035 8764 Acked 9763 Write 4 Done Done New false
25036 8765 Acked 9764 Read 3 Done Done New false
25037 8766 Acked 9765 Flush 0 Done Done New false
25038 STATES DS:0 DS:1 DS:2 TOTAL
25039 New 0 0 303 303
25040 Sent 0 0 0 0
25041 Done 303 303 0 606
25042 Skipped 0 0 0 0
25043 Error 0 0 0 0
25044 Last Flush: 9765 9765 9462
25045 Downstairs last five completed: 9462 9461 9460 9459 9458 9457
25046 Upstairs last five completed: 8766 8765 8764 8763 8762 8761
25047 CLIENT: Up:0 ds:303 act:2
250482023-10-13T04:47:12.917ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
25049 [8830] got start action
25050 [8830] I am going to restart
25051 Starting downstairs at port 8830
25052 Make output file at "/tmp/dsc/downstairs-8830.txt"
25053 [8830][2] reports Starting
25054 Downstairs /var/tmp/dsc/region/8830 port 8830 PID:Some(1205)
25055 [8830][2] reports Running
250562023-10-13T04:47:13.918ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 2
250572023-10-13T04:47:13.918ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state Offline
250582023-10-13T04:47:13.918ZINFOcrucible: [2] client re-new 303 jobs since flush 9462 = downstairs
250592023-10-13T04:47:13.919ZINFOcrucible: [2] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
250602023-10-13T04:47:13.920ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
250612023-10-13T04:47:13.920ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
250622023-10-13T04:47:13.920ZINFOcrucible: Returning client:2 UUID:12345678-0000-0000-0000-000000008830 matches
250632023-10-13T04:47:13.920ZINFOcrucible: [2] send last flush ID to this DS: 9462
250642023-10-13T04:47:13.920ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Offline
250652023-10-13T04:47:13.920ZINFOcrucible: [2] Replied this last flush ID: 9462
250662023-10-13T04:47:13.920ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Offline ds_transition to Replay
250672023-10-13T04:47:13.920ZINFOcrucible: [2] Transition from Offline to Replay
250682023-10-13T04:47:13.920ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
250692023-10-13T04:47:13.920ZINFOcrucible: [2] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Active Replay ds_transition to Active
250702023-10-13T04:47:13.920ZINFOcrucible: [2] Transition from Replay to Active
250712023-10-13T04:47:13.920ZINFOcrucible: [2] Starts cmd_loop
250722023-10-13T04:47:13.920ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
250732023-10-13T04:47:13.920ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
250742023-10-13T04:47:13.920ZINFOcrucible: Set check for repair
250752023-10-13T04:47:14.921ZINFOcrucible: Checking if live repair is needed
250762023-10-13T04:47:14.921ZINFOcrucible: No Live Repair required at this time
250772023-10-13T04:47:14.921ZWARNcrucible: [2] flow control end
25078 ----------------------------------------------------------------
25079 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25080 Downstairs last five completed: 9765 9764 9763 9762 9761 9760
25081 Upstairs last five completed: 8766 8765 8764 8763 8762 8761
25082 CLIENT: Up:0 ds:0 act:3
25083 Replay: All jobs finished, all DS active.
25084 Main task has work to do, go find it
25085 got dsc Stop(1)
25086 stop 1
25087 Oct 13 04:47:15.817 INFO request completed, latency_us: 495, response_code: 204, uri: /stop/cid/1, method: POST, req_id: f7c8d29e-3fe4-4963-ae6e-6ccdb0728a9a, remote_addr: 127.0.0.1:52046, local_addr: 127[8820] Got stop action so:false kr:false
25088 .0.0.1:9998
25089 Oct 13 04:47:15.818 INFO request completed, latency_us: 208, response_code: 200, uri: /state/cid/1, method: GET, req_id: e05b6659-f8c7-420a-b10b-c93bc9eb57b6, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
250902023-10-13T04:47:15.819ZWARNcrucible: [1] downstairs disconnected
250912023-10-13T04:47:15.819ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Gone missing, transition from Active to Offline
250922023-10-13T04:47:15.819ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b connection to 127.0.0.1:8820 closed looper = 1
250932023-10-13T04:47:15.819ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:false
250942023-10-13T04:47:15.819ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
250952023-10-13T04:47:15.819ZINFOcrucible: [1] 127.0.0.1:8820 task reports offline
250962023-10-13T04:47:15.819ZWARNcrucible: [1] pm_task rx.recv() is None
25097 [8820] Exited with: ExitStatus(unix_wait_status(9))
25098 [8820][1] reports Exit
250992023-10-13T04:47:18.822ZINFOcrucible: [1] connecting to 127.0.0.1:8820 looper = 1
25100 Oct 13 04:47:19.821 INFO request completed, latency_us: 254, response_code: 200, uri: /state/cid/1, method: GET, req_id: 6763c133-bc66-46ab-9dfe-349d6f0feb2a, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
25101 001/300 Read block 1027 len 24576
25102 002/300 Write block 4324 len 20480 data: 4 3 3 4 4
25103 003/300 Write block 1095 len 28672 data: 7 7 8 10 8 8 9
25104 004/300 Flush
25105 005/300 Read block 2604 len 36864
25106 006/300 Read block 2397 len 36864
25107 007/300 Write block 4202 len 40960 data: 7 7 6 6 6 6 7 7 8 10
25108 008/300 Read block 1386 len 40960
25109 009/300 Write block 4811 len 40960 data: 6 6 6 5 4 5 7 6 7 5
25110 010/300 Read block 2569 len 32768
25111 011/300 Read block 4630 len 20480
25112 012/300 Read block 11 len 20480
25113 013/300 Read block 3535 len 4096
25114 014/300 Write block 1630 len 4096 data: 9
25115 015/300 Read block 1748 len 40960
25116 016/300 Read block 3567 len 36864
25117 017/300 Read block 367 len 4096
25118 018/300 Read block 1442 len 16384
25119 019/300 Write block 196 len 8192 data: 9 8
25120 020/300 Read block 3812 len 28672
25121 021/300 Read block 3802 len 40960
25122 022/300 Read block 606 len 4096
25123 023/300 Write block 2841 len 16384 data: 7 7 5 5
25124 024/300 Write block 4866 len 8192 data: 8 5
25125 025/300 Write block 1492 len 12288 data: 8 8 8
25126 026/300 Write block 730 len 4096 data: 5
25127 027/300 Flush
25128 028/300 Write block 2208 len 32768 data: 5 6 6 6 6 5 4 4
25129 029/300 Write block 1638 len 20480 data: 9 7 7 6 7
25130 030/300 Read block 3640 len 24576
25131 031/300 Write block 1273 len 32768 data: 6 6 6 5 7 5 4 6
25132 032/300 Write block 3731 len 16384 data: 4 3 2 2
25133 033/300 Write block 2666 len 12288 data: 4 4 4
25134 034/300 Read block 4528 len 8192
25135 035/300 Write block 4803 len 32768 data: 5 5 4 3 5 6 6 5
25136 036/300 Read block 3857 len 36864
25137 037/300 Read block 2973 len 36864
25138 038/300 Read block 3688 len 24576
25139 039/300 Read block 630 len 16384
25140 040/300 Write block 1896 len 16384 data: 4 4 5 5
25141 041/300 Read block 2800 len 28672
25142 042/300 Read block 3933 len 40960
25143 043/300 Write block 3077 len 12288 data: 5 6 7
25144 044/300 Flush
25145 045/300 Write block 1810 len 12288 data: 7 8 8
25146 046/300 Write block 4133 len 4096 data: 7
25147 047/300 Read block 4649 len 40960
25148 048/300 Read block 1818 len 8192
25149 049/300 Write block 393 len 8192 data: 5 4
25150 050/300 Read block 4669 len 4096
25151 051/300 Write block 4353 len 28672 data: 7 5 5 5 6 6 6
25152 052/300 Write block 4617 len 32768 data: 7 6 6 5 7 9 9 9
25153 053/300 Read block 3038 len 28672
25154 054/300 Write block 1975 len 4096 data: 5
25155 055/300 Read block 3299 len 4096
25156 056/300 Read block 4676 len 12288
25157 057/300 Write block 1170 len 20480 data: 7 4 4 5 3
25158 058/300 Write block 206 len 4096 data: 7
25159 059/300 Write block 4776 len 16384 data: 7 9 10 11
25160 060/300 Flush
25161 061/300 Read block 1202 len 12288
25162 062/300 Write block 2958 len 4096 data: 4
25163 063/300 Read block 1612 len 12288
25164 064/300 Read block 1162 len 4096
25165 065/300 Read block 1751 len 36864
25166 066/300 Read block 2412 len 40960
25167 067/300 Read block 48 len 12288
25168 068/300 Read block 3007 len 36864
25169 069/300 Write block 1986 len 16384 data: 3 4 4 5
25170 070/300 Write block 690 len 40960 data: 3 2 6 7 6 8 8 7 7 6
25171 071/300 Read block 130 len 32768
25172 072/300 Read block 4215 len 16384
25173 073/300 Write block 3565 len 16384 data: 4 3 4 3
25174 074/300 Write block 4234 len 12288 data: 3 3 3
25175 075/300 Read block 2294 len 32768
25176 076/300 Write block 766 len 40960 data: 5 3 3 3 4 4 5 4 4 6
25177 077/300 Write block 617 len 28672 data: 9 8 10 7 8 7 5
25178 078/300 Read block 1991 len 12288
25179 079/300 Read block 4641 len 32768
25180 080/300 Write block 4025 len 4096 data: 9
25181 081/300 Write block 1903 len 8192 data: 5 4
25182 082/300 Read block 598 len 12288
25183 083/300 Flush
25184 084/300 Read block 1472 len 40960
25185 085/300 Read block 4895 len 24576
25186 086/300 Write block 2227 len 20480 data: 3 3 3 3 5
25187 087/300 Write block 4961 len 12288 data: 3 3 4
25188 088/300 Read block 3164 len 28672
25189 089/300 Write block 1823 len 40960 data: 7 7 7 7 7 6 6 8 6 6
25190 090/300 Read block 3986 len 40960
25191 091/300 Flush
25192 092/300 Read block 4691 len 16384
25193 093/300 Read block 1638 len 16384
25194 094/300 Read block 548 len 36864
25195 095/300 Read block 2747 len 36864
25196 096/300 Read block 1065 len 28672
25197 097/300 Read block 1502 len 20480
25198 098/300 Read block 1270 len 32768
25199 099/300 Write block 4324 len 20480 data: 5 4 4 5 5
25200 100/300 Read block 491 len 16384
25201 101/300 Write block 177 len 12288 data: 6 6 6
25202 102/300 Read block 268 len 24576
25203 103/300 Read block 475 len 32768
25204 104/300 Read block 1443 len 24576
25205 105/300 Flush
25206 106/300 Read block 1770 len 40960
25207 107/300 Read block 4622 len 32768
25208 108/300 Write block 2332 len 8192 data: 5 4
25209 109/300 Read block 1264 len 36864
25210 110/300 Read block 2183 len 20480
25211 111/300 Read block 1223 len 24576
25212 112/300 Read block 278 len 16384
25213 113/300 Read block 810 len 36864
25214 114/300 Read block 4710 len 40960
25215 115/300 Write block 4139 len 36864 data: 5 4 4 5 3 3 4 4 5
25216 116/300 Write block 1910 len 24576 data: 3 5 6 5 5 6
25217 117/300 Write block 2309 len 12288 data: 5 5 5
25218 118/300 Write block 2820 len 16384 data: 3 4 4 3
25219 119/300 Read block 860 len 12288
25220 120/300 Read block 3170 len 16384
25221 121/300 Read block 3578 len 36864
25222 122/300 Read block 2303 len 32768
25223 123/300 Read block 2555 len 16384
25224 124/300 Flush
25225 125/300 Read block 4461 len 28672
25226 126/300 Write block 2 len 8192 data: 4 6
25227 127/300 Read block 3630 len 16384
25228 128/300 Flush
25229 129/300 Read block 4889 len 8192
25230 130/300 Flush
25231 131/300 Write block 2662 len 20480 data: 7 5 3 3 5
25232 132/300 Write block 504 len 32768 data: 6 5 8 8 7 8 8 10
25233 133/300 Read block 1981 len 12288
25234 134/300 Read block 455 len 40960
25235 135/300 Write block 1409 len 36864 data: 5 5 5 5 6 7 7 6 7
25236 136/300 Write block 406 len 36864 data: 6 5 4 5 4 6 6 6 6
25237 137/300 Read block 126 len 8192
25238 138/300 Write block 3376 len 12288 data: 4 3 3
25239 139/300 Write block 799 len 20480 data: 4 5 5 7 7
25240 140/300 Write block 168 len 28672 data: 6 6 5 5 5 5 5
25241 141/300 Write block 309 len 36864 data: 6 6 8 6 6 6 5 4 3
25242 142/300 Read block 826 len 32768
25243 143/300 Write block 4865 len 12288 data: 7 9 6
25244 144/300 Read block 2750 len 24576
25245 145/300 Read block 3090 len 8192
25246 146/300 Read block 4903 len 28672
25247 147/300 Read block 2983 len 12288
25248 148/300 Write block 3020 len 28672 data: 6 6 4 5 6 8 8
25249 149/300 Read block 594 len 36864
25250 150/300 Flush
25251 151/300 Write block 3754 len 28672 data: 7 6 5 6 6 6 5
25252 152/300 Read block 4312 len 20480
25253 153/300 Read block 1686 len 16384
25254 154/300 Write block 3564 len 4096 data: 4
25255 155/300 Read block 4819 len 4096
25256 156/300 Write block 3431 len 20480 data: 9 9 8 6 6
25257 157/300 Write block 3312 len 12288 data: 4 6 6
25258 158/300 Read block 3067 len 12288
25259 159/300 Read block 3098 len 8192
25260 160/300 Read block 1675 len 4096
25261 161/300 Read block 3022 len 4096
25262 162/300 Write block 3894 len 4096 data: 6
25263 163/300 Read block 760 len 16384
25264 164/300 Flush
25265 165/300 Read block 2123 len 16384
25266 166/300 Read block 2033 len 16384
25267 167/300 Read block 3175 len 12288
25268 168/300 Write block 4101 len 4096 data: 9
25269 169/300 Write block 1502 len 40960 data: 6 7 7 7 10 10 7 7 7 7
25270 170/300 Write block 4877 len 36864 data: 4 3 2 2 2 3 4 5 5
25271 171/300 Write block 3247 len 40960 data: 7 8 7 8 6 6 5 3 3 5
25272 172/300 Write block 2839 len 28672 data: 7 6 8 8 6 6 7
25273 173/300 Write block 772 len 40960 data: 6 5 5 7 6 6 5 4 5 6
25274 174/300 Read block 2979 len 32768
25275 175/300 Write block 3591 len 28672 data: 4 4 3 4 3 3 4
25276 176/300 Write block 124 len 20480 data: 4 3 3 3 4
25277 177/300 Flush
25278 178/300 Read block 1042 len 8192
25279 179/300 Flush
25280 180/300 Read block 4741 len 36864
25281 181/300 Read block 664 len 8192
25282 182/300 Write block 3809 len 8192 data: 8 6
25283 183/300 Flush
25284 184/300 Write block 960 len 36864 data: 4 4 4 5 5 5 5 6 8
25285 185/300 Write block 2628 len 40960 data: 7 5 5 5 4 4 4 4 3 2
25286 186/300 Read block 2679 len 24576
25287 187/300 Flush
25288 188/300 Read block 1254 len 16384
25289 189/300 Read block 1576 len 12288
25290 190/300 Read block 3583 len 28672
25291 191/300 Flush
25292 192/300 Write block 2484 len 32768 data: 7 7 7 8 7 7 9 9
25293 193/300 Write block 3294 len 12288 data: 3 4 6
25294 194/300 Read block 2297 len 16384
25295 195/300 Read block 1099 len 16384
25296 196/300 Write block 75 len 4096 data: 3
25297 197/300 Read block 821 len 36864
25298 198/300 Write block 3652 len 4096 data: 3
25299 199/300 Write block 2020 len 32768 data: 5 5 6 5 5 5 5 5
25300 200/300 Write block 2285 len 40960 data: 7 7 5 8 7 7 6 6 7 8
25301 201/300 Write block 2177 len 20480 data: 6 5 5 5 2
25302 202/300 Read block 2002 len 4096
25303 203/300 Write block 1185 len 12288 data: 8 7 6
25304 204/300 Write block 4133 len 12288 data: 8 5 7
25305 205/300 Read block 3003 len 40960
25306 206/300 Flush
25307 207/300 Write block 4477 len 16384 data: 5 3 4 5
25308 208/300 Write block 4662 len 24576 data: 11 11 10 8 6 6
25309 209/300 Read block 350 len 28672
25310 210/300 Read block 2769 len 40960
25311 211/300 Read block 1586 len 40960
25312 212/300 Flush
25313 213/300 Write block 677 len 12288 data: 6 6 5
25314 214/300 Read block 1458 len 12288
25315 215/300 Write block 1613 len 28672 data: 7 6 6 5 4 5 6
25316 216/300 Read block 4057 len 16384
25317 217/300 Write block 4282 len 12288 data: 7 6 5
25318 218/300 Write block 1530 len 12288 data: 7 9 9
25319 219/300 Write block 206 len 8192 data: 8 7
25320 220/300 Read block 1906 len 28672
25321 221/300 Read block 4335 len 16384
25322 222/300 Write block 2335 len 28672 data: 4 5 6 5 5 6 5
25323 223/300 Read block 4523 len 24576
25324 224/300 Write block 4217 len 36864 data: 8 7 6 6 3 3 3 4 4
25325 225/300 Read block 450 len 24576
25326 226/300 Write block 1705 len 4096 data: 6
25327 227/300 Write block 152 len 40960 data: 12 11 11 9 7 5 4 5 6 4
25328 228/300 Read block 4829 len 32768
25329 229/300 Flush
25330 230/300 Read block 2282 len 8192
25331 231/300 Read block 4313 len 32768
25332 232/300 Read block 4182 len 40960
25333 233/300 Write block 3920 len 8192 data: 3 3
25334 234/300 Read block 140 len 28672
25335 235/300 Read block 623 len 24576
25336 236/300 Read block 1684 len 20480
25337 237/300 Read block 2973 len 28672
25338 238/300 Write block 16 len 36864 data: 6 6 7 7 7 7 5 5 5
25339 239/300 Read block 736 len 20480
25340 240/300 Write block 1013 len 36864 data: 8 9 8 9 9 9 8 6 7
25341 241/300 Read block 3330 len 24576
25342 242/300 Write block 1800 len 36864 data: 5 4 6 5 6 5 4 5 4
25343 243/300 Read block 3559 len 20480
25344 244/300 Flush
25345 245/300 Flush
25346 246/300 Read block 3293 len 12288
25347 247/300 Write block 3225 len 20480 data: 6 5 5 5 4
25348 248/300 Read block 2484 len 12288
25349 249/300 Read block 4512 len 40960
25350 250/300 Read block 4761 len 12288
25351 251/300 Write block 916 len 36864 data: 6 6 7 7 4 5 5 4 5
25352 252/300 Flush
25353 253/300 Read block 3700 len 20480
25354 254/300 Read block 2490 len 20480
25355 255/300 Read block 1066 len 40960
25356 256/300 Read block 3142 len 4096
25357 257/300 Read block 1088 len 16384
25358 258/300 Write block 3688 len 36864 data: 3 3 5 4 4 5 5 5 5
25359 259/300 Read block 3779 len 28672
25360 260/300 Write block 921 len 28672 data: 6 6 5 6 5 6 7
25361 261/300 Read block 1668 len 4096
25362 262/300 Read block 3387 len 12288
25363 263/300 Read block 4563 len 8192
25364 264/300 Write block 542 len 24576 data: 5 4 5 4 6 6
25365 265/300 Read block 1171 len 8192
25366 266/300 Read block 665 len 12288
25367 267/300 Flush
25368 268/300 Read block 3858 len 40960
25369 269/300 Read block 3203 len 28672
25370 270/300 Write block 1962 len 20480 data: 3 3 3 3 3
25371 271/300 Write block 4775 len 36864 data: 6 8 10 11 12 10 9 11 9
25372 272/300 Read block 1568 len 28672
25373 273/300 Write block 4738 len 20480 data: 3 3 2 2 3
25374 274/300 Read block 4143 len 36864
25375 275/300 Read block 3834 len 16384
25376 276/300 Read block 1256 len 40960
25377 277/300 Read block 4187 len 20480
25378 278/300 Write block 1145 len 8192 data: 6 5
25379 279/300 Write block 2658 len 36864 data: 8 8 7 7 8 6 4 4 6
25380 280/300 Flush
25381 281/300 Read block 3884 len 4096
25382 282/300 Read block 1051 len 20480
25383 283/300 Flush
25384 284/300 Read block 3897 len 16384
25385 285/300 Write block 787 len 20480 data: 4 5 4 3 3
25386 286/300 Write block 1879 len 40960 data: 4 4 4 2 3 3 4 5 6 5
25387 287/300 Write block 1071 len 4096 data: 6
25388 288/300 Read block 3363 len 4096
25389 289/300 Read block 3420 len 20480
25390 290/300 Read block 3550 len 28672
25391 291/300 Read block 1089 len 40960
25392 292/300 Write block 4122 len 16384 data: 6 7 7 7
25393 293/300 Read block 1618 len 12288
25394 294/300 Read block 4140 len 4096
25395 295/300 Flush
25396 296/300 Read block 4314 len 20480
25397 297/300 Read block 502 len 36864
25398 298/300 Write block 642 len 12288 data: 9 8 6
25399 299/300 Flush
25400 300/300 Write block 1546 len 40960 data: 5 4 5 4 6 6 6 6 7 7
25401 Main task has work to do, go find it
25402 got dsc Start(1)
25403 start 1
25404 Oct 13 04:47:20.756 INFO request completed, latency_us: 309, response_code: 204, uri: /start/cid/1, method: POST, req_id: 3f77ef86-51fb-4fb6-8cde-049b7153b178, remote_addr: 127.0.0.1:52046, local_addr: 127.0.0.1:9998
25405 Replay: started 1, returned:Ok(())
25406 ----------------------------------------------------------------
25407 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:302
25408 GW_ID ACK DSID TYPE BLOCKS DS:0 DS:1 DS:2 REPLAY
25409 8767 Acked 9766 Read 6 Done New Done false
25410 8768 Acked 9767 Write 5 Done New Done false
25411 8769 Acked 9768 Write 7 Done New Done false
25412 8770 Acked 9769 Flush 0 Done New Done false
25413 8771 Acked 9770 Read 9 Done New Done false
25414 8772 Acked 9771 Read 9 Done New Done false
25415 8773 Acked 9772 Write 10 Done New Done false
25416 8774 Acked 9773 Read 10 Done New Done false
25417 8775 Acked 9774 Write 10 Done New Done false
25418 8776 Acked 9775 Read 8 Done New Done false
25419 8777 Acked 9776 Read 5 Done New Done false
25420 8778 Acked 9777 Read 5 Done New Done false
25421 8779 Acked 9778 Read 1 Done New Done false
25422 8780 Acked 9779 Write 1 Done New Done false
25423 8781 Acked 9780 Read 10 Done New Done false
25424 8782 Acked 9781 Read 9 Done New Done false
25425 8783 Acked 9782 Read 1 Done New Done false
25426 8784 Acked 9783 Read 4 Done New Done false
25427 8785 Acked 9784 Write 2 Done New Done false
25428 8786 Acked 9785 Read 7 Done New Done false
25429 8787 Acked 9786 Read 10 Done New Done false
25430 8788 Acked 9787 Read 1 Done New Done false
25431 8789 Acked 9788 Write 4 Done New Done false
25432 8790 Acked 9789 Write 2 Done New Done false
25433 8791 Acked 9790 Write 3 Done New Done false
25434 8792 Acked 9791 Write 1 Done New Done false
25435 8793 Acked 9792 Flush 0 Done New Done false
25436 8794 Acked 9793 Write 8 Done New Done false
25437 8795 Acked 9794 Write 5 Done New Done false
25438 8796 Acked 9795 Read 6 Done New Done false
25439 8797 Acked 9796 Write 8 Done New Done false
25440 8798 Acked 9797 Write 4 Done New Done false
25441 8799 Acked 9798 Write 3 Done New Done false
25442 8800 Acked 9799 Read 2 Done New Done false
25443 8801 Acked 9800 Write 8 Done New Done false
25444 8802 Acked 9801 Read 9 Done New Done false
25445 8803 Acked 9802 Read 9 Done New Done false
25446 8804 Acked 9803 Read 6 Done New Done false
25447 8805 Acked 9804 Read 4 Done New Done false
25448 8806 Acked 9805 Write 4 Done New Done false
25449 8807 Acked 9806 Read 7 Done New Done false
25450 8808 Acked 9807 Read 10 Done New Done false
25451 8809 Acked 9808 Write 3 Done New Done false
25452 8810 Acked 9809 Flush 0 Done New Done false
25453 8811 Acked 9810 Write 3 Done New Done false
25454 8812 Acked 9811 Write 1 Done New Done false
25455 8813 Acked 9812 Read 10 Done New Done false
25456 8814 Acked 9813 Read 2 Done New Done false
25457 8815 Acked 9814 Write 2 Done New Done false
25458 8816 Acked 9815 Read 1 Done New Done false
25459 8817 Acked 9816 Write 7 Done New Done false
25460 8818 Acked 9817 Write 8 Done New Done false
25461 8819 Acked 9818 Read 7 Done New Done false
25462 8820 Acked 9819 Write 1 Done New Done false
25463 8821 Acked 9820 Read 1 Done New Done false
25464 8822 Acked 9821 Read 3 Done New Done false
25465 8823 Acked 9822 Write 5 Done New Done false
25466 8824 Acked 9823 Write 1 Done New Done false
25467 8825 Acked 9824 Write 4 Done New Done false
25468 8826 Acked 9825 Flush 0 Done New Done false
25469 8827 Acked 9826 Read 3 Done New Done false
25470 8828 Acked 9827 Write 1 Done New Done false
25471 8829 Acked 9828 Read 3 Done New Done false
25472 8830 Acked 9829 Flush 0 Done New Done false
25473 8831 Acked 9830 Read 1 Done New Done false
25474 8832 Acked 9831 Read 9 Done New Done false
25475 8833 Acked 9832 Read 10 Done New Done false
25476 8834 Acked 9833 Read 3 Done New Done false
25477 8835 Acked 9834 Read 9 Done New Done false
25478 8836 Acked 9835 Write 4 Done New Done false
25479 8837 Acked 9836 Write 10 Done New Done false
25480 8838 Acked 9837 Read 8 Done New Done false
25481 8839 Acked 9838 Read 4 Done New Done false
25482 8840 Acked 9839 Write 4 Done New Done false
25483 8841 Acked 9840 Write 3 Done New Done false
25484 8842 Acked 9841 Read 8 Done New Done false
25485 8843 Acked 9842 Write 10 Done New Done false
25486 8844 Acked 9843 Write 7 Done New Done false
25487 8845 Acked 9844 Read 3 Done New Done false
25488 8846 Acked 9845 Read 8 Done New Done false
25489 8847 Acked 9846 Write 1 Done New Done false
25490 8848 Acked 9847 Write 2 Done New Done false
25491 8849 Acked 9848 Read 3 Done New Done false
25492 8850 Acked 9849 Flush 0 Done New Done false
25493 8851 Acked 9850 Read 10 Done New Done false
25494 8852 Acked 9851 Read 6 Done New Done false
25495 8853 Acked 9852 Write 5 Done New Done false
25496 8854 Acked 9853 Write 3 Done New Done false
25497 8855 Acked 9854 Read 7 Done New Done false
25498 8856 Acked 9855 Write 10 Done New Done false
25499 8857 Acked 9856 Read 10 Done New Done false
25500 8858 Acked 9857 Flush 0 Done New Done false
25501 8859 Acked 9858 Read 4 Done New Done false
25502 8860 Acked 9859 Read 4 Done New Done false
25503 8861 Acked 9860 Read 9 Done New Done false
25504 8862 Acked 9861 Read 9 Done New Done false
25505 8863 Acked 9862 Read 7 Done New Done false
25506 8864 Acked 9863 Read 5 Done New Done false
25507 8865 Acked 9864 Read 8 Done New Done false
25508 8866 Acked 9865 Write 5 Done New Done false
25509 8867 Acked 9866 Read 4 Done New Done false
25510 8868 Acked 9867 Write 3 Done New Done false
25511 8869 Acked 9868 Read 6 Done New Done false
25512 8870 Acked 9869 Read 8 Done New Done false
25513 8871 Acked 9870 Read 6 Done New Done false
25514 8872 Acked 9871 Flush 0 Done New Done false
25515 8873 Acked 9872 Read 10 Done New Done false
25516 8874 Acked 9873 Read 8 Done New Done false
25517 8875 Acked 9874 Write 2 Done New Done false
25518 8876 Acked 9875 Read 9 Done New Done false
25519 8877 Acked 9876 Read 5 Done New Done false
25520 8878 Acked 9877 Read 6 Done New Done false
25521 8879 Acked 9878 Read 4 Done New Done false
25522 8880 Acked 9879 Read 9 Done New Done false
25523 8881 Acked 9880 Read 10 Done New Done false
25524 8882 Acked 9881 Write 9 Done New Done false
25525 8883 Acked 9882 Write 6 Done New Done false
25526 8884 Acked 9883 Write 3 Done New Done false
25527 8885 Acked 9884 Write 4 Done New Done false
25528 8886 Acked 9885 Read 3 Done New Done false
25529 8887 Acked 9886 Read 4 Done New Done false
25530 8888 Acked 9887 Read 9 Done New Done false
25531 8889 Acked 9888 Read 8 Done New Done false
25532 8890 Acked 9889 Read 4 Done New Done false
25533 8891 Acked 9890 Flush 0 Done New Done false
25534 8892 Acked 9891 Read 7 Done New Done false
25535 8893 Acked 9892 Write 2 Done New Done false
25536 8894 Acked 9893 Read 4 Done New Done false
25537 8895 Acked 9894 Flush 0 Done New Done false
25538 8896 Acked 9895 Read 2 Done New Done false
25539 8897 Acked 9896 Flush 0 Done New Done false
25540 8898 Acked 9897 Write 5 Done New Done false
25541 8899 Acked 9898 Write 8 Done New Done false
25542 8900 Acked 9899 Read 3 Done New Done false
25543 8901 Acked 9900 Read 10 Done New Done false
25544 8902 Acked 9901 Write 9 Done New Done false
25545 8903 Acked 9902 Write 9 Done New Done false
25546 8904 Acked 9903 Read 2 Done New Done false
25547 8905 Acked 9904 Write 3 Done New Done false
25548 8906 Acked 9905 Write 5 Done New Done false
25549 8907 Acked 9906 Write 7 Done New Done false
25550 8908 Acked 9907 Write 9 Done New Done false
25551 8909 Acked 9908 Read 8 Done New Done false
25552 8910 Acked 9909 Write 3 Done New Done false
25553 8911 Acked 9910 Read 6 Done New Done false
25554 8912 Acked 9911 Read 2 Done New Done false
25555 8913 Acked 9912 Read 7 Done New Done false
25556 8914 Acked 9913 Read 3 Done New Done false
25557 8915 Acked 9914 Write 7 Done New Done false
25558 8916 Acked 9915 Read 9 Done New Done false
25559 8917 Acked 9916 Flush 0 Done New Done false
25560 8918 Acked 9917 Write 7 Done New Done false
25561 8919 Acked 9918 Read 5 Done New Done false
25562 8920 Acked 9919 Read 4 Done New Done false
25563 8921 Acked 9920 Write 1 Done New Done false
25564 8922 Acked 9921 Read 1 Done New Done false
25565 8923 Acked 9922 Write 5 Done New Done false
25566 8924 Acked 9923 Write 3 Done New Done false
25567 8925 Acked 9924 Read 3 Done New Done false
25568 8926 Acked 9925 Read 2 Done New Done false
25569 8927 Acked 9926 Read 1 Done New Done false
25570 8928 Acked 9927 Read 1 Done New Done false
25571 8929 Acked 9928 Write 1 Done New Done false
25572 8930 Acked 9929 Read 4 Done New Done false
25573 8931 Acked 9930 Flush 0 Done New Done false
25574 8932 Acked 9931 Read 4 Done New Done false
25575 8933 Acked 9932 Read 4 Done New Done false
25576 8934 Acked 9933 Read 3 Done New Done false
25577 8935 Acked 9934 Write 1 Done New Done false
25578 8936 Acked 9935 Write 10 Done New Done false
25579 8937 Acked 9936 Write 9 Done New Done false
25580 8938 Acked 9937 Write 10 Done New Done false
25581 8939 Acked 9938 Write 7 Done New Done false
25582 8940 Acked 9939 Write 10 Done New Done false
25583 8941 Acked 9940 Read 8 Done New Done false
25584 8942 Acked 9941 Write 7 Done New Done false
25585 8943 Acked 9942 Write 5 Done New Done false
25586 8944 Acked 9943 Flush 0 Done New Done false
25587 8945 Acked 9944 Read 2 Done New Done false
25588 8946 Acked 9945 Flush 0 Done New Done false
25589 8947 Acked 9946 Read 9 Done New Done false
25590 8948 Acked 9947 Read 2 Done New Done false
25591 8949 Acked 9948 Write 2 Done New Done false
25592 8950 Acked 9949 Flush 0 Done New Done false
25593 8951 Acked 9950 Write 9 Done New Done false
25594 8952 Acked 9951 Write 10 Done New Done false
25595 8953 Acked 9952 Read 6 Done New Done false
25596 8954 Acked 9953 Flush 0 Done New Done false
25597 8955 Acked 9954 Read 4 Done New Done false
25598 8956 Acked 9955 Read 3 Done New Done false
25599 8957 Acked 9956 Read 7 Done New Done false
25600 8958 Acked 9957 Flush 0 Done New Done false
25601 8959 Acked 9958 Write 8 Done New Done false
25602 8960 Acked 9959 Write 3 Done New Done false
25603 8961 Acked 9960 Read 4 Done New Done false
25604 8962 Acked 9961 Read 4 Done New Done false
25605 8963 Acked 9962 Write 1 Done New Done false
25606 8964 Acked 9963 Read 9 Done New Done false
25607 8965 Acked 9964 Write 1 Done New Done false
25608 8966 Acked 9965 Write 8 Done New Done false
25609 8967 Acked 9966 Write 10 Done New Done false
25610 8968 Acked 9967 Write 5 Done New Done false
25611 8969 Acked 9968 Read 1 Done New Done false
25612 8970 Acked 9969 Write 3 Done New Done false
25613 8971 Acked 9970 Write 3 Done New Done false
25614 8972 Acked 9971 Read 10 Done New Done false
25615 8973 Acked 9972 Flush 0 Done New Done false
25616 8974 Acked 9973 Write 4 Done New Done false
25617 8975 Acked 9974 Write 6 Done New Done false
25618 8976 Acked 9975 Read 7 Done New Done false
25619 8977 Acked 9976 Read 10 Done New Done false
25620 8978 Acked 9977 Read 10 Done New Done false
25621 8979 Acked 9978 Flush 0 Done New Done false
25622 8980 Acked 9979 Write 3 Done New Done false
25623 8981 Acked 9980 Read 3 Done New Done false
25624 8982 Acked 9981 Write 7 Done New Done false
25625 8983 Acked 9982 Read 4 Done New Done false
25626 8984 Acked 9983 Write 3 Done New Done false
25627 8985 Acked 9984 Write 3 Done New Done false
25628 8986 Acked 9985 Write 2 Done New Done false
25629 8987 Acked 9986 Read 7 Done New Done false
25630 8988 Acked 9987 Flush 0 Done New Done false
25631 8989 Acked 9988 Read 4 Done New Done false
25632 8990 Acked 9989 Write 7 Done New Done false
25633 8991 Acked 9990 Read 6 Done New Done false
25634 8992 Acked 9991 Write 9 Done New Done false
25635 8993 Acked 9992 Read 6 Done New Done false
25636 8994 Acked 9993 Write 1 Done New Done false
25637 8995 Acked 9994 Write 10 Done New Done false
25638 8996 Acked 9995 Read 8 Done New Done false
25639 8997 Acked 9996 Flush 0 Done New Done false
25640 8998 Acked 9997 Read 2 Done New Done false
25641 8999 Acked 9998 Read 8 Done New Done false
25642 9000 Acked 9999 Read 10 Done New Done false
25643 9001 Acked 10000 Write 2 Done New Done false
25644 9002 Acked 10001 Read 7 Done New Done false
25645 9003 Acked 10002 Read 6 Done New Done false
25646 9004 Acked 10003 Read 5 Done New Done false
25647 9005 Acked 10004 Read 7 Done New Done false
25648 9006 Acked 10005 Write 9 Done New Done false
25649 9007 Acked 10006 Read 5 Done New Done false
25650 9008 Acked 10007 Write 9 Done New Done false
25651 9009 Acked 10008 Read 6 Done New Done false
25652 9010 Acked 10009 Write 9 Done New Done false
25653 9011 Acked 10010 Read 5 Done New Done false
25654 9012 Acked 10011 Flush 0 Done New Done false
25655 9013 Acked 10012 Flush 0 Done New Done false
25656 9014 Acked 10013 Read 3 Done New Done false
25657 9015 Acked 10014 Write 5 Done New Done false
25658 9016 Acked 10015 Read 3 Done New Done false
25659 9017 Acked 10016 Read 10 Done New Done false
25660 9018 Acked 10017 Read 3 Done New Done false
25661 9019 Acked 10018 Write 9 Done New Done false
25662 9020 Acked 10019 Flush 0 Done New Done false
25663 9021 Acked 10020 Read 5 Done New Done false
25664 9022 Acked 10021 Read 5 Done New Done false
25665 9023 Acked 10022 Read 10 Done New Done false
25666 9024 Acked 10023 Read 1 Done New Done false
25667 9025 Acked 10024 Read 4 Done New Done false
25668 9026 Acked 10025 Write 9 Done New Done false
25669 9027 Acked 10026 Read 7 Done New Done false
25670 9028 Acked 10027 Write 7 Done New Done false
25671 9029 Acked 10028 Read 1 Done New Done false
25672 9030 Acked 10029 Read 3 Done New Done false
25673 9031 Acked 10030 Read 2 Done New Done false
25674 9032 Acked 10031 Write 6 Done New Done false
25675 9033 Acked 10032 Read 2 Done New Done false
25676 9034 Acked 10033 Read 3 Done New Done false
25677 9035 Acked 10034 Flush 0 Done New Done false
25678 9036 Acked 10035 Read 10 Done New Done false
25679 9037 Acked 10036 Read 7 Done New Done false
25680 9038 Acked 10037 Write 5 Done New Done false
25681 9039 Acked 10038 Write 9 Done New Done false
25682 9040 Acked 10039 Read 7 Done New Done false
25683 9041 Acked 10040 Write 5 Done New Done false
25684 9042 Acked 10041 Read 9 Done New Done false
25685 9043 Acked 10042 Read 4 Done New Done false
25686 9044 Acked 10043 Read 10 Done New Done false
25687 9045 Acked 10044 Read 5 Done New Done false
25688 9046 Acked 10045 Write 2 Done New Done false
25689 9047 Acked 10046 Write 9 Done New Done false
25690 9048 Acked 10047 Flush 0 Done New Done false
25691 9049 Acked 10048 Read 1 Done New Done false
25692 9050 Acked 10049 Read 5 Done New Done false
25693 9051 Acked 10050 Flush 0 Done New Done false
25694 9052 Acked 10051 Read 4 Done New Done false
25695 9053 Acked 10052 Write 5 Done New Done false
25696 9054 Acked 10053 Write 10 Done New Done false
25697 9055 Acked 10054 Write 1 Done New Done false
25698 9056 Acked 10055 Read 1 Done New Done false
25699 9057 Acked 10056 Read 5 Done New Done false
25700 9058 Acked 10057 Read 7 Done New Done false
25701 9059 Acked 10058 Read 10 Done New Done false
25702 9060 Acked 10059 Write 4 Done New Done false
25703 9061 Acked 10060 Read 3 Done New Done false
25704 9062 Acked 10061 Read 1 Done New Done false
25705 9063 Acked 10062 Flush 0 Done New Done false
25706 9064 Acked 10063 Read 5 Done New Done false
25707 9065 Acked 10064 Read 9 Done New Done false
25708 9066 Acked 10065 Write 3 Done New Done false
25709 9067 Acked 10066 Flush 0 Done New Done false
25710 9068 Acked 10067 Write 10 Sent New Sent false
25711 STATES DS:0 DS:1 DS:2 TOTAL
25712 New 0 302 0 302
25713 Sent 1 0 1 2
25714 Done 301 0 301 602
25715 Skipped 0 0 0 0
25716 Error 0 0 0 0
25717 Last Flush: 10066 9765 10066
25718 Downstairs last five completed: 9765 9764 9763 9762 9761 9760
25719 Upstairs last five completed: 9068 9067 9066 9065 9064 9063
25720 CLIENT: Up:0 ds:302 act:2
25721 [8820] got start action
25722 [8820] I am going to restart
25723 Starting downstairs at port 8820
25724 Make output file at "/tmp/dsc/downstairs-8820.txt"
25725 [8820][1] reports Starting
25726 Downstairs /var/tmp/dsc/region/8820 port 8820 PID:Some(1206)
25727 [8820][1] reports Running
257282023-10-13T04:47:22.829ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b looper connected looper = 1
257292023-10-13T04:47:22.829ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state Offline
257302023-10-13T04:47:22.829ZINFOcrucible: [1] client re-new 303 jobs since flush 9765 = downstairs
257312023-10-13T04:47:22.830ZINFOcrucible: [1] upstairs guest_io_ready=TRUE, promote! session 373d0eb1-b2be-4ec1-8e62-d92421d49954
257322023-10-13T04:47:22.830ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
257332023-10-13T04:47:22.830ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
257342023-10-13T04:47:22.830ZINFOcrucible: Returning client:1 UUID:12345678-0000-0000-0000-000000008820 matches
257352023-10-13T04:47:22.830ZINFOcrucible: [1] send last flush ID to this DS: 9765
257362023-10-13T04:47:22.830ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Offline Active
257372023-10-13T04:47:22.830ZINFOcrucible: [1] Replied this last flush ID: 9765
257382023-10-13T04:47:22.831ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Offline Active ds_transition to Replay
257392023-10-13T04:47:22.831ZINFOcrucible: [1] Transition from Offline to Replay
257402023-10-13T04:47:22.831ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Transition from Replay to Active
257412023-10-13T04:47:22.831ZINFOcrucible: [1] 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b (373d0eb1-b2be-4ec1-8e62-d92421d49954) Active Replay Active ds_transition to Active
257422023-10-13T04:47:22.831ZINFOcrucible: [1] Transition from Replay to Active
257432023-10-13T04:47:22.831ZINFOcrucible: [1] Starts cmd_loop
257442023-10-13T04:47:22.831ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
257452023-10-13T04:47:22.831ZINFOcrucible: 44a0c453-5787-4dc8-8a4a-5665d0ed9f9b Active Active Active
257462023-10-13T04:47:22.831ZINFOcrucible: Set check for repair
257472023-10-13T04:47:23.832ZWARNcrucible: [1] flow control end
257482023-10-13T04:47:23.833ZINFOcrucible: Checking if live repair is needed
257492023-10-13T04:47:23.833ZINFOcrucible: No Live Repair required at this time
25750 ----------------------------------------------------------------
25751 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25752 Downstairs last five completed: 10068 10067 10066 10065 10064 10063
25753 Upstairs last five completed: 9069 9068 9067 9066 9065 9064
25754 CLIENT: Up:0 ds:0 act:3
25755 Replay: All jobs finished, all DS active.
25756 Test replay has completed
25757 Wrote out file "alan"
25758 CLIENT: Tests done. All submitted work has been ACK'd
25759 ----------------------------------------------------------------
25760 Crucible gen:2 GIO:true work queues: Upstairs:0 downstairs:0
25761 Downstairs last five completed: 10068 10067 10066 10065 10064 10063
25762 Upstairs last five completed: 9069 9068 9067 9066 9065 9064
25763 CLIENT: Up:0 ds:0 act:3
25764 CLIENT: All jobs finished, all DS active.
25765 Replays:30 time: 5:35
25766 Do final verify
257672023-10-13T04:47:24.806ZINFOcrucible: Upstairs starts
257682023-10-13T04:47:24.807ZINFOcrucible: Crucible Version: BuildInfo { version: "0.0.1", git_sha: "0a6b3facda3a9a8c75c70b591db2d34571e435e8", git_commit_timestamp: "2023-10-13T04:02:07.000000000Z", git_branch: "alan/unwrapless", rustc_semver: "1.70.0", rustc_channel: "stable", rustc_host_triple: "x86_64-unknown-illumos", rustc_commit_sha: "90c541806f23a127002de5b4038be731ba1458ca", cargo_triple: "x86_64-unknown-illumos", debug: true, opt_level: 0, }
257692023-10-13T04:47:24.807ZINFOcrucible: Upstairs <-> Downstairs Message Version: 4
257702023-10-13T04:47:24.807ZINFOcrucible: Upstairs opts: Upstairs UUID: a0d7c876-6432-4606-819c-cd0a312e8c7e, Targets: [127.0.0.1:8810, 127.0.0.1:8820, 127.0.0.1:8830], lossy: false, flush_timeout: None, key populated: false, cert_pem populated: false, key_pem populated: false, root_cert_pem populated: false, Control: None, read_only: false
257712023-10-13T04:47:24.807ZINFOcrucible: Crucible stats registered with UUID: a0d7c876-6432-4606-819c-cd0a312e8c7e
257722023-10-13T04:47:24.807ZINFOcrucible: Crucible a0d7c876-6432-4606-819c-cd0a312e8c7e has session id: d221e741-8682-4a50-aa7c-9d1cde229dac
25773 Crucible runtime is spawned
25774 The guest has requested activation with gen:3
257752023-10-13T04:47:24.807ZINFOcrucible: up_listen starts task = up_listen
257762023-10-13T04:47:24.807ZINFOcrucible: Wait for all three downstairs to come online
257772023-10-13T04:47:24.808ZINFOcrucible: Flush timeout: 0.5
257782023-10-13T04:47:24.808ZINFOcrucible: [2] connecting to 127.0.0.1:8830 looper = 2
257792023-10-13T04:47:24.808ZINFOcrucible: [1] connecting to 127.0.0.1:8820 looper = 1
257802023-10-13T04:47:24.808ZINFOcrucible: [2] a0d7c876-6432-4606-819c-cd0a312e8c7e looper connected looper = 2
257812023-10-13T04:47:24.808ZINFOcrucible: [0] connecting to 127.0.0.1:8810 looper = 0
257822023-10-13T04:47:24.808ZINFOcrucible: [1] a0d7c876-6432-4606-819c-cd0a312e8c7e looper connected looper = 1
257832023-10-13T04:47:24.808ZINFOcrucible: [0] a0d7c876-6432-4606-819c-cd0a312e8c7e looper connected looper = 0
257842023-10-13T04:47:24.809ZINFOcrucible: a0d7c876-6432-4606-819c-cd0a312e8c7e active request set
257852023-10-13T04:47:24.809ZINFOcrucible: Set desired generation to :3
257862023-10-13T04:47:24.809ZINFOcrucible: [2] Proc runs for 127.0.0.1:8830 in state New
257872023-10-13T04:47:24.809ZINFOcrucible: [1] Proc runs for 127.0.0.1:8820 in state New
257882023-10-13T04:47:24.809ZINFOcrucible: [0] Proc runs for 127.0.0.1:8810 in state New
257892023-10-13T04:47:24.809ZINFOcrucible: [2] a0d7c876-6432-4606-819c-cd0a312e8c7e (8cfb70a4-b24e-4c8a-9976-64e5de8e6692) New New New ds_transition to WaitActive
257902023-10-13T04:47:24.809ZINFOcrucible: [2] Transition from New to WaitActive
257912023-10-13T04:47:24.809ZINFOcrucible: [2] client is_active_req TRUE, promote! session 8cfb70a4-b24e-4c8a-9976-64e5de8e6692
257922023-10-13T04:47:24.809ZINFOcrucible: [1] a0d7c876-6432-4606-819c-cd0a312e8c7e (8cfb70a4-b24e-4c8a-9976-64e5de8e6692) New New WaitActive ds_transition to WaitActive
257932023-10-13T04:47:24.809ZINFOcrucible: [1] Transition from New to WaitActive
257942023-10-13T04:47:24.809ZINFOcrucible: [1] client is_active_req TRUE, promote! session 8cfb70a4-b24e-4c8a-9976-64e5de8e6692
257952023-10-13T04:47:24.810ZINFOcrucible: [0] a0d7c876-6432-4606-819c-cd0a312e8c7e (8cfb70a4-b24e-4c8a-9976-64e5de8e6692) New WaitActive WaitActive ds_transition to WaitActive
257962023-10-13T04:47:24.810ZINFOcrucible: [0] Transition from New to WaitActive
257972023-10-13T04:47:24.810ZINFOcrucible: [0] client is_active_req TRUE, promote! session 8cfb70a4-b24e-4c8a-9976-64e5de8e6692
257982023-10-13T04:47:24.810ZINFOcrucible: [1] downstairs client at 127.0.0.1:8820 has UUID 12345678-0000-0000-0000-000000008820
257992023-10-13T04:47:24.810ZINFOcrucible: [2] downstairs client at 127.0.0.1:8830 has UUID 12345678-0000-0000-0000-000000008830
258002023-10-13T04:47:24.810ZINFOcrucible: [1] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008820, encrypted: false, database_read_version: 1, database_write_version: 1 }
258012023-10-13T04:47:24.810ZINFOcrucible: [2] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008830, encrypted: false, database_read_version: 1, database_write_version: 1 }
258022023-10-13T04:47:24.810ZINFOcrucible: [0] downstairs client at 127.0.0.1:8810 has UUID 12345678-0000-0000-0000-000000008810
258032023-10-13T04:47:24.810ZINFOcrucible: [0] Got region def RegionDefinition { block_size: 4096, extent_size: Block { value: 100, shift: 12 }, extent_count: 50, uuid: 12345678-0000-0000-0000-000000008810, encrypted: false, database_read_version: 1, database_write_version: 1 }
258042023-10-13T04:47:24.810ZINFOcrucible: a0d7c876-6432-4606-819c-cd0a312e8c7e WaitActive WaitActive WaitActive
258052023-10-13T04:47:24.810ZINFOcrucible: a0d7c876-6432-4606-819c-cd0a312e8c7e WaitActive WaitActive WaitActive
258062023-10-13T04:47:24.811ZINFOcrucible: a0d7c876-6432-4606-819c-cd0a312e8c7e WaitActive WaitActive WaitActive
258072023-10-13T04:47:24.813ZINFOcrucible: [1] a0d7c876-6432-4606-819c-cd0a312e8c7e (8cfb70a4-b24e-4c8a-9976-64e5de8e6692) WaitActive WaitActive WaitActive ds_transition to WaitQuorum
258082023-10-13T04:47:24.814ZINFOcrucible: [1] Transition from WaitActive to WaitQuorum
258092023-10-13T04:47:24.814ZWARNcrucible: [1] new RM replaced this: None
258102023-10-13T04:47:24.814ZINFOcrucible: [1] 127.0.0.1:8820 task reports connection:true
258112023-10-13T04:47:24.814ZINFOcrucible: [2] a0d7c876-6432-4606-819c-cd0a312e8c7e (8cfb70a4-b24e-4c8a-9976-64e5de8e6692) WaitActive WaitQuorum WaitActive ds_transition to WaitQuorum
258122023-10-13T04:47:24.814ZINFOcrucible: [2] Transition from WaitActive to WaitQuorum
258132023-10-13T04:47:24.814ZWARNcrucible: [2] new RM replaced this: None
258142023-10-13T04:47:24.814ZINFOcrucible: a0d7c876-6432-4606-819c-cd0a312e8c7e WaitActive WaitQuorum WaitQuorum
258152023-10-13T04:47:24.814ZINFOcrucible: [0] a0d7c876-6432-4606-819c-cd0a312e8c7e (8cfb70a4-b24e-4c8a-9976-64e5de8e6692) WaitActive WaitQuorum WaitQuorum ds_transition to WaitQuorum
258162023-10-13T04:47:24.814ZINFOcrucible: [0] Transition from WaitActive to WaitQuorum
258172023-10-13T04:47:24.814ZWARNcrucible: [0] new RM replaced this: None
258182023-10-13T04:47:24.814ZINFOcrucible: [1] Starts reconcile loop
258192023-10-13T04:47:24.814ZINFOcrucible: [2] Starts reconcile loop
258202023-10-13T04:47:24.814ZINFOcrucible: [0]R flush_numbers[0..12]: [978, 977, 976, 967, 967, 981, 985, 984, 967, 981, 984, 982]
258212023-10-13T04:47:24.814ZINFOcrucible: [0]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]
258222023-10-13T04:47:24.814ZINFOcrucible: [0]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]
258232023-10-13T04:47:24.814ZINFOcrucible: [1]R flush_numbers[0..12]: [978, 977, 976, 967, 967, 981, 985, 984, 967, 981, 984, 982]
258242023-10-13T04:47:24.814ZINFOcrucible: [1]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]
258252023-10-13T04:47:24.814ZINFOcrucible: [1]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]
258262023-10-13T04:47:24.814ZINFOcrucible: [2]R flush_numbers[0..12]: [978, 977, 976, 967, 967, 981, 985, 984, 967, 981, 984, 982]
258272023-10-13T04:47:24.814ZINFOcrucible: [2]R generation[0..12]: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2]
258282023-10-13T04:47:24.814ZINFOcrucible: [2]R dirty[0..12]: [false, false, false, false, false, false, false, false, false, false, false, false]
258292023-10-13T04:47:24.814ZINFOcrucible: Max found gen is 3
25830 {"msg":"Generation requested: 3 >= found:3","v":0,"name":"crucible","level":30,"time":"2023-10-13T04:47:24.814883933ZThe guest has finished waiting for activation with:3
25831 ","Wait for a query_work_queue command to finish before sending IO
25832 hostname":"ip-10-150-1-57.us-west-2.compute.internal","pid":1211}
258332023-10-13T04:47:24.814ZINFOcrucible: Next flush: 987
258342023-10-13T04:47:24.814ZINFOcrucible: All extents match
258352023-10-13T04:47:24.814ZINFOcrucible: No downstairs repair required
258362023-10-13T04:47:24.814ZINFOcrucible: [0] Starts reconcile loop
258372023-10-13T04:47:24.814ZINFOcrucible: No initial repair work was required
258382023-10-13T04:47:24.814ZINFOcrucible: Set Downstairs and Upstairs active
258392023-10-13T04:47:24.815ZINFOcrucible: a0d7c876-6432-4606-819c-cd0a312e8c7e is now active with session: 8cfb70a4-b24e-4c8a-9976-64e5de8e6692
258402023-10-13T04:47:24.815ZINFOcrucible: a0d7c876-6432-4606-819c-cd0a312e8c7e Set Active after no repair
258412023-10-13T04:47:24.815ZINFOcrucible: Notify all downstairs, region set compare is done.
258422023-10-13T04:47:24.815ZINFOcrucible: Set check for repair
258432023-10-13T04:47:24.815ZINFOcrucible: [2] 127.0.0.1:8830 task reports connection:true
258442023-10-13T04:47:24.815ZINFOcrucible: a0d7c876-6432-4606-819c-cd0a312e8c7e Active Active Active
258452023-10-13T04:47:24.815ZINFOcrucible: Set check for repair
258462023-10-13T04:47:24.815ZINFOcrucible: [0] received reconcile message
258472023-10-13T04:47:24.815ZINFOcrucible: [0] All repairs completed, exit
258482023-10-13T04:47:24.815ZINFOcrucible: [1] received reconcile message
258492023-10-13T04:47:24.815ZINFOcrucible: [0] Starts cmd_loop
258502023-10-13T04:47:24.815ZINFOcrucible: [2] received reconcile message
258512023-10-13T04:47:24.815ZINFOcrucible: [0] 127.0.0.1:8810 task reports connection:true
258522023-10-13T04:47:24.815ZINFOcrucible: [1] All repairs completed, exit
258532023-10-13T04:47:24.815ZINFOcrucible: [1] Starts cmd_loop
258542023-10-13T04:47:24.815ZINFOcrucible: [2] All repairs completed, exit
258552023-10-13T04:47:24.815ZINFOcrucible: [2] Starts cmd_loop
258562023-10-13T04:47:24.815ZINFOcrucible: a0d7c876-6432-4606-819c-cd0a312e8c7e Active Active Active
258572023-10-13T04:47:24.815ZINFOcrucible: Set check for repair
25858 Region: es:100 ec:50 bs:4096 ts:20480000 tb:5000 max_io:256 or 1048576
25859 Loading write count information from file "alan"
25860 Read and Verify all blocks (0..5000 range:false)
258612023-10-13T04:47:25.820ZINFOcrucible: Checking if live repair is needed
258622023-10-13T04:47:25.840ZINFOcrucible: No Live Repair required at this time
25863 Wrote out file "alan"
25864 Verify test completed
25865 Wrote out file "alan"
25866 CLIENT: Tests done. All submitted work has been ACK'd
25867 ----------------------------------------------------------------
25868 Crucible gen:3 GIO:true work queues: Upstairs:0 downstairs:0
25869 Downstairs last five completed: 1053 1052 1051 1050 1049 1048
25870 Upstairs last five completed: 54 53 52 51 50 49
25871 CLIENT: Up:0 ds:0 act:3
25872 CLIENT: All crucible jobs finished, exiting program
25873 Oct 13 04:47:26.997 INFO accepted connection, remote_addr: 127.0.0.1:64742, local_addr: 127.0.0.1:9998
25874 Main task has work to do, go find it
25875 got dsc Shutdown
25876 Shutdown
25877 Shut it down
25878 Oct 13 04:47:26.998 INFO request completed, latency_us: 246, response_code: 204, uri:[8810] Disable keep_running
25879 /shutdown[8810] Got stop action so:false kr:false
25880 , method: POST, req_id: f3746899-07b9-4747-8ee2-f679db499a4a, remote_addr: 127.0.0.1:64742[8820] Disable keep_running
25881 , local_addr:[8820] Got stop action so:false kr:false
25882 127.0.0.1:9998
25883 [8830] Disable keep_running
25884 [8830] Got stop action so:false kr:false
25885 [8810] Exited with: ExitStatus(unix_wait_status(9))
25886 [8810][0] reports Exit
25887 [8820] Exited with: ExitStatus(unix_wait_status(9))
25888 [8820][1] reports Exit
25889 [8830] Exited with: ExitStatus(unix_wait_status(9))
25890 [8830][2] reports Exit
25891 October 13, 2023 at 04:47:32 AM UTC Test ends with 0